Welcome to MilkyWay@home

bad wus in the database

Message boards : Number crunching : bad wus in the database
Message board moderation

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
Profile niterobin

Send message
Joined: 11 Mar 08
Posts: 28
Credit: 818,194
RAC: 0
Message 3883 - Posted: 19 Jun 2008, 19:45:57 UTC

I have work units starting with 3732, 606 and 607 here and everything seems to be fine.

Thank you for the work you've put in to get everything running smoothly again.
ID: 3883 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Aug 07
Posts: 2046
Credit: 26,480
RAC: 0
Message 3885 - Posted: 19 Jun 2008, 20:02:58 UTC - in response to Message 3880.  

... that when the purge starts, it's so expensive that it slows down the rest of the system and can't catch up, while pretty much makes everything slow and unresponsive. so to get the purge done we have to stop the assimilator and work generation.

Can't you automatically delete any WU that's been done for 24 hours or more? So rather than doing a full purge every 24 hours you'd be purging WUs continuously at roughly the same rate that they're coming in..

Perhaps keep a log of completed WUs (sorted by reported completion time by definition), and purge old (>24h) entries whenever you add a new one (i.e. a WU is reported)

if(thisWU was completed more than 24 hours ago) delete and examine next item;
else exit;

To put it in pseudo-code <.< (had a brain-fart, so that was the best I could do)


i'll try and give this a shot and see if the server gets overloaded again. a day of WUs (at ~6 WUs a second) is quite a lot... which is part of the problem
ID: 3885 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Stefan Ver3

Send message
Joined: 17 May 08
Posts: 16
Credit: 528,507
RAC: 0
Message 3887 - Posted: 19 Jun 2008, 20:52:05 UTC - in response to Message 3885.  

... that when the purge starts, it's so expensive that it slows down the rest of the system and can't catch up, while pretty much makes everything slow and unresponsive. so to get the purge done we have to stop the assimilator and work generation.

Can't you automatically delete any WU that's been done for 24 hours or more? So rather than doing a full purge every 24 hours you'd be purging WUs continuously at roughly the same rate that they're coming in..

Perhaps keep a log of completed WUs (sorted by reported completion time by definition), and purge old (>24h) entries whenever you add a new one (i.e. a WU is reported)

if(thisWU was completed more than 24 hours ago) delete and examine next item;
else exit;

To put it in pseudo-code <.< (had a brain-fart, so that was the best I could do)


i'll try and give this a shot and see if the server gets overloaded again. a day of WUs (at ~6 WUs a second) is quite a lot... which is part of the problem


Are you sure you want to do that just yet Travis? It's working really good right now. I'd sleep on it for the night, and celebrate Pint Night at the Ruck. :D :D :D :D
ID: 3887 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Emanuel

Send message
Joined: 18 Nov 07
Posts: 280
Credit: 2,442,757
RAC: 0
Message 3894 - Posted: 19 Jun 2008, 22:36:46 UTC - in response to Message 3885.  

i'll try and give this a shot and see if the server gets overloaded again. a day of WUs (at ~6 WUs a second) is quite a lot... which is part of the problem

As long as you make a variable out of the, er.. completed WU retention time, you should just be able to start with something relatively short like an hour, then increase it if you think it can take more. Like Stefan Ver3 said, take it slow! We're all enjoying the current flow of WUs :)
ID: 3894 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Odd-Rod

Send message
Joined: 7 Sep 07
Posts: 444
Credit: 5,712,451
RAC: 0
Message 3900 - Posted: 20 Jun 2008, 7:02:23 UTC - in response to Message 3894.  

i'll try and give this a shot and see if the server gets overloaded again. a day of WUs (at ~6 WUs a second) is quite a lot... which is part of the problem

As long as you make a variable out of the, er.. completed WU retention time, you should just be able to start with something relatively short like an hour, then increase it if you think it can take more. Like Stefan Ver3 said, take it slow! We're all enjoying the current flow of WUs :)


I agree. Do we need 24 hours? A few hours would be nice for those who want to check. But what's the point of being able to check if there are no WUs to check? ;)
ID: 3900 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Aug 07
Posts: 2046
Credit: 26,480
RAC: 0
Message 3901 - Posted: 20 Jun 2008, 8:13:37 UTC - in response to Message 3900.  

i'll try and give this a shot and see if the server gets overloaded again. a day of WUs (at ~6 WUs a second) is quite a lot... which is part of the problem

As long as you make a variable out of the, er.. completed WU retention time, you should just be able to start with something relatively short like an hour, then increase it if you think it can take more. Like Stefan Ver3 said, take it slow! We're all enjoying the current flow of WUs :)


I agree. Do we need 24 hours? A few hours would be nice for those who want to check. But what's the point of being able to check if there are no WUs to check? ;)


right now the purge function has it built only for days. if 1 day starts to make the server run slowly, i'll recode it for 3 or 4. right now things seem to be working fine so i'm hoping everything will be ok and WUs will be retained for a day.
ID: 3901 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 · 2

Message boards : Number crunching : bad wus in the database

©2024 Astroinformatics Group