Welcome to MilkyWay@home

Clearing results from database quickly?


Advanced search

Message boards : Number crunching : Clearing results from database quickly?
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
ProfileWang Solutions
Avatar

Send message
Joined: 22 Dec 07
Posts: 13
Credit: 46,606,530
RAC: 0
30 million credit badge10 year member badge
Message 4208 - Posted: 17 Jul 2008, 11:25:12 UTC

Since the change to the longer units I have been noticing a few things that are a little strange. Firstly, in my results I am seeing a lot of tasks supposedly not returned by deadline, but I have not seen any evidence of this actually happening on my machines. Indeed, one machine supposedly had 17 results not returned by deadline, but it had been set to no new work for several weeks and hasn't had anything to return! The others have all been going through work steadily and returning promptly, yet all are shown as having work not returned by deadline. I cannot work it out.

Secondly, are you removing finished results from the database very quickly now? I just completed a few tasks, and within 5 minutes went to see the credit that they received but could find no trace of ANY completed tasks in my results.

How long are you planning to leave the results in the database?
Proud member of BOINC@AUSTRALIA
ID: 4208 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileConan
Avatar

Send message
Joined: 2 Jan 08
Posts: 105
Credit: 65,396,973
RAC: 0
50 million credit badge10 year member badge
Message 4210 - Posted: 17 Jul 2008, 13:13:17 UTC - in response to Message 4208.  

Since the change to the longer units I have been noticing a few things that are a little strange. Firstly, in my results I am seeing a lot of tasks supposedly not returned by deadline, but I have not seen any evidence of this actually happening on my machines. Indeed, one machine supposedly had 17 results not returned by deadline, but it had been set to no new work for several weeks and hasn't had anything to return! The others have all been going through work steadily and returning promptly, yet all are shown as having work not returned by deadline. I cannot work it out.

Secondly, are you removing finished results from the database very quickly now? I just completed a few tasks, and within 5 minutes went to see the credit that they received but could find no trace of ANY completed tasks in my results.

How long are you planning to leave the results in the database?


G'Day Wang Solutions,
From what I have seen and timed, you will get about 5 minutes from the time you return your result till they are purged from the database.
So you will have to be quick.
I have not seen evidence of the first part of the problem you have reported with any of my computers, no results that are reporting past deadline.
ID: 4210 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Irishgeezah
Avatar

Send message
Joined: 10 Nov 07
Posts: 37
Credit: 11,443,363
RAC: 0
10 million credit badge10 year member badge
Message 4218 - Posted: 17 Jul 2008, 18:58:13 UTC

I'd like to see the results being available for more than a few minutes too, like many here, i run several projects and not being able to analyse the results to make the best use of my systems by assigning them to more appropriate projects is a bit of a headache at the moment.......no point having machines crunching when they'll return the work after the deadline is up!!
Surely there's a way to archive the results for more than a few minutes.


ID: 4218 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Keith T.

Send message
Joined: 18 Jun 08
Posts: 5
Credit: 20,484
RAC: 0
10 thousand credit badge10 year member badge
Message 4220 - Posted: 17 Jul 2008, 19:25:18 UTC - in response to Message 4218.  

I'd like to see the results being available for more than a few minutes too, like many here, i run several projects and not being able to analyse the results to make the best use of my systems by assigning them to more appropriate projects is a bit of a headache at the moment.......no point having machines crunching when they'll return the work after the deadline is up!!
Surely there's a way to archive the results for more than a few minutes.


BoincView and BoincLogX can both log results.

Stats sites like BOINCstats can also give you data about your recent results about 24 hr after they were returned.
ID: 4220 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Alinator

Send message
Joined: 7 Jun 08
Posts: 464
Credit: 56,639,936
RAC: 0
50 million credit badge10 year member badge
Message 4227 - Posted: 17 Jul 2008, 19:56:15 UTC
Last modified: 17 Jul 2008, 19:57:29 UTC

The only problem is you don't get the Granted Credit for the tasks, be able to review the stderr data (if any and may not apply to BoincLogX.), and have no simple way to relate what's in the logs to what is shown on the Host Summary.

Alinator
ID: 4227 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileJayargh
Avatar

Send message
Joined: 8 Oct 07
Posts: 289
Credit: 3,690,838
RAC: 0
3 million credit badge10 year member badge
Message 4228 - Posted: 17 Jul 2008, 21:02:18 UTC

I think when Travis is back he will change the purge time....the only reason it is so fast now was the load on the server when they were short.....Travis is the only person with permissions to change this and other parameters so we are just waiting for him to get back and adjust the server to the new task length ;)
ID: 4228 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileWang Solutions
Avatar

Send message
Joined: 22 Dec 07
Posts: 13
Credit: 46,606,530
RAC: 0
30 million credit badge10 year member badge
Message 4232 - Posted: 18 Jul 2008, 11:29:36 UTC

I have now had time to look into the first problem a little more, and I still cannot explain it. According to the host record here:

http://milkyway.cs.rpi.edu/milkyway/results.php?hostid=11188

I still have 5 tasks in cache, with the other ones listed having not been returned by deadline. But this PC actually has NO Milky Way work on it currently, and has not done so for several days.

I suspect that the database might have been corrupted and that these tasks are actually from some other host, or have in fact been completed and returned by this host but not recognised for some reason (or were listed twice).

Does anyone have any other thoughts?
Proud member of BOINC@AUSTRALIA
ID: 4232 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Alinator

Send message
Joined: 7 Jun 08
Posts: 464
Credit: 56,639,936
RAC: 0
50 million credit badge10 year member badge
Message 4236 - Posted: 18 Jul 2008, 15:46:44 UTC

Most likely they are just 'ghosts'.

Ghosts are tasks which the project thinks it sent to you, but the scheduler reply which would have told your host about them never makes it back to the host.

Given the severe overload the backend was experiencing recently, it's not too surprising that some tasks ended up getting ghosted.

In any event, they are pretty much harmless and there isn't very much you can do about them yourself anyway. Usually the best thing to do is just ignore them and let them timeout on deadline.

HTH,

Alinator

ID: 4236 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileWang Solutions
Avatar

Send message
Joined: 22 Dec 07
Posts: 13
Credit: 46,606,530
RAC: 0
30 million credit badge10 year member badge
Message 4240 - Posted: 19 Jul 2008, 1:04:53 UTC

Thanks, I was not really worried about it from my point of view, but just wondering if anyone else was experiencing it and if in fact it needed to be checked on for reasons of data integrity.
Proud member of BOINC@AUSTRALIA
ID: 4240 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Peter-B

Send message
Joined: 14 Jul 08
Posts: 5
Credit: 44,130,280
RAC: 0
30 million credit badge10 year member badge
Message 4246 - Posted: 19 Jul 2008, 13:55:40 UTC

Hi, I'm new to Milky Way.
I also find it peculiar that the stats for each WU are removed so quickly.
Why can't they be removed after24/48hrs?

ID: 4246 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile[BAT] tutta55
Avatar

Send message
Joined: 27 Aug 07
Posts: 46
Credit: 8,529,766
RAC: 0
5 million credit badge10 year member badge
Message 4636 - Posted: 6 Aug 2008, 13:36:35 UTC

*burp* ... erm, I mean *bump*

BOINC.BE: For Belgians who love the smell of glowing red cpu's in the morning
Tutta55's Lair
ID: 4636 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Dr Who Fan
Avatar

Send message
Joined: 8 Aug 08
Posts: 21
Credit: 330,700
RAC: 0
100 thousand credit badge10 year member badge
Message 4657 - Posted: 9 Aug 2008, 2:54:19 UTC

I also noticed completed work disappearing from the work unit list in less than 5 min... As others have said would be nice to have them be available for 24/48 hrs.

ID: 4657 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileThunder
Avatar

Send message
Joined: 9 Jul 08
Posts: 85
Credit: 44,842,651
RAC: 0
30 million credit badge10 year member badge
Message 4658 - Posted: 9 Aug 2008, 4:02:38 UTC
Last modified: 9 Aug 2008, 4:07:25 UTC

Okay, I've held off on commenting about this and a few other subjects because I figured that eventually Travis would come along, read and take care of the many outstanding issues, but...

This is getting completely rediculous. It's great that the server can seemingly run without intervention now, but good grief! Travis, we've not seen a single post addressing anything for over a MONTH now. :-P

The database is only keeping 14k-15k tasks in progress at a time right now which is a tiny, miniscule fraction of what it was before. The odds of keeping results in the database for a bit longer affecting the operation of the project at this point is virtually nil. The setting to increase the time to keep workunits in the database before purging is annoyingly simple... it would probably take you 1 minute to make the change.

If the whole project is going to take a sabbatical for a month, the least you can do is grant your volunteers the common courtesy of spending 2 minutes to make a post saying so. Otherwise, please, PLEASE get off your rear-ends and demonstrate to us that you still have a pulse and can fog a mirror. 5+ years of volunteering for BOINC projects has shown me repeatedly that NOTHING will drive volunteers away from a BOINC project faster than a sense that the project staff isn't responsive and it's bloody OBVIOUS from the posts lately that this is what your volunteers are feeling.

[edit]Sorry if this post sounds a bit overly harsh, but I'm sincerely getting sick of the projects I volunteer for just mysteriously curling up and dying lately. It's really starting to grate on my nerves. It's not just MW, so my fuse is being shortened by more than just this project. [/edit]
ID: 4658 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Odd-Rod

Send message
Joined: 7 Sep 07
Posts: 442
Credit: 1,404,502
RAC: 63
1 million credit badge10 year member badge
Message 4659 - Posted: 9 Aug 2008, 5:34:09 UTC - in response to Message 4246.  

Hi, I'm new to Milky Way.
I also find it peculiar that the stats for each WU are removed so quickly.
Why can't they be removed after24/48hrs?


Hi and welcome, but to what I'm not sure now ;)

A brief history of what happenend before you joined.
This project use to have very short WUs - my 3GHz P4 completed them in about 10m33s and another host of mine in around 7min. These short WUs meant that the database had many (many!) entries and this caused problems. I think they were related to the purging process. (Sorry, don't know the programming side, but the forums will have it all). In an effort to to sort this out, settings were changed to what we have now - I think the WUs remain for a maximum of 5min.
Since then, the WU duration was increased, initially by about 60 times, and soon afterwards by about 30 times the original duration. My P4: 10h17m and 5h32m.

Now at this point the database purging could have been returned to its old setting, but at the same time the admins have disappeared.

So, the real problem here is the lack of admins, which is a pity as this project had very responsive admins until that moment.

Hoping this sheds some light on the situation, and that people realize that there's not much point in complaining about the problems here until there someone to respond to it.

One thing's for sure, no response will lose a lot of crunchers. I'll stay while the project works, in the hope for a return of the admins. (Good movie title that: "Return of the Admins") :)

Rod
ID: 4659 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileArion
Avatar

Send message
Joined: 10 Aug 08
Posts: 218
Credit: 41,846,854
RAC: 0
30 million credit badge10 year member badge
Message 4667 - Posted: 11 Aug 2008, 7:49:10 UTC - in response to Message 4659.  
Last modified: 11 Aug 2008, 7:58:50 UTC

So, the real problem here is the lack of admins, which is a pity as this project had very responsive admins until that moment.

Hoping this sheds some light on the situation, and that people realize that there's not much point in complaining about the problems here until there someone to respond to it.

One thing's for sure, no response will lose a lot of crunchers. I'll stay while the project works, in the hope for a return of the admins. (Good movie title that: "Return of the Admins") :)

Rod


I guess a lot of us are out and about looking for either a new primary or secondary project to crunch on with all the change overs/updates on equipment, server versions, kredit kops, etc. Is this a normal policy with the admins here to just go missing? Whose minding the store when the servers go down, etc.?

[edit] I don't know why I bothered to post this even after looking through the messages for an hour. So... Does this mean that we are part of the experiment titled "How to Attract and Lose Participants!"? One more thing, I've seen a lot of information about the computing aspects of the project. Is there any publications or anything to show what's actually been done along the astronomy aspect yet?
[/edit]
ID: 4667 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
STE\/E

Send message
Joined: 29 Aug 07
Posts: 486
Credit: 573,905,473
RAC: 1,841
500 million credit badge10 year member badge
Message 4668 - Posted: 11 Aug 2008, 9:50:59 UTC

Is this a normal policy with the admins here to just go missing? Whose minding the store when the servers go down, etc.?


It's quite normal @ a lot of the projects once they've been running for awhile, especially during the summer month's, maybe this fall things will pick up as far as communications from the Dev's goes, but who knows ... ????
ID: 4668 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Odd-Rod

Send message
Joined: 7 Sep 07
Posts: 442
Credit: 1,404,502
RAC: 63
1 million credit badge10 year member badge
Message 4676 - Posted: 11 Aug 2008, 16:43:05 UTC - in response to Message 4667.  

Is this a normal policy with the admins here to just go missing?

Not at this project. This project was one of the nicest projects with regular responses from the admins. This strange disappearance has really got me puzzled!

Oh well, lets hope they'll be back, because it doesn't take long for the Boinc community to turn against a project.
Rod

ID: 4676 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Misfit
Avatar

Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,319
RAC: 0
1 million credit badge10 year member badge
Message 4691 - Posted: 14 Aug 2008, 5:14:53 UTC - in response to Message 4676.  

Some projects are run by grad students who disappear on summer vacation.
me@rescam.org
ID: 4691 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Odd-Rod

Send message
Joined: 7 Sep 07
Posts: 442
Credit: 1,404,502
RAC: 63
1 million credit badge10 year member badge
Message 4692 - Posted: 14 Aug 2008, 6:07:51 UTC - in response to Message 4691.  

Some projects are run by grad students who disappear on summer vacation.

If they are on summer vacation when would they likely to be back? Our summer vacation here in South Africa is over Christmas, so I've no idea when their vacation ends.

But still very strange that they would suddenly vanish after being so active, without at least saying goodbye...

Rod
ID: 4692 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileTravis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Aug 07
Posts: 2046
Credit: 26,480
RAC: 0
10 thousand credit badge10 year member badge
Message 4693 - Posted: 14 Aug 2008, 18:20:15 UTC - in response to Message 4691.  

Some projects are run by grad students who disappear on summer vacation.



shh.

school starts back up in 3 weeks or so.

we're clearing results from the database quickly because the server would get overloaded and things would come to a screeching halt. it seems that having the purge running constantly was the only way to fix that.

on another note, nate and i have been pretty busy working on a new version of the application and assimilator, which will mean even more serious workunits; and hopefully less server load. there'll be more about this in the next week or two when we start it up.
ID: 4693 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Number crunching : Clearing results from database quickly?

©2020 Astroinformatics Group