Welcome to MilkyWay@home

Why is it so hard to get work?


Advanced search

Message boards : Number crunching : Why is it so hard to get work?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · Next

AuthorMessage
BarryAZ

Send message
Joined: 1 Sep 08
Posts: 519
Credit: 289,496,332
RAC: 27,561
200 million credit badge13 year member badgeextraordinary contributions badge
Message 23386 - Posted: 26 May 2009, 7:50:24 UTC - in response to Message 23346.  

The 'week or two' might well be longer unless the optimized ATI GPU application is in that timeframe. A lot of the load here is from the ATI crunchers -- after all, this is the ONLY project with an ATI_GPU application in the BOINC world. One really can't blame them for the load here, it is much a matter of there being no other home for those cycles. When the millenial day of ATI_GPU support existing in multiple BOINC projects occurs, then life might be sweet indeed.






I'm really hoping to put this all behind us once we get milkyway_gpu up and running (which is getting really close). Probably another week or two.



ID: 23386 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Profilebanditwolf
Avatar

Send message
Joined: 12 Nov 07
Posts: 2425
Credit: 524,164
RAC: 0
500 thousand credit badge14 year member badge
Message 23395 - Posted: 26 May 2009, 12:17:51 UTC

To be honest, before people started using scripts to hammer the server, we were getting around 9-11 workunits a second. Now we're seeing around 6-7 workunits a second.


Now shouldn't this get some/all to let off their scripts so they could actually get more work? What more proof is needed that the scripts are hindering the project?
Doesn't expecting the unexpected make the unexpected the expected?
If it makes sense, DON'T do it.
ID: 23395 · Rating: 0 · rate: Rate + / Rate - Report as offensive
ProfileJames Sotherden
Avatar

Send message
Joined: 3 Jan 09
Posts: 139
Credit: 30,373,963
RAC: 17,104
30 million credit badge12 year member badge
Message 23399 - Posted: 26 May 2009, 12:51:25 UTC

Well i have noticed that since the GPU code came out that i have'nt had hardly any work at all. But thats all right seti has taking up the slack and I'm all most at the half million mark. I came here when seti was having big problems, ill stay here now that Milkyway is having problems. I want to get to that half million mark here to. As the doctor said to the patient with kidney stones, this to shall pass. And besides I really want to see the finished 3D model.
ID: 23399 · Rating: 0 · rate: Rate + / Rate - Report as offensive
ProfileGalaxyIce
Avatar

Send message
Joined: 6 Apr 08
Posts: 2018
Credit: 100,142,856
RAC: 0
100 million credit badge13 year member badge
Message 23403 - Posted: 26 May 2009, 13:10:23 UTC - in response to Message 23395.  

To be honest, before people started using scripts to hammer the server, we were getting around 9-11 workunits a second. Now we're seeing around 6-7 workunits a second.


Now shouldn't this get some/all to let off their scripts so they could actually get more work? What more proof is needed that the scripts are hindering the project?

Proof? I wonder what 'proof' you are talking about? That the only people using scripts are those crunching GPU WUs? Or that most, by far, of crunchers here are CPU crunchers, and that most, by far of script users are CPU crunchers? Or that the user base has increased rapidly since word got out that pretty damn good credits can be had by using optimized apps at MilkyWay? And that most of these new crunchers are CPU crunchers and not GPU crunchers? And that the talk and detailed publication of script useage has made it more and more likely that more and more are taking to script usage?

Or a combination of some or all of the above? Let's hear the 'proof' please, and perhaps think before making unsubstantiated allegations?"



ID: 23403 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Brian Silvers

Send message
Joined: 21 Aug 08
Posts: 625
Credit: 558,425
RAC: 0
500 thousand credit badge13 year member badge
Message 23405 - Posted: 26 May 2009, 13:38:12 UTC - in response to Message 23403.  


Or a combination of some or all of the above? Let's hear the 'proof' please, and perhaps think before making unsubstantiated allegations?"


The scripts simply cannot be helping, unless you are only concerned with your limited viewpoint of your own machines, and even then it might not be helping long-term. A classic example of how people don't understand how things work long-term with BOINC is the assertion that "Project 'X' is hogging my CPU" when the CPU scheduler has started trying to make sure that work gets done by deadline. The resource shares get honored over the long-term, but might not be honored to exacting precision short-term.

To give another example, if there's a fire in a building and only one exit, more people can get out if people don't panic and stampede the exit. It is apparently just as hard to get you folks to understand that you might be throttling yourselves as it is to get people to remain calm in a fire...
ID: 23405 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Bill & Patsy
Avatar

Send message
Joined: 7 Jul 08
Posts: 47
Credit: 13,629,944
RAC: 0
10 million credit badge13 year member badge
Message 23407 - Posted: 26 May 2009, 13:57:57 UTC
Last modified: 26 May 2009, 13:58:52 UTC

Ice, the most recent postings from banditwolf and Brian were platform independent. It's about overloading the server. Sure, as you point out, the heavy server traffic is caused by many factors, but as they and Travis and others have pointed out, the scripts are just making things worse, and adding to the decrease in available work throughput.
--Bill

ID: 23407 · Rating: 0 · rate: Rate + / Rate - Report as offensive
John Clark

Send message
Joined: 4 Oct 08
Posts: 1734
Credit: 64,228,409
RAC: 0
50 million credit badge13 year member badge
Message 23409 - Posted: 26 May 2009, 14:43:10 UTC
Last modified: 26 May 2009, 14:45:43 UTC

It is probably true that script users are slowing down the transfer of Milkyway WUs to both GPU and CPU users. But other factors enter in to the slow down including limited shared memory on the server for the WU transfer.

I have this quad core running 2 projects - Malaria Control and Milkyway - concurrently on the CPU with a 50:50 share. This means that 2 cores are dedicated to each project. I have run the PC long enough to have settled down the project RACs, and for Milkyway this box gives an RAC of 8,413. When I ran all 4 cores of this CPU on Milkyway the RAC was heading up north of 16,000.

This can get a goodly percentage of the slower ATI GPU outputs.

I find that my CPU boxes are not having that much difficulty getting work, and maintaining their limited caches.

Perhaps I am very luck getting work, but things should calm down in a few weeks time with MW_GPU. So continuous whining is not going to help your concerns or corner.
Go away, I was asleep


ID: 23409 · Rating: 0 · rate: Rate + / Rate - Report as offensive
ProfileAriZonaMoon*
Avatar

Send message
Joined: 29 Sep 08
Posts: 1618
Credit: 46,511,893
RAC: 0
30 million credit badge13 year member badge
Message 23415 - Posted: 26 May 2009, 15:42:22 UTC

How nice to see everyone so happy! ;-D


My Rac is falling.. and its not anything I can blame
MW for.. That would have been too easy. Aye. ;-)


ID: 23415 · Rating: 0 · rate: Rate + / Rate - Report as offensive
John Clark

Send message
Joined: 4 Oct 08
Posts: 1734
Credit: 64,228,409
RAC: 0
50 million credit badge13 year member badge
Message 23417 - Posted: 26 May 2009, 15:57:34 UTC - in response to Message 23415.  

How nice to see everyone so happy! ;-D


My Rac is falling.. and its not anything I can blame
MW for.. That would have been too easy. Aye. ;-)



Good afternoon Moon.

Stealth should be joining us in the other place today.
Go away, I was asleep


ID: 23417 · Rating: 0 · rate: Rate + / Rate - Report as offensive
ProfileAriZonaMoon*
Avatar

Send message
Joined: 29 Sep 08
Posts: 1618
Credit: 46,511,893
RAC: 0
30 million credit badge13 year member badge
Message 23419 - Posted: 26 May 2009, 16:08:06 UTC - in response to Message 23417.  

How nice to see everyone so happy! ;-D


My Rac is falling.. and its not anything I can blame
MW for.. That would have been too easy. Aye. ;-)



Good afternoon Moon.

Stealth should be joining us in the other place today.


Would it help my RAC??

LoL. Cya around John. ;-)
ID: 23419 · Rating: 0 · rate: Rate + / Rate - Report as offensive
BarryAZ

Send message
Joined: 1 Sep 08
Posts: 519
Credit: 289,496,332
RAC: 27,561
200 million credit badge13 year member badgeextraordinary contributions badge
Message 23420 - Posted: 26 May 2009, 16:21:15 UTC - in response to Message 23407.  
Last modified: 26 May 2009, 16:24:17 UTC

I'd suggest that scripts, which might be making things unhappy, especially for those not using scripts, were in fact a reaction to what was seen as an existing difficulty in getting work. That difficulty in getting work *PRECEDED* the frequent use of scripts. The timing as I see it (others have mentioned this before in addition to me), is something like this.

Last summer increasing numbers of people became aware of the high credit values offered in MilkyWay and so migrated CPU cycles this way.

As the code was open for optimization, a few hardy souls figured out how to optimize the applications while producing legitimate results. At the the time these optimizations were privately used and not broadly available, but even with only a few running optimized code, the total amount of work being completed jumped big time.

Adjustments were made as to credits per work units and maximum number of completed work units in order to respond to the vastly more efficient optimized code.

Then, in the early fall (late September), the optimized code went public (and somewhat viral). This resulted in a huge increase in the number of work units requested and increasing the load on the servers. Also, the rapid rise of participants really took off. There were some server issues dealing with the increased load, and there was a further adjustment to credits (perhaps under influence from DA). Cache sizes were reduced as well.

Later in the fall, folks optimizing the application code were able to complete an optimized application with support for double precision ATI video cards (38xx and 48xx). As this was (AND IS) the only place for ATI video cards to participate with those resources in a BOINC project, there was an additional draw to new users.

By late October, early November, the vast increase in CPU and GPU resources available to run MilkyWay along with the vast increase in efficiency, basically outstripped the ability of the server have adequate work available for all comers. There were some project issues resulting from this overload.

When the servers were tweaked to respond to the vastly higher demand, the work units being requested by the 4x increase in users, 10x increase in CPU efficiency and the totally new work unit capability of ATI GPU's resulted in a sparse work unit availability scenario. Which still exists today.

Some users (both CPU and GPU) began then in response to the difficulty in obtaining work, to use scripts which request work from the server at scripted intervals. The effect of these scripts has been first to in effect redirect available work to those running scripts -- resulting in some folks who are running scripts achieving (with the optimized GPU application in particular) some VERY HIGH RAC's (in the >100K range). Folks with CPU's are also running those scripts, though with the significant processing difference, few if any CPU only folks can achieve RAC's in the >50K range.

The contention and strife going on here at the moment has some GPU advocates (many of whom are running scripts) flailing at the CPU folks for wasting energy and whining about the morally superior way of the GPU. The CPU folks (and some few GPU folks who aren't running scripts) are complaining about the GPU folks and scriptors in general in their immoral redistribution of work.

When (if) the MilkyWay_GPU project gets off the ground, it may do much to mitigate the contention -- though until that project includes an optimized ATI GPU application I doubt that will happen. Travis has indicated that when the Milkyway_GPU project is running (and I'm assuming when it is running along with an optimized ATI GPU application), the current MilkyWay project will revert to a CPU only project (via control methods as yet unannounced).

That will certainly reduce the load on this (CPU) side of the research effort. Will that load reduction be adequate to render scripts of no value in getting work? Fair question -- we shall quite possibly know the answer to this in the summer.

In the meantime, I think folks are wasting a LOT of organic cycles on yammering on one another.


Ice, the most recent postings from banditwolf and Brian were platform independent. It's about overloading the server. Sure, as you point out, the heavy server traffic is caused by many factors, but as they and Travis and others have pointed out, the scripts are just making things worse, and adding to the decrease in available work throughput.

ID: 23420 · Rating: 0 · rate: Rate + / Rate - Report as offensive
ProfilePhil
Avatar

Send message
Joined: 13 Feb 08
Posts: 1124
Credit: 46,740
RAC: 0
10 thousand credit badge13 year member badge
Message 23424 - Posted: 26 May 2009, 17:00:53 UTC - in response to Message 23368.  
Last modified: 26 May 2009, 17:07:21 UTC

"If you don't like my scripting screwing up the project, then you should shut up and leave..."

So, am I going to get banned too?

Well that depends on your aims.
Caring Sharing eco-friendly contributors who wish to ensure a constant output of scientific throughput will be welcomed of course, its just them script-kiddies who are after personal gain that'll be ditched.
ID: 23424 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Misfit
Avatar

Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,319
RAC: 0
1 million credit badge14 year member badge
Message 23425 - Posted: 26 May 2009, 17:04:01 UTC - in response to Message 23399.  

As the doctor said to the patient with kidney stones, this to shall pass.

Painfully.
me@rescam.org
ID: 23425 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Brian Silvers

Send message
Joined: 21 Aug 08
Posts: 625
Credit: 558,425
RAC: 0
500 thousand credit badge13 year member badge
Message 23436 - Posted: 26 May 2009, 18:50:49 UTC - in response to Message 23420.  

I'd suggest that scripts, which might be making things unhappy, especially for those not using scripts, were in fact a reaction to what was seen as an existing difficulty in getting work. That difficulty in getting work *PRECEDED* the frequent use of scripts.


All technically correct, but the issue is that using scripts in the environment that we're in, which is a work shortage for everyone, is akin to saying "I don't have enough, so I'm going to try to take some of yours". That's all well and good and stuff, but I think if someone were to do this, say, in a grocery store or in Wal-Mart, people might be rightfully offended. If work was plentiful and/or shortages were few and far between, it would be a different matter. However, the shortage is constant and it takes a certain bit of gall that I don't have to attempt to butt in line like some are doing.

However, getting past all of the greed aspects, there's another underlying issue...which is that in all likelihood, the scripts themselves are actually throttling the work distribution. The irony of that, if it is indeed the case, and I'm still curious about what Travis said yesterday, but if the scripts are throttling the outbound distribution, then people clamoring for more are simply shooting themselves in the feet. Of course getting people to realize this is next to impossible, probably a combination of not wanting to believe it and that they're not going to stop doing it because they know that whoever they are competing with is likely not going to stop...


When (if) the MilkyWay_GPU project gets off the ground, it may do much to mitigate the contention -- though until that project includes an optimized ATI GPU application I doubt that will happen. Travis has indicated that when the Milkyway_GPU project is running (and I'm assuming when it is running along with an optimized ATI GPU application), the current MilkyWay project will revert to a CPU only project (via control methods as yet unannounced).


I truly hope that the project gets running sometime within the next few weeks...
ID: 23436 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Chris S
Avatar

Send message
Joined: 20 Sep 08
Posts: 1391
Credit: 203,118,321
RAC: 22,783
200 million credit badge13 year member badge
Message 23438 - Posted: 26 May 2009, 19:10:00 UTC

..its just them script-kiddies who are after personal gain that'll be ditched.


Good comment Phil, I agree.
ID: 23438 · Rating: 0 · rate: Rate + / Rate - Report as offensive
ProfileThe Gas Giant
Avatar

Send message
Joined: 24 Dec 07
Posts: 1947
Credit: 240,884,648
RAC: 0
200 million credit badge13 year member badge
Message 23443 - Posted: 26 May 2009, 19:53:37 UTC

As I pointed out in this message and to Travis...

To stop the scriptors hitting the project so hard, you could increase the minimum time between host contacts at the server end. LHC@home increased theirs to just over 15 minutes....maybe you could try 2 minutes and see what happens. I believe it is a simple server side setting.
ID: 23443 · Rating: 0 · rate: Rate + / Rate - Report as offensive
ProfileLord Tedric
Avatar

Send message
Joined: 9 Nov 07
Posts: 151
Credit: 8,391,608
RAC: 0
5 million credit badge14 year member badge
Message 23445 - Posted: 26 May 2009, 20:07:04 UTC - in response to Message 23436.  


All technically correct, but the issue is that using scripts in the environment that we're in, which is a work shortage for everyone, is akin to saying "I don't have enough, so I'm going to try to take some of yours". That's all well and good and stuff, but I think if someone were to do this, say, in a grocery store or in Wal-Mart, people might be rightfully offended.


Q. When does a work unit become 'yours'?
A. When it's on your system and your crunching it!

Q. When are they your groceries?
A. When you've paid for them!
ID: 23445 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Brian Silvers

Send message
Joined: 21 Aug 08
Posts: 625
Credit: 558,425
RAC: 0
500 thousand credit badge13 year member badge
Message 23448 - Posted: 26 May 2009, 21:04:27 UTC - in response to Message 23443.  
Last modified: 26 May 2009, 21:05:14 UTC

As I pointed out in this message and to Travis...

To stop the scriptors hitting the project so hard, you could increase the minimum time between host contacts at the server end. LHC@home increased theirs to just over 15 minutes....maybe you could try 2 minutes and see what happens. I believe it is a simple server side setting.


That's what Bill was suggesting, and got shot down for it... Personally, I don't think 2 minutes is long enough. Needs to be at least 10. I'm sure though that 10 will cause a huge amount of complaining about how that means that someone might have 3-5 minutes of not being able to get anything, although I'm not sure how that's different from the current situation, but then again, I view this from the perspective of solving a problem, not from the eyes of a competitor in a competition.
ID: 23448 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Brian Silvers

Send message
Joined: 21 Aug 08
Posts: 625
Credit: 558,425
RAC: 0
500 thousand credit badge13 year member badge
Message 23449 - Posted: 26 May 2009, 21:09:33 UTC - in response to Message 23445.  
Last modified: 26 May 2009, 21:15:00 UTC


All technically correct, but the issue is that using scripts in the environment that we're in, which is a work shortage for everyone, is akin to saying "I don't have enough, so I'm going to try to take some of yours". That's all well and good and stuff, but I think if someone were to do this, say, in a grocery store or in Wal-Mart, people might be rightfully offended.


Q. When does a work unit become 'yours'?
A. When it's on your system and your crunching it!

Q. When are they your groceries?
A. When you've paid for them!


Do you ever wonder why it is that movies about global destruction from a comet, asteroid, or whatever seem to initially focus on the fact that none of us make any kind of overt attempt at being cooperative with each other until disaster looms large, and even then it might take a lot of convincing? It's all about who can get the most of whatever, no matter what the detrement to others. While I don't advocate Socialism or Communism, I think it would be nice if we were not as petty and greedy all the time...

Think about this:

If I were to follow your logic to the fullest extent, then I would be within my rights to reach into your shopping cart and take the item that is in your cart that I want, because you haven't paid for it yet.

Or how about people in a flooded area that are in need of water, and I just take all of yours and tell you "tough luck"?

Behavior like this only ends up in escalation of bad feelings / hostility if it is over something very important (like food, water, etc...)

However, as I said in my previous post, which you left out, the irony is that the greed might be causing a further slowdown. So much for the "more science than ever" claim too...
ID: 23449 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Profile[KWSN]John Galt 007
Avatar

Send message
Joined: 12 Dec 08
Posts: 56
Credit: 269,889,439
RAC: 0
200 million credit badge12 year member badge
Message 23450 - Posted: 26 May 2009, 21:11:38 UTC - in response to Message 23448.  

As I pointed out in this message and to Travis...

To stop the scriptors hitting the project so hard, you could increase the minimum time between host contacts at the server end. LHC@home increased theirs to just over 15 minutes....maybe you could try 2 minutes and see what happens. I believe it is a simple server side setting.


That's what Bill was suggesting, and got shot down for it... Personally, I don't think 2 minutes is long enough. Needs to be at least 10. I'm sure though that 10 will cause a huge amount of complaining about how that means that someone might have 3-5 minutes of not being able to get anything, although I'm not sure how that's different from the current situation, but then again, I view this from the perspective of solving a problem, not from the eyes of a competitor in a competition.



And something I posted got shot down as well...

If you limit the # of WUs per core, there should be plenty for all. 5k per CPU is a bit much, even if you run only CPUs...
Click to help Seti City.




ID: 23450 · Rating: 0 · rate: Rate + / Rate - Report as offensive
Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · Next

Message boards : Number crunching : Why is it so hard to get work?

©2021 Astroinformatics Group