Welcome to MilkyWay@home

Separation Project Coming To An End

Message boards : News : Separation Project Coming To An End
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 . . . 13 · Next

AuthorMessage
Profile Tom Donlon
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 10 Apr 19
Posts: 408
Credit: 120,203,200
RAC: 0
Message 75485 - Posted: 13 Jun 2023, 16:17:05 UTC

Dear Volunteers,

We are beginning to sunset the Separation project (note that this is different from the N-Body project, which still needs your help!!!). Thanks to your computational contributions, we have completed the goals that we set out to accomplish with this the project, so we are going to shut down the Separation project in the near future. We are planning on submitting a paper containing the final results of the Separation project to an academic journal in the near future. It is possible that the reviewer may request revisions that involve re-running Separation data, in which case we might bring it back for a short time, but until that happens we do not need your computer time for this. We would like to sincerely thank you for all the time and effort that you spent helping us complete this substantial task. We owe the success of the project to you all.

With this announcement comes a few updates to MilkyWay@home: MilkyWay@home will continue to exist, but only as the Nbody project. Because of the complexity of running Nbody tree codes on GPUs, there is little speed-up from running Nbody on a GPU compared to an equivalent CPU. Although we did beta-test a full GPU Nbody application, we have decided not to deploy it due to concerns over maintaining its support; among other issues, supporting GPU Nbody would require changing the GPU code every time the CPU code is updated, and making sure every change is tested and works identically on both platforms. Since we are actively working on extensions of the Nbody code that include such things as the effects of the Large Magellanic Cloud's gravity and the consequences of self-interacting dark matter on the formation of tidal streams, this parallel support would be difficult. So, when Separation shuts down, we will be ending GPU support for MilkyWay@home. This will have the greatest impact on people who crunch lots of Separation work units on high-end GPUs; these users can expect to earn a much lower amount of credit per time than they did before. If you contribute only CPU cycles, your ability to earn credit will be unaffected. We still want as many people to contribute CPU time to MilkyWay@home as possible! However, we understand that if your goal is to use GPUs to obtain a large amount of credit quickly, there are more efficient ways to do this and other projects that can better use your resources.

We would appreciate your input on this because we expect that it will probably take some time for GPU-oriented users to swap that hardware over to different projects. How long would you like us to wait before we shut down Separation?

Again, thank you so much for letting us use your processors! They have not only increased our knowledge of the Milky Way galaxy that we live in, but they have also contributed to five PhD theses and trained dozens of undergraduate students to build and maintain large and complex computing systems. These students and I thank you for your contributions. We will be posting the results of the final Separation project results probably later this year, after they have gone through the peer review process (which takes about six months on average, with wide variability).

Best,
Tom & Prof. Newberg
ID: 75485 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Keith Myers
Avatar

Send message
Joined: 24 Jan 11
Posts: 709
Credit: 547,415,492
RAC: 61,215
Message 75486 - Posted: 13 Jun 2023, 16:28:18 UTC - in response to Message 75485.  

I would just set the shutdown at the deadline after the last task is sent out. That allows the slowest host with too large a cache to get most of the work returned.

Any news or plans for a research paper using the results from the Separation search?
ID: 75486 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Skillz

Send message
Joined: 28 May 17
Posts: 76
Credit: 4,398,257,447
RAC: 44,222
Message 75487 - Posted: 13 Jun 2023, 16:32:14 UTC - in response to Message 75485.  

. How long would you like us to wait before we shut down Separation?

Best,
Tom & Prof. Newberg


Around 2030. :(
ID: 75487 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ian&Steve C.
Avatar

Send message
Joined: 18 Nov 22
Posts: 84
Credit: 640,530,617
RAC: 12,364
Message 75488 - Posted: 13 Jun 2023, 16:49:36 UTC - in response to Message 75485.  

I would not say that the goal of GPU users is to "gain large amounts of credit quickly". that's a pretty large mischaracterization in my opinion.

most, if not all, of us use GPUs because they are massively more power efficient for the work being done. hundreds or thousands of times more work done per watt than CPUs when properly coded/optimized. since most of us pay for our own electricity to donate free compute resources, it makes sense that people will go for a power efficient device.

sad to see the Separation project going away as I personally just came to the project. but I do understand shutting down a project for which there is no longer a benefit. I agree that there's no reason to keep a project up to crunch data that no one will look at just to placate the users.

as far as timeline, in line with my last statement, that's probably best left up to the project scientists rather than the users. we have a lot of bias in that decision and most users will say keep it up as long as possible. the best answer will be to shut it down when YOU are ready and no longer need the results.

ID: 75488 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Tom Donlon
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 10 Apr 19
Posts: 408
Credit: 120,203,200
RAC: 0
Message 75489 - Posted: 13 Jun 2023, 17:03:52 UTC - in response to Message 75486.  

I would just set the shutdown at the deadline after the last task is sent out. That allows the slowest host with too large a cache to get most of the work returned.

Any news or plans for a research paper using the results from the Separation search?


Yes! We are working on one. It has had a very long lead time, but it's getting done.
ID: 75489 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile crashtech

Send message
Joined: 2 Oct 16
Posts: 4
Credit: 5,433,038,917
RAC: 14,443
Message 75490 - Posted: 13 Jun 2023, 17:26:17 UTC

I bought much of my hardware with Separation's rather unique FP64 requirements in mind. It's a shame that it is ending.
Is there an expected time frame for tasks to run out?
ID: 75490 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
bluestang

Send message
Joined: 13 Oct 16
Posts: 112
Credit: 1,174,293,644
RAC: 0
Message 75491 - Posted: 13 Jun 2023, 17:35:15 UTC

Hate to say it, but a lot of users will not stay/come back to just run CPU on this project as they are used to what their GPU can do. So expect a large drop off of users after.

However, maybe spend sometime optimizing the N-Body GPU app so that maybe they will stay/come back? Especially if you can find a way to utilize NVDIA CUDA more to bring those users in?
ID: 75491 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Tom Donlon
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 10 Apr 19
Posts: 408
Credit: 120,203,200
RAC: 0
Message 75492 - Posted: 13 Jun 2023, 18:10:17 UTC - in response to Message 75491.  

Hate to say it, but a lot of users will not stay/come back to just run CPU on this project as they are used to what their GPU can do. So expect a large drop off of users after.

However, maybe spend sometime optimizing the N-Body GPU app so that maybe they will stay/come back? Especially if you can find a way to utilize NVDIA CUDA more to bring those users in?


You're right, we expect a large drop-off in the number of users.

During our benchmark tests, other GPU N-body codes performed similarly to our GPU code. I don't think that it is an issue with optimization of the GPU code implementation, I think that there is just too much overhead with building the particle tree on the GPU that there is no substantial speed-up compared to running on CPU. It's possible that there are improvements that could be made, but it isn't really a feasible thing to support with our current group and infrastructure.
ID: 75492 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ian&Steve C.
Avatar

Send message
Joined: 18 Nov 22
Posts: 84
Credit: 640,530,617
RAC: 12,364
Message 75493 - Posted: 13 Jun 2023, 19:30:15 UTC - in response to Message 75492.  

Hate to say it, but a lot of users will not stay/come back to just run CPU on this project as they are used to what their GPU can do. So expect a large drop off of users after.

However, maybe spend sometime optimizing the N-Body GPU app so that maybe they will stay/come back? Especially if you can find a way to utilize NVDIA CUDA more to bring those users in?


You're right, we expect a large drop-off in the number of users.

During our benchmark tests, other GPU N-body codes performed similarly to our GPU code. I don't think that it is an issue with optimization of the GPU code implementation, I think that there is just too much overhead with building the particle tree on the GPU that there is no substantial speed-up compared to running on CPU. It's possible that there are improvements that could be made, but it isn't really a feasible thing to support with our current group and infrastructure.


is the GPU n-body code on your github page? we have a very competent GPU developer on our team who is great at GPU optimization. we've produced optimized GPU applications for several other projects.

if it's not on your github, would you be willing to share what code you have already so that we can take a look?

ID: 75493 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
.clair.

Send message
Joined: 3 Mar 13
Posts: 84
Credit: 779,527,712
RAC: 7
Message 75494 - Posted: 13 Jun 2023, 19:33:07 UTC
Last modified: 13 Jun 2023, 19:40:56 UTC

It is a pity to see separation go , though also nice to know the research will be complete and written up .
But
I have 869 tasks stuck in the database from the January 2021 crash , I am shure there will be many other participants also with stuck tasks , across all participants this could add up to many thousands of work units in limbo .
Half of mine are still `pending` ,
Before the separation work is removed from server could someone have a look to see if anything can be rescued or made useable from them .
ID: 75494 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Nik343

Send message
Joined: 16 Nov 20
Posts: 1
Credit: 75,312,176
RAC: 0
Message 75495 - Posted: 13 Jun 2023, 20:03:56 UTC

Please keep Separation running! I'm willing to contibute hardware that specifically excels in FP64 for the help of the project. It would be a shame if I had to transition elsewhere.. :(
ID: 75495 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Skillz

Send message
Joined: 28 May 17
Posts: 76
Credit: 4,398,257,447
RAC: 44,222
Message 75496 - Posted: 13 Jun 2023, 20:33:12 UTC - in response to Message 75495.  

Please keep Separation running! I'm willing to contibute hardware that specifically excels in FP64 for the help of the project. It would be a shame if I had to transition elsewhere.. :(


I don't think you read the reason for shutting down separation. The project has enough data for their research.
ID: 75496 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
alanb1951

Send message
Joined: 16 Mar 10
Posts: 211
Credit: 107,266,727
RAC: 18,297
Message 75497 - Posted: 13 Jun 2023, 20:58:43 UTC

Tom,

It's always a bit sad to see a project come to an end, but it's also good to know the goals have been achieved. Well done to all concerned!

Regarding actual shut-down, I'm inclined to agree with Ian&Steve C's comment on the timeline in his first post. Has the current Separation run already reached a point where it's effectively work for works sake, or is there still serious value in processing the rest of this batch? It really has to be your call as to when to stop generating new work :-)

If it's a matter of trying to get the current run to converge as quickly as possible, I'll stick around until that happens (as, I suspect, will many others); otherwise I might as well switch to N-Body only at once, resuming GPU work if and only if you have to do re-runs based on the review process.

Regarding GPU users finding other things to do, I suspect quite a few of us will just give more time to Einstein@home :-)

Good luck with the paper! I'll continue to watch progress on all projects here with great interest...

Cheers - Al.

P.S. there is still the occasional post on the SETI@home site asking when that is going to restart -- expect the same to happen here :-)
ID: 75497 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
bluestang

Send message
Joined: 13 Oct 16
Posts: 112
Credit: 1,174,293,644
RAC: 0
Message 75498 - Posted: 13 Jun 2023, 21:55:25 UTC - in response to Message 75493.  

Hate to say it, but a lot of users will not stay/come back to just run CPU on this project as they are used to what their GPU can do. So expect a large drop off of users after.

However, maybe spend sometime optimizing the N-Body GPU app so that maybe they will stay/come back? Especially if you can find a way to utilize NVDIA CUDA more to bring those users in?


You're right, we expect a large drop-off in the number of users.

During our benchmark tests, other GPU N-body codes performed similarly to our GPU code. I don't think that it is an issue with optimization of the GPU code implementation, I think that there is just too much overhead with building the particle tree on the GPU that there is no substantial speed-up compared to running on CPU. It's possible that there are improvements that could be made, but it isn't really a feasible thing to support with our current group and infrastructure.


is the GPU n-body code on your github page? we have a very competent GPU developer on our team who is great at GPU optimization. we've produced optimized GPU applications for several other projects.

if it's not on your github, would you be willing to share what code you have already so that we can take a look?


As long as it is not Linux only :)
ID: 75498 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Keith Myers
Avatar

Send message
Joined: 24 Jan 11
Posts: 709
Credit: 547,415,492
RAC: 61,215
Message 75499 - Posted: 13 Jun 2023, 22:03:23 UTC - in response to Message 75498.  


As long as it is not Linux only :)

It is much, much easier to develop in Linux compared to Windows.

Nothing preventing a good Windows developer from tackling a N-body gpu app.

But no reason to NOT develop an enhanced N-body gpu app for Linux users.

The user always has the option of running Linux.
ID: 75499 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile [AF>Le_Pommier] Jerome_C2005

Send message
Joined: 1 Apr 08
Posts: 30
Credit: 84,628,667
RAC: 894
Message 75500 - Posted: 13 Jun 2023, 23:11:48 UTC

Gee, this is the saddest news of the year so far, I'll never get my big blue star :(

But happy to read that what we did was useful somehow though ;)
ID: 75500 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3329
Credit: 524,004,144
RAC: 33,776
Message 75501 - Posted: 13 Jun 2023, 23:18:09 UTC - in response to Message 75492.  

Hate to say it, but a lot of users will not stay/come back to just run CPU on this project as they are used to what their GPU can do. So expect a large drop off of users after.

However, maybe spend sometime optimizing the N-Body GPU app so that maybe they will stay/come back? Especially if you can find a way to utilize NVDIA CUDA more to bring those users in?


You're right, we expect a large drop-off in the number of users.

During our benchmark tests, other GPU N-body codes performed similarly to our GPU code. I don't think that it is an issue with optimization of the GPU code implementation, I think that there is just too much overhead with building the particle tree on the GPU that there is no substantial speed-up compared to running on CPU. It's possible that there are improvements that could be made, but it isn't really a feasible thing to support with our current group and infrastructure.


I think if you could bring up the N-Body gpu tasks before you sunset the Separation gpu tasks it would be an easy transfer of resources from one app to the other and the loss of users would be minimal. Getting the data to Ian&Steve's teammate would at least get the Linux side of things up and running pretty quickly and then maybe you guys inhouse could develop the Windows app using his tweaks or reaching out to someone else who has the skillset, perhaps even within RPI itself.
ID: 75501 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Cthulhu

Send message
Joined: 15 Dec 20
Posts: 1
Credit: 21,648,260
RAC: 7,093
Message 75503 - Posted: 13 Jun 2023, 23:38:34 UTC - in response to Message 75485.  

Already disconnected the project from my machines after all available current work completed this evening.
ID: 75503 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
carmar

Send message
Joined: 29 Apr 22
Posts: 2
Credit: 341,311
RAC: 0
Message 75504 - Posted: 14 Jun 2023, 1:01:37 UTC

My request is till the end of this year, if it’s possible.

How soon will we know the duration decision (sorry if this has already been answered)?
ID: 75504 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
San-Fernando-Valley

Send message
Joined: 13 Apr 17
Posts: 256
Credit: 604,411,638
RAC: 0
Message 75505 - Posted: 14 Jun 2023, 6:15:36 UTC - in response to Message 75488.  

Ian&Steve C. :

Very well said ...

S-F-V
ID: 75505 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · 3 · 4 . . . 13 · Next

Message boards : News : Separation Project Coming To An End

©2024 Astroinformatics Group