Welcome to MilkyWay@home

MilkyWay steals all my CPUs

Questions and Answers : Preferences : MilkyWay steals all my CPUs
Message board moderation

To post messages, you must log in.

AuthorMessage
Chris Rampson
Avatar

Send message
Joined: 14 May 11
Posts: 7
Credit: 87,497,784
RAC: 0
Message 69763 - Posted: 29 Apr 2020, 13:11:05 UTC

Milkyway grabs all my CPUs and crunches away while other projects miss their deadlines - even when Milkyway has a LOWER resource share and it's deadline is far in the future.

Can this be fixed in Preferences, or is it a coding issue?

RH Linux 7 with 8-core AMD
ID: 69763 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3315
Credit: 519,950,357
RAC: 21,982
Message 69764 - Posted: 29 Apr 2020, 17:28:44 UTC - in response to Message 69763.  

Milkyway grabs all my CPUs and crunches away while other projects miss their deadlines - even when Milkyway has a LOWER resource share and it's deadline is far in the future.

Can this be fixed in Preferences, or is it a coding issue?

RH Linux 7 with 8-core AMD


Need a few more details...what is the resource share for each Project you are attached to?
What is your cache setting for Boinc? Do you use the same setting for every Project?
I see you are doing both cpu and gpu workunits here, are you doing the same thing at your other Projects?

One problem could be your cache size, you have 314 tasks right now waiting to run, is that a 10 day cache? If so try lowering your cache to about 1 to 2 days, assuming you have always on internet, and let Boinc figure things out much easier. Right now you could have soooo many MilkyWay workunits that Boinc thinks you will never get thru them in time unless it crunches nonstop 24/7 especially since you are doing the mt cpu tasks.

My suggestion would be to abort all the cpu workunits from MilkyWay and just crunch gpu units from here and cpu workunits from your other Projects. ALL cpu workunits here at MilkyWay are mt tasks, mt meaning multi-threaded meaning multiple cpu cores all working on a single workunit.
ID: 69764 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ghend

Send message
Joined: 10 May 09
Posts: 4
Credit: 51,806,357
RAC: 0
Message 69776 - Posted: 8 May 2020, 23:41:00 UTC
Last modified: 8 May 2020, 23:46:35 UTC

Milkyway seems to look at your number of days to cache (1.5 in my case) and the percentage of CPUs that BOINC is allowed and then fills every second of that allocation with work. N-body tasks want to use 16 out of the 18 CPUs that I have allocated. A large number of single CPU separation jobs are also sent. There is no way to limit this that I have found.

One thing I have done is allow Milkyway to send work then set No New Tasks until it has depleted its queue and other projects have had time to catch up, then allow another batch of tasks. I have also limited tasks to GPU only. After a few days you get a nag notice that you have CPU work turned off with instructions for how to turn it back on. When I have had enough of manually managing the project I shut it off for several weeks.

Other projects seem to play better with others. Rosetta appears to some degree adjust how many jobs they send relative the the max_concurrent setting in app_config.xml. LHC and Cosmology will let you set hard limits on how many jobs are assigned at any one time and limit how many CPUs the multithreaded jobs can use. This method is very effective.

I read somewhere that it is possible to have more than one instance of BOINC running at the same time. This would allow running Milkyway separately with limited allocated resources and its own separate scheduler. I have not been motivated enough to try it.

It would be nice if the Milkyway developers could implement something similar to what LHC and Cosmology have done for CPU jobs. Then I could leave the project running 24/7.
ID: 69776 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3315
Credit: 519,950,357
RAC: 21,982
Message 69803 - Posted: 10 May 2020, 16:54:15 UTC - in response to Message 69776.  

Milkyway seems to look at your number of days to cache (1.5 in my case) and the percentage of CPUs that BOINC is allowed and then fills every second of that allocation with work. N-body tasks want to use 16 out of the 18 CPUs that I have allocated. A large number of single CPU separation jobs are also sent. There is no way to limit this that I have found.

One thing I have done is allow Milkyway to send work then set No New Tasks until it has depleted its queue and other projects have had time to catch up, then allow another batch of tasks. I have also limited tasks to GPU only. After a few days you get a nag notice that you have CPU work turned off with instructions for how to turn it back on. When I have had enough of manually managing the project I shut it off for several weeks.

Other projects seem to play better with others. Rosetta appears to some degree adjust how many jobs they send relative the the max_concurrent setting in app_config.xml. LHC and Cosmology will let you set hard limits on how many jobs are assigned at any one time and limit how many CPUs the multithreaded jobs can use. This method is very effective.

I read somewhere that it is possible to have more than one instance of BOINC running at the same time. This would allow running Milkyway separately with limited allocated resources and its own separate scheduler. I have not been motivated enough to try it.

It would be nice if the Milkyway developers could implement something similar to what LHC and Cosmology have done for CPU jobs. Then I could leave the project running 24/7.


Try adjusting the resource share to a lower number for MillyWay than the other Projects and see if that helps. BUT even then you will probably end up running all MilkyWay tasks one day and all of another Project another day, or for a few days, etc etc. This is just a fuction of how Boinc works but if you give it time it will even out.Ihavemy resource share for MilkyWay set at 1 and another Project at 10, so 10 times more for the other Project, and it does work out like that for the daily credits which is what resource share measures.
ID: 69803 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ghend

Send message
Joined: 10 May 09
Posts: 4
Credit: 51,806,357
RAC: 0
Message 69805 - Posted: 11 May 2020, 22:04:24 UTC - in response to Message 69803.  

Milkyway seems to look at your number of days to cache (1.5 in my case) and the percentage of CPUs that BOINC is allowed and then fills every second of that allocation with work. N-body tasks want to use 16 out of the 18 CPUs that I have allocated. A large number of single CPU separation jobs are also sent. There is no way to limit this that I have found.

One thing I have done is allow Milkyway to send work then set No New Tasks until it has depleted its queue and other projects have had time to catch up, then allow another batch of tasks. I have also limited tasks to GPU only. After a few days you get a nag notice that you have CPU work turned off with instructions for how to turn it back on. When I have had enough of manually managing the project I shut it off for several weeks.

Other projects seem to play better with others. Rosetta appears to some degree adjust how many jobs they send relative the the max_concurrent setting in app_config.xml. LHC and Cosmology will let you set hard limits on how many jobs are assigned at any one time and limit how many CPUs the multithreaded jobs can use. This method is very effective.

I read somewhere that it is possible to have more than one instance of BOINC running at the same time. This would allow running Milkyway separately with limited allocated resources and its own separate scheduler. I have not been motivated enough to try it.

It would be nice if the Milkyway developers could implement something similar to what LHC and Cosmology have done for CPU jobs. Then I could leave the project running 24/7.


Try adjusting the resource share to a lower number for MillyWay than the other Projects and see if that helps. BUT even then you will probably end up running all MilkyWay tasks one day and all of another Project another day, or for a few days, etc etc. This is just a fuction of how Boinc works but if you give it time it will even out.Ihavemy resource share for MilkyWay set at 1 and another Project at 10, so 10 times more for the other Project, and it does work out like that for the daily credits which is what resource share measures.


I have tried that, does not make any real difference. project_max_concurrent is set in app_config.xml for all of my BOINC projects so they don't have a chance to kill each other and also to control fan noise. I also usually don't allow n_body to run because there is no way to control how many CPUs it wants to use. (This is something that I started doing recently that I neglected to mention above.) For Milkyway, project_max_concurrent is set to 3. That allows 1 GPU and only 2 CPU jobs to run, or 3 CPU jobs if GPU jobs are not available. Milkyway will still send jobs for the 18 max CPUs that are allocated to BOINC, but since the jobs are only for Separation and the due date is far enough out they tend to finish in plenty of time.

Look here if you are not familiar with app_config.xml: https://boinc.berkeley.edu/wiki/Client_configuration
ID: 69805 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Keith Myers
Avatar

Send message
Joined: 24 Jan 11
Posts: 696
Credit: 540,061,643
RAC: 86,741
Message 69811 - Posted: 13 May 2020, 18:37:28 UTC - in response to Message 69805.  

I also usually don't allow n_body to run because there is no way to control how many CPUs it wants to use.

Incorrect. You have posted the link to the configuration document that shows explicitly an example for multi-threaded applications. Set the number of nthreads(cpu cores) you allow the mt application to use. Use a max_concurrent statement to restrict the number of mt tasks running at any one time.

...
   [<app_version>
       <app_name>Application_Name</app_name>
       [<plan_class>mt</plan_class>]
       [<avg_ncpus>x</avg_ncpus>]
       [<ngpus>x</ngpus>]
       [<cmdline>--nthreads 7</cmdline>]
   </app_version>]
   ...
   [<project_max_concurrent>N</project_max_concurrent>]

ID: 69811 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Keith Myers
Avatar

Send message
Joined: 24 Jan 11
Posts: 696
Credit: 540,061,643
RAC: 86,741
Message 69812 - Posted: 13 May 2020, 18:43:27 UTC

When starting ANY new project, ALWAYS change the REC parameter in cc_config first from the default 10 days to a single day. That allows the project credit debts to balance out much faster and you will have less likelihood of the new project commandeering the host's crunching time and preventing your other projects from crunching. In fact leave REC set at 1 day for the future after editing. The client runs much smoother on multiple project hosts that way.

<rec_half_life_days>1.000000</rec_half_life_days>
ID: 69812 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ghend

Send message
Joined: 10 May 09
Posts: 4
Credit: 51,806,357
RAC: 0
Message 69815 - Posted: 14 May 2020, 0:54:56 UTC - in response to Message 69811.  

I also usually don't allow n_body to run because there is no way to control how many CPUs it wants to use.

Incorrect. You have posted the link to the configuration document that shows explicitly an example for multi-threaded applications. Set the number of nthreads(cpu cores) you allow the mt application to use. Use a max_concurrent statement to restrict the number of mt tasks running at any one time.

...
   [<app_version>
       <app_name>Application_Name</app_name>
       [<plan_class>mt</plan_class>]
       [<avg_ncpus>x</avg_ncpus>]
       [<ngpus>x</ngpus>]
       [<cmdline>--nthreads 7</cmdline>]
   </app_version>]
   ...
   [<project_max_concurrent>N</project_max_concurrent>]



I have played with those settings but could not come up with anything that works. I will give this a try. Thank you. It would still be nice if they would add some kind of direct control on the prefs page. More casual users will just give up and delete the project.
ID: 69815 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Keith Myers
Avatar

Send message
Joined: 24 Jan 11
Posts: 696
Credit: 540,061,643
RAC: 86,741
Message 69817 - Posted: 14 May 2020, 1:20:13 UTC - in response to Message 69815.  
Last modified: 14 May 2020, 1:23:23 UTC

I have played with those settings but could not come up with anything that works. I will give this a try. Thank you. It would still be nice if they would add some kind of direct control on the prefs page. More casual users will just give up and delete the project.

This app_config.xml would for example limit your host to running 1 mt nbody task using 4 threads per task and only one task at a time.

<app_config>
   <app_version>
       <app_name>milkyway_nbody</app_name>
       <plan_class>mt</plan_class>
       <avg_ncpus>4</avg_ncpus>
       <cmdline>--nthreads 4</cmdline>
   </app_version>
   <max_concurrent>1</max_concurrent>
</app_config>


I understand that some other projects do in fact give you control over the number of threads used on multi-thread tasks directly in the application preferences. This project does not however and you will need to use the app_config for finer control.
ID: 69817 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ghend

Send message
Joined: 10 May 09
Posts: 4
Credit: 51,806,357
RAC: 0
Message 69818 - Posted: 14 May 2020, 2:29:00 UTC - in response to Message 69815.  

I also usually don't allow n_body to run because there is no way to control how many CPUs it wants to use.

Incorrect. You have posted the link to the configuration document that shows explicitly an example for multi-threaded applications. Set the number of nthreads(cpu cores) you allow the mt application to use. Use a max_concurrent statement to restrict the number of mt tasks running at any one time.

...
   [<app_version>
       <app_name>Application_Name</app_name>
       [<plan_class>mt</plan_class>]
       [<avg_ncpus>x</avg_ncpus>]
       [<ngpus>x</ngpus>]
       [<cmdline>--nthreads 7</cmdline>]
   </app_version>]
   ...
   [<project_max_concurrent>N</project_max_concurrent>]


I have played with those settings but could not come up with anything that works. I will give this a try. Thank you. It would still be nice if they would add some kind of direct control on the prefs page. More casual users will just give up and delete the project.


I don't know what is different about this from previous attempts but this works. Two 2 CPU nbody jobs are waiting to run. It had been a while since I had tried this. Thanks for nudging me into trying again.

<?xml version="1.0" ?>
<app_config>
<project_max_concurrent>3</project_max_concurrent>
<app>
<name>milkyway_nbody</name>
<max_concurrent>1</max_concurrent>
</app>
<app_version>
<app_name>milkyway_nbody</app_name>
<plan_class>mt</plan_class>
<avg_ncpus>2</avg_ncpus>
<cmdline>--nthreads 2</cmdline>
</app_version>
</app_config>
ID: 69818 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile d_worms

Send message
Joined: 9 Feb 12
Posts: 2
Credit: 2,627,308
RAC: 0
Message 70430 - Posted: 23 Jan 2021, 6:00:51 UTC

That what I was looking for thanks




link to website www.whocalledmeoz.info
ID: 70430 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile UnionJack

Send message
Joined: 8 Jan 10
Posts: 21
Credit: 33,211,690
RAC: 0
Message 71261 - Posted: 22 Oct 2021, 14:15:15 UTC - in response to Message 69815.  

| It would still be nice if they would add some kind of direct control on the prefs page. More casual users will just give up and delete the project.

I agree. Other projects have done it, so why not MW@H? It's almost as though the project didn't want us to find these things out without working at it.
Rgds
Peter.
ID: 71261 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3315
Credit: 519,950,357
RAC: 21,982
Message 71264 - Posted: 23 Oct 2021, 10:49:15 UTC - in response to Message 71261.  

| It would still be nice if they would add some kind of direct control on the prefs page. More casual users will just give up and delete the project.

I agree. Other projects have done it, so why not MW@H? It's almost as though the project didn't want us to find these things out without working at it.


I think you are missing their goal...they want to crunch the units using all available cpu cores so they get thru them faster. Limiting your dual core or 12 core pc to only 1 or 2 cpu cores makes the units take longer to run, that defeats the purpose. Since Boinc already offers ways to change those settings the Project is happy as things are, now would it be nice if they added some fine tuning controls...yes it would but it is what it is and as you found out the Boinc Community can usually step in with some help if people just ask.
ID: 71264 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Hurr1cane78

Send message
Joined: 7 May 14
Posts: 57
Credit: 201,121,413
RAC: 24,622
Message 74821 - Posted: 16 Dec 2022, 21:56:25 UTC

my boinc playlist gpu calculations all videos https://www.youtube.com/watch?v=pYVW9oA9yWk&list=PL_r97NUMjf7x4TgZT3bwQYL8KV_Vbo2J8
ID: 74821 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Questions and Answers : Preferences : MilkyWay steals all my CPUs

©2024 Astroinformatics Group