Welcome to MilkyWay@home

Admin Updates Discussion

Message boards : News : Admin Updates Discussion
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 3 · 4 · 5 · 6

AuthorMessage
Link
Avatar

Send message
Joined: 19 Jul 10
Posts: 595
Credit: 18,976,206
RAC: 5,560
Message 77068 - Posted: 14 Apr 2024, 7:58:29 UTC - in response to Message 77067.  
Last modified: 14 Apr 2024, 7:58:42 UTC

In my experience, the BOINC client often resorts to put various tasks into waiting state — or starts tasks from the queue of downloaded tasks in an order which is unexpected to the user — whenever it is faced with a mixture of tasks with various thread counts in its buffer.
That's not just your experience and was the reason for all the requests for a single core version of N-Body in the past years as well as questions about limiting the thread count for N-Body, in particular from people running more than one project at the same time.


"Max # CPUs = 2…256" — The server assigns only tasks for the multi-threaded app_version to the host. Furthermore, the application's built in upper limit of thread count per task is overridden by this setting.
That's easy to check, I've set now my to 2, i.e. what I have in my app_config.xml. If that's how it works, than again, the default should be 16 as it was until now to avoid complanis from users, who were happy with the situation.

On the positive side, with some ST tasks in the mix, the CPU utilization and energy usage is higher while the CPU is still constantly boosting at 4.6 GHz, so the production should be higher with the ST application.
ID: 77068 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3321
Credit: 520,651,133
RAC: 32,692
Message 77069 - Posted: 14 Apr 2024, 10:57:02 UTC - in response to Message 77067.  

Link wrote:
now there are 4 ST tasks waiting to run while 8 2-thread tasks are running. Doesn't make sense at all.
In my experience, the BOINC client often resorts to put various tasks into waiting state — or starts tasks from the queue of downloaded tasks in an order which is unexpected to the user — whenever it is faced with a mixture of tasks with various thread counts in its buffer. Obviously the client has to have a scheduling algorithm which aims among else for good host utilization. The outcome of this algorithm may not always be what the user believes to be expected or optimal.

--------

On the new settings: This is what I *think* what they do:

"Max # jobs" — means "limit of tasks in progress" (or more precisely said, limit of MilkyWay@Home tasks in progress)

"Max # CPUs" — means "thread count per task" (or more precisely said, thread count per MilkyWay@Home task)

"Max # CPUs = "No limit" — The server assigns a mixture of tasks for the single-threaded app_version and for the multithreaded app_version to the host. The choice of thread count of MT tasks is left to the application. (The MT app_version behaves as before the introduction of the Max # CPUs preferences setting). Standard(?) BOINC server behaviour is to watch the computing performance of each app_version on the host, and from some point on assign only tasks of the better performing app_version to the host.

"Max # CPUs = 1" — The server assigns only tasks for the single-threaded app_version to the host.

"Max # CPUs = 2…256" — The server assigns only tasks for the multi-threaded app_version to the host. Furthermore, the application's built in upper limit of thread count per task is overridden by this setting.

(Again, that's only what I *think* how it works.)
On top of that, the user can still override the thread count of the multithreaded app_version by means of app_config.xml, or/and enforce the use of desired application binaries by means of app_info.xml.


Wouldn't it be far easier for MilkyWay to just add a check box for the single and MT tasks, so we can check one or both of them and let Boinc figure it out. That way everyone with an app_config file wouldn't have to do anything at all, except uncheck the ST tasks box if they choose too, and those only wanting the ST tasks can get them.
ID: 77069 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Magenta

Send message
Joined: 14 Oct 23
Posts: 3
Credit: 75,621
RAC: 880
Message 77072 - Posted: 14 Apr 2024, 11:59:39 UTC - in response to Message 77069.  

Wouldn't it be far easier for MilkyWay to just add a check box for the single and MT tasks, so we can check one or both of them

Let's not add more complexity than necessary - both for the users and the administrator. First it should be checked if the mt application can run one thread just like it can run any other number. If it can there's no need to handle two applications and three combinations of those. Okay, you couldn't have different numbers of threads at the same time but I can't think of a good reason why I'd want that anyway.

That way everyone with an app_config file wouldn't have to do anything at all

Wasn't this about getting rid of the need for app_config? With its most popular settings now available through the GUI that's a good opportunity to remove it instead of creating a new use. Many users don't feel comfortable with editing config files or don't even know about them. And I can do with one less place of configuration.
ID: 77072 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Link
Avatar

Send message
Joined: 19 Jul 10
Posts: 595
Credit: 18,976,206
RAC: 5,560
Message 77073 - Posted: 14 Apr 2024, 14:57:06 UTC - in response to Message 77068.  

"Max # CPUs = 2…256" — The server assigns only tasks for the multi-threaded app_version to the host. Furthermore, the application's built in upper limit of thread count per task is overridden by this setting.
That's easy to check, I've set now my to 2, i.e. what I have in my app_config.xml. If that's how it works, than again, the default should be 16 as it was until now to avoid complanis from users, who were happy with the situation.
Seems to work so far, no new ST tasks anymore. Will let it run for a while like that to confirm and than switch to ST tasks and compare performance.
ID: 77073 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Magenta

Send message
Joined: 14 Oct 23
Posts: 3
Credit: 75,621
RAC: 880
Message 77074 - Posted: 14 Apr 2024, 17:27:15 UTC - in response to Message 77072.  

First it should be checked if the mt application can run one thread

It works. I can't comment on the speed though, the tasks are too heterogeneous. Plus no real st tasks for me at this time, despite resetting to "unlimited".
ID: 77074 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
xii5ku

Send message
Joined: 1 Jan 17
Posts: 34
Credit: 100,564,926
RAC: 284,108
Message 77075 - Posted: 15 Apr 2024, 6:49:26 UTC
Last modified: 15 Apr 2024, 7:01:14 UTC

Mixed thread counts of concurrent tasks in the same client:
Link wrote:
That's not just your experience and was the reason for all the requests for a single core version of N-Body in the past years as well as questions about limiting the thread count for N-Body, in particular from people running more than one project at the same time.
If tasks with very differing runtime behaviour come from different projects though, then the user has a few more instruments to deal with this.


app_config.xml and web prefs interaction:
Magenta wrote:
Let's not add more complexity than necessary - both for the users and the administrator.
I agree. Those who write an app_config.xml or app_info.xml generally need to be prepared to watch what goes on at the projects and to adapt when necessary anyway. Since the addition of the new preferences items here at MW@H, app_config.xml owners can simply either switch the new prefs once such that they best work with their app_config.xml, or opt to remove their apt_config.xml for good. One hurdle is that the new options aren't named very intuitively, but at least they are the same as at some other projects with multithreaded applications, apparently. (Just the 1T + MT mix in case of #CPUs=Unlimited seems to be unique.)

PS,
@Kevin Roux, thanks for the continuous care and improvements to the project.
ID: 77075 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Sutaru Tsureku

Send message
Joined: 30 Apr 09
Posts: 101
Credit: 29,871,946
RAC: 288
Message 77080 - Posted: 18 Apr 2024, 16:17:37 UTC - in response to Message 77062.  

I guess the following settings should be for just the multi thread app on a 4 thread CPU:

Maximale Anzahl Aufgaben 1 (max. tasks)
Maximale Anzahl CPUs 4 (max. threads)

I test this now.
Is it working as expected? Because "Max # of jobs for this project" isn't very specific, is it like max_concurrent in the app_info.xml or is it max task on the computer from this project, i.e. something like WCG has for each of their subprojects? Also "Max # of CPUs for this project" is pretty much wrong if it's actually "Max # of threads per task for this project".


The standard settings should be max. tasks 1 and max. threads no limit and not at both no limit.
The standard settings should be no limit for number of max. concurrent tasks and 16 for number of threads, i.e. what we had until now, than no one would complain about changes they didn't ask for. ;-)

And than of course, the ST application should only be send to users, who set 1 as limit for threads (or have a single core CPU or allow BOINC to use just one core) and everyone else should get only MT application. The issue with the current implementation is that it's "max" x threads and not like for example on PrimeGrid "exactly" x threads.

On my new Ryzen with 16 threads, on which I set "--nthreads 2" in the app_info.xml, so far BOINC decided to skip all single thread tasks and when it finishes a 2-core task, it starts again a 2-core tasks instead of 2 single core tasks which actually should run first. Let's see what it will do when it runs out of 2 core tasks.


It's nice that you are still also BOINC-active like me :D after SETI@home closed :(

For a perfectionist like me, it don't work so well... ;)

I have an app_config.xml with <cmdline>--nthreads 4</cmdline>. (on a 4 core (thread) CPU ;)
The above project preferences set. 1 / 4
Maximale Anzahl Aufgaben	1
Maximale Anzahl CPUs		4


and:
Startet nur die ausgewählte Anwendung/en	Milkyway@home N-Body Simulation: nein
						Milkyway@home N-Body Simulation with Orbit Fitting: ja
Wenn keine Aufgaben für die ausgewählten Anwendungen vorhanden sind, Aufgaben von anderen Anwendungen akzeptieren?	yes


My WU cache is 0 / 0 days and additional days.

Because of this WU cache, BOINC ask for new WU around 2 mins. before the WU will finish.
This happened:
------------------------------------------------------------------------------------------
14.04.24 | 02:37:33 Sending scheduler request: To fetch work.
14.04.24 | 02:37:33 Requesting new tasks for CPU
14.04.24 | 02:37:36 Scheduler request completed: got 0 new tasks
14.04.24 | 02:37:36 No tasks sent
14.04.24 | 02:37:36 No tasks are available for Milkyway@home N-Body Simulation with Orbit Fitting
14.04.24 | 02:37:36 This computer has reached a limit on tasks in progress

14.04.24 | 02:37:36 Project requested delay of 91 seconds
14.04.24 | 02:39:27 Sending scheduler request: To fetch work.
14.04.24 | 02:39:27 Reporting 1 completed tasks
14.04.24 | 02:39:27 Requesting new tasks for CPU
14.04.24 | 02:39:30 Scheduler request completed: got 1 new tasks
14.04.24 | 02:39:30 Project requested delay of 91 seconds
------------------------------------------------------------------------------------------

For a perfectionist, the red lines shouldn't happen. ;)
The CPU idled a few seconds, because BOINC got not a WU after the 1. work request. :(
ID: 77080 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Link
Avatar

Send message
Joined: 19 Jul 10
Posts: 595
Credit: 18,976,206
RAC: 5,560
Message 77081 - Posted: 18 Apr 2024, 18:43:46 UTC - in response to Message 77080.  

For a 4 core / 4 threads CPU you don't need app_config with <cmdline>--nthreads 4</cmdline>, it does exactly nothing, just like I didn't need <cmdline>--nthreads 2</cmdline> on my Core 2 Duo, I used that however on my new Ryzen until it became possible to configure that on the server, now it's removed from it too, only <fraction_done_exact/> left for both applications and everything works as it should.

No idea, why you have no cache, that will always result in some idle time, not every scheduler request is successful. I think you should be used to have some cache after all those years of SETI. ;-)

And than of course you should have "no limit" for number of tasks and 4 CPUs, than everything will work as you want it to.

But good to know, how "Max # jobs" work, thanks for posting it here.
ID: 77081 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Link
Avatar

Send message
Joined: 19 Jul 10
Posts: 595
Credit: 18,976,206
RAC: 5,560
Message 77098 - Posted: 24 Apr 2024, 15:21:02 UTC

Kevin Roux wrote:
After some testing, the single-threaded executable and multi-threaded executables running on one thread have very little difference in between them with the MT executable even being slightly faster in most cases
Thanks for testing this.
ID: 77098 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
magic_sam

Send message
Joined: 8 Nov 22
Posts: 3
Credit: 4,580,769
RAC: 44,300
Message 77099 - Posted: 24 Apr 2024, 18:19:31 UTC

Hello all,

Looks like "Max # of simultaneous MilkyWay@home tasks" actually means "Total number of tasks, running and cached"...

I set my preferences to:

Max # of simultaneous Milkyway@home tasks 7
Max # of threads for each Milkyway@home task 4

In order to have 28 cores dedicated to Milkyway@home.

I sometimes get the same results as Sutaru Tsureku, i.e: "This computer has reached a limit on tasks in progress".

BOINC (v.8.0.1) is not caching any work for Milkyway@home, even though it should:

Store at least 0.2 days of work
Store up to an additional 0.3 days of work

What am I doing wrong ?

Cheers, Sam
ID: 77099 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Link
Avatar

Send message
Joined: 19 Jul 10
Posts: 595
Credit: 18,976,206
RAC: 5,560
Message 77100 - Posted: 24 Apr 2024, 19:00:08 UTC - in response to Message 77099.  
Last modified: 24 Apr 2024, 19:04:50 UTC

Max # of simultaneous Milkyway@home tasks 7
Set this to "no limit". This setting is actually intended for projects with applications, which for example use a lot of RAM and running more than x of them simultaneously might cause issues. No need for that here for the average user, the cache settings alone will make sure, that you have your 0.2+0.3 days of work in your cache if that's what you want (I however suggest changing it to something like 0.49+0.01 or simply to 0.5+0.0, in particular if you are running more than one project simultaneously).
ID: 77100 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Bill F
Avatar

Send message
Joined: 4 Jul 09
Posts: 87
Credit: 16,787,235
RAC: 2,905
Message 77101 - Posted: 24 Apr 2024, 19:21:48 UTC - in response to Message 77100.  

Max # of simultaneous Milkyway@home tasks 7
Set this to "no limit". This setting is actually intended for projects with applications, which for example use a lot of RAM and running more than x of them simultaneously might cause issues. No need for that here for the average user, the cache settings alone will make sure, that you have your 0.2+0.3 days of work in your cache if that's what you want (I however suggest changing it to something like 0.49+0.01 or simply to 0.5+0.0, in particular if you are running more than one project simultaneously).



Link

From above your suggestion of ....simply to 0.5+0.0, if you are running more than one project simultaneously).

Would your advice change on the mix of programs or the power / resources of the systems attached ? I run Windows and Android only and I am currently set at 0.2 days and 1

Thanks
Bill
In October of 1969 I took an oath to support and defend the Constitution of the United States against all enemies, foreign and domestic;
There was no expiration date.


ID: 77101 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Link
Avatar

Send message
Joined: 19 Jul 10
Posts: 595
Credit: 18,976,206
RAC: 5,560
Message 77102 - Posted: 25 Apr 2024, 8:44:22 UTC - in response to Message 77101.  
Last modified: 25 Apr 2024, 9:04:33 UTC

Would your advice change on the mix of programs or the power / resources of the systems attached ?
Ressource share between projects is a different setting, my advice will only make it easier for the BOINC client to follow that setting. It might change it in case your additional 1 day made it hard for your BOINC client to follow your resource share setting by forcing it into "high priority" mode. So, yes, the result might look a bit like a change of ressource share, but it isn't.

In general you should simply set the cache to what you want to have it, "additional days" are only to limit the amount of scheduler requests and are basically a leftover from dial-up days and old BOINC versions, which reported completed tasks up to 24 hours after file upload. With current versions, which report latest 1 hour after upload, they don't really make much sense. Small values, which equal 1-2 hours can be used to avoid unnecessary load on project servers, in particular if running only one project nothing can go wrong with that, but if running more projects, this will result in projects reporting completed tasks filling up the cache and projects, which actually should request work, not requesting it because there's enough in the cache, in particular with settings like yours, where the additional days are several times of the actuall cache settings.
ID: 77102 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 . . . 3 · 4 · 5 · 6

Message boards : News : Admin Updates Discussion

©2024 Astroinformatics Group