Welcome to MilkyWay@home

Message from server: double precision GPU required

Message boards : Number crunching : Message from server: double precision GPU required
Message board moderation

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
Zydor
Avatar

Send message
Joined: 24 Feb 09
Posts: 620
Credit: 100,587,625
RAC: 0
Message 46541 - Posted: 11 Mar 2011, 16:50:06 UTC - in response to Message 46540.  
Last modified: 11 Mar 2011, 17:16:46 UTC

Bit more than that - about 2/3%, which is expected as a CPU app running full tilt on 100% utilisation will slow a GPU slightly when the GPU asks for help (say loading / unloading data to GPU), that can cause a very small brief wait state in the GPU whilst the CPU finishes what its doing before going to help the GPU.

Its also mindful to keep an eye on the needs for GPU apps. For example, the Collatz app by default takes up 30% of a GPUs memory - each GPU, not total for all GPUs. The same size of memory space is needed in main PC memory to load and unload the data to and from that 30% of GPU memory. This means those with multiple GPUs, and "only" 2Gb main memory, need to do a quick check on whats left for standard domestic use if they run multiple GPUs - the load on main memory quickly adds up.

Not normaly an issue on 4Gb machines, but it can be a problem on 2Gb machines depending on what applications are running, and what domestic memory needs are. It applies to all GPU applications (quantity of main memory needed as a percentage of GPU memory will vary from application to application), Collatz was just an example, not indicating any bad aspect of Collatz - in fact Collatz is one of the better GPU apps out there.

Just need to be aware these days of real GPU needs, as more is done on PCs, and more have multiple GPUs. The memory effect on main memory was never an issue with single GPUs, so was not real visible as a potential issue. Doesnt happen a lot even now, but its sufficiently "near the wire" these days to start thinking about it and highlighting this behaviour, as it eats PC main memory sometimes without the User realising it.

EDIT:
Just realised, sorry that was my typing - by 2/3% I meant 2 or 3% - wasnt the best way to express it - my bad :)
Regards
Zy
ID: 46541 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile kashi

Send message
Joined: 30 Dec 07
Posts: 311
Credit: 149,490,184
RAC: 0
Message 46545 - Posted: 12 Mar 2011, 3:25:51 UTC

There are other things you can try but with your current configuration you probably need to disable your onboard graphics. BOINC is trying to run 2 tasks and when the onboard graphics is detected as not double precision capable ("Device 0 not available or not supported") then 2 tasks are being run concurrently on your HD 5870.

In my opinion disabling onboard graphics completely would be the best option for your configuration. This would free up some memory. I don't know how much memory current Einstein tasks use, but running 6 Einstein tasks, 2 MilkyWay tasks concurrently and supporting onboard graphics may mean that the 3GB you have is not enough. Even when Task Manager shows there is free memory it is possible for GPU crunching to need more memory.

If you then have lag problems you could install the MilkyWay Anonymous platform version and adjust the parameters in the app_info.xml file to minimise it. Or you could instead disable onboard graphics in BOINC only with a cc_config.xml file if you wanted to try it. <ignore_ati_dev>0</ignore_ati_dev>

The other thing that could be mucking things up is the Catalyst driver causing your HD 5870 to go into power saving mode. This would most likely happen when you turned off the screen. This is more likely to give problems when there are 2 GPUs in a computer. It can be usually be cured by a process of trial and error. In my case I use a dummy plug and change EnableUlps from 1 to 0 in the registry, but there are other ways that others use. I use Windows 7 and you use Windows XP so you may need a different fix. Disabling onboard graphics completely may be a lot easier and quicker way to fix this if it is happening on your computer.

You haven't mentioned whether it happens when you turn off the screen but if it does happen then, a quick fix until you can find a better solution is to use a blank screensaver and not turn off the screen.
ID: 46545 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Sunny129
Avatar

Send message
Joined: 25 Jan 11
Posts: 271
Credit: 346,072,284
RAC: 0
Message 46582 - Posted: 17 Mar 2011, 3:42:21 UTC - in response to Message 46545.  
Last modified: 17 Mar 2011, 3:43:30 UTC

I don't know how much memory current Einstein tasks use, but running 6 Einstein tasks, 2 MilkyWay tasks concurrently and supporting onboard graphics may mean that the 3GB you have is not enough. Even when Task Manager shows there is free memory it is possible for GPU crunching to need more memory.

i don't know how much system memory either an Einstein GC task or an Einstein BRPS task uses. likewise, i don't know how much GPU memory a MW@H task uses, and therefore i don't know how much system memory a MW@H task will use when loading/unloading data from the GPU to the system memory and vice versa. however, i do believe it was a memory problem all along. while i do have 4GB of system memory, i'm running WinXP 32-bit, and so only ~3GB is recognized. since then, i have scaled back the number of CPU cores usable by BOINC from 6 to 5. this way there is always a free core available to help with loading/unloading data from the GPU to the system memory and vice versa, and these transfers won't slow to a crawl b/c the free CPU core won't be crunching an Einstein task. at any rate, it seems to be working - i haven't had any connectivity issues since then.

If you then have lag problems you could install the MilkyWay Anonymous platform version and adjust the parameters in the app_info.xml file to minimise it. Or you could instead disable onboard graphics in BOINC only with a cc_config.xml file if you wanted to try it. <ignore_ati_dev>0</ignore_ati_dev>

i've already tried <ignore_ati_dev>0</ignore_ati_dev> in an attempt to fix a different problem that i'm troubleshooting on the SETI@Home message boards. while its fixed some problems and created others, it did nothing for my connectivity issues. but those seem to be fixed now anyways. i'll update the thread if this problem comes back.

thanks for everyone's help,
Eric
ID: 46582 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Matt Arsenault
Volunteer moderator
Project developer
Project tester
Project scientist

Send message
Joined: 8 May 10
Posts: 576
Credit: 15,979,383
RAC: 0
Message 46587 - Posted: 17 Mar 2011, 14:18:56 UTC - in response to Message 46582.  

I don't know how much memory current Einstein tasks use, but running 6 Einstein tasks, 2 MilkyWay tasks concurrently and supporting onboard graphics may mean that the 3GB you have is not enough. Even when Task Manager shows there is free memory it is possible for GPU crunching to need more memory.
Task manager shows you nothing about GPU memory usage. You need to find special tools to do that, and last time I looked for something to do that for ATI, I only found people saying the ATI driver for the last few years doesn't let you get GPU memory usage information.

...i don't know how much GPU memory a MW@H task uses, and therefore i don't know how much system memory a MW@H task will use when loading/unloading data from the GPU to the system memory and vice versa.
A GPU separation workunit should need (maybe 2) * mu * (number streams + 1) * sizeof(double) bytes. For a typical workunit, this would be something like (1400 * 1600 * (3 + 1) * 8 / (1024^2) ~= 70 (maybe 140) MB of GPU= memory. System memory is hardly needed. The CPU (separation) workunits use next to nothing. Nbodies require more memory, but at most should be around 15MB (for now at least).

ID: 46587 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Sunny129
Avatar

Send message
Joined: 25 Jan 11
Posts: 271
Credit: 346,072,284
RAC: 0
Message 46590 - Posted: 17 Mar 2011, 14:51:34 UTC - in response to Message 46587.  

I don't know how much memory current Einstein tasks use, but running 6 Einstein tasks, 2 MilkyWay tasks concurrently and supporting onboard graphics may mean that the 3GB you have is not enough. Even when Task Manager shows there is free memory it is possible for GPU crunching to need more memory.
Task manager shows you nothing about GPU memory usage. You need to find special tools to do that, and last time I looked for something to do that for ATI, I only found people saying the ATI driver for the last few years doesn't let you get GPU memory usage information.

i don't think he is implying that the task manager does tell you anything about GPU memory usage. rather i think the point he is trying to make is that the task manager tells you how much system memory is left unused (available), and that the specific amount of system memory available after taking into account the amount being used to crunch 6 E@H CPU tasks simultaneously might not be enough to handle the additional (yet only occasional) GPU data loading/unloading whenever a MW@H task starts or finishes.


A GPU separation workunit should need (maybe 2) * mu * (number streams + 1) * sizeof(double) bytes. For a typical workunit, this would be something like (1400 * 1600 * (3 + 1) * 8 / (1024^2) ~= 70 (maybe 140) MB of GPU= memory. System memory is hardly needed. The CPU (separation) workunits use next to nothing. Nbodies require more memory, but at most should be around 15MB (for now at least).

first i'd have to confirm how much system memory my 6 E@H CPU tasks are consuming and note how much is left unused (still available). using that number, i can calculate whether or not i have enough system memory left over to handle a MW@H GPU task or two. in the mean time, like i said previously, ever since i scaled down the number of CPU cores running E@H tasks from 6 to 5, i haven't had any BOINC GUI or connectivity issues. so while i haven't done exact calculations to be sure, i'm fairly confident that my problem had something to do with trying to use more memory than i had available.
ID: 46590 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile kashi

Send message
Joined: 30 Dec 07
Posts: 311
Credit: 149,490,184
RAC: 0
Message 46592 - Posted: 17 Mar 2011, 18:37:23 UTC - in response to Message 46590.  

Yes that is correct, all I meant was that a shortage of system memory can still exist and affect GPU processing even though Task Manager shows that there is unused memory still available.

The reason I mentioned it, is that a number of people have had errors running Collatz GPU when the application was unable to create a buffer. When I suggested a way to reduce the memory used by the application because they had insufficient memory to run Collatz some replied that Task Manager showed that they had ample memory left. Example

This is different to the problem you experienced. I'm not certain in your case that it is definitely only memory related. There could be contention issues relating to computer resources other than just memory causing BOINC Manager to lose contact with boinc.exe.

I noticed a large slowdown in MilkyWay GPU processing speed when I ran Einstein on all 8 cores. This was shortly after the MilkyWay GPU application was first introduced. Since that time I have always left one or two CPU cores free to support GPU processing depending on how many GPU cores I had running.

I thought of this with your problem but was reluctant to suggest running CPU projects on less than the full number of CPU cores because some find this concept objectionable. Depending on the number of CPU cores and the particular project it may not actually result in much reduction in total CPU project throughput. This is because the CPU tasks will often be completed more quickly. The GPU processing usually speeds up too, particularly for those with multiple GPUs.

Glad you have it sorted out.
ID: 46592 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Sunny129
Avatar

Send message
Joined: 25 Jan 11
Posts: 271
Credit: 346,072,284
RAC: 0
Message 46594 - Posted: 17 Mar 2011, 20:03:16 UTC - in response to Message 46592.  

Yes that is correct, all I meant was that a shortage of system memory can still exist and affect GPU processing even though Task Manager shows that there is unused memory still available.

i wonder - if we keep a close eye on the task manager during the brief moment that an GPU task loads from system memory to the GPU (or during the brief moment that a GPU task unloads from the GPU to the system memory), will the task manager actually show the brief reduction in available system memory? in other words, does the amount of system memory allocated to the above events (whether a task is just starting, just finishing, or suspending/restarting "mid-crunch") manifest itself as a brief reduction in available system memory visible through the system tray? or must we leave a "buffer" b/c that brief reduction in available system memory never shows up in the task manager?


I thought of this with your problem but was reluctant to suggest running CPU projects on less than the full number of CPU cores because some find this concept objectionable. Depending on the number of CPU cores and the particular project it may not actually result in much reduction in total CPU project throughput. This is because the CPU tasks will often be completed more quickly. The GPU processing usually speeds up too, particularly for those with multiple GPUs.

Glad you have it sorted out.

i'll take any suggestion i can get when it comes to sorting out problems w/ BOINC or individual projects/applications. and i know exactly what you mean about hesitating to suggest using one less CPU core...its a tough decision trying to decide whether to use all CPU cores or to free 1 or 2 cores up for GPU assistance, especially if you haven't yet experimented with the particular host in question to see which is the most efficient mix of DC applications and resources.

given that i haven't gotten that far with experimentation yet, here's how i look at it: going from 6 CPU cores to 5 CPU cores will reduce the crunching efficiency of E@H CPU tasks by ~17%. but if that shaves a few seconds off of the average GPU task (whether S@H or MW@H), it may more than make up for any production lost to disabling 1 CPU core in BOINC. in addition, GPU tasks of a particular project typically take far less time to finish than their CPU task counterparts...granted i'm not comparing GPU tasks with CPU tasks of the same project since i'm running E@H on the CPU, and S@H & MW@H on the GPU...but i think i have a valid point with respect to crunching efficiency in general. the only downside to this way of looking at things is that no two DC projects award points the same way - i could do 10 hours worth of S@H GPU tasks and MW@H GPU tasks each, and typically i'll wind up with FAR more credit for the 10 hours of MW@H work than i would for the 10 hours of S@H work. fortunately i'm in it for the science, not the BOINC credit or the points standings.
ID: 46594 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile kashi

Send message
Joined: 30 Dec 07
Posts: 311
Credit: 149,490,184
RAC: 0
Message 46605 - Posted: 18 Mar 2011, 3:09:44 UTC - in response to Message 46594.  

Yes considering the daily throughput of CPU only, what I was suggesting is you will possibly discover that the loss is less than 17%. This is particularly so when the CPU is also supporting GPU processing.

In the case of Intel CPUs with 4 physical cores and 8 virtual cores (hyperthreading), the loss from using only 7 cores is sometimes much less than 12.5%. This varies greatly with different project applications. There was one particular PrimeGrid application where the total throughput was similar for my CPU when using only 4 cores with hyperthreading turned off as it was when running on all 8 cores with hyperthreading turned on. In other words the tasks took almost exactly twice as long to complete running on 8 cores as they did running on 4.

This speed up of individual task processing when using less than the full number of cores may not happen as much on your 6 core AMD CPU however.

Depending on the CPU architecture and operating system, efficiency can sometimes be improved by running a mix of different CPU projects at once or assigning a task to a particular core so it doesn't jump around from core to core. Einstein is one project that has been shown to benefit from such experimentation. Assigning tasks to a particular core would mainly benefit CPUs with hyperthreading though.

You would be aware of these issues, just letting you know that Einstein processing efficiency can benefit from experimentation when you get around to it. There is some information about this topic on the Einstein forum.
ID: 46605 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 · 2

Message boards : Number crunching : Message from server: double precision GPU required

©2024 Astroinformatics Group