WU: de_separation_23_3s
GPU: NVIDIA GTX 260

Looks like the problem with GTX2xx and WUs de_separation_23_3s Matt is aware of and going to fix in the next version.
Other WUs should run on your GPU.
The only WU with error I still could find in your list is Workunit 228247254 and there is an ATI card (HD5850?) that finished the WU and is waiting for validation (not chrashed).

<core_client_version>6.10.17</core_client_version>
<![CDATA[
<message>
Unzulässige Funktion. (0x1) - exit code 1 (0x1)
</message>
<stderr_txt>
<search_application> milkywayathome separation 0.50 Windows x86 double OpenCL </search_application>
Found 1 platforms
Platform 0 information:
  Platform name:       NVIDIA CUDA
  Platform version:    OpenCL 1.0 CUDA 3.2.1
  Platform vendor:     
  Platform profile:    
  Platform extensions: cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing cl_nv_d3d9_sharing cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll 
Using device 0 on platform 0
Found 2 CL devices
Device GeForce GTX 260 (NVIDIA Corporation:0x10de)
Type:                CL_DEVICE_TYPE_GPU
Driver version:      266.58
Version:             OpenCL 1.0 CUDA
Compute capability:  1.3
Little endian:       CL_TRUE
Error correction:    CL_FALSE
Image support:       CL_TRUE
Address bits:        32
Max compute units:   27
Clock frequency:     1104 Mhz
Global mem size:     939327488
Max mem alloc:       234831872
Global mem cache:    0
Cacheline size:      0
Local mem type:      CL_LOCAL
Local mem size:      16384
Max const args:      9
Max const buf size:  65536
Max parameter size:  4352
Max work group size: 512
Max work item dim:   3
Max work item sizes: { 512, 512, 64 }
Mem base addr align: 2048
Min type align size: 128
Timer resolution:    1000 ns
Double extension:    MW_CL_KHR_FP64
Extensions:          cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing cl_nv_d3d9_sharing cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll  cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_fp64 
Found a compute capability 1.3 device. Using -cl-nv-maxrregcount=32 

Compiler flags:
-cl-mad-enable -cl-no-signed-zeros -cl-strict-aliasing -cl-finite-math-only -DUSE_CL_MATH_TYPES=0 -DUSE_MAD=1 -DUSE_FMA=0 -cl-nv-verbose  -cl-nv-maxrregcount=32  -DDOUBLEPREC=1 -DMILKYWAY_MATH_COMPILATION -DNSTREAM=3 -DFAST_H_PROB=1 -DAUX_BG_PROFILE=0 -DUSE_IMAGES=1 -DI_DONT_KNOW_WHY_THIS_DOESNT_WORK_HERE=0  

Build status: CL_BUILD_SUCCESS
Build log: 

: Considering profile 'compute_13' for gpu='sm_13' in 'cuModuleLoadDataEx_4'
: Retrieving binary for 'cuModuleLoadDataEx_4', for gpu='sm_13', usage mode='  --verbose --maxrregcount 32  '
: Considering profile 'compute_13' for gpu='sm_13' in 'cuModuleLoadDataEx_4'
: Control flags for 'cuModuleLoadDataEx_4' disable search path
: Ptx binary found for 'cuModuleLoadDataEx_4', architecture='compute_13'
: Ptx compilation for 'cuModuleLoadDataEx_4', for gpu='sm_13', ocg options='  --verbose --maxrregcount 32  '
ptxas info    : Compiling entry function 'mu_sum_kernel' for 'sm_13'
ptxas info    : Used 32 registers, 800+0 bytes lmem, 48+16 bytes smem, 56 bytes cmem[1], 4 bytes cmem[2], 4 bytes cmem[3], 4 bytes cmem[4], 4 bytes cmem[5], 4 bytes cmem[6]
Kernel work group info:
  Work group size = 512
  Kernel local mem size = 64
  Compile work group size = { 0, 0, 0 }
Group size = 64, per CU = 8, threads per CU = 512
Block size = 13824
Desired = 163
Min sol: 163 13312
Lower n solution: n = 163, x = 13312
Higher n solution: n = 163, x = 13312
Using solution: n = 163, x = 13312
Range:          { nu_steps = 640, mu_steps = 1600, r_steps = 1400 }
Iteration area: 2240000
Chunk estimate: 163
Num chunks:     163
Added area:     13312
Effective area: 2253312
Integration time: 957.124801 s. Average time per iteration = 1495.507502 ms
Kernel work group info:
  Work group size = 512
  Kernel local mem size = 64
  Compile work group size = { 0, 0, 0 }
Group size = 64, per CU = 8, threads per CU = 512
Block size = 13824
Desired = 21
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Min sol: 1 0
Didn't find a solution. Using fallback solution n = 20, x = 0
Using solution: n = 20, x = 0
Range:          { nu_steps = 160, mu_steps = 400, r_steps = 700 }
Iteration area: 280000
Chunk estimate: 21
Num chunks:     20
Added area:     0
Effective area: 280000
Global dimensions not divisible by local
Failed to find good run sizes
Failed to calculate integral 1
12:30:48 (2372): called boinc_finish

</stderr_txt>
]]>

The memory in the GPU was not being initialized properly, I have to install Visual Studio on Windows so it may take a little longer than expected.

The bug has been identified and fixed. It is currently going through testing and a new version should be up sometime within the next 48 hours (pending the results of the test, and it will most likely be within an hour, but I like to give myself a larger window for unforeseable problems). The new version also contains new performance enhancements that give a 5-10% decrease in running times.

Thank you for your patience.

I am beginning to think there is something very wrong with 195.62.

I keep loosing GPUs. 4 to 3 to 2 after a WU crash. Reboot and now I see only 3. So I tried booting a Vista partion with 185.xx drivers. 4 GPUs and I can blast Furmark with no issues nor GPU failures.

So on my part I am thinking of winding back a few driver versions and see if my WU pass and stop loosing GPUs after a crash. Its so bad it needs a power off rather than a reboot to free them back up.

Also my 260 is pretty much erroring 3 out of 4 units and thats up on 195.62 as well.

So whilst its not just that driver version as others are seeing the WU errors on other versions for me at least 195.62 is not a good place to be.

Is this still an issue?

If so please let me know how much Video RAM the graphics card has. The larger work units appear to be using 312MB of Video RAM.

I don't know the technical details of how GPU applications use system memory or use it to remap the video memory, I just know that they do. Errors were reported a while ago with the Collatz ATI application from people who were running multiple 1GB graphics memory ATI GPUs and had only 1 GB of system memory. More recently someone with only 2GB of memory also got those buffer errors when trying to run 2 HD 5970 cards. The solution in both cases was to limit the amount of video memory used by the application with the use of the r parameter.

I know Collatz ATI is a different application from MilkyWay ATI or CUDA and is more dependent on video card memory bandwidth. However system memory would still need to be used and when the MilkyWay WU is 4 times longer it possiubly uses more than previously.

Because I had noticed that some who reported no problems even with multiple nVidia cards had 8GB or 12GB of system memory and some of those who were reporting problems had 3GB reported as available or 4GB available with multiple GPU cores I reasoned that memory buffer issues may be causing a problem. Could be wrong, just raising the possibility to try and help narrow down what is causing these errors for some while others are unaffected.

Therefore it is possible that those who have a large amount of system memory installed and use a Windows 64-bit operating system may not encounter these errors. Also graphics card memory needs to be remapped to system memory so multiple cards could still cause a problem even if one of them is not being used for MilkyWay CUDA processing.

Dude, there's a config file you can create to track operations of WUs, but I've never used it. This is the link:

Client Configuration

[Milkyway@home] [coproc_debug] Assigning CUDA instance 0 to de_s222_3s_best_1p_01r_41_701094_1259781480_0

My box is only like 3 or 4 months old...
asus a6t motherboard Intel I7 920 2.67 o/c to 3.4 and I have 12gigs of DDR3 ram
the video card are bothe evga gtx260-216 core factory o/c which I have o/c even more.

I've been following this thread and I'm just tossing up ideas...

What is the physical set up for the computers that return invalid wu's? Could it be that the power supply is a little light? Is there an issue with the BIOS of the motherboard so the rev needs updating?

It is possible that this is a memory buffer issue. If so those with a smaller amount of system memory or less bandwidth on an older architecture and/or multiple nVidia GPU cores would be more likely to experience the issue than those with a larger amount of memory/newer system and/or only a single nVidia GPU core.

I have no experience with nVidia cards, this is only theory, it would need to be tested by someone who has the errors either installing more memory or only running one nVidia GPU per box and seeing if it fixed it. I suppose you could also test it by swapping an nVidia card that is erroring to another system that has more available memory.

These are the clock speeds for a GTX260 from nVidia site specs
Graphics 576
Processor 1242
Memory 999
hope it helps
;-p