Welcome to MilkyWay@home

RX6600/Ubuntu 22.10 > Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY

Questions and Answers : Unix/Linux : RX6600/Ubuntu 22.10 > Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY
Message board moderation

To post messages, you must log in.

AuthorMessage
alex

Send message
Joined: 1 Oct 14
Posts: 3
Credit: 19,130,914
RAC: 4,492
Message 74927 - Posted: 19 Jan 2023, 22:28:35 UTC
Last modified: 19 Jan 2023, 22:45:36 UTC

Hi guys,

I very much would like to contribute to the project GPU-wise. But I saw in the other recent post that most likely it is a general issue for Radeon RX 6000 series apparently due to usage of ROCR. I've bought this card recently and have the latest drivers of course.

Are there any plans to fix it in the near future?

I get this error for each GPU task, although Einstein and PrimeGrid work fine:

<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 250 (0xfa, -6)</message>
<stderr_txt>
<search_application> milkyway_separation 1.46 Linux x86_64 double OpenCL </search_application>
Reading preferences ended prematurely
BOINC GPU type suggests using OpenCL vendor 'Advanced Micro Devices, Inc.'
Setting process priority to 0 (13): Permission denied
Error loading Lua script 'astronomy_parameters.txt': [string "number_parameters: 4..."]:1: '<name>' expected near '4'
Switching to Parameter File 'astronomy_parameters.txt'
<number_WUs> 5 </number_WUs>
<number_params_per_WU> 20 </number_params_per_WU>
Using AVX path
Found 1 platform
Platform 0 information:
Name: AMD Accelerated Parallel Processing
Version: OpenCL 2.1 AMD-APP (3513.0)
Vendor: Advanced Micro Devices, Inc.
Extensions: cl_khr_icd cl_amd_event_callback
Profile: FULL_PROFILE
Using device 0 on platform 0
Found 1 CL device
Device 'gfx1032' (Advanced Micro Devices, Inc.:0x1002) (CL_DEVICE_TYPE_GPU)
Board: AMD Radeon RX 6600
Driver version: 3513.0 (HSA1.1,LC)
Version: OpenCL 2.0
Compute capability: 0.0
Max compute units: 14
Clock frequency: 2750 Mhz
Global mem size: 8573157376
Local mem size: 65536
Max const buf size: 7287183768
Double extension: cl_khr_fp64
Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY
Error getting device and context (-6): CL_OUT_OF_HOST_MEMORY
Failed to calculate likelihood
Using AVX path
Found 1 platform
Platform 0 information:
Name: AMD Accelerated Parallel Processing
Version: OpenCL 2.1 AMD-APP (3513.0)
Vendor: Advanced Micro Devices, Inc.
Extensions: cl_khr_icd cl_amd_event_callback
Profile: FULL_PROFILE
Using device 0 on platform 0
Found 1 CL device
Device 'gfx1032' (Advanced Micro Devices, Inc.:0x1002) (CL_DEVICE_TYPE_GPU)
Board: AMD Radeon RX 6600
Driver version: 3513.0 (HSA1.1,LC)
Version: OpenCL 2.0
Compute capability: 0.0
Max compute units: 14
Clock frequency: 2750 Mhz
Global mem size: 8573157376
Local mem size: 65536
Max const buf size: 7287183768
Double extension: cl_khr_fp64
Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY
Error getting device and context (-6): CL_OUT_OF_HOST_MEMORY
Failed to calculate likelihood
Using AVX path
Found 1 platform
Platform 0 information:
Name: AMD Accelerated Parallel Processing
Version: OpenCL 2.1 AMD-APP (3513.0)
Vendor: Advanced Micro Devices, Inc.
Extensions: cl_khr_icd cl_amd_event_callback
Profile: FULL_PROFILE
Using device 0 on platform 0
Found 1 CL device
Device 'gfx1032' (Advanced Micro Devices, Inc.:0x1002) (CL_DEVICE_TYPE_GPU)
Board: AMD Radeon RX 6600
Driver version: 3513.0 (HSA1.1,LC)
Version: OpenCL 2.0
Compute capability: 0.0
Max compute units: 14
Clock frequency: 2750 Mhz
Global mem size: 8573157376
Local mem size: 65536
Max const buf size: 7287183768
Double extension: cl_khr_fp64
Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY
Error getting device and context (-6): CL_OUT_OF_HOST_MEMORY
Failed to calculate likelihood
Using AVX path
Found 1 platform
Platform 0 information:
Name: AMD Accelerated Parallel Processing
Version: OpenCL 2.1 AMD-APP (3513.0)
Vendor: Advanced Micro Devices, Inc.
Extensions: cl_khr_icd cl_amd_event_callback
Profile: FULL_PROFILE
Using device 0 on platform 0
Found 1 CL device
Device 'gfx1032' (Advanced Micro Devices, Inc.:0x1002) (CL_DEVICE_TYPE_GPU)
Board: AMD Radeon RX 6600
Driver version: 3513.0 (HSA1.1,LC)
Version: OpenCL 2.0
Compute capability: 0.0
Max compute units: 14
Clock frequency: 2750 Mhz
Global mem size: 8573157376
Local mem size: 65536
Max const buf size: 7287183768
Double extension: cl_khr_fp64
Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY
Error getting device and context (-6): CL_OUT_OF_HOST_MEMORY
Failed to calculate likelihood
Using AVX path
Found 1 platform
Platform 0 information:
Name: AMD Accelerated Parallel Processing
Version: OpenCL 2.1 AMD-APP (3513.0)
Vendor: Advanced Micro Devices, Inc.
Extensions: cl_khr_icd cl_amd_event_callback
Profile: FULL_PROFILE
Using device 0 on platform 0
Found 1 CL device
Device 'gfx1032' (Advanced Micro Devices, Inc.:0x1002) (CL_DEVICE_TYPE_GPU)
Board: AMD Radeon RX 6600
Driver version: 3513.0 (HSA1.1,LC)
Version: OpenCL 2.0
Compute capability: 0.0
Max compute units: 14
Clock frequency: 2750 Mhz
Global mem size: 8573157376
Local mem size: 65536
Max const buf size: 7287183768
Double extension: cl_khr_fp64
Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY
Error getting device and context (-6): CL_OUT_OF_HOST_MEMORY
Failed to calculate likelihood
00:22:59 (590444): called boinc_finish(-6)
ID: 74927 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
magic_sam

Send message
Joined: 8 Nov 22
Posts: 2
Credit: 2,354,264
RAC: 34,697
Message 75075 - Posted: 21 Feb 2023, 16:24:22 UTC

Hi,

Please have a look at this other thread:

https://milkyway.cs.rpi.edu/milkyway/forum_thread.php?id=4885&postid=75074#75074

Looks like we are several participants with a similar issue.

Cheers, Samuel
ID: 75075 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
alex

Send message
Joined: 1 Oct 14
Posts: 3
Credit: 19,130,914
RAC: 4,492
Message 75158 - Posted: 16 Mar 2023, 8:27:51 UTC - in response to Message 75075.  

Yeah, I saw it, thanks though. Fortunately I have got a second computer with nvidia gpus so it's not that bad right now. I can use amd gpus for einstein, primegrid and amicable numbers quite nicely.
ID: 75158 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3310
Credit: 519,244,124
RAC: 20,888
Message 75159 - Posted: 16 Mar 2023, 11:27:36 UTC - in response to Message 75158.  

Yeah, I saw it, thanks though. Fortunately I have got a second computer with nvidia gpus so it's not that bad right now. I can use amd gpus for einstein, primegrid and amicable numbers quite nicely.


A cruncher at Einstein said if you downclock your gpu just a little bit the errors will stop, other people say they don't think so but he's gone from more than 10 errors per day to zero errors per day. MSIAfterburner is one program to help you do that and it works for both AMD and Nvidia gpu's, I believe gpuZ is another one that will let you change the clock speeds as well.
ID: 75159 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Technik007[CZ]

Send message
Joined: 12 Sep 10
Posts: 6
Credit: 55,439,405
RAC: 0
Message 75183 - Posted: 22 Mar 2023, 15:20:55 UTC - in response to Message 74927.  

Hi, I can confirm the same even with older card.
Few days ago I installed Radeon VII on my PC with Ubuntu 22.04 and latest "pro" drivers from amd website. Whilst other gpu boinc project like Einstein@Home or NumberFileds@Home were runing fine on this setup Milkyway was generating errors about every 2sec per WU.
ID: 75183 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Technik007[CZ]

Send message
Joined: 12 Sep 10
Posts: 6
Credit: 55,439,405
RAC: 0
Message 75193 - Posted: 23 Mar 2023, 5:31:05 UTC - in response to Message 75183.  

Stderr output

<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 250 (0xfa, -6)</message>
<stderr_txt>
<search_application> milkyway_separation 1.46 Linux x86_64 double OpenCL </search_application>
Reading preferences ended prematurely
BOINC GPU type suggests using OpenCL vendor 'Advanced Micro Devices, Inc.'
Setting process priority to 0 (13): Permission denied
Error loading Lua script 'astronomy_parameters.txt': [string "number_parameters: 4..."]:1: '<name>' expected near '4'
Switching to Parameter File 'astronomy_parameters.txt'
<number_WUs> 5 </number_WUs>
<number_params_per_WU> 20 </number_params_per_WU>
Using AVX path
Found 1 platform
Platform 0 information:
Name: AMD Accelerated Parallel Processing
Version: OpenCL 2.1 AMD-APP (3513.0)
Vendor: Advanced Micro Devices, Inc.
Extensions: cl_khr_icd cl_amd_event_callback
Profile: FULL_PROFILE
Using device 0 on platform 0
Found 2 CL devices
Device 'gfx906:sramecc+:xnack-' (Advanced Micro Devices, Inc.:0x1002) (CL_DEVICE_TYPE_GPU)
Board: AMD Radeon VII
Driver version: 3513.0 (HSA1.1,LC)
Version: OpenCL 2.0
Compute capability: 0.0
Max compute units: 60
Clock frequency: 1801 Mhz
Global mem size: 17163091968
Local mem size: 65536
Max const buf size: 14588628168
Double extension: cl_khr_fp64
Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY
Error getting device and context (-6): CL_OUT_OF_HOST_MEMORY
Failed to calculate likelihood
Using AVX path
Found 1 platform
Platform 0 information:
Name: AMD Accelerated Parallel Processing
Version: OpenCL 2.1 AMD-APP (3513.0)
Vendor: Advanced Micro Devices, Inc.
Extensions: cl_khr_icd cl_amd_event_callback
Profile: FULL_PROFILE
Using device 0 on platform 0
Found 2 CL devices
Device 'gfx906:sramecc+:xnack-' (Advanced Micro Devices, Inc.:0x1002) (CL_DEVICE_TYPE_GPU)
Board: AMD Radeon VII
Driver version: 3513.0 (HSA1.1,LC)
Version: OpenCL 2.0
Compute capability: 0.0
Max compute units: 60
Clock frequency: 1801 Mhz
Global mem size: 17163091968
Local mem size: 65536
Max const buf size: 14588628168
Double extension: cl_khr_fp64
Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY
Error getting device and context (-6): CL_OUT_OF_HOST_MEMORY
Failed to calculate likelihood
Using AVX path
Found 1 platform
Platform 0 information:
Name: AMD Accelerated Parallel Processing
Version: OpenCL 2.1 AMD-APP (3513.0)
Vendor: Advanced Micro Devices, Inc.
Extensions: cl_khr_icd cl_amd_event_callback
Profile: FULL_PROFILE
Using device 0 on platform 0
Found 2 CL devices
Device 'gfx906:sramecc+:xnack-' (Advanced Micro Devices, Inc.:0x1002) (CL_DEVICE_TYPE_GPU)
Board: AMD Radeon VII
Driver version: 3513.0 (HSA1.1,LC)
Version: OpenCL 2.0
Compute capability: 0.0
Max compute units: 60
Clock frequency: 1801 Mhz
Global mem size: 17163091968
Local mem size: 65536
Max const buf size: 14588628168
Double extension: cl_khr_fp64
Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY
Error getting device and context (-6): CL_OUT_OF_HOST_MEMORY
Failed to calculate likelihood
Using AVX path
Found 1 platform
Platform 0 information:
Name: AMD Accelerated Parallel Processing
Version: OpenCL 2.1 AMD-APP (3513.0)
Vendor: Advanced Micro Devices, Inc.
Extensions: cl_khr_icd cl_amd_event_callback
Profile: FULL_PROFILE
Using device 0 on platform 0
Found 2 CL devices
Device 'gfx906:sramecc+:xnack-' (Advanced Micro Devices, Inc.:0x1002) (CL_DEVICE_TYPE_GPU)
Board: AMD Radeon VII
Driver version: 3513.0 (HSA1.1,LC)
Version: OpenCL 2.0
Compute capability: 0.0
Max compute units: 60
Clock frequency: 1801 Mhz
Global mem size: 17163091968
Local mem size: 65536
Max const buf size: 14588628168
Double extension: cl_khr_fp64
Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY
Error getting device and context (-6): CL_OUT_OF_HOST_MEMORY
Failed to calculate likelihood
Using AVX path
Found 1 platform
Platform 0 information:
Name: AMD Accelerated Parallel Processing
Version: OpenCL 2.1 AMD-APP (3513.0)
Vendor: Advanced Micro Devices, Inc.
Extensions: cl_khr_icd cl_amd_event_callback
Profile: FULL_PROFILE
Using device 0 on platform 0
Found 2 CL devices
Device 'gfx906:sramecc+:xnack-' (Advanced Micro Devices, Inc.:0x1002) (CL_DEVICE_TYPE_GPU)
Board: AMD Radeon VII
Driver version: 3513.0 (HSA1.1,LC)
Version: OpenCL 2.0
Compute capability: 0.0
Max compute units: 60
Clock frequency: 1801 Mhz
Global mem size: 17163091968
Local mem size: 65536
Max const buf size: 14588628168
Double extension: cl_khr_fp64
Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY
Error getting device and context (-6): CL_OUT_OF_HOST_MEMORY
Failed to calculate likelihood
05:28:34 (50609): called boinc_finish(-6)

</stderr_txt>
]]>
ID: 75193 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Technik007[CZ]

Send message
Joined: 12 Sep 10
Posts: 6
Credit: 55,439,405
RAC: 0
Message 75224 - Posted: 27 Mar 2023, 14:04:40 UTC
Last modified: 27 Mar 2023, 14:07:14 UTC

Quite interesting thing is this project on same hw but Win10 runs fine so it is not hw failure.
Projects like NumberFileds@home, Primegrid, Srbase and Einstein@home return results on Ubuntu fine without error and WUs are validated ok.
But Moo! Wrapper does not send WU when on Ubuntu while it does when runs on Windows. (is there something missing, some dll? they have app for linux and ati/amd)
ID: 75224 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ian&Steve C.
Avatar

Send message
Joined: 18 Nov 22
Posts: 81
Credit: 636,355,130
RAC: 0
Message 75225 - Posted: 27 Mar 2023, 15:46:26 UTC
Last modified: 27 Mar 2023, 15:52:30 UTC

all of these newer Radeon GPUs require the new drivers and are not supported by legacy drivers.

my guess is that the Linux application requires/uses some OpenCL function that's been depreciated in newer versions. or both the Windows and Linux apps need it, but only the Linux driver doesn't have the support built in for the older functions. I'd learn more toward the source of the issue being the driver itself though. since Nvidia doesnt seem to have a problem including support in their drivers on the newest devices and AMD is notorious for GPU driver issues, especially on Linux.

http://man.opencl.org/deprecated.html
OpenCL Features Deprecated in OpenCL 2.0

clCreateCommandQueue



Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY


ID: 75225 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Technik007[CZ]

Send message
Joined: 12 Sep 10
Posts: 6
Credit: 55,439,405
RAC: 0
Message 75239 - Posted: 29 Mar 2023, 9:39:09 UTC - in response to Message 75225.  

Milkyway@home is still using app from 2017 which is simply not ready for that OpenCL level I would say because backward compatibility for OpenCL 2.0 is not mandatory. And Nvidia is using OpenCL 3.0 on their cards which is using OpenCL 1.2 grounds.

And other projects have apps updated eg Einstein@home got apps in 2022 or 2023 and it runs fine.
ID: 75239 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ian&Steve C.
Avatar

Send message
Joined: 18 Nov 22
Posts: 81
Credit: 636,355,130
RAC: 0
Message 75240 - Posted: 29 Mar 2023, 11:41:08 UTC - in response to Message 75239.  
Last modified: 29 Mar 2023, 11:41:32 UTC

Einstein also has AMD apps from 2016/2017 that work fine that havent been updated.

it's not really that the app needs updated. could an app update fix it? maybe. but the root cause seems to be the driver not supporting some legacy features. the fact that these cards work on Windows is a big clue. they took the time to properly implement the features for the Windows driver, but not Linux.

ID: 75240 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Technik007[CZ]

Send message
Joined: 12 Sep 10
Posts: 6
Credit: 55,439,405
RAC: 0
Message 75276 - Posted: 3 Apr 2023, 11:17:51 UTC - in response to Message 75240.  
Last modified: 3 Apr 2023, 11:19:36 UTC

the root cause seems to be the driver not supporting some legacy features. the fact that these cards work on Windows is a big clue

you are right but...

To be honest to you I want to run one project which runs on linux and cpu better and its name is Universe@home. And that difference in computing time is something like 3h vs 40m for same wu. I would like to stay with MilkyWay@home as-well but simply I cannot unless there is solution found.
So I am going to run Einstein@home which run fine on that broken linux AMD driver with their either old or new app ( interesting is it ? they have old app and it works fine on linux ) and then Universe@home and other projects working fine or sometimes better on linux.

And I might try run Universe@home in VM on Windows host to test if there is some computing penalty.
ID: 75276 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ian&Steve C.
Avatar

Send message
Joined: 18 Nov 22
Posts: 81
Credit: 636,355,130
RAC: 0
Message 75277 - Posted: 3 Apr 2023, 21:47:34 UTC - in response to Message 75276.  

I’ve seen others report that Universe works well using a VM under WSL/WSL2. That way you could run windows for the AMD card and Universe in the WSL VM.

ID: 75277 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Neo_Chen

Send message
Joined: 30 Sep 20
Posts: 1
Credit: 228
RAC: 0
Message 75338 - Posted: 21 Apr 2023, 15:30:35 UTC

I can also confirm this happens on my PC.

Spec:
OS: Arch Linux
CPU: Ryzen 5 5600X
GPU: RX6500XT


on both AMDGPU-Pro OpenCL and ROCm OpenCL driver, I get the same errors
ID: 75338 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Skip Da Shu
Avatar

Send message
Joined: 11 Apr 08
Posts: 81
Credit: 57,755,743
RAC: 0
Message 75339 - Posted: 24 Apr 2023, 22:55:27 UTC
Last modified: 24 Apr 2023, 23:10:44 UTC

Ditto, same problem. Replaced RX 580 in a box with an RX 6600 (going from opencl=legacy to rocm. Works on multiple E@H WUs and multiple PrimeGrid WUs but not MW Sep tasks. Haven't run it with any WCG WUs yet.

Thanx Ian&Steve C.:

http://man.opencl.org/deprecated.html

OpenCL Features Deprecated in OpenCL 2.0

clCreateCommandQueue


Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY


Although sounds like easy app code fix to me.

Skip
ID: 75339 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Technik007[CZ]

Send message
Joined: 12 Sep 10
Posts: 6
Credit: 55,439,405
RAC: 0
Message 75407 - Posted: 28 May 2023, 21:30:39 UTC - in response to Message 75339.  
Last modified: 28 May 2023, 21:31:07 UTC

It would be good idea to contact AMD support and ask them why this app works fine on windows but not on linux.
ID: 75407 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
tictoc
Avatar

Send message
Joined: 31 Dec 11
Posts: 17
Credit: 3,171,557,895
RAC: 1
Message 75415 - Posted: 31 May 2023, 2:43:27 UTC

MilkyWay@Home needs what is probably a fairly minor update in order to run on the ROCm OpenCL drivers. While quite a few older OpenCL applications will run without modifications on the newer ROCm stack, that is not the case for everything. For example, over on Einstein@Home, while the FGRPB1G app ran with no modifications, O2MDF (Gravitational Wave tasks from 2020) needed a few small modifications, in order to run on both the ROCm open source drivers, and the now deprecated AMDGPU-Pro drivers. https://einsteinathome.org/goto/comment/175741

There was probably a bit of AMD special sauce in the closed source Linux drivers, that most likely still exists in the Windows driver, which allows MilkyWay to run on the latest Windows driver. I did do a simple rebuild of MilkyWay on a modern ROCm Linux stack, but the rebuilt app still failed with the same errors. It has been on my much too long todo list to go through the MilkyWay app and see if it is a simple fix, but I haven't taken the time to do that.

Since I recently started crunching MilkyWay again, I am just going the easy route and passing a few Radeon VII's through to a VM that is using the OpenCL bits from the old AMDGPU-Pro driver before it transitioned to ROCm for the compute backend (AMDGPU-Pro 20.30.1109583-1). This allows me to run MilkyWay on a fully up-to-date system (kernel 6.3.5|libc 2.37) until the MilkyWay app is updated.
ID: 75415 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote

Questions and Answers : Unix/Linux : RX6600/Ubuntu 22.10 > Error creating command queue (-6): CL_OUT_OF_HOST_MEMORY

©2024 Astroinformatics Group