@ Tom:

no problem ...
glad you replied ...
try to have a nice day ....
cheers

I did eventually get WU's about 30 Mins ago...obviously a big lag for some reason.

Hmm. That's really strange. I hope that things didn't get "desynced" with the DB when the drive failed. I'm not sure exactly what that would mean in terms of tech, but if the server thinks that there are 1.6M WUs that are ready to be sent out, but they are actually stuck somewhere, then it probably wouldn't make more WUs that could actually get sent out.

Maybe I can clear out the WUs that are waiting in the queue somehow. I suppose this could also happen with the WUs in the validation queue, which could explain the huge amount of WUs waiting for validation that apparently aren't getting sent out either.

I'll have to dig around. If I end up purging everything I'll have to figure out a way to give out the proper credits to everyone who has WUs that have been crunched and are sitting in limbo.

Today is busy for me, I have to train students on our observatory telescope, have several meetings, and am about to teach class. But I'll try to look into things when I get a chance.

This isn't strange... the server status page is correct, you need to reduce the time it takes to look up tasks ready to be sent. You are asking the database too much with over 18M tasks ready to be sent. The scheduler process doesn't hit the database every time someone asks for work, it instead looks in a shared memory pool that is fed by the feeder. This feeder process hits the database for tasks marked ready to send(up til the shared memory size), so you need to reduce the number with this as it does take the database some time to return an output and the bigger the database the longer it takes to return said output especially given there are lots of processes hitting it(ie web forum like this response, scheduler setting tasks to sent, feeder requesting tasks to be put into scheduler, stats generation, etc etc).

If this is a problem, try increasing the shared memory size and it should store more in the buffer
Or the other option is to turn off the workunit generator only, and you can see if jobs are still being sent out.

Thanks for the insight Kiska, that is very helpful. I am going to shut down the milkyway processes for a bit in order to remove the pile-up of unsent WUs, and then I will also look at increasing the shared memory size.

I've just been notified that the server rebuild has completed, so hopefully we can get things running as per normal soon! I'll be paying attention to things throughout the day in order to fix any weirdness that comes along.

I'm guessing your rebuild took ages because the disks were under heavy load. Yes I know I've mentioned it before, but please get some SSDs from our donations. And ask for donations on the home page!

Our shared memory limit appears to be the max value it can be for 64 bit machines (SHMMAX = 18446744073692774399 bytes), so I don't think that the shared memory allowance is the issue here, unless I'm missing something. I did kill all jobs that hadn't been sent out yet though.

Thanks for the insight Kiska, that is very helpful. I am going to shut down the milkyway processes for a bit in order to remove the pile-up of unsent WUs, and then I will also look at increasing the shared memory size.

Tom,

I see you've turned off almost everything... Surely if you turn off the feeder there is no way work can be sent out as nothing will be put in the shared memory to be sent out! Also, if the transitioner isn't running result data won't move from one state to another. Kiska's recommendation was to turn off the work generator(s) only...

I'm confused - hopefully Kiska will chip in again...

[Edit. Just seen your most recent comment regarding deleting unsent work. And the status page now shows everything turned on again, though it seems to be an older version of the status page...]

Cheers - Al.

I see you've turned off almost everything... Surely if you turn off the feeder there is no way work can be sent out as nothing will be put in the shared memory to be sent out! Also, if the transitioner isn't running result data won't move from one state to another. Kiska's recommendation was to turn off the work generator(s) only...

I'm confused - hopefully Kiska will chip in again...

Cheers - Al.

Everything is back on. I think you just happened to look at the server status page while things were coming back up, so some processes were still reporting as off. I've noticed that there's a little bit of lag there, and sometimes people catch it in weird moments while I'm doing maintenance, and then they get confused because it looks like I'm turning random things off and on.

Hi Tom,

At the moment I'm getting the following error:

29-3-2022 21:43:37 | Milkyway@Home | update requested by user
29-3-2022 21:43:37 | Milkyway@Home | Sending scheduler request: Requested by user.
29-3-2022 21:43:37 | Milkyway@Home | Requesting new tasks for NVIDIA GPU
29-3-2022 21:43:48 | Milkyway@Home | Scheduler request completed: got 0 new tasks
29-3-2022 21:43:48 | Milkyway@Home | Server error: feeder not running
29-3-2022 21:43:48 | Milkyway@Home | Project requested delay of 400 seconds

Is everything OK?

Regards,
Arjen

Tue Mar 29 12:46:22 2022 | Milkyway@Home | Reporting 2 completed tasks
Tue Mar 29 12:46:22 2022 | Milkyway@Home | Requesting new tasks for CPU and Intel GPU
Tue Mar 29 12:46:33 2022 | Milkyway@Home | Scheduler request completed: got 0 new tasks
Tue Mar 29 12:46:33 2022 | Milkyway@Home | Server error: feeder not running

Everything is back on. I think you just happened to look at the server status page while things were coming back up, so some processes were still reporting as off. I've noticed that there's a little bit of lag there, and sometimes people catch it in weird moments while I'm doing maintenance, and then they get confused because it looks like I'm turning random things off and on.

Just tried to get GPU work and got:

1606 Milkyway@Home 29-03-2022 08:46 PM Requesting new tasks for CPU and AMD/ATI GPU
1607 Milkyway@Home 29-03-2022 08:46 PM [sched_op] CPU work request: 280707.60 seconds; 0.00 devices
1608 Milkyway@Home 29-03-2022 08:46 PM [sched_op] AMD/ATI GPU work request: 21780.00 seconds; 1.00 devices
1609 Milkyway@Home 29-03-2022 08:46 PM Scheduler request completed: got 0 new tasks
1610 Milkyway@Home 29-03-2022 08:46 PM Server error: feeder not running

Looks like the same thing as last time. I'll restart the server processes and see if the feeder comes back up.

No clue why it does that.

There's no 'feeder not running' error for me now:

29-3-2022 21:53:45 | Milkyway@Home | update requested by user
29-3-2022 21:53:45 | Milkyway@Home | Sending scheduler request: Requested by user.
29-3-2022 21:53:45 | Milkyway@Home | Requesting new tasks for NVIDIA GPU
29-3-2022 21:53:46 | Milkyway@Home | Scheduler request completed: got 0 new tasks
29-3-2022 21:53:46 | Milkyway@Home | Project requested delay of 91 seconds

So I'll just have to wait a bit I guess :)

Image from Stat's page now

Download server milkyway.cs.rpi.edu Running
Upload server milkyway.cs.rpi.edu Running
Scheduler milkyway Not Running
feeder milkyway Not Running
transitioner milkyway Not Running
db_purge milkyway Not Running
file_deleter milkyway Not Running
stream_fit_validator (milkyway) milkyway Not Running
stream_fit_assimilator (milkyway ) milkyway Not Running
stream_fit_work_generator (milkyway ) milkyway Not Running
nbody_validator (milkyway_nbody ) milkyway Not Running
nbody_assimilator (milkyway_nbody ) milkyway Not Running
nbody_work_generator (milkyway_nbody) milkyway Not Running

Milkyway@home N-Body Simulation 13966487 576029 0.17 (0.01 - 1.3) 185
Milkyway@home Separation 4961115 655282 0.6 (0.01 - 25.08) 919

Upstream server release: 1.0.4
Database schema version: 27028

Task data as of 29 Mar 2022, 18:24:00 UTC

Image from Stat's page now

Download server milkyway.cs.rpi.edu Running
Upload server milkyway.cs.rpi.edu Running
Scheduler milkyway Not Running
feeder milkyway Not Running
transitioner milkyway Not Running
db_purge milkyway Not Running
file_deleter milkyway Not Running
stream_fit_validator (milkyway) milkyway Not Running
stream_fit_assimilator (milkyway ) milkyway Not Running
stream_fit_work_generator (milkyway ) milkyway Not Running
nbody_validator (milkyway_nbody ) milkyway Not Running
nbody_assimilator (milkyway_nbody ) milkyway Not Running
nbody_work_generator (milkyway_nbody) milkyway Not Running

Milkyway@home N-Body Simulation 13966487 576029 0.17 (0.01 - 1.3) 185
Milkyway@home Separation 4961115 655282 0.6 (0.01 - 25.08) 919

Upstream server release: 1.0.4
Database schema version: 27028

Task data as of 29 Mar 2022, 18:24:00 UTC

Weird, I see the same 18:24 UTC, but everything says running. But Boinc returns "Milkyway@Home 29-03-2022 09:00 PM Project is temporarily shut down for maintenance" (I'm on UTC+1 here)

Our shared memory limit appears to be the max value it can be for 64 bit machines (SHMMAX = 18446744073692774399 bytes), so I don't think that the shared memory allowance is the issue here, unless I'm missing something. I did kill all jobs that hadn't been sent out yet though.

No no no, please don't set that, something more sensible please? Unless you actually have that amount of RAM its not going to help.

You also need to consider that the database also needs to use memory for its own caching ability.

Maybe something like 512MiB or 1GiB should be sufficient for shared memory.

If you use /etc/sysctl.conf for system config, just edit then use sysctl -p to reload the changes

As for the turning thing off ONLY these 2 should be turned off to test if the database tasks table is just really full:
stream_fit_work_generator (milkyway )
nbody_work_generator (milkyway_nbody)

Tom - thanks for the clarification.

I'd just edited my earlier message to acknowledge seeing your "killed tasks" comment, but I'll leave the edit as is...

The task numbers appear unchanged (as the page on display at 20:00 UTC is apparently from before your most recent activity) -- I presume the server status page task data won't get updated again today - do you know how often it should update those numbers? (It didn't seem to be updated more than once a day whilst the server was struggling...)

Thanks for your efforts, especially given the other calls on your time! I just hope it sorts out soon, and you can get a bit of relative peace and quiet...

Cheers - Al.

The task numbers appear unchanged (as the page on display at 20:00 UTC is apparently from before your most recent activity) -- I presume the server status page task data won't get updated again today - do you know how often it should update those numbers? (It didn't seem to be updated more than once a day whilst the server was struggling...)

Cheers - Al.

Seems to update once in a while see https://grafana.kiska.pw/goto/8ChLths7k?orgId=1

Click on one of the fields on the right hand side of the graph:

or

29-3-2022 22:57:44 | Milkyway@Home | Sending scheduler request: To fetch work.
29-3-2022 22:57:44 | Milkyway@Home | Requesting new tasks for NVIDIA GPU
29-3-2022 22:57:46 | Milkyway@Home | Scheduler request completed: got 0 new tasks
29-3-2022 22:57:46 | Milkyway@Home | No work available
29-3-2022 22:57:46 | Milkyway@Home | Project requested delay of 91 seconds

29-3-2022 22:57:44 | Milkyway@Home | Sending scheduler request: To fetch work.
29-3-2022 22:57:44 | Milkyway@Home | Requesting new tasks for NVIDIA GPU
29-3-2022 22:57:46 | Milkyway@Home | Scheduler request completed: got 0 new tasks
29-3-2022 22:57:46 | Milkyway@Home | No work available
29-3-2022 22:57:46 | Milkyway@Home | Project requested delay of 91 seconds

I keep getting CPU Nbody work, but there seems to be a problem with server keeping up with the GPUs. If there was a way to send out bigger GPU tasks that would be nice. I think the huge number of them being sent back and forth could be overloading things?