Welcome to MilkyWay@home

Server Issues

Message boards : News : Server Issues
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · Next

AuthorMessage
JohnDK
Avatar

Send message
Joined: 18 Feb 10
Posts: 53
Credit: 221,698,710
RAC: 5,265
Message 74937 - Posted: 25 Jan 2023, 16:54:53 UTC

25/01/2023 17.51.14 | Milkyway@Home | Server error: feeder not running

Not able to report or get new work.
ID: 74937 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Kevin Roux
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 9 Aug 22
Posts: 69
Credit: 1,158,801
RAC: 7,965
Message 74939 - Posted: 25 Jan 2023, 17:27:22 UTC - in response to Message 74937.  

25/01/2023 17.51.14 | Milkyway@Home | Server error: feeder not running

Not able to report or get new work.


Server should be working now.
ID: 74939 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
JohnDK
Avatar

Send message
Joined: 18 Feb 10
Posts: 53
Credit: 221,698,710
RAC: 5,265
Message 74940 - Posted: 25 Jan 2023, 17:49:36 UTC

Yes, great!
ID: 74940 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Link
Avatar

Send message
Joined: 19 Jul 10
Posts: 588
Credit: 18,925,749
RAC: 4,898
Message 75216 - Posted: 26 Mar 2023, 14:52:47 UTC

Workunits waiting for validation are piling up since two days, now already over 1 million. Something needs a kick.
ID: 75216 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Speedy51

Send message
Joined: 12 Jun 10
Posts: 57
Credit: 6,163,587
RAC: 19
Message 75219 - Posted: 26 Mar 2023, 22:08:39 UTC - in response to Message 75216.  

Something needs a kick

Whatever you kick hopefully you won't hurt your foot. :-)

Yesterday 26/3/23 I got 54 or so GPU results to validate quickly by processing numbers higher than _1 in my queue. Things will eventually work themselves out. They always seem too
ID: 75219 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Link
Avatar

Send message
Joined: 19 Jul 10
Posts: 588
Credit: 18,925,749
RAC: 4,898
Message 75233 - Posted: 28 Mar 2023, 13:59:24 UTC - in response to Message 75219.  

Things will eventually work themselves out. They always seem too

As we see once again, they don't without some little help from the admins and everything just stops at one point.
ID: 75233 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Captiosus

Send message
Joined: 9 Apr 14
Posts: 35
Credit: 9,708,616
RAC: 0
Message 75241 - Posted: 29 Mar 2023, 20:45:08 UTC

Was wondering why my MW@H RAC was dropping. Someone go give the server a kick.
ID: 75241 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
San-Fernando-Valley

Send message
Joined: 13 Apr 17
Posts: 256
Credit: 604,411,638
RAC: 0
Message 75244 - Posted: 30 Mar 2023, 5:50:26 UTC - in response to Message 75241.  

... Someone go give the server a kick.

As already said before:
The kick is no go, since the admins feet are still hurting from last time ...
Parience is needed ...
Have you all a nice day!
ID: 75244 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
San-Fernando-Valley

Send message
Joined: 13 Apr 17
Posts: 256
Credit: 604,411,638
RAC: 0
Message 75248 - Posted: 30 Mar 2023, 14:20:28 UTC

Separation has run out of tasks ...

MW overall response time very high

cheers
S-F-V
ID: 75248 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
JohnDK
Avatar

Send message
Joined: 18 Feb 10
Posts: 53
Credit: 221,698,710
RAC: 5,265
Message 75251 - Posted: 30 Mar 2023, 16:01:09 UTC

Yes gussing MW servers is ready to break down

30/03/2023 17.59.50 | Milkyway@Home | Requesting new tasks for CPU
30/03/2023 18.00.03 | Milkyway@Home | Scheduler request failed: Server returned nothing (no headers, no data)
ID: 75251 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Aaron S

Send message
Joined: 10 May 14
Posts: 1
Credit: 17,434,480
RAC: 0
Message 75253 - Posted: 1 Apr 2023, 3:58:23 UTC

I just checked my pending validation tasks and noticed the same thing (they seem to be increasing daily). I do still seem to be receiving credit although my daily average seems to have dropped recently. I’ve set MW to “no new tasks” for the time being until things get sorted.
ID: 75253 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3319
Credit: 520,303,301
RAC: 20,458
Message 75257 - Posted: 1 Apr 2023, 10:56:37 UTC - in response to Message 75251.  

Yes gussing MW servers is ready to break down

30/03/2023 17.59.50 | Milkyway@Home | Requesting new tasks for CPU
30/03/2023 18.00.03 | Milkyway@Home | Scheduler request failed: Server returned nothing (no headers, no data)


Every once in awhile the Server fails to do what it's supposed to be doing and the Admins just need to manually reset it and then the tasks start flowing again and the invalids start dropping. This time no Admin seems to have acknowledged the problem yet so it's taking longer than normal for the reset to happen.
ID: 75257 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
.clair.

Send message
Joined: 3 Mar 13
Posts: 84
Credit: 779,527,671
RAC: 3,042
Message 75263 - Posted: 2 Apr 2023, 0:24:36 UTC

Ooo dear,
someone forgot to feed the hamster this afternoon
Back up to speed again now :-)
ID: 75263 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Link
Avatar

Send message
Joined: 19 Jul 10
Posts: 588
Credit: 18,925,749
RAC: 4,898
Message 75264 - Posted: 2 Apr 2023, 8:25:00 UTC - in response to Message 75263.  

Back up to speed again now :-)
Not sure it will last long, results waiting for validation still increasing, even if not as fast as before.
ID: 75264 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3319
Credit: 520,303,301
RAC: 20,458
Message 75265 - Posted: 2 Apr 2023, 10:37:06 UTC - in response to Message 75264.  

Back up to speed again now :-)


Not sure it will last long, results waiting for validation still increasing, even if not as fast as before.


Isn't that because they keep sending out the _0 tasks before they send out the _1 tasks that are often needed? Remember the _1 _2 _3 etc tasks all go to the end of the queue instead of the beginning.
ID: 75265 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Marcin
Avatar

Send message
Joined: 18 Jun 09
Posts: 35
Credit: 11,223,092
RAC: 10,835
Message 75267 - Posted: 2 Apr 2023, 14:17:07 UTC

how much ram does the server have? is it possible things are falling apart after some time due to out-of-memory kill?
ID: 75267 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
San-Fernando-Valley

Send message
Joined: 13 Apr 17
Posts: 256
Credit: 604,411,638
RAC: 0
Message 75268 - Posted: 2 Apr 2023, 15:13:15 UTC

Zero SEPARATION tasks today (04.Apr.23) available.

We had the same situation on 30.Mar.23 !

Crunched 10,000 tasks in 3 days ?

The server speed is a pain ...

cheers
S-F-V
ID: 75268 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Link
Avatar

Send message
Joined: 19 Jul 10
Posts: 588
Credit: 18,925,749
RAC: 4,898
Message 75271 - Posted: 2 Apr 2023, 16:59:38 UTC - in response to Message 75265.  

Isn't that because they keep sending out the _0 tasks before they send out the _1 tasks that are often needed? Remember the _1 _2 _3 etc tasks all go to the end of the queue instead of the beginning.

No, the usual 10000 ready to send tasks last just few minutes, so it doesn't matter. The issue is that the validators are not keeping up for whatever reason, we had that issue before, last time from mid September until beginning of November with up to over 6 million waiting for validation, hopefully they can fix it a bit faster this time.

Take de_modfit_71_bundle5_3s_south_pt2_4_1674667491_95541421_2 as example, 7 minutes between created and send, that's nothing. However some of my returned task are waiting already since more than two days for validation.
ID: 75271 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Kevin Roux
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 9 Aug 22
Posts: 69
Credit: 1,158,801
RAC: 7,965
Message 75272 - Posted: 2 Apr 2023, 20:24:37 UTC

Hello everyone,

The server has been struggling lately. The number of workunits waiting for validation spiked for some reason which has caused the server to use a lot of memory. This leads to the sorts of problems we are experiencing with the server as it works through the workunits. We are looking into what is causing the issues and are keeping an eye on it while it goes through this process.
We apologize for late response and thank you for you patience.

Thanks,
Kevin
ID: 75272 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
.clair.

Send message
Joined: 3 Mar 13
Posts: 84
Credit: 779,527,671
RAC: 3,042
Message 75273 - Posted: 2 Apr 2023, 20:24:55 UTC
Last modified: 2 Apr 2023, 20:28:32 UTC

Ah !!
Got it . .
Tell the hamster generator to run the other way round
They realy do need to find a way to send the resends out first
Wich iz wot I am realy saying

edit - an answer 31 seconds before I posted
ok , I know I type as fast as any housewife wearing oven gloves :-)
ID: 75273 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · Next

Message boards : News : Server Issues

©2024 Astroinformatics Group