Welcome to MilkyWay@home

High number of validation inconclusive (pending)

Message boards : Number crunching : High number of validation inconclusive (pending)
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · Next

AuthorMessage
MindCrime

Send message
Joined: 5 Mar 14
Posts: 24
Credit: 500,964,006
RAC: 0
Message 70949 - Posted: 10 Jul 2021, 2:40:26 UTC

On a particular hardware setup, I was able to hit about 2.25-2.75 Million per day and I was carrying about 3000-3600 validation inconclusive (pending) at the time. I took a little time off, historic heatwave, and now I'm ramping back up with the same hardware. I now have 6000 validation inconclusive (pending) and I'm getting like 1.75-1.9 million per day.

Anyone else seeing higher than normal amounts of (pendings)?
ID: 70949 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Tom Donlon
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 10 Apr 19
Posts: 408
Credit: 120,203,200
RAC: 0
Message 70954 - Posted: 10 Jul 2021, 17:16:02 UTC

Please see the thread here: https://milkyway.cs.rpi.edu/milkyway/forum_thread.php?id=4732&postid=70953#70953

I am testing a new validator that should fix these problems. I hope to roll it out early next week.
ID: 70954 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Marsinph

Send message
Joined: 13 Nov 10
Posts: 23
Credit: 108,282,839
RAC: 0
Message 71255 - Posted: 19 Oct 2021, 16:32:07 UTC - in response to Message 70954.  

Please see the thread here: https://milkyway.cs.rpi.edu/milkyway/forum_thread.php?id=4732&postid=70953#70953

I am testing a new validator that should fix these problems. I hope to roll it out early next week.



Hello Tom
I think it is still not fixed. After pushing my new computer (id 825437 ), 15 inconclusive for 1 valid !

Best regards
ID: 71255 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Tom Donlon
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 10 Apr 19
Posts: 408
Credit: 120,203,200
RAC: 0
Message 71257 - Posted: 19 Oct 2021, 22:32:55 UTC
Last modified: 19 Oct 2021, 22:33:25 UTC

It is "fixed", but the system got another oversized WU stuck in it again. I'm hoping this was a fluke. I cleared that oversized WU out, and hopefully you should see the errors dropping again.

I still need to figure out why these things get generated in the first place.
ID: 71257 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Septimus

Send message
Joined: 8 Nov 11
Posts: 205
Credit: 2,893,342
RAC: 358
Message 71675 - Posted: 4 Feb 2022, 14:48:12 UTC

Has the Validator got a problem, my validation pending total is going up steadily.
ID: 71675 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
JohnDK
Avatar

Send message
Joined: 18 Feb 10
Posts: 53
Credit: 221,700,755
RAC: 5,085
Message 71676 - Posted: 4 Feb 2022, 18:26:04 UTC

I see the Workunits waiting for validation on the server status page, has gone up from some 8000 to over 17000 in about 2 hours.
ID: 71676 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3319
Credit: 520,321,590
RAC: 20,772
Message 71680 - Posted: 5 Feb 2022, 10:47:54 UTC - in response to Message 71676.  

I see the Workunits waiting for validation on the server status page, has gone up from some 8000 to over 17000 in about 2 hours.


I wonder if AMD or Intel is testing a new cpu and using MilkyWay to do it.
ID: 71680 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Septimus

Send message
Joined: 8 Nov 11
Posts: 205
Credit: 2,893,342
RAC: 358
Message 71846 - Posted: 2 Mar 2022, 15:11:13 UTC

Looks like the validator is stuck again for some reason, queue is steadily building up.
ID: 71846 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3319
Credit: 520,321,590
RAC: 20,772
Message 71851 - Posted: 3 Mar 2022, 12:54:30 UTC - in response to Message 71846.  

Looks like the validator is stuck again for some reason, queue is steadily building up.


The News section says a drive is out clogging up everything
ID: 71851 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
unixchick
Avatar

Send message
Joined: 21 Feb 22
Posts: 66
Credit: 817,008
RAC: 0
Message 71856 - Posted: 3 Mar 2022, 20:01:42 UTC

Currently the status says Workunits waiting for validation 675900

any guesses at what number the system will starting having problems? So far I'm getting plenty of WUs to run, and the website isn't slow.
ID: 71856 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
mmonnin

Send message
Joined: 2 Oct 16
Posts: 167
Credit: 1,005,838,225
RAC: 49,078
Message 71857 - Posted: 3 Mar 2022, 22:45:59 UTC - in response to Message 71856.  

Currently the status says Workunits waiting for validation 675900

any guesses at what number the system will starting having problems? So far I'm getting plenty of WUs to run, and the website isn't slow.


https://milkyway.cs.rpi.edu/milkyway/forum_thread.php?id=4842&postid=71849#71849
ID: 71857 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
unixchick
Avatar

Send message
Joined: 21 Feb 22
Posts: 66
Credit: 817,008
RAC: 0
Message 71859 - Posted: 4 Mar 2022, 1:23:33 UTC - in response to Message 71857.  
Last modified: 4 Mar 2022, 1:24:31 UTC

Currently the status says Workunits waiting for validation 675900

any guesses at what number the system will starting having problems? So far I'm getting plenty of WUs to run, and the website isn't slow.


https://milkyway.cs.rpi.edu/milkyway/forum_thread.php?id=4842&postid=71849#71849


This is a good link explaining why the WU waiting for validation is rising for anyone still wondering why

I was more curious as to what point it all goes boom. The website is starting to lag a bit for me and the WU waiting validation is 700k+ (edit to add 772546)

Just a bit of fun as a guessing game. I'm new here, so anyone know from past experience when it is too much?
I will also send the admin (Tom?) my thoughts and wishes on a speedy arrival of the drive, and an easy adding of said drive :-)
ID: 71859 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Tom Donlon
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 10 Apr 19
Posts: 408
Credit: 120,203,200
RAC: 0
Message 71872 - Posted: 4 Mar 2022, 16:07:09 UTC

I sure hope there isn't a point where things go boom! We should be able to store ~2 billion workunits pending validation, so in theory we will be fine for a long time.

We haven't heard back about the drive yet, but it's been sent to the manufacturer.
ID: 71872 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
unixchick
Avatar

Send message
Joined: 21 Feb 22
Posts: 66
Credit: 817,008
RAC: 0
Message 71879 - Posted: 5 Mar 2022, 16:41:48 UTC
Last modified: 5 Mar 2022, 16:43:30 UTC

We have passed the 2 million mark of waiting for validation (2264700) and the system is still functioning pretty well. kudos Tom! 2 billion is a high mark, and I doubt we will reach that. Nice to know the system has capacity for that much. Thanks for the info. I'll worry less now.
ID: 71879 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
JohnDK
Avatar

Send message
Joined: 18 Feb 10
Posts: 53
Credit: 221,700,755
RAC: 5,085
Message 71880 - Posted: 5 Mar 2022, 18:39:05 UTC

2.3 mill now. It was 1.7 mill yesterday, forum is slow, not that good if you ask me.
ID: 71880 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
JohnDK
Avatar

Send message
Joined: 18 Feb 10
Posts: 53
Credit: 221,700,755
RAC: 5,085
Message 71881 - Posted: 5 Mar 2022, 19:37:54 UTC

And now I'm not getting new work: There's 0 tasks ready to send according to server status :(
ID: 71881 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
unixchick
Avatar

Send message
Joined: 21 Feb 22
Posts: 66
Credit: 817,008
RAC: 0
Message 71883 - Posted: 6 Mar 2022, 16:28:09 UTC

Thank you to whoever (Tom?) is working on a Sunday to get the server up again. I just got some tasks (resends). The server is going to take some time until it can send enough WUs to make us all happy though.
Here are some stats on the server, so I can quantify the recovery.
Transitioner backlog (hours) 19.25
Workunits waiting for validation 2446326
Tasks in progress 1438848 (This number is surprising as I would have thought it would be lower with the pause in sending out)

Thanks again to those in on a Sunday to fix things.
ID: 71883 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jimbocous
Avatar

Send message
Joined: 7 Mar 20
Posts: 22
Credit: 104,984,999
RAC: 12,225
Message 71923 - Posted: 11 Mar 2022, 8:27:46 UTC

Bit of strangeness on these, however.
Here's a sample WU:
https://milkyway.cs.rpi.edu/milkyway/workunit.php?wuid=386958166
Pending validation, but since initial replication is 1, there's no other work in progress to be completed to validate against, at least for now.
Am I missing something here?
I have over 900 of these pending at this point. Was wondering why my RAC landed in the dumps.
ID: 71923 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3319
Credit: 520,321,590
RAC: 20,772
Message 71924 - Posted: 11 Mar 2022, 11:26:35 UTC - in response to Message 71923.  

Bit of strangeness on these, however.
Here's a sample WU:
https://milkyway.cs.rpi.edu/milkyway/workunit.php?wuid=386958166
Pending validation, but since initial replication is 1, there's no other work in progress to be completed to validate against, at least for now.
Am I missing something here?
I have over 900 of these pending at this point. Was wondering why my RAC landed in the dumps.


They still have to be processed by the Server before any credits get awarded, they don't want any bad actors scamming the system so to be safe they scan them all.
ID: 71924 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jimbocous
Avatar

Send message
Joined: 7 Mar 20
Posts: 22
Credit: 104,984,999
RAC: 12,225
Message 71927 - Posted: 11 Mar 2022, 19:31:26 UTC - in response to Message 71924.  
Last modified: 11 Mar 2022, 19:46:58 UTC


They still have to be processed by the Server before any credits get awarded, they don't want any bad actors scamming the system so to be safe they scan them all.

I think you missed the point here. Unless I misunderstand, in addition to the basics of checking the completed workunit for proper formation and content, validation is the process of comparing the results of computation of an identical result from different computers. The presumption is that different machines with different hardware providing a comparable output is an indicator of a valid task. With only one computer doing the work, validation per that definition is not possible. With only 1 returned result, that result can never validate. If that's not how the software is currently operating, we're not doing science here.
ID: 71927 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · 3 · Next

Message boards : Number crunching : High number of validation inconclusive (pending)

©2024 Astroinformatics Group