Welcome to MilkyWay@home

Validation Pending too many tasks

Message boards : Number crunching : Validation Pending too many tasks
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4

AuthorMessage
Septimus

Send message
Joined: 8 Nov 11
Posts: 205
Credit: 2,882,853
RAC: 272
Message 74519 - Posted: 21 Oct 2022, 10:23:41 UTC - in response to Message 74518.  

It would be nice to know the breakdown of the waiting for validation backlog between the applications. My guess is that Nbody is growing exponentially since the changes in profile from September. My small separation backlog has at last got to zero after 3 weeks. Whether the inconclusive Nbody’s will ever validate I don’t know. Have moved onto to other projects in the interim.
ID: 74519 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Septimus

Send message
Joined: 8 Nov 11
Posts: 205
Credit: 2,882,853
RAC: 272
Message 74520 - Posted: 21 Oct 2022, 10:24:04 UTC - in response to Message 74518.  

It would be nice to know the breakdown of the waiting for validation backlog between the applications. My guess is that Nbody is growing exponentially since the changes in profile from September. My small separation backlog has at last got to zero after 3 weeks. Whether the inconclusive Nbody’s will ever validate I don’t know. Have moved onto to other projects in the interim.
ID: 74520 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3315
Credit: 519,940,055
RAC: 22,560
Message 74521 - Posted: 21 Oct 2022, 11:25:56 UTC - in response to Message 74515.  

Validation is still happening just at a very slow pace so if one stops crunching a reduction in one's Validation Pending is to be expected. Even without stopping there are occasional temporary reductions. However, users stopping is likely to slow things down even more as there will be even less machines doing the little validation that is happening.

Unfortunately, it seems unlikely that things will get fixed until after the server migration.

I haven't ran N-Body much over the last few months. Is N-Body validation also a problem or is it just Separation?


I don't know but the Separation tasks are not getting a lot better, here's my previous postings about numbers

posted on 19 Oct:
In progress (957) · Validation pending (7855) · Validation inconclusive (484) · Valid (3588) · Invalid (2) · Error (5)

posted on 17 Oct:
In progress (747) · Validation pending (4510) · Validation inconclusive (256) · Valid (3269) · Invalid (2) · Error (1)[/quote]

Today on 21 Oct the numbers are
In progress (1045) · Validation pending (7992) · Validation inconclusive (368) · Valid (3800) · Invalid (2) · Error (2)

These are just Separation tasks that includes both cpu and gpu tasks
ID: 74521 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
hoppisaur

Send message
Joined: 16 Jun 16
Posts: 9
Credit: 778,481,857
RAC: 34,308
Message 74522 - Posted: 21 Oct 2022, 12:06:47 UTC
Last modified: 21 Oct 2022, 12:08:53 UTC

My tasks status as of today (10.21). Server migration can't come soon enough...

In progress (1158) · Validation pending (25563) · Validation inconclusive (1850) · Valid (11077) · Invalid (95) · Error (1)

Edit: this is 99.99+% separation.
ID: 74522 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Kiska

Send message
Joined: 31 Mar 12
Posts: 94
Credit: 151,905,988
RAC: 12,364
Message 74524 - Posted: 21 Oct 2022, 14:52:48 UTC

Here is mine

State: All (353) · In progress (120) · Validation pending (0) · Validation inconclusive (170) · Valid (63) · Invalid (0) · Error (0)
Application: All (361) · Milkyway@home N-Body Simulation (353) · Milkyway@home Separation (8)


btw I have started a thread posting daily graphs of the workunits graph so come and say hi?

Here is the link to the thread: https://milkyway.cs.rpi.edu/milkyway/forum_thread.php?id=4940
ID: 74524 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
San-Fernando-Valley

Send message
Joined: 13 Apr 17
Posts: 256
Credit: 604,411,638
RAC: 0
Message 74536 - Posted: 22 Oct 2022, 4:54:43 UTC - in response to Message 74515.  
Last modified: 22 Oct 2022, 4:58:18 UTC

Validation is still happening just at a very slow pace so if one stops crunching a reduction in one's Validation Pending is to be expected. Even without stopping there are occasional temporary reductions. However, users stopping is likely to slow things down even more as there will be even less machines doing the little validation that is happening.

Unfortunately, it seems unlikely that things will get fixed until after the server migration.

I haven't ran N-Body much over the last few months. Is N-Body validation also a problem or is it just Separation?

N-Body validation is doing fine and has been all along -- As I have mention a few times already.
The only problem N-Body has, is the extrem long running time of each task since a few weeks.
But, as Tom said, it is a known problem and they are working on it.

I wish fellow crunchers would specify in their post which project they are referring to.
ID: 74536 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3315
Credit: 519,940,055
RAC: 22,560
Message 74538 - Posted: 22 Oct 2022, 11:03:37 UTC - in response to Message 74521.  

Validation is still happening just at a very slow pace so if one stops crunching a reduction in one's Validation Pending is to be expected. Even without stopping there are occasional temporary reductions. However, users stopping is likely to slow things down even more as there will be even less machines doing the little validation that is happening.

Unfortunately, it seems unlikely that things will get fixed until after the server migration.

I haven't ran N-Body much over the last few months. Is N-Body validation also a problem or is it just Separation?


I don't know but the Separation tasks are not getting a lot better, here's my previous postings about numbers

posted on 17 Oct:
In progress (747) · Validation pending (4510) · Validation inconclusive (256) · Valid (3269) · Invalid (2) · Error (1)


posted on 19 Oct:
In progress (957) · Validation pending (7855) · Validation inconclusive (484) · Valid (3588) · Invalid (2) · Error (5)

Today on 21 Oct the numbers are
In progress (1045) · Validation pending (7992) · Validation inconclusive (368) · Valid (3800) · Invalid (2) · Error (2)

These are just Separation tasks that includes both cpu and gpu tasks[/quote]

Today 22 Oct my Separation numbers are:
In progress (1021) · Validation pending (6844) · Validation inconclusive (333) · Valid (2608) · Invalid (2) · Error (0)

As you can see there HAS been progress made in my Separation numbers so things ARE getting better!!!
ID: 74538 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Slywy

Send message
Joined: 22 Jul 12
Posts: 11
Credit: 1,008,373
RAC: 0
Message 74539 - Posted: 22 Oct 2022, 16:00:32 UTC - in response to Message 74536.  

Validation is still happening just at a very slow pace so if one stops crunching a reduction in one's Validation Pending is to be expected. Even without stopping there are occasional temporary reductions. However, users stopping is likely to slow things down even more as there will be even less machines doing the little validation that is happening.

Unfortunately, it seems unlikely that things will get fixed until after the server migration.

I haven't ran N-Body much over the last few months. Is N-Body validation also a problem or is it just Separation?

N-Body validation is doing fine and has been all along -- As I have mention a few times already.
The only problem N-Body has, is the extrem long running time of each task since a few weeks.
But, as Tom said, it is a known problem and they are working on it.

I wish fellow crunchers would specify in their post which project they are referring to.


Separation, 36 pending for a couple of days. Maybe a few.
ID: 74539 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3315
Credit: 519,940,055
RAC: 22,560
Message 74542 - Posted: 23 Oct 2022, 11:21:31 UTC - in response to Message 74538.  

Validation is still happening just at a very slow pace so if one stops crunching a reduction in one's Validation Pending is to be expected. Even without stopping there are occasional temporary reductions. However, users stopping is likely to slow things down even more as there will be even less machines doing the little validation that is happening.

Unfortunately, it seems unlikely that things will get fixed until after the server migration.

I haven't ran N-Body much over the last few months. Is N-Body validation also a problem or is it just Separation?


I don't know but the Separation tasks are not getting a lot better, here's my previous postings about numbers

posted on 17 Oct:
In progress (747) · Validation pending (4510) · Validation inconclusive (256) · Valid (3269) · Invalid (2) · Error (1)


posted on 19 Oct:
In progress (957) · Validation pending (7855) · Validation inconclusive (484) · Valid (3588) · Invalid (2) · Error (5)

Today on 21 Oct the numbers are
In progress (1045) · Validation pending (7992) · Validation inconclusive (368) · Valid (3800) · Invalid (2) · Error (2)

These are just Separation tasks that includes both cpu and gpu tasks


Today 22 Oct my Separation numbers are:
In progress (1021) · Validation pending (6844) · Validation inconclusive (333) · Valid (2608) · Invalid (2) · Error (0)

As you can see there HAS been progress made in my Separation numbers so things ARE getting better!!![/quote]

Today on 23 Oct my Separation numbers are:
In progress (997) · Validation pending (5311) · Validation inconclusive (278) · Valid (2882) · Invalid (2) · Error (0)

You can see the in progress numbers are fairly steady but the rest of the numbers ARE getting better!!!
ID: 74542 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Martin

Send message
Joined: 28 May 22
Posts: 17
Credit: 402,111,833
RAC: 0
Message 74578 - Posted: 27 Oct 2022, 1:31:22 UTC

"Waiting for Validation" is down ~500,000 since the last time I looked at it, on 10/23 it was around 5.8 million..

Looks like the queue is clearing out nicely. So far.

Martin
ID: 74578 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Captiosus

Send message
Joined: 9 Apr 14
Posts: 35
Credit: 9,708,616
RAC: 0
Message 74579 - Posted: 27 Oct 2022, 2:15:21 UTC
Last modified: 27 Oct 2022, 2:19:06 UTC

Yep, definitely noticed that something has become unstuck and WUs are getting processed. According to the BOINC statistics viewer in the client the last 2-3 days worth of RAC have been trending upwards.
ID: 74579 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3315
Credit: 519,940,055
RAC: 22,560
Message 74583 - Posted: 27 Oct 2022, 22:59:59 UTC - in response to Message 74579.  

Yep, definitely noticed that something has become unstuck and WUs are getting processed. According to the BOINC statistics viewer in the client the last 2-3 days worth of RAC have been trending upwards.


Yup mine was on 23 Oct my Separation numbers are:
In progress (997) · Validation pending (5311) · Validation inconclusive (278) · Valid (2882) · Invalid (2) · Error (0)

But today on 27 Oct they are:
In progress (1015) · Validation pending (5976) · Validation inconclusive (375) · Valid (4744) · Invalid (2) · Error (5)

And my RAC too is going way up as a result:

MilkyWay@Home today 484,969,900 yesterday 626,525 average 360,295
ID: 74583 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Septimus

Send message
Joined: 8 Nov 11
Posts: 205
Credit: 2,882,853
RAC: 272
Message 74615 - Posted: 1 Nov 2022, 18:28:41 UTC - in response to Message 74583.  

I am beginning to wonder if some long run Nbody WU’s will ever get validated. I have one that I completed, throughout its life back in September it has been Aborted twice, failed to start on time 3 times and is now back “in progress “. Have to wonder how many more there are like this.
ID: 74615 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3315
Credit: 519,940,055
RAC: 22,560
Message 74617 - Posted: 2 Nov 2022, 2:56:52 UTC - in response to Message 74615.  

I am beginning to wonder if some long run Nbody WU’s will ever get validated. I have one that I completed, throughout its life back in September it has been Aborted twice, failed to start on time 3 times and is now back “in progress “. Have to wonder how many more there are like this.


Since they are long runs and it took awhile before anyone at the project said anything about them I would guess there are a bunch of them as people thought they were problematic and just aborted them or didn't get to them as they ran other long tasks
ID: 74617 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
San-Fernando-Valley

Send message
Joined: 13 Apr 17
Posts: 256
Credit: 604,411,638
RAC: 0
Message 74618 - Posted: 2 Nov 2022, 5:44:07 UTC - in response to Message 74615.  

I am beginning to wonder if some long run Nbody WU’s will ever get validated. I have one that I completed, throughout its life back in September it has been Aborted twice, failed to start on time 3 times and is now back “in progress “. Have to wonder how many more there are like this.

I've been having many N-Body long-runs for weeks. Up to 24 hours on my "old" and "slow" PC.
Never had to wait very long for validation.
So no problems that I, personally, can see.
ID: 74618 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Septimus

Send message
Joined: 8 Nov 11
Posts: 205
Credit: 2,882,853
RAC: 272
Message 74619 - Posted: 2 Nov 2022, 8:47:20 UTC - in response to Message 74618.  

Thanks for your comments. I am not sure why that WU is being bounced around, it used 41,000 CPU seconds, I ran it on 8 CPUs in around 5,600 seconds. I have had much bigger ones validated.
ID: 74619 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Septimus

Send message
Joined: 8 Nov 11
Posts: 205
Credit: 2,882,853
RAC: 272
Message 74620 - Posted: 2 Nov 2022, 8:50:36 UTC - in response to Message 74619.  

Thanks for your comments. I am not sure why that WU is being bounced around, it used 41,000 CPU seconds, I ran it on 8 CPUs in around 5,600 seconds. I have had much bigger ones validated.


Just got validated at last !
ID: 74620 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile mikey
Avatar

Send message
Joined: 8 May 09
Posts: 3315
Credit: 519,940,055
RAC: 22,560
Message 74621 - Posted: 2 Nov 2022, 10:53:54 UTC - in response to Message 74620.  

Thanks for your comments. I am not sure why that WU is being bounced around, it used 41,000 CPU seconds, I ran it on 8 CPUs in around 5,600 seconds. I have had much bigger ones validated.


Just got validated at last !


That's very good, so the system is working as it should that's a very good thing!!
ID: 74621 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4

Message boards : Number crunching : Validation Pending too many tasks

©2024 Astroinformatics Group