Hello Everyone,

I have put up the following separation runs:

de_modfit_80_bundle4_4s_south4s_bgset_4
de_modfit_81_bundle4_4s_south4s_bgset_4
de_modfit_82_bundle4_4s_south4s_bgset_4
de_modfit_83_bundle4_4s_south4s_bgset_4
de_modfit_84_bundle4_4s_south4s_bgset_4
de_modfit_85_bundle4_4s_south4s_bgset_4
de_modfit_86_bundle4_4s_south4s_bgset_4

And I have taken down the following separation runs:

de_modfit_80_bundle4_4s_south4s_bgset_3
de_modfit_81_bundle4_4s_south4s_bgset_3
de_modfit_82_bundle4_4s_south4s_bgset_3
de_modfit_83_bundle4_4s_south4s_bgset_3
de_modfit_84_bundle4_4s_south4s_bgset_3
de_modfit_86_bundle4_4s_south4s_bgset_3

The new runs are copies of the previous bgset_3 runs and bgset_2 runs, in order to get yet another set of data to compare against. In the last set of runs, Stripe 84 didn't seem to cause problems like it did in the bgset_2 runs, but Stripe 85 did. As before, I will keep a close eye on these forums for a few weeks in order to see whether you are having any trouble with some of the Stripes.

You may continue to see workunits from bgset_2 runs for the next few days as workunits finish up and complete validation. This is expected and normal. You may start seeing the new runs as soon as your machine downloads new workunits from the server.

Thank you all for your continued support with this project. I hope you are staying safe in the pandemic world that we're stuck in for a little while longer.

Best,
Tom

Two wrong de_modfit_85. Very few, but still there.

Thanks for the update, I'm going to keep that stripe running for now, but I'll keep an eye on it.

I'm catching a few validate errors on de_modfit_85_bundle4_4s_south4s_bgset_4. Not sure if it need be reported but wanted to say something as it's the only validate errors I'm getting.

Thanks for the report. I'm going to try to keep these runs up until it starts becoming a problem, since it doesn't look like they've converged yet. If you start getting a large percentage of validate errors (>10% or so) please let me know and I can stop the bad stripes.

Hello, I'm now approaching that 10% of invalid tasks on de_modfit_85_bundle4_4s_south4s_bgset_4. I am at circa 8000 valid / 800 invalid.

I am up to 7% now from 5% yesterday.

---

Over 10% invalid and counting.

Right at 10% invalid now and climbing with the gpus running nothing but MW now since both Einstein and GPUGrid can't produce work.

---

I have taken down the de_modfit_85_bundle4_4s_south4s_bgset_4 run. It was not converged, but that's okay, and it would have only become more and more of a problem. Please let me know if Stripe 84 (de_modfit_84_bundle4_4s_south4s_bgset_4) workunits start causing any problems.

It is normal to see de_modfit_85_bundle4_4s_south4s_bgset_4 workunits trickle in over the next couple days as validations finish up. In a few days you should not be getting any more Stripe 85 workunits.

Best,
Tom

Sorry to inform you that Stripe 84 (de_modfit_84_bundle4_4s_south4s_bgset_4) workunits start causing problems this morning. For example in my case :
- de_modfit_84_bundle4_4s_south4s_bgset_4_1603804501_70487957
- de_modfit_84_bundle4_4s_south4s_bgset_4_1603804501_70487958
Regards.

Jean-Pierre

Confirmed. Seeing many failed validations on the 84 stripe as well. Not sure how to get accurate numbers as there are still stripe 85 validate errors trickling in, but my validate error rate is ~10%, and clicking on the first 10 errors found them all to be 84s. If there is a way and it would help I would be happy to narrow a search to find out exactly how many of which I'm looking at but I know it's going south for stripe 84.

Confirmed. Having the same high invalidate rate on the stripe 84 tasks.

---

Thanks for the updates everyone, I have also stopped the Stripe 84 (de_modfit_84_bundle4_4s_south4s_bgset_4) run. As usual, it may take a little time for the workunits to complete validating.

Yes, I'm getting lots of validation errors (84's)

I'm getting lots of errors (85,s) too.

Hello,
Have I missed something, or are there really no WU available for crunching? Quite surprising, it's the first time I see something like that since I joined MW.

Hello,

The answer is here : https://milkyway.cs.rpi.edu/milkyway/server_status.php

As you can see, the "stream_fit_work_generator (milkyway)" and the "nbody_work_generator (milkyway_nbody)" are stopped.
The questions are now : why MilkyWay stopped the generators yesterday and when MilkyWay intends to start them again ?

Tom, thank you to inform us.

Jean-Pierre

Yes, I saw that the generators are down and was wondering whether someone could shed some light on why, when, etc.
Also, how is it possible that there are 600K+ tasks in progress?! Someone knew that the generators will be down and was hoarding taks of what ?

The 600K+ tasks in progress are those sent before the generators stop and being calculated right now on the computers of Internet users participating in the MylkyWay project. Their number is steadily decreasing.
The question is to know why nobody at MilkyWay informed us about this shutdown.

I hope that Tom Donlon will inform us as soon as possible.