Welcome to MilkyWay@home

Server Downtime March 25, 2022 (24 hours starting 17:00 UTC)


Advanced search

Message boards : News : Server Downtime March 25, 2022 (24 hours starting 17:00 UTC)
Message board moderation

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
Kiska

Send message
Joined: 31 Mar 12
Posts: 88
Credit: 149,786,185
RAC: 35,892
100 million credit badge10 year member badge
Message 72250 - Posted: 26 Mar 2022, 16:19:00 UTC - in response to Message 72248.  

It looks like the server is back up and running, but the transitioner backlog is at 46 hours. Did the server rebuild the drive already?

Again, thanks for all the work that's being done for this project! It's fun for us to be able to support it.


It's down to a bit over 15 hours right now so YES it seems to be working!!
Well done Team MilkyWay!!!


Seems to be falling behind according to my stats: https://grafana.kiska.pw/goto/m27fH4snk?orgId=1
ID: 72250 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
unixchick
Avatar

Send message
Joined: 21 Feb 22
Posts: 66
Credit: 664,097
RAC: 5,271
500 thousand credit badge
Message 72251 - Posted: 27 Mar 2022, 4:07:48 UTC

Things still seem broken.
Workunits waiting for validation 4535388

I think you need to look at a log on the drives to see if the new drive is synced and see if any of the other drives are having issues.

Good Luck.
ID: 72251 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Septimus

Send message
Joined: 8 Nov 11
Posts: 186
Credit: 2,399,250
RAC: 4,664
2 million credit badge11 year member badge
Message 72252 - Posted: 27 Mar 2022, 6:46:09 UTC - in response to Message 72251.  
Last modified: 27 Mar 2022, 6:47:05 UTC

The system seems to be Ok, the numbers are still high but the backlog in days has reduced a lot. I have a higher than normal Validation Inconclusive count but most of my backlog is cleared. The system is a lot more responsive than it was.
ID: 72252 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Arne

Send message
Joined: 19 Mar 22
Posts: 2
Credit: 69,487
RAC: 0
10 thousand credit badge
Message 72253 - Posted: 27 Mar 2022, 8:18:30 UTC

What is frustrating is that I had to wait 8 days after registering, and downloading, computing, and uploading a work unit and its results to receive any credit and enable me to post anything, in particular to ask why the heck this takes so long. A FAQ would really do wonders. Like, I have (had) two computers crunching, one somewhat current (A) and the other a decade old (B). B's were validated first, despite B taking far longer to work the unit than any A received, and despite B downloading WU's at least a day later than A. (I actually had started crunching on B only to see if A was the problem for not receiving credits in any appreciable amount of time.)

Like, for days I wondered why I received no credit, because there is no section with or anything with information about it, or if there is it's not easy enough to jump out to me, and I couldn't ask anything because that requires at least one WU having been fully processed. Case in point: A has 0/321 WUs validated (70 pending, 160 inconclusive), while B got 1/28 valid (26 pending, 0 inconclusive). A FAQ that made it crayon-drawing clear the steps between WU being sent out and being credited with crunching them would really help, and If I am any metric, prevent much frustration and possible project-dropping due to apparently not being appreciated (with credits). To me it looked like I was simply burning computing time and I kept (and honestly still keep) wondering if MW@h was worth spending my crunching time on.

Keeping in mind I kept my old computer crunch a lot for the longest time (Rosetta, WCG, SETI) before I stopped a couple years ago.
ID: 72253 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
San-Fernando-Valley

Send message
Joined: 13 Apr 17
Posts: 215
Credit: 131,314,815
RAC: 15,424
100 million credit badge5 year member badgeextraordinary contributions badge
Message 72254 - Posted: 27 Mar 2022, 9:53:46 UTC - in response to Message 72253.  

Welcome to Milkway ...

@ARNE

What is frustrating is that I had to wait 8 days after registering, and downloading, computing, and uploading a work unit and its results to receive any credit and enable me to post anything, in particular to ask why the heck this takes so long.......

Please help me understand:
One does not have to be registered to be able to see the homepage.
There it says something about downtime.
Further thinking brings up the option to check the Message Board under the Community tab.
Then, really being smart, one sees the the first topic called "News"!
Clicking on that brings one to the "server downtime ...." thread, among others.
With a little patience one is able to recognise, after reading a little bit, that the project is/was having a "bad time".

So what is the problem?
One does have to be able to post to find out what is going on.

You're burning computing time AND getting, eventually, credits. That should be OK.

Of course you are frustrated, so are we.
I had, over the weekend, more than 55thousand tasks waiting for validation.
And let me tell you, they are now down to 40thousand waiting. And I'm still happy.

Just relax, enjoy the nice weather,
ID: 72254 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profilemikey
Avatar

Send message
Joined: 8 May 09
Posts: 2984
Credit: 497,558,105
RAC: 194,041
300 million credit badge13 year member badgeextraordinary contributions badge
Message 72255 - Posted: 27 Mar 2022, 11:44:40 UTC - in response to Message 72253.  

What is frustrating is that I had to wait 8 days after registering, and downloading, computing, and uploading a work unit and its results to receive any credit and enable me to post anything, in particular to ask why the heck this takes so long. A FAQ would really do wonders. Like, I have (had) two computers crunching, one somewhat current (A) and the other a decade old (B). B's were validated first, despite B taking far longer to work the unit than any A received, and despite B downloading WU's at least a day later than A. (I actually had started crunching on B only to see if A was the problem for not receiving credits in any appreciable amount of time.)

Like, for days I wondered why I received no credit, because there is no section with or anything with information about it, or if there is it's not easy enough to jump out to me, and I couldn't ask anything because that requires at least one WU having been fully processed. Case in point: A has 0/321 WUs validated (70 pending, 160 inconclusive), while B got 1/28 valid (26 pending, 0 inconclusive). A FAQ that made it crayon-drawing clear the steps between WU being sent out and being credited with crunching them would really help, and If I am any metric, prevent much frustration and possible project-dropping due to apparently not being appreciated (with credits). To me it looked like I was simply burning computing time and I kept (and honestly still keep) wondering if MW@h was worth spending my crunching time on.

Keeping in mind I kept my old computer crunch a lot for the longest time (Rosetta, WCG, SETI) before I stopped a couple years ago.


It's just MilkyWay going thru a rough patch right now, there is a Forum I think you can post in with zero credits called Getting Started if you ever get to that point again try it. As for why computer B got credits before computer A did that's just the luck of the draw as they say in poker.

In addition LOTS of older computers can and do successfully get and finish tasks at alot of projects and each and every one of them is appreciated by the project they crunch for.
ID: 72255 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Arne

Send message
Joined: 19 Mar 22
Posts: 2
Credit: 69,487
RAC: 0
10 thousand credit badge
Message 72257 - Posted: 27 Mar 2022, 19:00:19 UTC - in response to Message 72254.  

It's probably just the frustration talking, sorry about my tone. And yes, if I spend two hours clicking through the forum and half a dozen threads I do come to the conclusion that something's off, but that did require work I was not prepared to spend. I am, by habit and inclination, a lurker. I’d have been happy as a clam to contribute and see the numbers go up. It gives me a low dose of fuzzy feelings. I like numbers going up. Makes me feel useful.

Now let me paint the picture from my point of view:
Last time I used to crunch, about five years ago, I saw WU's being downloaded, analysed, results being uploaded, and credit being given in the span of hours to days. Just to frame my expectations.
Recently I got a new computer, one where I’m not feeling vaguely guilty by having it crunch numbers instead of putting it in standby or powering down (that old computer is over a decade old). I think to myself, hey, why not fire up BOINC and crunch again? So I do.
Two out the three previous projects I crunched for have gone belly up in some way, the scheduler doesn’t receive any WUs in any case. Okay, let’s see what projects resonate. I skim a couple descriptions, MW@h looks nice. Forum’s been actively posted in the last couple days, so it’s apparently not gone the way of the dodo.
I register, download WUs, crunch them, upload results, and wait for credits being received. I wait one day, because sure, they might be swamped. I wait two days, because eh, why not. I fire up BOINC on my old computer and let it crunch WUs for WH@h to see if my new computer is the problem. I wait three days and get nervous. What is taking so long?
I check the project website. There’s no FAQ. FAQ’s are like a staple, at least in my expectation, they’ve been around since the usenet. Whatever’s going on here?
I check the forums. 3 out of the ten most recent news threads include the word 'server'. Oh boy. "Server Downtime", "Server Trouble", "Server Outages". Oh boy oh boy.
There’s no FAQ, most recent news is about failing servers, what's going on? Did I join a dying project? Did I waste my crunch time? Maybe a closer look will help?
I read a couple entries, find out that a guy named Tom is responsible for the project, or at least many people refer to him and he's active in replying, but what’s going on with the forum? Why the heck are quotes bigger than the reply text below? I’m left with a vague feeling like I’ve stumbled into a backyard off some craigslist entry. Everyone is fully cognisant and accepting and accommodating.
Okay, this feels off. Better ask in the forum.
Huh. I can’t open threads or post replies until I got some credits. Apparently these guys had some spam problems, and this is a nifty way to circumvent that. Neat!
But this is the opposite to a solution to my problem.
I pause the crunching (or rather, I don’t download new WUs) and wait.
And wait.
And wait. Days passed. A new thread opens up with "Server Problems". I get more suspicious.
Finally one (1) WU gets validated. I can post.

But I post with a very sour taste in my mouth.

Don’t get me wrong, I can understand why y’all are fine with that when that’s within you expectations, or you’d been forewarned, but to me this felt like entering a different public bus line than usual and seeing a puddle of vomit in the aisle. It doesn’t matter that this only happens once a year or something, this first impression will undermine any goodwill and colour my expectations for the future.
ID: 72257 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileTom Donlon
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 10 Apr 19
Posts: 372
Credit: 97,447,146
RAC: 132,248
50 million credit badge3 year member badge
Message 72258 - Posted: 27 Mar 2022, 19:30:00 UTC - in response to Message 72257.  
Last modified: 27 Mar 2022, 19:35:47 UTC

Sorry to hear about your experience Arne. I understand that this leaves a sour taste in the mouth of new volunteers. Yes, the server is going through a rough patch at the moment and things aren't usually like this. We had a drive fail on the server that led to memory issues because the server was constantly rebuilding from parity. A new drive has been installed, but it is taking a long time to rebuild.

Believe me when I say that I am as frustrated as all of you are. I am trying to communicate these problems upwards.
ID: 72258 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
WMD

Send message
Joined: 15 Jun 13
Posts: 15
Credit: 1,716,357,901
RAC: 1,855,740
1 billion credit badge9 year member badgeextraordinary contributions badge
Message 72261 - Posted: 27 Mar 2022, 22:05:46 UTC - in response to Message 72249.  

I don't know if the drive is rebuilt yet, but I did turn off the WU generators.

Did you, actually? The number of "Tasks ready to send" is higher than it was yesterday, so it seems more work is still being generated.
ID: 72261 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileTom Donlon
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 10 Apr 19
Posts: 372
Credit: 97,447,146
RAC: 132,248
50 million credit badge3 year member badge
Message 72263 - Posted: 28 Mar 2022, 14:37:48 UTC - in response to Message 72261.  

I believe that the "Tasks ready to send" also includes tasks that are going out for their second/third rounds of validation. So it will populate for a little while even without the WU generators being turned on.

To the best of my knowledge, the WU generators are turned off. I can't find any processes that correspond to them, and they've disappeared from the server status page.
ID: 72263 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileTom Donlon
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 10 Apr 19
Posts: 372
Credit: 97,447,146
RAC: 132,248
50 million credit badge3 year member badge
Message 72269 - Posted: 28 Mar 2022, 16:23:38 UTC

FYI I did just turn the WU generators back on in order to see if the server can catch up now that it is all the way rebuilt.
ID: 72269 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profilett2012tt

Send message
Joined: 30 Apr 20
Posts: 1
Credit: 1,028,583
RAC: 0
1 million credit badge2 year member badge
Message 72759 - Posted: 13 Apr 2022, 14:55:31 UTC - in response to Message 72212.  

Tom, is that Tom from MySpace? If you are you should tell us, I think we'd all like that ;) you should also change your profile pic to that classic myspace "Tom" profile pic and see if anyone remembers, roflol 🤣

BTW thank you for ALL your hard work and effort! 💯
ID: 72759 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 · 2

Message boards : News : Server Downtime March 25, 2022 (24 hours starting 17:00 UTC)

©2022 Astroinformatics Group