Welcome to MilkyWay@home

Server Downtime March 28, 2022 (12 hours starting 00:00 UTC)


Advanced search

Message boards : News : Server Downtime March 28, 2022 (12 hours starting 00:00 UTC)
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · 13 . . . 15 · Next

AuthorMessage
arcturus

Send message
Joined: 20 Nov 07
Posts: 54
Credit: 2,576,446
RAC: 7,760
2 million credit badge15 year member badge
Message 72514 - Posted: 5 Apr 2022, 15:22:55 UTC

At the snail's pace it's taking to convert inconclusive to validated, probably exacerbated by defections, I figure it's going to be long time before things return to normal, barring another server mishap.

One has to question the long term viability and commitment of a project operating on a shoestring budget, eliciting contributions from volunteers, and having no contingencies in place to avoid a repeat. Other projects have hick-ups now & then tho I'm hard pressed to point one where issues have lasted this long. Unfortunately it's sometimes a sign of forthcoming project mortality. The FLOPS this project gets is cheap computational power compared to the outlay. So hello project managers, what's your game plan? What kind of commitment can you give us? I'm happily crunching other projects for now and we would certainly appreciate some outreach from you.
ID: 72514 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
unixchick
Avatar

Send message
Joined: 21 Feb 22
Posts: 66
Credit: 654,295
RAC: 6,849
500 thousand credit badge
Message 72519 - Posted: 5 Apr 2022, 17:57:44 UTC

looks like the hiccup the system had is almost over. My invalids are going down (slowly, but it is progress) and while waiting for validation is rising (Workunits waiting for validation 68398) it is still low. Hiccups happen, but it is nice to have Tom communicate what is going on while fixes are happening.
ID: 72519 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Atratus

Send message
Joined: 21 Mar 09
Posts: 2
Credit: 36,642,641
RAC: 0
30 million credit badge13 year member badge
Message 72524 - Posted: 5 Apr 2022, 20:20:24 UTC - in response to Message 72475.  

I have suggested we all chip in and buy some up to date hardware. Quite how Tom came up with 10 grand just for SSDs I don't know.


Or you could rent some cloud resources and attach them to the project to help speed up clearing the ready to send queue?

I am renting out some azure and aws offerings to help speed this up a bit


Hi Tom, Hi all,

this is just a calculation.

I have compared what my Raid6 will cost to Backup in a cloud and it were too expensive.
So, I had bought my 8 x8TB Hard Drives and built my own Raid6@Home

And, what should I say: It runs since 5 Years like a Charm 24/7 :-)

The only things I had to do were some managements (installing Upgrades, Cleaning the Fans).

Here is the german price from today:
https://www.idealo.de/preisvergleich/OffersOfProduct/6639979_-ironwolf-8tb-st8000vn004-seagate.html

There is no need to choose the cheapest Drive.
If you have some more traffic, just choose a helium (it's cooler) or a higher level Drive (lasts longer).

Just for the high frequented Spaces you should choose a ssd which is just acting like a buffer...

This is just my opinion, but I have also my own Raid6 with just 64TB (Useable Space ~45TB)
This is a lot of fun :-)

--
Kind Regards
Atratus
ID: 72524 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
poppinfresh99

Send message
Joined: 28 Feb 22
Posts: 14
Credit: 1,589,949
RAC: 10,012
1 million credit badge
Message 72529 - Posted: 6 Apr 2022, 3:05:07 UTC - in response to Message 72514.  

One has to question the long term viability and commitment of a project operating on a shoestring budget, eliciting contributions from volunteers, and having no contingencies in place to avoid a repeat. Other projects have hick-ups now & then tho I'm hard pressed to point one where issues have lasted this long. Unfortunately it's sometimes a sign of forthcoming project mortality. The FLOPS this project gets is cheap computational power compared to the outlay. So hello project managers, what's your game plan? What kind of commitment can you give us? I'm happily crunching other projects for now and we would certainly appreciate some outreach from you.


Because this project accepts and receives donations, it should be able to handle issues like drive failure or temporary WCG crunchers. Accepting donations seems to imply that they intend to grow the GFLOPS of the project, though, to be fair, they never say what donations will be used for.

For what purpose does this project accept donations? Does this project want or need more GFLOPS?
ID: 72529 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Peter Hucker
Avatar

Send message
Joined: 5 Jul 11
Posts: 741
Credit: 334,518,263
RAC: 88,397
300 million credit badge11 year member badge
Message 72530 - Posted: 6 Apr 2022, 3:20:16 UTC - in response to Message 72529.  

One has to question the long term viability and commitment of a project operating on a shoestring budget, eliciting contributions from volunteers, and having no contingencies in place to avoid a repeat. Other projects have hick-ups now & then tho I'm hard pressed to point one where issues have lasted this long. Unfortunately it's sometimes a sign of forthcoming project mortality. The FLOPS this project gets is cheap computational power compared to the outlay. So hello project managers, what's your game plan? What kind of commitment can you give us? I'm happily crunching other projects for now and we would certainly appreciate some outreach from you.


Because this project accepts and receives donations, it should be able to handle issues like drive failure or temporary WCG crunchers. Accepting donations seems to imply that they intend to grow the GFLOPS of the project, though, to be fair, they never say what donations will be used for.

For what purpose does this project accept donations? Does this project want or need more GFLOPS?
I assume the humans' pay is a big part of the money used.
ID: 72530 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileBill F
Avatar

Send message
Joined: 4 Jul 09
Posts: 53
Credit: 13,468,925
RAC: 10,740
10 million credit badge13 year member badge
Message 72543 - Posted: 6 Apr 2022, 22:44:57 UTC - in response to Message 72530.  

And why would you assume that human pay is either a part or a big part ?

Thanks
Bill F
In October of 1969 I took an oath to support and defend the Constitution of the United States against all enemies, foreign and domestic;
There was no expiration date.


ID: 72543 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Peter Hucker
Avatar

Send message
Joined: 5 Jul 11
Posts: 741
Credit: 334,518,263
RAC: 88,397
300 million credit badge11 year member badge
Message 72544 - Posted: 6 Apr 2022, 23:12:41 UTC - in response to Message 72543.  

And why would you assume that human pay is either a part or a big part ?
Because most people don't work for free. I'm quite surprised you asked.
ID: 72544 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Skillz

Send message
Joined: 28 May 17
Posts: 21
Credit: 1,777,280,629
RAC: 18,640,059
1 billion credit badge5 year member badge
Message 72545 - Posted: 7 Apr 2022, 2:23:09 UTC - in response to Message 72544.  

And why would you assume that human pay is either a part or a big part ?
Because most people don't work for free. I'm quite surprised you asked.


Well considering its a university and most of the "staff" are students at the University who actually pay to attend. I'd say they not only work for free, but they pay to work on the project in the form of tuition.
ID: 72545 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Peter Hucker
Avatar

Send message
Joined: 5 Jul 11
Posts: 741
Credit: 334,518,263
RAC: 88,397
300 million credit badge11 year member badge
Message 72546 - Posted: 7 Apr 2022, 3:10:34 UTC - in response to Message 72545.  

And why would you assume that human pay is either a part or a big part ?
Because most people don't work for free. I'm quite surprised you asked.
Well considering its a university and most of the "staff" are students at the University who actually pay to attend. I'd say they not only work for free, but they pay to work on the project in the form of tuition.
The professors get paid. There are running costs to everything. I can't believe you think this stuff is all free. I used to work in a university and I can tell you everything you've said is utter nonsense.
ID: 72546 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Skillz

Send message
Joined: 28 May 17
Posts: 21
Credit: 1,777,280,629
RAC: 18,640,059
1 billion credit badge5 year member badge
Message 72547 - Posted: 7 Apr 2022, 3:37:58 UTC - in response to Message 72546.  

And why would you assume that human pay is either a part or a big part ?
Because most people don't work for free. I'm quite surprised you asked.
Well considering its a university and most of the "staff" are students at the University who actually pay to attend. I'd say they not only work for free, but they pay to work on the project in the form of tuition.
The professors get paid. There are running costs to everything. I can't believe you think this stuff is all free. I used to work in a university and I can tell you everything you've said is utter nonsense.


Yeah, the professors do get paid, but it's not the "MW@Home" budget and donations to the project won't go towards the professors salaries.
The grad students that help manage the project get paid with a degree.

Much like all the work we, the volunteers are doing, is also... free.

The only utter nonsense here is thinking people HAVE to get paid before they're willing to help out on anything.
ID: 72547 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
unixchick
Avatar

Send message
Joined: 21 Feb 22
Posts: 66
Credit: 654,295
RAC: 6,849
500 thousand credit badge
Message 72548 - Posted: 7 Apr 2022, 3:46:16 UTC

Hey Tom.

Is it normal for the waiting for validation to grow? or it is a sign of an issue? Workunits waiting for validation 412415

The status page hasn't updated in a while (Task data as of 6 Apr 2022, 20:37:54 UTC) which also concerns me.

Thanks.
ID: 72548 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Peter Hucker
Avatar

Send message
Joined: 5 Jul 11
Posts: 741
Credit: 334,518,263
RAC: 88,397
300 million credit badge11 year member badge
Message 72549 - Posted: 7 Apr 2022, 3:54:36 UTC - in response to Message 72547.  
Last modified: 7 Apr 2022, 3:55:32 UTC

Yeah, the professors do get paid, but it's not the "MW@Home" budget and donations to the project won't go towards the professors salaries.
The grad students that help manage the project get paid with a degree.

Much like all the work we, the volunteers are doing, is also... free.

The only utter nonsense here is thinking people HAVE to get paid before they're willing to help out on anything.
Maybe the students are free, but others are paid for. Secretaries, IT support, etc. When I worked in a university, every department had to rent a number of rooms from the central university budget, so there's that. The server I presume is in a room. And I'm guessing for Milkyway, they have to buy telescope data.
ID: 72549 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Kiska

Send message
Joined: 31 Mar 12
Posts: 88
Credit: 149,784,912
RAC: 58,765
100 million credit badge10 year member badge
Message 72550 - Posted: 7 Apr 2022, 5:27:29 UTC - in response to Message 72549.  

When I was working in university it support. I know for mine for public projects you also had to pay for bandwidth that it consumed billed at 90th percentile for both inbound and outbound transfers.

Guess what I am trying to say is don't be too harsh on Tom since he is a phd student and that activity takes time away from here(such as teaching, grading, writing his thesis etc etc)

Also for a system administrator working full time it's roughly a $50k starting position
ID: 72550 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Kiska

Send message
Joined: 31 Mar 12
Posts: 88
Credit: 149,784,912
RAC: 58,765
100 million credit badge10 year member badge
Message 72551 - Posted: 7 Apr 2022, 5:29:47 UTC - in response to Message 72548.  

Hey Tom.

Is it normal for the waiting for validation to grow? or it is a sign of an issue? Workunits waiting for validation 412415

The status page hasn't updated in a while (Task data as of 6 Apr 2022, 20:37:54 UTC) which also concerns me.

Thanks.


I believe that is normal as people submit tasks back it adds to validation queue and if we are starting to make way in the work unit backlog. As far as I can see the validator is working just slowly
ID: 72551 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Peter Hucker
Avatar

Send message
Joined: 5 Jul 11
Posts: 741
Credit: 334,518,263
RAC: 88,397
300 million credit badge11 year member badge
Message 72552 - Posted: 7 Apr 2022, 5:36:17 UTC - in response to Message 72550.  
Last modified: 7 Apr 2022, 5:36:39 UTC

Also for a system administrator working full time it's roughly a $50k starting position
Lucky you, I got $26K, after 5 years. The UK doesn't pay workers very well. I assume you're talking US dollars?
ID: 72552 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
arcturus

Send message
Joined: 20 Nov 07
Posts: 54
Credit: 2,576,446
RAC: 7,760
2 million credit badge15 year member badge
Message 72555 - Posted: 7 Apr 2022, 14:04:37 UTC - in response to Message 72551.  

As far as I can see the validator is working just slowly


That's an understatement. In a week maybe 3% of mine have been validated. Christmas season will be knocking before they're caught up at this rate.
ID: 72555 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileTom Donlon
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 10 Apr 19
Posts: 372
Credit: 96,823,521
RAC: 134,535
50 million credit badge3 year member badge
Message 72556 - Posted: 7 Apr 2022, 14:54:24 UTC

I just killed some stale processes in the DB. It seems like some queries get hung up because the WU tables are quite large at the moment, and then they slow everything down because they lock certain parts of the DB while they run. Killing them every so often seems to help. I had been trying to avoid doing that to see if the server could handle things on its own at this point, but apparently it still needs the regular kicks now and then to keep functioning as per normal.

Sorry about the validation backlog and the slowness, hopefully what I just did speeds things up a bit. I'll keep killing stale tasks regularly until the validation backlog AND the WUs ready to send both decrease.

As for the pay, if you donate to milkyway, none of that goes into my (or any other milkyway personnel's) pocket. The funds are used strictly for project upgrades and development. Our salaries are funded by grants and/or teaching, not volunteer donations. We don't ask for donations because they make us richer, they just help us make the project better. Just wanted to clear that up!
ID: 72556 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Kiska

Send message
Joined: 31 Mar 12
Posts: 88
Credit: 149,784,912
RAC: 58,765
100 million credit badge10 year member badge
Message 72557 - Posted: 7 Apr 2022, 16:17:10 UTC - in response to Message 72555.  

As far as I can see the validator is working just slowly


That's an understatement. In a week maybe 3% of mine have been validated. Christmas season will be knocking before they're caught up at this rate.


Remember that your "validation inconclusive" also contains workunits where the other computer hasn't sent back yet.
ID: 72557 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Kiska

Send message
Joined: 31 Mar 12
Posts: 88
Credit: 149,784,912
RAC: 58,765
100 million credit badge10 year member badge
Message 72558 - Posted: 7 Apr 2022, 16:20:59 UTC - in response to Message 72552.  

Also for a system administrator working full time it's roughly a $50k starting position
Lucky you, I got $26K, after 5 years. The UK doesn't pay workers very well. I assume you're talking US dollars?


I live in Australia, and for us "junior system admin" is a $50-80k per year salary depending on experience. Also I was quoting Australian dollar. So USD would be $37k starting
ID: 72558 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Kiska

Send message
Joined: 31 Mar 12
Posts: 88
Credit: 149,784,912
RAC: 58,765
100 million credit badge10 year member badge
Message 72559 - Posted: 7 Apr 2022, 16:25:18 UTC

Hey Tom:

I guess now that the Separation queue ready to send is almost 0, you can turn that work generator back on
ID: 72559 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · 13 . . . 15 · Next

Message boards : News : Server Downtime March 28, 2022 (12 hours starting 00:00 UTC)

©2022 Astroinformatics Group