I guess its time to do daily postings of what the graph looks like for workunits and discussions(?):

I guess its time to do daily postings of what the graph looks like for workunits and discussions(?):

5 Million waiting for validation by the end of tomorrow?

I guess its time to do daily postings of what the graph looks like for workunits and discussions(?):

5 Million waiting for validation by the end of tomorrow?

I am going to omit the graph from the quote.

But yes it does seem to be trending that way

Yes, it's obvious that the computers at Milkyway are not able to keep up with the WU's we are producing. I don't understand why they are not able to keep up since we are seemingly not finishing anymore now than when it suddenly stopped being able to keep up. The System always says everything is running and I've heard very little from Tom on the subject.
Wish we knew what is wrong.

I guess its time to do daily postings of what the graph looks like for workunits and discussions(?):

Yes, please do.

Yes, it's obvious that the computers at Milkyway are not able to keep up with the WU's we are producing. I don't understand why they are not able to keep up since we are seemingly not finishing anymore now than when it suddenly stopped being able to keep up. The System always says everything is running and I've heard very little from Tom on the subject.
Wish we knew what is wrong.

The workunit generators don't generate tasks (wingman tasks or initial tasks) if the WU pools have more tasks than they should. So when the nbody pool had like 100k tasks in it, any tasks that you sent back were essentially put on hold because the WU generator wouldn't make any wingman tasks for validation until the pool was cleared.

It's not so much that the "computer can't keep up" as it is that the WU pool being overfilled is a nasty bug. I want to migrate to the new server hardware ASAP but other people have to get things compiled and sorted out before that can happen. Currently, the server is slow becaus eit is short on memory, so in that regard the computer can't keep up.

I guess its that time for a daily graph of server_status:

Yes, it's obvious that the computers at Milkyway are not able to keep up with the WU's we are producing. I don't understand why they are not able to keep up since we are seemingly not finishing anymore now than when it suddenly stopped being able to keep up. The System always says everything is running and I've heard very little from Tom on the subject.
Wish we knew what is wrong.

The workunit generators don't generate tasks (wingman tasks or initial tasks) if the WU pools have more tasks than they should. So when the nbody pool had like 100k tasks in it, any tasks that you sent back were essentially put on hold because the WU generator wouldn't make any wingman tasks for validation until the pool was cleared.

It's not so much that the "computer can't keep up" as it is that the WU pool being overfilled is a nasty bug. I want to migrate to the new server hardware ASAP but other people have to get things compiled and sorted out before that can happen. Currently, the server is slow becaus eit is short on memory, so in that regard the computer can't keep up.

Thanks Tom, really appreciate the update info. Is the memory the type that we can help you with or is it "special" and only something we can contribute money for you to buy or is it even worth it, since you are talking about migrating to a new server anyway? It certainly will be nice when everything is updated and running smoothly again.
Thanks for your great efforts!!

Allen

Yes, it's obvious that the computers at Milkyway are not able to keep up with the WU's we are producing. I don't understand why they are not able to keep up since we are seemingly not finishing anymore now than when it suddenly stopped being able to keep up. The System always says everything is running and I've heard very little from Tom on the subject.
Wish we knew what is wrong.

The workunit generators don't generate tasks (wingman tasks or initial tasks) if the WU pools have more tasks than they should. So when the nbody pool had like 100k tasks in it, any tasks that you sent back were essentially put on hold because the WU generator wouldn't make any wingman tasks for validation until the pool was cleared.

It's not so much that the "computer can't keep up" as it is that the WU pool being overfilled is a nasty bug. I want to migrate to the new server hardware ASAP but other people have to get things compiled and sorted out before that can happen. Currently, the server is slow becaus eit is short on memory, so in that regard the computer can't keep up.

Thanks Tom, really appreciate the update info. Is the memory the type that we can help you with or is it "special" and only something we can contribute money for you to buy or is it even worth it, since you are talking about migrating to a new server anyway? It certainly will be nice when everything is updated and running smoothly again.
Thanks for your great efforts!!

Allen

+1 What Allen said. How much $$ to bulk up the memory? If the memory is upgraded after the new sever gets online, the old one can be used as a backup, or it can be run in parallel with he new one. Maybe they could split the load 50/50. I presume you are going to need more hardware for the multi-galaxy sims anyway, so there's also that....

Is it possible to track 2 items on a graph ..would be interesting to see Nbody average run time and total waiting for validation on the same graph. My own, probably misguided view is things have not been the same since we got these very long Nbody Simulation WU’s.

I have messed around with the graphs and going back 90 days there seems to be a massive spike in Nbody average run time that coincides with the growth of waiting for validation.Average Nbody run time is around 15 times higher than it used to be. I may be misguided in thinking the two are linked.

A daily graph:


Seems we have a small recovery

Is it possible to track 2 items on a graph ..would be interesting to see Nbody average run time and total waiting for validation on the same graph. My own, probably misguided view is things have not been the same since we got these very long Nbody Simulation WU’s.

I have messed around with the graphs and going back 90 days there seems to be a massive spike in Nbody average run time that coincides with the growth of waiting for validation.Average Nbody run time is around 15 times higher than it used to be. I may be misguided in thinking the two are linked.

You'll need to add this query:


Then add this override:


Then you'll get this graph:

Thanks for that Kiska most informative. There seems to be a good correlation to me , especially latterly.

Thanks again Kiska would it be possible to make that the daily graph please.

Here is the daily graph:


I am sure you'll note my weird posting times, that is because I do it when I get home from work :D

Also another change is the addition of average run times on the right axis

Thanks Tom, really appreciate the update info. Is the memory the type that we can help you with or is it "special" and only something we can contribute money for you to buy or is it even worth it, since you are talking about migrating to a new server anyway? It certainly will be nice when everything is updated and running smoothly again.
Thanks for your great efforts!!

Allen

The new server hardware has more than 2x the amount of memory as the current server, so once we migrate it should be less of a problem. The problem is migrating...!

Here is another graph I decided to add:

Thanks Tom, really appreciate the update info. Is the memory the type that we can help you with or is it "special" and only something we can contribute money for you to buy or is it even worth it, since you are talking about migrating to a new server anyway? It certainly will be nice when everything is updated and running smoothly again.
Thanks for your great efforts!!

Allen

The new server hardware has more than 2x the amount of memory as the current server, so once we migrate it should be less of a problem. The problem is migrating...!

Hope you're finding these graphs useful :D

btw sorry for the 1 request per minute to the server_status page

Thanks very useful…..

Thanks very useful…..