Welcome to MilkyWay@home

Problems downloading Stars and Volume files

Message boards : Number crunching : Problems downloading Stars and Volume files
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Barbud [USA]
Avatar

Send message
Joined: 30 Aug 07
Posts: 15
Credit: 6,390
RAC: 0
Message 207 - Posted: 8 Nov 2007, 13:47:12 UTC

I may have spoken to soon on the other thread but, it still appears that the stars.txt and volume.txt files are hanging up on download. I know that during the night(~11:30pmCST) when I was on the website that the website locked up with too many concurrent connections. So maybe there is a problem with too many people trying to download the files at once.
ID: 207 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
darkpella

Send message
Joined: 8 Oct 07
Posts: 6
Credit: 75,628
RAC: 0
Message 214 - Posted: 8 Nov 2007, 16:17:07 UTC - in response to Message 207.  

I may have spoken to soon on the other thread but, it still appears that the stars.txt and volume.txt files are hanging up on download. I know that during the night(~11:30pmCST) when I was on the website that the website locked up with too many concurrent connections. So maybe there is a problem with too many people trying to download the files at once.


Looks like I got the same problem now (didn't meet it with many other WUs my hosts downloaded earlier today

Here's BOINC's log:
08/11/2007 17.13.14|Milkyway@home|[file_xfer] Started download of file stars.txt
08/11/2007 17.13.16|Milkyway@home|[file_xfer] Temporarily failed download of stars.txt: file not found
08/11/2007 17.13.16|Milkyway@home|Backing off 14 min 26 sec on download of file stars.txt
08/11/2007 17.13.18|Milkyway@home|[file_xfer] Started download of file parameters_generated_1194572017_9188
08/11/2007 17.13.19|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194572017_9188: file not found
08/11/2007 17.13.19|Milkyway@home|Backing off 7 min 45 sec on download of file parameters_generated_1194572017_9188
ID: 214 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
BobCat13

Send message
Joined: 29 Aug 07
Posts: 2
Credit: 346,188
RAC: 0
Message 215 - Posted: 8 Nov 2007, 16:40:59 UTC - in response to Message 207.  

Even if you get stars.txt and volume.txt to download, once you run out of tasks they are deleted from the project directory. So now when more tasks become available, stars.txt and volume.txt will have to be downloaded again.

If stars.txt and volume.txt are the same for all tasks, shouldn't they be set as non-delete in the server configuration? I solved this by making those two files read-only, but if the project changes them then all tasks will fail for me.

Here are the two messages in the boinc log showing that the project tried to delete these files when I ran out of work:

Milkyway@home 11/8/2007 10:22:24 AM [error] Couldn't delete file projects/milkyway.cs.rpi.edu_milkyway/stars.txt
Milkyway@home 11/8/2007 10:22:31 AM [error] Couldn't delete file projects/milkyway.cs.rpi.edu_milkyway/volume.txt

ID: 215 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Ray Murray
Avatar

Send message
Joined: 8 Oct 07
Posts: 24
Credit: 111,325
RAC: 0
Message 219 - Posted: 8 Nov 2007, 18:51:44 UTC
Last modified: 8 Nov 2007, 18:52:06 UTC

I would guess that there will be a limit as to how many times or for how long it will request the missing files before giving up. Any idea how long that might be?
All excited about getting my first pair of wus, to find them stuck in download for the last 5 hours.

Candidate for ATA membership.
ID: 219 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
[PST]Howard
Avatar

Send message
Joined: 31 Aug 07
Posts: 21
Credit: 21,004,179
RAC: 0
Message 220 - Posted: 8 Nov 2007, 19:09:48 UTC
Last modified: 8 Nov 2007, 20:01:31 UTC

There may have been a problem with the server as there were 4700 (and rising) WUs waiting for validation, but that seems to be resolved now.
ID: 220 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Aug 07
Posts: 2046
Credit: 26,480
RAC: 0
Message 226 - Posted: 8 Nov 2007, 20:29:04 UTC - in response to Message 220.  

There may have been a problem with the server as there were 4700 (and rising) WUs waiting for validation, but that seems to be resolved now.



We've been testing the first version of using a genetic search to automatically generate assimilate and generate work units - and it looks like there's been a problem where it was creating too many work units. For some reason the stars and volume.txt files got lost in the process of doing the search, so thats why you all had problems. Hopefully the next one we run (today or tomorrow) will work a bit more smoothly :)

We definitely appreciate your work and hope you hang in there with us while we get all the bugs sorted out.
ID: 226 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
[PST]Howard
Avatar

Send message
Joined: 31 Aug 07
Posts: 21
Credit: 21,004,179
RAC: 0
Message 229 - Posted: 8 Nov 2007, 20:32:21 UTC

Thanx Travis
ID: 229 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Barbud [USA]
Avatar

Send message
Joined: 30 Aug 07
Posts: 15
Credit: 6,390
RAC: 0
Message 238 - Posted: 8 Nov 2007, 22:03:25 UTC - in response to Message 226.  

We definitely appreciate your work and hope you hang in there with us while we get all the bugs sorted out.

Still having issues and now I'm starting to see other files not downloading as well:

11/8/2007 4:01:44 PM|Milkyway@home|[file_xfer] Started download of file volume.txt
11/8/2007 4:01:45 PM|Milkyway@home|[file_xfer] Started download of file stars.txt
11/8/2007 4:01:46 PM|Milkyway@home|[file_xfer] Temporarily failed download of stars.txt: file not found
11/8/2007 4:01:46 PM|Milkyway@home|[file_xfer] Temporarily failed download of volume.txt: file not found
11/8/2007 4:01:46 PM|Milkyway@home|[file_xfer] Started download of file parameters_generated_1194561965_872
11/8/2007 4:01:47 PM|Milkyway@home|[file_xfer] Started download of file parameters_generated_1194561964_863
11/8/2007 4:01:48 PM|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194561964_863: file not found
11/8/2007 4:01:48 PM|Milkyway@home|[file_xfer] Started download of file parameters_generated_1194561964_867
11/8/2007 4:01:48 PM|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194561965_872: file not found
11/8/2007 4:01:49 PM|Milkyway@home|[file_xfer] Started download of file parameters_generated_1194561964_862
11/8/2007 4:01:49 PM|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194561964_867: file not found
11/8/2007 4:01:50 PM|Milkyway@home|[file_xfer] Started download of file parameters_generated_1194561964_861
11/8/2007 4:01:50 PM|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194561964_862: file not found
11/8/2007 4:01:51 PM|Milkyway@home|[file_xfer] Started download of file parameters_generated_1194561964_860
11/8/2007 4:01:51 PM|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194561964_861: file not found
11/8/2007 4:01:52 PM|Milkyway@home|[file_xfer] Started download of file parameters_generated_1194561963_858
11/8/2007 4:01:52 PM|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194561964_860: file not found
11/8/2007 4:01:53 PM|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194561963_858: file not found

Are there other issues here besides the server being overwhelmed by requests?

ID: 238 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Aug 07
Posts: 2046
Credit: 26,480
RAC: 0
Message 240 - Posted: 8 Nov 2007, 22:20:34 UTC - in response to Message 238.  

I think the server is doing ok... i'm not quite sure why files are going missing. I've started up a new search and hopefully that will fix the problem of too many work units being generated -- i think that had something to do with files going missing. Hopefully I'll figure out the problem soon.
ID: 240 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 29 Aug 07
Posts: 115
Credit: 502,662,458
RAC: 3,243
Message 243 - Posted: 8 Nov 2007, 23:02:44 UTC - in response to Message 240.  

I think the server is doing ok... i'm not quite sure why files are going missing. I've started up a new search and hopefully that will fix the problem of too many work units being generated -- i think that had something to do with files going missing. Hopefully I'll figure out the problem soon.


About half of my machines picked up work, and about half of those had stuck files downloading.



ID: 243 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Bill

Send message
Joined: 3 Oct 07
Posts: 21
Credit: 49,862
RAC: 0
Message 247 - Posted: 8 Nov 2007, 23:23:09 UTC - in response to Message 243.  

I think the server is doing ok... i'm not quite sure why files are going missing. I've started up a new search and hopefully that will fix the problem of too many work units being generated -- i think that had something to do with files going missing. Hopefully I'll figure out the problem soon.


About half of my machines picked up work, and about half of those had stuck files downloading.


I just posted this problem in Q&A - Windows
The project directory didn't exist. I manually created it and the downloads started right up
ID: 247 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 29 Aug 07
Posts: 115
Credit: 502,662,458
RAC: 3,243
Message 248 - Posted: 8 Nov 2007, 23:37:06 UTC - in response to Message 247.  

I think the server is doing ok... i'm not quite sure why files are going missing. I've started up a new search and hopefully that will fix the problem of too many work units being generated -- i think that had something to do with files going missing. Hopefully I'll figure out the problem soon.


About half of my machines picked up work, and about half of those had stuck files downloading.


I just posted this problem in Q&A - Windows
The project directory didn't exist. I manually created it and the downloads started right up

The project directory does exist on mine, but still I have stuck downloads.

ID: 248 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Barbud [USA]
Avatar

Send message
Joined: 30 Aug 07
Posts: 15
Credit: 6,390
RAC: 0
Message 254 - Posted: 9 Nov 2007, 3:46:38 UTC

It's interesting how the files get stuck in the same position on the progress meter in the transfers tab:

Parameters file is always at 32.36%
Stars file is at 0.00%
Volume file is at 100.00%

Even though the volume.txt file is at 100% it is still in the transfer tab. I have to abort some downloads because they were not being recognized as being downloaded at the client but the server would say I had the WU for hours. I wonder if the scheduler is clearing out the files to fast after the transfer? I am just shooting spitballs here. Anybody else got an idea?
ID: 254 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Pepo
Avatar

Send message
Joined: 18 Sep 07
Posts: 7
Credit: 33,809
RAC: 0
Message 257 - Posted: 9 Nov 2007, 7:03:33 UTC - in response to Message 254.  
Last modified: 9 Nov 2007, 7:05:13 UTC

It's interesting how the files get stuck in the same position on the progress meter in the transfers tab:

Parameters file is always at 32.36%
Stars file is at 0.00%
Volume file is at 100.00%

Even though the volume.txt file is at 100% it is still in the transfer tab.

From my experiences with failing transmissions behind NTLM proxy server - Boinc receives some data packet sized around 300 kB, which tells (in HTML language, suitable for browsers), that e.g. the authentication was wrong or omitted or something else went wonky. Boinc just counts these bytes in into the amount to be transmitted - thus it was already enough for volume.txt, or already the first third of parameters_... or still less than 0.01% of stars.txt to be displayed. After the amount of bytes transferred is over (volume.txt) or the connection is cleanly closed (stars.txt, parameters_...), some higher code instance notices, that the checksum does not match at all and the false contents is thrown away. In the case of the large files, they are possibly tried to download the full amount...

In this case it can be different, but maybe there is something common in the scenario?

Peter
ID: 257 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Sou'westerly

Send message
Joined: 19 Sep 07
Posts: 5
Credit: 5,401
RAC: 0
Message 261 - Posted: 9 Nov 2007, 9:19:40 UTC

I’ve noticed this morning on this winXP machine that all files ending with a 4 figured number starting with a 3 have downloaded successfully, e.g.
09/11/2007 09:07:32|Milkyway@home|Finished download of parameters_generated_1194648761_3785
However all files ending with a 5 figure number beginning with a 4 have failed because the file has not been found, e.g.
09/11/2007 08:57:38|Milkyway@home|Giving up on download of parameters_generated_1194582769_44937: file not found
Hope that helps, Dave.
ID: 261 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Jayargh
Avatar

Send message
Joined: 8 Oct 07
Posts: 289
Credit: 3,690,838
RAC: 0
Message 264 - Posted: 9 Nov 2007, 16:46:34 UTC
Last modified: 9 Nov 2007, 16:46:53 UTC

Is the best course of action to just abort the downloads missing files? I assume these will never dl...right?
ID: 264 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Buster Gunn

Send message
Joined: 3 Oct 07
Posts: 5
Credit: 329,770
RAC: 0
Message 266 - Posted: 9 Nov 2007, 17:08:21 UTC
Last modified: 9 Nov 2007, 17:09:52 UTC

Stars & volume.txt downloaded and I also got about 300 w/u's to run, which I did. Now NEW W/U files to download are having the same problem as Stars and volume had before. Can't find file. I've aborted the downloads because they just keep trying but failing. ????????????
ID: 266 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile [B^S] ShanerX
Avatar

Send message
Joined: 28 Aug 07
Posts: 2
Credit: 1,093,644
RAC: 0
Message 269 - Posted: 9 Nov 2007, 17:38:09 UTC
Last modified: 9 Nov 2007, 17:41:29 UTC

I just posted in 'Platform Specific - Windows' ... should've read through these posts first. Getting same problem on one WinXPSP2 pc, only one d/l at a time.

11/9/2007 12:08:32 PM|Milkyway@home|[file_xfer] Started download of file parameters_generated_1194582644_44195
11/9/2007 12:08:34 PM|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194582644_44195: file not found
11/9/2007 12:08:34 PM|Milkyway@home|Backing off 2 hr 2 min 8 sec on download of file parameters_generated_1194582644_44195

File transferred 326/1000 bytes 32.60% Boinc Client: 5.8.15

I aborted the others, but will check stats if I get another. This file also is a five figure number beginning with '4', so that was a good suggestion to research from there if possible.

ID: 269 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 29 Aug 07
Posts: 115
Credit: 502,662,458
RAC: 3,243
Message 270 - Posted: 9 Nov 2007, 17:42:06 UTC

I am having stuck downloads for parameters* files quite frequently now. The stars and volume files, not quite as often.

ID: 270 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Ray Murray
Avatar

Send message
Joined: 8 Oct 07
Posts: 24
Credit: 111,325
RAC: 0
Message 271 - Posted: 9 Nov 2007, 19:04:16 UTC

I got the stars and volume files at the first attempt at this batch but can't get the parameters file for any of these 5 wus:

09/11/2007 16:46:28|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194569879_2880: file not found
09/11/2007 16:46:30|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194569879_2881: file not found
09/11/2007 16:46:33|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194569879_2882: file not found
09/11/2007 16:46:34|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194569880_2892: file not found
09/11/2007 16:46:35|Milkyway@home|[file_xfer] Temporarily failed download of parameters_generated_1194569880_2893: file not found

All have 3 failed downloads and a No Reply against them already so I'm not very hopeful 8¬(
ID: 271 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Number crunching : Problems downloading Stars and Volume files

©2024 Astroinformatics Group