Hello everyone, I have been receiving this error on my WU for a few days now:

<core_client_version>8.2.5</core_client_version>
<![CDATA[
<message>
Handle non valido.
(0x6) - exit code 6 (0x6)</message>
<stderr_txt>
<search_application> milkyway_nbody 1.93 Windows x86_64 double OpenMP, Crlibm </search_application>
Using OpenMP 16 max threads on a system with 16 processors
Running MilkyWay@home Nbody v1.93
Optimal Softening Length = 0.016610605295128 kpc
Dwarf Initial Position: [-2.402221970317321,-15.498762100105402,10.843819263173673]
Dwarf Initial Velocity: [175.826707232158043,18.114181126003757,85.239728932862391]
Failed to move file 'nbody_checkpoint_tmp_14324' to 'nbody_checkpoint' (15100): (null)Failed to move file 'nbody_checkpoint_tmp_14324' to 'nbody_checkpoint' (15105): (null)Failed to move file 'nbody_checkpoint_tmp_14324' to 'nbody_checkpoint' (15105): (null)Failed to move file 'nbody_checkpoint_tmp_14324' to 'nbody_checkpoint' (15105): (null)Failed to move file 'nbody_checkpoint_tmp_14324' to 'nbody_checkpoint' (15105): (null)Failed to move file 'nbody_checkpoint_tmp_14324' to 'nbody_checkpoint' (15105): (null)Failed to move file 'nbody_checkpoint_tmp_14324' to 'nbody_checkpoint' (15105): (null)Failed to update checkpoint 'nbody_checkpoint' with temporary (0): No error
Failed to write checkpoint
Error running system: NBODY_CHECKPOINT_ERROR (64)
strftime() failed called boinc_finish(6)

</stderr_txt>

I have already tried disconnecting and reconnecting the project, but without results.
I have also tried uninstalling and reinstalling the BOINC app, but without results.
I tried changing the project settings to process on only 8 CPUs and then on 1 CPU, in order to switch processing apps, but there were no results.
I haven't made any hardware changes to my PC. The only software change I made was creating a development drive with the ReFS filesystem (instead of NTFS) where my documents, pictures, videos, music, downloads, and the BOINC data directory are saved.
I have no issues with the other 33 BOINC projects I am connected to, but only with Milkyway@home. Can someone tell me why and help me find a solution?
Thank you very much.

Here are the details of my PC:
https://milkyway.cs.rpi.edu/milkyway/show_host_detail.php?hostid=600642

Might be antivirus blocking the temp checkpoint file by scanning it if your BOINC data dir is not excluded from scanning, so try that first.

---

Thank you for the response Link,
I use Microsoft Defender Antivirus and I had already excluded the BOINC data folder from the scan.
In the settings, I also added the app "milkyway_nbody_orbit_fitting_1.93_windows_x86_64__mt.exe" as allowed, for extra security, but it didn’t solve the problem.
Yet, before the creation of the development unit, on the same SSD and the same identification drive letter D:, it was working fine, even without excluding the folder from antivirus scans and including the nbody app as allowed.
It really seems that it doesn’t like the ReFS file system, while it does like NTFS.
It seems strange to me, but it is the only BOINC project that gives me problems.

Does the issue go away, if you move BOINC data back to NTFS drive? If yes, than it might be not the optimal solution for you, but simply keep it there, I have never used it, but I don't think there are any advantages of using ReFS for BOINC data (or you can try to hardlink the slots dir to NTFS drive if you want to keep the rest on the ReFS drive for some reason).

---

Here I am again, after performing various tests, which I will list below, and before each test, I closed the BOINC manager, waited for the processes to finish, changed the setting, restarted the PC, waited for it to complete its startup procedures, and restarted the BOINC manager to see what would happen. If it didn't work, I detached the MilkyWay@Home project, closed the BOINC manager, waited for the processes to finish, restarted the PC, waited for it to complete its startup procedures, and restarted the BOINC manager, reattached the MilkyWay@Home project to see what would happen, and if it didn't work, I repeated the procedure from the beginning with another test:

1) disabled "development drive protection" in Windows security settings and it doesn't work.

2) disabled "real-time protection" in Windows security settings and it doesn't work. So, Windows security can be ruled out as the problem.

3) I stopped requesting new WUs from all the other BOINC projects I was involved in.Connected, waited for them to finish processing all the projects, after which I disconnected all of them leaving only the MilkyWay@home project, but it didn't work.

4) Uninstalled BOINC manager, restarted the PC, moved the BOINC data directory from D: (SATA SSD drive with ReFS filesystem) to C: (NVMe system SSD drive with NTFS filesystem), installed BOINC manager and at the end of the installation started it to see if it worked and IT WORKS!!! So I would say that there is a bug in the application 'Milkyway@home N-Body Simulation with Orbit Fitting 1.93 (mt)' that does not write the checkpoint file to the ReFS filesystem.

I don't know if it may be relevant or not to my problem, but I read at the beginning of the files "EMD_v193_OCS_north_no_orbit.lua", "EMD_v193_OCS_north_orbit_fitting.lua" and "EMD_v193_OCS_north_orbit_fitting_MW2014.lua", in the directory "D:BOINCprojectsmilkyway.cs.rpi.edu_milkyway", this:
"-- /* Copyright (c) 2016 - 2018 Siddhartha Shelton */-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- ---- Client side Lua file. If using this compile with DNBODY_DEV_OPTIONS=OFF-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --"

After these tests, I discovered by chance that on drive D: with ReFS filesystem it did not support TRIM functionality (according to Microsoft it is supported only on disk pools and not on a single drive like in my case) and so I went back, deleted the D: volume, recreated it as a basic unit and then formatted it to the NTFS filesystem.
Then I uninstalled BOINC Manager, restarted the PC, moved the BOINC data directory from C: (the system disk SSD NVMe with NTFS filesystem) to D: (SATA SSD with NTFS filesystem), installed BOINC Manager, and at the end of the installation I started it to see if it worked, and IT WORKS!!! After that, I reconnected all the previous BOINC projects and everything works.

So I would say that there is a bug in the application 'Milkyway@home N-Body Simulation with Orbit Fitting 1.93 (mt)' that does not write the checkpoint file to the ReFS filesystem.

That's more than weird, an application should actually not need to care, what filesystem it is writing to, that's operating system's job. Creating a file and writing to it is done via API provided by Windows, so basically the application tells Windows to write that data to this file and Windows does it, not the application itself. It would be insane for many reasons, if all applications would be able to write files directly into the filesystem by itself.

I discovered by chance that on drive D: with ReFS filesystem it did not support TRIM functionality (according to Microsoft it is supported only on disk pools and not on a single drive like in my case)

It this case it's completely unsuitable for single SSDs.

---

So I would say that there is a bug in the application 'Milkyway@home N-Body Simulation with Orbit Fitting 1.93 (mt)' that does not write the checkpoint file to the ReFS filesystem.

That's more than weird, an application should actually not need to care, what filesystem it is writing to, that's operating system's job. Creating a file and writing to it is done via API provided by Windows, so basically the application tells Windows to write that data to this file and Windows does it, not the application itself. It would be insane for many reasons, if all applications would be able to write files directly into the filesystem by itself.

I understand your reasoning, but how do you explain to me that this is the only application, among all the other BOINC projects, that gives me issues?
I don't want to say something silly, but perhaps the compilation option 'NBODY_DEV_OPTIONS=OFF' could be the problem or some other compilation parameter of the application.
I forgot to mention that among the tests I did, when it was only processing this project, I tried to increase the time between each checkpoint, in the options > processing preferences > general data of the BOINC manager, from the usual 60 seconds to 10 minutes, but it didn't work.
I tried 20 minutes, but nothing.
I tried 30 minutes but nothing.
I tried 60 minutes, so it finished processing the WU before the checkpoint; in fact, it reached 100% processing, but in the end it has to write the result to the file, and thus it gives me the same error and incorrect processing.

Well, I don't have an explanation, never had anything like that. It's just weird. The only issues with applications not being able to write to a specific filesystem were applications, which need to write files over 4GB. That wasn't possible with FAT32, so it was necessary to run them either on NTFS or exFAT (or avoid files over 4GB if possible, than they worked absolutely fine on FAT32). So this was a limitation of the filesystem.

Here I have no idea what's happening, according to the std_err, the application is not unable to write the file, it's unable to rename it to the final name (checkpoint files are created under temporary name, than the previous checkpoint file is deleted and than the new file is renamed), however it's not written wether it's unable to do it because of the file doesn't exist or what else is the reason. Well, the devs have to look at that (if they can replicate it).

You could of course try to see what happens on ReFS formated HDD if you have one laying around, it might be some weird issue with that SSD when formated with ReFS and running without TRIM support. At least I would do it out of curiosity. :D

---

I forgot to mention that in the various previous tests, I had also tried with Bitlocker activated and deactivated, on the drive with ReFS filesystem, but it didn't change anything.
For more information on the C: drive, I only have the Windows 11 PRO 24H2 operating system and it's a Samsung 970 EVO 250 GB NVMe SSD with NTFS filesystem, while on the D: drive I have documents, downloads, images, shortcuts, music, favorites, searches, game saves, videos, and the BOINC data directory and it's a Samsung 860 EVO 250 GB SATA SSD with ReFS filesystem in the previous tests, now returned to NTFS for the following tests.So, I did some other tests such as:

1) I created a virtual disk vhdx on C: of 50 GB (minimum size to create a development unit with ReFS filesystem), formatted in ReFS, followed the procedures explained in previous posts, but it does not work and gives me the same error.
2) I moved the same virtual disk to D:, just to change drives to see if it would make any difference, but nothing.
3) I tried another old SSD that I had at home, an OCZ VERTEX 3 3.5" SATA of 150 GB, with a single partition formatted as ReFS, changed the drive letter to D: to avoid having to install BOINC, copied the BOINC data directory and repeated the previous procedures, but nothing, same error.
4) Since I had another OCZ VERTEX 3 3.5" SATA SSD of 150 GB, I did the same thing on this one too, but still nothing, same error.
5) Considering I have these 2 SSDs, I wanted to try to create a mirrored pool and format it as ReFS, but when I tried to copy the BOINC directory from the Samsung 860 to the pool, it became corrupted and I couldn't perform the tests. I tried to recreate the pool and reformat it in ReFS and everything went well, but as soon as I tried to copy the BOINC data directory again, the pool became corrupted once more and thus the end of the tests.

Soon I will try with HDDs as soon as I manage to find them somewhere in the house.

Well, than it's at least not specific to this drive, which should make it easier for the devs to replicate it.

Question: I don't think that's the issue here, but when doing all those tests, did you reinstall BOINC everytime pointing to the new data dir just to make sure all the permissions are set right? Don't need to uninstall first, just install it once more over the existing installation.

---

Question: I don't think that's the issue here, but when doing all those tests, did you reinstall BOINC everytime pointing to the new data dir just to make sure all the permissions are set right? Don't need to uninstall first, just install it once more over the existing installation.

It wasn't for the permissions that I had to reinstall BOINC, but because every time I created the development drive and then formatted it with the ReFS file system, it assigned the drive letter F:. As I had mentioned earlier, C: is where the operating system is, D: is my data, and E: is the BluRay burner (which I hadn't mentioned), so the first available letter is F:. Unfortunately, in this case, you have to uninstall and reinstall BOINC to change the data directory, because if you reinstall it over the previous installation, you get two options: either to repair the previous installation, which points to the old directory, or to uninstall the application. In fact, in my latest attempts to avoid continuously installing and uninstalling BOINC, I changed the drive letter of my data to F:, so that every time I created the development drive, it would assign D: as the letter, which pointed to the data directory of the last installation.

By doing these tests, I also discovered that the BOINC data directory must be called BOINC, because if you copy all the data from D:BOINC to F:, when you start the BOINC manager it doesn't recognize the video card, and consequently, if there were any GPU processing tasks, you lose them and it results in an error. Another thing I found out is that when you uninstall and reinstall BOINC to change the data directory, for example from D:BOINC to F:BOINC or vice versa, you need to ensure that both directories exist; otherwise, the installation gets stuck and you cannot proceed.

Here I am again with an update.
I took advantage of the release of Windows version 25H2 for a clean installation of both the Operating System and BOINC.
I created the usual F: development drive to have the ReFS file system and to create the BOINC data directory from scratch.
I restarted the PC and then BOINC, connected to the MilkyWay@home project, it did its initialization, downloaded everything needed, and finally the WU.
The processing started and shortly after, the same error returned with the application "Milkyway@home N-Body Simulation with Orbit Fitting 1.93 (mt)".
I recently noticed that the new version of the app "Milkyway@home N-Body Simulation 1.94 (mt)" and the related WUs have been released, so I wanted to try again to create the F: development drive with the ReFS file system and copy the BOINC data directory.
Reinstallation of BOINC for the data directory permissions.
Restarted PC and BOINC, downloads WU for the app "Milkyway@home N-Body Simulation 1.94 (mt)", the processing starts and shortly after the exact same processing error of the application "Milkyway@home N-Body Simulation with Orbit Fitting 1.93 (mt)" occurs again.

So, I would say that after all these tests, the two applications have a bug when writing the checkpoint file to the ReFS file system, which perhaps few or no one uses, and therefore it doesn't have priority compared to other more important bugs to fix.

The MilkyWay@home app on Windows uses Transactional NTFS to try to make its checkpointing robust, but without checking whether the filesystem supports it or falling back to an alternative method if it fails. TxF does not work on ReFS, hence all the “Failed to move file” errors. (It’s worth bearing in mind that this part of the MW code was written (and labelled “temporary”) fourteen years ago – before ReFS was released…)

You could try raising an issue at GitHub. (It’s worth bearing in mind that the last time an issue was addressed there was seven years ago…)

Thanks for the reply, Brian, I finally understood why it was only this BOINC project, out of the 34 I am connected to, that gave me problems with the ReFS file system.

But reading the Microsoft documentation from the link you provided, on the 'Transactional NTFS (TxF)' feature it says:

'[Microsoft strongly advises developers to use alternative means to meet the application's needs. Many scenarios for which TxF was developed can be achieved through simpler and more readily available techniques. Additionally, TxF may not be available in future versions of Microsoft Windows. For more information and alternatives to TxF, see Alternatives to Using Transactional NTFS.]'

So applications will need to be modified in the future to make them work.

Taking a look at the other BOINC projects, I noticed that the checkpoint files are about a kilobyte in size, not around 10 megabytes like the last WU processed by MW.

Isn't there another way to report the problem besides GitHub?

Thank you.

There’s the Application Code Discussion forum here, but that’s pretty quiet too…

FWIW I’ve opened an issue in the MilkyWay GitHub repository to report this bug.

Thank you Brian for opening the issue on GitHub.

In the meantime, I have started a discussion in the forum section you pointed out to me here:

https://milkyway.cs.rpi.edu/milkyway/forum_thread.php?id=5203

Let's see if this way we can awaken the interest in solving the problem.

Hey guys

Thanks for the work you've done looking into this. Since most of the work on this project is done by students that come and go it can be easy for us to lose track of some of things like this that need to be updated. We greatly appreciate any info on bugs you're able to give us.

If you do have issues to report we are more likely to see it if you post here in the forums than if you open an issue on github (no harm in doing both of course). I did somehow miss this thread though, so thanks for opening the new one.

As for this issue, you are right that it is not high priority for us since it only impacts a small number of individual users. I'll make sure to look into it but it may be pushed aside if I'm needed for something else.

Thank you very much for the reply, gimmyk.

Please keep us updated as soon as you have any developments on the applications in the future.

Good work.

I've added the missing fallback to the boinc version of the code, as well as an additional fallback to reading/writing the checkpoint that should help with some similar issues. We will keep using TxF for now; no reason to worry about changing that as long as it still works. But I think that these changes will eliminate a lot of the checkpoint errors we have been seeing.

Since this is a smaller issue we will wait until we have other features to update, so it will probably be some time until this fix goes live.

Thank you very much, gimmyk, for the update.

I saw that a new version of the "Milkyway@home N-Body Simulation with Orbit Fitting" application has been released. Does it include the changes you mentioned?

I need to do some testing to see how this new app behaves with the ReFS filesystem, or is it pointless as a test?