Welcome to MilkyWay@home

source v0.18


Advanced search

Message boards : Application Code Discussion : source v0.18
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · Next

AuthorMessage
ProfileTravis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Aug 07
Posts: 2046
Credit: 26,480
RAC: 0
10 thousand credit badge14 year member badge
Message 10641 - Posted: 14 Feb 2009, 9:30:38 UTC - in response to Message 10585.  

meh :( i cut like... 10 multiplies and a divide out of the calculate probabilities :(
ID: 10641 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profilespeedimic
Avatar

Send message
Joined: 22 Feb 08
Posts: 260
Credit: 57,387,048
RAC: 0
50 million credit badge13 year member badge
Message 10644 - Posted: 14 Feb 2009, 9:55:36 UTC - in response to Message 10641.  

meh :( i cut like... 10 multiplies and a divide out of the calculate probabilities :(


well, the improvement for the stock gcc-compiled app should be more than 30sec...

BTW, how do you like the new stderr?
mic.


ID: 10644 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profilespeedimic
Avatar

Send message
Joined: 22 Feb 08
Posts: 260
Credit: 57,387,048
RAC: 0
50 million credit badge13 year member badge
Message 10710 - Posted: 14 Feb 2009, 20:11:34 UTC

If someone feels like testing, here are the new v18(beta) apps:

Linux32 Pack

Linux64 Pack

LinuxAMD Pack

Report problems & errors here...
mic.


ID: 10710 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
WimTea

Send message
Joined: 16 Nov 07
Posts: 23
Credit: 4,774,710
RAC: 0
3 million credit badge14 year member badge
Message 10723 - Posted: 14 Feb 2009, 20:56:59 UTC

Guess i'll have to install some *nix real soon now on my new quad...

It runs XP64 after a few tries with drivers and stuff, but tx to a crappy GPU bios (reports itself as an engineering version, would you believe that?) I have no GPU crunching atm.

Now I only need a few 'recreational' hours (plus a few, plus a few, hey it's IT...) to install *nix. Any recommendations for the flavour? I'm not a total newbie, did some AIX sysadm in my early days (and not only smitty). But I would also like to use some regular programs like a spreadsheet, mail, photoeditor and stuff.
ID: 10723 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profilespeedimic
Avatar

Send message
Joined: 22 Feb 08
Posts: 260
Credit: 57,387,048
RAC: 0
50 million credit badge13 year member badge
Message 10724 - Posted: 14 Feb 2009, 21:08:50 UTC - in response to Message 10723.  

Guess i'll have to install some *nix real soon now on my new quad...

It runs XP64 after a few tries with drivers and stuff, but tx to a crappy GPU bios (reports itself as an engineering version, would you believe that?) I have no GPU crunching atm.

Now I only need a few 'recreational' hours (plus a few, plus a few, hey it's IT...) to install *nix. Any recommendations for the flavour? I'm not a total newbie, did some AIX sysadm in my early days (and not only smitty). But I would also like to use some regular programs like a spreadsheet, mail, photoeditor and stuff.


I usually use ubuntu for desktops and notebooks, debian for servers.

mic.


ID: 10724 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfilePhil
Avatar

Send message
Joined: 13 Feb 08
Posts: 1124
Credit: 46,740
RAC: 0
10 thousand credit badge13 year member badge
Message 10740 - Posted: 15 Feb 2009, 0:20:36 UTC - in response to Message 10710.  

If someone feels like testing, here are the new v18(beta) apps:

LinuxAMD Pack

Report problems & errors here...


No problems, ran test units and new ps work fine ... checkpointing OK.

ID: 10740 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
John Clark

Send message
Joined: 4 Oct 08
Posts: 1734
Credit: 64,228,409
RAC: 0
50 million credit badge13 year member badge
Message 10742 - Posted: 15 Feb 2009, 0:55:10 UTC

Are Windows opti compiled applications coming soon from the V0.18?
ID: 10742 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileKSMarksPsych
Avatar

Send message
Joined: 9 Sep 07
Posts: 22
Credit: 320,035
RAC: 0
100 thousand credit badge14 year member badge
Message 10834 - Posted: 15 Feb 2009, 9:41:01 UTC - in response to Message 10723.  

Now I only need a few 'recreational' hours (plus a few, plus a few, hey it's IT...) to install *nix. Any recommendations for the flavour? I'm not a total newbie, did some AIX sysadm in my early days (and not only smitty). But I would also like to use some regular programs like a spreadsheet, mail, photoeditor and stuff.


I like Fedora. But Ubuntu is OK too. I use Open Office for word processing and spread sheet type stuff. GIMP is good for high power photo stuff. For simple resizing and cropping I use KolourPaint (included with KDE), which is more powerful than MS Paint.

And back on topic, I'm going to install the new beta app (32 bit) in a few minutes.
Kathryn :o)
The BOINC FAQ Service
The Unofficial BOINC Wiki
The Trac System
More BOINC information than you can shake a stick of RAM at.
ID: 10834 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Cluster Physik

Send message
Joined: 26 Jul 08
Posts: 627
Credit: 94,940,203
RAC: 0
50 million credit badge13 year member badgeextraordinary contributions badge
Message 10840 - Posted: 15 Feb 2009, 10:39:23 UTC - in response to Message 10742.  
Last modified: 15 Feb 2009, 10:40:55 UTC

Are Windows opti compiled applications coming soon from the V0.18?

No, 0.18 is still too slow ;)
The current 0.16 apps (they are also not compiled from the 0.16 sources) will just be relabeled as 0.18.
ID: 10840 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Cluster Physik

Send message
Joined: 26 Jul 08
Posts: 627
Credit: 94,940,203
RAC: 0
50 million credit badge13 year member badgeextraordinary contributions badge
Message 10935 - Posted: 15 Feb 2009, 22:31:46 UTC - in response to Message 10644.  

meh :( i cut like... 10 multiplies and a divide out of the calculate probabilities :(


well, the improvement for the stock gcc-compiled app should be more than 30sec...

@speedimic:
If you want to get closer to the performance of the optimized Windows apps, prepare the compilation of the 0.18d sources, hopefully coming tomorrow.
They will save a bit more than 30s ;)
ID: 10935 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileConan
Avatar

Send message
Joined: 2 Jan 08
Posts: 108
Credit: 66,460,328
RAC: 1,216
50 million credit badge13 year member badge
Message 10952 - Posted: 15 Feb 2009, 23:23:04 UTC - in response to Message 10710.  

Installed the AMD pack and used the 32 bit source.

0.18c is running well and validating.

It did do a strange thing when I changed over to the new application.
The currently running jobs all had the percentage done reset to zero and start again, the time taken did not change but kept ticking over.

This meant that the first 3 or 4 jobs I processed with the new app had much longer process times than the ones that have been done with the new app alone.

Other than that all running smoothly and about 1 to 2 minutes quicker than the 0.16 version
(on my AMD Opteron this means it drops from 33 to 34 minutes to 31 to 32 minutes with Linux Fedora Core 3 and 6,
compare this to the Windows app that takes just 8 minutes with the same hardware, but still much better than an hour on the original stock app).

Thanks to both Speedimic and Cluster Physik for the much improved optimized applications and to Travis and Dave for providing the app in the first place.
ID: 10952 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profilespeedimic
Avatar

Send message
Joined: 22 Feb 08
Posts: 260
Credit: 57,387,048
RAC: 0
50 million credit badge13 year member badge
Message 10958 - Posted: 15 Feb 2009, 23:57:49 UTC - in response to Message 10935.  

meh :( i cut like... 10 multiplies and a divide out of the calculate probabilities :(


well, the improvement for the stock gcc-compiled app should be more than 30sec...

@speedimic:
If you want to get closer to the performance of the optimized Windows apps, prepare the compilation of the 0.18d sources, hopefully coming tomorrow.
They will save a bit more than 30s ;)


Looking forward to that.
My compiler is ready and waiting...
mic.


ID: 10958 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profilespeedimic
Avatar

Send message
Joined: 22 Feb 08
Posts: 260
Credit: 57,387,048
RAC: 0
50 million credit badge13 year member badge
Message 10959 - Posted: 16 Feb 2009, 0:00:17 UTC - in response to Message 10952.  

...
It did do a strange thing when I changed over to the new application.
The currently running jobs all had the percentage done reset to zero and start again, the time taken did not change but kept ticking over.

This meant that the first 3 or 4 jobs I processed with the new app had much longer process times than the ones that have been done with the new app alone.
...


must be the new checkpointing...
mic.


ID: 10959 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileTravis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Aug 07
Posts: 2046
Credit: 26,480
RAC: 0
10 thousand credit badge14 year member badge
Message 10975 - Posted: 16 Feb 2009, 3:55:24 UTC - in response to Message 10959.  

...
It did do a strange thing when I changed over to the new application.
The currently running jobs all had the percentage done reset to zero and start again, the time taken did not change but kept ticking over.

This meant that the first 3 or 4 jobs I processed with the new app had much longer process times than the ones that have been done with the new app alone.
...


must be the new checkpointing...


Yeah that was probably the new checkpointing.

I've updated the v0.18 code for everyone. It's pretty fast. It gave me around a 30% speedup on my machine with the current WU sizes, going from 25m12s down to 18m38s on stripe 86.
ID: 10975 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfileTravis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 30 Aug 07
Posts: 2046
Credit: 26,480
RAC: 0
10 thousand credit badge14 year member badge
Message 10990 - Posted: 16 Feb 2009, 6:59:51 UTC - in response to Message 10975.  
Last modified: 16 Feb 2009, 7:00:58 UTC

The major change from 0.18c to 0.18d is the following:

In calculating bg_prob (from calculate probabilities) we're now doing:
<pre>
                (*bg_prob) = 0;
                if (alpha == delta == 1) {
                        for (i = 0; i < ap->convolve; i++) {
                                xyz[i][2] = r_point[i] * sinb;
                                zp = r_point[i] * cosb;
                                xyz[i][0] = zp * cosl - lbr_r;
                                xyz[i][1] = zp * sinl;

                                rg = sqrt(xyz[i][0]*xyz[i][0] + xyz[i][1]*xyz[i][1] + (xyz[i][2]*xyz[i][2])/(q*q));
                                rs = rg + r0;

                                (*bg_prob) += qw_r3_N[i] / (rg * rs * rs * rs);
                        }
                } else {
                        for (i = 0; i < ap->convolve; i++) {
                                xyz[i][2] = r_point[i] * sinb;
                                zp = r_point[i] * cosb;
                                xyz[i][0] = zp * cosl - lbr_r;
                                xyz[i][1] = zp * sinl;

                                rg = sqrt(xyz[i][0]*xyz[i][0] + xyz[i][1]*xyz[i][1] + (xyz[i][2]*xyz[i][2])/(q*q));

                                (*bg_prob) += qw_r3_N[i] / (pow(rg, alpha) * pow(rg + r0, alpha_delta3));
                        }
                }
                (*bg_prob) *= reff_xr_rp3;
</pre>


Since the parameter files we've been using have a fixed alpha and delta, which also happens to be 1, we can cut out those two calls to pow. This speeds things up quite a bit :)
ID: 10990 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
jedirock
Avatar

Send message
Joined: 8 Nov 08
Posts: 178
Credit: 6,140,854
RAC: 0
5 million credit badge13 year member badge
Message 14161 - Posted: 6 Mar 2009, 21:04:38 UTC

Just tried profiling my custom build of the MW app earlier today. Seems that almost half of the time in the application, 47.8%, is from the exp function, which as I can tell, calculates e^x. The majority of those calls are from calculate_integral. The other half, 47.4%, is in the calculate_probabilities function proper. After that, with 0.4 and 0.2% are the sin and cos functions, respectively. Hope that helps optimizing the application somewhat. ;)
ID: 14161 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Emanuel

Send message
Joined: 18 Nov 07
Posts: 280
Credit: 2,442,757
RAC: 0
2 million credit badge14 year member badge
Message 14186 - Posted: 6 Mar 2009, 23:52:18 UTC
Last modified: 6 Mar 2009, 23:53:03 UTC

Here's where exp() is called:
(it is also called in probability.c, but calculate_integrals() doesn't use functions from that file)

function . . . . .. . . . . . frequency . param
set_probability_constants() . 1 . . . . . sigmoid_curve_params[1] * (gPrime - sigmoid_curve_params[2])
set_probability_constants() . n . . . . . -((g-gPrime) * (g-gPrime) / (2 * stdev * stdev))
calculate_probabilities() . . n^2 . . . . -xyz_norm / stream_sigma_sq2[i]
calculate_likelyhood() . .. . 1 . . . . . ap->background_weight
calculate_likelyhood() . .. . n . . . . . ap->stream_weights[i]
calculate_likelyhood() . .. . n . . . . . [same as above, presumably optimized out]

From those, their frequency in calculate_integrals():

function . . . . .. . . . . . frequency
set_probability_constants() . n
calculate_probabilities() . . n^3

So calculate_probabilities is called with O(n^5) frequency!

PS: sorry about the dots, the code tags are broken and remove extra spaces.
ID: 14186 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProfilePaul D. Buck

Send message
Joined: 12 Apr 08
Posts: 621
Credit: 161,934,067
RAC: 0
100 million credit badge13 year member badge
Message 14219 - Posted: 7 Mar 2009, 7:50:54 UTC

Any thoughts on the memory leak mentioned in the other thread?

I note with my ATI crunching host I get about 1.6 G memory use in 24 hours or so. Stopping BOINC and killing the left over processes and restarting are required ... otherwise the system continues to slow ...
ID: 14219 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profilebanditwolf
Avatar

Send message
Joined: 12 Nov 07
Posts: 2425
Credit: 524,164
RAC: 0
500 thousand credit badge14 year member badge
Message 15442 - Posted: 15 Mar 2009, 3:30:31 UTC

Would something like this help to find the error when a wu errors out? Or add any other usefull information. I'm sure you've thought of some of this Cluster. I think your addition of the CPU & wall time is great.

Rosetta's stderr out (most):

BOINC:: Initializing ... ok.
[2009- 3-14 9:12: 1:] :: BOINC :: boinc_init()
BOINC:: Setting up shared resources ... ok.
BOINC:: Setting up semaphores ... ok.
BOINC:: Updating status ... ok.
BOINC:: Registering timer callback... ok.
BOINC:: Worker initialized successfully.
Registering options..
Registered extra options.
Initializing core...
Initializing options.... ok
Loaded options.... ok
Processed options.... ok
Initializing random generators... ok
Initialization complete.
Setting WU description ...
Firstarg=true; pp=-d./
firstarg: <-d./>
End of unzipping.
Unpacking WU data ...
Firstarg=true; pp=-d./
firstarg: <-d./>
End of unzipping.
Setting database description ...
Setting up checkpointing ...
BOINC:: Worker startup.
Starting watchdog...
Watchdog active.
# cpu_run_time_pref: 36000
======================================================
DONE :: 1 starting structures 34685.1 cpu seconds
This process generated 25 decoys from 25 attempts
======================================================

Doesn't expecting the unexpected make the unexpected the expected?
If it makes sense, DON'T do it.
ID: 15442 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Cluster Physik

Send message
Joined: 26 Jul 08
Posts: 627
Credit: 94,940,203
RAC: 0
50 million credit badge13 year member badgeextraordinary contributions badge
Message 15460 - Posted: 15 Mar 2009, 10:12:28 UTC - in response to Message 14219.  

Any thoughts on the memory leak mentioned in the other thread?

I note with my ATI crunching host I get about 1.6 G memory use in 24 hours or so. Stopping BOINC and killing the left over processes and restarting are required ... otherwise the system continues to slow ...

That's strange. My apps (CPU and GPU) should not suffer from the leaks mentioned in the other thread. I use a different layout of the arrays than the stock app and have therefore also changed the memory allocation and deallocation.
ID: 15460 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · Next

Message boards : Application Code Discussion : source v0.18

©2021 Astroinformatics Group