search progress visualization
log in

Advanced search

Message boards : Number crunching : search progress visualization

Author Message
Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 11115 - Posted: 17 Feb 2009 | 6:53:01 UTC
Last modified: 9 Jun 2009 | 15:52:21 UTC

So I've been thinking about make things a little more interesting on your end. Dave and myself are going to be trying to get some small visualizations available (that should be updated quasi-real time) that should show how our different searches are progressing.

We'd also like to put some other (possibly interesting) information in these, like what user crunched the most accurate fit, and things of that nature, as well as having the graphs be a bit more dynamic.

So I'm starting this thread as a preliminary version, and hopefully you guys can give us some good suggestions. I'm working on a script that will generate these images automatically as something we can have running every couple minutes in the background to keep the plots up to date.

At any rate, here's what the current searches are doing:

Search Plots
____________

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 11117 - Posted: 17 Feb 2009 | 7:22:35 UTC - in response to Message 11115.
Last modified: 17 Feb 2009 | 8:35:44 UTC

For a little more information about what you're looking at:

The graphs are showing the particle swarm population being updated over time (ie, the best fitness in the population, the worst fitness in the population, and the average fitness in the population).

Typical particle swarm optimization updates the jth parameter of particle i as follows:

p[i][j] = w * v[i][j] + c1 * r1 * (g[j] - p[i][j]) + c2 * r2 * (l[i][j] - p[i][j])

w, c1 and c2 are constants, typically ~1, 2.0 and 2.0. However for the milkyway application, we've found that having w around 1 doesn't converge at all, so with these searches we're trying different w values. ps_x_2 is using w = 0.4, ps_x_3 is using w = 0.6 and ps_x_4 is using w = 0.8.

r1 and r2 are numbers randomly generated between 0 and 1.

v[i][j] refers to the previous velocity of the jth parameter of particle i (how far it moved from it's last position), g[j] refers to the jth parameter of the globally best found particle, l[i][j] refers to jth parameter of the ith particles best found position.

So as the search progresses, the particles are updated by moving according to their previous velocity, but pulled towards the best found position of the search ("global" knowledge) and the particles on previously best found position ("local" knowledge).

A lot of optimization people like this approach because as you can see, it's pretty simple but offers rather good results.

What we're doing is a little different because instead of generating a population of particles and waiting for the results, we're letting the particles continue to fly around the search space, and only updating the global and local best positions when the results arrive.

Interestingly enough, so far it seems that our asynchronous particle swarm optimization is giving much better results than the newton method searches we were doing, because the newton method searches would converge to a local minima rather quickly and not be able to escape it.
____________

Otter
Send message
Joined: 20 Jan 09
Posts: 9
Credit: 17,289,621
RAC: 0
Message 11120 - Posted: 17 Feb 2009 | 8:14:49 UTC

Interesting stuff.

Travis wrote:
p[i][j] = w * v[i][j] + c1 * r1 * (g[j] - p[i][j]) + c2 * r2 * (l[i][j] - p[i][j])


So tell me, your equation looks similar to the SOR iterative method (but not). Is that what W is? (The relaxation constant). I have studied (introductory) iterative methods, so please be gentle ><

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 11125 - Posted: 17 Feb 2009 | 8:33:47 UTC - in response to Message 11120.
Last modified: 17 Feb 2009 | 8:38:18 UTC

Interesting stuff.

Travis wrote:
p[i][j] = w * v[i][j] + c1 * r1 * (g[j] - p[i][j]) + c2 * r2 * (l[i][j] - p[i][j])


So tell me, your equation looks similar to the SOR iterative method (but not). Is that what W is? (The relaxation constant). I have studied (introductory) iterative methods, so please be gentle ><


That's interesting. I've actually never heard of the SOR method, so I just looked it up ;) But just from looking, they do seem somewhat similar.

What w is doing here is putting some kind of limit on the particles velocity. If w is too high, the particle's velocity will increase over time, while if it's too low the particle's velocity will slow down over time.

Since the search space that we're looking at here isn't particularly pretty, having a high w can be nice in that it will help a particle escape a local minima by increasing it's velocity until it can jump out. However, if it's too high the particles wont ever converge to a decent value in any of the minima found (making it more or less a random search). So in that way it's rather similar to the temperature used by simulated annealing.

You can pretty much see this happening in the graphs above. The searches with a low w (ps_s79_2, ps_s82_2 and ps_s86_2) are converging rather quickly, while the other searches are keeping a wider populuation, meaning they're searching a larger area and not converging as quickly.

Basically the tradeoff is with a low w you can converge to a solution really quickly, however you run a much higher risk of the solution being a local minima instead of the globally best point.
____________

Otter
Send message
Joined: 20 Jan 09
Posts: 9
Credit: 17,289,621
RAC: 0
Message 11128 - Posted: 17 Feb 2009 | 9:24:23 UTC

Fun stuff!

SOR is just an optimal Gauss-Seidel method, which is just an optimized Jacobi method. I'll be honest, I have almost no idea what you are talking about, but I think I get the big picture. But when you say convergence - what the particles converging to? Also in the big picture how does this relate to images of the galaxy?

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 11129 - Posted: 17 Feb 2009 | 9:29:50 UTC - in response to Message 11128.
Last modified: 17 Feb 2009 | 9:34:44 UTC

Fun stuff!

SOR is just an optimal Gauss-Seidel method, which is just an optimized Jacobi method. I'll be honest, I have almost no idea what you are talking about, but I think I get the big picture. But when you say convergence - what the particles converging to? Also in the big picture how does this relate to images of the galaxy?


The particles are trying to find the optimum fit of a cylinder to the sagittarius stream in the milkyway galaxy in the corresponding stripe of the sky. The parameters to our search basically define things like the size and shape of the cylinder, it's density and other information about how the background stars (the stars not in the stream) are distributed. The stripes are thin enough that we can get away with using a cylinder :P

Once we have a accurate fits of these cylinders across multiple stripes we can fit them together to get an image of the sagittarius stream.


*edit* So basically, each particle is a set of parameters, and we use these different particles to generate new sets of parameters to try and find one that gives the best fit to the data.
____________

Profile Daniel
Send message
Joined: 25 Nov 07
Posts: 25
Credit: 35,622,831
RAC: 93,625
Message 11171 - Posted: 17 Feb 2009 | 15:59:28 UTC

I think that is the most thorough explanation of a project I've ever read. Thank you Travis, I knew there was a reason this is my favorite project.

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 11397 - Posted: 18 Feb 2009 | 12:21:45 UTC - in response to Message 11171.

I saw some interesting things in the last set of searches, so I started up a new batch. Each of these have a w = 0.8 so they're rather exploratory.

In one of the s79 searches we found a point point that was a bit higher than we've seen before so we'd like to see if we could find it again (to verify our stuff isn't going crazy).

Also, I noticed there was a bit of a problem with the optimization over a few of the parameters (which are in radians so they don't have bounds), so I made a few code changes to helpfully make that situation a bit better -- capping out the velocity of those parameters at PI or -PI, and having the parameters wrap around.
____________

Otter
Send message
Joined: 20 Jan 09
Posts: 9
Credit: 17,289,621
RAC: 0
Message 11468 - Posted: 18 Feb 2009 | 18:10:22 UTC

Bump, I would highly recommend you keep the little blurb about this thread on the main page. It is definitely more interesting than credit/code changes!

Profile DoctorNow
Avatar
Send message
Joined: 28 Aug 07
Posts: 146
Credit: 5,183,509
RAC: 0
Message 11479 - Posted: 18 Feb 2009 | 19:03:42 UTC
Last modified: 18 Feb 2009 | 19:12:07 UTC

Hehe, seems I misunderstood the news about that a bit.
I thought visualization in real-time would be a screensaver. ;-D
Now I see the diagrams in the first post are regularly updated, that's pretty cool also. :-)
I know there was another (rather old) thread asking the same, but how about a screensaver anyway, is it possible?
____________
Member of BOINC@Heidelberg and ATA!

My BOINCstats

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 11494 - Posted: 18 Feb 2009 | 19:42:46 UTC - in response to Message 11479.

Hehe, seems I misunderstood the news about that a bit.
I thought visualization in real-time would be a screensaver. ;-D
Now I see the diagrams in the first post are regularly updated, that's pretty cool also. :-)
I know there was another (rather old) thread asking the same, but how about a screensaver anyway, is it possible?


Well, I don't know about a screen saver because what we're doing on the individual processors isn't particularly interesting... it's much more interesting when you put it all together as a whole.

What we have in this post is pretty much preliminary work, so we'd really like to hear your ideas and suggestions as to what you'd like to know is going on server-side, and what we're doing as a whole.

Right now we would like to get these graphs (and some other ones) automatically generated in their own webpage, with information like what users have given us the best fitness and things like that.

Pretty much we could have whatever information you guys think would be interesting, and we'll also have things we're interested about and looking into on their for our own benefit.

Basically, we'd like a webpage we could check to easily know the status of our different searches, and I thought it would be more interesting with some user input, so we could make that information more interesting to you guys, and also add things you'd be interested in as crunchers.
____________

Otter
Send message
Joined: 20 Jan 09
Posts: 9
Credit: 17,289,621
RAC: 0
Message 11557 - Posted: 19 Feb 2009 | 2:19:50 UTC
Last modified: 19 Feb 2009 | 2:21:37 UTC

I'd like to know the definition of fitness.

Also when you say the particles are 'swarming' - how do stars swarm? Or better - what are particles? My naive thoughts are - I look at the sky from time to time, and (from my perspective), those stars move reallllly slowly. So if you are looking at telescopic imagery, how are you getting motion?

Edit- I am being naive on purpose, as people that crunch aren't necessarily astrophysicists, so simple things are undefined to them, the galaxy just boils down to stars and space, and they may have little knowledge of anything more!

Profile Paul D. Buck
Send message
Joined: 12 Apr 08
Posts: 621
Credit: 161,934,067
RAC: 0
Message 11580 - Posted: 19 Feb 2009 | 6:40:47 UTC - in response to Message 11494.

Well, I don't know about a screen saver because what we're doing on the individual processors isn't particularly interesting... it's much more interesting when you put it all together as a whole.


The screen saver does not have to be "perfectly" real if that is not possible. LHC for example has this "swarm" display (I think the UBW has an image of it) which is nonsesnse with the exception of the turn counter which is accurate.

It is just a thought to consider, to make the application more "real" to those that like the graphics ...

Priority wise I would put it below the change to the client to detect ATI cards and the OpenCL application (conserve resources, make the one additional application instead of four) ...

Profile Misfit
Avatar
Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,115
RAC: 0
Message 11581 - Posted: 19 Feb 2009 | 6:42:54 UTC - in response to Message 11494.

Screensavers won't be a part of a fully optimized app anyway. So you'll need to determine what percentage use the stock app and prioritize accordingly.
____________

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 11587 - Posted: 19 Feb 2009 | 9:17:51 UTC - in response to Message 11557.

I'd like to know the definition of fitness.


The astronomy application does a maximum likelihood fit of the stars in a stripe to a model of star stream(s) and the background stars.

Also when you say the particles are 'swarming' - how do stars swarm? Or better - what are particles? My naive thoughts are - I look at the sky from time to time, and (from my perspective), those stars move reallllly slowly. So if you are looking at telescopic imagery, how are you getting motion?


It's basically just terminology that the particular search uses. In essence each particle is a set of parameters that gets updated over time. The particles continue to move in their previous direction in the search space, but are also pulled to their own previous best found position, and the best found position by the entire swarm.

Basically, we have some function:

fitness = f(parameters)

and we want to find parameters that give us the best fitness. What each of our applications is doing is calculating that function f with a different set of parameters.

The higher the fitness is, the higher the likelihood of those set of parameters being the most accurate fit of the model to the data.

Edit- I am being naive on purpose, as people that crunch aren't necessarily astrophysicists, so simple things are undefined to them, the galaxy just boils down to stars and space, and they may have little knowledge of anything more!


That's good! Part of the point of this thread is to try and accurately describe what we're doing to all our crunchers, so if you guys have any questions please feel free to fire away :)
____________

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 11588 - Posted: 19 Feb 2009 | 9:20:27 UTC - in response to Message 11580.

Well, I don't know about a screen saver because what we're doing on the individual processors isn't particularly interesting... it's much more interesting when you put it all together as a whole.


The screen saver does not have to be "perfectly" real if that is not possible. LHC for example has this "swarm" display (I think the UBW has an image of it) which is nonsesnse with the exception of the turn counter which is accurate.

It is just a thought to consider, to make the application more "real" to those that like the graphics ...

Priority wise I would put it below the change to the client to detect ATI cards and the OpenCL application (conserve resources, make the one additional application instead of four) ...


Whenever we're looking for a new undergrad to work with us, usually one of the first things we throw out there is "you could make a screen saver for our application." And usually they want to do something else :P

We have kind of a tight ship right here, so the real issue is finding someone with the knowledge and desire to make one. Personally, I have no experience doing any kind of graphical design, so I think my skills are best used elsewhere... reducing credit and things of that nature :D
____________

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 11590 - Posted: 19 Feb 2009 | 9:22:16 UTC - in response to Message 11581.

Screensavers won't be a part of a fully optimized app anyway. So you'll need to determine what percentage use the stock app and prioritize accordingly.


I want so say maybe 90% or more of the WUs returned aren't from a stock app :P

... it might actually be interesting to count what WUs are generated from what applications and make that information available for you guys. Would let whoever compiled whichever application strut their stuff a bit. :)
____________

Profile Glenn Rogers
Avatar
Send message
Joined: 4 Jul 08
Posts: 165
Credit: 363,844
RAC: 0
Message 11592 - Posted: 19 Feb 2009 | 9:27:08 UTC - in response to Message 11590.

Isnt that interesting Hmmm!! So so the results from these are good one would presume??
____________

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 11595 - Posted: 19 Feb 2009 | 9:40:52 UTC - in response to Message 11592.

Isnt that interesting Hmmm!! So so the results from these are good one would presume??


So far so good, every now and then we're having a little trouble with the outlier detection (which is why you see the best fitness go up, then back down), but the results we're getting using particle swarm for stripes 20, 21, 79, 82 and 86 are comparable (or better!) than what we've gotten doing more traditional searches (conjugate gradient descent) on the bluegene at RPI.

We don't have any information about the values that stripes 22 and 23 should come out to... these are entirely new and we don't have any results for them. So you guys are really doing new astronomy with those two :)
____________

Profile Glenn Rogers
Avatar
Send message
Joined: 4 Jul 08
Posts: 165
Credit: 363,844
RAC: 0
Message 11596 - Posted: 19 Feb 2009 | 9:44:15 UTC - in response to Message 11595.
Last modified: 19 Feb 2009 | 9:47:35 UTC

Im about 50% through s23 now have to see how it reports..

EDIT: heres one i found...
Task ID 5603831
Name ps_s23_9_179749_1235009280_0
Workunit 5428244
Created 19 Feb 2009 2:08:04 UTC
Sent 19 Feb 2009 2:09:27 UTC
Received 19 Feb 2009 7:15:03 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x0)
Computer ID 21898
Report deadline 22 Feb 2009 2:09:27 UTC
CPU time 1491.703
stderr out <core_client_version>6.4.5</core_client_version>
<![CDATA[
<stderr_txt>
Running Milkyway@home version 0.19 by Gipsel
CPU: Genuine Intel(R) CPU T2300 @ 1.66GHz (2 cores/threads) 1.66678 GHz (889ms)

WU completed. It took 1491.7 seconds CPU time and 1510.59 seconds wall clock time @ 1.66678 GHz.

</stderr_txt>
]]>

Validate state Valid
Claimed credit 4.06538192328865
Granted credit 12.732925
application version 0.19
____________

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 11605 - Posted: 19 Feb 2009 | 10:47:50 UTC - in response to Message 11596.

As of this morning (feb 19) we started using Mersenne twister instead of drand48() to generate the random numbers for our particle swarm optimization. I'm hoping that this will improve the results a little bit, by adding some more variety to the random numbers we're generating.
____________

Emanuel
Send message
Joined: 18 Nov 07
Posts: 280
Credit: 2,442,674
RAC: 756
Message 11613 - Posted: 19 Feb 2009 | 11:51:48 UTC

Good call.

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 11633 - Posted: 19 Feb 2009 | 14:50:59 UTC - in response to Message 11613.

I also added median values to the output and graphs. It'll probably take them awhile to catch up with the median values however.
____________

EigenState
Send message
Joined: 19 Apr 08
Posts: 13
Credit: 84,666
RAC: 0
Message 11671 - Posted: 19 Feb 2009 | 20:31:38 UTC - in response to Message 11494.

What we have in this post is pretty much preliminary work, so we'd really like to hear your ideas and suggestions as to what you'd like to know is going on server-side, and what we're doing as a whole.

Right now we would like to get these graphs (and some other ones) automatically generated in their own webpage, with information like what users have given us the best fitness and things like that.

Pretty much we could have whatever information you guys think would be interesting, and we'll also have things we're interested about and looking into on their for our own benefit.

Basically, we'd like a webpage we could check to easily know the status of our different searches, and I thought it would be more interesting with some user input, so we could make that information more interesting to you guys, and also add things you'd be interested in as crunchers.


I would suggest, no urge that the project utilize this new webpage as a means to provide a vastly greater educational opportunity for the volunteer participants. Thus, I believe it should provide information on both the fundamental physics and the computational aspects of the project. By computational aspects, I most certainly do not mean any discussion of credits or who has the bigger, faster hardware--such discussions are all too prevalent as it is on the forum. Some suggested reading materials on both subject matters would be excellent additions.

Allow your volunteers to learn if they so choose. The lack of such learning resources is, in my opinion, a great weakness in the BOINC concept.

Best regards,
EigenState

Emanuel
Send message
Joined: 18 Nov 07
Posts: 280
Credit: 2,442,674
RAC: 756
Message 12245 - Posted: 22 Feb 2009 | 0:00:17 UTC

The current situation in ps_s21_9 looks interesting. Is this a case of the particles moving from one local maximum to another, or are we seeing some outliers?

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 12345 - Posted: 22 Feb 2009 | 12:03:47 UTC - in response to Message 12245.

The current situation in ps_s21_9 looks interesting. Is this a case of the particles moving from one local maximum to another, or are we seeing some outliers?


Not quite sure :) That's why I'm letting it keep running.
____________

Emanuel
Send message
Joined: 18 Nov 07
Posts: 280
Credit: 2,442,674
RAC: 756
Message 12368 - Posted: 22 Feb 2009 | 13:58:10 UTC
Last modified: 22 Feb 2009 | 14:04:29 UTC

Hehe, that's cool. What about ps_s23_9? You can still just about see when the 'best' values shot up. I'm still not sure what you're aiming for - do you want the worst values to converge to the best ones, while finding the highest average overall? (if so, why aren't you showing the worst values in your graph?)

Emanuel
Send message
Joined: 18 Nov 07
Posts: 280
Credit: 2,442,674
RAC: 756
Message 12468 - Posted: 23 Feb 2009 | 0:36:09 UTC
Last modified: 23 Feb 2009 | 0:36:32 UTC

By the way, if the goal is for the particles to converge (i.e. all of them moving toward the global maximum?), could you add the mean absolute deviation or the interquartile range? (I guess if you add them at the bottom of the graphs you'd need a second range for the vertical axis, which could go on the right)

Profile banditwolf
Avatar
Send message
Joined: 12 Nov 07
Posts: 2425
Credit: 295,133
RAC: 0
Message 12478 - Posted: 23 Feb 2009 | 2:32:45 UTC

I would suggest that the bottom line of numbers be made X10000, to eliminate the mess of 0's.
____________
Doesn't expecting the unexpected make the unexpected the expected?
If it makes sense, DON'T do it.

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 12513 - Posted: 23 Feb 2009 | 12:43:22 UTC - in response to Message 12368.

Hehe, that's cool. What about ps_s23_9? You can still just about see when the 'best' values shot up. I'm still not sure what you're aiming for - do you want the worst values to converge to the best ones, while finding the highest average overall? (if so, why aren't you showing the worst values in your graph?)


I just was playing around and took out the worst values because you could see a bit better how the best was progressing that way. It would be nice for the worst values to converge to the best, because that's an indication that the search is over, but it's not really necessary. We're just looking to find the highest fitness

By the way, if the goal is for the particles to converge (i.e. all of them moving toward the global maximum?), could you add the mean absolute deviation or the interquartile range? (I guess if you add them at the bottom of the graphs you'd need a second range for the vertical axis, which could go on the right)


Yeah I have our searches generate standard deviation and a few other values to the logs but haven't made a graph for that yet. It should be pretty easy to add one :)
____________

Profile Paul D. Buck
Send message
Joined: 12 Apr 08
Posts: 621
Credit: 161,934,067
RAC: 0
Message 12776 - Posted: 24 Feb 2009 | 21:57:03 UTC

With regard to a screensaver, I have to admit that at times it is nice to be able to look at the data, but, with the runtimes so short now it does not seem to make much sense. My "slowest" system when I was running it last week I think was down to 10 minutes per task ... that is not much time to do much of anything of interest...

and the high end at 12 seconds, well, blink and you missed it all ...

Emanuel
Send message
Joined: 18 Nov 07
Posts: 280
Credit: 2,442,674
RAC: 756
Message 13051 - Posted: 27 Feb 2009 | 14:26:47 UTC

Looks like the graphs haven't been updated in a few days - I thought they were scripted to run automatically.. Perhaps you could add a timestamp to the bottom of the post so we know when it was last updated.

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 13055 - Posted: 27 Feb 2009 | 16:20:30 UTC - in response to Message 13051.

Looks like the graphs haven't been updated in a few days - I thought they were scripted to run automatically.. Perhaps you could add a timestamp to the bottom of the post so we know when it was last updated.


Dave was supposed to have gotten them run automatically :P I guess he didn't... I'm going to bug him some more.
____________

Profile banditwolf
Avatar
Send message
Joined: 12 Nov 07
Posts: 2425
Credit: 295,133
RAC: 0
Message 13091 - Posted: 27 Feb 2009 | 18:46:51 UTC - in response to Message 13055.

Dave was supposed to have gotten them run automatically :P I guess he didn't... I'm going to bug him some more.


Keep that cattle prod handy!
____________
Doesn't expecting the unexpected make the unexpected the expected?
If it makes sense, DON'T do it.

Profile Phil
Avatar
Send message
Joined: 13 Feb 08
Posts: 1124
Credit: 46,740
RAC: 0
Message 13097 - Posted: 27 Feb 2009 | 19:21:00 UTC - in response to Message 13091.
Last modified: 27 Feb 2009 | 19:21:16 UTC

Dave was supposed to have gotten them run automatically :P I guess he didn't... I'm going to bug him some more.


Keep that cattle prod handy!


I'll click the Red X on him.

Profile Misfit
Avatar
Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,115
RAC: 0
Message 13125 - Posted: 27 Feb 2009 | 22:46:58 UTC - in response to Message 13097.

Dave was supposed to have gotten them run automatically :P I guess he didn't... I'm going to bug him some more.

Keep that cattle prod handy!

I'll click the Red X on him.

Dave will blame it on a freshman.
____________

Profile GalaxyIce
Avatar
Send message
Joined: 6 Apr 08
Posts: 2018
Credit: 100,142,856
RAC: 0
Message 13126 - Posted: 27 Feb 2009 | 22:52:59 UTC - in response to Message 13125.

Dave was supposed to have gotten them run automatically :P I guess he didn't... I'm going to bug him some more.

Keep that cattle prod handy!

I'll click the Red X on him.

Dave will blame it on a freshman.

I think you have something there. A freshman prodding Dave with a cattle prod. Now that's an idea for a screensaver :P

____________

Profile Misfit
Avatar
Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,115
RAC: 0
Message 13128 - Posted: 27 Feb 2009 | 22:55:39 UTC - in response to Message 13126.

Dave was supposed to have gotten them run automatically :P I guess he didn't... I'm going to bug him some more.

Keep that cattle prod handy!

I'll click the Red X on him.

Dave will blame it on a freshman.

I think you have something there. A freshman prodding Dave with a cattle prod. Now that's an idea for a screensaver :P

I might have to enroll.
____________

Profile banditwolf
Avatar
Send message
Joined: 12 Nov 07
Posts: 2425
Credit: 295,133
RAC: 0
Message 13136 - Posted: 27 Feb 2009 | 23:00:48 UTC - in response to Message 13128.

Dave was supposed to have gotten them run automatically :P I guess he didn't... I'm going to bug him some more.

Keep that cattle prod handy!

I'll click the Red X on him.

Dave will blame it on a freshman.

I think you have something there. A freshman prodding Dave with a cattle prod. Now that's an idea for a screensaver :P

I might have to enroll.


The next hit internet game. Prod Dave & the sequel Prod Dave Again!!
____________
Doesn't expecting the unexpected make the unexpected the expected?
If it makes sense, DON'T do it.

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 13160 - Posted: 27 Feb 2009 | 23:21:16 UTC - in response to Message 13136.

Updated these to the newer searches which are using some limited redundancy to check new local/global best particles ;) Hopefully these should improve better than the last set.
____________

Profile Misfit
Avatar
Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,115
RAC: 0
Message 13190 - Posted: 28 Feb 2009 | 1:45:52 UTC - in response to Message 13160.

Updated these to the newer searches which are using some limited redundancy to check new local/global best particles ;) Hopefully these should improve better than the last set.

How was the prodding?
____________

Profile banditwolf
Avatar
Send message
Joined: 12 Nov 07
Posts: 2425
Credit: 295,133
RAC: 0
Message 15018 - Posted: 12 Mar 2009 | 14:37:03 UTC

The median seems to generally be where the average is. I know they aren't the same but I think it seems kind of redundant to show both on the graphs, unless the graphs could be stretched to show the difference better. Is it possible to change the left column to show the difference between each fittness? Otherwise they all show the same number, which may be confusing.
____________
Doesn't expecting the unexpected make the unexpected the expected?
If it makes sense, DON'T do it.

Emanuel
Send message
Joined: 18 Nov 07
Posts: 280
Credit: 2,442,674
RAC: 756
Message 15029 - Posted: 12 Mar 2009 | 17:21:52 UTC
Last modified: 12 Mar 2009 | 17:27:06 UTC

The median is useful because it isn't effected as much by outliers, so it's probably not as useful now that the amount of outliers has been reduced.

Edit: also, what is purple?

Profile banditwolf
Avatar
Send message
Joined: 12 Nov 07
Posts: 2425
Credit: 295,133
RAC: 0
Message 15033 - Posted: 12 Mar 2009 | 18:13:18 UTC - in response to Message 15029.

The median is useful because it isn't effected as much by outliers, so it's probably not as useful now that the amount of outliers has been reduced.

Edit: also, what is purple?


Purple is worst. I know the different marks are useful. It just isn't as easy to view with them both being about the same spots.
____________
Doesn't expecting the unexpected make the unexpected the expected?
If it makes sense, DON'T do it.

Profile Misfit
Avatar
Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,115
RAC: 0
Message 15052 - Posted: 12 Mar 2009 | 20:50:00 UTC - in response to Message 15033.

The median is useful because it isn't effected as much by outliers, so it's probably not as useful now that the amount of outliers has been reduced.

Edit: also, what is purple?


Purple is worst. I know the different marks are useful. It just isn't as easy to view with them both being about the same spots.

What rhymes with purple?

And what rhymes with orange?

And what TV series had an episode that refered to "purple and orange squadrons"?
____________

Profile Paul D. Buck
Send message
Joined: 12 Apr 08
Posts: 621
Credit: 161,934,067
RAC: 0
Message 15142 - Posted: 13 Mar 2009 | 7:42:20 UTC - in response to Message 15052.

The median is useful because it isn't effected as much by outliers, so it's probably not as useful now that the amount of outliers has been reduced.

Edit: also, what is purple?


Purple is worst. I know the different marks are useful. It just isn't as easy to view with them both being about the same spots.

What rhymes with purple?

And what rhymes with orange?

And what TV series had an episode that refered to "purple and orange squadrons"?

Roger Miller seemed to think it was Maple Surple

Emanuel
Send message
Joined: 18 Nov 07
Posts: 280
Credit: 2,442,674
RAC: 756
Message 15359 - Posted: 14 Mar 2009 | 14:41:04 UTC

It looks like all the current searches have stopped improving - how do you determine when to stop a search? Do you have objective criteria for when a search is over?

Divide Overflow
Avatar
Send message
Joined: 16 Feb 09
Posts: 109
Credit: 11,089,510
RAC: 0
Message 15924 - Posted: 18 Mar 2009 | 5:32:14 UTC

Every time I turn in a work unit or check the latest visualization reports, I feel the urge to rush to my Newtonian telescope to check if the stars have changed much. ;)

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 16849 - Posted: 25 Mar 2009 | 20:57:28 UTC - in response to Message 15359.

It looks like all the current searches have stopped improving - how do you determine when to stop a search? Do you have objective criteria for when a search is over?


Still working on Dave to automate the script that generates the graphs. The one's we're doing right now are pretty close to done. For some streams (79, 82, 86) they're the highest fitness we've ever seen. The other stripes are in the same boat, however the parameters are a little weird so we're looking into the cause of that.
____________

Emanuel
Send message
Joined: 18 Nov 07
Posts: 280
Credit: 2,442,674
RAC: 756
Message 16863 - Posted: 25 Mar 2009 | 22:14:31 UTC - in response to Message 16849.

The one's we're doing right now are pretty close to done. For some streams (79, 82, 86) they're the highest fitness we've ever seen. The other stripes are in the same boat, however the parameters are a little weird so we're looking into the cause of that.

That's pretty cool - how much higher (closer?) are they? Sounds like we're really breaking some new ground here :)

Odd-Rod
Send message
Joined: 7 Sep 07
Posts: 355
Credit: 989,636
RAC: 836
Message 17155 - Posted: 30 Mar 2009 | 7:46:39 UTC - in response to Message 15052.

Sorry, I know this is off-topic, but I can't resist!

What rhymes with purple?

curple:
the rump, derriere, buttocks, fundament

hirple:
to walk with a limp

And what rhymes with orange?

sporange:
A single-celled or many-celled structure in which spores are produced, as in fungi, algae, mosses, and ferns. Also called spore case.

And what TV series had an episode that refered to "purple and orange squadrons"?


I was going to guess Captain Scarlet with it's 'multi-coloured' characters, but since you mention 'an episode' I pass on this one.

Debs
Send message
Joined: 15 Jan 09
Posts: 169
Credit: 6,734,481
RAC: 0
Message 17185 - Posted: 31 Mar 2009 | 0:46:04 UTC - in response to Message 17155.

And what TV series had an episode that refered to "purple and orange squadrons"?


I was going to guess Captain Scarlet with it's 'multi-coloured' characters, but since you mention 'an episode' I pass on this one.



While google is your friend, Battlestar Galactica is the saviour of mankind :)
____________

Profile The Gas Giant
Avatar
Send message
Joined: 24 Dec 07
Posts: 1947
Credit: 240,865,573
RAC: 0
Message 17459 - Posted: 3 Apr 2009 | 18:30:09 UTC - in response to Message 17185.
Last modified: 3 Apr 2009 | 18:43:56 UTC

And what TV series had an episode that refered to "purple and orange squadrons"?


I was going to guess Captain Scarlet with it's 'multi-coloured' characters, but since you mention 'an episode' I pass on this one.



While google is your friend, Battlestar Galactica is the saviour of mankind :)

And thank Gods for bittorent...the saviour of bad network programming decisions!

Profile Misfit
Avatar
Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,115
RAC: 0
Message 17502 - Posted: 4 Apr 2009 | 0:18:12 UTC - in response to Message 17459.

And what TV series had an episode that refered to "purple and orange squadrons"?

I was going to guess Captain Scarlet with it's 'multi-coloured' characters, but since you mention 'an episode' I pass on this one.

While google is your friend, Battlestar Galactica is the saviour of mankind :)

And thank Gods for bittorent...the saviour of bad network programming decisions!

Oh those poor trackers for getting hammered like that.
____________

Profile The Gas Giant
Avatar
Send message
Joined: 24 Dec 07
Posts: 1947
Credit: 240,865,573
RAC: 0
Message 17513 - Posted: 4 Apr 2009 | 4:35:16 UTC - in response to Message 17502.

And what TV series had an episode that refered to "purple and orange squadrons"?

I was going to guess Captain Scarlet with it's 'multi-coloured' characters, but since you mention 'an episode' I pass on this one.

While google is your friend, Battlestar Galactica is the saviour of mankind :)

And thank Gods for bittorent...the saviour of bad network programming decisions!

Oh those poor trackers for getting hammered like that.

Better than hammering the Milkyway schedular!

Profile Misfit
Avatar
Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,115
RAC: 0
Message 17572 - Posted: 4 Apr 2009 | 19:37:30 UTC - in response to Message 17513.

And what TV series had an episode that refered to "purple and orange squadrons"?

I was going to guess Captain Scarlet with it's 'multi-coloured' characters, but since you mention 'an episode' I pass on this one.

While google is your friend, Battlestar Galactica is the saviour of mankind :)

And thank Gods for bittorent...the saviour of bad network programming decisions!

Oh those poor trackers for getting hammered like that.

Better than hammering the Milkyway schedular!

Tell that to your demonoid script.
____________

Profile Phil
Avatar
Send message
Joined: 13 Feb 08
Posts: 1124
Credit: 46,740
RAC: 0
Message 17660 - Posted: 5 Apr 2009 | 20:17:49 UTC - in response to Message 17502.

And what TV series had an episode that refered to "purple and orange squadrons"?

I was going to guess Captain Scarlet with it's 'multi-coloured' characters, but since you mention 'an episode' I pass on this one.

While google is your friend, Battlestar Galactica is the saviour of mankind :)

And thank Gods for bittorent...the saviour of bad network programming decisions!

Oh those poor trackers for getting hammered like that.

I know, I have a script that does "update tracker" every 5 seconds...

Profile Misfit
Avatar
Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,115
RAC: 0
Message 17700 - Posted: 6 Apr 2009 | 2:23:15 UTC - in response to Message 17660.

And what TV series had an episode that refered to "purple and orange squadrons"?

I was going to guess Captain Scarlet with it's 'multi-coloured' characters, but since you mention 'an episode' I pass on this one.

While google is your friend, Battlestar Galactica is the saviour of mankind :)

And thank Gods for bittorent...the saviour of bad network programming decisions!

Oh those poor trackers for getting hammered like that.

I know, I have a script that does "update tracker" every 5 seconds...

Maybe that's why I have all those repetitive PG2 log entries.
____________

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 24685 - Posted: 9 Jun 2009 | 16:04:15 UTC - in response to Message 11115.

Just an update here, I found a little php script to display the plots of how our searches are performing, so to take a look and see how things are running you can go to:

http://milkyway.cs.rpi.edu/milkyway/download/plots/plots.php

Does anyone know how I could get that php inside a forum post so the one at top would always be up to date?
____________

Profile Misfit
Avatar
Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,115
RAC: 0
Message 24687 - Posted: 9 Jun 2009 | 16:42:44 UTC - in response to Message 24685.

Just an update here, I found a little php script to display the plots of how our searches are performing, so to take a look and see how things are running you can go to:

http://milkyway.cs.rpi.edu/milkyway/download/plots/plots.php

Does anyone know how I could get that php inside a forum post so the one at top would always be up to date?

BBCODE won't allow you to do that if I'm understanding you correctly. You'd have to show each individual image.



To keep it updated you'd have to manually replace the image, with the exact same name, on your end. Even then some browsers may not automatically refresh the image.
____________

Profile Misfit
Avatar
Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,115
RAC: 0
Message 24688 - Posted: 9 Jun 2009 | 16:47:47 UTC - in response to Message 24687.

See my APOTD thread in Science for a nice example. Create a thread, in the first post make a link to the php url. Sticky the thread. Then all you need to do is update the php page. Since you'll most likely be adding new (and removing old?) images with different filenames they should be properly displayed in the browser.
____________

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 24701 - Posted: 9 Jun 2009 | 18:39:04 UTC - in response to Message 24687.

Just an update here, I found a little php script to display the plots of how our searches are performing, so to take a look and see how things are running you can go to:

http://milkyway.cs.rpi.edu/milkyway/download/plots/plots.php

Does anyone know how I could get that php inside a forum post so the one at top would always be up to date?

BBCODE won't allow you to do that if I'm understanding you correctly. You'd have to show each individual image.



To keep it updated you'd have to manually replace the image, with the exact same name, on your end. Even then some browsers may not automatically refresh the image.


Yeah I was trying to find some way to automatically put the images in a post, as opposed to just linking the php page (as in the first post in this thread).
____________

Profile Misfit
Avatar
Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,115
RAC: 0
Message 24706 - Posted: 9 Jun 2009 | 19:19:24 UTC - in response to Message 24701.

Just an update here, I found a little php script to display the plots of how our searches are performing, so to take a look and see how things are running you can go to:

http://milkyway.cs.rpi.edu/milkyway/download/plots/plots.php

Does anyone know how I could get that php inside a forum post so the one at top would always be up to date?

BBCODE won't allow you to do that if I'm understanding you correctly. You'd have to show each individual image.



To keep it updated you'd have to manually replace the image, with the exact same name, on your end. Even then some browsers may not automatically refresh the image.


Yeah I was trying to find some way to automatically put the images in a post, as opposed to just linking the php page (as in the first post in this thread).

Well you do have an unlimited post edit time. So every time there is a new image you could edit your post with the new filename.
____________

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 24713 - Posted: 9 Jun 2009 | 19:38:19 UTC - in response to Message 24706.

Well you do have an unlimited post edit time. So every time there is a new image you could edit your post with the new filename.


The lazy computer scientist in my hates that solution :D
____________

Profile Misfit
Avatar
Send message
Joined: 27 Aug 07
Posts: 915
Credit: 1,503,115
RAC: 0
Message 24717 - Posted: 9 Jun 2009 | 19:55:56 UTC - in response to Message 24713.

Well you do have an unlimited post edit time. So every time there is a new image you could edit your post with the new filename.

The lazy computer scientist in my hates that solution :D

Plan B. Find an unused monitor. Display the image on that monitor. Point a webcam at the monitor. Post a link to the webcam. If an image gets burned in get another monitor and repeat the steps.
____________

Emanuel
Send message
Joined: 18 Nov 07
Posts: 280
Credit: 2,442,674
RAC: 756
Message 24741 - Posted: 9 Jun 2009 | 22:24:26 UTC

Is it just me, or are those fitness values a lot better than before? :O New coordinate system helping?

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 24833 - Posted: 10 Jun 2009 | 18:47:39 UTC - in response to Message 24741.

I'm unstickying this because it's now on the main webpage and being automatically updated :)
____________

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 24834 - Posted: 10 Jun 2009 | 18:48:24 UTC - in response to Message 24741.

Is it just me, or are those fitness values a lot better than before? :O New coordinate system helping?


Each stripe will have a different best fitness. I think the reason the new ones are a bit lower is because of the coordinate system, but who knows? :)
____________

Profile Martin Chartrand
Avatar
Send message
Joined: 25 Mar 09
Posts: 62
Credit: 37,838,030
RAC: 0
Message 24835 - Posted: 10 Jun 2009 | 19:01:56 UTC

This probably the wrong place to ask this but it seems appropriate since it talks of the visualization.
Is there a site or URL that shows the progress of the 3D dimensionnal map created by MW?
Since we are all crunching for it, there should be some kind of progress as to the 3D model of the milkyway?
Excuse my ignorance in this matter but I am far from being a scientist.
Would this 3D model look like the 3D model of Google Space?
Or this is only a 3D model of mathematical proportions and none of actual images?
Does this 3D model encompass only points to show the stars?
Is it avaivable to the public?
Does it exist?

Martin Chartrand

Emanuel
Send message
Joined: 18 Nov 07
Posts: 280
Credit: 2,442,674
RAC: 756
Message 24840 - Posted: 10 Jun 2009 | 19:55:03 UTC - in response to Message 24834.

Each stripe will have a different best fitness. I think the reason the new ones are a bit lower is because of the coordinate system, but who knows? :)

Are you intending to rerun the search of the previous stripe for the sake of comparison? (and uh, potentially improved results)

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 24850 - Posted: 10 Jun 2009 | 20:34:05 UTC - in response to Message 24835.

This probably the wrong place to ask this but it seems appropriate since it talks of the visualization.
Is there a site or URL that shows the progress of the 3D dimensionnal map created by MW?
Since we are all crunching for it, there should be some kind of progress as to the 3D model of the milkyway?
Excuse my ignorance in this matter but I am far from being a scientist.
Would this 3D model look like the 3D model of Google Space?
Or this is only a 3D model of mathematical proportions and none of actual images?
Does this 3D model encompass only points to show the stars?
Is it avaivable to the public?
Does it exist?

Martin Chartrand


I think the astronomers are working on getting some interesting figures for you guys showing what you've already crunched.
____________

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 24852 - Posted: 10 Jun 2009 | 20:34:57 UTC - in response to Message 24840.

Each stripe will have a different best fitness. I think the reason the new ones are a bit lower is because of the coordinate system, but who knows? :)

Are you intending to rerun the search of the previous stripe for the sake of comparison? (and uh, potentially improved results)


I dont think the new coordinate system applies to the older streams.
____________

Profile banditwolf
Avatar
Send message
Joined: 12 Nov 07
Posts: 2425
Credit: 295,133
RAC: 0
Message 24900 - Posted: 11 Jun 2009 | 0:22:01 UTC

I think the graphs would fit better under the Boinc logo at the bottom. They widen the page a bit.
____________
Doesn't expecting the unexpected make the unexpected the expected?
If it makes sense, DON'T do it.

Profile Arif Mert Kapicioglu
Send message
Joined: 14 Dec 09
Posts: 158
Credit: 572,866,449
RAC: 4,073
Message 36217 - Posted: 2 Feb 2010 | 8:12:25 UTC

The fitness gets better, especially the de_s222_opt_1_v01 evolution. I wonder about the maps astronomers work on. Is there an estimated release date?

Rob
Send message
Joined: 6 Apr 10
Posts: 1
Credit: 25,275
RAC: 0
Message 38746 - Posted: 15 Apr 2010 | 1:11:22 UTC

Hi my name is Rob and I'm new to the MilkyWay@Home project, but I thought I would give my 2 cents anyways.

I am a computer science student currently enrolled at Ramapo College and will be graduating in December. Looking for a graduate program, I found RPI, and soon after MilkyWay@Home.

I've been reading the posts from earlier and I found a lot of people would like something graphical and my suggestion follows that line of thought.

My idea would be a Milkyway Earth (think Google Earth but in space) that would give us at least an idea (may not be accurate) of what our computers are actually helping Heidi Newberg and other professors compute.


Congrats on your great work and maybe one day soon I could help with your project.

Thanks for listening.
-- Rob

Profile Travis
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 30 Aug 07
Posts: 1976
Credit: 26,480
RAC: 0
Message 38748 - Posted: 15 Apr 2010 | 4:02:23 UTC - in response to Message 38746.

Hi my name is Rob and I'm new to the MilkyWay@Home project, but I thought I would give my 2 cents anyways.

I am a computer science student currently enrolled at Ramapo College and will be graduating in December. Looking for a graduate program, I found RPI, and soon after MilkyWay@Home.

I've been reading the posts from earlier and I found a lot of people would like something graphical and my suggestion follows that line of thought.

My idea would be a Milkyway Earth (think Google Earth but in space) that would give us at least an idea (may not be accurate) of what our computers are actually helping Heidi Newberg and other professors compute.


Congrats on your great work and maybe one day soon I could help with your project.

Thanks for listening.
-- Rob


Hi Rob,

If you're accepted to RPI let us know :) What graduate department did you apply to?

One of our undergrads is currently working on some data visualization, I hope it goes somewhere :)
____________

Post to thread

Message boards : Number crunching : search progress visualization


Main page · Your account · Message boards


Copyright © 2013 AstroInformatics Group