Dear MWAH Staff,

I'm looking for your help here. MWAH WUs are continually causing BOINC Manager to hang at the system level. I addressed this issue here...

WU Freezes BOINC Manager...:
http://milkyway.cs.rpi.edu/milkyway/forum_thread.php?id=2639

...back in October...resurrected the thread back in January...and, to date, have received zero feedback from anyone at MWAH. The problem persists and is now occurring with a less-than-48-hour regularity.

Symptomology
Desktop Clues...
Due to the frequency of this problem I have had to resort to running Activity Monitor in OSX continuously with the floating CPU meter on my desktop to let me know when there is a problem with MWAH/BOINC.

When everything is running normally this meter shows 8 green bars at 100% full levels indicating that all 8 CPUs are running at maximum performance.

When the MWAH/BOINC freeze occurs the floating CPU meters show only 4 green bars at approximately 45% levels. I know it's time to attend to MWAH/BOINC at that point. Sometimes hours go by before I'm at the computer to discover that the software has hung.

MWAH/BOINC Clues...
The Task tab in BOINC Manager shows 1 MWAH WU Completed Ready to report at 100% and 7 or 8 running MWAH WUs but the Progress/Elapsed/Remaining columns show no changes in percentages or time for those supposedly running 7 or 8 WUs. The software has now hung.

System Clues...
Activity Monitor sometimes shows that the supposed Running WUs are still in system memory (using 0.0% CPU) and sometimes the WUs have been dumped from system memory.

BOINC (user "me") is still in memory but using 0.0% CPU. boinc (user "boinc_master") is still running using ~87% of the total CPUs.

Of note is that two system processes are working their tails off at this point also...syslog @ ~47% and mDNSResponder @ ~ 37%. Total CPU load: 87 + 47 + 37 = 171% total CPU = 4 green bars at ~ 45%.

More MWAH/BOINC Clues...
Trying to do anything in MWAH/BOINC at this point results in the appearance of the dreaded Communicating with BOINC Client. Please wait... dialog box. MWAH/BOINC will never recover or clear itself from this state.

The only option at this point is to click the Quit BOINC Manager button in that dialog box and start over.

Note: One can avoid the appearance of that dialog box by just going to "Quit BOINC Manger" in the File menu.

New System Clues...
Since my original post I decided to see if I could discover a bit more as to what was happening to my system when this software hang occurred...here's a breakdown of events from the latest occurrence...


1. I notice the 4 x 45% green CPU bars at 22:50...yep, another 4-CPU MWAH/BOINC software hang. Last BOINC log entry at 21:26...not bad, only lost about an hour and a half of crunching this time. I quit/restart BOINC Manager.

2. stdoutdae log of this...

23-Feb-2012 21:26:12 [Milkyway@Home] [task] result ps_separation_82_2s_mix2_3_6073084_0 checkpointed
23-Feb-2012 21:26:12 [Milkyway@Home] [task] result ps_separation_82_2s_mix2_3_6073029_0 checkpointed
23-Feb-2012 21:26:12 [Milkyway@Home] [task] result ps_separation_82_2s_mix3_3_6076873_0 checkpointed
23-Feb-2012 21:26:12 [Milkyway@Home] [task] result ps_separation_82_2s_mix0_3_6077609_0 checkpointed
23-Feb-2012 21:26:12 [Milkyway@Home] [task] result ps_separation_82_2s_mix1_3_6075129_0 checkpointed
23-Feb-2012 21:26:12 [Milkyway@Home] [task] result ps_separation_82_2s_mix3_3_6079305_0 checkpointed
23-Feb-2012 21:26:12 [Milkyway@Home] [task] result ps_separation_82_2s_mix2_3_6079646_0 checkpointed
23-Feb-2012 21:26:12 [Milkyway@Home] [task] result ps_separation_82_2s_mix3_3_6086074_0 checkpointed
23-Feb-2012 21:26:29 [Milkyway@Home] [task] Process for ps_separation_82_2s_mix2_3_6073084_0 exited
23-Feb-2012 21:26:29 [Milkyway@Home] [task] task_state=EXITED for ps_separation_82_2s_mix2_3_6073084_0 from handle_exited_app
23-Feb-2012 21:26:29 [Milkyway@Home] [task] process exited with status 0
23-Feb-2012 21:26:29 [Milkyway@Home] Computation for task ps_separation_82_2s_mix2_3_6073084_0 finished
23-Feb-2012 21:26:29 [Milkyway@Home] [task] result state=FILES_UPLOADING for ps_separation_82_2s_mix2_3_6073084_0 from CS::app_finished
23-Feb-2012 21:26:29 [Milkyway@Home] [task] result state=FILES_UPLOADED for ps_separation_82_2s_mix2_3_6073084_0 from CS::update_results
23-Feb-2012 21:26:29 [Milkyway@Home] [task] ACTIVE_TASK::start(): forked process: pid 12521
23-Feb-2012 21:26:29 [Milkyway@Home] [task] task_state=EXECUTING for ps_separation_82_2s_mix2_3_6086089_0 from start
23-Feb-2012 21:26:29 [Milkyway@Home] Starting task ps_separation_82_2s_mix2_3_6086089_0 using milkyway version 101
23-Feb-2012 22:50:05 [---] Starting BOINC client version 6.12.35 for x86_64-apple-darwin
23-Feb-2012 22:50:05 [---] log flags: file_xfer, sched_ops, task, task_debug
23-Feb-2012 22:50:05 [---] Libraries: libcurl/7.19.7 OpenSSL/0.9.7l zlib/1.2.3 c-ares/1.6.0
23-Feb-2012 22:50:05 [---] Data directory: /Library/Application Support/BOINC Data
23-Feb-2012 22:50:05 [---] Processor: 8 GenuineIntel Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz [x86 Family 6 Model 42 Stepping 7]
23-Feb-2012 22:50:05 [---] Processor features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX SMX EST TM2 SSSE3 CX16 TPR PDCM SSE4.1 SSE4.2 xAPIC POPCNT AES XSAVE OSXSAVE PCID TSCTMR AVX1.0
23-Feb-2012 22:50:05 [---] OS: Mac OS X 10.6.8 (Darwin 10.8.0)
23-Feb-2012 22:50:05 [---] Memory: 16.00 GB physical, 359.23 GB virtual
23-Feb-2012 22:50:05 [---] Disk: 465.75 GB total, 358.99 GB free
23-Feb-2012 22:50:05 [---] Local time is UTC -5 hours
23-Feb-2012 22:50:05 [---] No usable GPUs found
23-Feb-2012 22:50:05 [climateprediction.net] URL http://climateprediction.net/; Computer ID 1171126; resource share 100
23-Feb-2012 22:50:05 [Milkyway@Home] URL http://milkyway.cs.rpi.edu/milkyway/; Computer ID 330727; resource share 100
23-Feb-2012 22:50:05 [SETI@home] URL http://setiathome.berkeley.edu/; Computer ID 6180003; resource share 100
23-Feb-2012 22:50:05 [Milkyway@Home] General prefs: from Milkyway@Home (last modified 22-Feb-2012 15:54:46)
23-Feb-2012 22:50:05 [Milkyway@Home] Computer location: home
23-Feb-2012 22:50:05 [Milkyway@Home] General prefs: no separate prefs for home; using your defaults
23-Feb-2012 22:50:05 [---] Reading preferences override file
23-Feb-2012 22:50:05 [---] Preferences:
23-Feb-2012 22:50:05 [---] max memory usage when active: 12288.00MB
23-Feb-2012 22:50:05 [---] max memory usage when idle: 14745.60MB
23-Feb-2012 22:50:05 [---] max disk usage: 100.00GB
23-Feb-2012 22:50:05 [---] (to change preferences, visit the web site of an attached project, or select Preferences in the Manager)
23-Feb-2012 22:50:05 [---] Not using a proxy
Initialization completed
23-Feb-2012 22:50:05 [Milkyway@Home] [task] ACTIVE_TASK::start(): forked process: pid 152
23-Feb-2012 22:50:05 [Milkyway@Home] [task] task_state=EXECUTING for ps_separation_82_2s_mix2_3_6073029_0 from start
23-Feb-2012 22:50:05 [Milkyway@Home] Restarting task ps_separation_82_2s_mix2_3_6073029_0 using milkyway version 101
23-Feb-2012 22:50:05 [Milkyway@Home] [task] ACTIVE_TASK::start(): forked process: pid 153
23-Feb-2012 22:50:05 [Milkyway@Home] [task] task_state=EXECUTING for ps_separation_82_2s_mix3_3_6076873_0 from start
23-Feb-2012 22:50:05 [Milkyway@Home] Restarting task ps_separation_82_2s_mix3_3_6076873_0 using milkyway version 101
23-Feb-2012 22:50:05 [Milkyway@Home] [task] ACTIVE_TASK::start(): forked process: pid 154
23-Feb-2012 22:50:05 [Milkyway@Home] [task] task_state=EXECUTING for ps_separation_82_2s_mix0_3_6077609_0 from start
23-Feb-2012 22:50:05 [Milkyway@Home] Restarting task ps_separation_82_2s_mix0_3_6077609_0 using milkyway version 101
23-Feb-2012 22:50:05 [Milkyway@Home] [task] ACTIVE_TASK::start(): forked process: pid 155
23-Feb-2012 22:50:05 [Milkyway@Home] [task] task_state=EXECUTING for ps_separation_82_2s_mix1_3_6075129_0 from start
23-Feb-2012 22:50:05 [Milkyway@Home] Restarting task ps_separation_82_2s_mix1_3_6075129_0 using milkyway version 101
23-Feb-2012 22:50:05 [Milkyway@Home] [task] ACTIVE_TASK::start(): forked process: pid 156
23-Feb-2012 22:50:05 [Milkyway@Home] [task] task_state=EXECUTING for ps_separation_82_2s_mix3_3_6079305_0 from start
23-Feb-2012 22:50:05 [Milkyway@Home] Restarting task ps_separation_82_2s_mix3_3_6079305_0 using milkyway version 101
23-Feb-2012 22:50:05 [Milkyway@Home] [task] ACTIVE_TASK::start(): forked process: pid 157
23-Feb-2012 22:50:05 [Milkyway@Home] [task] task_state=EXECUTING for ps_separation_82_2s_mix2_3_6079646_0 from start
23-Feb-2012 22:50:05 [Milkyway@Home] Restarting task ps_separation_82_2s_mix2_3_6079646_0 using milkyway version 101
23-Feb-2012 22:50:05 [Milkyway@Home] [task] ACTIVE_TASK::start(): forked process: pid 158
23-Feb-2012 22:50:05 [Milkyway@Home] [task] task_state=EXECUTING for ps_separation_82_2s_mix3_3_6086074_0 from start
23-Feb-2012 22:50:05 [Milkyway@Home] Restarting task ps_separation_82_2s_mix3_3_6086074_0 using milkyway version 101
23-Feb-2012 22:50:05 [Milkyway@Home] [task] ACTIVE_TASK::start(): forked process: pid 159
23-Feb-2012 22:50:05 [Milkyway@Home] [task] task_state=EXECUTING for ps_separation_82_2s_mix2_3_6086089_0 from start
23-Feb-2012 22:50:05 [Milkyway@Home] Restarting task ps_separation_82_2s_mix2_3_6086089_0 using milkyway version 101
23-Feb-2012 22:50:05 [Milkyway@Home] Sending scheduler request: To fetch work.
23-Feb-2012 22:50:05 [Milkyway@Home] Reporting 1 completed tasks, requesting new tasks for CPU

As per usual, no clues here.

3. I discovered a few weeks back that at the system level BOINC Manager is stuck in an endless loop. Here's a snippet of my OSX syslog from this time period today beginning at the MWAH/BOINC hang at 21:26...

Feb 23 21:26:30 imac /Applications/BOINCManager.app/Contents/Resources/boinc[151]: dnssd_clientstub deliver_request: socketpair failed 24 (Too many open files)
Feb 23 21:26:30: --- last message repeated 499 times ---
Feb 23 21:26:30 imac /Applications/BOINCManager.app/Contents/Resources/boinc[151]: *** process 151 exceeded 500 log message per second limit - remaining messages this second discarded ***
Feb 23 21:26:30 imac /Applications/BOINCManager.app/Contents/Resources/boinc[151]: dnssd_clientstub deliver_request: socketpair failed 24 (Too many open files)
Feb 23 21:26:31: --- last message repeated 499 times ---
Feb 23 21:26:31 imac /Applications/BOINCManager.app/Contents/Resources/boinc[151]: *** process 151 exceeded 500 log message per second limit - remaining messages this second discarded ***
Feb 23 21:26:31 imac /Applications/BOINCManager.app/Contents/Resources/boinc[151]: dnssd_clientstub deliver_request: socketpair failed 24 (Too many open files)

...and this continues unabated for the full time until I intervene by quitting BOINC manager at 22:50 and do a system restart...

Feb 23 22:48:48 imac /Applications/BOINCManager.app/Contents/Resources/boinc[151]: *** process 151 exceeded 500 log message per second limit - remaining messages this second discarded ***
Feb 23 22:48:49 imac /Applications/BOINCManager.app/Contents/Resources/boinc[151]: dnssd_clientstub deliver_request: socketpair failed 24 (Too many open files)
Feb 23 22:48:49: --- last message repeated 499 times ---
Feb 23 22:48:49 imac /Applications/BOINCManager.app/Contents/Resources/boinc[151]: *** process 151 exceeded 500 log message per second limit - remaining messages this second discarded ***
Feb 23 22:48:49 imac /Applications/BOINCManager.app/Contents/Resources/boinc[151]: dnssd_clientstub deliver_request: socketpair failed 24 (Too many open files)
Feb 23 22:48:50: --- last message repeated 499 times ---
Feb 23 22:48:50 imac /Applications/BOINCManager.app/Contents/Resources/boinc[151]: *** process 151 exceeded 500 log message per second limit - remaining messages this second discarded ***
Feb 23 22:49:08 iMac configd[13]: setting hostname to "iMac.local"
Feb 23 22:49:08 iMac configd[13]: network configuration changed.
Feb 23 22:49:11 iMac [0x0-0x32032].org.xxx[6041]: NOTE: child process received `Goodbye', closing down
Feb 23 22:49:11: --- last message repeated 2 times ---
Feb 23 22:49:11 iMac com.apple.launchd.peruser.501[103] ([0x0-0x9009].com.xxx[131]): Exited: Killed
Feb 23 22:49:11 iMac com.apple.launchd.peruser.501[103] ([0x0-0xf00f].com.apple.ActivityMonitor[176]): Exited: Killed
Feb 23 22:49:11 iMac loginwindow[30]: DEAD_PROCESS: 30 console
Feb 23 22:49:12 iMac shutdown[12578]: reboot by Owner:

...this accounts for why syslog was using 47% CPU and mDNSResponder was using 37%!

Final Note:
The system and MWAH/BOINC will now run fine for the next day or so (not more than two!) until this occurs again...

Any help, ideas, suggestions would be greatly appreciated?!

Best to you,
:)

BOINC freezing isn't really something milkyway can do. Can you try using the sample process thing in Activity Monitor on the boinc processes when it does happen (it's there on Lion)?

BOINC freezing isn't really something milkyway can do. Can you try using the sample process thing in Activity Monitor on the boinc processes when it does happen (it's there on Lion)?

Can do, Matt. Thanks for the suggestion.

I posted this problem here as the folks over at BOINC informed me that they were no longer servicing any per-version 7 releases of BOINC Manager...my hope was that someone here might have some "legacy" fluency with this particular misbehavior. If the problem was a bit more infrequent I'd probably not care too much, but having to reset things 2-3-4 times a week was starting to get to me.

Thanks again!
:)

One possibility is the scheduling/handover interaction between mt tasks and single threaded tasks. I remember when I did AQUA mt tasks I sometimes had the problem of idle cores, so the combination of mt tasks and single threaded tasks has a history of causing trouble in BOINC. You may have the opposite problem of an nbody mt task starting or attempting to start causing some separation tasks to freeze and boinc.exe becoming unresponsive.

I would not process separation tasks on a CPU myself because I feel CPU resources are more efficiently used to do nbody tasks or other CPU projects. I don't see the point in taking 2 hours on a CPU core for a task that can be done in under a minute on the last three generations of ATI/AMD cards.

If the nbody tasks are currently too short and cause work fetch issues I would share CPU resources with another project. Or just give the MilkyWay project a miss if I had no GPU until the nbody tasks are a more reasonable length and/or the number of tasks allowed in the cache is increased so that selecting nbody tasks only no longer causes work fetch problems. 5 seconds runtime is just too short with the current number of tasks allowed. Also the shorter that mt tasks are the greater the number of times that BOINC is switching from and to single threaded tasks when the mt tasks start or finish. So the greater potential for trouble to occur and for it to happen more often. If the CPU is busy at the time of the switch and it is delayed/interrupted then this could possibly cause the problem, especially as you are using all 8 cores for BOINC.

The first thing you could try is configuring BOINC to use 7 cores. This leaves a core free for operating system processes and so minimises/avoids momentary CPU overcommitment. Some do not like this idea because it reduces BOINC contribution slightly, but due to contention issues the reduction is less than the 12.5% expected because the tasks speed up slightly. In the case of high cache/memory intense CPU tasks the reduction in total work performed is so low that it is of no concern.

If that didn't work you could then perhaps try limiting the number of CPU cores used by nbody tasks to 5 with an app_info.xml file. With the current short tasks you could possibly even limit nbody to 4, 3 or 2 cores but this would allow more than 1 nbody task to run at the same time. You would need to include both the nbody and the separation application in the app_info.xml if you wanted to do both. The theory here is that a smaller number of CPU cores being switched by BOINC when mt tasks start or finish has less potential to cause trouble. May not work of course, just something to experiment with if you wish.

A further option is to try a future development version of BOINC which has had some changes made in mt tasks job scheduling. This is designed to fix the idle CPUs problem but the changes may also affect your issues. These changes were only made 3 days ago so you will have have to wait until a development BOINC version with the changes included is released. Changeset 25312 in boinc

You should only try a development version of BOINC after you have exhausted all other options. Only use as a last resort. The usual disclaimers apply about development versions of BOINC. May be unstable, may cause loss of all work, etc., etc. One particular thing to note is that you can't revert to a current release version of BOINC from development 7.0.xx versions without losing all work in progress. This is of particular importance to those who contribute to projects with very long running tasks such as Climate Prediction.

...

I would not process separation tasks on a CPU myself because I feel CPU resources are more efficiently used to do nbody tasks or other CPU projects. I don't see the point in taking 2 hours on a CPU core for a task that can be done in under a minute on the last three generations of ATI/AMD cards.

...

Hi Kashi,

Thanks for taking the time to put all of that together, I do appreciate it... :)

Allocating GPU cycles for MWAH/BOINC projects is a near-impossibility for those of us crunching away in Apple's "Walled Garden of Hardware", with the exception of their MacPro line (which can run DP GPUs for MWAH), and Hackintoshes, all of their computers are based off of mobile platforms with SP GPUs.

To update my original observed symptomology, it is not just N-Body's that seem to trigger these hangs.

As for your suggestions of rolling back CPU allocations from 100%, I had considered this initially but the issue I'm describing here seems to be a network communications one and so I'm thinking that CPU usage levels are not relevant here. The never-ending log entries...

Feb 23 21:26:30 imac /Applications/BOINCManager.app/Contents/Resources/boinc[151]: dnssd_clientstub deliver_request: socketpair failed 24 (Too many open files)
Feb 23 21:26:30: --- last message repeated 499 times ---
Feb 23 21:26:30 imac /Applications/BOINCManager.app/Contents/Resources/boinc[151]: *** process 151 exceeded 500 log message per second limit - remaining messages this second discarded ***

...look to me to be a exitless-loop of DNS socketpair attempts by MWAH/BOINC Manager.

My (very) crude understanding of how dnssd_clientstub.c works at the system level indicates to me that there is a network communications issue occurring in the MWAH/BOINC chain-of-events. Not being a coder I do not know how to fully interpret these things, but the dnssd_clientstub deliver_request: socketpair failed 24 (Too many open files) reports are clearly being generated from there...

dnssd_clientstub.c:
http://opensource.apple.com/source/mDNSResponder/mDNSResponder-212.1/mDNSShared/dnssd_clientstub.c

...(Warning: 10.5 Leopard version...I couldn't locate the 10.6 Snow Leopard or 10.7 Lion resources!). There was a reference in there about mDNSResponder and the Bonjour API being broken for 64-bit apps. (Important?)

(A quick google search for dnssd_clientstub deliver_request socketpair failed 24 reveals that a lot of other software developer folks are encountering this same problem.)

My pitiful understanding of how BOINC projects and BOINC Manager communicate with the mothership to both report and receive new WUs has led me here to the boards in search of answers. :D

I've run SETI and CPDN for days-long stretches at 100% and have never seen either of those two projects cause this problem...it seems isolated to MWAH/BOINC.

Hope this was helpful!

:)

Fair enough, I wasn't aware that MilkyWay GPU was not available for many Apple computers.

You had mentioned nbody tasks in your previous thread so I thought they may be involved somehow. So if it still occurs even when the nbody application is not selected and only separation tasks are processed, it cannot be due to one application or the other but to MilkyWay itself.

So what is different about MilkyWay compared to other projects, possibly in the network area? I haven't done any nbody tasks so I don't know if they are the same but I know that separation tasks no longer send an upload file but include the results when they are reported. Perhaps Mac BOINC version is waiting for uploads or acknowledgements that never happen? This may not be relevant but it's the only difference to other projects I can think of.

I can't offer any Apple specific help at all, I have never used an Apple computer and know almost nothing about them.

Hi Kashi,

Yeah, while Apple does do a grand job making sure everything inside their boxes work to a tee they've always been restrictive on hardware choices and upgrade paths...sigh...

My fault for not being a bit more observant when this problem first surfaced re: NBody's being the only culprit...oops!

I thought the whole MWAH/BOINC connection odd, too...ergo these posts. In the meantime I continue to hold out hope that someone with some previous experience and/or understanding with this error can help shine some light on things!

I've been using Apple products for nearly 20 years now and I can only pretend to know anything about them! :)

I do appreciate your help and insights here!

Best, :)

Hi Matt,

There are 5 ways to examine the Sample Process data in Activity Monitor...
1. Sample Text,
2. # of Samples,
3. # of Seconds,
4. % of Parent and
5. % of Thread.

Below is the Sample Text view of boinc (User: boinc_master) which is running at 87% CPU from my latest 4-CPU hang...

I can zip up and send you any/all of the following saved Sample Process views...

boic 1–5 (boinc_master)
BOINC 1–5 (Me)
mDNSResponder 1–5
syslog 1–5

...just PM me with an email address. If there are any other things you'd like me to check, report on, etc. kindly let me know!
______________

Note: MWAH/Boinc is currently hung...all Progress, Elapsed, Remaining activity is frozen in the current states...

Current Status of MWAH/BOINC Manager
100.000% Ready to report 82_2s_mix1_3_6662031_1
79.766% Waiting to run 82_2s_mix2_3_6666333_0
23.147% Waiting to run 82_2s_mix4_3_6667392_0
21.191% Waiting to run 82_2s_mix3_3_6667847_0
20.698% Waiting to run 82_2s_mix1_3_6665662_0
17.514% Waiting to run 82_2s_mix2_3_6667202_0
13.145% Waiting to run 82_2s_mix4_3_6668219_0
11.250% Waiting to run 82_2s_mix0_3_6669847_0
100.000% Ready to report nbody-Plum_Embedded_13182472_0
100.000% Running nbody-Plum_Embedded_13183500_0
_______

Last Boinc log entry at Mon Feb 27 16:50 (currently 19:26)
_______

Hope this helps!
:)
_______

Analysis of sampling boinc (pid 16118) every 1 millisecond
Process: boinc [16118]
Path: /Applications/BOINCManager.app/Contents/Resources/boinc
Load Address: 0x1000
Identifier: boinc
Version: ??? (???)
Code Type: X86 (Native)
Parent Process: BOINCManager [16113]

Date/Time: 2012-02-27 17:36:43.773 -0500
OS Version: Mac OS X 10.6.8 (10K549)
Report Version: 7

Call graph:
2629 Thread_133420 DispatchQueue_1: com.apple.main-thread (serial)
2629 0x10df9
2629 0x10ed2
2629 0x3ba9e
2629 0x5d5a4
2629 0x3005d
2629 0x609d7
2629 0x2f613
2629 0x39a5f
2629 0x3934e
2629 0x489a1
2629 getaddrinfo
2629 si_addrinfo
2629 search_addrinfo
2629 _mdns_addrinfo
2629 _mdns_search
2629 _mdns_query_mDNSResponder
1390 _mdns_query_start
1385 DNSServiceQueryRecord
1256 deliver_request
1114 syslog
1109 vsyslog
213 vm_allocate
209 mach_msg
205 mach_msg_trap
4 mach_msg
3 mig_get_reply_port
1 vm_allocate
199 asl_format_message
191 _asl_append_string
94 reallocf
87 realloc
45 malloc_zone_realloc
43 szone_realloc
10 szone_free
9 szone_free
1 tiny_free_list_add_ptr
10 szone_malloc_should_clear
7 tiny_malloc_from_free_list
5 tiny_malloc_from_free_list
2 tiny_free_list_add_ptr
3 szone_malloc_should_clear
7 szone_realloc
5 szone_size
4 __memcpy
4 __spin_lock
2 tiny_free_list_remove_ptr
1 tiny_free_list_add_ptr
2 malloc_zone_realloc
29 szone_size
13 realloc
7 reallocf
89 _asl_append_string
5 strlen
2 __memcpy
1 malloc
1 malloc_zone_malloc
1 szone_malloc_should_clear
1 __cpu_number
8 asl_format_message
166 _asl_server_message
165 mach_msg
163 mach_msg_trap
2 mach_msg
1 _asl_server_message
84 asl_set
63 asl_set_query
25 strdup
14 malloc
12 malloc_zone_malloc
9 szone_malloc_should_clear
8 tiny_malloc_from_free_list
1 szone_malloc_should_clear
1 DYLD-STUB$$_spin_unlock
1 malloc_zone_malloc
1 szone_malloc
2 malloc
5 __memcpy
5 strdup
1 strlen
14 reallocf
10 realloc
9 malloc_zone_realloc
9 szone_realloc
4 szone_free
2 szone_free
2 tiny_free_list_add_ptr
2 __spin_lock
1 szone_malloc_should_clear
1 __cpu_number
1 szone_realloc
1 szone_size
1 realloc
4 reallocf
12 asl_set_query
6 asprintf
4 __vfprintf
2 malloc
2 malloc_zone_malloc
2 szone_malloc_should_clear
1 szone_malloc_should_clear
1 tiny_malloc_from_free_list
5 atoi
4 strtol_l
1 atoi
1 DYLD-STUB$$strtol_l
20 strcmp
1 DYLD-STUB$$strcmp
83 gethostname
79 __sysctl
4 sysctl
77 __bzero
61 asprintf
42 __vfprintf
26 __vfprintf
6 __ultoa
6 localeconv_l
4 __sfvwrite
3 __sfvwrite
1 __memcpy
9 asprintf
6 malloc
5 malloc_zone_malloc
4 szone_malloc_should_clear
3 szone_malloc_should_clear
1 __cpu_number
1 __spin_lock
1 malloc
2 __bzero
1 DYLD-STUB$$localeconv_l
1 pthread_getspecific
53 asl_free
53 free
26 szone_free_definite_size
17 szone_free_definite_size
8 tiny_free_list_add_ptr
1 tiny_free_list_remove_ptr
11 szone_size
10 __spin_lock
4 free
2 OSSpinLockUnlock
37 snprintf
30 __vfprintf
16 __vfprintf
9 __sfvwrite
4 __memcpy
4 __sfvwrite
1 memcpy
3 localeconv_l
1 __ultoa
1 strlen
6 snprintf
1 __bzero
33 vasprintf
27 __vfprintf
20 __vfprintf
5 __sfvwrite
4 __sfvwrite
1 __memcpy
1 __ultoa
1 strlen
3 malloc
2 malloc
1 malloc_zone_malloc
3 vasprintf
31 free
10 szone_size
8 __spin_lock
8 szone_free_definite_size
7 szone_free_definite_size
1 tiny_free_list_add_ptr
5 free
15 asl_new
10 strdup
6 malloc
5 malloc_zone_malloc
4 szone_malloc_should_clear
2 szone_malloc_should_clear
2 tiny_malloc_from_free_list
1 malloc_zone_malloc
1 malloc
2 strlen
1 __memcpy
1 strdup
5 calloc
3 malloc_zone_calloc
3 szone_malloc_should_clear
1 __bzero
1 szone_malloc_should_clear
1 tiny_malloc_from_free_list
2 calloc
14 getuid
12 vsyslog
9 getgid
9 notify_check
6 token_table_find
3 pthread_mutex_lock
2 pthread_mutex_lock
1 OSSpinLockUnlock
2 _nc_table_find_n
1 pthread_mutex_unlock
3 notify_check
7 gettimeofday
7 __gettimeofday
5 __nanotime
2 __gettimeofday
3 strcmp
2 strncpy
1 __bzero
1 strncpy
1 asl_syslog_faciliy_num_to_name
3 syslog
2 DYLD-STUB$$asl_format_message
130 socketpair$UNIX2003
4 deliver_request
4 strerror$UNIX2003
3 free
2 szone_free_definite_size
1 free
1 __error
107 DNSServiceRefDeallocate
92 write_all
90 send$NOCANCEL$UNIX2003
90 sendto$NOCANCEL$UNIX2003
1 _sysenter_trap
1 write_all
9 create_hdr
7 malloc
7 malloc_zone_malloc
6 szone_malloc_should_clear
6 tiny_malloc_from_free_list
1 __spin_lock
1 __bzero
1 create_hdr
2 DNSServiceRefDeallocate
2 FreeDNSServiceOp
2 free
1 free
1 szone_size
2 free
1 free
1 szone_free_definite_size
14 create_hdr
4 malloc
4 malloc_zone_malloc
3 szone_malloc_should_clear
3 tiny_malloc_from_free_list
1 malloc_zone_malloc
3 create_hdr
3 put_string
1 DYLD-STUB$$strcpy
1 DYLD-STUB$$strlen
1 __bzero
1 strcpy
3 DNSServiceQueryRecord
1 ConnectToServer
1 malloc
1 malloc_zone_malloc
1 cerror
1 cthread_set_errno_self
1 __error
1 put_uint16
1 put_uint32
1 strcpy
4 strlen
1 _mdns_query_start
1136 DNSServiceCreateConnection
598 ConnectToServer
373 connect$NOCANCEL$UNIX2003
153 socket
66 setsockopt
6 ConnectToServer
529 deliver_request
485 read_all
484 recv$NOCANCEL$UNIX2003
484 recvfrom$NOCANCEL$UNIX2003
1 read_all
30 write_all
29 send$NOCANCEL$UNIX2003
29 sendto$NOCANCEL$UNIX2003
1 write_all
10 free
5 szone_size
3 free
2 szone_free_definite_size
1 OSSpinLockLock
1 szone_free_definite_size
3 deliver_request
1 ConvertHeaderBytes
4 create_hdr
4 malloc
3 malloc_zone_malloc
3 szone_malloc_should_clear
1 OSSpinLockLock
1 OSSpinLockUnlock
1 __cpu_number
1 malloc
2 DNSServiceCreateConnection
2 _sysenter_trap
1 DYLD-STUB$$setsockopt
97 DNSServiceRefDeallocate
95 close
1 DNSServiceRefDeallocate
1 FreeDNSServiceOp
1 free
3 _mdns_query_mDNSResponder
1 DYLD-STUB$$strlen
1 _mdns_query_clear
1 pthread_mutex_lock
1 OSSpinLockLock
2629 Thread_133422 DispatchQueue_2: com.apple.libdispatch-manager (serial)
2629 start_wqthread
2629 _pthread_wqthread
2629 _dispatch_worker_thread2
2629 _dispatch_queue_invoke
2629 _dispatch_mgr_invoke
2629 kevent

Total number in stack (recursive counted multiple, when >=5):
11 szone_malloc_should_clear
10 malloc
10 malloc_zone_malloc
7 __bzero
7 __memcpy
7 free
7 szone_size
7 tiny_malloc_from_free_list
6 __spin_lock
6 strlen
6 tiny_free_list_add_ptr
5 szone_free_definite_size

Sort by top of stack, same collapsed (when >= 5):
kevent 2629
recvfrom$NOCANCEL$UNIX2003 484
connect$NOCANCEL$UNIX2003 373
mach_msg_trap 368
socket 153
socketpair$UNIX2003 130
sendto$NOCANCEL$UNIX2003 119
close 95
_asl_append_string 89
__bzero 84
__sysctl 79
__vfprintf 66
setsockopt 66
szone_size 62
szone_free_definite_size 28
__spin_lock 26
tiny_malloc_from_free_list 26
strcmp 23
__memcpy 18
free 16
tiny_free_list_add_ptr 15
getuid 14
realloc 14
strlen 14
asl_set_query 12
vsyslog 12
__sfvwrite 11
reallocf 11
szone_free 11
szone_malloc_should_clear 11
asprintf 9
getgid 9
localeconv_l 9
__ultoa 8
asl_format_message 8
szone_realloc 8
deliver_request 7
malloc 7
ConnectToServer 6
mach_msg 6
snprintf 6
strdup 6
__nanotime 5
malloc_zone_malloc 5

Binary Images:
0x1000 - 0xcdff7 +boinc (??? - ???) <9993FA00-F29B-6854-E41C-3592E2AA86B2> /Applications/BOINCManager.app/Contents/Resources/boinc
0xe4000 - 0x199fe7 libcrypto.0.9.7.dylib (0.9.7 - compatibility 0.9.7) <78B6EEB6-50D9-4479-D1A5-B4C381A07E4D> /usr/lib/libcrypto.0.9.7.dylib
0x1df000 - 0x205fff libssl.0.9.7.dylib (0.9.7 - compatibility 0.9.7) <1C17D1BE-1E54-70CF-07B2-86E5DA647C81> /usr/lib/libssl.0.9.7.dylib
0x8fe00000 - 0x8fe4163b dyld (132.1 - ???) <67866EA0-11AD-E175-066C-48E996265855> /usr/lib/dyld
0x90003000 - 0x90040ff7 com.apple.SystemConfiguration (1.10.9 - 1.10.2) <5C0F72C2-177E-675A-8677-9BC4205B4A98> /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
0x90041000 - 0x90074ff7 com.apple.AE (496.5 - 496.5) <BF9673D5-2419-7120-26A3-83D264C75222> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
0x9036f000 - 0x903b3ff3 com.apple.coreui (2 - 114) <1A3C3B7F-3837-6477-3114-47F6BFD56CB2> /System/Library/PrivateFrameworks/CoreUI.framework/Versions/A/CoreUI
0x9111d000 - 0x91170ff7 com.apple.HIServices (1.8.3 - ???) <1D3C4587-6318-C339-BD0F-1988F246BE2E> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
0x91171000 - 0x9120efe3 com.apple.LaunchServices (362.3 - 362.3) <15B47388-16C8-97DA-EEBB-1709E136169E> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
0x9120f000 - 0x91391fe7 libicucore.A.dylib (40.0.0 - compatibility 1.0.0) <D5980817-6D19-9636-51C3-E82BAE26776B> /usr/lib/libicucore.A.dylib
0x91392000 - 0x916b6fef com.apple.HIToolbox (1.6.5 - ???) <F21289A3-A00E-0BC3-66F1-7557862CE7EB> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0x91864000 - 0x918fffe7 com.apple.ApplicationServices.ATS (275.19 - ???) <9FA31967-CF14-B033-EB8D-570561D12A13> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
0x91b1f000 - 0x91b8eff7 libvMisc.dylib (268.0.1 - compatibility 1.0.0) <B3DEACA1-9375-CF8F-0898-AA2C5F8159DC> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
0x91c84000 - 0x91fa4ff3 com.apple.CoreServices.CarbonCore (861.39 - 861.39) <5C59805C-AF39-9010-B8B5-D673C9C38538> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0x91fbe000 - 0x91fcaff7 libkxld.dylib (??? - ???) <9A441C48-2D18-E716-5F38-CBEAE6A0BB3E> /usr/lib/system/libkxld.dylib
0x92f62000 - 0x9300affb com.apple.QD (3.36 - ???) <5A93B258-3853-636F-DB26-223642DA2779> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/QD.framework/Versions/A/QD
0x9300b000 - 0x9301cff7 com.apple.LangAnalysis (1.6.6 - 1.6.6) <B59157A1-8796-CE8F-2508-EB96F093F6A7> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/LangAnalysis.framework/Versions/A/LangAnalysis
0x9301d000 - 0x9307efe7 com.apple.CoreText (151.10 - ???) <5C2DEFBE-D54B-4DC7-D456-9ED02880BE98> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreText.framework/Versions/A/CoreText
0x93087000 - 0x9349dff7 libBLAS.dylib (219.0.0 - compatibility 1.0.0) <9D89FCB3-24C9-8FCF-DB49-27B184AC3222> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
0x9367d000 - 0x9369efe7 com.apple.opencl (12.3.6 - 12.3.6) <6BD7A385-02BC-AAC9-D654-93BD1805EA23> /System/Library/Frameworks/OpenCL.framework/Versions/A/OpenCL
0x9369f000 - 0x936fcff7 com.apple.framework.IOKit (2.0 - ???) <482CF2CC-DF02-2B5B-2133-18F5EA0052A8> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
0x9381d000 - 0x938cafe7 libobjc.A.dylib (227.0.0 - compatibility 1.0.0) <9F8413A6-736D-37D9-8EB3-7986D4699957> /usr/lib/libobjc.A.dylib
0x93903000 - 0x93aaaff7 libSystem.B.dylib (125.2.11 - compatibility 1.0.0) <2DCD13E3-1BD1-6F25-119A-3863A3848B90> /usr/lib/libSystem.B.dylib
0x943fc000 - 0x94424ff7 libxslt.1.dylib (3.24.0 - compatibility 3.0.0) <315D97C2-4E1F-A95F-A759-4A3FA5639E75> /usr/lib/libxslt.1.dylib
0x9456e000 - 0x9458afe3 com.apple.openscripting (1.3.1 - ???) <0E6B81D1-C1BD-1B5F-836F-256E6701B5DE> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/OpenScripting.framework/Versions/A/OpenScripting
0x945cc000 - 0x946d8ff7 libGLProgrammability.dylib (??? - ???) <04D7E5C3-B0C3-054B-DF49-3B333DCDEE22> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLProgrammability.dylib
0x94807000 - 0x94809ff7 libRadiance.dylib (??? - ???) <5920EB69-8D7F-5EFD-70AD-590FCB5C9E6C> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
0x9480a000 - 0x94818fe7 libz.1.dylib (1.2.3 - compatibility 1.0.0) <E1B922F4-23DC-467E-631F-7E1B9C9F51CB> /usr/lib/libz.1.dylib
0x94902000 - 0x94902ff7 com.apple.Accelerate.vecLib (3.6 - vecLib 3.6) <ABF97DA4-3BDF-6FFD-6239-B023CA1F7974> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
0x9495a000 - 0x949befff com.apple.htmlrendering (72 - 1.1.4) <0D22B190-513B-7FF6-39FC-9D336285DE08> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HTMLRendering.framework/Versions/A/HTMLRendering
0x949bf000 - 0x949daff7 libPng.dylib (??? - ???) <25DF2360-BFD3-0165-51AC-0BDAF7899DEC> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
0x949db000 - 0x94a1eff7 libGLU.dylib (??? - ???) <FB26DD53-03F4-A7D7-8804-EBC5B3B37FA3> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib
0x94a2c000 - 0x94a52ffb com.apple.DictionaryServices (1.1.2 - 1.1.2) <5A8D5D84-06EB-F9B5-BA0A-A87A0A2C0035> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
0x94c9c000 - 0x94c9cff7 com.apple.CoreServices (44 - 44) <B9461120-F72A-D28F-D0CF-5647958BD3F9> /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
0x94c9d000 - 0x94cbdfe7 libresolv.9.dylib (41.0.0 - compatibility 1.0.0) <584B2B30-DC65-6930-F59F-C49FD5604B67> /usr/lib/libresolv.9.dylib
0x94cbe000 - 0x94cd6ff7 com.apple.CFOpenDirectory (10.6 - 10.6) <F11D1CCA-0921-0BBA-C0D3-95BF21483E95> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
0x94dcc000 - 0x94e5efe7 com.apple.print.framework.PrintCore (6.3 - 312.7) <EE9ED7BE-7D97-B759-C063-1E4B7DBD3775> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/PrintCore.framework/Versions/A/PrintCore
0x94eb5000 - 0x94f2ffff com.apple.audio.CoreAudio (3.2.6 - 3.2.6) <156A532C-0B60-55B0-EE27-D02B82AA6217> /System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
0x94f75000 - 0x95077fe7 libcrypto.0.9.8.dylib (0.9.8 - compatibility 0.9.8) <015563C4-81E2-8C8A-82AC-31B38D904A42> /usr/lib/libcrypto.0.9.8.dylib
0x95084000 - 0x95098ffb com.apple.speech.synthesis.framework (3.10.35 - 3.10.35) <9F5CE4F7-D05C-8C14-4B76-E43D07A8A680> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
0x950e7000 - 0x95130fe7 libTIFF.dylib (??? - ???) <579DC328-567D-A74C-4BCE-1D1C729E3F6D> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
0x95131000 - 0x95920557 com.apple.CoreGraphics (1.545.0 - ???) <1D9DC7A5-228B-42CB-7018-66F42C3A9BB3> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
0x95958000 - 0x95958ff7 com.apple.ApplicationServices (38 - 38) <EAF1BC8C-4FD4-4300-B8F7-4B24E49125E2> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
0x95959000 - 0x9595cff7 libCoreVMClient.dylib (??? - ???) <F58BDFC1-7408-53C8-0B08-48BA2F25CA43> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib
0x95baa000 - 0x95bbffff com.apple.ImageCapture (6.1 - 6.1) <B909459A-EAC9-A7C8-F2A9-CD757CDB59E8> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/ImageCapture.framework/Versions/A/ImageCapture
0x960ca000 - 0x960d3ff7 com.apple.DiskArbitration (2.3.1 - 2.3.1) <AAE0185D-4A47-1833-27A3-03FDF734A3BA> /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
0x960d4000 - 0x960d7fe7 libmathCommon.A.dylib (315.0.0 - compatibility 1.0.0) <1622A54F-1A98-2CBE-B6A4-2122981A500E> /usr/lib/system/libmathCommon.A.dylib
0x960d8000 - 0x961d9fe7 libxml2.2.dylib (10.3.0 - compatibility 10.0.0) <E1A44C67-FA6C-0188-F46F-EA14CEE557BB> /usr/lib/libxml2.2.dylib
0x961fc000 - 0x96240fe7 com.apple.Metadata (10.6.3 - 507.15) <460BEF23-B89F-6F4C-4940-45556C0671B5> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
0x96241000 - 0x964b2fef com.apple.Foundation (6.6.8 - 751.63) <69B3441C-B196-F2AD-07F8-D8DD24E4CD8C> /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
0x964b3000 - 0x964c7fe7 libbsm.0.dylib (??? - ???) <B328FA0A-899C-4FC4-F2AC-2FDC08819CD2> /usr/lib/libbsm.0.dylib
0x96508000 - 0x9650aff7 com.apple.securityhi (4.0 - 36638) <BF4D0F7F-4FAE-602A-6CCA-37733F59BA49> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/SecurityHI.framework/Versions/A/SecurityHI
0x96809000 - 0x96984fe7 com.apple.CoreFoundation (6.6.6 - 550.44) <F88C95CD-1264-782D-A1F5-204739847E93> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
0x96a08000 - 0x96a0cff7 libGFXShared.dylib (??? - ???) <801B2C2C-1692-475A-BAD6-99F85B6E7C25> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib
0x9718c000 - 0x9723aff3 com.apple.ink.framework (1.3.3 - 107) <E215794C-F3E9-C067-8882-9054796DDA8A> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/Ink.framework/Versions/A/Ink
0x9723b000 - 0x97281ff7 libauto.dylib (??? - ???) <7CB1AB76-50A2-8E56-66E4-CF51CA75B177> /usr/lib/libauto.dylib
0x9728c000 - 0x972abff7 com.apple.CoreVideo (1.6.2 - 45.6) <F9E52A13-E181-D302-3B0E-0799ED6D6130> /System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo
0x972ac000 - 0x972fcff7 com.apple.framework.familycontrols (2.0.2 - 2020) <9062C95B-EE1C-977C-9D74-EF96CE888160> /System/Library/PrivateFrameworks/FamilyControls.framework/Versions/A/FamilyControls
0x972fd000 - 0x973c8fef com.apple.CoreServices.OSServices (359.2 - 359.2) <7C16D9C8-6F41-5754-17F7-2659D9DD9579> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
0x973c9000 - 0x977feff7 libLAPACK.dylib (219.0.0 - compatibility 1.0.0) <4D2F47EF-BD32-1E3C-6A0A-438896ADE2BE> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
0x977ff000 - 0x97881ffb SecurityFoundation (??? - ???) <6BDC767F-230D-3FE7-6174-10A329E241FA> /System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
0x9788b000 - 0x97896ff7 libGL.dylib (??? - ???) <3E34468F-E9A7-8EFB-FF66-5204BD5B4E21> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib
0x97897000 - 0x978daff7 com.apple.NavigationServices (3.5.4 - 182) <FE19D51B-233A-69EC-9E36-B89D0EA438C1> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/NavigationServices.framework/Versions/A/NavigationServices
0x979ba000 - 0x979c0fe7 com.apple.CommerceCore (1.0 - 9.1) <521D067B-3BDA-D04E-E1FA-CFA526C87EB5> /System/Library/PrivateFrameworks/CommerceKit.framework/Versions/A/Frameworks/CommerceCore.framework/Versions/A/CommerceCore
0x979c1000 - 0x979f9ff7 com.apple.LDAPFramework (2.0 - 120.1) <FDBA63D6-E765-407A-AD1F-858116AECE8C> /System/Library/Frameworks/LDAP.framework/Versions/A/LDAP
0x97a0a000 - 0x97a11ff3 com.apple.print.framework.Print (6.1 - 237.1) <726A7F31-8C27-8403-0016-71E022EDC14C> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/Print.framework/Versions/A/Print
0x97a8f000 - 0x97a8fff7 com.apple.vecLib (3.6 - vecLib 3.6) <FF4DC8B6-0AB0-DEE8-ADA8-7B57645A1F36> /System/Library/Frameworks/vecLib.framework/Versions/A/vecLib
0x98841000 - 0x98841ff7 com.apple.Carbon (150 - 152) <00542B48-EE9B-39E6-F67D-06CEF503519F> /System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
0x98855000 - 0x98862ff7 com.apple.NetFS (3.2.2 - 3.2.2) <12E7F3CD-F898-0FF6-A5DC-A70A8EF6B0E8> /System/Library/Frameworks/NetFS.framework/Versions/A/NetFS
0x98863000 - 0x98943fe7 com.apple.vImage (4.1 - 4.1) <AEF800AE-65DA-5E79-5B01-165AF51BAAF6> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
0x98b8f000 - 0x98b90ff7 com.apple.audio.units.AudioUnit (1.6.7 - 1.6.7) <838E1760-F7D9-3239-B3A8-20E25EFD1379> /System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
0x98b91000 - 0x98b9fff7 com.apple.opengl (1.6.13 - 1.6.13) <025A905D-C1A3-B24A-1585-37C328D77148> /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
0x98bfc000 - 0x98c00ff7 IOSurface (??? - ???) <8C403D51-2644-EBD6-A636-19F8ADEB0A09> /System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface
0x98c01000 - 0x98c3bfe7 libssl.0.9.8.dylib (0.9.8 - compatibility 0.9.8) <C62A7753-99A2-6782-92E7-6628A6190A90> /usr/lib/libssl.0.9.8.dylib
0x990f8000 - 0x991b0feb libFontParser.dylib (??? - ???) <D57D3834-9395-FD58-092A-49B3708E8C89> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.dylib
0x991c4000 - 0x991ceffb com.apple.speech.recognition.framework (3.11.1 - 3.11.1) <90C38107-AEE7-AE55-5C51-28D129B19BCD> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/SpeechRecognition.framework/Versions/A/SpeechRecognition
0x993fb000 - 0x9941fff7 libJPEG.dylib (??? - ???) <EA97DEC5-6E16-B51C-BF55-F6E8D23526AD> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
0x99420000 - 0x994ccfe7 com.apple.CFNetwork (454.12.4 - 454.12.4) <DEDCD006-389F-967F-3405-EDF541F406D7> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
0x994cd000 - 0x9968ffeb com.apple.ImageIO.framework (3.0.4 - 3.0.4) <027F55DF-7E4E-2310-1536-3F470CB8847B> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/ImageIO
0x99890000 - 0x9994cfff com.apple.ColorSync (4.6.6 - 4.6.6) <7CD8B191-039A-02C3-EA5E-4194EC59995B> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ColorSync.framework/Versions/A/ColorSync
0x99b55000 - 0x99b5bfff com.apple.CommonPanels (1.2.4 - 91) <CE92759E-865E-8A3B-1488-ECD497E4074D> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/CommonPanels.framework/Versions/A/CommonPanels
0x99b5c000 - 0x99c36fff com.apple.DesktopServices (1.5.11 - 1.5.11) <800F2040-9211-81A7-B438-7712BF51DEE3> /System/Library/PrivateFrameworks/DesktopServicesPriv.framework/Versions/A/DesktopServicesPriv
0x99c37000 - 0x99d74fe7 com.apple.audio.toolbox.AudioToolbox (1.6.7 - 1.6.7) <2D31CC6F-32CC-72FF-34EC-AB40CEE496A7> /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
0x99d75000 - 0x99dafff7 libcups.2.dylib (2.8.0 - compatibility 2.0.0) <6875335E-0993-0D77-4E80-41763A8477CF> /usr/lib/libcups.2.dylib
0x99df6000 - 0x99e06ff7 libsasl2.2.dylib (3.15.0 - compatibility 3.0.0) <E276514D-394B-2FDD-6264-07A444AA6A4E> /usr/lib/libsasl2.2.dylib
0x99e07000 - 0x99e11fe7 com.apple.audio.SoundManager (3.9.3 - 3.9.3) <DE0E0EF6-8190-3F65-6BDD-5AC9D8A025D6> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/CarbonSound.framework/Versions/A/CarbonSound
0x99e1a000 - 0x99eb2fe7 edu.mit.Kerberos (6.5.11 - 6.5.11) <49579559-BD62-A988-81DD-7E87D6A82C11> /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
0x99eb3000 - 0x99eb7ff7 libGIF.dylib (??? - ???) <2123645B-AC89-C4E2-8757-85834CAE3DD2> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
0x99eb8000 - 0x99eb9ff7 com.apple.TrustEvaluationAgent (1.1 - 1) <06484720-AB50-6FD9-B5BF-05F5A640C9E5> /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent
0x99eba000 - 0x99edcfef com.apple.DirectoryService.Framework (3.6 - 621.11) <E71F5EA4-A5C4-EDE7-EF52-69135D7DD798> /System/Library/Frameworks/DirectoryService.framework/Versions/A/DirectoryService
0x99edd000 - 0x99eefff7 com.apple.MultitouchSupport.framework (207.11 - 207.11) <6FF4F2D6-B8CD-AE13-56CB-17437EE5B741> /System/Library/PrivateFrameworks/MultitouchSupport.framework/Versions/A/MultitouchSupport
0x99f8f000 - 0x99f94ff7 com.apple.OpenDirectory (10.6 - 10.6) <48487E27-1AC5-8F8D-9A7D-AA41866A9BFF> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/OpenDirectory
0x99fa3000 - 0x99fe5ff7 libvDSP.dylib (268.0.1 - compatibility 1.0.0) <1AE34B00-8A62-1E51-935F-BB3F0E4BE50F> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
0x99fe6000 - 0x99fe6ff7 com.apple.Accelerate (1.6 - Accelerate 1.6) <3891A689-4F38-FACD-38B2-4BF937DE30CF> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
0x9a021000 - 0x9a0dafe7 libsqlite3.dylib (9.6.0 - compatibility 9.0.0) <03186216-0A12-4AA9-D7BA-3E3551A1195C> /usr/lib/libsqlite3.dylib
0x9a11c000 - 0x9a169feb com.apple.DirectoryService.PasswordServerFramework (6.1 - 6.1) <70E88670-86AC-8B94-2EED-4D945DC6398F> /System/Library/PrivateFrameworks/PasswordServer.framework/Versions/A/PasswordServer
0x9a2ab000 - 0x9a315fe7 libstdc++.6.dylib (7.9.0 - compatibility 7.0.0) <411D87F4-B7E1-44EB-F201-F8B4F9227213> /usr/lib/libstdc++.6.dylib
0x9a33c000 - 0x9a33fffb com.apple.help (1.3.2 - 41.1) <8AC20B01-4A3B-94BA-D8AF-E39034B97D8C> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/Help.framework/Versions/A/Help
0x9a340000 - 0x9a371ff7 libGLImage.dylib (??? - ???) <0EE86397-A867-0BBA-E5B1-B800E43FC5CF> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
0x9a37e000 - 0x9a3fefeb com.apple.SearchKit (1.3.0 - 1.3.0) <7AE32A31-2B8E-E271-C03A-7A0F7BAFC85C> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
0x9a55f000 - 0x9a7c4feb com.apple.security (6.1.2 - 55002) <7F00A51B-F22A-0EBC-A321-923472D686BD> /System/Library/Frameworks/Security.framework/Versions/A/Security
0x9a7c5000 - 0x9ab30ff7 com.apple.QuartzCore (1.6.3 - 227.37) <E323A5CC-499E-CA9E-9BC3-537231449CAA> /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
0xffff0000 - 0xffff1fff libSystem.B.dylib (??? - ???) <2DCD13E3-1BD1-6F25-119A-3863A3848B90> /usr/lib/libSystem.B.dylib

By way of follow up, Matt...I discovered this while researching socketpair on the internets...

#1141 (boinc client hangs) – BOINC:
http://boinc.berkeley.edu/trac/ticket/1141

:)

"fixed in 6.12.41"

http://boinc.berkeley.edu/trac/ticket/1144#comment:1

http://boinc.berkeley.edu/dl/?C=M;O=D

"fixed in 6.12.41"

http://boinc.berkeley.edu/trac/ticket/1144#comment:1

http://boinc.berkeley.edu/dl/?C=M;O=D

Hi Kashi,

Thanks so much for ferreting this out! Looks like this fits my situation to a tee...Mac OSX 10.6.8 BOINC 6.12.35...Bingo!

Ticket #1144 (closed Defect: fixed)

Opened 5 months ago

Last modified 5 months ago
6.12.35 on OS X 10.6.8 is leaking file descriptors
Reported by: scasady Owned by:
Priority: Undetermined Milestone: Undetermined
Component: Undetermined Version: 6.12.35
Keywords: Cc:
Description

The file descriptor acquired to open a slot directory is not always closed. They slowly accumulate until the boinc client exits or in some cases hangs.
Change History
comment:1 Changed 5 months ago by scasady

Status changed from new to closed
Resolution set to fixed

fixed in 6.12.41

This looks to be the solution for both this problem (4-CPU hang/dnssd_clientstub deliver_request: socketpair failed 24 (Too many open files) and for the crash/auto-reboot issues I was encountering here (Second Issue: unattended spontaneous BOINC Manager restart))...

(Waiting on GPU) Returns!...Repeatable on demand!:
http://boinc.berkeley.edu/dev/forum_thread.php?id=7212&nowrap=true#42278

...and here (BOINC Manager Spontaneous Restart)...

Long Post - BM Spontaneous Restart:
http://boinc.berkeley.edu/dev/forum_thread.php?id=7237#42284

...and both here (auto-reboot about a minute later and here (WU freeze while forcing 4 CPU's to run continuously)) from the same thread...

N-Body's Trashing Task & Project Queues...:
http://milkyway.cs.rpi.edu/milkyway/forum_thread.php?id=2718&nowrap=true#52142

N-Body's Trashing Task & Project Queues...:
http://milkyway.cs.rpi.edu/milkyway/forum_thread.php?id=2718&nowrap=true#52146

...and, of course, the original thread...

WU Freezes BOINC Manager...:
http://milkyway.cs.rpi.edu/milkyway/forum_thread.php?id=2639

...super news! You're my superhero for the rest of the week! :) (Now I'll have to go find something else to fret about!)

Some side notes: It was getting to the point of predictability with this issue, leave MWAH/BOINC running for extended periods and it will inevitably crash from a socketpair too many open files buildup. Also, I began to notice that just prior to the crash the BOINC Manager Event Log window (which I was leaving open all the time) would momentarily go blank whenever I would open up the OSX System Preferences panels in the Apple Menu (to change my Display Sleep settings from 1–15 minutes or back)...I knew the next MWAH WU in the queue would trigger the glitch...it was something like to an idiot light going off telling me that the MWAH/BOINC socketpair open files was full and wasn't gonna take it anymore!

Matt A. should be grateful here, as well...I was going to offer to send him the contents of the...

sched_request_milkyway.cs.rpi.edu_milkyway.xml
stdoutdae.txt
job_log_milkyway.cs.rpi.edu_milkyway.txt
client_state.xml
client_state_prev.xml
statistics_milkyway.cs.rpi.edu_milkyway.xml
daily_xfer_history.xml
sched_reply_milkyway.cs.rpi.edu_milkyway.xml

...files which all show final "modified" status at the moment of the crash! (Phew!) Of course if he still wants them I have them put aside in a MWAH/BOINC forensics folder!

And, finally, for the time being I'll just clear MWAH/BOINC with a daily restart to keep the issue at bay until I'm ready to devote some time to chasing BOINC v.7 glitches!

Thanks so much Kashi and Matt for your interest in this issue!
:)


P.S. Added linkbacks to here at those other threads in case anyone else has been running into this problem!

....And, finally, for the time being I'll just clear MWAH/BOINC with a daily restart to keep the issue at bay until I'm ready to devote some time to chasing BOINC v.7 glitches!....

Not going to try 6.12.41 or 6.12.43 from my second link? Perhaps the best option if you don't mind restarting every day. Who knows if those versions have something else that could cause a problem.

I would never have found the reference to the fix if you hadn't found the first ticket reference from scasady..

....And, finally, for the time being I'll just clear MWAH/BOINC with a daily restart to keep the issue at bay until I'm ready to devote some time to chasing BOINC v.7 glitches!....

Not going to try 6.12.41 or 6.12.43 from my second link? Perhaps the best option if you don't mind restarting every day. Who knows if those versions have something else that could cause a problem.

I would never have found the reference to the fix if you hadn't found the first ticket reference from scasady..

You'll forgive my ignorance here, but I thought 6.12.41/43 were actually just preproduction beta v.7's?

As for finding the scasady ticket, well, coffee before bed time will do that for you, too. :)

I think 6.12.43 was the last of the older kind similar to the current release version. As far as I know the new 7.0.xx type with the new scheduling code started at 6.13.0.

I think 6.12.43 was the last of the older kind similar to the current release version. As far as I know the new 7.0.xx type with the new scheduling code started at 6.13.0.

My confusion, yes, it looks like 6.12.43 was the last of the 6.12's...

Development BOINC 6.12.43 change log:
http://setiathome.berkeley.edu/forum_thread.php?id=64241

...I was concerned that BOINC isn't pushing that release to newcomers on its OSX download page...

BOINC: compute for science:
http://boinc.berkeley.edu/download.php

...or recommending it on its client software page...

Download BOINC client software:
http://boinc.berkeley.edu/download_all.php

...I'm the jumpy sort when it comes to installing possibly unstable software (though you'd think that with some of the issues I've had 6.12.35 would have gotten me over that!...sorry...) though I feel good about .43. I'll try to muster up some courage and consider downloading it this weekend when I've got some time to play with it.

:)

Being careful about development BOINC versions is a healthy attitude, I understand your reluctance. I have used two of the 7.0.xx versions due to POEM OpenCL requirements so am a bit more comfortable now about using development versions to overcome specific problems. The very unstable versions with severe problems are usually quickly removed from the list of downloadable versions.

MB Atlanos is using 6.10.43 on a number of different projects on a Core2 Duo with Darwin 10.8.0. Also Cap is using 6.12.41 on a 24 core Xeon X5650 with Darwin 10.8.0.

Just because it doesn't become the recommended version doesn't necessarily mean a development version has problems. It can be that the improvements/changes are only minor or only needed by a relatively small number of users, so the recommended version is not changed. If the recommended version is changed to suit a very small number of users then the number of problems caused by a large number of people updating to the new recommended BOINC version outweighs the benefit of the fixes for the few. This is possibly why the recommended version is not changed very often.

The last few 6.12.xx development versions had more limited testing than usual as resources were moved to developing 6.13.xx versions. Many of the changes in 6.12.xx versions after 6.12.36 related to virtual machine issues related to Test for Theory.

Being careful about development BOINC versions is a healthy attitude, I understand your reluctance. I have used two of the 7.0.xx versions due to POEM OpenCL requirements so am a bit more comfortable now about using development versions to overcome specific problems. The very unstable versions with severe problems are usually quickly removed from the list of downloadable versions.

MB Atlanos is using 6.10.43 on a number of different projects on a Core2 Duo with Darwin 10.8.0. Also Cap is using 6.12.41 on a 24 core Xeon X5650 with Darwin 10.8.0.

Just because it doesn't become the recommended version doesn't necessarily mean a development version has problems. It can be that the improvements/changes are only minor or only needed by a relatively small number of users, so the recommended version is not changed. If the recommended version is changed to suit a very small number of users then the number of problems caused by a large number of people updating to the new recommended BOINC version outweighs the benefit of the fixes for the few. This is possibly why the recommended version is not changed very often.

The last few 6.12.xx development versions had more limited testing than usual as resources were moved to developing 6.13.xx versions. Many of the changes in 6.12.xx versions after 6.12.36 related to virtual machine issues related to Test for Theory.

Hi Kashi,

I decided to get a jump on the weekend and went ahead and updated the iMac (it's the faster cruncher of my two machines) to 6.12.43 (after doing a backup of the BOINC User Data folder...just in case). Smooth as silk. I'll be letting it run MWAH flat-out with no interruptions for the next few days to see how it behaves. I'll be sure to post back any successes or failures...ha! Thanks for the encouragement on this!

On a side note...how did you navigate FreeDC to find the view for MB Atlanos and Cap that you provided?, I can't seem to locate that stat view for my CPUs on their site!

Thanks so much for all your time and efforts on this!
:)

Hope 6.12.43 works OK for you.

To find the details of your hosts.

From project website: Your account>Computers on this account>click Free-DC(or BOINCstats) logo under first column, Computer ID.

From Free-DC website: On your "User by CPID Stats page" click your score for a current project. Hosts will be shown on the page you go to, click ID of host you are interested in. Only works if hosts aren't hidden.

Changes in host ID for the same computer means not all projects always show under the one host ID, especially projects you have not recently contributed to, if you are no longer attached.

Hope 6.12.43 works OK for you.

To find the details of your hosts.

From project website: Your account>Computers on this account>click Free-DC(or BOINCstats) logo under first column, Computer ID.

From Free-DC website: On your "User by CPID Stats page" click your score for a current project. Hosts will be shown on the page you go to, click ID of host you are interested in. Only works if hosts aren't hidden.

Changes in host ID for the same computer means not all projects always show under the one host ID, especially projects you have not recently contributed to, if you are no longer attached.

Thanks for the well-wishes!

Your instructions for finding that particular stats page was spot-on! Thanks!

:)

...and, the laptop just performed its final 4-cpu crash so I decided to upgrade that to 6.12.43 as well...here's to smooth sailing!...(jinx?!)...

:)