Is there anyone who has resources on clustering? Can Boinc computations be run on clusters? Do they need individual instances on virtual machines or can they send bits of work to each node? I know very little about cluster computing but want to set one up to learn more. I have hardware Intel Xeon E5-2699v3 and Nvidia P100s). If anyone could be so kind as to point me to a starting point here I would appreciate it.

Can Boinc computations be run on clusters? Do they need individual instances on virtual machines or can they send bits of work to each node?

No, BOINC does not support that, it's made to run on PCs. Of course it also runs on server hardware, but not on clusters. If you have the hardware, run it on bare metal, no VMs or other crap in between doing nothing else than slowing it down.

---

Some years back people did try to run SETI@Home on clusters of Raspberry Pi and similar low cost SBC. I can't recall how successful or otherwise they were, but BOINC does run on Raspberry Pi when running Rasperian(sp?). If you search the S@H forum you should come across a few threads discussing clusters (be warned, the S@H forum are incredibly slow just now, and you may/will hit time-outs.....)
A "starter for ten" thread: https://setiathome.berkeley.edu/forum_thread.php?id=69174#1276637

Getting BOINC to run on a cluster is one thing, finding projects that have applications that will run on it is another.

There are some people who run BOINC on computers with multiple Intel Xeon to some success.

The NVIDIA P100 family is quite some beast for number crunching, but I'm not too sure how many projects have applications that support it.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

No, BOINC does not support that, it's made to run on PCs. Of course it also runs on server hardware

When initially developed BOINC was intended to run on PCs and a limited number of other platforms. Over the years the range of devices that could run BOINC increased, using either "official" or "unofficial" versions of BOINC (remember BOINC is Open Source) Apart from the obvious SBC like Raspberry Pi people have successfully run BOINC on Amazon FireStick, managed switches and routers, small NAS devices (not the sort that are used in server farms, but the SOHO types). Then there a few real outliers - I managed to get BOINC running on an Intel i960 development board, and ported one of the SETI@Home apps to run on it, on a TMS320 development board, but never got the S@H app to any degree of stability. At least one nutter got some time on an IBM 360 and ported BOINC and a couple of different project apps to run on it!
I dare say some have tried on older 8/16 bit hardware but I can't think of any just now, but if they did I would imagine they were among the slowest hosts ever.

Basically, BOINC can be ported to just about any device that has a (recentish - say post 2000) CPU, but the real problem is getting projects to spend time developing/porting their applications to run on that hardware.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

BOINC does run on Raspberry Pi when running Rasperian(sp?)

Yes, many people use them, but they run them as farm, not cluster, simply because it doesn't make any sense. BOINC applications are designed to run on single devices, most on a single CPU core. So even if you would get BOINC to run somehow on a cluster, all you get as a "reward" is a lot less work done with the available hardware.

Basically, BOINC can be ported to just about any device that has a (recentish - say post 2000) CPU, but the real problem is getting projects to spend time developing/porting their applications to run on that hardware.

Well, here we don't have anything special, that requires any porting, just a single (?) x64 system with some CUDA cards (Tesla Pascal). I've seen some P100 and similar cards on Einstein, they run the stock applications without any issues (usually on Linux). The most adventurous thing, that might be necessary (or at least useful) here, is either running more than one BOINC client, one on each CPU or at least making sure, that applications are not moved between the CPUs. But aside of that, it's just a simple computer, install OS, install BOINC, check the GPU drivers, attach to projects of your choice and you should be done.

---

So even if you would get BOINC to run somehow on a cluster, all you get as a "reward" is a lot less work done with the available hardware.

For some the reward is not in accumulating worthless credits but doing the apparently impossible. Do not denigrate them as it is through their apparently pointless activities many years ago we can now have multi-threaded applications, hybrid applications using mixed CPU & GPU systems.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

It is possible, but it depends on the nature of your cluster and how you want to operate it…

The standard out-the-box BOINC client doesn’t have any support for clustering in general. If a cluster is homogeneous and the OS can present all the CPU cores as though they belong to a single host and take care of distributing tasks among nodes behind the scenes, it might just work.

If your cluster is “just a bunch of PCs” then you could simply run BOINC on each node individually – but there’ll be no coordination, and the standard GUI is not practical for managing large herds.

Beyond that it gets rather more involved. As BOINC is open-source it is possible to customise the client to manage the clustering aspects while appearing to project servers as a single host. This has definitely been done: a user at Rosetta@home ran a cluster through a small number of clients, leading to each host achieving “impossible” RAC levels. But even that was a relatively simple setup in the sense that it was homogeneous and using commodity hardware and OS. If you have a heterogeneous cluster you might need to present one host per platform type to get projects to send the right application executables. If the nodes use “exotic” hardware or software you will be limited to projects whose applications are also open-source so you can build them yourself and run them as anonymous platform.

If your cluster is “just a bunch of PCs” then you could simply run BOINC on each node individually – but there’ll be no coordination, and the standard GUI is not practical for managing large herds.

CrawfordCluster didn't specify it exactly, but I think we are talking about one single computer. Even if we were talking about few computers (I don't think we are talking about hundreds of them), you can easily manage them with BOINC Manager. Once you get them running, there's not much to do. To my Android device I don't log in even once a month, it has a very reliable main project, it has a backup project, that's all you need for crunching 24/7. OK, I also see it's hours on WUprop. If you want it more comfortable, use BoincTasks.

---

If YOU want to run BOINC Tasks then run it, but don't expect everyone else to do so - some do explore "if it can be done, then why not do it?". We are talking about people who want to run a compute-cluster rather than using apps like BOINC Tasks.
It is possible to run BOINC on a cluster, and there are probably quite a few running on different projects today - in the latter days of SETI there certainly were, and as has been pointed out to you there was at least one running Rosetta (I would even suggest that there may be more than one lurking in the depths of MilkyWay).

So, back on subject, should the OP desire to run a cluster based on the hardware to hand it would be worth investigating one of the branches of Beowulf Cluster technology, but be prepared for some work to get both BOINC and the project's apps ported]

(This talk of porting brings up a fairly common problem, but one that is not for this thread)

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

The reason I was asking about if BOINC could be run on a cluster would be to find relevant applications that can be used to demonstrate the project's capabilities. Like I said before, I don't know a ton about setting up a cluster and was hoping to find a one-stop-shop or someone to help guide me through learning more about them. The hardware specifically two towers running 2 p100s each and a Xeon 2699 v3 each. I would like to learn to use a software like Kubernetes or Proxmox to scale applications across the hardware. I just selected BOINC because it is for a good cause.

OK, so a Beowulf cluster wouldn't really work with just two compute nodes. But from what I see either Kubernetes or Proxmox would.
BOINC will scale readily across your two computers, and will probably not need to be ported as both the hardware and operating systems are well known.
Hardware? So long as one of the pair has two Ethernet connections no problem. To keep things simple, a direct connection between the two computers, say 1 or 10GB and one line out to the big bad world.
Software? Apart from your selected clustering software BOINC, on the one with the connection to the outside. As there is, in theory, no limit to the number of cores or GPUs the pair should appear to be a single computer with 144 cores, 4 GPUs and a massive amount of memory.
Project applications? They should run very well, indeed blindingly quickly if you launch all 144 core at a task! The GPUs will probably not cluster, but you will have four, each should be capable of running multiple tasks simultaneously.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

They should run very well, indeed blindingly quickly if you launch all 144 core at a task!

Considering the penalty of running a single task on two CCDs of the same CPU, I'd not recommend running a single task on two CPUs, in particular if those two CPUs are sitting in different computers. Running one BOINC client for both computers might indeed be quite interesting experiment to see how close they can get to that Threadripper in the top of the top list, but I expect a lot better results when running more concurrent tasks, like at least 4 per CPU. Simply consider "pinning an application to specific core(s) of a cluster" as something additional to learn about.

---

You've never tried cluster computing and so your view is totally invalid. Let's see what happens, you may actually learn something rather than just spouting your pet theory.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Isn't the whole idea of BOINC to avoid projects needing a cluster / supercomputer to do their computing? The huge computing job is broken up into bite size individual chunks and sent out to various "normal" computers. Those could be in the Projects own server racks, or scattered across the planet. Doesn't matter if it's a quad Xeon with a couple of P1000s attached, or a Raspberry Pi, it gets given the tasks, and returns the results, in an hour, or a day, or a week.

Trying to then gather up a bunch of those to compute on a unified cluster of machines just doesn't seem very efficient? Each box running with a central console for management should be able to do more work, with no exotic interconnections needed, just basic ethernet.

As mentioned Bare Metal is the way to go with BOINC. Proxmox is simple to setup and is based on Debian so you can install BOINC on top of Proxmox

If you want to get complicated then you could use OpenShift.

Realistically setup Proxmox and play with VM's and LXC

Trying to then gather up a bunch of those to compute on a unified cluster of machines just doesn't seem very efficient?

No, it's not efficient because of the overhead, which might be acceptable when you run stuff, that simply can't be completed on a single machine, but for BOINC it's just waste of ressources. I mean, considering that for CPUs with more than one CCD it's recommended to make sure, that project applications do not swap between them or that a MT application runs on more than one of them, how likely does it make sense to run a MT application on few CPUs sitting in different computers? It's going to be way worse than trying to run it on both CPUs of a dual-CPU system, even that is a bad idea and they sit at least on the same motherboard.

Each box running with a central console for management should be able to do more work, with no exotic interconnections needed, just basic ethernet.

I think if I was going to set up a dedicated crunching machine, I'd try this. Or something similar. When setting up a dedicated crunchbox, perfection is not reached when you can't add anything more, it's when you can't remove anything more (without loosing performance). So simple OS on bare metal, no VMs or any other crap slowing everything down.

---