[gentoo-user] Bootstrapping a Heterogeneous compile farm?
Ok... I'm starting a rather ambitious project, and perhaps it's too ambitious! I'm having a hard time putting it into non-confusing words, for some reason, I can picture it in my head, but describing it to a mailing list is harder than I thought! What I'd like to setup is a compile farm/cluster, using two of my existing machines. The number will eventually grow, but my wife won't let me bring my old computers out of storage in this tiny apartment. :) I'd like to be able to compile customized Gentoo systems (perhaps for use on LiveCD's, or available for Etherboot), but build them all from the same set of sources. I have a computer setup running Gentoo (very well except for some minor problems like the old mplayer issue that isn't fixed yet, and some perl wierdness that is probably due to the recent changes), and I want to use it to bootstrap the rest of my machines. The problem is, none of the machines are the same, the network is 100% heterogeneous, but all are X86 based so far, so it's not going to be cross- compiling per-se (although that functionality would be rather nice). The machine I'm running now is running an OpenMosix kernel. I'm assuming it's working well, but with only one machine in my cluster, it's hard to know for sure. What I'd like to do is setup a LiveCD for my wife's computer, so that when it's booted off that CD it becomes part of my cluster, and it's power can be used to compile everything (well, except for those things that don't compile across the network properly, like X for example). Is what I want to do possible, and if so, can someone give me some pointers or help on where to start? If possible, I'd like to minimize the amount of duplication, but I'm also not sure how possible that is. I'm pretty sure I could do something like this using chroot and just having separate directories setup, complete, for each machine. The problem I see with that, besides the huge amount of duplication, is that the directories will probably be too big to fit onto CD, and will have a lot of unnecessary stuff for running. I hope this question wasn't too obscure, if clarification is necessary please feel free to ask (either through the ML or private mail). Thanks for everyone's help! Adam -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Bootstrapping a Heterogeneous compile farm?
On 8/8/03 3:02 am, Adam Scriven [EMAIL PROTECTED] wrote: Well, I'm kinda wondering about distcc and ccache. I've got both running now over the cluster, but I'm wondering if that's overkill, or if they will interfere with the cluster's functioning. Well, I thought that distcc was unnecessary in a clustered environment, and that you just compiled with a higher -j number. But according to this HOWTO http://howto.ipng.be/openMosix-HOWTO/c1377.html I was incorrect. So it seems like you might well need to use distcc. Certainly, I admire you for your project - do let us all know how it goes. Stroller. -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Bootstrapping a Heterogeneous compile farm?
On Fri, Aug 08, 2003 at 03:10:16AM +0200, Renat Golubchyk wrote: On Friday 08 August 2003 02:00, Adam Scriven wrote: I'm not sure whether I understand it right, but if you just want to build a compile farm you could use ClusterKnoppix (Knoppix with OpenMosix kernel). http://bofh.be/clusterknoppix/ Yeah, I could use clusterknoppix, but some of the hardware I'm thinking about is very old and not supported. It seems like a bit of a bloated solution, when a lot of the computers will be headless. I may use it as a temporary solution, just to test the cluster setup I have to make sure it works, but I would much rather roll-my-own in the end. I don't know whether it includes distcc and ccache, but there was some talk about this on a debian mailing list. If it is not included you could add it yourself - just follow the instructions on the Knoppix website. Yeah, and Google is your friend :-) Well, I'm kinda wondering about distcc and ccache. I've got both running now over the cluster, but I'm wondering if that's overkill, or if they will interfere with the cluster's functioning. Right now, with only one node (and so no process migration), everything is working fine, and is stable as a rock. Thanks Renat! Adam -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Bootstrapping a Heterogeneous compile farm?
On Friday 08 August 2003 02:00, Adam Scriven wrote: What I'd like to setup is a compile farm/cluster, using two of my existing machines. The number will eventually grow, but my wife won't let me bring my old computers out of storage in this tiny apartment. :) I'd like to be able to compile customized Gentoo systems (perhaps for use on LiveCD's, or available for Etherboot), but build them all from the same set of sources. I have a computer setup running Gentoo (very well except for some minor problems like the old mplayer issue that isn't fixed yet, and some perl wierdness that is probably due to the recent changes), and I want to use it to bootstrap the rest of my machines. The problem is, none of the machines are the same, the network is 100% heterogeneous, but all are X86 based so far, so it's not going to be cross- compiling per-se (although that functionality would be rather nice). The machine I'm running now is running an OpenMosix kernel. I'm assuming it's working well, but with only one machine in my cluster, it's hard to know for sure. What I'd like to do is setup a LiveCD for my wife's computer, so that when it's booted off that CD it becomes part of my cluster, and it's power can be used to compile everything (well, except for those things that don't compile across the network properly, like X for example). I'm not sure whether I understand it right, but if you just want to build a compile farm you could use ClusterKnoppix (Knoppix with OpenMosix kernel). http://bofh.be/clusterknoppix/ I don't know whether it includes distcc and ccache, but there was some talk about this on a debian mailing list. If it is not included you could add it yourself - just follow the instructions on the Knoppix website. Yeah, and Google is your friend :-) Hope it helps. Cheers, Renat -- [EMAIL PROTECTED] mailing list
