Re: [9fans] xcpu note
| Probably apples and oranges, but Jim Kent wrote a job scheduler for his | kilocluster that nicely handled about 1M jobs in six hours. It's the | standard thing for whole genome sequence alignments at ucsc. | | I think that's neat, I would like to learn more. Was this scheduler for | an arbitrary job mix, or specialized to that app? Well, it was designed to do what we needed and no more, but it's still pretty general. The input is a file of commands, and it runs them all until they are all done (with a way to retry the ones that failed.) http://www.cse.ucsc.edu/~kent/ http://www.soe.ucsc.edu/~donnak/eng/parasol.htm
Re: [9fans] httpd
What was (is?) the vision for httpd? what was (is?) the vision for the question?
[9fans] drawterm
I made some changes to drawterm yesterday that fix some of the snarf problems and might even fix the random crashes that some have seen (and others have not). I also replaced my pipe-based pseudo-locks with real pthreads locks, so if it seemed slow before, maybe it will be better now. Best of all, I have a test case that reliably crashes drawterm. If you connect to somewhere and then run fcp /mnt/term/some/big/file /dev/null then eventually drawterm or the remote kernel or both get confused about what's going on on the wire and someone hangs up. It never gets through more than a couple megabytes before this happens. Drawterm gets a read error from bad digest, and the kernel sometimes sees unexpected rpc messages, usually Rreads to Treads that it is trying to flush (but the Rflush is sent only after the Rread, so the Rread is legitimate). I'm out of ideas at the moment. If someone else wants to debug, please do so. cvs -d :pserver:[EMAIL PROTECTED]:/cvs co drawterm Russ
[9fans] xcpu update
I was afraid I would get in trouble with you for all those uncontrolled debug prints, so have fixed all them so that they use the debug() function. So -D 0 will kill all debug prints, and you can turn them on as needed. xcpusrv is much quieter now. I also fixed the docs. Now if I could figure out this weird seeming race condition that locks up the main thread ... :-) ron
Re: [9fans] httpd
What was (is?) the vision for httpd? what was (is?) the vision for the question? generally, are/were there any plans for things like j2ee connector to access off world web services? specifically, how do i access a servlet (paypal) on a j2ee application server (tomcat) from plan9 httpd? a portion of the http request needs to be handled on the plan9 server so moving everything off of plan9 will still require apache/plan9 connectivity and auth considerations, etc.
Re: [9fans] xcpu note
David Leimbach wrote: Clustermatic is pretty cool, I think it's what was installed on one of the other clusters I used at LANL as a contractor at the time. I recall a companion tool for bproc to request nodes, sort of an ad-hoc scheduler. I had to integrate support for this in our MPI's start up that I was testing on that machine. the simple scheduler, bjs, was written by erik hendriks (now at Google, sigh) and was rock-solid. It ran on one cluster, unattended, scheduling 128 2-cpu nodes with a very diverse job mix, for one year. It was a great piece of software. It was far faster, and far more reliable, than any scheduler we have ever seen, then or now. In one test, we ran about 20,000 jobs through it on about an hour, on a 1024-node cluster, just to test. Note that it could probably have scheduled a lot more jobs, but the run-time of the job was non-zero. No other scheduler we have used comes close to this kind of performance. Scheduler overhead was basically insignificant. Yeah, when I came to the lab last it was a surprise to find out that I not only had to support bproc but bjs though. Luckilly it took about 10 minutes to figure it out and add support to our mpirun startup script. It was pretty neat. I'm curious to see how this all fits together with xcpu, if there is such a resource allocation setup needed etc. we're going to take bjs and have it schedule nodes to give to users. Note one thing we are going to do with xcpu: attach nodes to a user's desktop machine, rather than make users log in to the cluster. So users will get interactive clusters that look like they own them. This will, we hope, kill batch mode. Plan 9 ideas make this possible. It's going to be a big change, one we hope users will like. Hmm, planning to create a multi-hosted xcpu resource all bound to the user's namespace? Or one host per set of files? Is there a way to launch multiple jobs in one shot ala MPI startup this way that's easy? If you look at how most clusters are used today, they closely resemble the batch world of the 1960s. It is actually kind of shocking. I downloaded a JCL manual a year or two ago, and compared what JCL did to what people wanted batch schedulers for clusters to do, and the correspondance was a little depressing. The Data General ad said it best: Batch is a bitch. Yeah, I've been comparing them to punch card systems for a while now. Some are even almost the same size as those old machines now that we've stacked them up. MPI jobs have turned modern machines into huge monoliths that basically throw out the advantages of a multi-user system. In fact having worked with CPlant for a while with Ron Brightwell over at SNL, they had a design optimized for one process per machine. One CPU [no SMP hardware contention], Myrinet with Portals for RDMA and OS bypass reasons [low overheads], no threads [though I was somewhat taunted with them at one point], and this Yod and Yod2 scheduler for job startup. It was very unique, and very interesting to work on and not a lot of fun to debug running code on. :) The closest thing I've seen to this kind of design in production has to be Blue Gene [which is a much different architecture of course but similar in that it is very custom designed for a few purposes]. Oh yeah, if anyone has a copy of that ad (Google does not), i'd like it in .pdf :-) It appeared in the late 70s IIRC. ron p.s. go ahead, google JCL, and you can find very recent manuals on how to use it. I will be happy to post the JCL for sort + copy if anyone wants to see it. Please god no!!! :) Dave
Re: [9fans] xcpu note
David Leimbach wrote: Clustermatic is pretty cool, I think it's what was installed on one of the other clusters I used at LANL as a contractor at the time. I recall a companion tool for bproc to request nodes, sort of an ad-hoc scheduler. I had to integrate support for this in our MPI's start up that I was testing on that machine. the simple scheduler, bjs, was written by erik hendriks (now at Google, sigh) and was rock-solid. It ran on one cluster, unattended, scheduling 128 2-cpu nodes with a very diverse job mix, for one year. It was a great piece of software. It was far faster, and far more reliable, than any scheduler we have ever seen, then or now. In one test, we ran about 20,000 jobs through it on about an hour, on a 1024-node cluster, just to test. Note that it could probably have scheduled a lot more jobs, but the run-time of the job was non-zero. No other scheduler we have used comes close to this kind of performance. Scheduler overhead was basically insignificant. Yeah, when I came to the lab last it was a surprise to find out that I not only had to support bproc but bjs though. Luckilly it took about 10 minutes to figure it out and add support to our mpirun startup script. It was pretty neat. I'm curious to see how this all fits together with xcpu, if there is such a resource allocation setup needed etc. we're going to take bjs and have it schedule nodes to give to users. Note one thing we are going to do with xcpu: attach nodes to a user's desktop machine, rather than make users log in to the cluster. So users will get interactive clusters that look like they own them. This will, we hope, kill batch mode. Plan 9 ideas make this possible. It's going to be a big change, one we hope users will like. Hmm, planning to create a multi-hosted xcpu resource all bound to the user's namespace? Or one host per set of files? Is there a way to launch multiple jobs in one shot ala MPI startup this way that's easy? If you look at how most clusters are used today, they closely resemble the batch world of the 1960s. It is actually kind of shocking. I downloaded a JCL manual a year or two ago, and compared what JCL did to what people wanted batch schedulers for clusters to do, and the correspondance was a little depressing. The Data General ad said it best: Batch is a bitch. Yeah, I've been comparing them to punch card systems for a while now. Some are even almost the same size as those old machines now that we've stacked them up. MPI jobs have turned modern machines into huge monoliths that basically throw out the advantages of a multi-user system. In fact having worked with CPlant for a while with Ron Brightwell over at SNL, they had a design optimized for one process per machine. One CPU [no SMP hardware contention], Myrinet with Portals for RDMA and OS bypass reasons [low overheads], no threads [though I was somewhat taunted with them at one point], and this Yod and Yod2 scheduler for job startup. It was very unique, and very interesting to work on and not a lot of fun to debug running code on. :) The closest thing I've seen to this kind of design in production has to be Blue Gene [which is a much different architecture of course but similar in that it is very custom designed for a few purposes]. Oh yeah, if anyone has a copy of that ad (Google does not), i'd like it in .pdf :-) It appeared in the late 70s IIRC. ron p.s. go ahead, google JCL, and you can find very recent manuals on how to use it. I will be happy to post the JCL for sort + copy if anyone wants to see it. Please god no!!! :) Dave
Re: [9fans] xcpu note
David Leimbach wrote: Clustermatic is pretty cool, I think it's what was installed on one of the other clusters I used at LANL as a contractor at the time. I recall a companion tool for bproc to request nodes, sort of an ad-hoc scheduler. I had to integrate support for this in our MPI's start up that I was testing on that machine. the simple scheduler, bjs, was written by erik hendriks (now at Google, sigh) and was rock-solid. It ran on one cluster, unattended, scheduling 128 2-cpu nodes with a very diverse job mix, for one year. It was a great piece of software. It was far faster, and far more reliable, than any scheduler we have ever seen, then or now. In one test, we ran about 20,000 jobs through it on about an hour, on a 1024-node cluster, just to test. Note that it could probably have scheduled a lot more jobs, but the run-time of the job was non-zero. No other scheduler we have used comes close to this kind of performance. Scheduler overhead was basically insignificant. Yeah, when I came to the lab last it was a surprise to find out that I not only had to support bproc but bjs though. Luckilly it took about 10 minutes to figure it out and add support to our mpirun startup script. It was pretty neat. I'm curious to see how this all fits together with xcpu, if there is such a resource allocation setup needed etc. we're going to take bjs and have it schedule nodes to give to users. Note one thing we are going to do with xcpu: attach nodes to a user's desktop machine, rather than make users log in to the cluster. So users will get interactive clusters that look like they own them. This will, we hope, kill batch mode. Plan 9 ideas make this possible. It's going to be a big change, one we hope users will like. Hmm, planning to create a multi-hosted xcpu resource all bound to the user's namespace? Or one host per set of files? Is there a way to launch multiple jobs in one shot ala MPI startup this way that's easy? If you look at how most clusters are used today, they closely resemble the batch world of the 1960s. It is actually kind of shocking. I downloaded a JCL manual a year or two ago, and compared what JCL did to what people wanted batch schedulers for clusters to do, and the correspondance was a little depressing. The Data General ad said it best: Batch is a bitch. Yeah, I've been comparing them to punch card systems for a while now. Some are even almost the same size as those old machines now that we've stacked them up. MPI jobs have turned modern machines into huge monoliths that basically throw out the advantages of a multi-user system. In fact having worked with CPlant for a while with Ron Brightwell over at SNL, they had a design optimized for one process per machine. One CPU [no SMP hardware contention], Myrinet with Portals for RDMA and OS bypass reasons [low overheads], no threads [though I was somewhat taunted with them at one point], and this Yod and Yod2 scheduler for job startup. It was very unique, and very interesting to work on and not a lot of fun to debug running code on. :) The closest thing I've seen to this kind of design in production has to be Blue Gene [which is a much different architecture of course but similar in that it is very custom designed for a few purposes]. Oh yeah, if anyone has a copy of that ad (Google does not), i'd like it in .pdf :-) It appeared in the late 70s IIRC. ron p.s. go ahead, google JCL, and you can find very recent manuals on how to use it. I will be happy to post the JCL for sort + copy if anyone wants to see it. Please god no!!! :) Dave
[9fans] triple post?
My mouse bugged out a bit there... strange that it managed to cause Acme mail to send 3 emails by causing click events. Maybe USB Legacy Mode isn't all it's cracked up to be? Dave
Re: [9fans] httpd
On Tue, Oct 18, 2005 at 10:24:13AM -0700, Skip Tavakkolian wrote: What was (is?) the vision for httpd? what was (is?) the vision for the question? generally, are/were there any plans for things like j2ee connector to access off world web services? specifically, how do i access a servlet (paypal) on a j2ee application server (tomcat) from plan9 httpd? a portion of the http request needs to be handled on the plan9 server so moving everything off of plan9 will still require apache/plan9 connectivity and auth considerations, etc. If you are going to have to use java you could as well forget Plan 9. (In theory you could make httpd speak whatever retarded protocol tomcat uses this week, back in the day there were three or four, each with it's own set of bugs and flaws, and requiring a different apache module, but really, what does that earn you aside from extra pain?) What I would like is something 'servlet-like'(the servlets API itself was not all that bad back when I had to deal with it, but that is ignoring JSP, J2EE and all that crap), but in Limbo. Last I heard Inferno's httpd was in rather bad shape(HTTP 1.0 only?) but I think caerwyn was working on bringing it back to the level of Plan 9's httpd. I think someone mentioned that brucee had something in OzInferno for web app development, but you probably know more about that than me(I would love to hear more ;)). uriel
Re: [9fans] SuperComputing in Seattle.
On Mon, Oct 17, 2005 at 08:35:34AM -0600, Ronald G Minnich wrote: Roman Shaposhnick wrote: Guys, I know that at least some of you use Plan9 for the cluster work and such, hence SuperComputing expo in Seattle might be of an interest to you. I guess depending on how many of us will be there (I, for one, will be) it could be a nice opportunity for Plan 9 beer bash or something. What do you think ? Thanks, Roman. I will be there with a booth. We may have a small plan 9 cluster there, in the lanl booth. andrey will also be in the lanl booth. I guess we'll be happy to see any of you who can make it. I can set up a Plan 9 bof, although beer bash has more appeal. Sounds like a plan, then! At least now we have three out of some pinned down to booths. Shouldn't be too difficult to find each other. Thanks, Roman.
[9fans] quick question on threads on p9p
I see this: 8894 pts/3S 0:00 ./o.xcpusrv -9 -s tcp!*!20001 -D 0 8898 pts/3S 0:00 ./o.xcpusrv -9 -s tcp!*!20001 -D 0 8900 pts/3S 0:00 9pserve -u tcp!*!20001 8902 pts/3S 0:00 9pserve -u tcp!*!20001 8904 pts/3S 0:00 ./o.xcpusrv -9 -s tcp!*!20001 -D 0 8906 pts/3S 0:00 9pserve -u tcp!*!20001 8908 pts/3S 0:00 9pserve -u tcp!*!20001 8910 pts/3S 0:00 9pserve -u tcp!*!20001 8919 pts/3R 0:00 ps ax [EMAIL PROTECTED]:~/src/xcpu/xcpu acid -l thread 8894 corpid 8894: text /home/rminnich/src/xcpu/xcpu/o.xcpusrv /lib/i686/libm.so.6: linux 386 shared object /lib/libutil.so.1: linux 386 shared object /lib/i686/libc.so.6: linux 386 shared object /lib/ld-linux.so.2: linux 386 shared object /usr/local/plan9/acid/port /usr/local/plan9/acid/thread /usr/local/plan9/acid/386 acid; threads() p=(Proc)0x8066050 // pthread 0x22be pid 8894 Sched p=(Proc)0x8066468 // pthread 0x22c2 pid 8898 Sched t=(_Thread)0x8066728 // Sleeping /home/rminnich/src/xcpu/xcpu/xcpusrv.c:1177 chanalt [fsthread] p=(Proc)0x8078bc0 // pthread 0x22c8 pid 8904 Running t=(_Thread)0x8078e80 // Running /usr/local/plan9/src/lib9p/srv.c:62 read9pmsg acid; I'm seeing a bit more entries in /proc than I see in the threads() display in acid. What am I missing here? thanks ron
Re: [9fans] quick question on threads on p9p
[EMAIL PROTECTED]:~/src/xcpu/xcpu acid -l thread 8894 corpid 8894: text /home/rminnich/src/xcpu/xcpu/o.xcpusrv /lib/i686/libm.so.6: linux 386 shared object /lib/libutil.so.1: linux 386 shared object /lib/i686/libc.so.6: linux 386 shared object /lib/ld-linux.so.2: linux 386 shared object /usr/local/plan9/acid/port /usr/local/plan9/acid/thread /usr/local/plan9/acid/386 acid; threads() p=(Proc)0x8066050 // pthread 0x22be pid 8894 Sched p=(Proc)0x8066468 // pthread 0x22c2 pid 8898 Sched t=(_Thread)0x8066728 // Sleeping /home/rminnich/src/xcpu/xcpu/xcpusrv.c:1177 chanalt [fsthread] p=(Proc)0x8078bc0 // pthread 0x22c8 pid 8904 Running t=(_Thread)0x8078e80 // Running /usr/local/plan9/src/lib9p/srv.c:62 read9pmsg acid; It's weird that 8894 has no threads. I'm seeing a bit more entries in /proc than I see in the threads() display in acid. What am I missing here? Those three are the only ones you listed in your ps output. What do you see in /proc? One could also attach with gdb and run thread apply all where to get an alternate view of the current thread in each proc. Useful for sanity checking, but little else. stacks() in acid will give you the stack of every thread, current or not. Russ
Re: [9fans] quick question on threads on p9p
thanks, the stacks() is very helpful. ron
