-----Original Message-----
From: Beowulf [mailto:[email protected]] On Behalf Of Joe Landman
Sent: Friday, July 27, 2018 11:54 AM
To: [email protected]
Subject: Re: [Beowulf] Jupyter and EP HPC
On 07/27/2018 02:47 PM, Lux, Jim (337K) wrote:
>
> I’ve just started using Jupyter to organize my Pythonic ramblings..
>
> What would be kind of cool is to have a high level way to do some
> embarrassingly parallel python stuff, and I’m sure it’s been done, but
> my google skills appear to be lacking (for all I know there’s someone
> at JPL who is doing this, among the 6000 people doing stuff here).
>
> What I’m thinking is this:
>
> I have a high level python script that iterates through a set of data
> values for some model parameter, and farms out running the model to
> nodes on a cluster, but then gathers the results back.
>
> So, I’d have N copies of the python model script on the nodes.
>
> Almost like a pythonic version of pdsh.
>
> Yeah, I’m sure I could use lots of subprocess() and execute() stuff
> (heck, I could shell pdsh), but like with all things python, someone
> has probably already done it before and has all the nice hooks into
> the Ipython kernel.
>
I didn't do this with ipython or python ... but this was effectively the way I
parallelized NCBI BLAST in 1998-1999 or so. Wrote a perl script to parse args,
construct jobs, move data, submit/manage jobs, recover results, reassemble
output. SGI turned that into a product.
-- yes.. but I was hoping someone had done that for Jupyter..
>>> for parametervalue in parametervaluelist:
.... result = simulation(parametervalue)
Results.append(result)
_______________________________________________
Beowulf mailing list, [email protected] sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
