Re: [julia-users] Running multiple scripts in parallel Julia sessions

[Tim Holy]

pmap?

--Tim

On Saturday, January 23, 2016 07:29:07 PM Ritchie Lee wrote:
> Do you mean using @spawn, success, or just from the command prompt?
> 
> The scripts are long-running experiments where I am also tracking things
> like CPU time.  I would like them queued and run 4 at a time, i.e., if 1
> finishes ahead of others, then the next one will start running on the free
> processor.  Is there a way to schedule into separate processes?
> 
> On Saturday, January 23, 2016 at 12:00:35 PM UTC-8, Stefan Karpinski wrote:
> > Any reason not to run them all as separate processes?
> > 
> > On Fri, Jan 22, 2016 at 11:08 PM, Ritchie Lee  > 
> > > wrote:
> >> Let's say I have 10 julia scripts, scripts = ["script1.jl", "script2.jl",
> >> , "script10.jl"] and I would like to run them in parallel in separate
> >> Julia sessions, but 4 at a time (since I only have 4 cores on my
> >> machine).
> >> Is there any way to do this programmatically?
> >> 
> >> I tried doing this:
> >> 
> >> addprocs(4)
> >> pmap(include, scripts)
> >> 
> >> or
> >> 
> >> addprocs(4)
> >> @parallel for s in scripts
> >> include(s)
> >> end
> >> 
> >> However, this seems to reuse the same session, so all the global consts
> >> in the script file are colliding.  I would like to make sure that the
> >> namespaces are completely separate.
> >> 
> >> Thanks!

[julia-users] Running multiple scripts in parallel Julia sessions

[Kenta Sato]

I think GNU parallel is the best tool for that purpose. 
http://www.gnu.org/software/parallel/

You can pass -j option to control the number of maximum jobs at a time.

Re: [julia-users] Running multiple scripts in parallel Julia sessions

[Tim Holy]

You could write your own pmap-like function (see its source code) that starts 
a new worker and then shuts it down again when done.

Or, just wrap all your scripts inside modules?

module Script1

# code that was in script1.jl

end

--Tim

On Sunday, January 24, 2016 01:13:21 PM Ritchie Lee wrote:
> The scripts contain a lot of global consts and other things.  pmap seems to
> mix the namespaces for all scripts executed on the same processor.
> 
> On Sunday, January 24, 2016 at 12:06:05 AM UTC-8, Tim Holy wrote:
> > pmap?
> > 
> > --Tim
> > 
> > On Saturday, January 23, 2016 07:29:07 PM Ritchie Lee wrote:
> > > Do you mean using @spawn, success, or just from the command prompt?
> > > 
> > > The scripts are long-running experiments where I am also tracking things
> > > like CPU time.  I would like them queued and run 4 at a time, i.e., if 1
> > > finishes ahead of others, then the next one will start running on the
> > 
> > free
> > 
> > > processor.  Is there a way to schedule into separate processes?
> > > 
> > > On Saturday, January 23, 2016 at 12:00:35 PM UTC-8, Stefan Karpinski
> > 
> > wrote:
> > > > Any reason not to run them all as separate processes?
> > > > 
> > > > On Fri, Jan 22, 2016 at 11:08 PM, Ritchie Lee  > > > 
> > > > > wrote:
> > > >> Let's say I have 10 julia scripts, scripts = ["script1.jl",
> > 
> > "script2.jl",
> > 
> > > >> , "script10.jl"] and I would like to run them in parallel in
> > 
> > separate
> > 
> > > >> Julia sessions, but 4 at a time (since I only have 4 cores on my
> > > >> machine).
> > > >> Is there any way to do this programmatically?
> > > >> 
> > > >> I tried doing this:
> > > >> 
> > > >> addprocs(4)
> > > >> pmap(include, scripts)
> > > >> 
> > > >> or
> > > >> 
> > > >> addprocs(4)
> > > >> @parallel for s in scripts
> > > >> include(s)
> > > >> end
> > > >> 
> > > >> However, this seems to reuse the same session, so all the global
> > 
> > consts
> > 
> > > >> in the script file are colliding.  I would like to make sure that the
> > > >> namespaces are completely separate.
> > > >> 
> > > >> Thanks!

Re: [julia-users] Running multiple scripts in parallel Julia sessions

[Ritchie Lee]

The scripts contain a lot of global consts and other things.  pmap seems to 
mix the namespaces for all scripts executed on the same processor.


On Sunday, January 24, 2016 at 12:06:05 AM UTC-8, Tim Holy wrote:
>
> pmap? 
>
> --Tim 
>
> On Saturday, January 23, 2016 07:29:07 PM Ritchie Lee wrote: 
> > Do you mean using @spawn, success, or just from the command prompt? 
> > 
> > The scripts are long-running experiments where I am also tracking things 
> > like CPU time.  I would like them queued and run 4 at a time, i.e., if 1 
> > finishes ahead of others, then the next one will start running on the 
> free 
> > processor.  Is there a way to schedule into separate processes? 
> > 
> > On Saturday, January 23, 2016 at 12:00:35 PM UTC-8, Stefan Karpinski 
> wrote: 
> > > Any reason not to run them all as separate processes? 
> > > 
> > > On Fri, Jan 22, 2016 at 11:08 PM, Ritchie Lee  > > 
> > > > wrote: 
> > >> Let's say I have 10 julia scripts, scripts = ["script1.jl", 
> "script2.jl", 
> > >> , "script10.jl"] and I would like to run them in parallel in 
> separate 
> > >> Julia sessions, but 4 at a time (since I only have 4 cores on my 
> > >> machine). 
> > >> Is there any way to do this programmatically? 
> > >> 
> > >> I tried doing this: 
> > >> 
> > >> addprocs(4) 
> > >> pmap(include, scripts) 
> > >> 
> > >> or 
> > >> 
> > >> addprocs(4) 
> > >> @parallel for s in scripts 
> > >> include(s) 
> > >> end 
> > >> 
> > >> However, this seems to reuse the same session, so all the global 
> consts 
> > >> in the script file are colliding.  I would like to make sure that the 
> > >> namespaces are completely separate. 
> > >> 
> > >> Thanks! 
>
>

Re: [julia-users] Running multiple scripts in parallel Julia sessions

[Stefan Karpinski]

Any reason not to run them all as separate processes?

On Fri, Jan 22, 2016 at 11:08 PM, Ritchie Lee  wrote:

> Let's say I have 10 julia scripts, scripts = ["script1.jl", "script2.jl",
> , "script10.jl"] and I would like to run them in parallel in separate
> Julia sessions, but 4 at a time (since I only have 4 cores on my machine).
> Is there any way to do this programmatically?
>
> I tried doing this:
>
> addprocs(4)
> pmap(include, scripts)
>
> or
>
> addprocs(4)
> @parallel for s in scripts
> include(s)
> end
>
> However, this seems to reuse the same session, so all the global consts in
> the script file are colliding.  I would like to make sure that the
> namespaces are completely separate.
>
> Thanks!
>

Re: [julia-users] Running multiple scripts in parallel Julia sessions

[Ritchie Lee]

Do you mean using @spawn, success, or just from the command prompt?

The scripts are long-running experiments where I am also tracking things 
like CPU time.  I would like them queued and run 4 at a time, i.e., if 1 
finishes ahead of others, then the next one will start running on the free 
processor.  Is there a way to schedule into separate processes?


On Saturday, January 23, 2016 at 12:00:35 PM UTC-8, Stefan Karpinski wrote:
>
> Any reason not to run them all as separate processes?
>
> On Fri, Jan 22, 2016 at 11:08 PM, Ritchie Lee  > wrote:
>
>> Let's say I have 10 julia scripts, scripts = ["script1.jl", "script2.jl", 
>> , "script10.jl"] and I would like to run them in parallel in separate 
>> Julia sessions, but 4 at a time (since I only have 4 cores on my machine).  
>> Is there any way to do this programmatically?
>>
>> I tried doing this:
>>
>> addprocs(4)
>> pmap(include, scripts)
>>
>> or
>>
>> addprocs(4)
>> @parallel for s in scripts
>> include(s)
>> end
>>
>> However, this seems to reuse the same session, so all the global consts 
>> in the script file are colliding.  I would like to make sure that the 
>> namespaces are completely separate.
>>
>> Thanks!
>>
>
>

[julia-users] Running multiple scripts in parallel Julia sessions

[Ritchie Lee]

Let's say I have 10 julia scripts, scripts = ["script1.jl", "script2.jl", 
, "script10.jl"] and I would like to run them in parallel in separate 
Julia sessions, but 4 at a time (since I only have 4 cores on my machine). 
 Is there any way to do this programmatically?

I tried doing this:

addprocs(4)
pmap(include, scripts)

or

addprocs(4)
@parallel for s in scripts
include(s)
end

However, this seems to reuse the same session, so all the global consts in 
the script file are colliding.  I would like to make sure that the 
namespaces are completely separate.

Thanks!