Andrew Warren wrote:
> So would the current correct method for setting up multi-threaded jobs on a
> cluster be to specify custom runners in the [galaxy:tool_runners] section of
> the universe config file for EVERY tool that uses a multiple threads
> (assuming the default is set to one)?
>
> For example, for the bowtie program and a queue named "galaxy":
> *bowtie = pbs:///galaxy/-l ppn=4,mem=16gb/*
Hi Andrew,
You'll need to use the tool id from the <tool> tag in the XML config
file. The bowtie file is 'tools/sr_mapping/bowtie_wrapper.xml' and the
tool id is 'bowtie_wrapper'.
Unfortunately, you also need to set the number of threads in the same
XML file, although 4 happens to be the default:
--threads="4"
Unfortunately this value isn't read from the config currently.
--nate
> *
> *
> Is this currently the only way for galaxy to inform the queuing system how
> many threads a program will use?
> And does this mean that without custom runners in the config file any
> muti-threaded program that has multiple instances in an asychronous workflow
> has the opportunity to overload a cluster node since the queuing system
> doesn't "know" how many threads the program will be using?
>
> Just want to make sure I'm not missing out on the latest and greatest method
> for process management. :)
>
> Thanks,
> Andrew
> *
> *
> Louise-Amélie Schmitt wrote:
>
> > >
> > > default_cluster_job_runner will remain for backwards compatibility, but
> > > we'll ship a sample job_conf.xml that runs everything locally by
> > > default.
> > >
> > > --nate
> >
> > Haha, and I did that before realizing I could do just what I needed by
> > writing tool-specific *pbs*:// URLs at the end of the config file... I'm
> such
> > an idiot.
>
> Haha, okay, I don't think i even noticed since I was distracted by your
> implementation being a step in the way we want to go with it.
>
> > But I really like what you did of it and I have a couple of questions.
> >
> > Concerning the single-threaded tools, what would happen if the number of
> > threads set in the xml file was >1 ?
>
> It'd consume extra slots, but the tool itself would just run as usual.
>
> > Could it be possible to forbid a tool to run on a given node?
>
> Hrm. In *PBS* you could do it using node properties/neednodes or resource
> requirements. I'd have to think a bit about how to do this in a more
> general way in the XML.
>
> --nate
>
> >
> > Thanks,
> > L-A
> >
> >
> > >
> > >>
> > >> Peter
> > >>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
> http://lists.bx.psu.edu/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
