Re: [galaxy-dev] Is there a way to dynamically assign Queues in Galaxy

[John Chilton]

I have produced an updated version of my changes to be compatible with
newer versions of galaxy, but but my local Galaxy instances are too
far behind to test the changes. Once I get some time to test this out
more I will update the pull request, but since you are interested now
the following commands should pull down the updated changeset and
apply it in isolation for minimum chance of breaking other things.

wget 
https://bitbucket.org/jmchilton/umn-galaxy-central/changeset/66395a9d870f/raw
-O dynamic_jobs.patch
hg import dynamic_jobs.patch

The original pull request was well tested, these changes aren't and I
am not sure how my changes will play with other changes that have been
made to Galaxy in meantime. If you chose to try this out, good luck
and let me know what goes wrong.

-John


John Chilton
Senior Software Developer
University of Minnesota Supercomputing Institute
Office: 612-625-0917
Cell: 612-226-9223

On Fri, Jun 1, 2012 at 1:32 PM, Anthonius deBoer  wrote:
> I would be very interested in this piece of code...
>
> Would I be wrong in assuming if I did the pull of that version I would be
> messing up my Galaxy system and I need to manually resolve it?
>
> I would not want to re-introduce many of the bugs etc. so could I not just
> pull in whatever code you changed? Could you indicate which python modules
> you changed so I can pull in just those?
>
> Thanks
>
> Thon
>
>
> On Jun 01, 2012, at 11:18 AM, John Chilton  wrote:
>
> I posted a message about this half a year ago and issued a pull
> request (which is now conflicted). I developed some code which was not
> a turn key solution but I thought it was a good first step and it was
> very expressive. It would allow you to do stuff like inspect input
> sizes or parameter values and dynamically change queues or queue
> parameters based on these (using raw python to express these rules).
>
> galaxy-dev@lists.bx.psu.edu/msg03010.html'>http://www.mail-archive.com/galaxy-dev@lists.bx.psu.edu/msg03010.html
>
> https://bitbucket.org/galaxy/galaxy-central/pull-request/12/dynamic-job-runners
>
> You are not the first person to follow up with requests like this, I
> would be happy to update the pull request if it were going to be acted
> on.
>
> -John
>
> On Fri, Jun 1, 2012 at 1:02 PM, Anthonius deBoer  wrote:
>> Could you give me a pointer where in the codebase I should start looking
>> to
>> implement this, please?
>> I'd be happy to share this with the community if it is useful for
>> others
>>
>> Thon
>>
>> On Jun 01, 2012, at 10:53 AM, Jeremy Goecks 
>> wrote:
>>
>> No, this is not configurable via XML; you'd have to extend the Galaxy
>> codebase to implement this behavior.
>>
>> Best,
>> J.
>>
>> On Jun 1, 2012, at 1:50 PM, Anthonius deBoer wrote:
>>
>> Hi Jeremy,
>>
>> Thanks for the info...But it's not clear from your message if I could use
>> this Key/Value approach at the moment to distinguish if a job should be
>> run
>> in the fast or in the slow queue.
>>
>> I would like to add a parameter to every tool that would have it determine
>> if it should be in the fast queue or in the slow queue...
>> It would be checked for interactive jobs and if someone created a workflow
>> with this tool, they could turn it off and it would run in the slow/high
>> memory queue
>> Could I add this today and what would the XML look like or are you saying
>> it
>> only works for the trackster example you gave...
>>
>> Thanks
>>
>> Thon
>>
>> On Jun 01, 2012, at 05:44 AM, Jeremy Goecks 
>> wrote:
>>
 Is there a way for a tool to sometimes be placed in the fast queue and
 sometimes in the long queue?
>>>
>>> Not through Galaxy as far as I know.
>>
>> Yes, this is possible using job parameterization. From
>> universe.ini.sample:
>>
>> --
>> # Per-tool job handler and runner overrides. Parameters can be included to
>> define multiple
>> # runners per tool. E.g. to run Cufflinks jobs initiated from Trackster
>> # differently than standard Cufflinks jobs:
>> #
>> # cufflinks = local:///
>> # cufflinks[source@trackster] = local:///
>> --
>>
>> This approach is definitely a beta feature, but the idea is that any set
>> of
>> key@value parameters should be able to be used to direct jobs to different
>> queues as needed.
>>
>> Job parameterization is done in only one place right now, the tracks.py
>> controller in rerun_tool The idea is that jobs run via Trackster are
>> short,
>> so they get a different queue:
>>
>> --
>> subset_job, subset_job_outputs = tool.execute( trans,
>> incoming=tool_params,
>> history=target_history,
>> job_params={ "source" : "trackster" } )
>> --
>>
>>
>>> Right now I'd like to be able to allocate jobs to different queues
>>> based on the input data size (and thus the expected compute time
>>> and resource needed), but that is rather complicated. e.g. If you
>>> have a low memory queue and a high memory query.
>>
>> To make this work, you'd want to modify the execute() method in th

Re: [galaxy-dev] Meme gives me UnicodeDecodeError. Is this an error from parsing the OUTPUT?

[Derrick Lin]

Okay, it turned out to be my execution hosts on the cluster didn't have the
required packages installed. I then install libxslt-devel and
ImageMagick-devel,
everything now works as expected.

Hope this help,
Derrick

On Wed, Jun 6, 2012 at 10:03 AM, Derrick Lin  wrote:

> I guess I know what the issue is (at least for my case). I checked the
> stdout, it contains:
>
>  MEME — Multiple EM for Motif Elicitation
>
> USAGE:
>
> meme  [optional arguments]
>
>
>   file containing sequences in FASTA format
>[-h]print this message
>[-o ]   name of directory for output files will not
>replace existing directory
>[-oc ]  name of directory for output files will
>replace existing directory
>[-text] output in text format (default is HTML)
>[-dna]  sequences use DNA alphabet
>[-protein]  sequences use protein alphabet
>[-mod oops|zoops|anr]   distribution of motifs
>
> ...
>
> Then  I checked the stderr, it has:
>
> mv: cannot stat 
> `/misc/NFS/app/galaxy/galaxy_dist/dev/database/job_working_directory/000/973/dataset_2527_files/meme.html':
>  No such file or directory
>
> Seems  to me, the meme was ran but return its help page and didn't return any 
> result. Still investigating why.
>
> Derrick
>
>
> On Wed, Jun 6, 2012 at 9:47 AM, Derrick Lin  wrote:
>
>> Hi,
>>
>> I am hitting the same issue. meme itself runs fine on commandline.
>>
>> I suspected the html or xml generated by meme contains non-ascii, thus
>> galaxy throw this error.
>>
>> Have you been able to fix that?
>>
>> MEME is not on the public galaxy server either..
>>
>> Regards,
>> Derrick
>>
>> On Thu, May 31, 2012 at 4:37 AM, Anthonius deBoer wrote:
>>
>>> Hi,
>>>
>>> I am trying to run MEME and I got this error below when running it.
>>> Is this an error complaining about the non-ASCII nature of the OUTPUT or
>>> of the program (or input?).
>>>
>>> Thanks
>>>
>>> Thon
>>>
>>> Error Traceback:
>>> View as:   Interactive  |  Text  |  XML (full)
>>> ⇝ UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>> 25: ordinal not in range(128)
>>> URL: http://srp106:8080/dataset/errors?id=176856
>>> Module weberror.evalexception.middleware:364 in respond  view
>>> >>  app_iter = self.application(environ, detect_start_response)
>>> Module paste.debug.prints:98 in __call__  view
>>> >>  environ, self.app)
>>> Module paste.wsgilib:539 in intercept_output  view
>>> >>  app_iter = application(environ, replacement_start_response)
>>> Module paste.recursive:80 in __call__  view
>>> >>  return self.application(environ, start_response)
>>> Module paste.httpexceptions:632 in __call__  view
>>> >>  return self.application(environ, start_response)
>>> Module galaxy.web.framework.base:160 in __call__  view
>>> >>  body = method( trans, **kwargs )
>>> Module galaxy.web.controllers.dataset:160 in errors  view
>>> >>  return trans.fill_template( "dataset/errors.mako", hda=hda )
>>> Module galaxy.web.framework:733 in fill_template  view
>>> >>  return self.fill_template_mako( filename, **kwargs )
>>> Module galaxy.web.framework:744 in fill_template_mako  view
>>> >>  return template.render( **data )
>>> Module mako.template:296 in render  view
>>> >>  return runtime._render(self, self.callable_, args, data)
>>> Module mako.runtime:660 in _render  view
>>> >>  **_kwargs_for_callable(callable_, data))
>>> Module mako.runtime:692 in _render_context  view
>>> >>  _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
>>> Module mako.runtime:718 in _exec_template  view
>>> >>  callable_(context, *args, **kwargs)
>>> Module dataset_errors_mako:74 in render_body  view
>>> >>  __M_writer(filters.html_escape(unicode(job.stdout )))
>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 25:
>>> ordinal not in range(128)
>>>
>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>
>>>  http://lists.bx.psu.edu/
>>>
>>
>>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Meme gives me UnicodeDecodeError. Is this an error from parsing the OUTPUT?

[Derrick Lin]

I guess I know what the issue is (at least for my case). I checked the
stdout, it contains:

 MEME — Multiple EM for Motif Elicitation

USAGE:

meme  [optional arguments]


  file containing sequences in FASTA format
   [-h]print this message
   [-o ]   name of directory for output files will not
   replace existing directory
   [-oc ]  name of directory for output files will
   replace existing directory
   [-text] output in text format (default is HTML)
   [-dna]  sequences use DNA alphabet
   [-protein]  sequences use protein alphabet
   [-mod oops|zoops|anr]   distribution of motifs

...

Then  I checked the stderr, it has:

mv: cannot stat
`/misc/NFS/app/galaxy/galaxy_dist/dev/database/job_working_directory/000/973/dataset_2527_files/meme.html':
No such file or directory

Seems  to me, the meme was ran but return its help page and didn't
return any result. Still investigating why.

Derrick


On Wed, Jun 6, 2012 at 9:47 AM, Derrick Lin  wrote:

> Hi,
>
> I am hitting the same issue. meme itself runs fine on commandline.
>
> I suspected the html or xml generated by meme contains non-ascii, thus
> galaxy throw this error.
>
> Have you been able to fix that?
>
> MEME is not on the public galaxy server either..
>
> Regards,
> Derrick
>
> On Thu, May 31, 2012 at 4:37 AM, Anthonius deBoer wrote:
>
>> Hi,
>>
>> I am trying to run MEME and I got this error below when running it.
>> Is this an error complaining about the non-ASCII nature of the OUTPUT or
>> of the program (or input?).
>>
>> Thanks
>>
>> Thon
>>
>> Error Traceback:
>> View as:   Interactive  |  Text  |  XML (full)
>> ⇝ UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> 25: ordinal not in range(128)
>> URL: http://srp106:8080/dataset/errors?id=176856
>> Module weberror.evalexception.middleware:364 in respond  view
>> >>  app_iter = self.application(environ, detect_start_response)
>> Module paste.debug.prints:98 in __call__  view
>> >>  environ, self.app)
>> Module paste.wsgilib:539 in intercept_output  view
>> >>  app_iter = application(environ, replacement_start_response)
>> Module paste.recursive:80 in __call__  view
>> >>  return self.application(environ, start_response)
>> Module paste.httpexceptions:632 in __call__  view
>> >>  return self.application(environ, start_response)
>> Module galaxy.web.framework.base:160 in __call__  view
>> >>  body = method( trans, **kwargs )
>> Module galaxy.web.controllers.dataset:160 in errors  view
>> >>  return trans.fill_template( "dataset/errors.mako", hda=hda )
>> Module galaxy.web.framework:733 in fill_template  view
>> >>  return self.fill_template_mako( filename, **kwargs )
>> Module galaxy.web.framework:744 in fill_template_mako  view
>> >>  return template.render( **data )
>> Module mako.template:296 in render  view
>> >>  return runtime._render(self, self.callable_, args, data)
>> Module mako.runtime:660 in _render  view
>> >>  **_kwargs_for_callable(callable_, data))
>> Module mako.runtime:692 in _render_context  view
>> >>  _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
>> Module mako.runtime:718 in _exec_template  view
>> >>  callable_(context, *args, **kwargs)
>> Module dataset_errors_mako:74 in render_body  view
>> >>  __M_writer(filters.html_escape(unicode(job.stdout )))
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 25:
>> ordinal not in range(128)
>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>
>>  http://lists.bx.psu.edu/
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Meme gives me UnicodeDecodeError. Is this an error from parsing the OUTPUT?

[Derrick Lin]

Hi,

I am hitting the same issue. meme itself runs fine on commandline.

I suspected the html or xml generated by meme contains non-ascii, thus
galaxy throw this error.

Have you been able to fix that?

MEME is not on the public galaxy server either..

Regards,
Derrick

On Thu, May 31, 2012 at 4:37 AM, Anthonius deBoer  wrote:

> Hi,
>
> I am trying to run MEME and I got this error below when running it.
> Is this an error complaining about the non-ASCII nature of the OUTPUT or
> of the program (or input?).
>
> Thanks
>
> Thon
>
> Error Traceback:
> View as:   Interactive  |  Text  |  XML (full)
> ⇝ UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 25:
> ordinal not in range(128)
> URL: http://srp106:8080/dataset/errors?id=176856
> Module weberror.evalexception.middleware:364 in respond  view
> >>  app_iter = self.application(environ, detect_start_response)
> Module paste.debug.prints:98 in __call__  view
> >>  environ, self.app)
> Module paste.wsgilib:539 in intercept_output  view
> >>  app_iter = application(environ, replacement_start_response)
> Module paste.recursive:80 in __call__  view
> >>  return self.application(environ, start_response)
> Module paste.httpexceptions:632 in __call__  view
> >>  return self.application(environ, start_response)
> Module galaxy.web.framework.base:160 in __call__  view
> >>  body = method( trans, **kwargs )
> Module galaxy.web.controllers.dataset:160 in errors  view
> >>  return trans.fill_template( "dataset/errors.mako", hda=hda )
> Module galaxy.web.framework:733 in fill_template  view
> >>  return self.fill_template_mako( filename, **kwargs )
> Module galaxy.web.framework:744 in fill_template_mako  view
> >>  return template.render( **data )
> Module mako.template:296 in render  view
> >>  return runtime._render(self, self.callable_, args, data)
> Module mako.runtime:660 in _render  view
> >>  **_kwargs_for_callable(callable_, data))
> Module mako.runtime:692 in _render_context  view
> >>  _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
> Module mako.runtime:718 in _exec_template  view
> >>  callable_(context, *args, **kwargs)
> Module dataset_errors_mako:74 in render_body  view
> >>  __M_writer(filters.html_escape(unicode(job.stdout )))
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 25:
> ordinal not in range(128)
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
>  http://lists.bx.psu.edu/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] How does Galaxy access datasets?

[Dorset, Daniel C]

I'm trying to troubleshoot why I can't retrieve output from my Galaxy cluster 
instance. I notice that when I click on any output, the URL is something like:

http://[root galaxy address]/datasets/[some 16-character hash]/display/[file 
name]

I'm not able to find the "datasets" directory on the local machine, and I 
couldn't figure anything out by searching the paster.log file and the apache 
access and error logs. Everytime I try to access output, it downloads a 
zero-byte file. The files that I want to download through Galaxy are in the 
correct subdirectory of /database/files/... If someone could explain to be 
what's going on "behind the scenes," it would help me quite a bit. I'm guessing 
that the absolute path is stored in a database, but beyond that I don't know 
any specifics. If that's the case, knowing the relevant database tables would 
be a huge hint.

Thanks!

Dan
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] populating history with input files automatically from remote site?

[Nikhil Joshi]

Hi all,

I have a galaxy instance running in the Amazon Cloud that I have
customized and I want to be able to transfer files (via scp) to the
instance (or its attached storage) and then have those files show up
automatically in the history.  Is this possible?  Alternatively, is
there some way to automatically populate the history of a registered
user, so that when that user logs in, the files are automatically in a
saved history?  A third possibility would be to transfer the files and
restart galaxy and then have the files show up automatically?  Just to
be clear, I do not want to use the Upload Files tool at all.  Is there
any way to implement any of these options?  Any help would be highly
appreciated.  Thanks!

- Nik.

-- 
Nikhil Joshi
Bioinformatics Analyst/Programmer
UC Davis Bioinformatics Core
http://bioinformatics.ucdavis.edu/
najoshi -at- ucdavis -dot- edu
530.752.2698 (w)
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Add new mako apge in galaxy front end

[Jeremy Goecks]

Shashi,

Here's our recommended approach:

(1) create the mako page in /templates or in an appropriate subdirectory

(2) write a controller method that processes inputs and then fills in the 
template. For example, from user/dbkeys:

return trans.fill_template( 'user/dbkeys.mako',
user=user,
dbkeys=dbkeys,
message=message,

installed_len_files=self.installed_len_files,
lines_skipped=lines_skipped,
fasta_hdas=fasta_hdas,
use_panels=kwds.get( 'use_panels', False ) )


Best,
J.

On Jun 5, 2012, at 1:47 AM, sshashi wrote:

> Hi Nate,
> 
> I am using local instance galaxy in my system . I want add one new mako
> page in galaxy .how can i add in galaxy ?
> 
> Regards
> shashi 
> 
> 
> ---
> 
> This e-mail is for the sole use of the intended recipient(s) and may
> contain confidential and privileged information. If you are not the
> intended recipient, please contact the sender by reply e-mail and destroy
> all copies and the original message. Any unauthorized review, use,
> disclosure, dissemination, forwarding, printing or copying of this email
> is strictly prohibited and appropriate legal action will be taken.
> ---
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> 
>  http://lists.bx.psu.edu/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/