Re: [galaxy-dev] galaxy job manager with celery and rabbitmq

[Peter van Heusden]

Galaxy implements the job manager using rabbitmq and celery? Surely you
mean Galaxy *does not* implement celery (at this point). As far as I know
it is planned for the future.

Peter

On 25 January 2016 at 15:37, 杜云飞  wrote:

> Hi,
>   We are reading the source code for galaxy job manager. We found that the
> job manager works like amqp. We want to know why galaxy implements the  job
> manager using rabbitmq and celery. What is the difference between  them?
> Thanks!
> Best regards,
>forest
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Python ValueError: too many values to unpack

[Hans-Rudolf Hotz]

Hi Matthias


Make sure 'listing' is a list of (displayName,value,boolean)

Though, there might be other errors as well


Regards, Hans-Rudolf

I am not sure, whether this
On 01/26/2016 01:22 PM, Matthias De Smet wrote:

Hi all,

I’m trying to develop a tool that lets you select a file on the server.
I’m using the dynamic_options setting to let Galaxy render a dropdown
list of available file.


However, when I try to test the tool in its current form, I get the
following error:


galaxy.web.framework.decorators ERROR 2016-01-26 13:08:35,649 Uncaught
exception in exposed API method:
Traceback (most recent call last):
   File "lib/galaxy/web/framework/decorators.py", line 260, in decorator
 rval = func( self, trans, *args, **kwargs)
   File "lib/galaxy/webapps/galaxy/api/tools.py", line 98, in build
 return tool.to_json(trans, kwd.get('inputs', kwd))
   File "lib/galaxy/tools/__init__.py", line 2602, in to_json
 populate_state(trans, self.inputs, state_inputs, state_errors,
params.__dict__)
   File "lib/galaxy/tools/__init__.py", line 2471, in populate_state
 state[input.name] = input.get_initial_value(trans, context,
history=history)
   File "lib/galaxy/tools/parameters/basic.py", line 984, in
get_initial_value
 value = [ optval for _, optval, selected in options if selected ]
ValueError: too many values to unpack


My tool xml looks like this:




   from the archive to a data library
   
   
 test.sh > $log
   
   
 
   
 
 
   
 
 
 

   
 
  
   
 
 
   
   
This tool copies data from and archive folder to your history. Keep in
mind this folder MUST be readable by Galaxy in order for this tool to work.
   




and the script called in the “code” tag looks like this


def list_files(dir,query): listing=[]
for fname in listdir(dir):
  fullpath = path.join(dir, fname) #only select datafiles, not md5
checksums
  if path.isfile(fullpath) and not fullpath.endswith(".md5"):
  listing.append( fname )
return listing


This script should return a list with filenames that should be displayed
in the dropdown list.

I’m tracking Galaxy release 15.10, running on a Ubuntu 14.04.3 server

Thanks for your help!

Matthias






___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Python ValueError: too many values to unpack

[Matthias De Smet]

Wow, great tip! This fixed it!

Thanks a bunch!
Matthias


> Op 26 jan. 2016, om 15:20 heeft Hans-Rudolf Hotz  het volgende 
> geschreven:
> 
> Hi Matthias
> 
> 
> Make sure 'listing' is a list of (displayName,value,boolean)
> 
> Though, there might be other errors as well
> 
> 
> Regards, Hans-Rudolf
> 
> I am not sure, whether this
> On 01/26/2016 01:22 PM, Matthias De Smet wrote:
>> Hi all,
>> 
>> I’m trying to develop a tool that lets you select a file on the server.
>> I’m using the dynamic_options setting to let Galaxy render a dropdown
>> list of available file.
>> 
>> 
>> However, when I try to test the tool in its current form, I get the
>> following error:
>> 
>> 
>> galaxy.web.framework.decorators ERROR 2016-01-26 13:08:35,649 Uncaught
>> exception in exposed API method:
>> Traceback (most recent call last):
>>   File "lib/galaxy/web/framework/decorators.py", line 260, in decorator
>> rval = func( self, trans, *args, **kwargs)
>>   File "lib/galaxy/webapps/galaxy/api/tools.py", line 98, in build
>> return tool.to_json(trans, kwd.get('inputs', kwd))
>>   File "lib/galaxy/tools/__init__.py", line 2602, in to_json
>> populate_state(trans, self.inputs, state_inputs, state_errors,
>> params.__dict__)
>>   File "lib/galaxy/tools/__init__.py", line 2471, in populate_state
>> state[input.name] = input.get_initial_value(trans, context,
>> history=history)
>>   File "lib/galaxy/tools/parameters/basic.py", line 984, in
>> get_initial_value
>> value = [ optval for _, optval, selected in options if selected ]
>> ValueError: too many values to unpack
>> 
>> 
>> My tool xml looks like this:
>> 
>> 
>> 
>> 
>>   from the archive to a data library
>>   
>>   
>> test.sh > $log
>>   
>>   
>> > label="Choose source archive">
>>   
>> 
>> 
>>   
>> 
>> 
>> > dynamic_options="list_files(archive_source,query)"/>
>> 
>>   
>> 
>>  
>>   
>> 
>> 
>>   
>>   
>> This tool copies data from and archive folder to your history. Keep in
>> mind this folder MUST be readable by Galaxy in order for this tool to work.
>>   
>> 
>> 
>> 
>> 
>> and the script called in the “code” tag looks like this
>> 
>> 
>> def list_files(dir,query): listing=[]
>> for fname in listdir(dir):
>>  fullpath = path.join(dir, fname) #only select datafiles, not md5
>> checksums
>>  if path.isfile(fullpath) and not fullpath.endswith(".md5"):
>>  listing.append( fname )
>> return listing
>> 
>> 
>> This script should return a list with filenames that should be displayed
>> in the dropdown list.
>> 
>> I’m tracking Galaxy release 15.10, running on a Ubuntu 14.04.3 server
>> 
>> Thanks for your help!
>> 
>> Matthias
>> 
>> 
>> 
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>> 
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>> 

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Removing set_metadata_externally from wiki documentation?

[Peter Cock]

On Mon, Jan 25, 2016 at 11:33 AM, Peter Cock  wrote:
> Hello all,
>
> We're currently looking at changing our Galaxy setup to link user accounts
> with Linux user accounts for better cluster integration (running jobs as the
> actual user on SGE). As part of this, we've tried setting up a fresh
> installation
> on a new VM which has thrown up some issues.
>
> First of all, what is the status of the old set_metadata_externally setting?
> This is described on the legacy documentation page as required under
> "Submitting Jobs as the Real User",
>
> https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster/Legacy
>
> Unfortunately, the same text is still present on the current page:
>
> https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster/
>
> According to the GitHub search, set_metadata_externally no longer
> exists in the code or default settings files:
>
> https://github.com/galaxyproject/galaxy/search?q=set_metadata_externally
>
> I eventually tracked this down to the following commit, first included in
> Galaxy v13.06, back when the config file was universe_wsgi.ini rather
> than config/galaxy.ini,
>
> https://github.com/galaxyproject/galaxy/commit/68065e123800a5a2e700d636db9354702ac42fe5
>
> So, can we assume that metadata is now always set externally, and
> should the wiki be updated accordingly?
>
> Thanks,
>
> Peter

I've gone ahead and removed the references to set_metadata_externally
from https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster/
but the text also covered outputs_to_working_directory as well:

"""
The set_metadata_externally and outputs_to_working_directory options
in the Galaxy config must be set to True. set_metadata_externally is
always recommended for Galaxy servers running outside of a single
user's development environment. outputs_to_working_directory ensures
that a tool/job's outputs are written to the temporary working
directory, which (when using the real user system) is owned by the
real user who submitted the job. If left set to the default (False),
the tool will attempt to write directly to the directory specified in
file_path (by default, galaxy-dist/database/files/), which must be
owned by the Galaxy user (and thus will not be writable by the real
user).
"""

Does the new version make sense?:

"""
The outputs_to_working_directory option in the Galaxy config must be
set to True. This ensures that a tool/job's outputs are written to the
temporary working directory, which (when using the real user system)
is owned by the real user who submitted the job. If left set to the
default (False), the tool will attempt to write directly to the
directory specified in file_path (by default,
galaxy-dist/database/files/), which must be owned by the Galaxy user
(and thus will not be writable by the real user).
"""

Diff: 
https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster?action=diff=53=54

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Importing workflows via command-line/Docker script

[Cameron Jack]

Hi all,


I've been developing a Dockerfile and repositories to enable users to set up an 
NGS forensics pipeline. So far so good, I can build the image (based on Bjoern 
Gruening's excellent galaxy-stable Docker image) and incorporate all the 
various tools I need, including custom ones with custom interfaces. I've also 
built some workflows within a running image and exported them to .ga files, but 
now I'm wondering if there is a way to import those back in to a new image via 
the Dockerfile? I'd rather have everything set up for the users upon building 
the image, rather than logging in as admin and then importing the workflow. Is 
this possible? If so, how?


Thanks for any and all answers.


Best regards,

Cam


Cameron Jack

Bioinformatician
ANU Bioinformatics Consulting Unit
The Australian National University

The John Curtin School of Medical Research
Building 131 Garran Road
Acton 0200, ACT

Ph (office): +61 2 612 51128
Ph (mobile): +61 4 2368 0745
Email: cameron.j...@anu.edu.au
Group email: a...@anu.edu.au
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Retain the dbkey specified for an input dataset through a Galaxy workflow?

[Greg Von Kuster]

I’ve tracked down how the dbkey is getting lost on tool output datasets that 
are part of a collection, but now I’m wondering if the tool’s 
  tag is lacking information about the dbkey and this is why 
it is getting lost.

At least the code implies this.  John, can you help here?

The populate_collection_elements() function in 
~/lib/galaxy/tools/parameters/output_collect.py looks for a match on dbkey from 
the  tag set, and if there is no match the default dbkey 
value “?” is associated with the output dataset in the collection.  An example 
tool that results in this behavior has this tag set:





I’ve not found an example anywhere in the Galaxy code or in tools that have 
been written to produce output collections that includes a dbkey designation in 
the  tag set, so I’m wondering if I am correctly 
understanding the intent of this code.

I have a work-around fix that works without adding a dbkey designation to the 
tag set.  The caller of the populate_collection_elements() function is a 
function named collect_dynamic_collections(), whose signature includes the 
input datasets from which the dbkey can be retained.

I can submit a PR that includes this approach to a fix, but if the fix is as 
simple as adding some kind of dbkey designation to the tag set, an example of 
what that should look like would be much appreciated.

Thanks very much!

Greg Von Kuster


> On Jan 25, 2016, at 10:11 AM, Greg Von Kuster  wrote:
> 
> I’ve disovered that this issue is related to tools rather than workflows, and 
> specifically with tools that produce dataset collections on output.  In the 
> "job.finish()" method, metadata that includes the input dataset’s dbkey 
> setting is generated correctly for output datasets that are not part of a 
> collection, but the dbkey (and possibly other metadata attributes) are lost 
> if the output dataset is part of a collection.  I’m still digging to find how 
> setting metadata for output dataset collections is handled differently than 
> regular output datasets.
> 
> 
>> On Jan 22, 2016, at 2:34 PM, Greg Von Kuster  wrote:
>> 
>> Hello Galaxians,
>> 
>> I’m running Galaxy 15.10 and running workflows that include tools that 
>> require reference genomes (e.g. Extract Genomic DNA).  I set the dbkey for 
>> the input dataset and it is retained for some tools, but not others.  
>> Running the workflow multiple times, it looks like the dbekey is lost at 
>> different tool points in the workflow.  Is this a known issue or is there 
>> some setting I’ve missed.  I’ve seen where the output dattype can be set for 
>> each tool, but not the dbkey.  This is a problem because any tools that 
>> require a dbkey downstream result in errors.
>> 
>> I was running the dev branch for a while, but workflow bugs in that branch 
>> forced me to revert back to 15.10.
>> 
>> I’ve searched biostar and the mail lists, but haven’t seen an answer for 
>> this specific issue, although there are several related threads from the 
>> past.  Sorry if it’s been answered and I missed it.
>> 
>> Thanks very much for any help you can provide,
>> 
>> Greg Von Kuster
>> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/