Re: [galaxy-dev] Problem with LWR (input size limitation)

[John Chilton]

On Wed, Nov 19, 2014 at 12:58 PM, Misharl Monsoor
 wrote:
> Hi John ,
>
> First all, thank you very much for your quick reply. I will test your
> suggestion about pycurl,  and tell you if it has worked. For the second
> question, i was wondering if it is possible with LWR to define a path (where
> are located the inputs) as an input (string input in the wrapper xml).

Still not certain I understand - but I do not believe this is possible
for datasets - this is something that must be configured for the LWR
itself. If you have inputs to the tool that are not datasets - such as
reference or index data - those could potentially be specified this
way as just 
> Thank you again for your help,
>
> Misharl Monsoor.
>
>
>
>
>
> Le 19/11/2014 15:30, John Chilton a écrit :
>
>> If you install pycurl (globally or in a Galaxy's virtualenv) and set
>> LWR_CURL_TRANSPORT=1 in Galaxy's environment than the LWR will stream
>> large files with CURL instead of the mmap hack.
>>
>> I don't understand your second question - the LWR should stream
>> whatever data inputs are selected by the user. If you want the tool to
>> be able to take many inputs you can use multiple="true" on your data
>> parameter or use a repeat block.
>> (https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax).
>>
>> Hope this helps,
>> -John
>>
>>
>> On Mon, Nov 17, 2014 at 3:04 AM, Misharl Monsoor 
>> wrote:
>>>
>>> Hi everybody,
>>>
>>> In our lab, we are trying to connect our Galaxy instance to a Windows 7
>>> 64
>>> bits server, in order to executes programs that need to be run within
>>> Windows. However we have a problem concerning the size of the input that
>>> is
>>> uploaded to the Windows server, it doesn't accept inputs with a size >
>>> 1.3
>>> Gb ? We have the following problem with mmap function:
>>>
>>>data = mmap.mmap(input.fileno(), 0, access=mmap.ACCESS_READ)
>>> error: [Errno 12] Cannot allocate memory
>>>
>>>
>>> I have another question concerning the inputs, is there a way to use
>>> several
>>> inputs that be uploaded to a Windows Server?
>>>
>>> Thank you very much in advance,
>>>
>>> Bests regards,
>>>
>>> Cheers,
>>>
>>> Misharl Monsoor
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   http://lists.bx.psu.edu/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/mailinglists/
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Creation of a backup/migrate script for data/users

[John Chilton]

This is a tough question - I suspect it will be difficult no matter
how you proceed. The API objects don't map 100% directly to database
objects and may not have complete coverage - but using the database
directly will require one to learn a lot of Galaxy's data model I
suspect.

A middle ground - would be to just use the Galaxy code and ORM layer
without running a Galaxy server (see scripts/db_shell.py for
instance).

My gut tells me it is probably be easiest (least difficult anyway) to
target the database directly - but this could be wrong.

-John

On Wed, Nov 19, 2014 at 6:21 AM, Rémy Dernat  wrote:
> Hi,
>
> I would like to create a code to backup and migrate(*) galaxy datas, based
> on the api and/or with the script I did here:
> https://github.com/remyd1/galaxy_debug_params/blob/master/source/Templates/parameters/generate_shell_script.mako
>
> To do that, I would use the galaxy model, to retrieve everything in the old
> one and recreate everything in the new one.
>
> However, I would like to be able to do that without my web server running.
> That is to say, a "cold" migration to avoid any current task to write
> additionnal things into the database. Obviously, that is a problem for the
> api because this one use the web server.
>
> Could you give some advice to do that ? Should I recreate a code from 0 ?
> Could I use the code in the api ? Is there a way to create a code that could
> use the api differently (recreate a copy of the api without the web server)
> ?
>
> Kind regards,
>
> Remy
>
>
> (*) this is important; that means you could import data from an old galaxy
> to a recent one (or the opposite); a simple backup of files and a dump of
> the db is not enough.
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Unable to add tools from tool shed

[John Chilton]

Is there a actually a "Repository Installation Error below" - it is
hard to say what went wrong without more logging?

It is certainly frustrating that that package installation is failing
- it seems to be a quite straight-forward installation definition.

-John

On Thu, Nov 13, 2014 at 3:19 PM, Ryan G  wrote:
> I'm following the guide on
> https://wiki.galaxyproject.org/Admin/Tools/AddToolFromToolShedTutorial to
> add a tool from the tool shed.
>
> The tutorial uses 'bwa_base', but I don't see that.  So instead, I'm using
> 'bwa_mem'.  I choose 'Preview and Install', then 'Install to Galaxy'.  I
> then select to have it installed in 'NGS: Mapping' tool panel, then click
> Install.
>
> The page then shows two tools:   bwa_mem, and package_bwa_0_7_7.  Both of
> which show 'Error' as the status.  Clicking on 'bwa_mem' shows some
> information with "This repository is not installed correctly (see the
> Repository Installation Error below)"
>
> What does this mean?  And how do I fix it?
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] master_api_key currently doesn't benefit from a real admin's key?!

[John Chilton]

Thanks for the detailed description - this gives me a better
understanding of what is going on. Okay - I think the right way to
handle this is probably to make the master api key work everywhere
where the user API key is insufficient.

It makes sense to me that the master API key for instance cannot
import a workflow - that operation requires a user. But something like
creating a data library and assigning user roles to it or installing
tool shed repositories - should be doable via the master API. I
understand that it probably cannot do these specific things - but lets
call that a bug or at least a deficiency and modify the permission
checking to allow the master API key to do these things.

I like the idea of cleaning up the permissions better than attaching a
fake or arbitrary user object to the session in the abstract - though
it may prove more difficult in some cases. Do you want to try to
attempt this and open a pull request - it may be as simple as
modifying a few if checks to see if trans.user_is_admin() before doing
other more elaborate checks? Even if the fixes are too complicated
even just outlining the API endpoints you want to be able to use with
the master_api_key, a bioblend script to exercise them, and the Galaxy
stack traces would allow me or another devteam member to modify the
checks much more quickly.

Hope this helps,

-John

On Wed, Nov 19, 2014 at 12:14 PM, Dooley, Damion  wrote:
> Ah! Key idea is the ease of installation/maintenance of tools that need admin 
> level api access.
>
> I was hoping that the master api key enabled my program to do any of the api 
> calls available (I'm using bioblend btw).  I am running workflows, uploading 
> datasets from user history to data library, linking files into data library 
> from server.  Alot of this work is done on behalf of a user - but I don't 
> want to give such a user elevated access to the server.  So yes I can 
> hard-code an admin api key into my galaxy tool for doing this work.
>
> But I would much rather just count on a master/admin/uber api key that is 
> defined in galaxy config for doing all this work.  This way, whenever a tool 
> is installed from toolshed that needs admin functionality, it can just start 
> working if the "uber" api key has been set up.  (No need to set up an admin 
> user, copy key into config or wherever, reset galaxy etc).  That's what I was 
> thinking the "master" api key did - I didn't realize it doesn't work with at 
> least a handful of the API calls because it lacks an associated user object.  
> i had tried to put an admin api key into the master_api_key field but that 
> failed on various needed api calls - it ignores the potential to use the 
> admin user's user object.
>
> By the way I do use the interacting user's non-privileged api key as well, it 
> is convenient for delineating what that user does and does not have access to 
> in terms of data libraries/datasets and workflows.  Its just the extra work I 
> trigger on their behalf has to be admin level.
>
> The tool itself is called the Data Versioning tool, and it is listing and 
> caching/retrieving versioned fasta (or other types of) data and regenerating 
> blast etc. databases as requested by users as an initial step to performing 
> their other workflows.  It aims to quickly deal with datasets that are 
> anywhere from megabytes to hundreds of gigabytes in size.
>
> Hope that helps?
>
> Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre 
> for Disease Control
> 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada
> 
> From: John Chilton [jmchil...@gmail.com]
> Sent: Wednesday, November 19, 2014 6:39 AM
> To: Dooley, Damion
> Cc: galaxy-...@lists.bx.psu.edu
> Subject: Re: [galaxy-dev] master_api_key currently doesn't benefit from a 
> real admin's key?!
>
> If you already have an admin key in the database - that is more useful
> than the master_api_key - I don't believe there is anything that the
> master API key can do that the admin API key cannot (let me know if I
> am wrong). I created master_api_key mostly just to bootstrap users and
> API keys for new installations. So my question is what are you trying
> to accomplish with a master API key if you already have an admin API
> key? I would be happy to comment on the different ideas you mentioned
> - but I want to understand what you are trying to accomplish.
>
> -John
>
> On Tue, Nov 18, 2014 at 8:35 PM, Dooley, Damion  
> wrote:
>> I made the erroneous assumption that if I put my own admin user API key into 
>> the galaxy configuration master_api_key field, it would accept that and run 
>> all the api functions that needed a key connected to a 

Re: [galaxy-dev] more control over the "docker" command

[John Chilton]

On Mon, Nov 24, 2014 at 10:54 AM, Dan Tenenbaum  wrote:
> Hi,
>
> Thanks for supporting the running of docker containers in Galaxy.
>
> I have two requests for more control over the docker command that is run.
>
> According to https://github.com/apetkau/galaxy-hackathon-2014 , the docker 
> command that is run when a docker-enabled tool is run might look something 
> like this:
>
> command is: sudo docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v 
> /home/aaron/Projects/galaxy-central:/home/aaron/Proje
> cts/galaxy-central:ro -v 
> /home/aaron/Projects/galaxy-central/tools/docker:/home/aaron/Projects/galaxy-central/tools/docker:ro
>  -v /home/aaron/Projects/galaxy-central/datab
> ase/job_working_directory/000/6:/home/aaron/Projects/galaxy-central/database/job_working_directory/000/6:rw
>  -v /home/aaron/Projects/galaxy-central/database/files:/home/aa
> ron/Projects/galaxy-central/database/files:rw -w 
> /home/aaron/Projects/galaxy-central/database/job_working_directory/000/6 
> --net none busybox:ubuntu-14.04 
> /home/aaron/Projects/galaxy-central/database/job_working_directory/000/6/container.sh;
>  return_code=$?; if [ -f 
> /home/aaron/Projects/galaxy-central/database/job_working_directory/000/6/wo
> rking_file ] ; then cp 
> /home/aaron/Projects/galaxy-central/database/job_working_directory/000/6/working_file
>  /home/aaron/Projects/galaxy-central/database/files/000/dataset_10.dat ; fi; 
> sh -c "exit $return_code"
>
> I'd like to be able to specify extra flags to be included in the command. In 
> my case I'd like to include "--link server:server" because I want to link 
> this container with another container that contains a long-running server 
> process (I can describe my use case in greater detail if desired).
> Can there be a way to do this in my tool wrapper?

I certainly don't mind adding more options - we've added a bunch as
different interesting things have been attempted (e.g. docker in
docker). I would like to keep the tool wrapper very general though and
expose these options in the job_conf.xml instead - since for the most
part they seem like they will be related to the tool's deployment and
not the abstract command being produced. I have added this
functionality with the following changeset -
https://bitbucket.org/galaxy/galaxy-central/commits/ebceda0721f3f3215447c19f94525f06f164afd1.
Simply add --link
server:server to your job destination. Let me know if this
doesn't work out.

>
> Secondly, rather than giving my galaxy user passwordless sudo, I'd like to 
> add it to the docker group, then I can run docker commands without prepending 
> sudo. This seems a lot safer. Could this be exposed, maybe by a sudo="false" 
> attribute in the tool wrapper?

This is currently possible - in your job_conf.xml just add to the job
destination parameter:

  false

Checkout the admittedly huge file config/job_conf.xml.sample_advanced
for a full list of the supported options. The docker related options
are all near each other and apply to any built-in job runner.

I suspect tot running with sudo really should be the default and I am
growing frustrated it is not. I am unsure if I should "fix" it now and
break the couple sites doing Docker in Galaxy or just leave it and
have everyone who wants to turn sudo off disable it explicitly.

At any rate - keep us informed about how things are going - I would
love to hear about what you are attempting to do with the linked
persistent docker container :).

-John

>
> Thanks,
> Dan
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] more control over the "docker" command

[John Chilton]

On Mon, Nov 24, 2014 at 12:56 PM, Dan Tenenbaum  wrote:
>
>
> - Original Message -
>> From: "John Chilton" 
>> To: "Dan Tenenbaum" 
>> Cc: galaxy-dev@lists.galaxyproject.org
>> Sent: Monday, November 24, 2014 9:30:50 AM
>> Subject: Re: [galaxy-dev] more control over the "docker" command
>>
>> On Mon, Nov 24, 2014 at 10:54 AM, Dan Tenenbaum
>>  wrote:
>> > Hi,
>> >
>> > Thanks for supporting the running of docker containers in Galaxy.
>> >
>> > I have two requests for more control over the docker command that
>> > is run.
>> >
>> > According to https://github.com/apetkau/galaxy-hackathon-2014 , the
>> > docker command that is run when a docker-enabled tool is run might
>> > look something like this:
>> >
>> > command is: sudo docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
>> > /home/aaron/Projects/galaxy-central:/home/aaron/Proje
>> > cts/galaxy-central:ro -v
>> > /home/aaron/Projects/galaxy-central/tools/docker:/home/aaron/Projects/galaxy-central/tools/docker:ro
>> > -v /home/aaron/Projects/galaxy-central/datab
>> > ase/job_working_directory/000/6:/home/aaron/Projects/galaxy-central/database/job_working_directory/000/6:rw
>> > -v /home/aaron/Projects/galaxy-central/database/files:/home/aa
>> > ron/Projects/galaxy-central/database/files:rw -w
>> > /home/aaron/Projects/galaxy-central/database/job_working_directory/000/6
>> > --net none busybox:ubuntu-14.04
>> > /home/aaron/Projects/galaxy-central/database/job_working_directory/000/6/container.sh;
>> > return_code=$?; if [ -f
>> > /home/aaron/Projects/galaxy-central/database/job_working_directory/000/6/wo
>> > rking_file ] ; then cp
>> > /home/aaron/Projects/galaxy-central/database/job_working_directory/000/6/working_file
>> > /home/aaron/Projects/galaxy-central/database/files/000/dataset_10.dat
>> > ; fi; sh -c "exit $return_code"
>> >
>> > I'd like to be able to specify extra flags to be included in the
>> > command. In my case I'd like to include "--link server:server"
>> > because I want to link this container with another container that
>> > contains a long-running server process (I can describe my use case
>> > in greater detail if desired).
>> > Can there be a way to do this in my tool wrapper?
>>
>> I certainly don't mind adding more options - we've added a bunch as
>> different interesting things have been attempted (e.g. docker in
>> docker). I would like to keep the tool wrapper very general though
>> and
>> expose these options in the job_conf.xml instead - since for the most
>> part they seem like they will be related to the tool's deployment and
>> not the abstract command being produced. I have added this
>> functionality with the following changeset -
>> https://bitbucket.org/galaxy/galaxy-central/commits/ebceda0721f3f3215447c19f94525f06f164afd1.
>> Simply add --link
>> server:server to your job destination. Let me know if this
>> doesn't work out.
>
> Thanks very much for the quick turnaround!
>
> I think it will work out. I am not an expert in job_conf.xml. My main concern 
> is being able to pass extra arguments to some docker-enabled tools but not to 
> others, is that possible?

Many job destinations can map to the same underlying compute resource
(cluster, server, etc...) - so I would just duplicate your job
destination definition and customize each one for the different tool
and then your tool section - make sure a given tool maps to the
correct destination. There will be some unfortunate duplication - but
it shouldn't be too bad unless you are doing a bunch of very cool
things with dozens of different tools.
(https://trello.com/c/wt1LflNh).

>
>
>>
>> >
>> > Secondly, rather than giving my galaxy user passwordless sudo, I'd
>> > like to add it to the docker group, then I can run docker commands
>> > without prepending sudo. This seems a lot safer. Could this be
>> > exposed, maybe by a sudo="false" attribute in the tool wrapper?
>>
>> This is currently possible - in your job_conf.xml just add to the job
>> destination parameter:
>>
>>   false
>>
>> Checkout the admittedly huge file config/job_conf.xml.sample_advanced
>> for a full list of the supported options. The docker related options
>> are all near each other and apply to any built-in job runner.
>>
>> I suspect tot running with sudo really should be the default and I am
>&

Re: [galaxy-dev] rename output dataset in workflow - input dataset variable

[John Chilton]

All -

Certainly this is a very real and important problem.

The devteam hasn't moved on the tagging approach outlined in the dev
thread referenced by Peter and I suspect that is because I the
prevailing thought on the team is that dataset naming is not the most
appropriate abstraction to use to address that (though personally I
would be keen to merge a pull request for the compromise approach I
outlined if someone wants to put it together).

Outside the realm of dataset naming however - the devteam is actively
working on this problem in at least two ways.

 - If one is performing an interactive analysis with a few initial
inputs - showing the structure and connection between datasets in the
history I suspect will be a more robust way to track connections and
inputs throughout an analysis than dataset names. Carl has prototyped
and demonstrated some stuff internally for showing such structures - I
would assume it is coming in a future release.

- If you have many samples - I suspect no approach based around
individual datasets will be sufficient. Dataset collections however
have been designed from the ground up with sample tracking in mind and
I think with very little effort on the part of tool developers users
get a very effective sample tracking. Dataset lists and lists of
paired datasets (say representing replicates, samples, conditions, or
patients, etc...) or more deeply nested data structures (representing
hierarchical combinations of those things) are created with element
identifiers at each level of the hierarchy that are preserved
throughout a complex analysis transparently in a way that names are
not - and with very little effort tool developers can leverage these
at merging steps - to produce reports, etc
(bit.ly/gcc2014workflows).

I am not claiming the problem has been solved - but I did want to
express that the devteam is working on it very actively and things
will continue to improve in this realm.

Thanks for the comments,
-John


On Fri, Nov 21, 2014 at 4:20 AM, Peter Cock  wrote:
> Yes :(
>
> There's been some past discussion of this from a tool developer
> perspective, e.g. https://trello.com/c/JnhOEqow and
> http://dev.list.galaxyproject.org/Using-input-dataset-names-in-output-dataset-names-td4662481.html
>
> The best individual tool authors can do is something like
> "$input.name processed with XXX" or "XXX on $input.name"
> which in a long pipeline results in extremely long names with
> tools sometimes prefixed and sometimes postfixed. :(
>
> Of course, things get really complicated when a tool has multiple
> input files - in some cases the tool author could regard one set of
> files as primary and preserve their name/tag only,
>
> Naming things is hard.
>
> Peter
>
> On Wed, Nov 19, 2014 at 8:34 PM, Curtis Hendrickson (Campus)
>  wrote:
>>
>> Brad et al,
>>
>>
>>
>> I would like second the issue you raise so succinctly. The failure to 
>> automatically
>> track the original sample name throughout the analysis (that and  array 
>> selection
>> of paired end reads) is one of the biggest barriers people face for doing 
>> work on
>> many samples in galaxy. It just gets very confusing unless you spend a lot 
>> of time
>> workarounds (creating workflows to rename things, editing datasets 
>> individually,
>> etc) – especially for non-programmer users, for whom workflows with variables
>> and API calls are beyond the pale.
>>
>>
>>
>> Regards,
>>
>> Curtis
>>
>>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] tool xml data_columns interface

[John Chilton]

Hello Jorrit,

This makes a great deal of sense to me - opened a pull request with
the requested change here
https://bitbucket.org/galaxy/galaxy-central/pull-request/575/allow-using-attribute-optional-for-data/diff.

-John

On Tue, Nov 25, 2014 at 9:58 AM, Jorrit Boekel
 wrote:
> Hi all,
>
> I’m playing around with the data_column param in my tool xml. Excellent 
> stuff. I only have a small point. The force_select attribute, can it not 
> instead be called optional (and reverse its behaviour)? That would make it 
> more similar to other params which have optional=true/false as an option 
> (e.g. type=data).
>
> cheers,
> —
> Jorrit Boekel
> Proteomics systems developer
> BILS / Lehtiö lab
> Scilifelab Stockholm, Sweden
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Is anyone using composite datatype uploads?

[John Chilton]

What is disappearing (sortof) is the UI for selecting the individual
pieces - I don't think anything in the API has changed yet.

I don't think datatypes have a way of picking apart an archive file
for their pieces - but it would make a lot of sense to implement.

Total Aside: I wonder if testing tools one at a time is always the
best approach - one can imagine (for instance in your indexer's case)
utility in being able to string together some pre-steps or post-steps
for things like conversions to ease testing. For some very yak-shavey
reasons I wrote a little YAML DSL
(https://gist.github.com/jmchilton/3b66f101c53c6734d3c5) for building
Galaxy workflows to test some tool and workflow features. I have been
considering extending it with the ability to specify input and outputs
conditions and adding it to planemo to test repositories . (By
assigning labels at a level above the actual RAW workflow - it solves
the biggest problem that was present with the workflow testing stuff
we briefly discussed last winder). It would be cool for local and CI
testing - but it is more difficult to see how that pattern could work
with the tool shed test framework where the ids are not simple and the
tools may be split across many repositories.

-John

On Wed, Nov 19, 2014 at 10:39 AM, Peter Cock  wrote:
> Hi John, Sam,
>
> I've not done it yet, but was hoping to implement uploading of
> BLAST databases at some point - mainly for use within the
> test framework, rather than expecting it to be useful for the
> end user.
>
> Is the issue here uploading an archive (e.g. .zip or .tar.gz) or
> offering a way to pick multiple files to be treated together as
> a composite dataset?
>
> Peter
>
> On Wed, Nov 19, 2014 at 3:32 PM, John Chilton  wrote:
>> Well there is at least one person using this functionality -
>> http://dev.list.galaxyproject.org/Problem-to-upload-data-to-Galaxy-when-using-pbed-file-format-td4666000.html.
>>
>> Just to make this more concrete - Sam has swapped the "upload file"
>> button to use the new upload widget this release cycle (targeted for
>> December 1st). So barring negative feedback - uploading pbed or velvet
>> report datatypes (or other similar composite datatypes) will no longer
>> be possible via the GUI.
>>
>> -John
>>
>> On Thu, Aug 14, 2014 at 2:32 PM, Aysam Guerler  
>> wrote:
>>> Hello everyone,
>>>
>>> We are considering to disable the deprecated upload tool form which is
>>> currently accessible through Tool panel > Get Data > Upload file. The new
>>> upload feature (icon at the top of the Tool panel) covers all of its
>>> functionality except uploading composite datatypes like e.g. Velvet.
>>>
>>> Please let us know if you are using the composite file upload functionality
>>> of the former tool form.
>>>
>>> Thanks,
>>> Sam
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   http://lists.bx.psu.edu/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/mailinglists/
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Fwd: Bug with multiple reverse proxies

[John Chilton]

Sounds great - Dannon merged your pull request
(https://bitbucket.org/galaxy/galaxy-central/pull-request/560/use-first-component-of-x-forwarded-for/diff)
this morning. Thanks for the fix.

-John

On Mon, Nov 24, 2014 at 5:30 AM, Jan Kanis  wrote:
> Hi John,
>
> The change in galaxy and the one in routes are two independent instances of
> the same problem, so the change in the galaxy PR can be merged immediately.
> It just won't solve my original problem without the routes changes.
>
> Jan
>
> On 19 November 2014 at 15:33, John Chilton  wrote:
>>
>> Hey Jan,
>>
>>   Thanks for the fixes. Sorry no one responded to your earlier e-mail.
>> I think the pull request should serve as contact point on this  - it
>> cannot be merged until this
>> (https://github.com/bbangert/routes/pull/33) is fixed and Galaxy
>> updated to target it right?
>>
>> On Thu, Nov 13, 2014 at 11:02 AM, Jan Kanis  wrote:
>> > Hi,
>> >
>> > We ran into a problem if galaxy is running behind multiple reverse
>> > proxies.
>> > Galaxy assumes that the X-Forwarded-Host header only contains a single
>> > host
>> > to which it redirects, but apache will append comma-separated components
>> > to
>> > it if it already exists, which is the case of multiple reverse proxies.
>> > The
>> > result is that galaxy sometimes (i.e. sometimes after user login,
>> > depending
>> > on browser specifics, and when installing toolshed tools) tries to
>> > redirect
>> > the browser to "hostname, hostname". Apache docs
>> >
>> > I have a patch for galaxy in this pull request. That change fixes a file
>> > that makes the same assumption. The actual fix requires a change in
>> > Routes.
>> > The Routes PR is here, so actually fixing this requires an update of
>> > Routes
>> > after that pr is merged there.
>> >
>> > Jan
>> >
>> >
>> >
>> > ___
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client.  To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> >   http://lists.bx.psu.edu/
>> >
>> > To search Galaxy mailing lists use the unified search at:
>> >   http://galaxyproject.org/search/mailinglists/
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] galaxy with torque

[John Chilton]

I am not sure we have a walkthrough for Torque specifically - but if
you have Galaxy up and running and you can qsub commands to torque -
hopefully you have done all of the hard parts.

You will need a DRMAA library for your torque setup -
https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster
suggests compiling pbs_drmaa and outlines how to set it up. After that
you just need to add a plugin and default destination to your
job_conf.xml file - also outlined on that wiki page.

Other good resources to consult if you are scaling up your Galaxy this way are:
https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer
https://wiki.galaxyproject.org/Events/GCC2014/TrainingDay/AdminWalkthrough

Good luck and let us know if you encounter any problems.

-John

On Fri, Nov 21, 2014 at 2:30 PM, Fernandez Edgar
 wrote:
> Hello all,
>
>
>
> My name is Edgar Fernandez. I’m a sys. admin. at University of Montreal.
>
> I’ve contacted you a while back about installing galaxy and I’ve
> successfully done it on a redhat 6 server.
>
>
>
> I see myself in a situation where I need to utilise all my redhat servers
> (who are identical to the server running the galaxy website).
>
>
>
> I’ve also successfully installed a server torque with compute notes and
> clients nodes.
>
>
>
> What are the last step to make the link between galaxy and torque?
>
> Also, once that connection is made, how will galaxy keep track of the jobs
> sent?
>
> I mean who will it know this job that just finished is for this user and not
> another ?
>
>
>
> Also, my torque installation is so that my server running galaxy is a submit
> node and a client node.
>
> I hope this is not a problem.
>
>
>
> Please help!
>
>
>
> Cordialement / Regards,
>
>
>
> Edgar Fernandez
>
> System Administrator (Linux)
>
> Direction Générale des Technologies de l'Information et de la Communication
>
> (  Bur. : 1-514-343-6111 poste 16568
>
>
>
> Université de Montréal
>
> PAVILLON ROGER-GAUDRY, bureau X-218
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Deleted tool temp dir

[John Chilton]

That Galaxy version is quite old at this point. There were a couple
fixes for the cleanup_job config handling over the last year that may
have fixed the problem. I run with this option enabled daily and I
have not seen any problems in months. If you can confirm this is still
a problem with latest stable Galaxy instance - I would be definitely
be eager to address any bugs.

-John

On Sun, Nov 30, 2014 at 5:43 PM, Pandori n  wrote:
> Hi all.
> Config:
> - galaxy-dist stable (changeset:   11247:a9a0ac9c1afa) [Dec/18/2013]
> - NFS
> - Cluster (torque)
>
> When i start my WF,  on step 2, in 'job_working_directory' created temp
> (intermediate) dir (for tool step 3). But when step 2 is completed, temp dir
> (in  'job_working_directory') removed(via galaxy) and step 3 failed. In
> 'universe_wsgi.ini'  'cleanup_job = never'. Why galaxy.jobs deleted temp
> dir?
>
> Any help is appreciated. Thanks.
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] galaxy with torque

[John Chilton]

That handle id seems wrong... at least it is not what I am used to. It
needs to match a server section specified in your galaxay ini file -
usually this is a simple string like handler0 or something.

It looks like the specific error is :

/home/galaxy/galaxy-dist/eggs/pbs_python-4.3.5-py2.6-linux-x86_64-ucs4.egg/_pbs.so:
undefined symbol: log_record

I am not sure what causes this - some subtle incompatibility.

So the pbs_python 4.4.0 egg is available on eggs.galaxyproject.org (
http://eggs.galaxyproject.org/pbs_python/pbs_python-4.4.0.tar.gz) - I think
it may be needed for newer torque versions - do you want to update the
version specified in eggs.ini, delete the old egg, and rescramble the egg
with 4.4.0?

-John

P.S. Since you are setting up a new server I would strongly suggest using
postgres instead of MySQL - but the previous comment about it not needing
to be accessed on the compute servers is correct.


On Tue, Dec 2, 2014 at 3:10 PM, Fernandez Edgar <
edgar.fernan...@umontreal.ca> wrote:

>  Hello again,
>
>
>
> I’m very close in making pbs_python work but I’m hitting a new wall.
>
> So I’ve created the file config/job_conf.xml which looks like this
>
> 
>
> 
>
> 
>
>  load="galaxy.jobs.runners.pbs:PBSJobRunner"/>
>
> 
>
> http://gavroche.esi.umontreal.ca>*">
>
> http://gavroche.esi.umontreal.ca>*" tags="pbs"/>
>
> 
>
> 
>
> 
>
>  tags="mycluster,longjobs">
>
> walltime=72:00:00
>
> 
>
> 
>
> 
>
> *gavroche.esi.umontreal.ca <http://gavroche.esi.umontreal.ca>* is my
> torque server.
>
> I know that in your documentation it doesn’t say to put in a handlers tag
> but galaxy doesn’t parse the xml without it.
>
>
>
> Now, once I try to start galaxy, I get the error you see in the file
> *paster.log* attached to this email.
>
> Can anyone help please?
>
>
>
> Cordialement / Regards,
>
> Edgar Fernandez
>
>
>
> *De :* Fernandez Edgar
> *Envoyé :* December-02-14 1:33 PM
> *À :* 'Rémy Dernat'
> *Cc :* John Chilton; galaxy-...@bx.psu.edu
> *Objet :* RE: Re : [galaxy-dev] galaxy with torque
>
>
>
> Thank you for that correction.
>
>
>
> Just a small FYI (maybe it will be useful to update the wiki)…
>
>
>
> I had to export three variables to make the scramble possible:
>
>
>
> export PBS_PYTHON_INCLUDEDIR=/usr/local/torque/include/
>
> export PBSCONFIG=/usr/local/torque/bin/pbs-config
>
> export LIBTORQUE_DIR=/usr/local/torque/lib/libtorque.so
>
> python scripts/scramble.py -e pbs_python
>
>
>
> Cordialement / Regards,
>
> Edgar Fernandez
>
>
>
> *De :* Rémy Dernat [mailto:remy...@gmail.com ]
> *Envoyé :* December-02-14 11:49 AM
> *À :* Fernandez Edgar
> *Cc :* John Chilton; galaxy-...@bx.psu.edu
> *Objet :* Re: Re : [galaxy-dev] galaxy with torque
>
>
>
> Sorry for answer 7. There is no benefit to do that. Once the egg is done,
> there is nothing to do from here, except if you change your python
> version... If that variable is empty, that is normal, because it is not an
> environment variable, it is just used by the following python command:
>
> LIBTORQUE_DIR=/path/to/libtorque python scripts/scramble.py -e pbs_python
>
>
>
> 2014-12-02 16:27 GMT+01:00 Rémy Dernat :
>
> Hi Edgar,
>
>
>
> You are right. It is very annoying...
>
>
>
> So, to answer your questions:
>
> 1/ First answer of google / wikipedia with DRMAA :
> http://en.wikipedia.org/wiki/DRMAA
>
> It is a pattern to talk with any DRM system (SGE, torque, whatever...)
>
>
>
> 2/ This is a library (python version) for your Torque installation.
>
>
>
> 3/ MySQL access is only needed by your galaxy frontend.
>
>
>
> 4/ Internet access is not required for your compute node (except the
> galaxy one), but it is better if you want to use a package manager on your
> compute nodes, for example...
>
>
>
> 5/ On my part, I use permissions like 760 on galaxy directory. It depends
> on your needs... Some applications might need an access to your galaxy
> installation, but you should split binaries, the galaxy installation and
> your data (datasets, libraries...). But do not forget to share this folders
> by NFS (if needed).
>
>
>
> 6/ Sorry, no idea; but I see no reason for that to become unavailable, if
> your proxy is well configured.
>
>
>
> 7/ You have to put this command line into a file which will be sourced
> like $HOME_GALAXY/.bashrc or your environment file
> ("environment_setup_file" in universe_wsgi.ini or co

Re: [galaxy-dev] Page generated by a Galaxy behind HTTPS(Nginx) makes HTTP calls

[John Chilton]

Thanks for the error report. I believe some more recent changes have
been made to address these particular issues:

https://bitbucket.org/galaxy/galaxy-central/commits/579d211c2b0fa8ef4195930cb999edbb9a0cf8eb
https://bitbucket.org/galaxy/galaxy-central/commits/7867dda60542072921153e30195fffde741e5333

If these don't address the problem - letting us know the exact path
through the UI to recreate the problem would be of great help in
tracking things down.

-John

On Tue, Dec 2, 2014 at 10:59 AM, Maxime Lévesque
 wrote:
>
> I have followed the instructions here :
>
> https://wiki.galaxyproject.org/Admin/Config/nginxProxy
>
> to run a Galaxy behind an NGinx proxy under with HTTPS,
> The relevant part of my NGinx config has :
>
> proxy_set_header REMOTE_USER "$loginOnHost";
> proxy_set_header   X-Forwarded-Host $host;
> proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
> proxy_set_header   X-URL-SCHEME https;
>
>
> it is working well, with the exception that some pages are making external
> calls
> to http://genome.ucsc.edu, and not https://genome.ucsc.edu .
>
> I get a "mixed content" error (in chrome) :
>
> I verified that the URL that the page is trying to access *does* exist under
> HTTPS,
> so it's just a matter of telling galaxy to generate HTTPS links. I would
> have thought that
> this is what "proxy_set_header   X-URL-SCHEME https;" did...
>
>
> https://galaxy-1hge11q-uvd.udes.genap.ca/' was loaded over HTTPS, but
> requested an insecure resource
> 'http://genome.ucsc.edu/cgi-bin/hgTables?GALAXY_URL=https%3A//galaxy-1hge11q…sc_table_direct1&hgta_compressType=none&sendToGalaxy=1&hgta_outputType=bed'.
> This request has been blocked; the content must be served over HTTPS.
>
>
> Thanks !
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] how to use api to log into Galaxy?

[John Chilton]

Hmm... as far as I am aware the API cannot be used to drive this. I
would suggest if at all possible to configure authentication external
to your Galaxy instance - using Apache or nginx - and then the two
applications (yours and Galaxy) can share a common authentication
mechanism and session.

If you really want to drive something like this directly - not using a
proxy - I think you are going to need to directly enter session
information into Galaxy's database.

-John

On Thu, Nov 27, 2014 at 2:16 AM, xlwang  wrote:
> Hi,
> I will integrate Galaxy with another system. When I logged into this system,
> and clicked a link jump to Galaxy, I hope this user has already logged into
> Galaxy. How to use api to deliver a session or somthing else to Galaxy?
> Now, I can use api to create a  new user in this system and  also in Galaxy.
>
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problem with LWR (input size limitation)

[John Chilton]

I have had one other report of this error previously - but I have not
been able to reproduce it. I have tried so hard to understand these
unicode/utf-8/Python 2 v 3/etc... issues and I must admit I simply
don't :(.

Reading through http://farmdev.com/talks/unicode/ I found a hack that
might work, if you place:

import sys
sys.setdefaultencoding('utf-8')

at the top of lwr/web/framework.py does that fix the problem?

-John

On Wed, Nov 26, 2014 at 2:53 AM, Misharl Monsoor  wrote:
> Hi John,
>
> I can confirm you that your advice putting "LWR_CURL_TRANSPORT=1" works very
> fine, thanks a lot! Sorry, i haven't expressed myself correctly for my
> second question, thanks for answering, it helps me.
> I have another problem with LWR, when executing the jobs on the Windows
> Server, i have the following bug (it happens a lot of time, and i must
> restart the LWR server in order for it to works):
>
>  File
> "C:\Python2764\lib\site-packages\paste-1.7.5.1-py2.7.egg\paste\httpserver
> py", line 287, in wsgi_execute
>self.wsgi_start_response)
>  File "C:\Desktop\lwr_master\lwr\web\framework.py", l
> ne 37, in __call__
>return controller(environ, start_response, **request_args)
>  File "C:\Desktop\lwr_master\lwr\web\framework.py", l
> ne 141, in controller_replacement
>resp = self.__build_response(result)
>  File "C:\Desktop\lwr_master\lwr\web\framework.py", l
> ne 129, in __build_response
>resp = Response(body=self.body(result))
>  File "C:\Desktop\lwr_master\lwr\web\framework.py", l
> ne 155, in body
>body = dumps(result)
>  File "C:\Python2764\lib\json\__init__.py", line 243, in dumps
>return _default_encoder.encode(obj)
>  File "C:\Python2764\lib\json\encoder.py", line 207, in encode
>chunks = self.iterencode(o, _one_shot=True)
>  File "C:\Python2764\lib\json\encoder.py", line 270, in iterencode
>return _iterencode(o, 0)
> nicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 718:
> invalid
> start byte
>
>
> Bests regards and thank you very much for your help,
>
> Misharl Monsoor.
>
>
>
> Le 21/11/2014 17:44, John Chilton a écrit :
>
>> On Wed, Nov 19, 2014 at 12:58 PM, Misharl Monsoor
>>  wrote:
>>>
>>> Hi John ,
>>>
>>> First all, thank you very much for your quick reply. I will test your
>>> suggestion about pycurl,  and tell you if it has worked. For the second
>>> question, i was wondering if it is possible with LWR to define a path
>>> (where
>>> are located the inputs) as an input (string input in the wrapper xml).
>>
>> Still not certain I understand - but I do not believe this is possible
>> for datasets - this is something that must be configured for the LWR
>> itself. If you have inputs to the tool that are not datasets - such as
>> reference or index data - those could potentially be specified this
>> way as just > be passed through on the command-line just like any other parameter.
>>
>> Hope this helps.
>> -John
>>
>>> Thank you again for your help,
>>>
>>> Misharl Monsoor.
>>>
>>>
>>>
>>>
>>>
>>> Le 19/11/2014 15:30, John Chilton a écrit :
>>>
>>>> If you install pycurl (globally or in a Galaxy's virtualenv) and set
>>>> LWR_CURL_TRANSPORT=1 in Galaxy's environment than the LWR will stream
>>>> large files with CURL instead of the mmap hack.
>>>>
>>>> I don't understand your second question - the LWR should stream
>>>> whatever data inputs are selected by the user. If you want the tool to
>>>> be able to take many inputs you can use multiple="true" on your data
>>>> parameter or use a repeat block.
>>>> (https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax).
>>>>
>>>> Hope this helps,
>>>> -John
>>>>
>>>>
>>>> On Mon, Nov 17, 2014 at 3:04 AM, Misharl Monsoor
>>>> 
>>>> wrote:
>>>>>
>>>>> Hi everybody,
>>>>>
>>>>> In our lab, we are trying to connect our Galaxy instance to a Windows 7
>>>>> 64
>>>>> bits server, in order to executes programs that need to be run within
>>>>> Windows. However we have a problem concerning the size of the input
>>>>> that
>>>>> is
>>>>> uploaded to the Windows server, it doesn't accept inputs with a size >

Re: [galaxy-dev] Nothing being tested on Test and main Tool Shed?

[John Chilton]

On Wed, Dec 10, 2014 at 5:46 AM, Peter Cock  wrote:
> On Thu, Nov 27, 2014 at 1:58 PM, Peter Cock  wrote:
>> Hi Guys,
>>
>> I retitled this since the TestToolShed does seem to be running tests
>> regularly again. However, I still have a fair number failing with the
>> cryptic status "Exception: History in error state."
>>
>> I guess the logging changes John tried are not propagated to the
>> user-facing output on the ToolShed, but might still offer you some
>> clues?
>>
>> It appears from the stack trace to be something with uploading
>> the test input files into the new history?
>>
>> My current list of tools with failing tests is:
>>
>> https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go
>> https://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr
>> https://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod
>> https://testtoolshed.g2.bx.psu.edu/view/peterjc/fastq_paired_unpaired
>> https://testtoolshed.g2.bx.psu.edu/view/peterjc/get_orfs_or_cdss
>> https://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler
>> https://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus
>> https://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus
>> https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_primer_clip
>>
>> I have updated some of the tools I listed below (in the older email),
>> so their latest revision has not yet been tested.
>>
>> Regards,
>>
>> Peter
>
> Hi guys,
>
> The good news is my list of failing tools has shrunk to the following,
> none of which seem to have been tested yet in December, stuck
> with "Exception: Job in error state.":
>
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/mira_assembler
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus
>
> Meanwhile, on the main tool,  "Exception: Job in error state."
> and not yet tested in December:
>
> https://toolshed.g2.bx.psu.edu/view/peterjc/blast2go
>
> It would be nice to have a more informative error here...

Ugh - yes it clearly would - very sorry about that.

https://trello.com/c/BZUaMnJB

>
> Have these tools been automatically earmarked not to be retested?
> Note that some tools have been tested earlier this month:
>
> I also have a legitimate failure for missing tests, run 2014-12-09,
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus
>

Interesting... some tests did run though? Does the framework not
report passing tests - seems like it would be hard to know if it
working in that case.

-John

> And again, a legitimate failure for some missing tests on the
> main Tool Shed, also run on 2014-12-09,
> https://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus
>
> Regards,
>
> Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] order of tools installed from toolshed

[John Chilton]

Peter -

  When you have previously asked about suite_config.xml I grepped
around and I couldn't find any references to it, did some more greps
just now and I still cannot. My best guess is that is not being used
at all - but I am not a tool shed expert.

Wolfgang -

  I am not aware of being able to enforce an ordering of these things.
It would be a nice feature obviously so I have created a Trello card
for the request - https://trello.com/c/lPOkD4BD.

-John

On Wed, Dec 10, 2014 at 6:08 AM, Peter Cock  wrote:
> Thanks Wolfgang,
>
> I think the suite_config.xml file was originally designed a subset of
> the tool_conf.xml file, inserted into it at run time. I think it could hold
> the tools in the desired order - probably with subsections and
> labels too.
>
> However, I suspect this may not be used with the current
> ToolShed setup - let's wait for the USA based Galaxy team to
> explain later today?
>
> Regards,
>
> Peter
>
> On Wed, Dec 10, 2014 at 10:40 AM, Wolfgang Maier
>  wrote:
>> Hi Peter,
>>
>> here's the link to the package:
>>
>> https://testtoolshed.g2.bx.psu.edu/view/wolma/suite_mimodd_0_1_5
>>
>> It is not using a suite_config.xml file (which I hear of for the first
>> time), but contains just a single repository_dependencies.xml file with this
>> content:
>>
>> 
>> 
>>  > name="package_mimodd_0_1_5" owner="wolma"
>> toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > owner="wolma" toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > name="mimodd_ngs_run_annotation" owner="wolma"
>> toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > owner="wolma" toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > owner="wolma" toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > owner="wolma" toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > name="mimodd_read_alignment" owner="wolma"
>> toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > name="mimodd_variant_calling" owner="wolma"
>> toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > name="mimodd_extract_variants" owner="wolma"
>> toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > name="mimodd_coverage_stats" owner="wolma"
>> toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > name="mimodd_deletion_prediction" owner="wolma"
>> toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > owner="wolma" toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > name="mimodd_annotate_variants" owner="wolma"
>> toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > name="mimodd_snpeff_genomes" owner="wolma"
>> toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>>  > name="mimodd_cloudmap_prepare" owner="wolma"
>> toolshed="https://testtoolshed.g2.bx.psu.edu"; />
>> 
>>
>> and I'd like Galaxy to add the corresponding tools in exactly this order to
>> the Tools bar. Do I need an additional file to specify this order ?
>>
>> Best,
>> Wolfgang
>>
>>
>>
>> On 12/10/2014 11:31 AM, Peter Cock wrote:
>>>
>>> Hi Wolfgang,
>>>
>>> You didn't include the name or a link to your Test Tool Shed packages.
>>> Are you talking about the suite_config.xml file?
>>>
>>> I'm not sure if Galaxy still uses that, no-one answered my earlier
>>> question:
>>>
>>> http://dev.list.galaxyproject.org/Role-of-suite-config-xml-in-current-Tool-Shed-tc4665887.html
>>>
>>> https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-October/020762.html
>>>
>>> e.g.
>>>
>>> https://github.com/peterjc/pico_galaxy/blob/master/tools/protein_analysis/suite_config.xml
>>> https://github.com/peterjc/pico_galaxy/issues/5
>>>
>>> Peter
>>>
>>> On Wed, Dec 10, 2014 at 10:18 AM, Wolfgang Maier
>>>  wrote:

 Dear all,

 I'm currently testing a software package we developed for installation
 from
 the testtoolshed. Things seem to work fine mostly, but there is one thing
 I
 don't know how to solve (if it can be solved).

 Our package consists of 14 individual tools, which I uploaded as 14
 separate
 repositories together with a repository suite definition that's taking
 care
 of installing all tools.

 Now, somewhat naively as it seems, I expected the order of repository
 declarations in the suite definition file to determine the order of the
 tools in the Galaxy Tools Bar, but, instead, they are added in seemingly
 random order.

 So I'm looking for a way to predefine the final order of the tools so
 that
 it makes sense for a typical workflow. Any advice how to address this?

 Thanks for your help,
 Wolfgang

 ___
 Please keep all replies on the list by using "reply all"
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] package_numpy_1_7 can't be installed

[John Chilton]

Looks like this - https://github.com/clarete/curdling/issues/19

I think you somehow need to install the package svn on your machine - if
you have root on that machine the command might be something like "sudo
zypper install svn" or may it is "sudo zypper install subversion". If you
don't have root on the machine I would ask the operating system maintainer
or try to compile and install svn locally and make it available to Galaxy.

Hope this helps,

-John

On Fri, Dec 12, 2014 at 7:14 AM, Wang, Yu  wrote:
>
>  Hi, I am running galaxy on SLSE11, when I tried to install bumpy 1.7.1, I
> got the following error.
> Can anyone point what goes wrong here?
>
>
>   Error Running from numpy source directory.
> /bin/sh: svnversion: command not found
>
> /home/wang/galaxy-dist/database/tmp/tmp-toolshed-mtdMR9ITC/numpy-1.7.1/numpy/distutils/system_info.py:1494:
>  UserWarning:
> Atlas (http://math-atlas.sourceforge.net/) libraries not found.
> Directories to search for the libraries can be specified in the
> numpy/distutils/site.cfg file (section [atlas]) or by setting
> the ATLAS environment variable.
>   warnings.warn(AtlasNotFoundError.__doc__)
>
> /home/wang/galaxy-dist/database/tmp/tmp-toolshed-mtdMR9ITC/numpy-1.7.1/numpy/distutils/system_info.py:1503:
>  UserWarning:
> Blas (http://www.netlib.org/blas/) libraries not found.
> Directories to search for the libraries can be specified in the
> numpy/distutils/site.cfg file (section [blas]) or by setting
> the BLAS environment variable.
>   warnings.warn(BlasNotFoundError.__doc__)
>
> /home/wang/galaxy-dist/database/tmp/tmp-toolshed-mtdMR9ITC/numpy-1.7.1/numpy/distutils/system_info.py:1506:
>  UserWarning:
> Blas (http://www.netlib.org/blas/) sources not found.
> Directories to search for the sources can be specified in the
> numpy/distutils/site.cfg file (section [blas_src]) or by setting
> the BLAS_SRC environment variable.
>   warnings.warn(BlasSrcNotFoundError.__doc__)
> /bin/sh: svnversion: command not found
>
> /home/wang/galaxy-dist/database/tmp/tmp-toolshed-mtdMR9ITC/numpy-1.7.1/numpy/distutils/system_info.py:1408:
>  UserWarning:
> Atlas (http://math-atlas.sourceforge.net/) libraries not found.
> Directories to search for the libraries can be specified in the
> numpy/distutils/site.cfg file (section [atlas]) or by setting
> the ATLAS environment variable.
>   warnings.warn(AtlasNotFoundError.__doc__)
>
> /home/wang/galaxy-dist/database/tmp/tmp-toolshed-mtdMR9ITC/numpy-1.7.1/numpy/distutils/system_info.py:1419:
>  UserWarning:
> Lapack (http://www.netlib.org/lapack/) libraries not found.
> Directories to search for the libraries can be specified in the
> numpy/distutils/site.cfg file (section [lapack]) or by setting
> the LAPACK environment variable.
>   warnings.warn(LapackNotFoundError.__doc__)
>
> /home/wang/galaxy-dist/database/tmp/tmp-toolshed-mtdMR9ITC/numpy-1.7.1/numpy/distutils/system_info.py:1422:
>  UserWarning:
> Lapack (http://www.netlib.org/lapack/) sources not found.
> Directories to search for the sources can be specified in the
> numpy/distutils/site.cfg file (section [lapack_src]) or by setting
> the LAPACK_SRC environment variable.
>   warnings.warn(LapackSrcNotFoundError.__doc__)
> error: must supply either home or prefix/exec-prefix -- not both
>
>
> Cheers,
> Yu
>
> di29her
> w...@lrz.de
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] How galaxy workflow work ?

[John Chilton]

Somehow your user object is not being set. Are you sure you have
initialized bioblend with the correct API key? Do other operations
work with that key?

-John

On Wed, Dec 3, 2014 at 11:00 AM, Bongsoo Park  wrote:
> Hi All,
>
> I faced a problem after I tested the Bio-Blend API. I cannot access the
> history anymore with the below error messages. Although, I tried
> installation from the scratch, but I got the same error message. Please let
> me know how to fix this problem. Thanks!
>
> (GALAXY User Interface)
>
> ===
>
>   "xhr": {
>
> "readyState": 4,
>
> "responseText": "{\"err_msg\": \"Uncaught exception in exposed API
> method:\", \"err_code\": 0}",
>
> "responseJSON": {
>
>   "err_msg": "Uncaught exception in exposed API method:",
>
>   "err_code": 0
>
> },
>
> "status": 500,
>
> "statusText": "Internal Server Error",
>
> "responseHeaders": {
>
>   "Date": "Wed, 03 Dec 2014 15:50:02 GMT\r",
>
>   "cache-control": "max-age=0,no-cache,no-store\r",
>
>   "Server": "PasteWSGIServer/0.5 Python/2.7.8\r",
>
>   "Connection": "close\r",
>
>   "content-type": "application/json\r"
>
> }
>
> ===
>
> I found the error message in the Galaxy instance shell interface.
>
> ===
>
> galaxy.web.framework.decorators ERROR 2014-12-03 10:50:02,279 Uncaught
> exception in exposed API method:
>
> Traceback (most recent call last):
>
>   File "/home/bxp12/galaxy-dist/lib/galaxy/web/framework/decorators.py",
> line 244, in decorator
>
> rval = func( self, trans, *args, **kwargs)
>
>   File
> "/home/bxp12/galaxy-ist/lib/galaxy/webapps/galaxy/api/history_contents.py",
> line 76, in index
>
> and ( history_id == trans.security.encode_id( trans.history.id ) ) ):
>
> AttributeError: 'NoneType' object has no attribute 'id'
>
> ===
>
> Best,
>
> Bongsoo
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Metadata error in uploading files to data libraries

[John Chilton]

Hello,

  Sorry we haven't made progress on this - and thanks for creating a
Trello card (https://trello.com/c/tw75nq1U).

  It looks like samtools is not on your path, can you use it from the
command-line? If not you should probably install it - I would suggest
install samtools 0.1.19 with homebrew (looks like you are a on Mac).
If you can run samtools from the command-line - can you do a which
`samtools` from your command-line and see where it is coming from and
then add that directory explicitly to your Galaxy PATH - say at the
top of run.sh in your Galaxy root (let me know if you need more
details on that).

-John

On Tue, Dec 9, 2014 at 3:02 AM, Jelle Scholtalbers
 wrote:
> Hi all,
>
> after an update to the following changeset(14859:7ba05957588a, stable,
> 05.12.14), our bam files that are uploaded(linked) to a data library, are no
> longer indexed. The metadata_xxx.dat is created, but it stays empty.
> The following error message appears in the log, although the state of the
> dataset is 'ok':
>
> galaxy.jobs WARNING 2014-12-05 12:47:02,218 Error accessing /g/K/K27.bam,
> will retry: [Errno 1] Operation not permitted: '/g/K/K27.bam'
> galaxy.jobs WARNING 2014-12-08 13:38:57,045 Error accessing /g/K/K2.bam,
> will retry: [Errno 1] Operation not permitted: '/g/K/K2.bam'
>
> All file permissions are correct (i.e. galaxy owns them). Furthermore,
> executing samtools index, just works on those files:
> samtools index /g/K/K2.bam
> /g/galaxy/galaxy_data/files/_metadata_files/006/metadata_6598.dat
>
> When uploading the file - "copy files into galaxy" - the samtools index just
> works.
>
> ==
>
> Now, on a clean local install(14874:885f940bff64, stable, 05.12.14) and
> samtools installed globally and with the bam file sorted, I get the
> following situation:
> When I try to upload this bam to a data library by linking the following
> error is shown on the dataset (note: here the dataset is set in error state,
> which does not happen on our server)
>
> Uploaded by: schol...@embl.de
> Date uploaded: Mon Dec 8 17:42:39 2014 (UTC)
> File size: 2.6 GB
> UUID: d23cf11a-0372-41cb-939a-7c8761d78b73
> Data type: auto
> Build: ?
> Miscellaneous information: Traceback (most recent call last): File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 407, in __main__() File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 396, in _
> Job Standard Error
>
> Traceback (most recent call last):
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 407, in
> __main__()
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 396, in __main__
> add_file( dataset, registry, json_file, output_path )
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 294, in add_file
> if datatype.dataset_content_needs_grooming( dataset.path ):
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
> line 147, in dataset_content_needs_grooming
> version = self._get_samtools_version()
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
> line 129, in _get_samtools_version
> output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE,
> stdout=subprocess.PIPE ).communicate()[1]
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
> line 711, in __init__
> errread, errwrite)
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
> line 1308, in _execute_child
> raise child_exception
> OSError: [Errno 2] No such file or directory
>
> error
> Database/Build: ?
> Number of data lines: None
> Disk file: /Users/scholtalbers/workspace/idr_data/WT1.sort.bam
>
> ===
> When uploading the bam without linking, I see the following processes:
> Upload->set meta->samtools index->'error state'
>
>  Miscellaneous information: uploaded bam file Traceback (most recent call
> last): File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 407, in __main__() File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.p
> Job Standard Error
>
> Traceback (most recent call last):
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 407, in
> __main__()
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 396, in __main__
> add_file( dataset, registry, json_file, output_path )
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
> line 324, in add_file
> if link_data_only == 'copy_files' and
> datatype.dataset_content_needs_grooming( output_path ):
>   File
> "/Users/scholtalbers/workspace/galaxy-dist-new/lib/galaxy/datatypes/binary.py",
> line 147, in dataset_cont

Re: [galaxy-dev] Functional tests using gzipped inputs have broken

[John Chilton]

Hey Peter,

Thanks for the very detailed bug report - it really helped track things down.

I have pushed a fix to next-stable and default (the bug never reached stable).

https://bitbucket.org/galaxy/galaxy-central/commits/a74d32014b0512613042c2a44d0b946703a1fbcd

The commit also includes a test case so hopefully this won't break again.

-John

On Wed, Dec 17, 2014 at 6:56 AM, Peter Cock  wrote:
> Hi all,
>
> Using gzipped input files in tool functional tests (which would be
> decompressed by the upload tool) used to work, e.g.
>
> https://travis-ci.org/peterjc/pico_galaxy/builds/42395911
> https://github.com/peterjc/pico_galaxy/blob/539260fa4348ed454ce464e48c9dbbccca22a012/tools/seq_filter_by_mapping/seq_filter_by_mapping.xml
>
> But with more recent versions of Galaxy this has broken, e.g.
>
> https://travis-ci.org/peterjc/pico_galaxy/builds/44038046
> https://github.com/peterjc/pico_galaxy/blob/e3064cc8f42253efd1f8eed56844d4c54645072e/tools/seq_filter_by_mapping/seq_filter_by_mapping.xml
>
> According to the TravisCI failure, the command run was:
>
> python 
> /home/travis/build/peterjc/pico_galaxy/galaxy-central-master/tools/seq_filter_by_mapping/seq_filter_by_mapping.py
> -i "None" -f "data" -m lax -p
> /tmp/tmpaCsMm5/tmpEpBIX1/database/files/000/dataset_50.dat
> /tmp/tmpaCsMm5/tmpEpBIX1/database/files/000/dataset_49.dat
>
> In this case the -i argument was $input_file, and -f was its
> extension/datatype. Somehow the gzipped FASTQ was
> not found, which is why the filename is just "None" and
> its datatype the default "data".
>
> My tool is sensibly returning error code 1, and:
>
> Missing input file: 'None'
>
> I have made some minor changes to this tool between those
> TravisCI tests, but the test is the same:
>
> 
> 
>  ftype="fastqsanger" />
>  value="SRR639755_sample_by_coord.sam" ftype="sam" />
> 
> 
>  ftype="fastqsanger" />
> 
> 
>  ftype="fastqsanger" />
>  value="SRR639755_sample_by_coord.sam" ftype="sam" />
> 
> 
>  ftype="fastqsanger" />
> 
> 
>
> --
>
> I could reproduce this on my development machine running an
> older Galaxy:
>
> $ hg log | head
> changeset:   16732:212e1d5e9be5
> branch:  stable
> tag: tip
> parent:  16716:2db0fb9594d6
> parent:  16731:7adac1842adf
> user:John Chilton 
> date:Wed Dec 10 12:20:55 2014 -0500
> summary: Merged in dannon/galaxy-central/stable (pull request #602)
>
> And again updating to the current tip,
>
> $ hg log | head
> changeset:   16850:e614c7c1e0e9
> tag: tip
> parent:  16848:1cf0af0a6324
> parent:  16849:d13f2b265dbf
> user:Dannon Baker 
> date:Tue Dec 16 15:48:44 2014 -0500
> summary: Merged in nitesh1989/galaxy-central3 (pull request #615)
>
> I can "fix" it by swapping the gzipped test input_file
> SRR639755_mito_pairs.fastq.gz to the uncompressed
> SRR639755_mito_pairs.fastq instead, as in this commit:
>
> https://github.com/peterjc/pico_galaxy/commit/126e3e716965016808ec2ba3a4be3b1075a2ebbf
> https://travis-ci.org/peterjc/pico_galaxy/builds/44317785
>
> I would like to revert this workaround since the uncompressed
> FASTQ file is a bit large to bundle as a test case - so I hope
> this is simply due to an accidental regression in Galaxy itself.
>
> Thanks,
>
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Error when opening bowtie2 tool page

[John Chilton]

Hello Anna,

Yeah - this is readily reproducible. I will work on this problem today
- sorry about that.

-John


On Thu, Dec 18, 2014 at 8:55 AM, Anna Terry  wrote:
> Deleting the newer 0.3 version of Bowtie fixes the problem with version 0.2.
>
> On 18 December 2014 at 13:29, Anna Terry  wrote:
>>
>> Hi again,
>> I have just updated Picard, and am getting a similar error with some tools
>> from that repository when I switch to older version from tool page.
>> I also noticed that some tools from the Picard repo appear as multiple
>> entries for different in the tool panel rather than selecting the different
>> versions from the drop down.
>> These errors are all on tools with version dropdowns, however I don't get
>> these errors on all of the version dropdowns, for instance I can switch to
>> an older version of fastqc no problem.
>> So are the new versions of tools breaking old versions, or is there a
>> problem with the tool panel?
>>
>> Any help appreciated!!
>>
>>
>> SamToFastq:
>> URL: http://../tool_runner/index
>> File
>> '/home/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py',
>> line 364 in respond
>>   app_iter = self.application(environ, detect_start_response)
>> File
>> '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py',
>> line 84 in __call__
>>   return self.application(environ, start_response)
>> File
>> '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/remoteuser.py',
>> line 107 in __call__
>>   return self.app( environ, start_response )
>> File
>> '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py',
>> line 633 in __call__
>>   return self.application(environ, start_response)
>> File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 132
>> in __call__
>>   return self.handle_request( environ, start_response )
>> File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 190
>> in handle_request
>>   body = method( trans, **kwargs )
>> File
>> '/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/tool_runner.py',
>> line 90 in index
>>   template, vars = tool.handle_input( trans, params.__dict__ )
>> File '/home/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py', line 2136 in
>> handle_input
>>   errors, params = self.__check_param_values( trans, expanded_incoming,
>> expanded_state, old_errors, process_state, history=history, source=source )
>> File '/home/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py', line 2251 in
>> __check_param_values
>>   errors = self.update_state( trans, inputs, state.inputs, incoming,
>> old_errors=old_errors or {}, source=source )
>> File '/home/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py', line 2514 in
>> update_state
>>   group_state = state[input.name]
>> KeyError: 'single_paired_end_type'
>>
>>
>> ReorderSam:
>> URL: http://../tool_runner/index
>> File
>> '/home/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py',
>> line 364 in respond
>>   app_iter = self.application(environ, detect_start_response)
>> File
>> '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py',
>> line 84 in __call__
>>   return self.application(environ, start_response)
>> File
>> '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/remoteuser.py',
>> line 107 in __call__
>>   return self.app( environ, start_response )
>> File
>> '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py',
>> line 633 in __call__
>>   return self.application(environ, start_response)
>> File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 132
>> in __call__
>>   return self.handle_request( environ, start_response )
>> File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 190
>> in handle_request
>>   body = method( trans, **kwargs )
>> File
>> '/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/tool_runner.py',
>> line 90 in index
>>   template, vars = tool.handle_input( trans, params.__dict__ )
>> File '/home/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py', line 2136 in
>> handle_input
>>   errors, params = self.__check_param_values( trans, expanded_incoming,
>> expanded_state, old_errors, process_state, history=history, source=source )
>> File '/home/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py', line 2251 in
>> __check_param_values
>>   errors = self.update_state( trans, inputs, state.inputs, incoming,
>> old_errors=old_errors or {}, source=source )
>> File '/home/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py', line 2514 in
>> update_state
>>   group_state = state[input.name]
>> KeyError: 'source'
>>
>> FastqToSam:
>> URL: http://./tool_runner/index
>> File
>> '/home/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py',
>> line 364 in respond
>>   app_iter = self.application(environ, detect_start_response)
>> File
>> '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recur

Re: [galaxy-dev] [Spam:*****] Re: How galaxy workflow work ?

[John Chilton]

There have been a ton of changes to the workflow API - I have been
trying to preserve backward compatibility though - e.g. I have API
tests in Galaxy for a bunch of legacy/deprecated features related to
workflows now.

Do you have a stack trace on the Galaxy side for the "Uncaught
exception in exposed API method"?

I will run through the blend4j test cases and see they turn up.

-John

On Thu, Dec 18, 2014 at 9:44 AM, Bruijn, Freek de  wrote:
> I am also having problems using the Galaxy API, with a similar error message 
> to the one reported by Bongsoo Park a few weeks ago (Uncaught exception in 
> exposed API method). We noticed it a few days ago and have not found the 
> cause yet. What we do know:
> - the problem occurs on several Galaxy servers, including the main instance 
> at https://usegalaxy.org/;
> - the problem seems to occur whilte using BioBlend, blend4j, and api Python 
> scripts;
> - even some of the blend4j unit tests are failing.
>
> One of my colleagues noticed a lot of changes in for example workflows.py (in 
> lib/galaxy/webapps/galaxy/api): 
> https://bitbucket.org/galaxy/galaxy-dist/annotate/579d211c2b0fa8ef4195930cb999edbb9a0cf8eb/lib/galaxy/webapps/galaxy/api/workflows.py#cl-117
>
> Did something change in the Galaxy API? Should we adjust the libraries and 
> the programs that use them?
>
> Cheers, Freek
>
>
> 
> From: galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf of 
> galaxy-dev-requ...@lists.galaxyproject.org 
> [galaxy-dev-requ...@lists.galaxyproject.org]
> Sent: Tuesday, December 16, 2014 18:00
> To: galaxy-dev@lists.galaxyproject.org
> Subject: galaxy-dev Digest, Vol 102, Issue 17
>
> Send galaxy-dev mailing list submissions to
> galaxy-dev@lists.galaxyproject.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.galaxyproject.org/listinfo/galaxy-dev
> or, via email, send a message with subject or body 'help' to
> galaxy-dev-requ...@lists.galaxyproject.org
>
> You can reach the person managing the list at
> galaxy-dev-ow...@lists.galaxyproject.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of galaxy-dev digest..."
>
>
> HEY!  This is important!  If you reply to a thread in a digest, please
> 1. Change the subject of your response from "Galaxy-dev Digest Vol ..." to 
> the original subject for the thread.
> 2. Strip out everything else in the digest that is not part of the thread you 
> are responding to.
>
> Why?
> 1. This will keep the subject meaningful.  People will have some idea from 
> the subject line if they should read it or not.
> 2. Not doing this greatly increases the number of emails that match search 
> queries, but that aren't actually informative.
>
> Today's Topics:
>
>1. Re: How galaxy workflow work ? (John Chilton)
>2. Re: Metadata error in uploading files to data libraries
>   (John Chilton)
>3. Re: package_numpy_1_7 can't be installed (Wang, Yu)
>4. Re: "join" produces an import error (Wang, Yu)
>5. Re: package_numpy_1_7 can't be installed (Björn Grüning)
>
>
> --
>
> Message: 1
> Date: Mon, 15 Dec 2014 14:24:13 -0500
> From: John Chilton 
> To: Bongsoo Park 
> Cc: galaxy-dev 
> Subject: Re: [galaxy-dev] How galaxy workflow work ?
> Message-ID:
> 
> Content-Type: text/plain; charset=UTF-8
>
> Somehow your user object is not being set. Are you sure you have
> initialized bioblend with the correct API key? Do other operations
> work with that key?
>
> -John
>
> On Wed, Dec 3, 2014 at 11:00 AM, Bongsoo Park  wrote:
>> Hi All,
>>
>> I faced a problem after I tested the Bio-Blend API. I cannot access the
>> history anymore with the below error messages. Although, I tried
>> installation from the scratch, but I got the same error message. Please let
>> me know how to fix this problem. Thanks!
>>
>> (GALAXY User Interface)
>>
>> ===
>>
>>   "xhr": {
>>
>> "readyState": 4,
>>
>> "responseText": "{\"err_msg\": \"Uncaught exception in exposed API
>> method:\", \"err_code\": 0}",
>>
>> "responseJSON": {
>>
>>   "err_msg": "Uncaught exception in exposed API method:",
>>
>>   "err_code": 0
>>
>> },
>>
>> "status": 500,
>>

Re: [galaxy-dev] [Spam:*****] Re: How galaxy workflow work ?

[John Chilton]

There was definitely a bug where I broke backward compatibility on
workflows in central a couple nights ago - I have patched that and
blend4j's test cases for running workflows now actually cause the
workflows to run again.

There are still some problems with blend4j where it is very particular
about what is in the response - these are blend4j specific issues that
I will try to clear up - I have created a github issue here:

https://github.com/jmchilton/blend4j/issues/26

Sorry about the problems.

-John


On Thu, Dec 18, 2014 at 9:59 AM, John Chilton  wrote:
> There have been a ton of changes to the workflow API - I have been
> trying to preserve backward compatibility though - e.g. I have API
> tests in Galaxy for a bunch of legacy/deprecated features related to
> workflows now.
>
> Do you have a stack trace on the Galaxy side for the "Uncaught
> exception in exposed API method"?
>
> I will run through the blend4j test cases and see they turn up.
>
> -John
>
> On Thu, Dec 18, 2014 at 9:44 AM, Bruijn, Freek de  wrote:
>> I am also having problems using the Galaxy API, with a similar error message 
>> to the one reported by Bongsoo Park a few weeks ago (Uncaught exception in 
>> exposed API method). We noticed it a few days ago and have not found the 
>> cause yet. What we do know:
>> - the problem occurs on several Galaxy servers, including the main instance 
>> at https://usegalaxy.org/;
>> - the problem seems to occur whilte using BioBlend, blend4j, and api Python 
>> scripts;
>> - even some of the blend4j unit tests are failing.
>>
>> One of my colleagues noticed a lot of changes in for example workflows.py 
>> (in lib/galaxy/webapps/galaxy/api): 
>> https://bitbucket.org/galaxy/galaxy-dist/annotate/579d211c2b0fa8ef4195930cb999edbb9a0cf8eb/lib/galaxy/webapps/galaxy/api/workflows.py#cl-117
>>
>> Did something change in the Galaxy API? Should we adjust the libraries and 
>> the programs that use them?
>>
>> Cheers, Freek
>>
>>
>> 
>> From: galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf of 
>> galaxy-dev-requ...@lists.galaxyproject.org 
>> [galaxy-dev-requ...@lists.galaxyproject.org]
>> Sent: Tuesday, December 16, 2014 18:00
>> To: galaxy-dev@lists.galaxyproject.org
>> Subject: galaxy-dev Digest, Vol 102, Issue 17
>>
>> Send galaxy-dev mailing list submissions to
>> galaxy-dev@lists.galaxyproject.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://lists.galaxyproject.org/listinfo/galaxy-dev
>> or, via email, send a message with subject or body 'help' to
>> galaxy-dev-requ...@lists.galaxyproject.org
>>
>> You can reach the person managing the list at
>> galaxy-dev-ow...@lists.galaxyproject.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of galaxy-dev digest..."
>>
>>
>> HEY!  This is important!  If you reply to a thread in a digest, please
>> 1. Change the subject of your response from "Galaxy-dev Digest Vol ..." to 
>> the original subject for the thread.
>> 2. Strip out everything else in the digest that is not part of the thread 
>> you are responding to.
>>
>> Why?
>> 1. This will keep the subject meaningful.  People will have some idea from 
>> the subject line if they should read it or not.
>> 2. Not doing this greatly increases the number of emails that match search 
>> queries, but that aren't actually informative.
>>
>> Today's Topics:
>>
>>1. Re: How galaxy workflow work ? (John Chilton)
>>2. Re: Metadata error in uploading files to data libraries
>>   (John Chilton)
>>3. Re: package_numpy_1_7 can't be installed (Wang, Yu)
>>4. Re: "join" produces an import error (Wang, Yu)
>>5. Re: package_numpy_1_7 can't be installed (Björn Grüning)
>>
>>
>> --
>>
>> Message: 1
>> Date: Mon, 15 Dec 2014 14:24:13 -0500
>> From: John Chilton 
>> To: Bongsoo Park 
>> Cc: galaxy-dev 
>> Subject: Re: [galaxy-dev] How galaxy workflow work ?
>> Message-ID:
>> 
>> Content-Type: text/plain; charset=UTF-8
>>
>> Somehow your user object is not being set. Are you sure you have
>> initialized bioblend with the correct API key? Do other operations
>> work with that key?
>>
>> -John
>>
>> On Wed, Dec 3, 2014 at 11:00 AM, Bongsoo Park  wrote:
>>&g

Re: [galaxy-dev] Error when opening bowtie2 tool page

[John Chilton]

Okay - so there are a bunch of issues here. To me the highest priority
is not being able to even load and run the old versions of the tool. I
have opened a pull request with a fix for that here
(https://bitbucket.org/galaxy/galaxy-central/pull-request/620/stable-dont-choke-on-tool-versions#comment-4374040).
I would expect within a couple days it is back-ported to the
latest_2014.10.06 tag of galaxy-dist.

There is also the problem of the duplicated tools. For me that goes
away after one restarts Galaxy (unless the tools aren't in a tool
section). Can you verify that restarting Galaxy fixes the problem or
that you have installed them outside a tool section (like they aren't
is a Picard section on the side or something like that). The
non-grouping of non-sectioned parameters is a known issue being
tracked here -
https://trello.com/c/l9bjV0vT/1973-tool-grouping-doesn-t-work-outside-of-tool-sections
- I don't think it has seen much movement because we generally install
tools into sections on usegalaxy.org.

The issue with some of the picard tools appearing twice is a problem
but we don't really have a way to resolve it. The fully qualified id
changed as part of a big update to the picard wrappers - some tools
have new "owners" in the tool shed. I don't think we will change the
id again - so hopefully it is not a problem going forward (and we will
try to limit this problem from happening again if we can). If you
don't want to uninstall the older versions that are by themselves -
one fix would be to open your shed_tool_conf.xml file and add a
hidden="true" tag to each - this should allow workflows and tool
re-running to still work without actually causing the tool to appear
in the tool panel. This is how we are planning to resolve the problem
on our servers. This is less than ideal - there is more about this
topic here https://github.com/galaxyproject/tools-devteam/pull/34 if
you are curious.

I am very sorry about all of these problems - certainly these are all
still very active development priorities (how to display multiple
versions, dealing with tool lineages, etc...) and I am confident there
will be improvements to all of this over the next year.

Let me know if I missed something or if there are still outstanding problems.

-John

On Thu, Dec 18, 2014 at 8:29 AM, Anna Terry  wrote:
> Hi again,
> I have just updated Picard, and am getting a similar error with some tools
> from that repository when I switch to older version from tool page.
> I also noticed that some tools from the Picard repo appear as multiple
> entries for different in the tool panel rather than selecting the different
> versions from the drop down.
> These errors are all on tools with version dropdowns, however I don't get
> these errors on all of the version dropdowns, for instance I can switch to
> an older version of fastqc no problem.
> So are the new versions of tools breaking old versions, or is there a
> problem with the tool panel?
>
> Any help appreciated!!
>
>
> SamToFastq:
> URL: http://../tool_runner/index
> File
> '/home/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py',
> line 364 in respond
>   app_iter = self.application(environ, detect_start_response)
> File
> '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py',
> line 84 in __call__
>   return self.application(environ, start_response)
> File
> '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/remoteuser.py',
> line 107 in __call__
>   return self.app( environ, start_response )
> File
> '/home/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py',
> line 633 in __call__
>   return self.application(environ, start_response)
> File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 132
> in __call__
>   return self.handle_request( environ, start_response )
> File '/home/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 190
> in handle_request
>   body = method( trans, **kwargs )
> File
> '/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/tool_runner.py',
> line 90 in index
>   template, vars = tool.handle_input( trans, params.__dict__ )
> File '/home/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py', line 2136 in
> handle_input
>   errors, params = self.__check_param_values( trans, expanded_incoming,
> expanded_state, old_errors, process_state, history=history, source=source )
> File '/home/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py', line 2251 in
> __check_param_values
>   errors = self.update_state( trans, inputs, state.inputs, incoming,
> old_errors=old_errors or {}, source=source )
> File '/home/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py', line 2514 in
> update_state
>   group_state = state[input.name]
> KeyError: 'single_paired_end_type'
>
>
> ReorderSam:
> URL: http://../tool_runner/index
> File
> '/home/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py',
> line 364 in respond
>   app_iter = s

Re: [galaxy-dev] Metadata error in uploading files to data libraries

[John Chilton]

So I have been able to reproduce this problem with
outputs_to_working_directory - though I am not sure how to address it
I have marked the Trello card as a confirmed bug.

The problem with shell=True is still confusing me - is not a normal
executable on your PATH? I feel like modifying your PATH to make sure
samtools is properly on it in someway is a better fix than sticking
shell=True in there. I'll think about it though - certainly a lot of
people get tripped up by the samtools requirement and Galaxy isn't
great a reporting the nature of the problem.

-John

On Thu, Dec 18, 2014 at 7:51 AM, Jelle Scholtalbers
 wrote:
> Hi John and others,
>
> The "OSError: [Errno 2] No such file or directory" is solved when putting
> the "shell=True" in the mentioned locations - as written in the Trello card.
>
> However, even after fixing that, linked BAM files where not indexed, or at
> least the "files/_metadata_files/000/metadata_XXX.dat" is empty on job
> completion. After spending too much time on this, I nailed down the problem
> to this setting:
> outputs_to_working_directory = True
>
> With this option set, the BAM index file is never moved to the right
> location (or is is not being created.)
> Setting this option to False (I think the setting was a left over from a
> failed attempt to use "real user job submission") this now just works.
>
> - Jelle
>
>
> On Mon, Dec 15, 2014 at 8:31 PM, John Chilton  wrote:
>>
>> Hello,
>>
>>   Sorry we haven't made progress on this - and thanks for creating a
>> Trello card (https://trello.com/c/tw75nq1U).
>>
>>   It looks like samtools is not on your path, can you use it from the
>> command-line? If not you should probably install it - I would suggest
>> install samtools 0.1.19 with homebrew (looks like you are a on Mac).
>> If you can run samtools from the command-line - can you do a which
>> `samtools` from your command-line and see where it is coming from and
>> then add that directory explicitly to your Galaxy PATH - say at the
>> top of run.sh in your Galaxy root (let me know if you need more
>> details on that).
>>
>> -John
>>
>> On Tue, Dec 9, 2014 at 3:02 AM, Jelle Scholtalbers
>>  wrote:
>> > Hi all,
>> >
>> > after an update to the following changeset(14859:7ba05957588a, stable,
>> > 05.12.14), our bam files that are uploaded(linked) to a data library,
>> > are no
>> > longer indexed. The metadata_xxx.dat is created, but it stays empty.
>> > The following error message appears in the log, although the state of
>> > the
>> > dataset is 'ok':
>> >
>> > galaxy.jobs WARNING 2014-12-05 12:47:02,218 Error accessing
>> > /g/K/K27.bam,
>> > will retry: [Errno 1] Operation not permitted: '/g/K/K27.bam'
>> > galaxy.jobs WARNING 2014-12-08 13:38:57,045 Error accessing /g/K/K2.bam,
>> > will retry: [Errno 1] Operation not permitted: '/g/K/K2.bam'
>> >
>> > All file permissions are correct (i.e. galaxy owns them). Furthermore,
>> > executing samtools index, just works on those files:
>> > samtools index /g/K/K2.bam
>> > /g/galaxy/galaxy_data/files/_metadata_files/006/metadata_6598.dat
>> >
>> > When uploading the file - "copy files into galaxy" - the samtools index
>> > just
>> > works.
>> >
>> > ==
>> >
>> > Now, on a clean local install(14874:885f940bff64, stable, 05.12.14) and
>> > samtools installed globally and with the bam file sorted, I get the
>> > following situation:
>> > When I try to upload this bam to a data library by linking the following
>> > error is shown on the dataset (note: here the dataset is set in error
>> > state,
>> > which does not happen on our server)
>> >
>> > Uploaded by: schol...@embl.de
>> > Date uploaded: Mon Dec 8 17:42:39 2014 (UTC)
>> > File size: 2.6 GB
>> > UUID: d23cf11a-0372-41cb-939a-7c8761d78b73
>> > Data type: auto
>> > Build: ?
>> > Miscellaneous information: Traceback (most recent call last): File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
>> > line 407, in __main__() File
>> >
>> > "/Users/scholtalbers/workspace/galaxy-dist-new/tools/data_source/upload.py",
>> > line 396, in _
>> > Job Standard Error
>> >
>> > Traceback (most recent call last):
>> >   File
>> >
>> > "/Users/scholtalbers/workspace/galaxy

Re: [galaxy-dev] Stop autoselecting in drop-down menus

[John Chilton]

Sorry for the delay - I think I understanding what you are saying now.

Are you connecting "inputs" to your datasets (they are at the botton
of the workflow editor menu and many people miss them)? It is much
more clear what the workflow runner has to do with these kind of
inputs.

Short of that though - I think more people than not consider it a
really nice feature that the inputs are prepopulated with items in the
history that match that input. If you have just a couple datasets in
in your history - it makes workflow running quicker and easier.

So unless there is an outcry from others that this is needed - I think
the devteam is not going to add this. (Though by all means others
should chime in and let me know if this would be a good feature and I
am wrong.)

If you have some developer time to devote to this - I would be happy
to walk you through where to look to modify this feature locally (with
the caveat that Sam is going to make big changes to the workflow run
page over the next few months so those diffs would like become
conflicted quickly.)

-John


On Thu, Oct 30, 2014 at 4:47 AM, Aleksey Jironkin
 wrote:
> Hi,
>
> Thanks for your reply. I don;t think I have described well exactly what
> happened. The autoselect of the top option happens in all different
> parameters.
>
> Because there is no "empty line" option in the select box, it "has" to
> select top element (whatever that may be). The elements get populated at
> run time so, if there has been new datasets generated with the same
> type, they will be populated above (seemingly) existing datasets. Thus
> making them autoselected when workflows run not necessarily the same
> workflow, but also different workflows altogether.
>
> Current behaviour automatically pre-select a value in drop down box, and
> I was wondering if we can insert:
>
> 
>
> Or something along those lines as a default for any dropdown, and the
> user not allowed to leave the dropdown box with "blank" value, which
> then forces the user to explicitely select something for that parameter.
>
>
>
> Alex
>
> On Wed, 2014-10-29 at 16:26 -0400, John Chilton wrote:
>> On Wed, Oct 29, 2014 at 7:45 AM, Aleksey Jironkin
>>  wrote:
>> >
>> > Hi,
>> >
>> > I was wondering if there is a way to prevent auto-selecting an
>> option
>> > for Running a workflow? Quiet often users (especially novices) would
>> > re-run workflows without checking much the data that goes in
>> thinking it
>> > will be the same as before. But because the options in drop down
>> menus
>> > are auto-populated it seems the top option is always selected by
>> > default. For example, previously we had a single fasta file as
>> reference
>> > and in the process of a workflow few other fasta files get
>> generated.
>> > they appear above the previous fasta in the selection, so without
>> paying
>> > attention people skip the input without really checking it.
>>
>> This behavior would be a bug - workflow parameters should be
>> preserved. What release are you on - we fixed a bug related to this
>> after the initial August release
>> (https://bitbucket.org/galaxy/galaxy-central/commits/5abd9fc76f7ad2eca013a1f81a77a4d3fe28328a)
>>  that should be available in the latest august stable tag.
>>
>> Is it possible you were on the August release but don't have this
>> fix?
>>
>> If it is a bug and is still present in the latest stable branch if you
>> could provide steps to reproduce it in a clean clone of Galaxy it
>> could go a long way toward helping us track down a fix.
>>
>> -John
>>
>> >
>> >
>> > Can we manually add a "blank" selection at the top to force users to
>> > select something?
>> >
>> >
>> >
>> > Alex
>> >
>> >
>> > --
>> > Dr Aleksey Jironkin
>> >
>> > Bioinformatician
>> > Bioinformatics Unit
>> > Infectious Disease Informatics
>> > Microbiologicy Services, Colindale
>> > Public Health England
>> > 61 Colindale Avenue
>> > London
>> > NW9 5EQ
>> >
>> > aleksey.jiron...@phe.gov.uk
>> >
>> > Tel: 020 83 276610
>> >
>> > www.gov.uk/phe Follow us on Twitter @PHE_uk
>> >
>> > Protecting and improving the nation’s health
>> >
>> >
>> >
>> > ___
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client. To manage your subscriptio

Re: [galaxy-dev] HOWTO share tool parameter settings?

[John Chilton]

Pieter -

Sorry for the delay - I am not sure how to respond - but I am trying
to get through my e-mail backlog before winter break.

I think this could be kind of cool - stuff has been added to the top
of that tool run screen lately and probably more will be added soon. I
think an option to basically take the JSON that an API request would
take and populate the form would be possible and a lot easier now that
Sam has rewritten the form in JavaScript so everything can be more
easily dynamic.

We had a tool developer meeting (which we probably should have
announced here on -dev instead of just on github in retrospect) and
one of the ideas that came out of that was the option to take an
existing tool run and basically extract a tool test XML block to stick
into the tool - I imagine the mechanism for doing tool parameter sets
could be similar.

It would also be something useful for disseminating say properties in
papers that is lighter than whole workflows or histories.

So I like it - but it is probably not a top priority for the devteam
at this time - happy to review pull requests though of course.

-John


On Thu, Oct 16, 2014 at 4:15 AM, Lukasse, Pieter  wrote:
> Hi John,
>
> I see that in my last email I replied to you with the fourth (and not the 
> third) option in mind :(.
>
> Just to come back to this related to the third option (i.e. " Have a dummy 
> tool to produce a settings file and allow the users to choose this file when 
> running the real tool"): I don't like this option so much because the file is 
> never as clear as the tool form that produced it. If users are used to 
> looking at the tool form for seeing details about the set parameters and now 
> they will have to look at a text file, this will be confusing. It is not a 
> huge problem, but it is one that I am trying to avoid. I.e. it would be best 
> if users are presented with only one way of checking parameters of executed 
> processes.
>
> In the scenario I am proposing, the settings files are disseminated just as 
> they are (e.g. a text file) either by sharing them in the normal ways or by 
> placing a number of them in pre-defined folders in Galaxy. The tool form 
> would allow the user to select one and would then fill in the values of the 
> parameters accordingly. I'm guessing similar binding logic is also happening 
> right now when a user clicks "rerun" on a step in his history.
>
> Best regards,
>
> Pieter.
>
> -Original Message-
> From: John Chilton [mailto:jmchil...@gmail.com]
> Sent: donderdag 9 oktober 2014 15:45
> To: Lukasse, Pieter
> Cc: galaxy-...@lists.bx.psu.edu
> Subject: Re: [galaxy-dev] HOWTO share tool parameter settings?
>
> How would this be different then the third option you listed?
>
> You want it to work with all tools and you as the developer want to be able 
> to construct these files without needing a dummy tool to produce the values?
>
> How would imagine these setting files would be disseminated to users and then 
> selected by users?
>
> -John
>
>
> On Thu, Oct 9, 2014 at 9:34 AM, Lukasse, Pieter  wrote:
>> Hi,
>>
>>
>>
>> Do we have a way (or plans) for sharing tool parameter settings in Galaxy?
>>
>>
>>
>> I know the following workarounds :
>>
>>
>>
>> · Share a history with all users: so users can import your step and
>> do “rerun” to run on their own file with your settings
>>
>> · Wrap the step in a workflow with all parameters set and publish
>> this workflow: users can run this “workflow”
>>
>> · Have a dummy tool to produce a settings file and allow the users
>> to choose this file when running the real tool
>>
>> · Use a conditional and many macros that are basically a copy of
>> each other, only differing in the parameter values
>>
>>
>>
>>
>>
>> But what I would like to have is a way to define bindings between a
>> settings file and the parameters in the tool form. Any plans, ideas?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Pieter Lukasse
>>
>> Wageningen UR, Plant Research International
>>
>> Department of Bioinformatics (Bioscience)
>>
>> Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB,
>> Wageningen, the Netherlands
>>
>> T: +31-317481122;
>> M: +31-628189540;
>> skype: pieter.lukasse.wur
>>
>> http://www.pri.wur.nl
>>
>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this and oth

Re: [galaxy-dev] Job wrapper object

[John Chilton]

Sorry for the long delay on this. You can get access to the job id and
perhaps metric information of a previous job by doing

$input.dataset.creating_job

That will give the job model object and from there you can use
.metrics to fetch the metrics. I consider it breaking Galaxy
abstractions so be careful if you decide to use it - we may break that
at some point.

-John


On Tue, Oct 7, 2014 at 7:13 PM, Alexandre Defelicibus
 wrote:
> Hi Lyad and John,
>
> Thanks for your replies and sorry for my no detailed question.
>
> Well, I'm developing a tool that is applied to analyse PDB files (proteomics
> stuffs) ,compare them, and create a ranking sort by the best solutions (PDB
> files), making some graphs and 3D images. This tool will have as output a
> HTML file with all these generated files.
>
> I would like to access and use the information about the job, like ID and
> metrics for example, and write on this HTML output file. So, to do that, I
> think, I need to be able to use the Galaxy lib, but I couldn't create
> objects using the available class, like JobWrapper. Maybe I'm looking to the
> wrong place to take these information.
>
> Is it possible?
>
> I hope I was more detailed.
>
> Best regards,
>
> Alexandre
>
>
> 2014-10-06 22:32 GMT-03:00 John Chilton :
>
>> Yes - I am with Iyad - having more context would probably help :). You
>> can inject job_wrapper related stuff into tool execution in a very
>> round about way perhaps by creating a dynamic job destination that
>> consumes the job_wrapper and then creates a job destination with
>> enviornment variables set to whatever it is you job needs. But that
>> solution is not very portable or simple - I can try to walk through
>> the steps in more detail though if it would be useful.
>>
>> -John
>>
>> On Mon, Oct 6, 2014 at 9:28 PM, Kandalaft, Iyad
>>  wrote:
>> > Tell us what you are trying to do and maybe someone might know the way
>> > to solve it.  It could be easier, the same, or more difficult than you
>> > imagined.
>> >
>> > Iyad Kandalaft
>> > Bioinformatics Programmer
>> > Microbial Biodiversity Bioinformatics
>> > Science & Technology Branch
>> > Agriculture & Agri-Food Canada
>> > iyad.kandal...@agr.gc.ca | (613) 759-1228
>> > 
>> > From: galaxy-dev-boun...@lists.bx.psu.edu
>> > [galaxy-dev-boun...@lists.bx.psu.edu] on behalf of Alexandre Defelicibus
>> > [adefelici...@gmail.com]
>> > Sent: October 6, 2014 4:50 PM
>> > To: 
>> > Subject: [galaxy-dev] Job wrapper object
>> >
>> > Hi all,
>> >
>> > I'm developing some tools on my Galaxy instance and I need to get some
>> > information about the job that a tool creates.
>> >
>> > The question is, how can I have a job_wrapper object in my tool wrapper?
>> > I tried to use the reserved variable $__app__, but I got errors.
>> > I couldn't import and use the galaxy lib in order to get the job
>> > information.
>> >
>> > Could someone show me the correct way to do that, with some examples?
>> >
>> > I appreciate all the help.
>> >
>> > Best regards,
>> >
>> > --
>> > Alexandre Defelicibus
>> > Mestrando em Bioengenharia
>> > Programa de Pós-Graduação em Bioengenharia
>> > Universidade de São Paulo - USP
>> >
>> > ___
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client.  To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> >   http://lists.bx.psu.edu/
>> >
>> > To search Galaxy mailing lists use the unified search at:
>> >   http://galaxyproject.org/search/mailinglists/
>
>
>
>
> --
> Alexandre Defelicibus
> Mestrando em Bioengenharia
> Programa de Pós-Graduação em Bioengenharia
> Universidade de São Paulo - USP
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problem with LWR (input size limitation)

[John Chilton]

I do have a test case that should exercise interesting UTF-8 stuff.

Perhaps it would help to narrow down whether this is a problem of the
test case being too weak or my system having some sort of
configuration that is different than yours. Can you try to run

python run_client_tests.py --url=http://localhost:8913

(replacing the URL with whatever it should actually be based on your
setup.) Does that pass or fail? It should cause some logging on the
LWR side but if the above call complete without throwing an error than
the test case passed.

Thanks for your continued patience and sorry I have not been able to
reproduce the problem.

-John


On Mon, Dec 8, 2014 at 3:48 AM, Misharl Monsoor  wrote:
> Hi John,
>
> Thanks again for you help. So i place "sys.setdefaultencoding('utf-8')" at
> the top of the "lwr/web/framework.py", and i still have this bug.
>
> I use Python version "2.7 64 bits" to executes LWR on the windows server,
> and i encounter a "common" problem with the "setdefaultencoding" function
> which can't be loaded when doing "import sys" and
> "sys.setdefaultencoding('utf-8')". The hack that i found was to do
> "reload(sys)" then "sys.setdefaultencoding('utf-8')". But i still get the
> "utf8' codec can't decode byte" bug. Can it come from my input data which
> contains utf8 characters?
>
> Thank you very much,
>
> Bests regards,
>
> Misharl Monsoor
>
>
>
> Le 04/12/2014 06:25, John Chilton a écrit :
>
>> I have had one other report of this error previously - but I have not
>> been able to reproduce it. I have tried so hard to understand these
>> unicode/utf-8/Python 2 v 3/etc... issues and I must admit I simply
>> don't :(.
>>
>> Reading through http://farmdev.com/talks/unicode/ I found a hack that
>> might work, if you place:
>>
>> import sys
>> sys.setdefaultencoding('utf-8')
>>
>> at the top of lwr/web/framework.py does that fix the problem?
>>
>> -John
>>
>> On Wed, Nov 26, 2014 at 2:53 AM, Misharl Monsoor 
>> wrote:
>>>
>>> Hi John,
>>>
>>> I can confirm you that your advice putting "LWR_CURL_TRANSPORT=1" works
>>> very
>>> fine, thanks a lot! Sorry, i haven't expressed myself correctly for my
>>> second question, thanks for answering, it helps me.
>>> I have another problem with LWR, when executing the jobs on the Windows
>>> Server, i have the following bug (it happens a lot of time, and i must
>>> restart the LWR server in order for it to works):
>>>
>>>   File
>>> "C:\Python2764\lib\site-packages\paste-1.7.5.1-py2.7.egg\paste\httpserver
>>> py", line 287, in wsgi_execute
>>> self.wsgi_start_response)
>>>   File "C:\Desktop\lwr_master\lwr\web\framework.py", l
>>> ne 37, in __call__
>>> return controller(environ, start_response, **request_args)
>>>   File "C:\Desktop\lwr_master\lwr\web\framework.py", l
>>> ne 141, in controller_replacement
>>> resp = self.__build_response(result)
>>>   File "C:\Desktop\lwr_master\lwr\web\framework.py", l
>>> ne 129, in __build_response
>>> resp = Response(body=self.body(result))
>>>   File "C:\Desktop\lwr_master\lwr\web\framework.py", l
>>> ne 155, in body
>>> body = dumps(result)
>>>   File "C:\Python2764\lib\json\__init__.py", line 243, in dumps
>>> return _default_encoder.encode(obj)
>>>   File "C:\Python2764\lib\json\encoder.py", line 207, in encode
>>> chunks = self.iterencode(o, _one_shot=True)
>>>   File "C:\Python2764\lib\json\encoder.py", line 270, in iterencode
>>> return _iterencode(o, 0)
>>> nicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 718:
>>> invalid
>>> start byte
>>>
>>>
>>> Bests regards and thank you very much for your help,
>>>
>>> Misharl Monsoor.
>>>
>>>
>>>
>>> Le 21/11/2014 17:44, John Chilton a écrit :
>>>
>>>> On Wed, Nov 19, 2014 at 12:58 PM, Misharl Monsoor
>>>>  wrote:
>>>>>
>>>>> Hi John ,
>>>>>
>>>>> First all, thank you very much for your quick reply. I will test your
>>>>> suggestion about pycurl,  and tell you if it has worked. For the second
>>>>> question, i was wondering if it is possible with LWR to define a 

Re: [galaxy-dev] [Spam:*****] Re: How galaxy workflow work ?

[John Chilton]

That is odd. Does that workflow run through the UI? It looks like it
was missing a tool during import. Certainly the API should indicate
that overtly instead of failing in this fashion.

If it does run, can you export it to JSON and send it to me?

-John

On Thu, Dec 18, 2014 at 10:55 AM, Ruslan Forostianov  wrote:
> galaxy.web.framework.decorators ERROR 2014-12-18 12:22:04,760 Uncaught
> exception in exposed API method:
> Traceback (most recent call last):
>   File "/home/galaxy/galaxy-dist/lib/galaxy/web/framework/decorators.py",
> line 244, in decorator
> rval = func( self, trans, *args, **kwargs)
>   File
> "/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/api/workflows.py", line
> 231, in create
> populate_state=True,
>   File "/home/galaxy/galaxy-dist/lib/galaxy/workflow/run.py", line 18, in
> invoke
> modules.populate_module_and_state( trans, workflow,
> workflow_run_config.param_map )
>   File "/home/galaxy/galaxy-dist/lib/galaxy/workflow/modules.py", line 843,
> in populate_module_and_state
> step_errors = module_injector.inject( step, step_args=step_args )
>   File "/home/galaxy/galaxy-dist/lib/galaxy/workflow/modules.py", line 830,
> in inject
> state, step_errors = module.compute_runtime_state( trans, step_args )
>   File "/home/galaxy/galaxy-dist/lib/galaxy/workflow/modules.py", line 264,
> in compute_runtime_state
> state = self.decode_runtime_state( trans, step_updates.pop( "tool_state"
> ) )
> KeyError: 'tool_state'
>
> Cheers,
> Ruslan Forostianov
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problem using FastQC in Galaxy

[John Chilton]

This looks like a bug - but unfortunately it appears you are running a
fairly old version of Galaxy  - I don't think the file
lib/galaxy/util/shed_util.py has existed for the better part of a
year. It should be possible to get the FastQC wrapper to work with the
old version if you figure out the exact version of fastqc you need -
but if you want to use the tool shed you will probably want to upgrade
to the latest stable release of Galaxy.

-John

On Mon, Jan 5, 2015 at 9:17 AM, Waldron, Michael H
 wrote:
> I attempted to install fastqc via the tool shed, and got an error. The log
> information is listed below. The main error appears to be "too many values
> to unpack".
>
> 152.2.204.5 - - [05/Jan/2015:09:09:00 -0400] "GET
> /admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/&repository_ids=ca249a25748b71a3&changeset_revisions=0b201de108b9
> HTTP/1.1" 500 -
> "https://toolshed.g2.bx.psu.edu/repository/preview_tools_in_changeset?repository_id=ca249a25748b71a3&changeset_revision=0b201de108b9";
> "Mozilla/5.0 (X11; Linux x86_64; rv:34.0) Gecko/20100101 Firefox/34.0"
> Error - : too many values to unpack
> URL:
> https://galaxy.its.unc.edu/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/&repository_ids=ca249a25748b71a3&changeset_revisions=0b201de108b9
> File
> '/nas02/apps/galaxy-prod/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/exceptions/errormiddleware.py',
> line 143 in __call__
>   app_iter = self.application(environ, start_response)
> File
> '/nas02/apps/galaxy-prod/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/recursive.py',
> line 80 in __call__
>   return self.application(environ, start_response)
> File
> '/nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/web/framework/middleware/remoteuser.py',
> line 91 in __call__
>   return self.app( environ, start_response )
> File
> '/nas02/apps/galaxy-prod/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py',
> line 632 in __call__
>   return self.application(environ, start_response)
> File '/nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/web/framework/base.py',
> line 132 in __call__
>   return self.handle_request( environ, start_response )
> File '/nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/web/framework/base.py',
> line 185 in handle_request
>   body = method( trans, **kwargs )
> File
> '/nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/web/framework/__init__.py',
> line 216 in decorator
>   return func( self, trans, *args, **kwargs )
> File
> '/nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py',
> line 1345 in prepare_for_install
>   shed_util.get_dependencies_for_repository( trans, tool_shed_url,
> repo_info_dict, includes_tool_dependencies )
> File '/nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/util/shed_util.py',
> line 601 in get_dependencies_for_repository
>   installed_rd, missing_rd =
> get_installed_and_missing_repository_dependencies_for_new_install( trans,
> repo_info_tuple )
> File '/nas02/apps/galaxy-prod/galaxy-dist/lib/galaxy/util/shed_util.py',
> line 715 in
> get_installed_and_missing_repository_dependencies_for_new_install
>   tool_shed, name, owner, changeset_revision = rd_tup
> ValueError: too many values to unpack
>
> Configuration
> -
>   __file__: '/nas02/apps/galaxy-prod/galaxy-dist/universe_wsgi.ini'
>   admin_users:
> 'roac...@email.unc.edu,kel...@email.unc.edu,mwald...@email.unc.edu,jen...@email.unc.edu'
>   allow_user_dataset_purge: 'True'
>   database_connection: 'postgres://galaxy@localhost/galaxy'
>   database_engine_option_server_side_cursors: 'True'
>   database_engine_option_strategy: 'threadlocal'
>   debug: 'False'
>   default_cluster_job_runner: 'drmaa://-q pgalaxy_q -M 4190 /'
>   enable_quotas: 'True'
>   enable_tracks: 'True'
>   file_path: '/proj/galaxy'
>   ftp_upload_dir: 'database/ftp'
>   ftp_upload_site: 'galaxy.its.unc.edu:2021'
>   here: '/nas02/apps/galaxy-prod/galaxy-dist'
>   job_working_directory: '/proj/galaxy/job_working_directory'
>   len_file_path: 'tool-data/shared/ucsc/chrom'
>   library_import_dir: '/proj/seq/galaxy'
>   local_job_queue_workers: '10'
>   nglims_config_file: 'tool-data/nglims.yaml'
>   remote_user_maildomain: 'email.unc.edu'
>   retry_job_output_collection: '30'
>   start_job_runners: 'drmaa'
>   static_cache_time: '360'
>   static_dir: '/nas02/apps/galaxy-prod/galaxy-dist/static/'
>   static_enabled: 'True'
>   static_favicon_dir:
> '/nas02/apps/galaxy-prod/galaxy-dist/static/favicon.ico'
>   static_images_dir: '/nas02/apps/galaxy-prod/galaxy-dist/static/images'
>   static_scripts_dir: '/nas02/apps/galaxy-prod/galaxy-dist/static/scripts/'
>   static_style_dir:
> '/nas02/apps/galaxy-prod/galaxy-dist/static/june_2007_style/blue'
>   tool_config_file: 'tool_conf.xml,shed_tool_conf.xml'
>   tool_dependency_dir: '../tool_dependencies'
>   tool_path: 'tools'
>   use_interactive: 'False'
>   use_nglims: 'False'
>   use_remote_user: 'True'
>
>
> WSGI Variables
> 

Re: [galaxy-dev] run_test.sh - ImportError: cannot import name _args_from_interpreter_flags

[John Chilton]

Hey Ryan,

This happens to me every time (or nearly every time) I run tests as
well and with no bearing on whether the test succeeds or not - so I
suspect this is not a serious problem. I am not sure what the problem
is but it seems to be a bug in Python itself.

Are you testing specific tools or Galaxy itself? If you are testing
tools - sometimes it helps to `export GALAXY_TEST_VERBOSE_ERRORS=True`
which can cause more details to be printed when things are failing.

-John

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
  File 
"/home/john/workspace/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/mixins.py",
line 165, in run
errors = (self.connection.connection_errors +
  File 
"/home/john/workspace/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/utils/__init__.py",
line 309, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
  File 
"/home/john/workspace/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/connection.py",
line 819, in connection_errors
return self.transport.connection_errors
  File 
"/home/john/workspace/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/connection.py",
line 782, in transport
self._transport = self.create_transport()
  File 
"/home/john/workspace/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/connection.py",
line 518, in create_transport
return self.get_transport_cls()(client=self)
  File 
"/home/john/workspace/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/connection.py",
line 524, in get_transport_cls
transport_cls = get_transport_cls(transport_cls)
  File 
"/home/john/workspace/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/transport/__init__.py",
line 108, in get_transport_cls
_transport_cache[transport] = resolve_transport(transport)
  File 
"/home/john/workspace/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/transport/__init__.py",
line 92, in resolve_transport
return symbol_by_name(transport)
  File 
"/home/john/workspace/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/utils/__init__.py",
line 92, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
  File 
"/home/john/workspace/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/transport/sqlalchemy/__init__.py",
line 13, in 
from kombu.transport import virtual
  File 
"/home/john/workspace/galaxy-central/eggs/kombu-3.0.13-py2.7.egg/kombu/transport/virtual/__init__.py",
line 19, in 
from multiprocessing.util import Finalize
  File "/usr/lib/python2.7/multiprocessing/__init__.py", line 65, in 
from multiprocessing.util import SUBDEBUG, SUBWARNING
  File "/usr/lib/python2.7/multiprocessing/util.py", line 41, in 
from subprocess import _args_from_interpreter_flags
ImportError: cannot import name _args_from_interpreter_flags

On Thu, Dec 25, 2014 at 3:04 PM, Ryan G  wrote:
> I'm trying to run the functional tests my local installation, and see this
> error come up when the test web server comes up.  Google didn't turn up much
> other than that other people see this.
>
> Most of my tests then proceed to fail so I suspect this is pretty important.
> How do I get around this?  Is there a solution?
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Mercurial is out of date for Toolshed

[John Chilton]

It is certainly the case that Galaxy should be using the latest
version of mercurial given a number of high profile bugs with older
versions. Unfortunately it doesn't seem possible to just drop the new
version it - there are some API changes that prevent Galaxy from
loading when doing this and the number of people who can add new eggs
to Galaxy is low.

I have created a Trello card to track this issue:

https://trello.com/c/9A9uIav0

Let us know if you happen to find a workaround for this issue.

-John


On Mon, Dec 22, 2014 at 2:55 PM, Ryan G  wrote:
> I've been trying to track down why I can't get anything from the toolshed
> installed and finally have it figured out.
>
> Whenever I tried to install anything I always got an Error with no
> explanation of what the error was.  After enabling Debug messages into the
> log file, I see the error is:
>
> tool_shed.util.hg_util DEBUG 2014-12-22 14:47:48,910 Error cloning
> repository: httpsconnection instance has no attribute '_set_hostport'
>
> I googled around and found out this is a known bug/issue with older version
> of Mercurial and was fixed in v3.
>
> I added a line to hg_util.py to see where it picks up hg.  Its using version
> 2.2.3.  Indeed, one of the eggs downloaded by Galaxy is mercurial-2.2.3.
>
> I have the newest version of mercurial installed in my site-packages folder
> but I guess that's not what galaxy wants.  So my question is, how do I get
> Galaxy to use the latest version of Mercurial?  And, Why did it download an
> older version?
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] [Spam:*****] Re: How galaxy workflow work ?

[John Chilton]

That post looks like it might be wrong. If 21 is the step id of the
tool (edgeR) then ds_map should probably reference steps 19 and 20
right?

Also - I will see what I can do about improving these workflow API
error messages - it is never exactly clear to me what is wrong when
there are problems.

-John

On Fri, Dec 19, 2014 at 4:21 PM, Ruslan Forostianov  wrote:
> Hi John,
>
> I did some investigation.
> I added logging statement to line 843 of lib/galaxy/workflow/modules.py
> file:
>
> for step in workflow.steps:
> step_args = param_map.get( step.id, {} )
> log.info('step.id=' + str(step.id) + ', step_args=' +
> str(step_args))
> step_errors = module_injector.inject( step, step_args=step_args )
>
> It gives:
> galaxy.workflow.modules INFO 2014-12-18 13:39:31,464 step.id=21,
> step_args={'contrast': u'Female-Male'}
>
> See http post below. It looks like somehow runtime parameter appears among
> steps.
>
> 4 * Client out-bound request
> 4 > POST
> http://galaxy.thehyve.net/api/workflows?key=5c4f870bc0108a60dc7435a000fb7874
> 4 > Content-Type: application/json
> 4 > Accept: application/json
> {"parameters":{"21":{"contrast":"Female-Male"}},"workflow_id":"d4bec7f9d7e8cf4d","history":"hist_id=59bd7ef52a8ce0e8","ds_map":{"21":{"id":"fb94345e0a2a5a2c","src":"hda"},"22":{"id":"9384bb69863b3f9e","src":"hda"}},"no_add_to_history":"true"}
>
> Here is link to ga file for completeness:
> https://github.com/thehyve/trait_workflow_runner/blob/master/src/main/resources/nl/vumc/biomedbridges/galaxy/rna-seq-dge.ga
>
> On Fri, Dec 19, 2014 at 7:47 PM, John Chilton  wrote:
>>
>> That is odd. Does that workflow run through the UI? It looks like it
>> was missing a tool during import. Certainly the API should indicate
>> that overtly instead of failing in this fashion.
>>
>> If it does run, can you export it to JSON and send it to me?
>>
>> -John
>>
>> On Thu, Dec 18, 2014 at 10:55 AM, Ruslan Forostianov 
>> wrote:
>> > galaxy.web.framework.decorators ERROR 2014-12-18 12:22:04,760 Uncaught
>> > exception in exposed API method:
>> > Traceback (most recent call last):
>> >   File
>> > "/home/galaxy/galaxy-dist/lib/galaxy/web/framework/decorators.py",
>> > line 244, in decorator
>> > rval = func( self, trans, *args, **kwargs)
>> >   File
>> > "/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/api/workflows.py",
>> > line
>> > 231, in create
>> > populate_state=True,
>> >   File "/home/galaxy/galaxy-dist/lib/galaxy/workflow/run.py", line 18,
>> > in
>> > invoke
>> > modules.populate_module_and_state( trans, workflow,
>> > workflow_run_config.param_map )
>> >   File "/home/galaxy/galaxy-dist/lib/galaxy/workflow/modules.py", line
>> > 843,
>> > in populate_module_and_state
>> > step_errors = module_injector.inject( step, step_args=step_args )
>> >   File "/home/galaxy/galaxy-dist/lib/galaxy/workflow/modules.py", line
>> > 830,
>> > in inject
>> > state, step_errors = module.compute_runtime_state( trans, step_args
>> > )
>> >   File "/home/galaxy/galaxy-dist/lib/galaxy/workflow/modules.py", line
>> > 264,
>> > in compute_runtime_state
>> > state = self.decode_runtime_state( trans, step_updates.pop(
>> > "tool_state"
>> > ) )
>> > KeyError: 'tool_state'
>> >
>> > Cheers,
>> > Ruslan Forostianov
>> >
>
>
>
>
> --
> Ruslan Forostianov,
> Software Developer
> www.thehyve.nl
> E rus...@thehyve.nl
> M +31(0)6 83 51 30 81
> Skype foro.ru
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] galaxy.tools.parameters.basic.RuntimeValue not str

[John Chilton]

It looks like the workflow submission code was unable to map your
parameter "reference_genome|index" to the correct tool runtime value.
Is there any chance you can post the tool and/or the workflow? Also -
I am assuming here the tool_id is mapper_easy - is that correct?

-John

On Fri, Jan 2, 2015 at 10:26 AM, Pandori n  wrote:
> Hi all!
> Maybe somebody faced with this problem:
>
> 1) in my wf used tool with:
> "u'model_class': u'Conditional',
>  u'type': u'conditional',
>  u'name': u'reference_genome'}], "
>
> 2) when i run wf via web - all ok, and on a second step:
> "Select a reference genome" is "test" (in view details)
>
> But, when i run wf via api:
>
> param = {u'mapper_easy' :{u'param': u'reference_genome|index', u'value':
> u'test'}}
> nuser.workflows.run_workflow(workflow_id=wf_id, dataset_map=dataset_map,
> params=param, history_id=hist_id)
>
> wf started and on a second step is failed... In details:
> "Select a reference genome" is " object at 0x7f5374266850>"
>
> Any help is appreciated. Thanks!
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Creating a new job handler for Galaxy

[John Chilton]

This sounds like a fun and challenging project - good luck!

The route I would recommend pursuing largely hinges on whether all 5
Galaxy instances have a shared file system and run as a single user.
If they do I would recommend implementing a Galaxy "job runner" - all
the runners bundled with Galaxy can be found in
lib/galaxy/jobs/runners/. The standard cluster runners in there would
be drmaa.py, pbs.py, and condor.py. drmaa.py and pbs.py demonstrating
hooking Galaxy up to a library for submitting jobs to a cluster.
condor.py demonstrates wrapping CLI tools. Along similar lines - there
is cli.py which is something of a general framework for submitting
jobs via CLI tools and can even be used to SSH before running the
submission scripts if that is useful.

That approach largely hinges on having a large shared cluster if you
want to submit many different tools. If you don't mind modifying the
tools themselves - one could move logic for staging files and
submitting to clusters into the tools themselves - I can send some
links of example tools that have done this.

If you don't have the shared cluster and have many different tools you
would like to manage this way - I would suggest looking at Pulsar
(https://github.com/galaxyproject/pulsar).  It can be used to
distribute jobs to remote clusters/machines. Pulsar has the concept of
"job managers" instead of "job runners" - they have a simpler
interface that would need to be implemented for ARC. Examples here
(https://github.com/galaxyproject/pulsar/tree/master/pulsar/managers).
Pulsar has a bunch of options for staging files (File system
copies/HTTP/scp/rsync) and these can be configured on a per-path basis
for each Galaxy instance allowing you to optimize the data transfer
for your 5 setups.

Pulsar can be deployed as a RESTful web service (in this case you
could probably do one web service for all 5 instances) or by
monitoring a message queue (without some small changes you would
probably need to stand-up one pulsar server for each of the 5 Galaxy
instances in this case).

I like to give the warning that Galaxy is designed for large shared
file systems - and Pulsar or other distributed strategies requires
more effort to deploy (and in your case will definitely require novel
development time as well).

It is probably out of scope - but I would also note that it would
possibly be significantly easier to just deploy one Galaxy instance
and route the jobs to local clusters and let them all share one large
file system and just provide 5 different "faces" to Galaxy. That
probably isn't possible due to hardware/institutional politics/etc...
but I just wanted to make sure.

Along the same lines - it is worth considering if writing a DRMAA
layer for ARC or plugging it into Condor somehow might be a more
robust solution that Galaxy can leverage without actually locking your
development efforts into Galaxy-specific solutions.

-John


On Mon, Jan 5, 2015 at 6:35 AM, Abdulrahman Azab  wrote:
> Hi Galaxy developers,
>
>
> In our Elixir project (http://www.elixir-europe.org/), Norway, we have five
> geographically distributed Galaxy instances. Each is working on a local
> cluster (mostly SLURM). We are planning to interconnect those five clusters
> using a meta-scheduler ARC (http://www.nordugrid.org/arc/), to achieve load
> balancing, so that a galaxy job can be reallocated to an external cluster in
> case that the local cluster is saturated.
>
>
> ARC manages the interconnection very well. What we need is to create a
> Galaxy job handler for ARC. Is there a general template or interface for a
> job handler, i.e. for defining job submission commands ... etc.?
>
> and how to compile and integrate this new job handler and integrate it into
> the Galaxy installation?
>
>
> Thank you,
>
> Yours sincerely,
> Abdulrahman Azab
>
> Head engineer, ELIXIR.NO / The Genomic HyperBrowser team
> Department of Informatics, University of Oslo, Boks 1072 Blindern, NO-0316
> OSLO, Norway
> Email: a...@ifi.uio.no, Cell-phone: +47 46797339
> 
> Senior Lecturer in Computer Engineering
> Faculty of Engineering, University of Mansoura, 35516-Mansoura, Egypt
> Email: abdulrahman.a...@mans.edu.eg
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Create a package with binaries

[John Chilton]

I could be wrong but I think the  tag should have some sort
of download action - I don't think tools are actually stuck in
$INSTALL_DIR like this - $INSTALL_DIR is where you can move or compile
downloaded files to. You might want to try $REPOSITORY_INSTALL_DIR
instead of $INSTALL_DIR - but really if you are going to just stick an
application next to your tool wrapper like this (I would consider this
an anti-pattern) I would just set your PATH in mytool.sh and not worry
about the tool shed install stuff at all. Maybe add something like

PATH=`dirname $0`/bin:$PATH
export PATH

To the wrapper mytool.sh.

As I mentioned - I think bundling applications with tools like this is
something of an anti-pattern even if you can make it work. The best
practice here would be to try to separate packages with binaries from
repositories that contain tools and create a format "package_mytools"
repository that is just responsible for installing the binary mytool.

Hopefully this helps,
-John

On Tue, Dec 16, 2014 at 9:43 AM, KRESS Arnaud (ESP)  wrote:
> Hi gentlemen,
> I am currently struggling to create a custom package (to share via a 
> toolshed) that would include a tool definition file and the associated 
> binary. Once installed, I can launch a job but I get the following error 
> message:
>
> mytool.sh: line 14: mytool: unknown command
>
> It seems that the PATH was not correctly set. What am I doing wrong ?
>
> My directory tree in my package:
> .
> ├── mytool.sh
> ├── mytool.xml
> ├── bin
> │   └── mytool
> └── tool_dependencies.xml
>
> Here is my tool_dependencies.xml file content:
>
> 
> 
> 
> 
> 
> 
>  action="prepend_to">$INSTALL_DIR/bin
> 
> 
> 
> 
> 
> 
> 
>
> Thank you,
> AK
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] The choice when I integrated slurm and galaxy in job_conf.xml

[John Chilton]

The Slurm job runner is probably a little more experimental but it is
what we use with usegalaxy.org and generally provides a better
end-user experience by clarifying different error conditions
correctly. I would start with that and if you encounter any issues
maybe report them to us and fallback to the DRMAA runner - which
should also work just fine.

-John

On Wed, Jan 7, 2015 at 8:30 PM, xlwang  wrote:
> hi,
> Slurm is the DRM in my cluster. When I modified the job_conf.xml, which
> plugin should I use? slurm or drmaa?
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Mercurial is out of date for Toolshed

[John Chilton]

Hey Ryan,

It looks like Dave B. has come to rescue with an egg and a patch that
implements this:

https://bitbucket.org/galaxy/galaxy-central/pull-request/626/upgrade-mercurial-egg-to-324

It probably won't be back-ported to the forthcoming January release
but I would suspect the raw patch
(https://api.bitbucket.org/2.0/repositories/galaxy/galaxy-central/diff/davebgx/galaxy-central:625bf1fd88ee..63d901ca0e6e)
will apply cleanly against the last few releases. If you have some
time and want to apply it to your release and let us know if it fixes
your problems that would probably help the pull request along.

-John

On Wed, Jan 7, 2015 at 10:56 AM, Ryan G  wrote:
> Yes, I think I found the root cause...its because the Galaxy egg for
> mercurial is using an older version of Mercurial that has this known bug.
> The only real fix is to upgrade the Galaxy egg to mercurial 3.x.
>
> The only way I know of to test this is to build a new Galaxy egg which I'm
> not familiar with.  But as I understand it the new Mercurial version will
> also break Galaxy.
>
>
> On Mon, Jan 5, 2015 at 6:19 PM, Björn Grüning 
> wrote:
>>
>> Hi Ryan,
>>
>> unfortunately not (yet). But I'm really surprised you are getting this
>> error. It working for me and in our docker containers and other
>> deployments. Can we try to detect the root cause of this error? Do you
>> have conflicting mercurial version.
>>
>> Cheers,
>> Bjoern
>>
>> > Thanks.  Is there an alternative way I can install tools from the
>> > Toolshed?  This problem pretty much renders the toolshed unusable for
>> > me...
>> >
>> > On Mon, Jan 5, 2015 at 11:14 AM, John Chilton 
>> > wrote:
>> >
>> >> It is certainly the case that Galaxy should be using the latest
>> >> version of mercurial given a number of high profile bugs with older
>> >> versions. Unfortunately it doesn't seem possible to just drop the new
>> >> version it - there are some API changes that prevent Galaxy from
>> >> loading when doing this and the number of people who can add new eggs
>> >> to Galaxy is low.
>> >>
>> >> I have created a Trello card to track this issue:
>> >>
>> >> https://trello.com/c/9A9uIav0
>> >>
>> >> Let us know if you happen to find a workaround for this issue.
>> >>
>> >> -John
>> >>
>> >>
>> >> On Mon, Dec 22, 2014 at 2:55 PM, Ryan G 
>> >> wrote:
>> >>> I've been trying to track down why I can't get anything from the
>> >>> toolshed
>> >>> installed and finally have it figured out.
>> >>>
>> >>> Whenever I tried to install anything I always got an Error with no
>> >>> explanation of what the error was.  After enabling Debug messages into
>> >> the
>> >>> log file, I see the error is:
>> >>>
>> >>> tool_shed.util.hg_util DEBUG 2014-12-22 14:47:48,910 Error cloning
>> >>> repository: httpsconnection instance has no attribute '_set_hostport'
>> >>>
>> >>> I googled around and found out this is a known bug/issue with older
>> >> version
>> >>> of Mercurial and was fixed in v3.
>> >>>
>> >>> I added a line to hg_util.py to see where it picks up hg.  Its using
>> >> version
>> >>> 2.2.3.  Indeed, one of the eggs downloaded by Galaxy is
>> >>> mercurial-2.2.3.
>> >>>
>> >>> I have the newest version of mercurial installed in my site-packages
>> >> folder
>> >>> but I guess that's not what galaxy wants.  So my question is, how do I
>> >> get
>> >>> Galaxy to use the latest version of Mercurial?  And, Why did it
>> >>> download
>> >> an
>> >>> older version?
>> >>>
>> >>> ___
>> >>> Please keep all replies on the list by using "reply all"
>> >>> in your mail client.  To manage your subscriptions to this
>> >>> and other Galaxy lists, please use the interface at:
>> >>>   https://lists.galaxyproject.org/
>> >>>
>> >>> To search Galaxy mailing lists use the unified search at:
>> >>>   http://galaxyproject.org/search/mailinglists/
>> >>
>> >
>> >
>> >
>> > ___
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client.  To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> >   https://lists.galaxyproject.org/
>> >
>> > To search Galaxy mailing lists use the unified search at:
>> >   http://galaxyproject.org/search/mailinglists/
>> >
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Text file busy

[John Chilton]

Hey Evan,

  Galaxy should perhaps be able to retry submissions that fail -
especially if they fail quickly and I have created a Trello card for
this here (https://trello.com/c/hxy2bcIb). Nate has added some
features for job state handling plugins
(https://bitbucket.org/galaxy/galaxy-central/commits/7b209e06ddb944e953d340754439f4e3e5dc339d)
and it may be possible to write a plugin to do this today though
immediate submissions failures maybe should be handled a level above
this by the framework... not sure.

  I am not really sure this is the appropriate solution though for
this particular problem though - this seems like an unfortunate
interplay between your file system and your cluster manager and it
would seem that any script or platform that automates the creation of
submissions of jobs would potentially be subject to the same problems.
Solving it in Galaxy would be a application level solution to a
system-level configuration problem in my opinion. Have you ran this
problem by the systems staff - it seems like it should be possible to
delay each submission by a half of a second or change the flushing
settings of the file system.

  As you mentioned - a local work around might be to `time.sleep(1)`
before `external_job_id = self.ds.runJob(jt)` in
lib/galaxy/jobs/runners/drmaa.py or similar line line pbs.py. Do you
want to try that and let us know if it addresses the problem?

  Finally, in terms of the workflow - if you rerun the failed step in
the GUI you should be given the option via a new checkbox on the tool
form to resume the workflow.

-John


On Mon, Jan 5, 2015 at 4:48 PM, Evan Bollig PhD  wrote:
> I get this error occasionally:
>
> "/bin/sh: 1: 
> /opt/galaxy/web/database/job_working_directory/000/100/galaxy_100.sh:
> Text file busy"
>
> When this occurs, the step fails outright. Resubmitting the step
> resolves the issue and things run no problem. If this error appears
> early in a long workflow, I have to manually resubmit ALL dependent
> steps... what a pain!
>
> Perhaps this is something the Galaxy job scheduler can look out for,
> flush() the system, sleep() a second or two to let the file write and
> close, and then rerun. A more fault-tolerant way of running workflows
> without unnecessary human intervention.
>
> Cheers,
> -Evan Bollig
> Research Associate | Application Developer | User Support Consultant
> Minnesota Supercomputing Institute
> 599 Walter Library
> 612 624 1447
> e...@msi.umn.edu
> boll0...@umn.edu
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Moving from mercurial to git

[John Chilton]

On Wed, Jan 14, 2015 at 11:41 AM, Martin Čech  wrote:
> Hello Abdulrahman,
>
> we have multiple active repositories on GH already that contain e.g. Galaxy
> tool wrappers (https://github.com/galaxyproject/tools-devteam) or software
> to help with Galaxy deployment
> (https://github.com/galaxyproject/usegalaxy-playbook).
>
> Regarding our main development repository (currently at
> https://bitbucket.org/galaxy/galaxy-central ) the move to the GH _might_
> happen sometimes this year and will be transparent for anybody deploying
> Galaxy (probably with https://bitbucket.org/galaxy/galaxy-dist remaining as
> a maintaned mirror).
>
> There is no deadline though as the percieved importance is not high.

For what it is worth... I perceive the importance as *VERY* high and I
know other members of the community do as well :). I doubt there is
any one step the devteam could take to encourage community
contributions or foster openness that would have more impact than
this.

Here is the Trello card to track progress and vote on:

https://trello.com/c/iiSBweRQ

-John


>
> Martin, Galaxy Team
>
>
> On Wed Jan 14 2015 at 11:30:54 AM Abdulrahman Azab  wrote:
>>
>> Hi Galaxy people,
>>
>>
>> I had a chat with some of you in the last workshop in Switzerland, and I
>> got information that you are planning to move to git.
>>
>>
>> Is this still the intention? If yes, then when should this happen?
>>
>>
>>
>> Thank you,
>>
>> Yours sincerely,
>> Abdulrahman Azab
>>
>> Head engineer, ELIXIR.NO / The Genomic HyperBrowser team
>> Department of Informatics, University of Oslo, Boks 1072 Blindern, NO-0316
>> OSLO, Norway
>> Email: a...@ifi.uio.no, Cell-phone: +47 46797339
>> 
>> Senior Lecturer in Computer Engineering
>> Faculty of Engineering, University of Mansoura, 35516-Mansoura, Egypt
>> Email: abdulrahman.a...@mans.edu.eg
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] galaxy-central and nginx

[John Chilton]

On Fri, Jan 9, 2015 at 10:42 AM, Jorrit Boekel
 wrote:
> Hi all,
>
> I’m testing the latest galaxy-central (rev 15961:63d901ca0e6e) to try out 
> dataset collections. First thing I run into is that things don’t seem to run 
> well when running it on a VM (port forwarded) with nginx. If I click on tools 
> I get a white screen instead of tool options, and the log show an error from 
> lib/galaxy/webapps/galaxy/api/tools.py : Checking parameter file failed.
>
> The setup: URL would be galaxydomain.com:3030, which the VM manager forwards 
> to port 80 on the galaxy VM, which there nginx would forward to port 8080. 
> Now, my nginx is set up to work with older galaxy versions, I don’t know if 
> that matters. I haven’t changed any config files at all yet.
>
> When I turn off nginx and serve directly on port 80 (using sudo), things look 
> like they work. At least the tools look like they should. Has the nginx 
> config to use changed recently or is this some weird artefact? Anyone else 
> seen it? Since I’m just goofing around to see how collections work, it is not 
> the end of the world for me, and I know galaxy-dist is the stable one, but 
> still.

I have not seen this and haven't heard of anyone else - it probably
doesn't have anything to do with -central versus -stable though - I
don't think I have seen any changes related to this recently. Are you
sure you have setup all the same configuration options in your new
Galaxy instance (e.g. nginx_x_archive_files_base,
nginx_x_accel_redirect_base, nginx_upload_store, nginx_upload_path,
remote_user, etc...)? Also do you have the same number of web threads
and handler threads or both instances - if you configured you older
instance to use multiple processes your new instance will need to be
configured to do this as well.

Otherwise I am out of ideas - sorry. Good luck with dataset
collections - there is still a lot of work that needs to be done in
-central for collections for sure but I am really excited about the
potential proteomics applications.

-John


>
> cheers,
> —
> Jorrit Boekel
> Proteomics systems developer
> BILS / Lehtiö lab
> Scilifelab Stockholm, Sweden
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Egg distribution error

[John Chilton]

Sorry for the delayed response - I am not really an expert on Galaxy's
egg stuff - but I believe running `python scripts/fetch_eggs.py`
before starting Galaxy usually fixes these problems for reasons I
don't understand. One probably shouldn't need to manually install mock
or pycrypto though.

Sorry for the problems and thanks for using Galaxy (and upgrading)!

-John

On Thu, Jan 15, 2015 at 10:42 AM, Ulf Schaefer  wrote:
> Hi all
>
> For what it's worth, I manually installed mock (and pycrypto because
> that was the next thing it complained about) and it worked after that.
>
> Ulf
>
> On 14/01/15 14:23, Ulf Schaefer wrote:
>> Hi all
>>
>> After the announcements this morning, I upgraded to the latest stable
>> release as suggested (latest_2015.01.13).
>>
>> Our server is now not starting with the following error message. Any
>> ideas what I might be doing wrong?
>>
>> Thanks a lot
>> Ulf
>>
>> ---
>>
>> galaxy.eggs DEBUG 2015-01-14 14:10:25,272 Fetched
>> http://eggs.galaxyproject.org/bioblend/bioblend-0.4.2-py2.6.egg
>> galaxy.eggs ERROR 2015-01-14 14:10:25,290 One of Galaxy's managed eggs
>> depends on something which is missing, this is almost certainly a bug in
>> the egg distribution.
>> galaxy.eggs ERROR 2015-01-14 14:10:25,291 Dependency "bioblend" requires
>> "mock"
>> Traceback (most recent call last):
>> File "./scripts/paster.py", line 33, in 
>>   serve.run()
>> File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py",
>> line 1049, in run
>>   invoke(command, command_name, options, args[1:])
>> File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py",
>> line 1055, in invoke
>>   exit_code = runner.run(args)
>> File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py",
>> line 220, in run
>>   result = self.command()
>> File "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/serve.py",
>> line 643, in command
>>   app = loadapp( app_spec, name=app_name, relative_to=base,
>> global_conf=vars)
>> File
>> "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line
>> 350, in loadapp
>>   return loadobj(APP, uri, name=name, **kw)
>> File
>> "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line
>> 375, in loadobj
>>   return context.create()
>> File
>> "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line
>> 813, in create
>>   return self.object_type.invoke(self)
>> File
>> "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line
>> 249, in invoke
>>   return fix_call(context.object, context.global_conf,
>> **context.local_conf)
>> File
>> "/home/galaxy/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py", line
>> 97, in fix_call
>>   val = callable(*args, **kw)
>> File
>> "/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py", line
>> 54, in app_factory
>>   webapp.add_ui_controllers( 'galaxy.webapps.galaxy.controllers', app )
>> File "/home/galaxy/galaxy-dist/lib/galaxy/web/framework/webapp.py",
>> line 119, in add_ui_controllers
>>   module = import_module( module_name )
>> File
>> "/home/galaxy/galaxy-dist/lib/galaxy/util/backports/importlib/__init__.py",
>> line 37, in import_module
>>   __import__(name)
>> File
>> "/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/cloudlaunch.py",
>> line 21, in 
>>   eggs.require('bioblend')
>> File "/home/galaxy/galaxy-dist/lib/galaxy/eggs/__init__.py", line
>> 411, in require
>>   return c[req.project_name].require()
>> File "/home/galaxy/galaxy-dist/lib/galaxy/eggs/__init__.py", line
>> 239, in require
>>   dists = self.resolve()
>> File "/home/galaxy/galaxy-dist/lib/galaxy/eggs/__init__.py", line
>> 170, in resolve
>>   dists = pkg_resources.working_set.resolve( (
>> self.distribution.as_requirement(), ), env, self.fetch )
>> File "/home/galaxy/galaxy-dist/lib/pkg_resources.py", line 569, in
>> resolve
>>   raise VersionConflict(dist,req) # XXX put more info here
>> pkg_resources.VersionConflict: (bioblend 0.4.2
>> (/home/galaxy/galaxy-dist/eggs/bioblend-0.4.2-py2.6.egg),
>> Requirement.parse('mock'))
>>
>> ---
>>
>> **
>> The information contained in the EMail and any attachments is confidential 
>> and intended solely and for the attention and use of the named addressee(s). 
>> It may not be disclosed to any other person without the express authority of 
>> Public Health England, or the intended recipient, or both. If you are not 
>> the intended recipient, you must not disclose, copy, distribute or retain 
>> this message or any part of it. This footnote also confirms that this EMail 
>> has been swept for computer viruses by Symantec.Cloud, but please re-sweep 
>> any attachments before opening or saving. http://www.gov.uk/PHE
>> **
>> _

Re: [galaxy-dev] Create a package with binaries

[John Chilton]

Yes, I think this is the current best practice pattern - where you
would stick the first tool_dependencies.xml in a "Tool Dependency
Definition" repository named package_ballast_1_0 and the the other
tool_dependencies.xml file would just sit in the tool_dependencies.xml
file corresponding to say an "unrestricted" repository called ballst
with your tools.

There are lots of examples of this pattern in the IUC repositories:

https://github.com/galaxyproject/tools-iuc

Hope this helps,
-John



On Wed, Jan 7, 2015 at 3:14 AM, KRESS Arnaud (ESP)  wrote:
> If I create two distincts packages, what should I indicate as package name in 
> the tool_dependencies.xml files ?
>
> For example, a binary package:
>
> 
> 
> 
> 
> http://myurl.com/...
> 
> ballast
> $INSTALL_DIR/bin
> 
>  mode="750">$INSTALL_DIR/bin/ballast
> 
>  action="prepend_to">$INSTALL_DIR/bin
> 
> 
> 
> 
> 
>
> and its corresponding tool package:
>
> 
> 
> 
> 
> 
>
> Should I set the same name ?
>
> Thank you,
> AK
>
>
> Le Lundi 5 Janvier 2015 20:23 CET, John Chilton  a écrit:
>
>> I could be wrong but I think the  tag should have some sort> of 
>> download action - I don't think tools are actually stuck in
>> $INSTALL_DIR like this - $INSTALL_DIR is where you can move or compile
>> downloaded files to. You might want to try $REPOSITORY_INSTALL_DIR
>> instead of $INSTALL_DIR - but really if you are going to just stick an
>> application next to your tool wrapper like this (I would consider this
>> an anti-pattern) I would just set your PATH in mytool.sh and not worry
>> about the tool shed install stuff at all. Maybe add something like
>>
>> PATH=`dirname $0`/bin:$PATH
>> export PATH
>>
>> To the wrapper mytool.sh.
>>
>> As I mentioned - I think bundling applications with tools like this is
>> something of an anti-pattern even if you can make it work. The best
>> practice here would be to try to separate packages with binaries from> 
>> repositories that contain tools and create a format "package_mytools"
>> repository that is just responsible for installing the binary mytool.>
>> Hopefully this helps,
>> -John
>>
>> On Tue, Dec 16, 2014 at 9:43 AM, KRESS Arnaud (ESP)  
>> wrote:
>> > Hi gentlemen,
>> > I am currently struggling to create a custom package (to share via a 
>> > toolshed) that would include a tool definition file and the associated 
>> > binary. Once installed, I can launch a job but I get the following error 
>> > message:
>> >
>> > mytool.sh: line 14: mytool: unknown command
>> >
>> > It seems that the PATH was not correctly set. What am I doing wrong ?
>> >
>> > My directory tree in my package:
>> > .
>> > ├── mytool.sh
>> > ├── mytool.xml
>> > ├── bin
>> > │   └── mytool
>> > └── tool_dependencies.xml
>> >
>> > Here is my tool_dependencies.xml file content:
>> >
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > > > action="prepend_to">$INSTALL_DIR/bin
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> >
>> > Thank you,
>> > AK
>> > ___
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client.  To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> >   https://lists.galaxyproject.org/
>> >
>> > To search Galaxy mailing lists use the unified search at:
>> >   http://galaxyproject.org/search/mailinglists/
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Display of file content in a combobox

[John Chilton]

I don't know how to do this exactly - but I think it is possible with
a custom datatype. I think you define a custom datatype (you will
probably want to extend the Galaxy tabular datatype), then I believe
you will need to override the set_meta method on the datatype to
collect a new metadata field (perhaps called factors), and finally
have the first step produce this datatype.

The second step then should define its input data parameter to consume
this type, and then define a "select" param type and within that

  
   

   
  

The example of a tool that works vaguely this way that I know about is
JJ's mothur wrappers (but hopefully someone can think of one that
actually works with tabular data).

Here is the repository:
https://toolshed.g2.bx.psu.edu/repos/jjohnson/mothur_toolsuite/file/tip/mothur

Here is the datatype definition - he extends Text data - but you would
want to extend Tabular instead - but this gives a sense of how to
override set_meta to collect metadata and extend a datatype.

https://toolshed.g2.bx.psu.edu/repos/jjohnson/mothur_toolsuite/file/tip/mothur/lib/galaxy/datatypes/metagenomics.py

Here is a tool than that uses this datatype and the filter on data_meta type.

https://toolshed.g2.bx.psu.edu/repos/jjohnson/mothur_toolsuite/file/tip/mothur/tools/mothur/classify.otu.xml

It will probably frustrating to get this pattern working the first
time - but it is possible.

Hope this helps,
-John


On Mon, Jan 12, 2015 at 10:31 AM, christof.piet...@kws.com
 wrote:
> Hello,
>
> I’d like to display the unique column content of a file in a combobox to
> give the user the possibility to select factor levels for subsequent
> analyses, e.g. for combining related factor levels in a meta-analyses.
>
> Unfortunately, I failed to connect the tool that reads out the factor levels
> with the tool that receive the selection via combobox as input argument.
>
> I’d be very thankful for any suggestion!
>
>
>
> Christof
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Getting 'No space left on device' at testtoolshed

[John Chilton]

Thanks for the bug report. This has been resolved for now - but it
does seem to happen more than it should.

There should ideally be nagios checks in place to make warn us before
it gets to this point - I will try to ask around about that.

-John

On Mon, Jan 12, 2015 at 6:18 PM, Jeltje van Baren
 wrote:
> Hi,
>
> I'm trying to upload a tarball to one of my repositories at
> https://testtoolshed.g2.bx.psu.edu/
>
> but I keep getting the error pasted below my signature. This started at
> about 1pm PST and is not resolved now, at 3pm. Am I the only one who sees
> this error and if so, what should I do?
>
> Thanks!
>
> -Jeltje van Baren
>
> Internal Server Error
>
> Galaxy was unable to successfully complete your request
>
> URL:
> http://testtoolshed.g2.bx.psu.edu/upload/upload?repository_id=2e4420df6d37d181
> Module galaxy.web.framework.middleware.error:149 in __call__
>>>  app_iter = self.application(environ, sr_checker)
> Module paste.debug.prints:106 in __call__
>>>  environ, self.app)
> Module paste.wsgilib:543 in intercept_output
>>>  app_iter = application(environ, replacement_start_response)
> Module paste.recursive:84 in __call__
>>>  return self.application(environ, start_response)
> Module paste.httpexceptions:633 in __call__
>>>  return self.application(environ, start_response)
> Module galaxy.web.framework.base:132 in __call__
>>>  return self.handle_request( environ, start_response )
> Module galaxy.web.framework.base:185 in handle_request
>>>  kwargs = trans.request.params.mixed()
> Module webob:900 in params
> Module webob:892 in str_params
> Module webob:818 in str_POST
> Module cgi:508 in __init__
>>>  self.read_multi(environ, keep_blank_values, strict_parsing)
> Module cgi:637 in read_multi
>>>  environ, keep_blank_values, strict_parsing)
> Module cgi:510 in __init__
>>>  self.read_single()
> Module cgi:648 in read_single
>>>  self.file.seek(0)
> IOError: [Errno 28] No space left on device
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] FastQC wrapper not seeing files at gzipped

[John Chilton]

Peter has already voted and if I recall correctly Ryan cannot access
Trello - so this might be a waste to bring up - but here is a Trello
card for voting on this issue and tracking progress
https://trello.com/c/3RkTDnIn.

To summarize previous discussion - this would be fantastic to have and
Galaxy needs this - but we solve this on usegalaxy.org by using a
compressed file system - a more elegant solution when it is a
possibility - so it has never been a tier one priority for the
devteam. The only update on this is that I don't think we are using a
compressed file system anymore so this might become and issue again
someday soon.

This would be non-trivial to implement - but I have always felt this
would be a fairly fun project to work on if anyone really tight on
space locally wants to try to tackle it :).

-John

On Tue, Jan 13, 2015 at 9:54 AM, Ryan G  wrote:
> Agreed.
>
> On Mon, Jan 12, 2015 at 10:24 PM, Peter Cock 
> wrote:
>>
>> Hi Ryan,
>>
>> That is the workaround I am using, which means
>> keeping an uncompressed copy of the FASTQ
>> file on our main storage from where Galaxy can
>> see it (for people to use within their histories).
>>
>> From a long term storage perspective this is not
>> ideal - so I am keen for better handling of gzipped
>> files within Galaxy (particularly within libraries
>> which we use for raw data).
>>
>> Peter
>>
>> On Mon, Jan 12, 2015 at 5:20 PM, Ryan G 
>> wrote:
>> > Yes, I'm doing a link to file on file system when doing a library
>> > import.
>> > Does this mean I should link to the the uncompressed file?
>> >
>> > On Mon, Jan 12, 2015 at 12:14 PM, Peter Cock 
>> > wrote:
>> >>
>> >> Ah. Then this is more subtle... are you using the
>> >> library import option where Galaxy just symlinks
>> >> to existing files? I thought that was not possible
>> >> with gzipped files (for the reasons given below).
>> >> Perhaps this is not being blocked, leading to the
>> >> confused state you're seeing?
>> >>
>> >> Peter
>> >>
>> >> On Mon, Jan 12, 2015 at 4:52 PM, Ryan G 
>> >> wrote:
>> >> > Galaxy is not decompressing the file.  The file is linked to on the
>> >> > filesystem.
>> >> >
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] galaxy and torque - resource allocation

[John Chilton]

Galaxy generally defers to the DRM (torque in your case) for dealing
with these things. In your job_conf.xml you can specify limits for
memory or CPUs and Galaxy will pass these along to the DRM at which
point it is up to the DRM to enforce these - details depend on if you
are using the PBS runner or the DRMAA runner (let me know which and I
can try to elobrate if that would be useful).

In your particular case - I don't believe torque "schedules" RAM so
things will generally only be... throttled... by CPUs counts. This is
what I was told by the admins at MSI when I worked there anyway. If
you want to place hard limits on RAM I think you need to "upgrade" to
the MOAB scheduler or switch over to a different DRM entirely like
SLURM. Even for DRMs that deal more directly with memory - Galaxy
doesn't provide a general mechanism for passing this along to the tool
(https://trello.com/c/3RkTDnIn) - so it would be up to the tool to
interact with the environment variables your DRM sets .

This sounds really terrible in the abstract - but it reality it
usually isn't an issue - most of Galaxy's multi-core mappers say have
relatively low memory per CPU usage - and for things like assemblers
where this is more important - one can usually just assign them to
their own CPU or something like that to ensure they get all the memory
available.

Unlike memory - Galaxy will attempt to pass the number of slots
allocated to a job to the tools by setting the GALAXY_SLOTS
environment variable. All of the multiple core devteam Galaxy tools at
this point use this and so should work - as should probably most
multi-core tools from the tool shed - at least the most popular ones.

Hope this helps,

-John


On Tue, Jan 13, 2015 at 11:52 AM, Fernandez Edgar
 wrote:
> Hello gents,
>
>
>
> Hope everyone had a great holiday break!
>
> Wish you guys all the best for 2015!
>
>
>
> I have a couple of questions about how resources (CPU and memory) is
> allocated when you have a galaxy and torque installation.
>
>
>
> So I’ve setup torque with some default and maximum amount of CPU and memory
> allocations.
>
> However, I have some worries when it comes to application (like tophat for
> example).
>
> By default, it takes half the CPU of a server unless specified otherwise.
>
> How is the CPU allocation is specified to application like tophat through
> galaxy?
>
>
>
> Also, how does galaxy react if a job needs more memory than the limit set by
> torque?
>
>
>
> Any information would help me a lot!
>
>
>
> My sincere salutations to you all!!!
>
>
>
> Cordialement / Regards,
>
>
>
> Edgar Fernandez
>
> System Administrator (Linux)
>
> Direction Générale des Technologies de l'Information et de la Communication
>
> (  Bur. : 1-514-343-6111 poste 16568
>
>
>
> Université de Montréal
>
> PAVILLON ROGER-GAUDRY, bureau X-218
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] EOF error with tool

[John Chilton]

There are a couple groups working on improved trinity wrappers - I
don't think they have reach the tool sheds though :(.

As for the details of your specific tool - I think I would really need
to see the tool to guess at the issue. Would it be possible to send us
the problematic tool XML file or paste it to a service like
gist.github.com?

As for reloading the tool - if you add yourself as an admin user in
galaxy's config/galaxy.ini file you should see a "Reload Tool
Configuration" option in the admin menu that will allow you to
manually reload the tool without restarting Galaxy (I generally
recommend keeping this open in its own tab).

Newer versions of Galaxy have some more experimental features designed
to help as well - for instance if you create a virtualenv for Galaxy
and install watchdog ("pip install watchdog") into it and then enable
"watch_tools = True " in config/galaxy.ini Galaxy should automatically
reload tools each time the tool config is updated. I am not sure
anyone is really using this feature so it might still have some bugs
(in particular I have noticed if the tool panel is really full there
are problems so trimming that down might be required to use it).

For the rare person like me who really hates using GUIs at all during
development - you might also want to write some tests for the tool and
run them using planemo (http://planemo.readthedocs.org/). It allows
linting and testing tools without a Galaxy interface so you can be
pretty confident the tool is working by the time you actually load it
into a web browser and view it.

-John


On Mon, Jan 12, 2015 at 9:36 AM, Francis, Warren
 wrote:
> Hello,
> I'm getting a strange error when trying to debug a tool.
>
> Traceback (most recent call last):
>   File "/var/project/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line
> 157, in prepare_job
> job_wrapper.prepare()
>   File "/var/project/galaxy-dist/lib/galaxy/jobs/__init__.py", line 811, in
> prepare
> self.command_line, self.extra_filenames = tool_evaluator.build()
>   File "/var/project/galaxy-dist/lib/galaxy/tools/evaluation.py", line 348,
> in build
> self.__build_command_line( )
>   File "/var/project/galaxy-dist/lib/galaxy/tools/evaluation.py", line 364,
> in __build_command_line
> command_line = fill_template( command, context=param_dict )
>   File "/var/project/galaxy-dist/lib/galaxy/util/template.py", line 9, in
> fill_template
> return str( Template( source=template_text, searchList=[context] ) )
>   File
> "/var/project/galaxy-dist/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg/Cheetah/Template.py",
> line 1004, in __str__
> return getattr(self, mainMethName)()
>   File "cheetah_DynamicallyCompiledCheetahTemplate_1421072368_91_78155.py",
> line 139, in respond
> EOFError: EOF when reading a line
>
> The tool is the assembler Trinity. The toolshed version is quite old and the
> one from the Trinity website does not work, so I tried to fix it (several
> problems to fix).
>
> What is the EOFError? There were a bunch of posts reporting this but no
> answer as to why this happens.
>
> Is there a way to debug a tool without having to restart galaxy each time?
> Something offline that can check for runtime errors, or at least see if it
> generates the command correctly.
>
> Best,
> -Warren
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Manually adding new tools from toolshed

[John Chilton]

I am skeptical that setting up a local tool shed is worth the effort -
it seems like overkill.  I think just downloading the tar balls,
manually modifying the datatypes_conf.xml is fine.

If you set the tool_dependencies_dir in your Galaxy configuration -
you can also create a isolated environments for tool requirements
(https://wiki.galaxyproject.org/Admin/Config/ToolDependencies)
allowing multiple versions of requirements like samtools to be
installed simultaneously. I recently added the ability to have
multiple versions of tools themselves installed like the tool shed but
without the tool shed
(https://bitbucket.org/galaxy/galaxy-central/commits/08f8850853d004bf8a456a147436a6694d0a1a11)
-  it is pretty experimental but it might help if you go down this
road.

As for how Galaxy knows about the tool shed installs - it is a
combination of file system files and database modifications (Galaxy
installs the dependencies into tool_dependencies_dir, the tool files
into the directory specified in shed_tools_conf.xml, updates the file
integrated_tool_panel_conf.xml, and registers tool lineages and
installation status in the Galaxy database). It should certainly be
possible to install tools from a different server than Galaxy is
running on - but you will need to run a Galaxy process on that server
that can access all the same files and the Galaxy database.

Hope this helps,

-John

On Fri, Jan 16, 2015 at 10:39 AM, Ryan G  wrote:
> I can try a local toolshed, but the organization I work at frowns upon
> automatic installation for various reasons.  That and it lets me see exactly
> how a tool is installed and works within my local instance.
>
> When a tool is installed from the toolshed, how does Galaxy know about it?
> From a DB entry, or from the XML file(s) being present somewhere?  I sort of
> like the idea of how /etc/profile.d gets scanned by /etc/bashrc when a user
> logs into a linux machine.  This way, there is no modification of files,
> just placing the right files in the right directory is all that is needed.
> Anyway, I need to understand how Galaxy works with the toolshed.
>
> On Fri, Jan 16, 2015 at 3:08 AM, Björn Grüning 
> wrote:
>>
>> Hi Ryan,
>>
>> what about a local bootstrapped ToolShed which is in sync with the
>> public one (at least for the users you are interested on). Than you can
>> install from this ToolShed. I guess this is less annoying than
>> maintaining everything on your own.
>>
>> Cheers,
>> Bjoern
>>
>> Am 16.01.2015 um 08:22 schrieb Peter Cock:
>> > On Thu, Jan 15, 2015 at 6:57 PM, Ryan G 
>> > wrote:
>> >> Because of the way the infrastructure for my Galaxy instance is set up,
>> >> I
>> >> need to download and install tools from the toolshed manually.  For
>> >> most of
>> >> the tools, this is pretty easy, however I'm now trying to add RSEM to
>> >> my
>> >> instance and it has some new datatypes.
>> >>
>> >> Along with the new data types is python class file.  I'm referring to
>> >> the
>> >> exist datatypes_conf.xml.sample but don't see a reference to
>> >> datatype_files
>> >> as is coded in the RSEM tool.
>> >>
>> >> So my question is, how can I manually install a tool such as RSEM from
>> >> the
>> >> toolshed?  I've cloned it but not sure where to put the files and what
>> >> to
>> >> modify in Galaxy's configuration to see the new files.  I can modify
>> >> the
>> >> datatypes_conf.xml and include the datatypes, but I get the sense this
>> >> is
>> >> not correct.
>> >>
>> >> Ryan
>> >
>> > The old pre-ToolShed way to add a new datatype was indeed to
>> > add it to datatypes_conf.xml and put any Python file into the
>> > galaxy-dist/lib/galaxy/datatypes/
>> >
>> > I'm still doing this on some of my locally developed tools, but
>> > now they are in the ToolShed perhaps I should switch to using
>> > that...
>> >
>> > Regards,
>> >
>> > Peter
>> >
>> > P.S. You also used to need to add an import line to the
>> > galaxy-dist/lib/galaxy/datatypes/registry.py but that is not
>> > longer required as of this bug fix:
>> >
>> > https://bitbucket.org/galaxy/galaxy-central/commits/1422966f1ca88472b8685015f5a8cdd3dd0f1db9
>> > ___
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client.  To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> >   https://lists.galaxyproject.org/
>> >
>> > To search Galaxy mailing lists use the unified search at:
>> >   http://galaxyproject.org/search/mailinglists/
>> >
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please ke

Re: [galaxy-dev] Unable to connect galaxy server after upgrade to latest version

[John Chilton]

No clue what is wrong - hopefully you figured this out.

If you haven't, can you send us your Galaxy log file for the web
server loading - you can send it to me personally and
galaxy-b...@bx.psu.edu - if you are worried about disclosing private
information on the public list?

Also it would be interesting to see the output produced when running
both "hg summary" and "hg status" from your Galaxy working directory.

-John

On Fri, Jan 16, 2015 at 6:11 PM, Ping Luo  wrote:
> I installed the latest galaxy server and started it without any change. The
> server stared on port 8080 and the log file looks fine. But when I typed
> http://localhost:8080, I got the error "Unable to connect". I had worked
> with version 2014_08_11 and had no problem at all. What could have been
> wrong?
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Fasta Datatype Adjustments

[John Chilton]

I think introducing a new a type like mutect.fasa or something is
probably the better idea here. You can create a tool that creates this
datatype and your main mutect tool wrapper can then take an input of
type  wrote:
> Hello Galaxy,
>
> I have a small dilemma regarding the fasta datatype.
>
> Currently, to my knowledge, the fasta datatype does not specify any
> metadata. I was curious how I should go about changing the fasta datatype if
> I wanted to include the .fai and .dict as metadata? Basically I want to
> avoid unnecessary recreating of the same file done automatically by MuTect.
> It seems very inefficient to have to produce these files every time a user
> were to call MuTect in galaxy (I.e. MuTect can't find them, so it makes them
> itself).
>
> I guess my question is, what are the consequences to adjusting the current
> implementation of the fasta datatype? Will users be able to pull this tool
> easily? (I.e. When one uploads a tool and that tools uses a different fasta
> definition, how does galaxy handle this?) Could this new fasta declaration
> potentially be adapted by galaxy? Should I just define a new mtfasta
> datatype special for MuTect purposes?
>
> Let me know if you have any advice,
>
> Marco
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Unable to get data from UCSC

[John Chilton]

Hello Anna,

Can you try something for me -


https://bitbucket.org/galaxy/galaxy-central/commits/579d211c2b0fa8ef4195930cb999edbb9a0cf8eb

On Mon, Jan 19, 2015 at 9:12 AM, Anna Terry  wrote:
> Update,
> Looking at the browser developer tools error logs in firefox I can see the
> following error:
> Blocked loading mixed active content
> "http://galaxy.hpc.qmul.ac.uk/tool_runner";
>
> similar error in chrome:
>
> (index):1 The page at 'https://galaxy.hpc.qmul.ac.uk/' was loaded over
> HTTPS, but displayed insecure content from
> 'http://galaxy.hpc.qmul.ac.uk/tool_runner': this content should also be
> loaded over HTTPS.
>
> genome.ucsc.edu/cgi-bin/hgTables:1 The page at
> 'https://genome.ucsc.edu/cgi-bin/hgTables' was loaded over HTTPS, but is
> submitting data to an insecure location at
> 'http://galaxy.hpc.qmul.ac.uk/tool_runner': this content should also be
> submitted over HTTPS.
>
> (index):1 Mixed Content: The page at 'https://galaxy.hpc.qmul.ac.uk/' was
> loaded over HTTPS, but requested an insecure form action
> 'http://galaxy.hpc.qmul.ac.uk/tool_runner'. This request has been blocked;
> the content must be served over HTTPS.
>
> Is there any way to change settings so that it points back to https?  The
> server should redirect to https but I guess the browser doesn't know this?
>
> TIA
> Anna
>
> On 13 January 2015 at 16:39, Anna Terry  wrote:
>>
>> Hi,
>>
>> I am having lots of problems with the get data tools installed on our
>> local galaxy server, I'll start with just one.
>>
>> With the UCSC Main one, I am unable to send data back to galaxy, when I
>> click "Send query to Galaxy" nothing happens, the button is unresponsive.
>>
>> This is in contrast to when I try the same query from the main galaxy
>> server, on the same browser and the content in correctly transferred back to
>> galaxy.
>>
>> I have no idea how to start debugging this, there is nothing in paster.log
>> Does anyone have any ideas?
>>
>> TIA,
>> Anna
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Unable to get data from UCSC

[John Chilton]

Ugh - hit send too soon accidentally.

I was trying to say - that I believe this was recently changed to
allow Galaxy served over HTTPS to access the UCSC in the linked
changeset. I was wondering if you could try to just undo that change
and see if things work for you again.

https://bitbucket.org/galaxy/galaxy-central/commits/579d211c2b0fa8ef4195930cb999edbb9a0cf8eb

If that does work - it might work to set that XML block to something like



which may correctly tell the web browser to infer HTTP or HTTPS and
that would be a change I would be happy to make in -central to fix
this problem.

Sorry again for all the e-mails.

-John

P.S. I would definitely serve Galaxy over HTTPS if at all possible -
so that might be something you just want to do now to side this
problem.


On Mon, Jan 19, 2015 at 8:36 PM, John Chilton  wrote:
> Hello Anna,
>
> Can you try something for me -
>
>
> https://bitbucket.org/galaxy/galaxy-central/commits/579d211c2b0fa8ef4195930cb999edbb9a0cf8eb
>
> On Mon, Jan 19, 2015 at 9:12 AM, Anna Terry  wrote:
>> Update,
>> Looking at the browser developer tools error logs in firefox I can see the
>> following error:
>> Blocked loading mixed active content
>> "http://galaxy.hpc.qmul.ac.uk/tool_runner";
>>
>> similar error in chrome:
>>
>> (index):1 The page at 'https://galaxy.hpc.qmul.ac.uk/' was loaded over
>> HTTPS, but displayed insecure content from
>> 'http://galaxy.hpc.qmul.ac.uk/tool_runner': this content should also be
>> loaded over HTTPS.
>>
>> genome.ucsc.edu/cgi-bin/hgTables:1 The page at
>> 'https://genome.ucsc.edu/cgi-bin/hgTables' was loaded over HTTPS, but is
>> submitting data to an insecure location at
>> 'http://galaxy.hpc.qmul.ac.uk/tool_runner': this content should also be
>> submitted over HTTPS.
>>
>> (index):1 Mixed Content: The page at 'https://galaxy.hpc.qmul.ac.uk/' was
>> loaded over HTTPS, but requested an insecure form action
>> 'http://galaxy.hpc.qmul.ac.uk/tool_runner'. This request has been blocked;
>> the content must be served over HTTPS.
>>
>> Is there any way to change settings so that it points back to https?  The
>> server should redirect to https but I guess the browser doesn't know this?
>>
>> TIA
>> Anna
>>
>> On 13 January 2015 at 16:39, Anna Terry  wrote:
>>>
>>> Hi,
>>>
>>> I am having lots of problems with the get data tools installed on our
>>> local galaxy server, I'll start with just one.
>>>
>>> With the UCSC Main one, I am unable to send data back to galaxy, when I
>>> click "Send query to Galaxy" nothing happens, the button is unresponsive.
>>>
>>> This is in contrast to when I try the same query from the main galaxy
>>> server, on the same browser and the content in correctly transferred back to
>>> galaxy.
>>>
>>> I have no idea how to start debugging this, there is nothing in paster.log
>>> Does anyone have any ideas?
>>>
>>> TIA,
>>> Anna
>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] PATH set in tool env.sh leaking through to set_metadata.py

[John Chilton]

Are you using the local job runner (this will be the case if you
haven't explicitly configured something like pbs or DRMAA in your
job_conf.xml file)?

-John

On Tue, Jan 20, 2015 at 12:34 PM, Wolfgang Maier
 wrote:
> On 01/20/2015 06:20 PM, Wolfgang Maier wrote:
>>
>>
>> I have not seen this error before (I believe not with latest_2014.10.06)
>
>
> update: confirmed this now. It's enough to hg update to latest_2014.10.06
> and things are working again.
>
> The difference is that when building the dependency shell command the latest
> release seems to put the call to set_metadata.sh into that command, while
> before it seems it was run separately.
>
>
> Wolfgang
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] PATH set in tool env.sh leaking through to set_metadata.py

[John Chilton]

So the tool dependency stuff didn't change - but the local job runner
now behaves like the other job runners (DRMAA, PBS, etc...) so you
have uncovered a bug for all of them I think. Is this a tool shed tool
or a locally created dependency?

I need to think about how to solve this more generally but if you want
a quick work around - I think it would work to just replace "python"
in $GALAXY_ROOT/set_metadata.sh with a  hard-coded path to Galaxy's
python or perhaps just "python2".

Sorry for the inconvenience and thanks for bringing this to our attention.

-John

On Tue, Jan 20, 2015 at 12:52 PM, John Chilton  wrote:
> Are you using the local job runner (this will be the case if you
> haven't explicitly configured something like pbs or DRMAA in your
> job_conf.xml file)?
>
> -John
>
> On Tue, Jan 20, 2015 at 12:34 PM, Wolfgang Maier
>  wrote:
>> On 01/20/2015 06:20 PM, Wolfgang Maier wrote:
>>>
>>>
>>> I have not seen this error before (I believe not with latest_2014.10.06)
>>
>>
>> update: confirmed this now. It's enough to hg update to latest_2014.10.06
>> and things are working again.
>>
>> The difference is that when building the dependency shell command the latest
>> release seems to put the call to set_metadata.sh into that command, while
>> before it seems it was run separately.
>>
>>
>> Wolfgang
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Manually installing rsem_datatypes package from toolshed

[John Chilton]

I believe this is an ancient bug that I has been fixed in Galaxy's
central branch but is not in the latest release:

https://bitbucket.org/galaxy/galaxy-central/commits/1422966f1ca88472b8685015f5a8cdd3dd0f1db9

The previous workaround for this bug was that one would need to import
the datatype at the top of registry.py (import rsem_datatypes) which
is another option for addressing this problem instead of applying the
patch.

Hope this helps,
-John


On Wed, Jan 21, 2015 at 2:16 PM, Ryan G  wrote:
> I'm trying to manually install rsem_datatypes package into my local Galaxy
> install.  I can't use the Toolshed for various reasons.
>
> Anyway, I downloaded the package using
>
> hg clone https://toolshed.g2.bx.psu.edu/repos/jjohnson/rsem_datatypes
>
> I see two files:
>
> 1)  datatypes_conf.xml, and
> 2) rsem.py
>
> I copied the  lines into my
> config/datatypes_conf.xml file.
> I also copied  rsem.py to lib/galaxy/datatypes/.
>
> When I restart Galaxy, it doesn't seem that Galaxy knows how to handle rsem
> datatypes, as I get this error:
>
> galaxy.datatypes.registry ERROR 2015-01-21 14:01:47,519 Error importing
> datatype module galaxy.datatypes.rsem: 'module' object has no attribute
> 'rsem'
> Traceback (most recent call last):
>   File "/data/galaxy/galaxy-dist/lib/galaxy/datatypes/registry.py", line
> 210, in load_datatypes
> module = getattr( module, mod )
> AttributeError: 'module' object has no attribute 'rsem'
> galaxy.datatypes.registry ERROR 2015-01-21 14:01:47,519 Error importing
> datatype module galaxy.datatypes.rsem: 'module' object has no attribute
> 'rsem'
> Traceback (most recent call last):
>   File "/data/galaxy/galaxy-dist/lib/galaxy/datatypes/registry.py", line
> 210, in load_datatypes
> module = getattr( module, mod )
> AttributeError: 'module' object has no attribute 'rsem'
> galaxy.datatypes.registry ERROR 2015-01-21 14:01:47,520 Error importing
> datatype module galaxy.datatypes.rsem: 'module' object has no attribute
> 'rsem'
> Traceback (most recent call last):
>   File "/data/galaxy/galaxy-dist/lib/galaxy/datatypes/registry.py", line
> 210, in load_datatypes
> module = getattr( module, mod )
> AttributeError: 'module' object has no attribute 'rsem'
>
> I'm obviously missing something but not sure what.  I suspect maybe the
> rsem.py file is not in the right place?
>
> Ryan
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] PATH set in tool env.sh leaking through to set_metadata.py

[John Chilton]

On Wed, Jan 21, 2015 at 6:54 PM, John Chilton  wrote:
> On Wed, Jan 21, 2015 at 10:35 AM, Wolfgang Maier
>  wrote:
>>
>>
>> On 21.01.2015 16:11, John Chilton wrote:
>>>
>>> So the tool dependency stuff didn't change - but the local job runner
>>> now behaves like the other job runners (DRMAA, PBS, etc...) so you
>>> have uncovered a bug for all of them I think. Is this a tool shed tool
>>> or a locally created dependency?
>>>
>>
>> This is a tool shed tool (package_mimodd_0_1_5), but one I uploaded myself
>> (I discovered the bug when I did a test install after updating our local
>> Galaxy).
>>
>>> I need to think about how to solve this more generally but if you want
>>> a quick work around - I think it would work to just replace "python"
>>> in $GALAXY_ROOT/set_metadata.sh with a  hard-coded path to Galaxy's
>>> python or perhaps just "python2".
>>
>>
>> Since I can change the tool shed tool a more "global" solution could be to
>> simply remove the "python" symlink from the python3 virtualenv after its
>> created and have the tool use "python3" explicitly at runtime. I'll try
>> whether that works, but I think it should.
>
> This sounds like a better short-term solution to me - this way older
> versions of Galaxy will still be able to use the recipe. I created a
> Trello issue to track the longer term fix of isolating metadata
> generation from the tool environment - which is needed but could be a
> little disruptive so I would rather push it off until after the next
> release.

Opps - I forgot the link and forgot to cc the mailing list.

https://trello.com/c/aMCwYZsN

-John

>
> Thanks!
>
> -John
>
>>
>>>
>>> Sorry for the inconvenience and thanks for bringing this to our attention.
>>>
>>
>> No problem. I'm glad if I could help with something instead of just asking
>> questions all the time.
>>
>> Wolfgang
>>
>>
>>
>>>
>>> On Tue, Jan 20, 2015 at 12:52 PM, John Chilton 
>>> wrote:
>>>>
>>>> Are you using the local job runner (this will be the case if you
>>>> haven't explicitly configured something like pbs or DRMAA in your
>>>> job_conf.xml file)?
>>>>
>>>> -John
>>>>
>>>> On Tue, Jan 20, 2015 at 12:34 PM, Wolfgang Maier
>>>>  wrote:
>>>>>
>>>>> On 01/20/2015 06:20 PM, Wolfgang Maier wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have not seen this error before (I believe not with
>>>>>> latest_2014.10.06)
>>>>>
>>>>>
>>>>>
>>>>> update: confirmed this now. It's enough to hg update to
>>>>> latest_2014.10.06
>>>>> and things are working again.
>>>>>
>>>>> The difference is that when building the dependency shell command the
>>>>> latest
>>>>> release seems to put the call to set_metadata.sh into that command,
>>>>> while
>>>>> before it seems it was run separately.
>>>>>
>>>>>
>>>>> Wolfgang
>>>>>
>>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Setting a file download link in Galaxy help

[John Chilton]

The help text is interpreted as reStructuredText (
http://docutils.sourceforge.net/rst.html) not HTML.

Here is an example link for the citation used in Galaxy's MAF tools - it
demonstrates one way to create links in tools using restructuredtext:

If you use this tool, please cite `Blankenberg D, Taylor J, Nekrutenko A;
The Galaxy Team. Making whole genome multiple alignments usable for
biologists. Bioinformatics. 2011 Sep 1;27(17):2426-2428. <
http://www.ncbi.nlm.nih.gov/pubmed/21775304>`_

Hope this helps,
-John


On Mon, Jan 19, 2015 at 10:49 PM, XiaoTao Jiang 
wrote:

> Dear all
>
> I have  installed Galaxy in local and I want to set a file download link
> in galaxy help. I want the user to download a small data-set from the help
> with a link. However, the general HTML download syntax does not take
> effect, look the figure I set.
>
> Anyone knows how to set the download link?
>
> [image: 内嵌图片 1]
>
> Thanks for all your attention.
>
> Regards
> Jimmy
>
>
> --
> Department of Engineering
> The University of HongKong
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] dataset collections

[John Chilton]

On Wed, Jan 21, 2015 at 11:02 AM, Jorrit Boekel
 wrote:
> Hi all,
>
> I’m toying around a little in galaxy-dist with the dataset collections 
> feature. Since I know this is work in progress, I was wondering about some 
> things I haven’t really found online.
>
> It seems to work really well to run a tool on a list of datasets, and a new 
> job is run for each list item. But when I want to reduce to a smaller amount 
> of list items, I understand I need to write some sort of merge tool myself, 
> dependent on the data (all proteomics data here currently). This works well 
> for reducing a dataset to a single file, but I am not sure about how to 
> reduce to a new smaller collection. In the tool I’m writing, I let the user 
> choose the size of the collection.
>
> Is there some way to tell galaxy dynamically how many outputs to expect AND 
> put them in a collection? Something like:
> 
> 
> 
> Where 3 is set by the user in a param also.

Through the January 2015 release - this is not possible. It is now
possible in central for tools to explicitly produce collections and
this functionality will be in the next release (I think it is still an
open question as to whether the team is aiming for February or March
2015 for that). There are a lot of details and examples linked to in
the following pull request that I merged last week:

https://bitbucket.org/galaxy/galaxy-central/pull-request/634/allow-tools-to-explicitly-produce-dataset

There are three style of outputs of increasing complexity - tools that
produce static collection (pairs or lists of fixed size), N->N
operations (like normalization), and finally fully dynamic
collections. Since one can pre-determine the size this would ideally
fall under the first type but I had not foreseen this use case so
there is currently no syntax like you described so you have to use the
third most complicated style.

I am going to assume this is some sort of binning operation? Lets
imagine - the user selects 3 bins and your tools creates a directory
and populates it with mzml files :

output/bin1.mzml
output/bin2.mzml
output/bin3.mzml

Then you could create an output collection like this:


  

  


After the job is complete a dataset collection will be populated with
three elements of type mzml and element identifiers bin1, bin2, and
bin3 (inferred from the name).

The syntax of the discover_datasets thing can be quite complex and you
can variously hard code properties like datatype, name, etc... or
infer them from the name on the file system.

>
>
> Also, when running with two or more lists as input, is there some sort of 
> correlation between the lists? It seems like it takes the files in dataset no 
> order, so just checking.

Yes definitely. The UI for creating lists is pretty limited and needs
to be updated to look a lot more like the UI for creating lists of
paired datasets and this would become a lot more clear I think. So
lists are ordered data structures and element identifiers are
preserved across executions.

So if you start with a list of raw files with identifiers sample1,
sample2, and sample3 and map and operation  like peak picking over
them you would get a new list with the same order and identifiers
(sample1, sample2, and sample3 in that order), then if you map and
operation like peptide identification on the picked files again you
would get a list with identifiers (sample1, sample2, and sample3).
Then if you have some sort of summary operation that takes in a raw
file and identification and you pass it the original list and the
result of the identifications - Galaxy should match everything up and
assign the create a resulting list with sample1, sample2, and sample3.
(The API lets you do more complicated things like cross-products over
subsets of inputs - but this isn't exposed in the GUI yet).

If you have two ordered lists and identifiers don't match up -
Galaxy's behavior should be considered undefined but it will assign
the resulting list the identifiers from one or the other inputs.

>
> By the way, thanks very much John and everyone else involved in collections 
> for doing and pushing this stuff.

Thanks - and it would be just a playground for me to write API tests
against without all the excellent work Carl has put into the UI to
make everything actually usable and useful.

> If there are smaller issues I can help with, I’d be thrilled.

The number one thing I encourage everyone to do to help is to build
awesome tools and workflows and put them in the tool shed.

> Can’t stress enough how much this feature means for galaxy adoption in our 
> lab and possibly field.

Shhh... don't tell them I am secretly wasting money they would like to
put into building a platform for sequencing data analysis to address
mass spec use cases - they will stop paying me.

Thanks for the kind note,
-John

>
> cheers,
> —
> Jorrit Boekel
> Proteomics systems developer
> BILS / Lehtiö lab
> Scilifelab Stockholm, Sweden
>
>
>
> 

Re: [galaxy-dev] galaxy and torque - resource allocation

[John Chilton]

Was hoping someone with an actual torque cluster would respond. I
think the way you would configure this might be for instance might be
setting native_specification like "-l nodes=1:ppn=n". Depending on how
things are configured or simply because I am ignorant - this may be
wrong - but I think what you really want should look a lot like your
arguments to qsub. So let say you have defined a job runner plugin
called pbs_drmaa at the top of your job_conf.xml file. Then you could
default everything to a single core by default and send "big" jobs to
a destination with 8 cores with destinations and tool sections that
look something like this...

...

  

  -l nodes=1:ppn=1


  -l nodes=1:ppn=8

  

and ...

  







  

Again - that native specification could be wrong - probably best to
test it out locally (or maybe Nate or JJ will step in and correct me).

Hope this helps,
-John


On Mon, Jan 19, 2015 at 8:05 AM, Fernandez Edgar
 wrote:
> Hello gents,
>
> Once again I would like to convey my most sincere appreciation for your 
> help!!!
> And yes I would like to hear your elaboration on my DRMAA runner which is 
> what I'm using.
> So my installation looks like: galaxy --> pbs_drmaa --> torque
>
> Cordialement / Regards,
> Edgar Fernandez
>
> -Message d'origine-
> De : John Chilton [mailto:jmchil...@gmail.com]
> Envoyé : January-18-15 8:59 PM
> À : Fernandez Edgar
> Cc : galaxy-...@bx.psu.edu
> Objet : Re: [galaxy-dev] galaxy and torque - resource allocation
>
> Galaxy generally defers to the DRM (torque in your case) for dealing with 
> these things. In your job_conf.xml you can specify limits for memory or CPUs 
> and Galaxy will pass these along to the DRM at which point it is up to the 
> DRM to enforce these - details depend on if you are using the PBS runner or 
> the DRMAA runner (let me know which and I can try to elobrate if that would 
> be useful).
>
> In your particular case - I don't believe torque "schedules" RAM so things 
> will generally only be... throttled... by CPUs counts. This is what I was 
> told by the admins at MSI when I worked there anyway. If you want to place 
> hard limits on RAM I think you need to "upgrade" to the MOAB scheduler or 
> switch over to a different DRM entirely like SLURM. Even for DRMs that deal 
> more directly with memory - Galaxy doesn't provide a general mechanism for 
> passing this along to the tool
> (https://trello.com/c/3RkTDnIn) - so it would be up to the tool to interact 
> with the environment variables your DRM sets .
>
> This sounds really terrible in the abstract - but it reality it usually isn't 
> an issue - most of Galaxy's multi-core mappers say have relatively low memory 
> per CPU usage - and for things like assemblers where this is more important - 
> one can usually just assign them to their own CPU or something like that to 
> ensure they get all the memory available.
>
> Unlike memory - Galaxy will attempt to pass the number of slots allocated to 
> a job to the tools by setting the GALAXY_SLOTS environment variable. All of 
> the multiple core devteam Galaxy tools at this point use this and so should 
> work - as should probably most multi-core tools from the tool shed - at least 
> the most popular ones.
>
> Hope this helps,
>
> -John
>
>
> On Tue, Jan 13, 2015 at 11:52 AM, Fernandez Edgar 
>  wrote:
>> Hello gents,
>>
>>
>>
>> Hope everyone had a great holiday break!
>>
>> Wish you guys all the best for 2015!
>>
>>
>>
>> I have a couple of questions about how resources (CPU and memory) is
>> allocated when you have a galaxy and torque installation.
>>
>>
>>
>> So I’ve setup torque with some default and maximum amount of CPU and
>> memory allocations.
>>
>> However, I have some worries when it comes to application (like tophat
>> for example).
>>
>> By default, it takes half the CPU of a server unless specified otherwise.
>>
>> How is the CPU allocation is specified to application like tophat
>> through galaxy?
>>
>>
>>
>> Also, how does galaxy react if a job needs more memory than the limit
>> set by torque?
>>
>>
>>
>> Any information would help me a lot!
>>
>>
>>
>> My sincere salutations to you all!!!
>>
>>
>>
>> Cordialement / Regards,
>>
>>
>>
>> Edgar Fernandez
>>
>> System Administrator (Linux)
>>
>> Direction Générale des Technologies de l'Information et de l

Re: [galaxy-dev] concatenate_datasets docker image can't create output

[John Chilton]

My first thought is the bug mentioned here -
https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-November/020892.html
along with potential work arounds. Are you able to confirm that this
is or is not the problem at all?

-John

On Fri, Jan 23, 2015 at 2:37 PM, Jeltje van Baren
 wrote:
> Hi,
>
> I'm following instructions at
> https://github.com/apetkau/galaxy-hackathon-2014.
>
> When I try to run the concatenate-datasets on two input files, two odd
> things happen. First, TWO nearly identical commands are generated in the
> History panel, only differing in their output filename. In paster.log, only
> the second one shows up. Second, the program fails, with this info in the
> History:
>
> An error occurred with this dataset:
> Galaxy slots passed through contain as 1
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh:
> line 2: can't create
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat
>
> This happens for both commands in the History panel, only the output
> filename in the other error is dataset_15.dat
>
> Oddly, the dataset_14.dat and dataset_15.dat are both created during this
> command, they just end up empty.
>
> Paster.log:
>
> galaxy.jobs.runners DEBUG 2015-01-23 11:08:35,973 (10) command is: docker
> inspect busybox:ubuntu-14.04 > /dev/null 2>&1
> [ $? -ne 0 ] && docker pull busybox:ubuntu-14.04 > /dev/null 2>&1
>
> docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:ro
> -v
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:ro
> -v
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10:rw
> -v
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw
> -w
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10
> --net none busybox:ubuntu-14.04
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh;
> return_code=$?; if [ -f
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file
> ] ; then cp
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/working_file
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_15.dat
> ; fi; sh -c "exit $return_code"
> galaxy.jobs.runners.local DEBUG 2015-01-23 11:08:36,040 (10) executing job
> script:
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/galaxy_10.sh
> galaxy.jobs DEBUG 2015-01-23 11:08:36,091 (10) Persisting job destination
> (destination id: docker_local)
>
>
> What could be happening here, and how do I fix it?
>
> Thanks,
>
> -Jeltje
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] LD_LIBRARY_PATH not exporting to galaxy

[John Chilton]

Had some idea but they were all dead ends. How are you sourcing the
modules - in Galaxy's environment, using an environment file, using
Galaxy dependencies (env.sh files) or using Galaxy's modules support?
Do you have an example of such a module that sets LD_LIBRARY_PATH?

One thing to check is the Galaxy's .bashrc file - make sure it isn't
resetting LD_LIBRARY_PATH somehow.

-John

On Fri, Jan 23, 2015 at 9:34 PM, Nikhil Joshi  wrote:
> Hi all,
>
> So we have forked the stable branch of galaxy and are making our own
> modifications. We are running galaxy on Ubuntu 14.04.1 LTS and we're using
> modules to load the underlying software, i.e. set environment variables.
> This works just fine for PATH, JAVA_JAR_PATH, PYTHONPATH, etc., however,
> LD_LIBRARY_PATH is not being passed to the tools. When a tool (such as
> cuffdiff) tries to access LD_LIBRARY_PATH, it is empty. We've checked to
> make sure that loading the cufflinks module correctly updates the
> LD_LIBRARY_PATH, however, when it is used in galaxy, all the other
> environment variables are set properly, but not LD_LIBRARY_PATH. Does anyone
> have any ideas on why this would happen and how to fix it?
>
> - Nik.
>
> --
> Nikhil Joshi
> Bioinformatics Analyst/Programmer
> UC Davis Bioinformatics Core
> http://bioinformatics.ucdavis.edu/
> najoshi -at- ucdavis -dot- edu
> 530.752.2698 (w)
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tool development - Selecting a single item from input dataset.

[John Chilton]

I have created a Trello card to track this request here
https://trello.com/c/qCtBBB8n.

Any chance you can share the tool with me? I understand that this
instinct is too simplistic - but it seems to me like the option to
repeatedly run a tool over many datasets should not be the concern of
the tool author - it is a user concern. If we add the option - perhaps
we could stipulate that it requests some other parameter to dependent
on it (I assume you have a dependent parameter in the repeat?).

-John

On Thu, Jan 22, 2015 at 5:50 AM, Vimalkumar Velayudhan
 wrote:
> Thanks Peter. I see how this feature would be useful, but the program I'm
> writing a wrapper for has an argument with values corresponding to the input
> files. I am using a  tag to maintain this order. With the multi-run
> option, files are selected in a random manner and added to the job queue. It
> is best not to display the multi-run option in this case.
>
> I see there is a TODO on this already:
> https://bitbucket.org/galaxy/galaxy-dist/src/a2308bdc93b897af974766b190abe019ade49e9a/lib/galaxy/tools/parameters/basic.py?at=default#cl-2084
>
> For now, I have set allow=False but I believe this is best set at the param
> tag level:
>
> 
>
>
> Vimal
>
> On Wed, Jan 21, 2015 at 2:26 AM, Peter Cock 
> wrote:
>>
>> I think this is the (relatively new) Galaxy ability to automatically
>> run N copies of your tool given N input files, making N outputs
>> and is related to the collections work.
>>
>> (This is possible if your tool takes a single input file)
>>
>> Peter
>>
>> On Tue, Jan 20, 2015 at 6:17 PM, Vimalkumar Velayudhan
>>  wrote:
>> > Hi all,
>> >
>> > I am trying to create a select box with the possibility of selecting
>> > only a
>> > single item from the input dataset (figure 1). This works fine but the
>> > option for selecting multiple files is still visible (figure 2). The
>> > multiple="false" attribute has no effect.
>> >
>> > Figure: http://i.imgur.com/oJVFCoF.png
>> >
>> > I have the following in my XML.
>> >
>> > > >label="Select Ribo-Seq alignment file" multiple="false" >
>> > 
>> >
>> > Any suggestions?
>> >
>> > galaxy-dist revision 5f4c13d622b8
>> >
>> >
>> > Regards,
>> > Vimalkumar Velayudhan
>> >
>> > ___
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client.  To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> >   https://lists.galaxyproject.org/
>> >
>> > To search Galaxy mailing lists use the unified search at:
>> >   http://galaxyproject.org/search/mailinglists/
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] data collections - workflow - bug?

[John Chilton]

Hey,

  Really intensive database operations including datasest collections
but other things too like multi-running tools or workflows over many
individual datasets for instance - can very easily overwhelm the
default sqlite database. This is frustrating and shouldn't happen -
but it does unfortunately. I would recommend using a postgres database
when testing out dataset collections. The good news is that it is
easier than ever to get a fully fledged production-quality server
thanks to Bjoern's docker server
(https://github.com/bgruening/docker-galaxy-stable) - it comes bundled
with Postgres and Slurm so it should be able to handle the collection
operations. If you need to run Galaxy on a non-containerized server
(for instance because that is where the software is) more information
on setting up Galaxy can be found here
https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer.

Here is a Trello card to track progress on the database optimization
efforts if you are interested https://trello.com/c/UPLsMKQI.

Very sorry.

-John



-John

On Mon, Jan 26, 2015 at 9:35 AM, Torsten Houwaart
 wrote:
> Hello Galaxy Devs,
>
> I was using data collections (for the first time) for a new workflow of ours
> and I ran into this problem. There was no complaint by the workflow-editor
> and I could start the workflow but then  happened.
> If you need more information about the workflow or otherwise let me know.
>
> Best,
> Torsten H.
>
>
> job traceback:
> Traceback (most recent call last):
>   File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py",
> line 565, in finish_job
> job_state.job_wrapper.finish( stdout, stderr, exit_code )
>   File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line
> 1250, in finish
> self.sa_session.flush()
>   File
> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/scoping.py",
> line 114, in do
> return getattr(self.registry(), name)(*args, **kwargs)
>   File
> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/session.py",
> line 1718, in flush
> self._flush(objects)
>   File
> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/session.py",
> line 1789, in _flush
> flush_context.execute()
>   File
> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/unitofwork.py",
> line 331, in execute
> rec.execute(self)
>   File
> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/unitofwork.py",
> line 475, in execute
> uow
>   File
> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/persistence.py",
> line 59, in save_obj
> mapper, table, update)
>   File
> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/persistence.py",
> line 485, in _emit_update_statements
> execute(statement, params)
>   File
> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
> line 1449, in execute
> params)
>   File
> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
> line 1584, in _execute_clauseelement
> compiled_sql, distilled_params
>   File
> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
> line 1698, in _execute_context
> context)
>   File
> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
> line 1691, in _execute_context
> context)
>   File
> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/default.py",
> line 331, in do_execute
> cursor.execute(statement, parameters)
> DBAPIError: (TransactionRollbackError) deadlock detected
> DETAIL:  Process 3144 waits for ShareLock on transaction 2517124; blocked by
> process 3143.
> Process 3143 waits for ShareLock on transaction 2517123; blocked by process
> 3144.
> HINT:  See server log for query details.
>  'UPDATE workflow_invocation SET update_time=%(update_time)s WHERE
> workflow_invocation.id = %(workflow_invocation_id)s' {'update_time':
> datetime.datetime(2015, 1, 26, 14, 20, 4, 155440), 'workflow_invocation_id':
> 5454}
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to t

Re: [galaxy-dev] concatenate_datasets docker image can't create output

[John Chilton]

Can you send me a screenshot of the Tool form right before you hit
submit. It is odd that two commands are executed barely differing at
all. There were some recent UI changes and I want to make sure that
you are passing two inputs once and not submitting one input twice in
the newly named "batch mode"?

container.sh is generated in command_factory.py using containers.py -
here are some relevant links.

https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/jobs/command_factory.py?at=default
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/containers.py?at=default
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/docker_util.py?at=default

-John


On Mon, Jan 26, 2015 at 2:17 PM, Jeltje van Baren
 wrote:
> I'm still trying to debug this.
>
> How do I change the working directory? I can't seem to find out how all
> those directory mounts get passed to docker - the catDocker.xml appears to
> generate a 'cat' command that I assume ends up in
> ()/job_working_directory/000/12/container.sh
> but I can't find where that's happening - container.sh isn't listed in any
> file under galaxy-dist.
> job_conf.xml lists the directories that get passed as variables - again,
> when and where are those defined?
>
> Thanks,
>
> -Jeltje
>
>
> On Fri, Jan 23, 2015 at 1:35 PM, Jeltje van Baren
>  wrote:
>>
>> I tried the second solution (setting everything to rw) and while I can
>> confirm that the command is changed accordingly, the results are the same:
>> Two empty output files and the same error message.
>>
>> Command from paster.log:
>> docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
>> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:rw
>> -v
>> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:rw
>> -v
>> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12:rw
>> -v
>> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files:rw
>> -w
>> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12
>> --net none busybox:ubuntu-14.04
>> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/container.sh;
>> return_code=$?; if [ -f
>> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file
>> ] ; then cp
>> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/working_file
>> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_19.dat
>> ; fi; sh -c "exit $return_code"
>>
>> -Jeltje
>>
>> On Fri, Jan 23, 2015 at 12:24 PM, John Chilton 
>> wrote:
>>>
>>> My first thought is the bug mentioned here -
>>>
>>> https://lists.galaxyproject.org/pipermail/galaxy-dev/2014-November/020892.html
>>> along with potential work arounds. Are you able to confirm that this
>>> is or is not the problem at all?
>>>
>>> -John
>>>
>>> On Fri, Jan 23, 2015 at 2:37 PM, Jeltje van Baren
>>>  wrote:
>>> > Hi,
>>> >
>>> > I'm following instructions at
>>> > https://github.com/apetkau/galaxy-hackathon-2014.
>>> >
>>> > When I try to run the concatenate-datasets on two input files, two odd
>>> > things happen. First, TWO nearly identical commands are generated in
>>> > the
>>> > History panel, only differing in their output filename. In paster.log,
>>> > only
>>> > the second one shows up. Second, the program fails, with this info in
>>> > the
>>> > History:
>>> >
>>> > An error occurred with this dataset:
>>> > Galaxy slots passed through contain as 1
>>> >
>>> > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/10/container.sh:
>>> > line 2: can't create
>>> >
>>> > /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/files/000/dataset_14.dat
>>> >
>>> > This happens for both commands in the History panel, only the output
>>> > filename in the other error is dataset_15.dat
>>> >
>>> > Oddly, the dataset_14.dat and dataset

Re: [galaxy-dev] data collections - workflow - bug?

[John Chilton]

Sorry again about this mixup (bringing conversation back on channel).
Does this happen everytime you run a workflow with collections? Do you
know how many job handlers Galaxy is configured with?

-John

On Mon, Jan 26, 2015 at 11:24 AM, John Chilton  wrote:
> Ugh misread sqlalchemy as sqlite. I will work on this.
>
> -John
>
> On Mon, Jan 26, 2015 at 11:16 AM, Torsten Houwaart
>  wrote:
>> Hi John,
>>
>> i ran this on the Galaxy server in Freiburg: galaxy.uni-freiburg.de
>> Björn (who sits immediately opposite of my desk :) ) told me it's a Postres
>> database and that shouldn't be the problem.
>>
>> Best,
>> Torsten
>>
>>
>> On 26.01.2015 16:48, John Chilton wrote:
>>>
>>> Hey,
>>>
>>>Really intensive database operations including datasest collections
>>> but other things too like multi-running tools or workflows over many
>>> individual datasets for instance - can very easily overwhelm the
>>> default sqlite database. This is frustrating and shouldn't happen -
>>> but it does unfortunately. I would recommend using a postgres database
>>> when testing out dataset collections. The good news is that it is
>>> easier than ever to get a fully fledged production-quality server
>>> thanks to Bjoern's docker server
>>> (https://github.com/bgruening/docker-galaxy-stable) - it comes bundled
>>> with Postgres and Slurm so it should be able to handle the collection
>>> operations. If you need to run Galaxy on a non-containerized server
>>> (for instance because that is where the software is) more information
>>> on setting up Galaxy can be found here
>>> https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer.
>>>
>>> Here is a Trello card to track progress on the database optimization
>>> efforts if you are interested https://trello.com/c/UPLsMKQI.
>>>
>>> Very sorry.
>>>
>>> -John
>>>
>>>
>>>
>>> -John
>>>
>>> On Mon, Jan 26, 2015 at 9:35 AM, Torsten Houwaart
>>>  wrote:
>>>>
>>>> Hello Galaxy Devs,
>>>>
>>>> I was using data collections (for the first time) for a new workflow of
>>>> ours
>>>> and I ran into this problem. There was no complaint by the
>>>> workflow-editor
>>>> and I could start the workflow but then  happened.
>>>> If you need more information about the workflow or otherwise let me know.
>>>>
>>>> Best,
>>>> Torsten H.
>>>>
>>>>
>>>> job traceback:
>>>> Traceback (most recent call last):
>>>>File
>>>> "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py",
>>>> line 565, in finish_job
>>>>  job_state.job_wrapper.finish( stdout, stderr, exit_code )
>>>>File "/usr/local/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line
>>>> 1250, in finish
>>>>  self.sa_session.flush()
>>>>File
>>>>
>>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/scoping.py",
>>>> line 114, in do
>>>>  return getattr(self.registry(), name)(*args, **kwargs)
>>>>File
>>>>
>>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/session.py",
>>>> line 1718, in flush
>>>>  self._flush(objects)
>>>>File
>>>>
>>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/session.py",
>>>> line 1789, in _flush
>>>>  flush_context.execute()
>>>>File
>>>>
>>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/unitofwork.py",
>>>> line 331, in execute
>>>>  rec.execute(self)
>>>>File
>>>>
>>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/unitofwork.py",
>>>> line 475, in execute
>>>>  uow
>>>>File
>>>>
>>>> "/usr/local/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/orm/persistence.py",
>>>> line 59, in save_obj
>>>>  mapper, table, update)
>>>>File
>>>>
>

Re: [galaxy-dev] condor jobs

[John Chilton]

On Fri, Jan 30, 2015 at 9:24 AM, Shrum, Donald C  wrote:
> Hi all,
>
>
>
> I’ve configured one of our tools to submit jobs to our condor cluster.  I
> can see the job is routed to the condor runner:
>
>
>
> ==> handler4.log <==
>
> galaxy.jobs.handler DEBUG 2015-01-30 09:14:58,092 (508) Dispatching to
> condor runner
>
> galaxy.jobs DEBUG 2015-01-30 09:14:58,204 (508) Persisting job destination
> (destination id: condor)
>
>
>
> I can see that indeed the job is submitted to the condor cluster:
>
> [root@galaxy galaxy-dist]# condor_q
>
> -- Submitter: galaxy.local : <10.177.61.90:55265> : galaxy.local
>
> ID  OWNERSUBMITTED RUN_TIME ST PRI SIZE CMD
>
>   21.0   galaxy  1/30 09:15   0+00:00:02 R  0   0.0  galaxy_509.sh
>
>
>
>
>
> The job begins to run:
>
> ==> handler4.log <==
>
> galaxy.jobs.runners.condor DEBUG 2015-01-30 09:15:03,827 (508/20) job is now
> running
>
> galaxy.jobs.runners.condor DEBUG 2015-01-30 09:15:05,183 (508/20) job has
> completed
>
>
>
> Galaxy is almost immediately removing the job working directory:
>
>
>
> Here is a snippet of the errors:
>
> ==> handler4.log <==
>
> galaxy.jobs.runners DEBUG 2015-01-30 09:15:06,372 (508/20) Unable to cleanup
> /panfs/storage.local/opt/galaxy-dist/database/pbs/galaxy_508.ec: [Errno 2]
> No such file or directory:
> '/panfs/storage.local/opt/galaxy-dist/database/pbs/galaxy_508.ec'
>
> galaxy.jobs DEBUG 2015-01-30 09:15:06,816 setting dataset state to ERROR
>
> galaxy.datatypes.metadata DEBUG 2015-01-30 09:15:06,996 Failed to cleanup
> MetadataTempFile temp files from
> /panfs/storage.local/galaxy-data/job_working_directory/000/508/metadata_out_HistoryDatasetAssociation_717_zrzVqh:
> No JSON object could be decoded
>
>
>
>
>
> Is it possible galaxy is attempting to query condor and see if the job is
> running, not finding anything and deciding that the job is not running and
> bailing out?
>
>
>
> I’ve reconstructed the process step by step using the logs but I have not
> been able to see exactly where the condor_submit command is shown so I can
> try to submit the same job manually.
>
>
>
> Does anyone have a suggestion for debugging this?

Two most relevant files would be:

https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/jobs/runners/condor.py
https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/jobs/runners/util/condor/__init__.py

The condor job logic strikes me as pretty brittle - it has always
worked for me when I have tested it but given how it is readiing logs
I would imagine very small changes to condor might cause it to fail.
So one thing to check is summarize_condor_log in
lib/galaxy/jobs/runners/util/condor/__init__.py - to see if that logic
matches the way your condor produces log files.

Given the symptoms - it might be also worth just sleeping for 5
seconds in condor_submit in
lib/galaxy/jobs/runners/util/condor/__init__.py and then verifying
that Galaxy is actually properly parsing the correct external id.

Here is an untested diff that adds the sleep and some more log
statements that might help:
https://gist.github.com/jmchilton/d0afd7242370642d5b43

If you are able to fix the problem - please let us know how so we can
fix it upstream.

-John

>
>
>
> Thanks,
>
> Don
>
> Florida State University
>
> Research Computing Center
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Pulsar on a remote (SGE) cluster

[John Chilton]

On Thu, Jan 29, 2015 at 4:43 PM, Joseph Brent Greer
 wrote:
> Hi all,
>
>
>
> We have Pulsar working in a synchronous fashion using RESTful services on
> our local network. Now, we’re trying to use Pulsar on a remote (SGE) cluster
> and we will not have the ability to mount a shared file system.
>
>
>
> What is the best way to use Galaxy to queue jobs?

Having a shared file system is definitely the best way to go if at all
possible. If you definitely cannot make that happen - than Pulsar is
probably the way to go - but it has limitations - it cannot for
instance leverage tool shed installed dependencies and it certainly
doesn't scale as well as Galaxy itself yet. I am working on these
problems and am happy to help work through the problems you encounter
- but I just like to throw out the warning about Pulsar.

> If Pulsar is the answer,
> how do we defer to the cluster’s queue and have Pulsar wait on it?

So it sounds like you have experience setting up Pulsar - you will now
need to set it up one a node connected to the remote cluster and open
a port for it (unless you want to setup a message queue - but I
recommend the RESTful mode when possible - it seems more robust
currently).

Once you have Pulsar setup - you will need to configure it to talk to
your SGE cluster. For that you will need to install an SGE drmaa
library on the node (it may already be available). Then copy
local_env.sh.sample to local_env.sh in Pulsar and setup
DRMAA_LIBRARY_PATH.

export DRMAA_LIBRARY_PATH=/path/to/your/libdrmaa.so

and finally setup a job_managers.ini file (cp job_managers.ini.sample
job_managers.ini) and change the default section to something like:

[manager:_default_]
type=queued_drmaa

More information about configuring job managers here
https://pulsar.readthedocs.org/en/latest/job_managers.html.

Finally - you need to configure Galaxy to use the correct Pulsar job
runner - it might look something like this -
https://gist.github.com/jmchilton/ca39ef1a3241d9074121.

That should be it - and then Galaxy should send jobs to the cluster as needed.

> How should Pulsar notify Galaxy when multiple jobs are done running?

Galaxy will poll the remote Pulsar server to determine when the jobs
are complete. If you cannot open a port for pulsar to do this polling
- then you will need to setup a message queue and then configure both
sides to use that - I would definitely try to bully your cluster
admins into opening that port before resorting to that though.

>
>
>
> We’ve looked through the documentation and listservs and haven’t found
> anything directly related.

Pulsar was previously called the LWR - so there are more conversations
related to it on galaxy-dev referring to it as the LWR.

>
>
>
> Any help or suggestions would be great.

Hopefully this helps and good luck!

-John

>
>
>
> Thanks,
>
>
>
> Joe Greer
>
> Proteomics Center of Excellence
>
> Northwestern University
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] concatenate_datasets docker image can't create output

[John Chilton]

Based on the screenshot you are calling the program correctly.

Can you tell me what version of Docker you are running and Galaxy you
are running? There have been a number of fixes in both programs over
the last six months to a year. Docker is known to consume a lot of
space in /var - it probably doesn't have much to do with Galaxy or
this tool right. It looks like you Docker command line does not
include -rm - which was added recently to have Docker dispose of
instances after it done with them. This might help with your /var
space issue - though in general Docker does use a lot a relatively
large amount of disk under /var - it is the nature of the technology
and many traditional HPC clusters for instance may allocate relatively
small /var partitions.

Upgrading Galaxy might also help the permission thing - I don't see a
'-u ' argument in your command-line - which is another newer
default to help ensure the resulting files have the correct
permissions (especially when sudo is disabled - which it seems it is
in your configuration).

Hope this helps,

-John



On Mon, Jan 26, 2015 at 7:02 PM, Jeltje van Baren
 wrote:
> Our sysadmin just notified me docker was taking up large amounts of space in
> /var/lib/docker/devicemapper/devicemapper/data
> and
> /var/lib/docker/devicemapper/devicemapper/metadata
>
> I noticed the following DEBUG message in paster.log:
>
> galaxy.jobs.runners.local DEBUG 2015-01-23 13:30:19,503 (12) executing job
> script:
> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/database/job_working_directory/000/12/galaxy_12.sh
> galaxy.jobs DEBUG 2015-01-23 13:30:19,552 (12) Persisting job destination
> (destination id: docker_local)
>
> Are these related? (the files I'm trying to concatenate are small text files
> so that's not it)
>
>
> On Mon, Jan 26, 2015 at 11:47 AM, Jeltje van Baren
>  wrote:
>>
>> See attached for screenshot!
>>
>> Thanks for the links.
>>
>>
>>
>> On Mon, Jan 26, 2015 at 11:37 AM, John Chilton 
>> wrote:
>>>
>>> Can you send me a screenshot of the Tool form right before you hit
>>> submit. It is odd that two commands are executed barely differing at
>>> all. There were some recent UI changes and I want to make sure that
>>> you are passing two inputs once and not submitting one input twice in
>>> the newly named "batch mode"?
>>>
>>> container.sh is generated in command_factory.py using containers.py -
>>> here are some relevant links.
>>>
>>>
>>> https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/jobs/command_factory.py?at=default
>>>
>>> https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/containers.py?at=default
>>>
>>> https://bitbucket.org/galaxy/galaxy-central/src/tip/lib/galaxy/tools/deps/docker_util.py?at=default
>>>
>>> -John
>>>
>>>
>>> On Mon, Jan 26, 2015 at 2:17 PM, Jeltje van Baren
>>>  wrote:
>>> > I'm still trying to debug this.
>>> >
>>> > How do I change the working directory? I can't seem to find out how all
>>> > those directory mounts get passed to docker - the catDocker.xml appears
>>> > to
>>> > generate a 'cat' command that I assume ends up in
>>> > ()/job_working_directory/000/12/container.sh
>>> > but I can't find where that's happening - container.sh isn't listed in
>>> > any
>>> > file under galaxy-dist.
>>> > job_conf.xml lists the directories that get passed as variables -
>>> > again,
>>> > when and where are those defined?
>>> >
>>> > Thanks,
>>> >
>>> > -Jeltje
>>> >
>>> >
>>> > On Fri, Jan 23, 2015 at 1:35 PM, Jeltje van Baren
>>> >  wrote:
>>> >>
>>> >> I tried the second solution (setting everything to rw) and while I can
>>> >> confirm that the command is changed accordingly, the results are the
>>> >> same:
>>> >> Two empty output files and the same error message.
>>> >>
>>> >> Command from paster.log:
>>> >> docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v
>>> >>
>>> >> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist:rw
>>> >> -v
>>> >>
>>> >> /inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/tools/docker:rw
>>> >> -v

Re: [galaxy-dev] Galaxy installation inside a secure environment

[John Chilton]

On Thu, Jan 15, 2015 at 10:05 AM, Abdulrahman Azab  wrote:
> Hi Galaxy people,
>
>
> At the University of Oslo, we have an infrastructure for storage and
> computation on sensitive data, called TSD (presented in the attached image).
> from inside, the infrastructure is logically divided into projects. Each
> project has a set of shared VMs (among its users) which are running over a
> file-system (HNAS). The shared project VMS has access to a computational
> SLURM cluster which is installed on another file-system (Colossus). There
> are directories in colossus which are mounted on HNAS for each project in
> order to allow each project users to push files into colossus and run jobs
> on the cluster. Users of a particular project can access the project main
> VMs through user VMs (one for each user). Each project VMs (both shared and
> user VMs) are in a separate subnet.
>
>
> All of this is inside the TSD. Now to access the TSD from outside, there is
> a complex authentication mechanism where a user can access his/her own VM.
> And to transfer data from/to the TSD is another complex story.  The
> important thing is that there is NO internet access in or out.
>
>
> There are two issues here:
>
>
> 1 - What we need to do is to install one Galaxy VM inside each project area,
> so that it is accessible by all project users. But we cannot use mercurial
> to access your distribution server. We can though install a bitbucket server
> inside the TSD and have the code-base there, so that It can be accessed by
> all project VMs, but I'm not sure what is the procedure here.

You can clone mercurial (or in the future git) repositories to some
file system that is accessible both internal to the firewalls and
external to them (I believe there has to be some file system like this
- even if it is just a USB stick - in order to install new software :)
). You can then treat the repository in the shared location as the
source of Galaxy and clone/update against it.

At a very high-level the initial clone process might look like this:

(desktop) % ssh login_node
(login_node) % cd /shared_directory/
(login_node) % hg clone https://bitbucket.org/galaxy/galaxy-dist galaxy-dist
(login_node) % exit
(desktop) % ssh secure_node
(secure_node) % cd /project_directory ; hg clone
/shared_directory/galaxy-dist galaxy-dist
(secure_node) % cd galaxy-dist ; hg update latest_2015.01.13
(secure_node) % exit
(desktop) %

then doing updates might look like this:

(desktop) % ssh login_node
(login_node) % cd /shared_directory/
(login_node) % hg pull https://bitbucket.org/galaxy/galaxy-dist galaxy-dist
(login_node) % exit
(desktop) % ssh secure_node
(secure_node) % cd /project_directory/galaxy-dist
(secure_node) % hg update latest_future_tag_name
(secure_node) % exit
(desktop) %


>
> 2- We are very concerned about the issue of regularly updating Galaxy
> instances in projects to the recent release. In many cases it causes many
> problems, e.g. tool versioning conflicts. So we have the idea of installing
> each of our tools together with all of its dependencies in a separate docker
> container, and run those as images on each Galaxy project VM. Is this
> possible and tested? Should this permanently solve the upgrading problem, or
> do you suggested another alternative?

Tool dependencies can be installed in Docker containers - several
people have tested it - and while there can be some problems getting
everything setup (a lot of moving pieces) I think it works fine once
setup.

I really like Aaron Petkau's tutorial here:
https://github.com/apetkau/galaxy-hackathon-2014
https://github.com/apetkau/galaxy-hackathon-2014/smalt

Another blog post with more information was put together by the Galaxy
User Group Grand Ouest here:
https://www.e-biogenouest.org/groups/guggo/wiki/FirstGenOuest

And Galaxy's wiki documentation can be found here:
https://wiki.galaxyproject.org/Admin/Tools/Docker

Kyle Ellrott is really getting the Dockerized tools to run at scale -
and we have worked with him to really scale things up across a large
cluster. I will try to update the wiki with some of that information.

Setting up a local tool shed might be another approach to address this
reproduciblity/maintenance problem - but I generally discourage the
use of local tool sheds (but cannot access the Internet might be a
very good reason to set this up).

Hope this helps,
-John

>
>
> Thank you,
>
> Yours sincerely,
> Abdulrahman Azab
>
> Head engineer, ELIXIR.NO / The Genomic HyperBrowser team
> Department of Informatics, University of Oslo, Boks 1072 Blindern, NO-0316
> OSLO, Norway
> Email: a...@ifi.uio.no, Cell-phone: +47 46797339
> 
> Senior Lecturer in Computer Engineering
> Faculty of Engineering, University of Mansoura, 35516-Mansoura, Egypt
> Email: abdulrahman.a...@mans.edu.eg
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and o

Re: [galaxy-dev] galaxy report page - server error solution?

[John Chilton]

Hey Guys,

  Thanks for the patches - it looks like Eric Enns came up with similar
ones a while back (
http://dev.list.galaxyproject.org/Patch-in-reference-to-trello-card-968-td4660438.html).
The bad news is I just went through and made a bunch PEP-8 fixes to the
reports code so all the patches are potentially invalid now - the good news
is I followed up with an alternative fix that should maintain the current
behavior for postgres but also work with MySQL based on your patches (
https://bitbucket.org/galaxy/galaxy-central/commits/ce2776f88e9db53886b8879a948b19ce2a513835)
- so hopefully Galaxy will work without modification with the next release
(March I suppose).

Thanks again!

-John

P.S. Like Hans said though - if you can migrate Galaxy off MySQL take the
opportunity to do so though.


On Thu, Feb 5, 2015 at 12:00 PM, Fernandez Edgar <
edgar.fernan...@umontreal.ca> wrote:

>  Here’s some other modifications I’ve made that should be logged
> somewhere :
>
>
>
> diff workflows.py ARCHIVES/workflows.py_20150205_11.37.57
>
> 143c143
>
> < q = sa.select( ( sa.func.date(
> model.StoredWorkflow.table.c.create_time ).label( 'date' ),sa.func.count(
> model.StoredWorkflow.table.c.id ).label( 'total_workflows' ) ),
>
> ---
>
> > q = sa.select( ( sa.func.date_trunc( 'month', sa.func.date(
> model.StoredWorkflow.table.c.create_time ) ).label( 'date' ),sa.func.count(
> model.StoredWorkflow.table.c.id ).label( 'total_workflows' ) ),
>
> 145c145
>
>  model.StoredWorkflow.table.c.create_time ), sa.func.month ( sa.func.date(
> model.StoredWorkflow.table.c.create_time ) ) ],
>
> ---
>
> >group_by = [ sa.func.date_trunc( 'month',
> sa.func.date( model.StoredWorkflow.table.c.create_time ) ) ],
>
> 177c177
>
> < q = sa.select( ( sa.func.date(
> model.StoredWorkflow.table.c.create_time ).label( 'date' ),
>
> ---
>
> > q = sa.select( ( sa.func.date_trunc( 'month', sa.func.date(
> model.StoredWorkflow.table.c.create_time ) ).label( 'date' ),
>
> 181c181
>
>  model.StoredWorkflow.table.c.create_time ), sa.func.month ( sa.func.date(
> model.StoredWorkflow.table.c.create_time ) )],
>
> ---
>
> >group_by = [ sa.func.date_trunc( 'month',
> sa.func.date( model.StoredWorkflow.table.c.create_time ) ) ],
>
>
>
>
>
> diff sample_tracking.py ARCHIVES/sample_tracking.py_20150205_11.37.50
>
> 136c136
>
> < q = sa.select( ( sa.func.date( model.Request.table.c.create_time
> ).label( 'date' ),sa.func.count( model.Request.table.c.id ).label(
> 'total' ) ),
>
> ---
>
> > q = sa.select( ( sa.func.date_trunc( 'month', sa.func.date(
> model.Request.table.c.create_time ) ).label( 'date' ),sa.func.count(
> model.Request.table.c.id ).label( 'total' ) ),
>
> 138c138
>
>  model.Request.table.c.create_time ), sa.func.month ( sa.func.date(
> model.Request.table.c.create_time ) ) ],
>
> ---
>
> >group_by = [ sa.func.date_trunc( 'month',
> sa.func.date( model.Request.table.c.create_time ) ) ],
>
> 169c169
>
> < q = sa.select( ( sa.func.date( model.Request.table.c.create_time
> ).label( 'date' ),
>
> ---
>
> > q = sa.select( ( sa.func.date_trunc( 'month', sa.func.date(
> model.Request.table.c.create_time ) ).label( 'date' ),
>
> 173c173
>
>  model.Request.table.c.create_time ), sa.func.month ( sa.func.date(
> model.Request.table.c.create_time ) ) ],
>
> ---
>
> >group_by = [ sa.func.date_trunc( 'month',
> sa.func.date( model.Request.table.c.create_time ) ) ],
>
>
>
>
>
>
>
> Cordialement / Regards,
>
> Edgar Fernandez
>
>
>
>
>
> -Message d'origine-
> De : Fernandez Edgar
> Envoyé : February-05-15 10:55 AM
> À : 'Hans-Rudolf Hotz'; galaxy-...@bx.psu.edu
> Objet : RE: [galaxy-dev] galaxy report page - server error solution?
>
>
>
> Good catch on that one!
>
> There wasn't originally the date_trunc function there but that groups the
> jobs by months together.
>
> Thanks!!!
>
>
>
> Cordialement / Regards,
>
> Edgar Fernandez
>
>
>
> -Message d'origine-
>
> De : Hans-Rudolf Hotz [mailto:h...@fmi.ch ] Envoyé :
> February-05-15 10:49 AM À : Fernandez Edgar; galaxy-...@bx.psu.edu
> Objet : Re: [galaxy-dev] galaxy report page - server error solution?
>
>
>
> Hi again
>
>
>
> for "jobs.py" I have modified the same lines as you,
>
>plus line 285 (group_by)
>
>
>
> for "users.py I have modified the same lines as you
>
>
>
>
>
> Hans-Rudolf
>
>
>
>
>
>
>
> On 02/05/2015 04:20 PM, Fernandez Edgar wrote:
>
> > Actually, here’s what I came up with :
>
> >
>
> > diff users.py ARCHIVES/users.py_20150205_9.28.18
>
> >
>
> > 23c23
>
> >
>
> > < q = sa.select( ( sa.func.date(
>
> > galaxy.model.User.table.c.create_time ).label( 'date' ),
>
> >
>
> > ---
>
> >
>
> >

Re: [galaxy-dev] "This job is waiting to run"

[John Chilton]

Assuming grid engine is working outside of Galaxy and the Galaxy user
can say qsub files just fine - then you need to install the sun grid
engine libdrmaa package - or build from scratch. Assuming you have
done that and the resulting .so file is
/usr/lib/gridengine-drmaa/lib/libdrmaa.so. Then all you should have to
do is create a config/job_conf.xml file in your Galaxy root that looks
something like this:





/usr/lib/gridengine-drmaa/lib/libdrmaa.so















A slightly more complicated setup with supervisor, uwsgi, multiple
handlers, etc... all on grid engine is covered in this tutorial that
Nate put together last year
(https://wiki.galaxyproject.org/Events/GCC2014/TrainingDay/AdminWalkthrough#Configure_Galaxy).
This job_conf.xml is a stripped down version of the one from that
tutorial.

In terms of rolling back and just getting jobs to start running again
- you need to remove $GALAXY_ROOT/config/job_conf.xml and
$GALAXY_ROOT/job_conf.xml and things should start working using the
local runner again. If this isn't working - there maybe a problem in
your galaxy.ini file - I would need to know what properties were being
set in there to know more though.

-John


On Mon, Feb 2, 2015 at 12:53 PM, Ryan G  wrote:
> I tried setting up Galaxy to use Sun Grid Engine, but failed miserably.  I
> removed my config/job_conf.xml hoping Galaxy would revert to its default
> state, but jobs are no longer being dispatched.  I don't see any errors in
> the log.
>
> How can I get Galaxy to 1) use SGE, or 2) revert to its default state for
> running jobs.  The page,
> https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster isn't 100%
> clear on how to set up a simply configuration for SGE.  I'd be happy to
> contribute mine as an example, if I can get it running.
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Bug? Uploaded files not adding to history (Nginx upload)

[John Chilton]

I believe we updated usegalaxy.org and Bjoern Docker galaxy images to
the latest Galaxy release - both which leverage this feature and
didn't encounter issues with either. Is it possible you updated nginx
as part of the release as well? At least in the past the upload
feature has had problems with compatibility across various versions of
nginx.

-John

On Tue, Feb 3, 2015 at 7:53 AM, Federico Zambelli
 wrote:
> Hi everybody,
>
> I have recently updated our lab Galaxy instance to the latest stable
> latest_2015.01.13. I have now a problem with uploaded files: they are not
> added to history.
> The problem disappears if I disable the nginx upload feature (that worked
> just fine until the update). Maybe it is of help to know that the files are
> correctly uploaded to the "nginx_upload_store" directory configured in
> galaxy.ini but then they are not added to the current history.
>
> Best,
> Federico Z.
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] tool xml to fetch choices from a URL

[John Chilton]

It may be possible to do this using a select field plus dynamic
options. The cummeRbund tool seems to use a select that uses arbitrary
Python to generate options - but I may be missing something.

https://github.com/galaxyproject/tools-devteam/tree/master/tools/cummeRbund

If it really does that - than it would seem to be possible to ping a
remote URL as part of that option building.

Hope this helps,

-John


On Sun, Feb 8, 2015 at 8:52 AM, Langhorst, Brad  wrote:
> Hi:
>
> I need users to specify a project when they run a tool…
> I would like this to be a select list whose set of options comes from a
> remote url.
>
> What’s the best way to achieve that UI?
>
> Should I try to add a from_url attribute to the options tag?
>
> Thanks,
>
>
> Brad
> --
> Brad Langhorst, Ph.D.
> Applications and Product Development Scientist
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] (no subject)

[John Chilton]

I am not sure what the problem is - any chance you can share your
nginx configuration file?

On Fri, Feb 20, 2015 at 1:13 PM, Briand, Sheldon
 wrote:
> Hi,
>
>
>
> I’m seeing the following error when trying to use upload file with a local
> file that is larger than 8Kb:
>
>
>
> galaxy.web.framework.decorators ERROR 2015-02-19 16:51:48,280 Uncaught
> exception in exposed API method:
>
> Traceback (most recent call last):
>
>   File "/BigData/galaxy/galaxy-dist/lib/galaxy/web/framework/decorators.py",
> line 135, in decorator
>
> rval = func( self, trans, *args, **kwargs)
>
>   File "/BigData/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/api/tools.py",
> line 124, in create
>
> tool = trans.app.toolbox.get_tool( payload[ 'tool_id' ] ) if 'tool_id'
> in payload else None
>
>   File "/BigData/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py", line 448,
> in get_tool
>
> if tool_id in self.tools_by_id and not get_all_versions:
>
> TypeError: unhashable type: 'list'
>
>
>
> The end of the file in the upload store:
>
> GGTGGATCACAAGGTCAGGAGATCGAGACCATCCTGGTTAACATGATGAAAGTGTCTACTAT
>
> ACAA-206772567414653
>
> Content-Disposition: form-data; name="history_id"
>
>
>
> Null
>
>
>
> I’m using nginx 1.6.2 with the upload_file module and the latest stable
> version of galaxy.
>
>
>
> What am I messing up?
>
>
>
> Thanks,
>
> -Sheldon
>
>
>
> Sheldon Briand
>
>
>
> NRC Research Computing Support Analyst / Operations, Science Portfolio
>
> Shared Services Canada / Government of Canada
>
> sheldon.bri...@ssc-spc.gc.ca / Tel: 902-426-1677
>
>
>
> CNRC Analyst de Support Informatique de Recherche/ Operations, Portefeuil
> des sciences
>
> Services partagés Canada /Gouvernement du Canada
>
> sheldon.bri...@ssc-spc.gc.ca / Tél: 902-426-1677
>
>
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] tophat error

[John Chilton]

Sorry for this messy situation. With the next release Galaxy will be
able to set metadata using any version of samtools due to an excellent
fix by Nate - so this hopefully will become less of an issue.

-John

On Mon, Feb 23, 2015 at 10:01 AM, Peter Cock  wrote:
> On Mon, Feb 23, 2015 at 2:13 PM, Michael Thon  wrote:
>>
>> I figured that galaxy must be finding a samtools v1.1 even though the tool
>> was not installed, according to the admin page. I uninstalled the samtools
>> that was installed system-wide on the server.  Now I get this error:
>>
>> Traceback (most recent call last):
>>   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py", line
>> 129, in queue_job
>> job_wrapper.finish( stdout, stderr, exit_code )
>>   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 1107, in
>> finish
>> dataset.datatype.set_meta( dataset, overwrite=False )  # call
>> datatype.set_meta directly for the initial set_meta call during dataset
>> creation
>>   File "/home/galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py", line 250,
>> in set_meta
>> raise Exception, "Error Setting BAM Metadata: %s" % stderr
>> Exception: Error Setting BAM Metadata: /bin/sh: 1: samtools: not found
>>
>> So, galaxy is not using the samtools from the installed galaxy package and
>> is instead using the system one.  I'll try to reinstall samtools and see if
>> that fixes it...
>
> In this case Galaxy itself expects samtools on the $PATH for managing
> BAM files separately from which ever tool you are using within Galaxy.
>
> The Galaxy Tool Shed dependencies are on a per-tool basis, and not
> used by Galaxy itself.
>
> This ought to be on the wiki somewhere, but I failed to find it...
>
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Hooking a dataset full deletion

[John Chilton]

I don't think the scripts that Galaxy has that deployers can opt to
run to say periodically purge deleted datasets, etc... would respect
such a hack anyway - so I would be hesitant to add a sort of porous
hook (not unwilling to, just hesitant). Setting up a cron job to
monitor the datasets table and figure out which new datasets have been
purged periodically might prove to be more robust and would allow you
to do this without modifying the core of Galaxy. There are heavy
tradeoffs with this approach obviously - it is more coding and there
is some time delay between deletion and the execution of desired task.

Hope this helps,

-John

On Wed, Feb 18, 2015 at 10:21 AM, Raffaele Montella
 wrote:
> Thanks Dannon!
> Actually I’d like to avoid code hacking enforcing NTCC (no touch core
> code!).
> Anyway, if there is no other way I’ll share the hack with the community,
> maybe could be part of next core improvements.
> Bests,
> — Raffaele
>
>
> On 18 Feb 2015, at 07:19, Dannon Baker  wrote:
>
> There isn't a hook that already exists for this, but if this is for a custom
> instance and you're looking to hack something in, you could add extra logic
> to lib/galaxy/managers/hdas.py in the 'purge' method.  Let me know if you
> need more information, good luck!
>
> -Dannon
>
> On Tue Feb 17 2015 at 6:37:52 PM Raffaele Montella
>  wrote:
>>
>> Hi,
>>
>> I need to perform some actions when a dataset if fully deleted.
>> Does it exist a hook in datatype definition to do that?
>> Thanks,
>>   Raffaele
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] custom java tools.xml : problem with classpath

[John Chilton]

Just to make sure - since it hasn't been explicitly stated - is this
tool being installed from the tool shed? The environment variable
setup in tool_dependencies.xml will only be available for tool shed
installs. You will also need a special requirement in your tool to
make it available
(https://github.com/bgruening/galaxytools/blob/master/augustus/augustus.xml#L6).

I consider this whole approach very heavy and not robust given that it
is explicitly tied to the tool shed. If there is any way you can
explicitly target the next release with your code there is a much
better solution to this problem - you can use $__tool_directory__ to
reference the directory tools are installed in
(https://bitbucket.org/galaxy/galaxy-central/commits/ddbe0d12bc86da406bb96213e23b8ddb273ed669).
It works for tool shed installations and tools just directly loaded
into Galaxy - it also doesn't require you to setup anything special in
tool_dependencies.xml or the requirements section. It is a cheetah
variable instead of a runtime environment variable so need to escape
the $ either.

If you are on an older version of Galaxy - the diff above is pretty
small - you should be possible to backport this enhancement locally
with minimal effort.

Hope this helps,

-John


On Thu, Feb 26, 2015 at 8:35 AM, Pierre Lindenbaum
 wrote:
>> You'll need to white list the semi-colon which Galaxy has replaced as a
>> security precaution (user free text can be exploited). See  here:
>> https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax Peter
>
>
> Thanks: adding :
>
>
>   
>  
> 
>  
>
>
> fixed the probem.
>
>> Could you confirm if the environment variable is being set now?
>
> Hum the variable is not set , I've changed  to
>
> 
>
>
> here is the content of tmp/error7.txt
>
> $ cat /tmp/error7.txt
> JVARKITDIR= expression=variant!=null;
> Error: Could not find or load main class
> com.github.lindenb.jvarkit.tools.vcffilterjs.VCFFilterJS
>
>
>
> P.
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Importing file via FTP can silently fail, [Errno ftp error] 550

[John Chilton]

Yeah - that is unfortunate - I agree completely that the resulting
datasets should be red. I have created a Trello card here:

https://trello.com/c/A6LrdjUU

-John

On Thu, Feb 26, 2015 at 5:19 AM, Peter Cock  wrote:
> On Wed, Feb 25, 2015 at 5:33 PM, Peter Cock  wrote:
>> Hi all,
>>
>> Using either the legacy upload tool, or the new upload widget, the
>> following FTP URL *appears* to load into Galaxy correctly giving
>> a green history entry.
>>
>> However, on closer inspection it is an empty file with this peep text:
>>
>> Unable to fetch
>> ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
>> [Errno ftp error] 550 pathogens: No such file or directory
>>
>> Clearly this should have been treated as a failure (red history entry).
>>
>> Should I file this as a bug on Trello?
>>
>> It seems there is a problem with the Sanger folder permissions which
>> trips up Galaxy and curl (but does not bother Firefox and wget):
>>
>> $ curl -O 
>> ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
>>   % Total% Received % Xferd  Average Speed   TimeTime Time  
>> Current
>>  Dload  Upload   Total   SpentLeft  Speed
>>   0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- >> 0
>> curl: (9) Server denied you to change to the given directory
>>
>> Using curl -v for verbose mode it tries to walk the path (as per the
>> https://www.ietf.org/rfc/rfc1738.txt standard) and fails with the
>> command "CWD pathogens" matching the Galaxy error.
>
> The reply from Sanger was that "pathogens" is actually an alias
> for "project/pathogens" and as a workaround this longer URL can
> be used with curl or Galaxy:
>
> ftp://ftp.sanger.ac.uk/pub/project/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
>
> The problem with Galaxy failing to detect the FTP error with the
> shorter URL remains:
>
> ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
>
> Regards,
>
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] custom java tools.xml : problem with classpath

[John Chilton]

Sorry I wasn't clear about that - it hasn't reached galaxy-dist yet -
you can download the latest snapshot of 15.03 release (I believe it is
running on usegalaxy.org as of yesterday) via this link instead
https://bitbucket.org/galaxy/galaxy-central/get/release_15.03.zip.

Also not need to escape $__tool_directory__  (so use
$__tool_directory__ instead of \$__tool_directory__).

-John

On Thu, Feb 26, 2015 at 9:40 AM, Pierre Lindenbaum
 wrote:
> On 02/26/2015 02:51 PM, John Chilton wrote:
>>
>> Just to make sure - since it hasn't been explicitly stated - is this
>> tool being installed from the tool shed? The environment variable
>> setup in tool_dependencies.xml will only be available for tool shed
>> installs.
>
>
> no, my future plan is to put it in the tool shed but for now, I'm just
> trying to make it run in my local instance of galaxy.
>
> I'm sorry if that wasn't clear.
>
>
> I've just downloaded the latest version of galaxy via
> https://bitbucket.org/galaxy/galaxy-dist/get/tip.zip (
> galaxy-galaxy-dist-02ddea42faad.zip )
> and re-tried.
>
>
> I tried to print the variable $__tool_directory__ with the tool.xml below
>
> but it's not defined
>
> $ cat /tmp/error7.txt
> dir= .
> Error: Could not find or load main class
> com.github.lindenb.jvarkit.tools.vcffilterjs.VCFFilterJS
>
>
> the xml:
>
>
>id="com.github.lindenb.jvarkit.tools.vcffilterjs.VCFFilterJS"
>version="1.0.0" name="vcffilterjs">
>   
> java
>   
>
>   
>   
> 
> 
> 
>  
>  
> 
> 
> 
>
> 
>   
> 
> 
>   
> 
> 
>   
>   
>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Importing file via FTP can silently fail, [Errno ftp error] 550

[John Chilton]

Yes - I believe it does. The top level layer (controller) is a little
different because it is coming through the API - but the next 10 or so
layers related to uploading are the same with the new method :).

-John

On Thu, Feb 26, 2015 at 9:57 AM, Peter Cock  wrote:
> On Thu, Feb 26, 2015 at 2:24 PM, John Chilton  wrote:
>> Yeah - that is unfortunate - I agree completely that the resulting
>> datasets should be red. I have created a Trello card here:
>>
>> https://trello.com/c/A6LrdjUU
>>
>> -John
>
> Thanks John,
>
> Does the new "Download from URL or upload files from disk" interface
> still end up calling the old Python script for the upload tool? i.e.
>
> https://github.com/galaxyproject/galaxy/blob/dev/tools/data_source/upload.py
>
> Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] custom java tools.xml : problem with classpath

[John Chilton]

Well - this version does feature Sam's complete rewrite of the front
end for tools - so things will be pretty different.

Oddly though your tool loaded for me - can you open up your JavaScript
console and let me know if there are any errors?

-John

On Thu, Feb 26, 2015 at 10:14 AM, Pierre Lindenbaum
 wrote:
> On 02/26/2015 03:43 PM, John Chilton wrote:
>>
>> Sorry I wasn't clear about that - it hasn't reached galaxy-dist yet -
>> you can download the latest snapshot of 15.03 release (I believe it is
>> running on usegalaxy.org as of yesterday) via this link instead
>> https://bitbucket.org/galaxy/galaxy-central/get/release_15.03.zip.
>>
>> Also not need to escape $__tool_directory__  (so use
>> $__tool_directory__ instead of \$__tool_directory__).
>>
>> -John
>>
>
> thank you for your help John & Peter
>
> hum... is there anything new with this version about tool.xml ?. I can see
> my tool in the right menu but the main page is blank (no input fields)
>
>127.0.0.1 - - [26/Feb/2015:16:09:07 +0200] "GET
>
> /tool_runner?tool_id=com.github.lindenb.jvarkit.tools.vcffilterjs.VCFFilterJS
>HTTP/1.1" 200 - "http://127.0.0.1:8080/"; "Mozilla/5.0 (X11; Ubuntu;
>Linux i686; rv:35.0) Gecko/20100101 Firefox/35.0"
>127.0.0.1 - - [26/Feb/2015:16:09:17 +0200] "GET
>
> /tool_runner?tool_id=com.github.lindenb.jvarkit.tools.vcffilterjs.VCFFilterJS
>HTTP/1.1" 200 - "http://127.0.0.1:8080/"; "Mozilla/5.0 (X11; Ubuntu;
>Linux i686; rv:35.0) Gecko/20100101 Firefox/35.0"
>127.0.0.1 - - [26/Feb/2015:16:09:27 +0200] "GET
>
> /tool_runner?tool_id=com.github.lindenb.jvarkit.tools.vcffilterjs.VCFFilterJS
>HTTP/1.1" 200 - "http://127.0.0.1:8080/"; "Mozilla/5.0 (X11; Ubuntu;
>Linux i686; rv:35.0) Gecko/20100101 Firefox/35.0"
>127.0.0.1 - - [26/Feb/2015:16:09:28 +0200] "GET
>
> /tool_runner?tool_id=com.github.lindenb.jvarkit.tools.vcffilterjs.VCFFilterJS
>HTTP/1.1" 200 - "http://127.0.0.1:8080/"; "Mozilla/5.0 (X11; Ubuntu;
>Linux i686; rv:35.0) Gecko/20100101 Firefox/35.0"
>
>
> $ xmllint --format tools/jvarkit2/vcffilterjs.xml
> 
>  id="com.github.lindenb.jvarkit.tools.vcffilterjs.VCFFilterJS"
> version="1.0.0" name="vcffilterjs">
>   
> java
>   
>   
>   
> 
> 
>   
> 
>   
> 
>   
> 
>   
>   
> 
>   
>   
> 
> 
>   
>   Help
> 
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] problem with latest toolshed devteam cuffmerge wrapper multi-select

[John Chilton]

Is there any chance I can get you to reproduce this problem on the Galaxy
test server:
https://test.galaxyproject.org/root?tool_id=toolshed.g2.bx.psu.edu%2Frepos%2Fdevteam%2Fcuffcompare%2Fcuffcompare%2F2.2.1.0
and
then share the history with me (jmchil...@gmail.com)?

We have fixed some bugs related to this recently and it would be
interesting to know if it works in the forthcoming release. My initial
attempts to recreate with the latest release have failed so this might be
fixed.

Any clue how old the Galaxy instance in your CloudMan setup is? I would be
eager to try to fix problems in the 15.01 release as well - older than that
though and we don't generally fix non-security related issues.

-John

On Fri, Feb 20, 2015 at 4:28 AM, Nikhil Joshi  wrote:

> So I am doing some debugging and it looks like in the from_html function
> in the SelectToolParameter class, it gets the legal values for the build
> using the get_legal_values function. This returns the correct values when I
> choose only one gtf file, but when I choose multiple gtf files the list of
> legal values is empty. I haven't figured out why this is happening.
> - Nik.
>
> On Wed, Feb 18, 2015 at 10:45 PM, Nikhil Joshi 
> wrote:
>
>> And, yes, all of the gtf files have the correct database build associated
>> with them.
>>
>> On Wed, Feb 18, 2015 at 6:29 PM, Nikhil Joshi 
>> wrote:
>>
>>> Hi all,
>>>
>>> We run our own custom galaxy instance in the Amazon Cloud and I am
>>> updating our instance with the latest toolshed files. For cuffmerge, in
>>> past revisions, we would just have a button to add another GTF input.
>>> However, this latest one (10:b6e3849293b1
>>> )
>>> that has been replaced with a multi-select. We also have a few built-in
>>> genomes that we use with cuffmerge. When I choose the first GTF file (using
>>> locally cached sequence data) it will populate the reference genome
>>> correctly with our built-in genomes. However, once I choose more than one
>>> GTF file, I get one of two errors:
>>>
>>>   No reference genome is available for the build associated with the
>>> selected input dataset
>>> OR
>>>  An invalid option was selected, please verify
>>>
>>> The built-in genomes are definitely there and available for cuffmerge,
>>> but for some reason choosing more than one GTF file in the multi-select
>>> causes an error. Is this a bug or am I doing something wrong?
>>>
>>> - Nik.
>>>
>>>
>>>
>>> --
>>> Nikhil Joshi
>>> Bioinformatics Analyst/Programmer
>>> UC Davis Bioinformatics Core
>>> http://bioinformatics.ucdavis.edu/
>>> najoshi -at- ucdavis -dot- edu
>>> 530.752.2698 (w)
>>>
>>
>>
>>
>> --
>> Nikhil Joshi
>> Bioinformatics Analyst/Programmer
>> UC Davis Bioinformatics Core
>> http://bioinformatics.ucdavis.edu/
>> najoshi -at- ucdavis -dot- edu
>> 530.752.2698 (w)
>>
>
>
>
> --
> Nikhil Joshi
> Bioinformatics Analyst/Programmer
> UC Davis Bioinformatics Core
> http://bioinformatics.ucdavis.edu/
> najoshi -at- ucdavis -dot- edu
> 530.752.2698 (w)
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] $GALAXY_TEST_DB_TEMPLATE failing with KeyError:

[John Chilton]

Hey Peter,

  That database is pretty new (not yet in stable) - but you are
tracking the master branch of galaxy on github in your tests (which is
sortof like stable). You will want to either track the dev branch (so
you can continue to catch all the errors right away :)) or track
master and use an older database.

  Someday soon I will move this logic into Galaxy's run_tests.sh so
that Galaxy always knows what version of the database it needs and we
don't have to deal with these version incompatibilities. Sorry for the
confusion.

Thanks,
-John




On Fri, Feb 27, 2015 at 9:07 AM, Peter Cock  wrote:
> Hi all,
>
> I've been using the SQLite template database feature to speed up
> functional testing. Rather than starting for the initial schema and
> applying all the incremental changes (as done when installing a
> fresh Galaxy instance), this lets the test suite start with a recent
> version of the database, by setting the environment variable
> $GALAXY_TEST_DB_TEMPLATE to a URL to fetch a template
> database, e.g. from:
>
> https://github.com/jmchilton/galaxy-downloads
>
> Database schema version 120 was working, changing to 127 broke:
>
> https://github.com/peterjc/pico_galaxy/commit/412d3fe88b8011f0b690956d129fd72f5b5d4c5c
>
> TravisCI using v120, worked:
> https://travis-ci.org/peterjc/pico_galaxy/builds/52308064
>
> TravisCI using v127, broke:
> https://travis-ci.org/peterjc/pico_galaxy/builds/52398778
>
> galaxy.model.migrate.check DEBUG 2015-02-27 10:05:33,274 pysqlite>=2
> egg successfully loaded for sqlite dialect
> Traceback (most recent call last):
>   File "./scripts/functional_tests.py", line 602, in 
> sys.exit( main() )
>   File "./scripts/functional_tests.py", line 409, in main
> app = UniverseApplication( **kwargs )
>   File 
> "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/app.py",
> line 49, in __init__
> self._configure_models( check_migrate_databases=True,
> check_migrate_tools=check_migrate_tools, config_file=config_file )
>   File 
> "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/config.py",
> line 779, in _configure_models
> create_or_verify_database( db_url, config_file,
> self.config.database_engine_options, app=self )
>   File 
> "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/model/migrate/check.py",
> line 61, in create_or_verify_database
> migrate()
>   File 
> "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/model/migrate/check.py",
> line 57, in migrate
> migrate_to_current_version( engine, db_schema )
>   File 
> "/home/travis/build/peterjc/pico_galaxy/galaxy-master/lib/galaxy/model/migrate/check.py",
> line 117, in migrate_to_current_version
> changeset = schema.changeset( None )
>   File 
> "/home/travis/build/peterjc/pico_galaxy/galaxy-master/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/schema.py",
> line 80, in changeset
> changeset = self.repository.changeset(database, start_ver, version)
>   File 
> "/home/travis/build/peterjc/pico_galaxy/galaxy-master/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/repository.py",
> line 225, in changeset
> changes = [self.version(v).script(database, op) for v in versions]
>   File 
> "/home/travis/build/peterjc/pico_galaxy/galaxy-master/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/repository.py",
> line 189, in version
> return self.versions.version(*p, **k)
>   File 
> "/home/travis/build/peterjc/pico_galaxy/galaxy-master/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/version.py",
> line 140, in version
> return self.versions[VerNum(vernum)]
> KeyError: 
>
> This is now testing against the (beta) Galaxy on GitHub:
> https://github.com/galaxyproject/galaxy
>
> Is it possible this is out of synch with the SQLite templates John
> posts here? https://github.com/jmchilton/galaxy-downloads
>
> Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Database deadlock with large workflows + dataset collections

[John Chilton]

Hey Aaron,

Thanks for the bug report - I have added it to Trello here
(https://trello.com/c/I0n23JEP). I assume this is the January stable
release (15.01)? Any clue if this workflow was being scheduled in a
web thread or a background job handler thread? (Did the submission
take forever - or was the web page responsive and the server bogged
down).

So there is a race condition here it would seem - and I don't have a
fix right away - but I do think (hope) the next release due out in a
couple of weeks will result in a massive speed up in workflow
scheduling - so hopefully we will be less likely to hit these
conditions.

-John

On Fri, Feb 27, 2015 at 10:11 AM, Aaron Petkau  wrote:
> Hey,
>
> I wanted to know if anyone else has had experience with database deadlock
> when using dataset collections and running a large number of samples through
> a workflow.
>
> Traceback (most recent call last):
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py",
> line 565, in finish_job
> job_state.job_wrapper.finish( stdout, stderr, exit_code )
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py",
> line 1250, in finish
> self.sa_session.flush()
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py",
> line 114, in do
> return getattr(self.registry(), name)(*args, **kwargs)
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py",
> line 1718, in flush
> self._flush(objects)
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py",
> line 1789, in _flush
> flush_context.execute()
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py",
> line 331, in execute
> rec.execute(self)
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.py",
> line 475, in execute
> uow
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py",
> line 59, in save_obj
> mapper, table, update)
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.py",
> line 485, in _emit_update_statements
> execute(statement, params)
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py",
> line 1449, in execute
> params)
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py",
> line 1584, in _execute_clauseelement
> compiled_sql, distilled_params
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py",
> line 1698, in _execute_context
> context)
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py",
> line 1691, in _execute_context
> context)
>   File
> "/Warehouse/Applications/irida/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.6-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.py",
> line 331, in do_execute
> cursor.execute(statement, parameters)
> DBAPIError: (TransactionRollbackError) deadlock detected
> DETAIL:  Process 25859 waits for ShareLock on transaction 144373; blocked by
> process 25858.
> Process 25858 waits for ShareLock on transaction 144372; blocked by process
> 25859.
> HINT:  See server log for query details.
>  'UPDATE workflow_invocation SET update_time=%(update_time)s WHERE
> workflow_invocation.id = %(workflow_invocation_id)s' {'update_time':
> datetime.datetime(2015, 2, 27, 3, 51, 57, 81403), 'workflow_invocation_id':
> 48}
>
> I saw this post
> http://dev.list.galaxyproject.org/data-collections-workflow-bug-td4666496.html
> with a similar issue and the solution was to make sure not to use a sqlite
> database, but I'm using a postgres database and still encountered this
> issue.  This was after running a very large number of samples (~200) using
> dataset collections.  Just wondering if anyone else was running into this
> issue?
>
> Thanks,
>
> Aaron
>
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep

Re: [galaxy-dev] Tool Testing: field inputs that occur inside repeat tag - resolved

[John Chilton]

Glad you got it working!

Tests that use tool data tables should work in planemo - however you
will need to configure the data table for the tool though. Here is an
example of the how to setup a test data for a tool data table:

https://github.com/jmchilton/picard/commit/4df8974384081ee1bb0f97e1bb8d7f935ba09d73

planemo will detect the tool data table file automatically if it is
sitting new to the tool or an explicit file can be passed in with
(--tool_data_table)
http://planemo.readthedocs.org/en/latest/commands.html#test-command.

-John

On Sat, Feb 28, 2015 at 7:15 AM, Peter Cock  wrote:
> On Fri, Feb 27, 2015 at 10:36 PM, Dooley, Damion  
> wrote:
>> Yay, with your tip, galaxy direct tests now work with tool_conf.xml AND data 
>> table in place!
>>
>> Galaxy DOC person: please update 
>> https://wiki.galaxyproject.org/Admin/RunningTests
>> to state that tool_conf.xml.sample is NOT used for testing!  It is stated in 
>> the
>> run_tests.sh help info half way down the page.
>
> Done - well spotted. I updated the run_test.sh help output that line was from.
>
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Parallelism using metadata

[John Chilton]

Hey Marco,

Thanks for the e-mail. This is an awesome idea, but I am worried it is
very hard to do this well in Galaxy. If you create symbolic links to
the original file - then Galaxy might delete the original file and the
derived files would all break without warning. Galaxy does have this
separate concept of datasets and history dataset associations so that
a dataset can exist in more than one place simultaneously - and one
could imagine sticking this metadata there and just sort of
dynamically splitting up the BAM file whenever it is used in a tool or
served out over the API - but this would be a large effort and would
require all sorts of modifications to various parts of Galaxy.

Something worth looking at is this work by Kyle Ellrott:

https://bitbucket.org/galaxy/galaxy-central/pull-request/175/parameter-based-bam-file-parallelization

This was a very localized effort to just work some of the ideas just
into the task splitting framework in Galaxy. This has the advantage of
not needing to mess with metadata and datasets, etc all the way up
the chain.

Kyle has abandoned that approach however, but it is promising start to
something like this I think - and it would be much less disruptive
than doing this with metadata and datasets (though admittedly more
limited as well).

If this will be primarily used for workflows - there are a couple of
recent developments that might make splitting more feasible. Dannon
introduced the ability to delete intermediate outputs from workflows a
few releases ago - and the upcoming release (15.03) will introduce the
ability to write tools that split up a single input into a collection.
The existing workflow and dataset collection framework can then apply
normal tools over every element of the collection and you can write a
tool to merge the results. More information can be found here -
https://bitbucket.org/galaxy/galaxy-central/pull-request/634/allow-tools-to-explicitly-produce-dataset.

These common pipelines where you split up a BAM files, run a bunch of
steps, and then merge the results will be executable in the near
future (though 15.03 won't have workflow editor support for it - I
will try to get to this by the following release - and you can
manually build up workflows to do this -
https://bitbucket.org/galaxy/galaxy-central/src/0468d285f89c799559926c94f300c42d05e8c47a/test/api/test_workflows.py?at=default#cl-544).

Thanks again,
-John


On Fri, Feb 27, 2015 at 10:04 PM, Marco Albuquerque
 wrote:
> Hello Galaxy Dev,
>
> I have a question regarding parallelism on a BAM file.
>
> I have currently implemented 3 split options for the BAM datatype
>
> 1) by_rname -> splits the bam into files based on the chromosome
> 2) by_interval -> splits the bam into files based on  a defined bp length,
> and does so across the entire genome present in the BAM file
> 3) by_read -> splits the bam into files based on the number of reads
> encountered (if multiple files, all other files match the interval as the
> first)
>
> Now, as you can imagine, reading and writing large BAM files is a pain, and
> I personally think this is not the best solution for Galaxy.
> What I was hoping to implement (but don't know how) is to create a new
> metadata option in bam (bam.metadata.bam_interval) which would generate the
> interval without creating a new file (essentially, I would create a symbolic
> link to the old large file, and then update the metadata.bam_interval, this
> would contain some string of the form chrom:start-end which could then be
> used in a variety of tools which accept an interval as an option (for
> example samtools view))
>
> This would be far more efficient then my first implementation, but the thing
> I don't know how to do is specify some kind of metadata at the split level.
> I was hoping maybe you could direct me to an example that does this?
>
> I have added the following to my metadata.py file:
>
> class IntervalParameter( MetadataParamter )
>
> def __init__( self, spec ):
> MetadataParamter.__init__( self, spec ):
> self.rname = self.spec.get( "rname" )
> self.start = self.spec.get( "start" )
> self.end = self.spec.get( "end" )
>
> def to_string(self):
> if self.rname = 'all':
> return ''
> else:
> return ''.join([self.rname, ':', self.start, '-', self.end])
>
> And the following to my binary.py file:
>
> ### UNDER THE BAM CLASS
>
> MetadataElement( name="bam_interval", desc="BAM Interval",
> param=metadata.IntervalParameter, rname="all", start="", end="",
> visible=False, optional=True)
>
>
> I somehow want rname="all" to be the default, but upon parallelism, I want
> to be able to adjust this parameter in the split functions.
>
> So,
>
>  split_mode="by_interval" split_size="5000" merge_outputs="output"/>
>
> Would actually change the metadata of each file, and not create sub-bams.
>
>
> PLEASE HELP!!!
>
> Marco
>
> ___
> 

Re: [galaxy-dev] RFC: new param attribute for Galaxy tool XML

[John Chilton]

Peter,

Galaxy parameters should be case sensitive I think - they are used in
plain dictionaries quite a bit and I have never seen any logic to make
them not case sensitive.

Bjoern,

Is this what you mean?

https://github.com/galaxyproject/galaxy/compare/dev...jmchilton:argument_name?expand=1

If it is what you want - I can open a pull request for it.

I guess I am a little skeptical this is useful. I would like to
believe that Galaxy tool for the most part don't need to map to a
single application underneath, but we don't have to rehash that
argument. If the parameter name and help is not more useful than the
underlying tool's parameter - why would the tool author change it in
the first place? Jen tells me the parameter name is important, you
tell me the parameter name is important, etc... I respect you guys
greatly and so I will defer to you - but it is frustrating we are
breaking abstractions that the tool author set up (presumably for a
reason).

-John


On Tue, Mar 3, 2015 at 11:38 AM, Peter Cock  wrote:
> Hi Björn,
>
> Command line arguments are often case sensitive
> (e.g. samtools switches), but are Galaxy parameter
> names?
>
> Peter
>
> On Sat, Feb 28, 2015 at 9:11 PM, Björn Grüning
>  wrote:
>> Hi all,
>>
>> we are planning to work on a project to implement a Galaxy fuse based
>> shell. Probably starting with the work from Clare [1].
>>
>> Next to our Galaxy IPython integration it should attract
>> more bioinformaticians and should offer a new way to interact with
>> Galaxy. This includes moving, deleting datasets, but also executing
>> tools and workflows. For the latter I would like to have some sort of
>> bash auto-completion. Type in your tool/workflow and you will see all
>> the parameters you can/should modify, in addition to your normal help page.
>>
>> Currently the parameter identifiers () do not need to
>> be meaningful. In fact many tools invent their own unique names, to
>> identify a parameter in the cheetah section. This name is always unique
>> but it's hard to guess it's meaning from just the name and are also not
>> mappable to the original parameter name. This makes tool execution from
>> the API sometimes hard.
>>
>> I would like to propose a new attribute for all  tags. This
>> should specify a unique value that 100% matches the original parameter
>> name of the underlying tool.
>>
>> This attribute could be used to:
>>  * automatically enhance the help text of each parameter. Currently best
>> practise it to include this parameter in brackets at the end of each
>> help text. We can do this now automatically, or only show it by mouse
>> over etc ...
>>  * In a Galaxy shell, the user could just type in a normal command (-i
>> history1/foo.bam -o history1/bar.sam) and Galaxy shell would be able
>> to map this parameter to the correct  tag in Galaxy
>>
>> I greatly appreciate any comments!
>> Thanks,
>> Bjoern
>>
>> [1]
>> https://github.com/claresloggett/gvl_commandline_utilities/blob/master/galaxy-fuse.py
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy Update Problem

[John Chilton]

Perhaps what is missing is that you need to run `hg pull` first before
doing `hg update stable`. update moves you to new changesets in your
local copy of the repository but it doesn't by default pull in updates
from the remote source for the repository.

Hope this helps,
-John

On Mon, Mar 2, 2015 at 1:32 AM, 孙兆朕  wrote:
> Hi all,
>
> I installed my Galaxy instance from
> https://bitbucket.org/galaxy/galaxy-dist/. But I did not use Mercurial to
> install Galaxy at that time, because I was not farmiliar with Mercurial and
> failed severed times. The instance imported many tool kits later, as it
> would be very difficult to restart everthing for me. I tried to use hg
> command:
>
>
> hg clone https://bitbucket.org/galaxy/galaxy-dist/
>
> cd galaxy-dist
>
> hg update stable
>
>
> to make a new copy and copied the ".hg" directory to the former instance and
> used the "hg update stable" command. Then it reports that there were not
> files to change. I wonder if there could be any way to update the instance.
>
> Thanks,
>
>
> Zhaozhen
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problem to effectively trace logs

[John Chilton]

So it looks like you are sorting by read name, and then Galaxy is
going to attempt to index the bam file using the command-line below:

% samtools sort -n tophat_out1h.bam -o - > test.bam
% samtools index test.bam test.bam.bai
[bam_index_core] the alignment is not sorted (test_mRNA_5_197_46): 166
> 54 in 1-th chr
[bam_index_build2] fail to index the BAM file.

I think Galaxy may want all tool outputs to be sorted by coordinate
and not read name.

Is it possible that sorting by read name and sorting by coordinate
produce identical ouptut in your smaller test data but not in your
larger file?

Not sure if there is a work around for where a difference sorting is
desired - maybe you have to implement a new datatype? We are wading
into bioinformatics stuffs I don't really know much about - hopefully
someone more knowledgeable will comment.

-John

On Wed, Mar 4, 2015 at 5:38 AM, Amandine VELT  wrote:
> Dear Galaxy dev team,
>
> I try to use a tool allowing to sort a bam file with the "samtools sort -n"
> command. This tool works on BAM test files but when I run it on a real BAM
> file (of 3.5 G), I got an error named "error" (see attachment) ... this does
> not help me.
>
> In the handler.log, the job is finished normally and I don't find the error
> line named "error". When I run the galaxy command directly on the server, it
> works. Is there a log file to follow more precisely the errors? Indeed, I
> don't find where the "error" comes from and so I can't resolve my problem.
>
> Here my log files :
>
>
> more galaxy_20175.e
>
> [bam_sort_core] merging from 34 files...
>
> more galaxy_20175.ec
>
> 0
>
> more galaxy_20175.o
>
> [bam_index_core] the alignment is not sorted
> (HISEQ:123:HBFN8ADXX:1:1101:1341:3979): 24-th chr > 13-th chr
> [bam_index_build2] fail to index the BAM file.
>
> These "errors" are not the problem, because I have the same on the BAM test
> files and they give no problem (I have a green box at the end of the run,
> only for tests data of course).
> Moreover, I got the following lines in my wrapper to be sure that they are
> only warnings :
>
> 
>  source="both"
> level="warning"
> description="Warning: fail to index the bam file" />
>  source="both"
> level="warning"
> description="Warning: the alignment is not sorted" />
> 
>
> more metadata_results_HistoryDatasetAssociation_35900_PbjjDh
>
> [true, "Metadata has been set successfully"]
>
> more handler0.log
>
> galaxy.jobs DEBUG 2015-03-03 10:06:25,824 (20175) Working directory for
> job is: /galaxy_data/job_working_directory/020/20175
> galaxy.jobs.handler DEBUG 2015-03-03 10:06:25,850 (20175) Dispatching to
> drmaa runner
> galaxy.jobs DEBUG 2015-03-03 10:06:25,943 (20175) Persisting job
> destination (destination id: sge)
> galaxy.jobs.handler INFO 2015-03-03 10:06:25,994 (20175) Job dispatched
> galaxy.jobs.runners.drmaa DEBUG 2015-03-03 10:06:26,584 (20175)
> submitting file /galaxy_data/job_working_directory/020/20175/galaxy_20175.sh
> galaxy.jobs.runners.drmaa DEBUG 2015-03-03 10:06:26,584 (20175) command
> is: samtools sort -n  /galaxy_data/files/028/dataset_28339.dat
> /galaxy_data/files/028/dataset_28631.dat ; mv
> /galaxy_data/files/028/dataset_28631.dat.bam
> /galaxy_data/files/028/dataset_28631.dat; return_code=$?; cd
> /data12/galaxy/galaxy-env/galaxy;
> /data12/galaxy/galaxy-env/galaxy/set_metadata.sh /galaxy_data/files
> /galaxy_data/job_working_directory/020/20175 .
> /data12/galaxy/galaxy-env/galaxy/universe_wsgi.ini
> /galaxy_data/tmp/tmp28hp1G
> /galaxy_data/job_working_directory/020/20175/galaxy.json
> /galaxy_data/job_working_directory/020/20175/metadata_in_HistoryDatasetAssociation_35900_vmNjk7,/galaxy_data/job_working_directory/020/20175/metadata_kwds_HistoryDatasetAssociation_35900_IHm6c9,/galaxy_data/job_working_directory/020/20175/metadata_out_HistoryDatasetAssociation_35900_CboGKT,/galaxy_data/job_working_directory/020/20175/metadata_results_HistoryDatasetAssociation_35900_PbjjDh,,/galaxy_data/job_working_directory/020/20175/metadata_override_HistoryDatasetAssociation_35900_XD1a0c;
> sh -c "exit $return_code"
> galaxy.jobs.runners.drmaa DEBUG 2015-03-03 10:06:26,585 (20175) native
> specification is: -q galaxy
> galaxy.jobs.runners.drmaa INFO 2015-03-03 10:06:26,818 (20175) queued as
> 160573
> galaxy.jobs DEBUG 2015-03-03 10:06:26,874 (20175) Persisting job
> destination (destination id: sge)
> galaxy.jobs.runners.drmaa DEBUG 2015-03-03 10:06:27,907 (20175/160573)
> state change: job is queued and active
> galaxy.jobs.runners.drmaa DEBUG 2015-03-03 10:06:35,538 (20175/160573)
> state change: job is running
> galaxy.jobs.runners.drmaa DEBUG 2015-03-03 10:12:10,200 (20173/160571)
> state change: job finished normally
> galaxy.datatypes.metadata DEBUG 2015-03-03 10:12:10,451 loading metadata
> from file for: HistoryDatasetAssociation 35898
> galaxy.jobs DEBUG 2015-03-

Re: [galaxy-dev] Comment on Training Documentation

[John Chilton]

Awesome work Damion - thanks for publishing and sharing these! I hope
the workshop goes well.

I recently created a wiki page just with tool development resource links

https://wiki.galaxyproject.org/Develop/ResourcesTools

that now gets embedded in other more common landing pages including

https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial
https://wiki.galaxyproject.org/Develop

I have added links to your content from to this resource list.

One quick thought on the content. The interpreter="python" idiom for
wrappers has been around for years and is used widely but it has some
serious drawbacks - in particular nothing can come before the wrapper
in the tool XML - no cheetah directives like #import or #set and no
linking in files (for instance the following setup idiom:

ln -s "${input_bam}" temp_input.bam &&
ln -s "${input_bam.metadata.bam_index}" temp_input.bam.bai &&
actual_command --args

In the planemo documentation I am not going to mention interpreter for
this reason - I think it is frustrating for people when it does not
work the way intuitively it should. The forthcoming 15.03 injects a
new variable called $__tool_directory__ that I think should be the new
best practice.

ln -s "${input_bam}" temp_input.bam &&
ln -s "${input_bam.metadata.bam_index}" temp_input.bam.bai &&
python $__tool_directory__/wrapper.py --args

I understand that it might be best to not base your tutorial on
features not even released yet :) - but I did want to take the
opportunity to mention this idiom.

As you mentioned I have also been working on updated tool development
documentation lately at (http://planemo.readthedocs.org/en/latest/). I
am trying to build a set of smaller resources that can be composed
into tailored resources (e.g. build tools without planemo, with
planemo, with planemo virtual appliance), (normal tutorial, verbose
tutorial, slide-based tutorial), (with and without Docker). I don't
know if rst is up to this task, but we will see.

Another effort along these lines worth checking out is Kyle's tool
tutorial - https://github.com/kellrott/galaxy-tooldev-docs/tree/master/docs.
This tutorial is focused on using the planemo appliance and
Dockerizing tools.

-John

On Fri, Mar 13, 2015 at 4:35 AM, Peter Cock  wrote:
> On Fri, Mar 13, 2015 at 12:23 AM, Martin Čech  wrote:
>> Your tutorial looks great Damion! Thank you for sharing.
>>
>> I am not sure about the vocabulary you are trying to estabilish though
>> (update vs revision). I understand what you mean but I would just stick with
>> 'If the tool changed behavior you have to bump the version.' No need to
>> specifically name the steps as it can cause confusion (revision has a
>> different meaning in version control world).
>
> I was also confused about your (Damion's) update vs revision terminology,
> and agree with Martin that any behaviour change (especially bug fixes,
> even minor ones) should come with a version change.
>
> I've not had time to look at your slides, but thanks for sharing them.
>
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] value_from_basic - KeyError: 'base_name', was: Tests not being run on toolsheds?

[John Chilton]

It seems to be happening during upload - so it is not unsurprising
that this error would affect multiple tools. I have checked out the
exact revision registered in the tests and the uploads seem fine to
me.

I am having trouble imagining a deployment problem short of an invalid
upload tool that might result in this error. Dave B, is it possible
that a problematic datatype is being installed by some repository and
affecting unrelated tools?

-John


On Mon, Mar 23, 2015 at 8:27 AM, Peter Cock  wrote:
> Hi Dave,
>
> Now that you've fixed some of the test back log, the bad
> news is this issue I reported last week appears to be a major
> regression affecting multiple tools on the Test Tool Shed:
>
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/align_back_trans
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod
> ...
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/nlstradamus
> ...
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs
> ...
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_select_by_id
>
> These are all recent test runs from 2015-03-18. Note that
> some tool tests from the same date are passing, e.g.
>
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/blast_rbh
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/seq_composition
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/mummer
>
> I have not yet spotted any pattern in this division.
>
> (However this is clearly not linked to the test expect_failure
> test I'd added to the sample_seqs tool, as I speculated last
> week.)
>
> Regards,
>
> Peter
>
> On Wed, Mar 18, 2015 at 2:14 PM, Peter Cock  wrote:
>> Hi Dave,
>>
>> The following looks like a regression on the Test Tool Shed, the tests
>> pass locally (using a recent revision), and on TravisCI using the current
>> galaxy dev branch on GitHub:
>> https://travis-ci.org/peterjc/pico_galaxy/builds/54870500
>>
>> Problem tool: https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs
>>
>> This may be unrelated, but by chance this is the first time I have
>> uploaded a tool to the Tool Shed which uses the new functionality
>> to test the stdout/stderr strings, and more importantly it includes a
>> test expected to fail via 
>>
>> Revision on my development repository:
>> https://github.com/peterjc/pico_galaxy/commit/55ebb308b911b4acef912cc3b03f4371f4c6dfe6
>>
>> Test Tool Shed output from last night:
>>
>> Automated test environment
>> Time tested: 2015-03-18 02:46:55
>> System: Linux 3.13.0-36-generic
>> Architecture: x86_64
>> Python version: 2.7.6
>> Galaxy revision: 17050:6395e7035143
>> Galaxy database version: 128
>> Tool shed revision: 16867:0468d285f89c
>> Tool shed database version: 25
>> Tool shed mercurial version: 3.2.4
>> Tests that failed
>> Tool id: sample_seqs
>> Tool version: sample_seqs
>> Test: test_tool_00
>> (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/sample_seqs/sample_seqs/0.2.1)
>> Stderr:
>> Traceback:
>> Traceback (most recent call last):
>>   File 
>> "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py",
>> line 268, in test_tool
>> self.do_it( td )
>>   File 
>> "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py",
>> line 37, in do_it
>> stage_data_in_history( galaxy_interactor, testdef.test_data(),
>> test_history, shed_tool_id )
>>   File 
>> "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py",
>> line 38, in stage_data_in_history
>> upload_wait()
>>   File 
>> "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py",
>> line 279, in wait
>> while not self.__history_ready( history_id ):
>>   File 
>> "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py",
>> line 297, in __history_ready
>> return self._state_ready( state, error_msg="History in error state." )
>>   File 
>> "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py",
>> line 356, in _state_ready
>> raise Exception( error_msg )
>> Exception: History in error state.
>> Traceback (most recent call last):
>>   File 
>> "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/jobs/runners/__init__.py",
>> line 158, in prepare_job
>> job_wrapper.prepare()
>>   File 
>> "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/jobs/__init__.py",
>> line 828, in prepare
>> tool_evaluator.set_compute_environment( compute_environment,
>> get_special=get_special )
>>   File 
>> "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/tools/evaluation.py",
>> line 53, in set_compute_environment
>> incoming = self.tool.params_from_strings( incoming, self.app )
>>   File 
>> "/tmp/buildslave/buildbot-install-te

Re: [galaxy-dev] test/base/twilltestcase.py - Converting local (test-data) bam to sam failed

[John Chilton]

Hey Peter,

  Thanks for continuing to apply pressure on these issues. I have
pushed some logging into Galaxy's development branch
(https://github.com/galaxyproject/galaxy/commit/8cb06d7fc2913b4d83ca01b50d76e9607bbe379d)
that should make that error more informative. I don't know when that
will get pushed out to bitbucket and the shed retested - but once
those two things happen we will hopefully have a better idea what the
problem is (probably some subtle deployment problem - some
incompatible version of samtools on the path? samtools on Galaxy's
path but not the test script's path?).

-John


On Mon, Mar 23, 2015 at 6:06 AM, Peter Cock  wrote:
> On Tue, Mar 3, 2015 at 9:55 AM, Peter Cock  wrote:
>> On Tue, Jan 27, 2015 at 2:23 PM, Peter Cock  
>> wrote:
>>> Hi all,
>>>
>>> I have a query about a failing tool test on the Tool Shed, where it
>>> seems Galaxy is trying to convert both the expected BAM output
>>> and the tool's BAM output into SAM for comparison - but this
>>> fails.
>>>
>>> https://toolshed.g2.bx.psu.edu/view/peterjc/samtools_depad
>>>
>>> 2015-01-27 02:13:51
>>> ...
>>> Converting local (test-data) bam to sam failed
>>>
>>> (and same error on the second test)
>>
>> https://toolshed.g2.bx.psu.edu/view/peterjc/samtools_depad is still failing,
>> although now with a more concise traceback:
>>
>> Test runs
>> 2015-01-29 02:26:31
>> Automated test environment
>> Tests that failed
>> Tool id: samtools_depad
>> Tool version: samtools_depad
>> Test: test_tool_00
>> (functional.test_toolbox.TestForTool_toolshed.g2.bx.psu.edu/repos/peterjc/samtools_depad/samtools_depad/0.0.1)
>> Stderr:
>> Traceback:
>> Traceback (most recent call last):
>> ...
>> Converting local (test-data) bam to sam failed
>>
>>> The same occurs on the Test Tool Shed (with a very slightly updated
>>> version of this tool), although with a more concise traceback:
>>>
>>> https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_depad
>>>
>>> 2015-01-25 08:21:59
>>> ...
>>> Tool id: samtools_depad
>>> Tool version: samtools_depad
>>> Test: test_tool_00
>>> (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/samtools_depad/samtools_depad/0.0.2)
>>> Stderr:
>>> Traceback:
>>> ...
>>> Converting local (test-data) bam to sam failed
>>>
>>
>> https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_depad also
>> still failing,
>>
>> Test runs
>> 2015-01-25 08:21:59
>> ...
>> Converting local (test-data) bam to sam failed
>
> Still failing on the Test Tool Shed, bug filed on Trello,
> https://trello.com/c/TL0IdLlG/2570-toolshed-converting-local-test-data-bam-to-sam-failed
>
> Tests currently stalled on the main Tool Shed, reported here:
> http://dev.list.galaxyproject.org/Tests-not-being-run-on-toolsheds-tc4666816.html
> https://lists.galaxyproject.org/pipermail/galaxy-dev/2015-March/021708.html
>
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Ideas: allow Fieldset in tool forms

[John Chilton]

Hello Gildas,

You may want to check out the discussion on this Trello card -
https://trello.com/c/KxlQK0FB. This is a frequently requested feature
with a lot of votes. I think Aysam is actively working on it - so I
would expect it to be in Galaxy soon.

-John



On Mon, Mar 23, 2015 at 7:33 AM, Gildas Le Corguille
 wrote:
> Dear Galaxy Team and al.,
>
> To improve the usability, it would be nice to group parameters in some kinds
> of "fieldset".
> We can already hide some advanced options with conditional but ...
>
> But it's just a personal wishes
> And as always, great job :)
>
> Cheers
>
> Gildas
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] simple way to refer to the repository install directory from a tool

[John Chilton]

Brad,

Yes - that environment variable trick is an anti-pattern in my mind
because it is heavy and because it only works for tool shed installed
tools. As of 15.03 you can now just use $__tool_directory__ to refer
to the directory the tool is in. I can point at the patch if you want
to back port this to an older Galaxy (the patch is relatively small).

Planemo's building a tool documentation has a section on writing
wrapper scripts that demonstrates the use of the new
$__tool_directory__ variable -
http://planemo.readthedocs.org/en/latest/writing_standalone.html#wrapping-a-script.

Hope this helps,
-John


On Thu, Mar 26, 2015 at 1:06 PM, Langhorst, Brad  wrote:
> I’m working on a very simple tool.
>
> 
> 
> Rscript
> samtools
> R
> 
> 
> 
> 
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> this won’t work because Rscript is not running in the tool install directory
> and can’t find simple_script.R…
>
> I’ve just been reading
>
> https://trello.com/c/9lQ7Sr8I/636-galaxy-tool-enhancement-to-accommodate-repository-install-dir
>
> and it seems that  I have to add another file…
>
> 
> 
> 
>  action="set_to">$REPOSITORY_INSTALL_DIR
> 
> 
>
> and add a set_enviornment dependency to bring it in.
> 
> ...
> TOOL_INSTALL_DIR
>  
>  
>
> I did that but the tool still won’t execute, in fact the .
> Maybe because I have to put this into a toolshed - or maybe I’ve done
> something wrong?
>
>
> I think this complexity could be avoided by setting an TOOL_INSTALL_DIR
> environment var as a tool is executed (though John doesn’t seem to like that
> idea due to a responsibility mixing issue)
>
> I think it’s a worthy goal to make tool development as frictionless as
> possible.
> Is there a simpler way to do this?
>
> Brad
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] problem with uploading SRA file to our custom galaxy

[John Chilton]

Binary datatypes like SRA need to be registered with Galaxy and have a
corresponding datatype definition class - out of the box no such
datatype is defined for SRA. Requiring this definition makes it a
little bit harder to for instance host malware on usegalaxy.org. Main
has a data source tool for fetching fastq files from EBI -

https://usegalaxy.org/tool_runner/data_source_redirect?tool_id=ebi_sra_main

To actually enable use of real SRA files locally - Matthew Shirley has
a nice tool shed package that should install the SRA toolkit and a
corresponding SRA datatypes that I would assume would allow SRA files
to be uploaded.

Hope this helps,
-John


On Tue, Mar 24, 2015 at 3:20 AM, Nikhil Joshi  wrote:
> It looks like it doesn't work in galaxy main either.
>
> On Fri, Mar 20, 2015 at 3:42 PM, Nikhil Joshi  wrote:
>>
>> Hi all,
>>
>> So it is time again for our next bioinformatics course using galaxy and I
>> am trying to add some new tools to our custom galaxy running in the Amazon
>> Cloud. I've installed the ncbi sra toolkit interface from the toolshed and
>> the sra toolkit is installed on our AMI. However, when I upload an SRA file
>> into our galaxy, the file is empty and I get this error:
>>
>> The uploaded binary file contains inappropriate content
>>
>> I am able to extract the fastq from the archive just fine on the command
>> line, so I know it's not a problem with the file itself. Is there a solution
>> to this?
>>
>> - Nik.
>>
>> --
>> Nikhil Joshi
>> Bioinformatics Analyst/Programmer
>> UC Davis Bioinformatics Core
>> http://bioinformatics.ucdavis.edu/
>> najoshi -at- ucdavis -dot- edu
>> 530.752.2698 (w)
>
>
>
>
> --
> Nikhil Joshi
> Bioinformatics Analyst/Programmer
> UC Davis Bioinformatics Core
> http://bioinformatics.ucdavis.edu/
> najoshi -at- ucdavis -dot- edu
> 530.752.2698 (w)
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Using synchronous method to upload multiple files

[John Chilton]

There was some work last summer to allow something that at least
vaguely sounds like what you are trying to accomplish - though I was
not involved so I probably am unable to help out much. The Trello card
that was tracking progress on this can be found here -
https://trello.com/c/YlADVbkI - and seems to provide some good
details. I am not sure if there is more formal documentation anywhere
though.

Hope this helps,

-John

On Fri, Mar 27, 2015 at 7:34 AM, Daniele P Colobraro
 wrote:
> Hi all,
>
> I've set up a simple tool using the synchronous method to upload a file on
> galaxy.
> I'm able to visit the external link, select and upload a file by the list.
> The script that I use is data_source.py and the 'command' part in the xml
> configuration is the same as the ucsc xml config (ucsc_tablebrowser.xml).
> But my purpose is to import multiple files.
>
> Have you any idea how I can set a way to import/upload multiple files by
> using the synchronous method?
>
> Thank in advance,
> Best regards
> Daniele
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/