Re: [galaxy-dev] Dynamic Tool UI based on script

[Steve Cassidy]

 I don’t like ‘hacks’!! I solved this by pulling the relevant bits of the 
module into my script so that it runs with no dependencies. It’s basically just 
making an HTTP GET request with an authentication header, so not too hard to 
achieve.  Should be safe to put in the toolshed.

Thanks, this is a big improvement in our UI (and I vote for not deprecating 
this feature, maybe even making it more robust).

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 7 Dec 2017, at 8:50 pm, Hans-Rudolf Hotz mailto:h...@fmi.ch>> 
wrote:


Sure

However, tools relying on "dynamic_options" and extra "code" scripts are 
usually 'hacks' for accessing external data/services in a local galaxy 
installation. And such tools are not intended for the toolshed anywayIMHO.


Hans-Rudolf


On 12/07/2017 09:37 AM, Steve Cassidy wrote:
I could do that but it would mean that no-one could install the tool
from the toolshed without also installing this package.

Maybe I just need to hack this together.

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 7 Dec 2017, at 6:59 pm, Hans-Rudolf Hotz mailto:h...@fmi.ch>> wrote:

Hi Steve

what happened, when you add this module to the environment the galaxy
server is running in? (e.g. by listing the path in the 'run.sh' file)

Regards, Hans-Rudolf


On 12/07/2017 03:05 AM, Steve Cassidy wrote:
Hi all,
 this would help in a tool I’m trying to write too but the problem I
have implementing this approach is that my script will need an external
python module - is there any way that I can list dependencies for the
script?  As it happens these are the same dependancies that the tool
itself will need once executed.

Thanks,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 24 Nov 2017, at 6:52 pm, Hans-Rudolf Hotz mailto:h...@fmi.ch>> wrote:

Hi Evan

Yes, this is possible with "dynamic_options"

e.g.:




   2017
   2016
   2015







...


...





The python function "ds_readFiles" is defined in a extra code file
"more_code.py", which is stored in the same directory as the tool
definition file. The function returns all files with a matching file
name (i.e.year), and could look like:


rDir = "/data/"

def ds_readFiles(year):

 l = os.listdir(rDir)
 p = re.compile('^%s_[0­9]{4}_M[0­9]{5}_[0­9]'%year)
 l.sort()
 path = rDir
 filesoptions = [(s) for s in l if os.path.exists(path + s)]
 files = [(s,s,False) for s in filesoptions if p.match(s)]
 return files



a similar case is also described here:

http://dev.list.galaxyproject.org/Dynamic-Tool-Parameter-Lists-td4175828.html


Hope this helps, Hans-Rudolf


On 11/23/2017 06:02 PM, Evan Clark wrote:
Is it possible to execute a script within a tool xml confit to retrieve
some data and append to the ui depending on the requested data. I.e
if I
wanted to get a list of files and provide them as select options before
hitting the execute button.


---
Sent from VMware Boxer


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Dynamic Tool UI based on script

[Steve Cassidy]

I could do that but it would mean that no-one could install the tool from the 
toolshed without also installing this package.

Maybe I just need to hack this together.

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 7 Dec 2017, at 6:59 pm, Hans-Rudolf Hotz mailto:h...@fmi.ch>> 
wrote:

Hi Steve

what happened, when you add this module to the environment the galaxy server is 
running in? (e.g. by listing the path in the 'run.sh' file)

Regards, Hans-Rudolf


On 12/07/2017 03:05 AM, Steve Cassidy wrote:
Hi all,
  this would help in a tool I’m trying to write too but the problem I
have implementing this approach is that my script will need an external
python module - is there any way that I can list dependencies for the
script?  As it happens these are the same dependancies that the tool
itself will need once executed.

Thanks,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 24 Nov 2017, at 6:52 pm, Hans-Rudolf Hotz mailto:h...@fmi.ch>> wrote:

Hi Evan

Yes, this is possible with "dynamic_options"

e.g.:




2017
2016
2015







...


...





The python function "ds_readFiles" is defined in a extra code file
"more_code.py", which is stored in the same directory as the tool
definition file. The function returns all files with a matching file
name (i.e.year), and could look like:


rDir = "/data/"

def ds_readFiles(year):

  l = os.listdir(rDir)
  p = re.compile('^%s_[0­9]{4}_M[0­9]{5}_[0­9]'%year)
  l.sort()
  path = rDir
  filesoptions = [(s) for s in l if os.path.exists(path + s)]
  files = [(s,s,False) for s in filesoptions if p.match(s)]
  return files



a similar case is also described here:

http://dev.list.galaxyproject.org/Dynamic-Tool-Parameter-Lists-td4175828.html


Hope this helps, Hans-Rudolf


On 11/23/2017 06:02 PM, Evan Clark wrote:
Is it possible to execute a script within a tool xml confit to retrieve
some data and append to the ui depending on the requested data. I.e if I
wanted to get a list of files and provide them as select options before
hitting the execute button.


---
Sent from VMware Boxer


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Dynamic Tool UI based on script

[Steve Cassidy]

Hi all,
  this would help in a tool I’m trying to write too but the problem I have 
implementing this approach is that my script will need an external python 
module - is there any way that I can list dependencies for the script?  As it 
happens these are the same dependancies that the tool itself will need once 
executed.

Thanks,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 24 Nov 2017, at 6:52 pm, Hans-Rudolf Hotz mailto:h...@fmi.ch>> 
wrote:

Hi Evan

Yes, this is possible with "dynamic_options"

e.g.:




 2017
 2016
 2015







...


...





The python function "ds_readFiles" is defined in a extra code file
"more_code.py", which is stored in the same directory as the tool definition 
file. The function returns all files with a matching file name (i.e.year), and 
could look like:


rDir = "/data/"

def ds_readFiles(year):

   l = os.listdir(rDir)
   p = re.compile('^%s_[0­9]{4}_M[0­9]{5}_[0­9]'%year)
   l.sort()
   path = rDir
   filesoptions = [(s) for s in l if os.path.exists(path + s)]
   files = [(s,s,False) for s in filesoptions if p.match(s)]
   return files



a similar case is also described here:

http://dev.list.galaxyproject.org/Dynamic-Tool-Parameter-Lists-td4175828.html


Hope this helps, Hans-Rudolf


On 11/23/2017 06:02 PM, Evan Clark wrote:
Is it possible to execute a script within a tool xml confit to retrieve
some data and append to the ui depending on the requested data. I.e if I
wanted to get a list of files and provide them as select options before
hitting the execute button.


---
Sent from VMware Boxer


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] planemo with --docker and --conda_dependency_resolution

[Steve Cassidy]

Hi,
 I’m trying to test a few tools, one of which uses a docker container, the 
others use dependencies that can be found in conda.

I can test each one by using either —docker or —conda_dependency_resolution 
with planemo, but when I try to give both of these, it seems to want to only 
use docker and fails because it can’t find a usable container.

I’ve hunted around the planemo sources but i can’t see what magic is creating 
this problem.  Am I just asking too much?

Any pointers appreciated.

Thanks,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tool needs a particular file extension -- now Datatype help

[Steve Cassidy]

Thanks all,
  it seems that my real problem is that the audio file (.wav) is not being 
identified as a valid datatype and ending up as a zero length text file. So, I 
need to start to explore the world of datatypes.

Following the docs 
(https://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes) I can 
modify datatypes_conf.xml in my Galaxy sources and add a new datatype for wav 
files:

http://web.science.mq.edu.au/~cassidy

On 21 Oct. 2016, at 12:58 pm, Peter Cock 
mailto:p.j.a.c...@googlemail.com>> wrote:

Using a soft link for this is a common pattern, and should be followed with &&
(ideally using XML CDATA to avoid escaping everything like & etc),
and quote the filenames just in case there are any spaces. e.g.

https://github.com/galaxyproject/tools-iuc/blob/master/tools/trinity/run_de_analysis.xml#L16


For reference, in tools-iuc there are over 400 soft link examples:

$ grep "ln -s" tools/*/*.xml | wc -l
446

Peter

On Fri, Oct 21, 2016 at 5:48 PM, Steve Cassidy  wrote:
Hi,
I’m wrapping a tool that needs it’s input to have a known file extension
(an audio file, eg. .wav).  Since Galaxy stores all data as .dat files the
tool is falling over since it doesn’t know what .dat is.

I thought I’d be able to get around this by hard linking the .dat file to
the same name with a .wav extension (dataset_1.dat.wav), this works when I
try it with the tool on the command line but within Galaxy it fails, here’s
my :

   ln $signal ${signal}.wav &
   /home/maus/maus OUTFORMAT=TextGrid LANGUAGE=$language
   BPF=$bpf INSKANTEXTGRID=$inskantextgrid
INSORTTEXTGRID=$insorttextgrid
   MODUS=$modus MAUSSHIFT=$mausshift MINPAUSLEN=$minpauslen
WEIGHT=$weight
   INSPROB=$insprob NOINITIALFINALSILENCE=$noinitialfinalsilence
OUTSYMBOL=$outsymbol
   OUT=$output SIGNAL=${signal}.wav

resulting in the job command line:

ln /tmp/tmp7AZvx7/files/000/dataset_2.dat
/tmp/tmp7AZvx7/files/000/dataset_2.dat.wav & /home/maus/maus
OUTFORMAT=TextGrid LANGUAGE=aus BPF=/tmp/tmp7AZvx7/files/000/dataset_1.dat
INSKANTEXTGRID=false INSORTTEXTGRID=false MODUS=standard MAUSSHIFT=10
MINPAUSLEN=5 WEIGHT=7.0 INSPROB=0.0 NOINITIALFINALSILENCE=no OUTSYMBOL=sampa
OUT=/tmp/tmp7AZvx7/files/000/dataset_3.dat
SIGNAL=/tmp/tmp7AZvx7/files/000/dataset_2.dat.wav

I’m getting an error message from the tool:

sox FAIL formats: can't open input file
`/tmp/tmp7AZvx7/files/000/dataset_2.dat.wav': WAVE: RIFF header not found

this suggests that the hard link didn’t get made.  I tried copying the file
instead but got the same result.

I could go in and patch the tool script to be more forgiving but it would be
good to find a solution that didn’t require that if possible.

Any pointers appreciated.

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Tool needs a particular file extension

[Steve Cassidy]

Hi,
 I’m wrapping a tool that needs it’s input to have a known file extension (an 
audio file, eg. .wav).  Since Galaxy stores all data as .dat files the tool is 
falling over since it doesn’t know what .dat is.

I thought I’d be able to get around this by hard linking the .dat file to the 
same name with a .wav extension (dataset_1.dat.wav), this works when I try it 
with the tool on the command line but within Galaxy it fails, here’s my 
:

ln $signal ${signal}.wav &
/home/maus/maus OUTFORMAT=TextGrid LANGUAGE=$language
BPF=$bpf INSKANTEXTGRID=$inskantextgrid INSORTTEXTGRID=$insorttextgrid
MODUS=$modus MAUSSHIFT=$mausshift MINPAUSLEN=$minpauslen WEIGHT=$weight
INSPROB=$insprob NOINITIALFINALSILENCE=$noinitialfinalsilence 
OUTSYMBOL=$outsymbol
OUT=$output SIGNAL=${signal}.wav

resulting in the job command line:

ln /tmp/tmp7AZvx7/files/000/dataset_2.dat 
/tmp/tmp7AZvx7/files/000/dataset_2.dat.wav & /home/maus/maus OUTFORMAT=TextGrid 
LANGUAGE=aus BPF=/tmp/tmp7AZvx7/files/000/dataset_1.dat INSKANTEXTGRID=false 
INSORTTEXTGRID=false MODUS=standard MAUSSHIFT=10 MINPAUSLEN=5 WEIGHT=7.0 
INSPROB=0.0 NOINITIALFINALSILENCE=no OUTSYMBOL=sampa 
OUT=/tmp/tmp7AZvx7/files/000/dataset_3.dat 
SIGNAL=/tmp/tmp7AZvx7/files/000/dataset_2.dat.wav

I’m getting an error message from the tool:

sox FAIL formats: can't open input file 
`/tmp/tmp7AZvx7/files/000/dataset_2.dat.wav': WAVE: RIFF header not found

this suggests that the hard link didn’t get made.  I tried copying the file 
instead but got the same result.

I could go in and patch the tool script to be more forgiving but it would be 
good to find a solution that didn’t require that if possible.

Any pointers appreciated.

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Storing data in the user session

[Steve Cassidy]

Thanks Bjoern, look forward to seeing this, happy to test it when it’s ready.

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 17 Oct. 2016, at 5:41 pm, Björn Grüning 
mailto:bjoern.gruen...@gmail.com>> wrote:

Hi Steve and Katherine,

we are working on this branch and it is working:
https://github.com/bgruening/galaxy/tree/extra_user_preferences

But are a little bit sidetracked by porting the user-preferences from
mako to JS before we can include this into Main Galaxy, but the main
idea should work for tools and all other features we have in mind.

Cheers,
Bjoern

Am 17.10.2016 um 23:31 schrieb Katherine Beaulieu:
Ah I see, we had a similar problem except our service used a password
and username so we ended up implementing a password parameter that
didn't persist in the database.

On Mon, Oct 17, 2016 at 5:29 PM, Steve Cassidy 
mailto:steve.cass...@mq.edu.au>
<mailto:steve.cass...@mq.edu.au>> wrote:

   Thanks Katherine, in my case I want to store the API key for an
   external service (not Galaxy) and get access to that from a tool so
   that the tool can act on behalf of the user.

   Steve

   —
   Department of Computing, Macquarie University
   http://web.science.mq.edu.au/~cassidy
   <http://web.science.mq.edu.au/%7Ecassidy>

   On 17 Oct. 2016, at 5:26 pm, Katherine Beaulieu
   mailto:katherine.beaulieu...@gmail.com>
   <mailto:katherine.beaulieu...@gmail.com>> wrote:

   Hi Steve,
   In terms of accessing the user's api key within a tool definition
   file, have a look at this thread I had started. John had suggested
   something but I haven't gotten it to work but maybe you will have
   more luck than me.
   
http://dev.list.galaxyproject.org/Workflow-ID-Galaxy-Instance-URL-API-Key-in-Galaxy-Tool-Config-td4669906.html#a4670104
   
<http://dev.list.galaxyproject.org/Workflow-ID-Galaxy-Instance-URL-API-Key-in-Galaxy-Tool-Config-td4669906.html#a4670104>
   Cheers,
   Katherine

   On Mon, Oct 17, 2016 at 5:17 PM, Steve Cassidy
   mailto:steve.cass...@mq.edu.au> 
<mailto:steve.cass...@mq.edu.au>> wrote:

   Hi Bjoern,
just wanting to revisit this question and perhaps get an
   update on what you were doing for extending the user profile.

   I just noticed in the galaxy sources the option to add ‘forms’
   to be filled out at registration time. The admin interface
   seems to allow me to create  a form but it doesn’t appear when
   I try to register.  The code for this seems to be 2-5 years
   old so perhaps it’s a forgotten corner?

   https://github.com/galaxyproject/galaxy/tree/dev/templates/admin/forms
   <https://github.com/galaxyproject/galaxy/tree/dev/templates/admin/forms>

   Anyway, if you have made any progress on this I’d be happy to
   hear about it.  I’d like to be able to store the user’s API
   key in the profile and access it from a tool definition file.

   Cheers,

   Steve
   —
   Department of Computing, Macquarie University
   http://web.science.mq.edu.au/~cassidy
   <http://web.science.mq.edu.au/%7Ecassidy>

   On 28 Jul. 2016, at 2:59 pm, Björn Grüning
   mailto:bjoern.gruen...@gmail.com>
   <mailto:bjoern.gruen...@gmail.com>> wrote:

   Hi Steve & Mohammed,

   I'm working on a user-profile extension to get credentials
   for a user,
   but this will take a few more weeks sorry.

   In the meantime can you look at the following idea:

   my_programm.py --credentials $USER_CREDENTIALS --arguments

   The variable $USER_CREDENTIALS is set during job execution
   with dynamic
   destination mappings
   
https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping
   
<https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping>

   What do you think, would this work for you?
   This assumes you as admin can pass along the user credentials
   if you
   know the username.

   If not please get in touch and I will tell you about my plans
   to extend
   the user-preferences maybe you can help me with this to speed
   things up.

   Ciao,
   Bjoern


   Hi Bjorn,
   sure, I don’t want to have the credentials stored in the
   tool but the tool
   needs to act on behalf of the user to retrieve data from the
   repository, so
   it needs access to credentials.  This could for example be a
   token generated
   via an OAuth2 exchange - basically something that is sent
   along with every
   http request to the remote resource to authenticate the user.

   My thought is that this could be stored in the user profile
   if this can be made
   available to the tool in the same way that __user_email__ is
   now.

   When you say ‘on the framework level’ do

Re: [galaxy-dev] Storing data in the user session

[Steve Cassidy]

Thanks Katherine, in my case I want to store the API key for an external 
service (not Galaxy) and get access to that from a tool so that the tool can 
act on behalf of the user.

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 17 Oct. 2016, at 5:26 pm, Katherine Beaulieu 
mailto:katherine.beaulieu...@gmail.com>> wrote:

Hi Steve,
In terms of accessing the user's api key within a tool definition file, have a 
look at this thread I had started. John had suggested something but I haven't 
gotten it to work but maybe you will have more luck than me.
http://dev.list.galaxyproject.org/Workflow-ID-Galaxy-Instance-URL-API-Key-in-Galaxy-Tool-Config-td4669906.html#a4670104
Cheers,
Katherine

On Mon, Oct 17, 2016 at 5:17 PM, Steve Cassidy 
mailto:steve.cass...@mq.edu.au>> wrote:
Hi Bjoern,
 just wanting to revisit this question and perhaps get an update on what you 
were doing for extending the user profile.

I just noticed in the galaxy sources the option to add ‘forms’ to be filled out 
at registration time. The admin interface seems to allow me to create  a form 
but it doesn’t appear when I try to register.  The code for this seems to be 
2-5 years old so perhaps it’s a forgotten corner?

https://github.com/galaxyproject/galaxy/tree/dev/templates/admin/forms

Anyway, if you have made any progress on this I’d be happy to hear about it.  
I’d like to be able to store the user’s API key in the profile and access it 
from a tool definition file.

Cheers,

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul. 2016, at 2:59 pm, Björn Grüning 
mailto:bjoern.gruen...@gmail.com>> wrote:

Hi Steve & Mohammed,

I'm working on a user-profile extension to get credentials for a user,
but this will take a few more weeks sorry.

In the meantime can you look at the following idea:

my_programm.py --credentials $USER_CREDENTIALS --arguments

The variable $USER_CREDENTIALS is set during job execution with dynamic
destination mappings
https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping

What do you think, would this work for you?
This assumes you as admin can pass along the user credentials if you
know the username.

If not please get in touch and I will tell you about my plans to extend
the user-preferences maybe you can help me with this to speed things up.

Ciao,
Bjoern


Hi Bjorn,
sure, I don’t want to have the credentials stored in the tool but the tool
needs to act on behalf of the user to retrieve data from the repository, so
it needs access to credentials.  This could for example be a token generated
via an OAuth2 exchange - basically something that is sent along with every
http request to the remote resource to authenticate the user.

My thought is that this could be stored in the user profile if this can be made
available to the tool in the same way that __user_email__ is now.

When you say ‘on the framework level’ do you mean by Galaxy?

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul 2016, at 5:23 PM, Björn Grüning 
mailto:bjoern.gruen...@gmail.com>> wrote:

Hi Steve,

can you explain the larger use-case behind it. Imho you should not
populate a tool with credentials, this should be handled on the
framework level.

Hope you are fine,
Bjoern

Am 28.07.2016 um 03:15 schrieb Steve Cassidy:
Hi again,
 I’m looking for the right way to store some user credentials in the
galaxy session so that tools can work on behalf of the user with our
repository.

Currently users have an API key and they need to upload it as a data
item to that is then passed to each tool that needs it as input.  This
doesn’t seem like the right solution since the API key becomes part of
the history and so would be shared if the history were shared.

What would be better would be a way of storing the API key in the user
session and then being able to pass that into a tool.

I note that there are a few user session variables available in the tool
xml file: __user__, __user_email__, __user_name__ and __user_id__.
There is also a user preferences page where they can fill out a few
details.  However, I can’t see a mechanism to extend this in any way and
have extra properties in the preferences pane that would then be
available to tools via the template file.

I can see that the standard practice for the data sources on
usegalaxy.org<http://usegalaxy.org/> 
<http://usegalaxy.org<http://usegalaxy.org/>> is to send the user off to a
repository website to find data that is then posted back to the galaxy
server.  This isn’t appropriate for us since the interaction with the
repository is not just for downloading a single dataset - tools will
query the repository, download various kinds of data and possibly upload
new data, so we need to store user credentials in some way.

Is there already a good way to achieve this or is thi

Re: [galaxy-dev] Storing data in the user session

[Steve Cassidy]

Hi Bjoern,
 just wanting to revisit this question and perhaps get an update on what you 
were doing for extending the user profile.

I just noticed in the galaxy sources the option to add ‘forms’ to be filled out 
at registration time. The admin interface seems to allow me to create  a form 
but it doesn’t appear when I try to register.  The code for this seems to be 
2-5 years old so perhaps it’s a forgotten corner?

https://github.com/galaxyproject/galaxy/tree/dev/templates/admin/forms

Anyway, if you have made any progress on this I’d be happy to hear about it.  
I’d like to be able to store the user’s API key in the profile and access it 
from a tool definition file.

Cheers,

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul. 2016, at 2:59 pm, Björn Grüning 
mailto:bjoern.gruen...@gmail.com>> wrote:

Hi Steve & Mohammed,

I'm working on a user-profile extension to get credentials for a user,
but this will take a few more weeks sorry.

In the meantime can you look at the following idea:

my_programm.py --credentials $USER_CREDENTIALS --arguments

The variable $USER_CREDENTIALS is set during job execution with dynamic
destination mappings
https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping

What do you think, would this work for you?
This assumes you as admin can pass along the user credentials if you
know the username.

If not please get in touch and I will tell you about my plans to extend
the user-preferences maybe you can help me with this to speed things up.

Ciao,
Bjoern


Hi Bjorn,
sure, I don’t want to have the credentials stored in the tool but the tool
needs to act on behalf of the user to retrieve data from the repository, so
it needs access to credentials.  This could for example be a token generated
via an OAuth2 exchange - basically something that is sent along with every
http request to the remote resource to authenticate the user.

My thought is that this could be stored in the user profile if this can be made
available to the tool in the same way that __user_email__ is now.

When you say ‘on the framework level’ do you mean by Galaxy?

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul 2016, at 5:23 PM, Björn Grüning  wrote:

Hi Steve,

can you explain the larger use-case behind it. Imho you should not
populate a tool with credentials, this should be handled on the
framework level.

Hope you are fine,
Bjoern

Am 28.07.2016 um 03:15 schrieb Steve Cassidy:
Hi again,
 I’m looking for the right way to store some user credentials in the
galaxy session so that tools can work on behalf of the user with our
repository.

Currently users have an API key and they need to upload it as a data
item to that is then passed to each tool that needs it as input.  This
doesn’t seem like the right solution since the API key becomes part of
the history and so would be shared if the history were shared.

What would be better would be a way of storing the API key in the user
session and then being able to pass that into a tool.

I note that there are a few user session variables available in the tool
xml file: __user__, __user_email__, __user_name__ and __user_id__.
There is also a user preferences page where they can fill out a few
details.  However, I can’t see a mechanism to extend this in any way and
have extra properties in the preferences pane that would then be
available to tools via the template file.

I can see that the standard practice for the data sources on
usegalaxy.org <http://usegalaxy.org> is to send the user off to a
repository website to find data that is then posted back to the galaxy
server.  This isn’t appropriate for us since the interaction with the
repository is not just for downloading a single dataset - tools will
query the repository, download various kinds of data and possibly upload
new data, so we need to store user credentials in some way.

Is there already a good way to achieve this or is this an enhancement to
Galaxy?

Thanks,

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy
<http://web.science.mq.edu.au/%7Ecassidy>



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Help wanted: setting up a production server

[Steve Cassidy]

Hi all,
  I’m working on setting up a Galaxy server for use with speech and language 
data.  While I can probably get there myself eventually, I’m thinking it would 
be good to get some (paid) help if it were available.

If you have some experience setting up a public facing Galaxy instance, and you 
have some time available in the next couple of weeks, please get in touch.

Any location is probably ok as long as we can work out how to reimburse you.  
We (and the servers) are based in Australia but I’m currently in New York.

Thanks,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Testtoolshed status

[Steve Cassidy]

I’m getting a 500 response from the testtoolshed when I try to upload a tool 
via planemo:

➜  alveo-galaxy-tools git:(master) ✗ planemo shed_update --shed_target 
testtoolshed apitools
cd '/Users/steve/projects/alveo-galaxy-tools/apitools' && git rev-parse HEAD
cd '/Users/steve/projects/alveo-galaxy-tools/apitools' && git diff --quiet
Could not update alveoimport
Unexpected HTTP status code: 500: 
  Internal Server Error
  
Internal Server Error
The server has either erred or is incapable of performing
the requested operation.




WSGI Server
  


Repository metadata updated successfully for repository alveoimport.
Failed to update a repository.

In addition, I just got a “Tool Shed could not be reached” message via the web 
interface, so i’m wondering if something is up.

Thanks,

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Conda: confused dependencies with multiple R versions and rpy

[Steve Cassidy]

Hi,
  trying to get my tools going, part 29.  I have a Docker image with the latest 
Galaxy release, patched to fix the Conda install bug I found earlier, my tools 
are installed ok and I’ve added a few from the toolshed for good measure.

One of my tools requires R 3.0.2 which is installed ok.   I also installed 
scatterplot from the toolshed which requires R 2.11; that is also installed ok 
by conda. However, I get this message when trying to run scatterplot:


Traceback (most recent call last):
  File 
"/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/scatterplot/d243056b22ed/scatterplot/scatterplot.py",
 line 5, in 
from rpy import *
  File 
"/tool_deps/rpy/1.0.3/devteam/package_rpy_1_0_3/82170c94ca7c/lib/python/rpy.py",
 line 134, in 
""" % RVERSION)
RuntimeError: No module named _rpy3002

  RPy module can not be imported. Please check if your rpy
  installation supports R 3.0.2. If you have multiple R versions
  installed, you may need to set RHOME before importing rpy. For
  example:

  >>> from rpy_options import set_options
  >>> set_options(RHOME='c:/progra~1/r/rw2011/')
  >>> from rpy import *


So it seems that rpy is installed and is aware of one version of R but not the 
other, and the other is being found (perhaps because it’s the default install 
in the Docker image?)

FYI my docker image recipe is here:

https://github.com/Alveo/docker-galaxy-alveo

and the image itself is here:

https://hub.docker.com/r/stevecassidy/alveo-galaxy/

Any pointers to the way forward appreciated.

Thanks,
Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Steve Cassidy]

Looks like I’m hitting this issue:

https://github.com/galaxyproject/galaxy/issues/2797

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 29 Aug 2016, at 11:09 AM, Steve Cassidy  wrote:
> 
> Thanks Marius,
>   I tried a build based on the image you mentioned and get the following 
> error:
> 
> /export/galaxy-central/database/job_working_directory/000/1/tool_script.sh: 
> line 9: /tool_deps/_conda/bin/activate: No such file or directory
> /export/galaxy-central/database/job_working_directory/000/1/galaxy_1.sh: line 
> 66: /tool_deps/_conda/bin/activate: No such file or directory
> 
> I tried again with the 16.07 release image but got the same result.  
> 
> Checking the /tool_deps/_conda/bin/ directory, indeed there is no activate 
> script, in fact it contains:
> 
> root@db343fbf7372:/galaxy-central# ls /tool_deps/_conda/bin/
> 2to3  conda  c_rehash  easy_install-3.5  idle3.5  pippydoc3   
>  python   python3.5
> python3.5m python3-config  pyvenv-3.5  tclsh8.5  wheelxz
> 2to3-3.5  conda-env  easy_install  idle3 openssl  pydoc  pydoc3.5 
>  
> python3  python3.5-config  python3.5m-config  pyvenv  sqlite3 
> unxz  wish8.5
> 
> 
> Any pointers appreciated.
> 
> Thanks,
> 
> Steve
> 
> —
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy
> 
>> On 26 Aug 2016, at 6:16 PM, Marius van den Beek  
>> wrote:
>> 
>> Hi Steve,
>> 
>> Looks like you're running an older version of galaxy in the docker 
>> container, since newer galaxy will build the metadata environment in a 
>> separate environment called 'conda-metadata-env',
>> 
>> and we have also changed the way that the handlers receive their Python 
>> environment (that's why sqlalachemy couldn't be found).
>> 
>> Can you try updating the container or building from the dev branch? I think 
>> the necessary changes for conda should be in the dev branch, which you can 
>> pull with docker pull quay.io/bgruening/galaxy:dev.
>> 
>> Best,
>> 
>> Marius
>> 
>> 
>> On Aug 26, 2016 6:49 AM, "Steve Cassidy"  wrote:
>> Ok, probing further trying to understand this. It looks like the job runner 
>> is trying to call set_meta() from galaxy_ext/metadata/set_metadata.py.  In 
>> there there’s a line that tries to set up the PATH:
>> 
>> # insert *this* galaxy before all others on sys.path
>> sys.path.insert( 1, os.path.abspath( os.path.join( os.path.dirname( __file__ 
>> ), os.pardir, os.pardir ) ) )
>> 
>> In my case the result is:
>> 
>> ['/export/galaxy-central/database/job_working_directory/000/3’,
>> '/galaxy-central/lib’,
>> '/galaxy-central/lib’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python27.zip’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/plat-linux2’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-tk’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-old’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-dynload’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/site-packages’,
>> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/site-packages/setuptools-25.1.6-py2.7.egg’
>> ]
>> 
>> Looking for sqlalchemy, I find it installed at:
>> 
>> /galaxy-central/.venv/lib/python2.7/site-packages/sqlalchemy
>> 
>> so clearly that line of code is not doing what it should. I’m guessing that 
>> the Conda resolver is running the script within a conda env and the BETA 
>> magic hasn’t quite made it to the right place yet…
>> 
>> BTW I was trying to understand the flags you mentioned but I can’t find 
>> GALAXY_CONFIG_ENABLE_BETA_TOOL_COMMAND_ISOLATION anywhere other than in 
>> sample Dockerfiles - is this some kind of magic incantation that creates a 
>> rift in space-time…enquiring minds want to know!!!
>> 
>> Steve
>> —
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy
>> 
>>> On 25 Aug 2016, at 9:54 PM, Steve Cassidy  wrote:
>>> 
>>> Nah, definitely baby steps…
>>> 
>>> so, in the repo you point to th

Re: [galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Steve Cassidy]

Thanks Marius,
   I tried a build based on the image you mentioned and get the following error:

/export/galaxy-central/database/job_working_directory/000/1/tool_script.sh: 
line 9: /tool_deps/_conda/bin/activate: No such file or directory
/export/galaxy-central/database/job_working_directory/000/1/galaxy_1.sh: line 
66: /tool_deps/_conda/bin/activate: No such file or directory

I tried again with the 16.07 release image but got the same result.  

Checking the /tool_deps/_conda/bin/ directory, indeed there is no activate 
script, in fact it contains:

root@db343fbf7372:/galaxy-central# ls /tool_deps/_conda/bin/
2to3  conda  c_rehash  easy_install-3.5  idle3.5  pippydoc3
python   python3.5
python3.5m python3-config  pyvenv-3.5  tclsh8.5  wheelxz
2to3-3.5  conda-env  easy_install  idle3 openssl  pydoc  pydoc3.5  
python3  python3.5-config  python3.5m-config  pyvenv  sqlite3 unxz  
wish8.5


Any pointers appreciated.

Thanks,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 26 Aug 2016, at 6:16 PM, Marius van den Beek  
> wrote:
> 
> Hi Steve,
> 
> Looks like you're running an older version of galaxy in the docker container, 
> since newer galaxy will build the metadata environment in a separate 
> environment called 'conda-metadata-env',
> 
> and we have also changed the way that the handlers receive their Python 
> environment (that's why sqlalachemy couldn't be found).
> 
> Can you try updating the container or building from the dev branch? I think 
> the necessary changes for conda should be in the dev branch, which you can 
> pull with docker pull quay.io/bgruening/galaxy:dev.
> 
> Best,
> 
> Marius
> 
> 
> On Aug 26, 2016 6:49 AM, "Steve Cassidy"  wrote:
> Ok, probing further trying to understand this. It looks like the job runner 
> is trying to call set_meta() from galaxy_ext/metadata/set_metadata.py.  In 
> there there’s a line that tries to set up the PATH:
> 
> # insert *this* galaxy before all others on sys.path
> sys.path.insert( 1, os.path.abspath( os.path.join( os.path.dirname( __file__ 
> ), os.pardir, os.pardir ) ) )
> 
> In my case the result is:
> 
> ['/export/galaxy-central/database/job_working_directory/000/3’,
> '/galaxy-central/lib’,
> '/galaxy-central/lib’,
>  
> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python27.zip’,
>  
> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7’,
>  
> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/plat-linux2’,
>  
> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-tk’,
> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-old’,
>  
> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-dynload’,
> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/site-packages’,
> '/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/site-packages/setuptools-25.1.6-py2.7.egg’
> ]
> 
> Looking for sqlalchemy, I find it installed at:
> 
> /galaxy-central/.venv/lib/python2.7/site-packages/sqlalchemy
> 
> so clearly that line of code is not doing what it should. I’m guessing that 
> the Conda resolver is running the script within a conda env and the BETA 
> magic hasn’t quite made it to the right place yet…
> 
> BTW I was trying to understand the flags you mentioned but I can’t find 
> GALAXY_CONFIG_ENABLE_BETA_TOOL_COMMAND_ISOLATION anywhere other than in 
> sample Dockerfiles - is this some kind of magic incantation that creates a 
> rift in space-time…enquiring minds want to know!!!
> 
> Steve
> —
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy
> 
> > On 25 Aug 2016, at 9:54 PM, Steve Cassidy  wrote:
> >
> > Nah, definitely baby steps…
> >
> > so, in the repo you point to there seems to be an error in the Dockerfile, 
> > the ENV line should use the var=value syntax to have more than one setting 
> > on one line (maybe that’s changed recently in docker).
> >
> > with this I built a new docker image, when I run my first tool it takes an 
> > age while it’s installing the deps, then it crashes with:
> >
> > Traceback (most recent call last):
> >  File 
> > "/export/galaxy-central/database/job_working_directory/000/1/set_metadata_QUJQLD.py",
> >  line 1, in 
> >from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
&g

Re: [galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Steve Cassidy]

Ok, probing further trying to understand this. It looks like the job runner is 
trying to call set_meta() from galaxy_ext/metadata/set_metadata.py.  In there 
there’s a line that tries to set up the PATH:

# insert *this* galaxy before all others on sys.path
sys.path.insert( 1, os.path.abspath( os.path.join( os.path.dirname( __file__ ), 
os.pardir, os.pardir ) ) )

In my case the result is:

['/export/galaxy-central/database/job_working_directory/000/3’, 
'/galaxy-central/lib’, 
'/galaxy-central/lib’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python27.zip’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/plat-linux2’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-tk’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-old’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-dynload’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/site-packages’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/site-packages/setuptools-25.1.6-py2.7.egg’
]

Looking for sqlalchemy, I find it installed at:

/galaxy-central/.venv/lib/python2.7/site-packages/sqlalchemy

so clearly that line of code is not doing what it should. I’m guessing that the 
Conda resolver is running the script within a conda env and the BETA magic 
hasn’t quite made it to the right place yet… 

BTW I was trying to understand the flags you mentioned but I can’t find 
GALAXY_CONFIG_ENABLE_BETA_TOOL_COMMAND_ISOLATION anywhere other than in sample 
Dockerfiles - is this some kind of magic incantation that creates a rift in 
space-time…enquiring minds want to know!!!

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 25 Aug 2016, at 9:54 PM, Steve Cassidy  wrote:
> 
> Nah, definitely baby steps…
> 
> so, in the repo you point to there seems to be an error in the Dockerfile, 
> the ENV line should use the var=value syntax to have more than one setting on 
> one line (maybe that’s changed recently in docker).
> 
> with this I built a new docker image, when I run my first tool it takes an 
> age while it’s installing the deps, then it crashes with:
> 
> Traceback (most recent call last):
>  File 
> "/export/galaxy-central/database/job_working_directory/000/1/set_metadata_QUJQLD.py",
>  line 1, in 
>from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
>  File "/galaxy-central/lib/galaxy_ext/metadata/set_metadata.py", line 23, in 
> 
>from sqlalchemy.orm import clear_mappers
> ImportError: No module named sqlalchemy.orm
> 
> and the output:
> 
> discarding /galaxy-central/tool_deps/_conda/bin from PATH
> prepending 
> /export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to 
> PATH
> discarding /galaxy-central/tool_deps/_conda/bin from PATH
> prepending 
> /export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to 
> PATH
> 
> I’m guessing this is some kind of conflict between python versions in and out 
> of conda environments? Surely sqlalchemy would be installed for Galaxy to 
> work?  
> 
> I’ll try to dig around this in the morning but if it rings a bell…
> 
> Steve
> —
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy
> 
>> On 25 Aug 2016, at 8:43 PM, Björn Grüning  wrote:
>> 
>> Hi Steve,
>> 
>> you call this baby-steps? I think this is huge! :)
>> 
>> All what you are missing is to enable conda in Galaxy.
>> Look at Gregs new flavour which is entirely Conda/Galaxy based.
>> 
>> You need to enable these env vars to make Galaxy conda enabled:
>> 
>> https://github.com/gregvonkuster/docker-galaxy-csg/blob/master/Dockerfile#L9
>> 
>> Hope this helps,
>> Bjoern
>> 
>> Am 25.08.2016 um 12:32 schrieb Steve Cassidy:
>>> Hi all,
>>> I’m making baby steps towards having a repeatable installation for my
>>> tools.  But I’m now stuck, so help would be appreciated.
>>> 
>>> I have a tool that works and is in the test toolshed (alveoimport in
>>> Data Sources). 
>>> 
>>> It depends on my python package which is now part of bioconda (pyalveo,
>>> version 0.6). 
>>> 
>>> I can run my tool via planemo, which works I think because I have
>>> pyalveo installed in a local venv.  
>>> 
>>> If I try to run the docker ima

Re: [galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Steve Cassidy]

Nah, definitely baby steps…

so, in the repo you point to there seems to be an error in the Dockerfile, the 
ENV line should use the var=value syntax to have more than one setting on one 
line (maybe that’s changed recently in docker).

with this I built a new docker image, when I run my first tool it takes an age 
while it’s installing the deps, then it crashes with:

Traceback (most recent call last):
  File 
"/export/galaxy-central/database/job_working_directory/000/1/set_metadata_QUJQLD.py",
 line 1, in 
from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
  File "/galaxy-central/lib/galaxy_ext/metadata/set_metadata.py", line 23, in 

from sqlalchemy.orm import clear_mappers
ImportError: No module named sqlalchemy.orm

and the output:

discarding /galaxy-central/tool_deps/_conda/bin from PATH
prepending 
/export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to 
PATH
discarding /galaxy-central/tool_deps/_conda/bin from PATH
prepending 
/export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to 
PATH

I’m guessing this is some kind of conflict between python versions in and out 
of conda environments? Surely sqlalchemy would be installed for Galaxy to work? 
 

I’ll try to dig around this in the morning but if it rings a bell…

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 25 Aug 2016, at 8:43 PM, Björn Grüning  wrote:
> 
> Hi Steve,
> 
> you call this baby-steps? I think this is huge! :)
> 
> All what you are missing is to enable conda in Galaxy.
> Look at Gregs new flavour which is entirely Conda/Galaxy based.
> 
> You need to enable these env vars to make Galaxy conda enabled:
> 
> https://github.com/gregvonkuster/docker-galaxy-csg/blob/master/Dockerfile#L9
> 
> Hope this helps,
> Bjoern
> 
> Am 25.08.2016 um 12:32 schrieb Steve Cassidy:
>> Hi all,
>>  I’m making baby steps towards having a repeatable installation for my
>> tools.  But I’m now stuck, so help would be appreciated.
>> 
>> I have a tool that works and is in the test toolshed (alveoimport in
>> Data Sources). 
>> 
>> It depends on my python package which is now part of bioconda (pyalveo,
>> version 0.6). 
>> 
>> I can run my tool via planemo, which works I think because I have
>> pyalveo installed in a local venv.  
>> 
>> If I try to run the docker image (derived from  bgruening/galaxy-stable
>> but with the testtoolshed added) I am able to install my tool, but it
>> doesn’t pick up the dependency, so it doesn’t work. 
>> 
>> I tried running with planemo turning on conda dependency resolution
>> (following https://pypi.python.org/pypi/planemo/): 
>> 
>> planemo serve --galaxy_branch release_16.01 --conda_dependency_resolution .
>> 
>> It seems to have a go, but fails:
>> 
>> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Building dependency
>> shell command for dependency 'pyalveo'
>> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Find dependency pyalveo
>> version 0.6
>> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned
>> 
>> (isnull? True)
>> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned
>> 
>> (isnull? True)
>> galaxy.tools.deps WARNING 2016-08-25 17:32:22,451 Failed to resolve
>> dependency on 'pyalveo', ignoring
>> 
>> So, what’s the easiest route to a Galaxy deployment with my tool
>> installed. The Docker route would be best I think, but what do I have to
>> add to bgruening/galaxy-stable to have conda find my dependencies.
>> 
>> Thanks in advance,
>> 
>> Steve
>> 
>> —
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy
>> <http://web.science.mq.edu.au/%7Ecassidy>
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>> 
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
>> 

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Steve Cassidy]

Hi all,
  I’m making baby steps towards having a repeatable installation for my tools.  
But I’m now stuck, so help would be appreciated.

I have a tool that works and is in the test toolshed (alveoimport in Data 
Sources).

It depends on my python package which is now part of bioconda (pyalveo, version 
0.6).

I can run my tool via planemo, which works I think because I have pyalveo 
installed in a local venv.

If I try to run the docker image (derived from  bgruening/galaxy-stable but 
with the testtoolshed added) I am able to install my tool, but it doesn’t pick 
up the dependency, so it doesn’t work.

I tried running with planemo turning on conda dependency resolution (following 
https://pypi.python.org/pypi/planemo/):

planemo serve --galaxy_branch release_16.01 --conda_dependency_resolution .

It seems to have a go, but fails:

galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Building dependency shell 
command for dependency 'pyalveo'
galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Find dependency pyalveo version 
0.6
galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned 
 (isnull? 
True)
galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned 
 (isnull? 
True)
galaxy.tools.deps WARNING 2016-08-25 17:32:22,451 Failed to resolve dependency 
on 'pyalveo', ignoring

So, what’s the easiest route to a Galaxy deployment with my tool installed. The 
Docker route would be best I think, but what do I have to add to 
bgruening/galaxy-stable to have conda find my dependencies.

Thanks in advance,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Storing data in the user session

[Steve Cassidy]

Hi Mohamed,

The problem with this solution is that the password again becomes part of the 
history since it is one of the inputs to the tool.  If I want to publish this 
workflow then the password can’t be visible.  For this reason password storage 
needs to be outside of workflows and handled somewhere else in Galaxy.

Steve



—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul 2016, at 6:04 PM, Mohamed Kassam 
mailto:k.mam...@gmail.com>> wrote:

Hi Steve,
there is another solution is that the user is giving his credential via the 
tool. The only thing is missing the tag password on the xml then you can 
encrypt the password.
I tried to add the new paramter on the build.py but I have some errors, don't 
know if someone arrived to fixed it.

Mohamed

2016-07-28 10:00 GMT+02:00 Steve Cassidy 
mailto:steve.cass...@mq.edu.au>>:
Hi Mohamed,
   In my case, the credentials on Galaxy are not generally the same as on the 
remote system so we need some way to store a second set of credentials in the 
galaxy session.

So, for example, user Fred has an account on Galaxy 
(f...@here.com<mailto:f...@here.com>) and wants to get data from Alveo as input 
to a workflow. For the Alveo tools to get data, they need Fred’s access token 
from Alveo which Fred can retrieve by logging in to the Alveo website.  Fred 
enters the access key into his preferences in Galaxy.  This key is then passed 
to one or more tools that access Alveo on his behalf:

 - the query tool finds words matching a given pattern from a spoken corpus 
(needs the token)
 - the get-data tool retrieves the audio files for each word (needs the token)
 - the automatic alignment tool finds the locations of the vowel sounds in each 
word
 - the annotation storer tool sends these locations back to Alveo as a new 
annotation set (needs the token)

and so on, at various times in the workflow, the tools need to talk to the 
repository to query, get data or add new data.

At the moment, I need the token to be a data item in the history, but it would 
be better if it were part of the user session in Galaxy.

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul 2016, at 5:46 PM, Mohamed Kassam 
mailto:k.mam...@gmail.com>> wrote:

Hello Cassidy,
I have the same problem like you because I need the user credential to access 
to an external tools.
I tried the user session variables available in the tool
 xml file: __user__, __user_email__, __user_name__ and __user_id__. I t works 
perfectly I don't know if you need the password, because this is the only 
information is missing.

Mohamed

2016-07-28 9:42 GMT+02:00 Steve Cassidy 
mailto:steve.cass...@mq.edu.au>>:
Hi Bjorn,
 sure, I don’t want to have the credentials stored in the tool but the tool
needs to act on behalf of the user to retrieve data from the repository, so
it needs access to credentials.  This could for example be a token generated
via an OAuth2 exchange - basically something that is sent along with every
http request to the remote resource to authenticate the user.

My thought is that this could be stored in the user profile if this can be made
available to the tool in the same way that __user_email__ is now.

When you say ‘on the framework level’ do you mean by Galaxy?

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 28 Jul 2016, at 5:23 PM, Björn Grüning 
> mailto:bjoern.gruen...@gmail.com>> wrote:
>
> Hi Steve,
>
> can you explain the larger use-case behind it. Imho you should not
> populate a tool with credentials, this should be handled on the
> framework level.
>
> Hope you are fine,
> Bjoern
>
> Am 28.07.2016 um 03:15 schrieb Steve Cassidy:
>> Hi again,
>>   I’m looking for the right way to store some user credentials in the
>> galaxy session so that tools can work on behalf of the user with our
>> repository.
>>
>> Currently users have an API key and they need to upload it as a data
>> item to that is then passed to each tool that needs it as input.  This
>> doesn’t seem like the right solution since the API key becomes part of
>> the history and so would be shared if the history were shared.
>>
>> What would be better would be a way of storing the API key in the user
>> session and then being able to pass that into a tool.
>>
>> I note that there are a few user session variables available in the tool
>> xml file: __user__, __user_email__, __user_name__ and __user_id__.
>> There is also a user preferences page where they can fill out a few
>> details.  However, I can’t see a mechanism to extend this in any way and
>> have extra properties in the preferences pane that would then be
>> available to tools via the template file.
>>
>> I can see that t

Re: [galaxy-dev] Storing data in the user session

[Steve Cassidy]

Hi Mohamed,
   In my case, the credentials on Galaxy are not generally the same as on the 
remote system so we need some way to store a second set of credentials in the 
galaxy session.

So, for example, user Fred has an account on Galaxy 
(f...@here.com<mailto:f...@here.com>) and wants to get data from Alveo as input 
to a workflow. For the Alveo tools to get data, they need Fred’s access token 
from Alveo which Fred can retrieve by logging in to the Alveo website.  Fred 
enters the access key into his preferences in Galaxy.  This key is then passed 
to one or more tools that access Alveo on his behalf:

 - the query tool finds words matching a given pattern from a spoken corpus 
(needs the token)
 - the get-data tool retrieves the audio files for each word (needs the token)
 - the automatic alignment tool finds the locations of the vowel sounds in each 
word
 - the annotation storer tool sends these locations back to Alveo as a new 
annotation set (needs the token)

and so on, at various times in the workflow, the tools need to talk to the 
repository to query, get data or add new data.

At the moment, I need the token to be a data item in the history, but it would 
be better if it were part of the user session in Galaxy.

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 28 Jul 2016, at 5:46 PM, Mohamed Kassam 
mailto:k.mam...@gmail.com>> wrote:

Hello Cassidy,
I have the same problem like you because I need the user credential to access 
to an external tools.
I tried the user session variables available in the tool
 xml file: __user__, __user_email__, __user_name__ and __user_id__. I t works 
perfectly I don't know if you need the password, because this is the only 
information is missing.

Mohamed

2016-07-28 9:42 GMT+02:00 Steve Cassidy 
mailto:steve.cass...@mq.edu.au>>:
Hi Bjorn,
 sure, I don’t want to have the credentials stored in the tool but the tool
needs to act on behalf of the user to retrieve data from the repository, so
it needs access to credentials.  This could for example be a token generated
via an OAuth2 exchange - basically something that is sent along with every
http request to the remote resource to authenticate the user.

My thought is that this could be stored in the user profile if this can be made
available to the tool in the same way that __user_email__ is now.

When you say ‘on the framework level’ do you mean by Galaxy?

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 28 Jul 2016, at 5:23 PM, Björn Grüning 
> mailto:bjoern.gruen...@gmail.com>> wrote:
>
> Hi Steve,
>
> can you explain the larger use-case behind it. Imho you should not
> populate a tool with credentials, this should be handled on the
> framework level.
>
> Hope you are fine,
> Bjoern
>
> Am 28.07.2016 um 03:15 schrieb Steve Cassidy:
>> Hi again,
>>   I’m looking for the right way to store some user credentials in the
>> galaxy session so that tools can work on behalf of the user with our
>> repository.
>>
>> Currently users have an API key and they need to upload it as a data
>> item to that is then passed to each tool that needs it as input.  This
>> doesn’t seem like the right solution since the API key becomes part of
>> the history and so would be shared if the history were shared.
>>
>> What would be better would be a way of storing the API key in the user
>> session and then being able to pass that into a tool.
>>
>> I note that there are a few user session variables available in the tool
>> xml file: __user__, __user_email__, __user_name__ and __user_id__.
>> There is also a user preferences page where they can fill out a few
>> details.  However, I can’t see a mechanism to extend this in any way and
>> have extra properties in the preferences pane that would then be
>> available to tools via the template file.
>>
>> I can see that the standard practice for the data sources on
>> usegalaxy.org<http://usegalaxy.org/> 
>> <http://usegalaxy.org<http://usegalaxy.org/>> is to send the user off to a
>> repository website to find data that is then posted back to the galaxy
>> server.  This isn’t appropriate for us since the interaction with the
>> repository is not just for downloading a single dataset - tools will
>> query the repository, download various kinds of data and possibly upload
>> new data, so we need to store user credentials in some way.
>>
>> Is there already a good way to achieve this or is this an enhancement to
>> Galaxy?
>>
>> Thanks,
>>
>> Steve
>>
>>
>> —
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy
>> <http://web.scie

Re: [galaxy-dev] Storing data in the user session

[Steve Cassidy]

Hi Bjorn,
 sure, I don’t want to have the credentials stored in the tool but the tool 
needs to act on behalf of the user to retrieve data from the repository, so
it needs access to credentials.  This could for example be a token generated
via an OAuth2 exchange - basically something that is sent along with every 
http request to the remote resource to authenticate the user.  

My thought is that this could be stored in the user profile if this can be made
available to the tool in the same way that __user_email__ is now. 

When you say ‘on the framework level’ do you mean by Galaxy? 

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 28 Jul 2016, at 5:23 PM, Björn Grüning  wrote:
> 
> Hi Steve,
> 
> can you explain the larger use-case behind it. Imho you should not
> populate a tool with credentials, this should be handled on the
> framework level.
> 
> Hope you are fine,
> Bjoern
> 
> Am 28.07.2016 um 03:15 schrieb Steve Cassidy:
>> Hi again,
>>   I’m looking for the right way to store some user credentials in the
>> galaxy session so that tools can work on behalf of the user with our
>> repository. 
>> 
>> Currently users have an API key and they need to upload it as a data
>> item to that is then passed to each tool that needs it as input.  This
>> doesn’t seem like the right solution since the API key becomes part of
>> the history and so would be shared if the history were shared.  
>> 
>> What would be better would be a way of storing the API key in the user
>> session and then being able to pass that into a tool.
>> 
>> I note that there are a few user session variables available in the tool
>> xml file: __user__, __user_email__, __user_name__ and __user_id__.
>> There is also a user preferences page where they can fill out a few
>> details.  However, I can’t see a mechanism to extend this in any way and
>> have extra properties in the preferences pane that would then be
>> available to tools via the template file.  
>> 
>> I can see that the standard practice for the data sources on
>> usegalaxy.org <http://usegalaxy.org> is to send the user off to a
>> repository website to find data that is then posted back to the galaxy
>> server.  This isn’t appropriate for us since the interaction with the
>> repository is not just for downloading a single dataset - tools will
>> query the repository, download various kinds of data and possibly upload
>> new data, so we need to store user credentials in some way. 
>> 
>> Is there already a good way to achieve this or is this an enhancement to
>> Galaxy? 
>> 
>> Thanks,
>> 
>> Steve
>> 
>> 
>> —
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy
>> <http://web.science.mq.edu.au/%7Ecassidy>
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>> 
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Storing data in the user session

[Steve Cassidy]

Hi again,
   I’m looking for the right way to store some user credentials in the galaxy 
session so that tools can work on behalf of the user with our repository.

Currently users have an API key and they need to upload it as a data item to 
that is then passed to each tool that needs it as input.  This doesn’t seem 
like the right solution since the API key becomes part of the history and so 
would be shared if the history were shared.

What would be better would be a way of storing the API key in the user session 
and then being able to pass that into a tool.

I note that there are a few user session variables available in the tool xml 
file: __user__, __user_email__, __user_name__ and __user_id__.  There is also a 
user preferences page where they can fill out a few details.  However, I can’t 
see a mechanism to extend this in any way and have extra properties in the 
preferences pane that would then be available to tools via the template file.

I can see that the standard practice for the data sources on 
usegalaxy.org is to send the user off to a repository 
website to find data that is then posted back to the galaxy server.  This isn’t 
appropriate for us since the interaction with the repository is not just for 
downloading a single dataset - tools will query the repository, download 
various kinds of data and possibly upload new data, so we need to store user 
credentials in some way.

Is there already a good way to achieve this or is this an enhancement to Galaxy?

Thanks,

Steve


—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Consuming dataset collections

[Steve Cassidy]

Hi John, 
 thanks for the response. So based on your updated documentation I’ve modified 
my script to take the identifiers as a second argument and with a bit of 
juggling I now have the command line:

query_textgrids.py --textgrid "${",".join(map(str, $textgrid))}" 
--identifier "${",".join(map(str, [t.element_identifier for t in $textgrid]))}" 
--tier $tier --regex '$regex' --output_path $output

Note the juggling needed with list comprehension to get the list of identifiers 
from the textgrid argument.  This works ok and I can now get a result from my 
tool that includes the identifier:

start   end durationlabel   identifier
0.590.830.246:  1_1308_1_2_092-ch6-speaker16.TextGrid
1.561.770.21I@  1_1308_1_2_094-ch6-speaker16.TextGrid
1.641.870.233:  1_1308_1_2_173-ch6-speaker16.TextGrid

in fact I’ll probably take the .TextGrid off the identifier so that it just 
names the recording I’m working with.  

What I’d like to do now is to write another tool that takes this as input along 
with another dataset collection who’s elements also have similar (or the same) 
identifiers but a different type (they will be acoustic features derived from 
an audio file.  I think I can see how to do this, the input to this tool will 
be similar to query_textgrids above and I’ll work through the identifiers and 
the table together.  

I saw your note on the issue re. galaxy.json and took a look in the sources for 
it, so this is a secret way of communicating dataset metadata back to the 
system? Sounds like it might be useful.  I may be able to get someone to work 
on this so if you have time to elaborate your ideas then please go ahead.

It seems that if I was to write my python script in the .xml file (as cheetah) 
I’d get access to a bunch of things that are opaque to a separate script. Would 
it be a useful goal to have a richer galaxy-tool interface that could make all 
information available to the tool wrapper visible to my Python script? One way 
to do that would just be to bundle everything up in JSON and send it to the 
script.  

Again, thanks for the help.

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 27 Jul 2016, at 12:29 AM, John Chilton  wrote:
> 
> Thanks for the questions - I have tried to revise the planemo docs to
> be more explicit about what collection identifiers are and where they
> come from 
> (https://github.com/galaxyproject/planemo/commit/a811e652f23d31682f862f858dc792c1ef5a99ce).
> http://planemo.readthedocs.io/en/latest/writing_advanced.html#collections
> 
> I think this might be a case where I'm too close the problem - I
> implemented collections, the tooling around them, and planemo docs so
> there is probably a lot that I just assume is implicit when it is
> completely non-obvious.
> 
> The collection identifier in your case is going to be something like:
> 1_1308_1_2_092-ch6-speaker16. The designation in the previous step -
> the outputing a collection with discovered datasets - if it is
> producing a collection should actually be called "identifier". The
> terms "desigination" and "identifier" are inter-changable from a
> Galaxy perspective - but I prefer using the term "identifier" for
> collections and the older "desigination" when discovered un-collected
> individual datasets.
> 
> There was a little warning explaining the odd whitespace replacement
> stuff that got shifted around at some point in the planemo docs - I
> think I have corrected that now. The explanation for fixing up the
> identifier was this:
> 
> "Here we are rewriting the element identifiers to assure everything is
> safe to put on the command-line. In the future, collections will not
> be able to contain keys that are potentially harmful and this won't be
> necessary."
> 
> So yes this is the name you are after.
> 
> As for the quesstion, "Is a manifest-based approach a silly idea?" -
> not at all, not in the least. I'd prefer to have both options
> available - this current option is nice because it doesn't require a
> "wrapper" script - you can build command lines and such from the
> cheetah template - but definitely people already working with and
> thinking about collections from inside some sort of script should have
> the option to consume and produce manifests of files.
> 
> I've created an issue for this idea here -
> https://github.com/galaxyproject/galaxy/issues/2658. I'm not sure if
> I'll have time to get to it anytime soon - but if you or someone else
> is eager to tackle the problem I could scope out an implementation
> plan for this.
> 
> Thanks for the e-mail and I hope this 

Re: [galaxy-dev] Papers that cite Galaxy workflows

[Steve Cassidy]

Thanks Dave, that’s just what I’m looking for, I found a few good examples of 
workflow citations.

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

On 27 Jul 2016, at 6:29 AM, Dave Clements 
mailto:cleme...@galaxyproject.org>> wrote:

Hi Steve,

I don't have a precise answer to your question, but the Galaxy CiteULike 
archive might be helpful here.  Any paper that uses, extends, or references 
Galaxy gets put in that library.  Papers are tagged with one or more of 17 
tags<https://wiki.galaxyproject.org/CiteULike>.  There aren't specific tags for 
"defines and publishes a workflow" or "reuses a previously published workflow" 
but I'm pretty sure I've come across many papers that do this.

A good place to start though would be papers with the shared 
tag<http://www.citeulike.org/group/16008/tag/shared>: Papers that have 
published/shared workflows, histories, datasets, pages, visualizations in a 
Galaxy instance.

Also, not sure if 
CloudMap<http://www.citeulike.org/group/16008/article/11450604> qualifies here, 
but if it does, many papers that reference it are reusing the pipeline defined 
in it.

Hope this helps,

Dave C

On Thu, Jul 21, 2016 at 7:21 PM, Steve Cassidy 
mailto:steve.cass...@mq.edu.au>> wrote:
Hi all,
 I’m writing a paper that will include some evangelism for Galaxy in language 
sciences and I’d like to be able to cite examples of papers in the Bio field 
that cite Galaxy workflows.  Even better would be the re-use of a Galaxy 
workflow cited in another paper.  Does anyone have any good example papers they 
could refer me to.

Thanks,

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/



--
http://galaxyproject.org/
http://getgalaxy.org/
http://usegalaxy.org/
https://wiki.galaxyproject.org/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Consuming dataset collections

[Steve Cassidy]

Hi all,
   I’m staring at the discussion of handling dataset collections:

http://planemo.readthedocs.io/en/latest/_writing_collections.html

but failing to see the solution to my problem.

I have a tool that creates a dataset collection, a group of files with names 
like 1_1308_1_2_092-ch6-speaker16.TextGrid where the 1_1308_1_2_092 part is a 
unique identifier that I’d like to keep track of.  I’ve used a 
discover_datasets tag in the tool xml file to match my output filenames and 
extract the designation (1_1308_1_2_092-ch6-speaker16.TextGrid) and the ext 
(TextGrid).

I have another tool that runs a query over these files and generates a single 
tabular result that will ideally include the identifier in some form. Here’s 
the command section for that tool:

query_textgrids.py --textgrid "${",".join(map(str, $textgrid))}" --tier 
$tier --regex '$regex' --output_path $output

where ‘$textgrid’ is one of my input parameters that has multiple=“true” set so 
that it can be a dataset collection.  That works ok but the input I get are the 
filenames (dataset_1.dat, etc.) not the name of the datasets.

The page above mentions something called the ‘element_identifier’ and gives 
this funky example:


merge_rows --name "${re.sub('[^\w\-_]', '_', $input.element_identifier)}" 
--file "$input" --to $output;


I can’t see what this element_identifier thing is - the suggestion is that it 
might be the dataset name, but I’m not sure.  Also I don’t understand why the 
command above is doing replacement of whitespace with underscores.

If this is the name I’m after, it would seem that I’d need to pass these names 
along with the textgrid files and then pair them up inside my script - is that 
what I need to do?

All of this cries out to me for a more explicit representation of a dataset 
collection that my tool can create and consume rather than this hacky treatment 
of filenames.   If I could generate a manifest file of some kind describing my 
dataset collection then none of this parsing of filenames would be needed.  I 
could also consume the manifest file as well and it could be used for 
collection level metadata.  Is this a silly idea?

Anyway, any help with my immediate problem would be appreciated.

Thanks,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Papers that cite Galaxy workflows

[Steve Cassidy]

Hi all,
 I’m writing a paper that will include some evangelism for Galaxy in language 
sciences and I’d like to be able to cite examples of papers in the Bio field 
that cite Galaxy workflows.  Even better would be the re-use of a Galaxy 
workflow cited in another paper.  Does anyone have any good example papers they 
could refer me to.

Thanks,

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] planemo shed_create bug? on planemo/server docker image

[Steve Cassidy]

Hi Bjorn,
  thanks for the response, very happy to go with the future way of doing 
things.  I did see the references to conda but couldn’t quite decipher what was 
needed. 

Are you saying that I should create a new recipe in bioconda for my dependency? 
Does it matter that this is not in the least ‘bio’ - these are for our language 
analysis tools, in the first instance this is a tool to pull data from our 
language repository.  

How does Galaxy know about the bioconda channel - is there a configuration 
setting pointing to the channels to find dependencies in? Could we create a 
language-conda channel to house language dependencies? Does that question make 
sense (not totally up on conda yet).

Steve 

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 14 Jul 2016, at 5:00 PM, Björn Grüning  wrote:
> 
> Hi Steve,
> 
> are you fine restricting your tools to a recent Galaxy version, let's
> say 6 month old?
> 
> If you forget about tool_depenencies.xml files, completely forget these!
> Conda packaging is the way to go and way more easy for you.
> 
> Let me know and I will give a short introduction or follow the
> description on https://github.com/bioconda/bioconda-recipes to create a
> PR and a package for your tool.
> 
> This can be used by Galaxy automatically.
> Thanks,
> Bjoern
> 
> Am 14.07.2016 um 03:49 schrieb Steve Cassidy:
>> I’m just getting my head around tool dependencies and in doing so trying to 
>> push my tools to a local toolshed which I can handily run with the 
>> planemo/server docker image via Kitematic.
>> 
>> That toolshed is set up with an admin user 
>> plan...@test.com<mailto:plan...@test.com> so I have a stanza in my 
>> .planemo.yml:
>> 
>> shed_username: "stevecassidy"
>> sheds:
>>  http://localhost:32768/:
>>key: ""
>>email: "plan...@test.com<mailto:plan...@test.com>"
>>password: "planemo"
>> 
>> Now I try to create a repository in the toolshed with planemo:
>> 
>> $ planemo shed_create --shed_target http://localhost:32768/
>> Repository created
>> cd '/Users/steve/projects/hcsvlab-galaxy/tools/alveo' && git rev-parse HEAD
>> cd '/Users/steve/projects/hcsvlab-galaxy/tools/alveo' && git diff --quiet
>> Repository [alveoimport] does not exist in the targeted Tool Shed.
>> 
>> 
>> Note the conflicting messages here. The repository is created (and I can 
>> confirm that it is there) but then I get the message that the repo does not 
>> exist in the Toolshed.  After a bit of digging it turns out that this is 
>> because the repo is owned by the user ‘planemo’ rather than my username as 
>> declared in .planemo.yml.  If I add —owner plane to the command then it 
>> works fine.
>> 
>> So I’m not sure if this is a bug in the shed_create code or a weird 
>> interaction with the setup on planemo/server.
>> 
>> Steve
>> 
>> —
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>> 
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] planemo shed_create bug? on planemo/server docker image

[Steve Cassidy]

I’m just getting my head around tool dependencies and in doing so trying to 
push my tools to a local toolshed which I can handily run with the 
planemo/server docker image via Kitematic.

That toolshed is set up with an admin user 
plan...@test.com so I have a stanza in my .planemo.yml:

shed_username: "stevecassidy"
sheds:
  http://localhost:32768/:
key: ""
email: "plan...@test.com"
password: "planemo"

Now I try to create a repository in the toolshed with planemo:

$ planemo shed_create --shed_target http://localhost:32768/
Repository created
cd '/Users/steve/projects/hcsvlab-galaxy/tools/alveo' && git rev-parse HEAD
cd '/Users/steve/projects/hcsvlab-galaxy/tools/alveo' && git diff --quiet
Repository [alveoimport] does not exist in the targeted Tool Shed.


Note the conflicting messages here. The repository is created (and I can 
confirm that it is there) but then I get the message that the repo does not 
exist in the Toolshed.  After a bit of digging it turns out that this is 
because the repo is owned by the user ‘planemo’ rather than my username as 
declared in .planemo.yml.  If I add —owner plane to the command then it works 
fine.

So I’m not sure if this is a bug in the shed_create code or a weird interaction 
with the setup on planemo/server.

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Position Vacant - Galaxy and Language Research in Sydney

[Steve Cassidy]

Hi all,
 I’d like to draw your attention to the following vacancy on our project in 
Sydney, Australia:

https://goo.gl/6NcY3g

This is on the Alveo project and I’m posting here because part of the platform 
is a Galaxy server with tools developed to work on language data (text and 
speech).   The role would suit someone with knowledge of Galaxy and some 
exposure to general eResearch problems; knowledge of language research methods 
would obviously be a bonus.

The ad mentions a part-time role for 6 months but in practice we have funding 
for longer than this and for a full time role for the right candidate. Please 
get in touch if you are interested in finding out about the post. International 
candidates are welcome to apply.

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Steve Cassidy]

A brief writeup of my experiences:

http://web.science.mq.edu.au/~cassidy/wordpress/2015/10/21/galaxy-tool-generating-datasets/

Steve

On 21 October 2015 at 18:27, Peter van Heusden  wrote:

> These names have meaning in CollectedDatasetMatch:
>
> designation means designation if that exists else name
> name means name
> dbkey means dbkey
> ext means ext
> visible means visible
>
>
> On 21 October 2015 at 08:44, Steve Cassidy 
> wrote:
>
>> Thanks, yes that works, though I'm not really sure what the difference is
>> between __name__ and __designation__. They both seem to have the same
>> effect in my example.
>>
>> Thanks for your help.
>>
>> Steve
>>
>>
>> On 21 October 2015 at 16:49, Peter van Heusden  wrote:
>>
>>> Since you're using a directory, you can use one of the built in patterns:
>>>
>>> DEFAULT_EXTRA_FILENAME_PATTERN = 
>>> r"primary_DATASET_ID_(?P[^_]+)_(?P[^_]+)_(?P[^_]+)(_(?P[^_]+))?"
>>>
>>> NAMED_PATTERNS = {
>>> "__default__": DEFAULT_EXTRA_FILENAME_PATTERN,
>>> "__name__": r"(?P.*)",
>>> "__designation__": r"(?P.*)",
>>> "__name_and_ext__": r"(?P.*)\.(?P[^\.]+)?",
>>> "__designation_and_ext__": r"(?P.*)\.(?P[^\._]+)?",
>>> }
>>>
>>> In terms of docs, I don't know what the future is - the Galaxy wiki or 
>>> http://galaxy.readthedocs.org/en/master/
>>>
>>>
>>> On 21 October 2015 at 04:26, Steve Cassidy 
>>> wrote:
>>>
>>>> Ah, thankyou, yes, I can now get results by using patterns to match the
>>>> output.  I used your example but prepend 'simple' to the filename and then
>>>> searches for that with:
>>>>
>>>> 
>>>> this solves the problem for the sample script but not generally since
>>>> in general I can't predict the filenames that will be generated - this is a
>>>> tool for downloading data from a repository which could be text, audio or
>>>> video data.
>>>>
>>>> If I don't use the 'simple' prefix and omit the file extension I still
>>>> get my data but I also get three other files which are temporary scripts
>>>> generated by galaxy and placed in the working directory.  So, back to
>>>> trying to put things in a subdirectory. It turns out that the issue I was
>>>> having was as you pointed out earlier, the directory attribute to
>>>> discover_datasets doesn't allow variables, so I need to write to a fixed
>>>> directory name:
>>>>
>>>> 
>>>> This now works!
>>>>
>>>> I had thought that I'd need to use a unique directory name but since
>>>> galaxy runs each job in a separate directory, this isn't required.  My real
>>>> tool now works too after following the same pattern.
>>>>
>>>> Thanks for your help.  I'll see if I can write this up in a blog post.
>>>>
>>>> Steve
>>>>
>>>>
>>>>
>>>> On 21 October 2015 at 00:06, Peter van Heusden  wrote:
>>>>
>>>>> I poked around at your tool XML and the code a bit and the problem is
>>>>> directory="$job_name". Galaxy expects to collect files from the job's
>>>>> working directory - basically the current working directory the job runs
>>>>> in. The directory= argument doesn't have variables expanded as far as I 
>>>>> can
>>>>> tell. In any event it is used in walk_over_extra_files() that is in
>>>>> lib/galaxy/tools/parameters/output_collect.py - if you look there you see
>>>>> that it is simply appended to the job's working directory.
>>>>>
>>>>> So if you use:
>>>>>
>>>>> 
>>>>>
>>>>> (note the < and > - this is effectively the regexp
>>>>> r"(?P.*)\.txt" with the < and > escaped out)
>>>>>
>>>>> And alter the code so that it just writes files to the current
>>>>> directory, then you'll pick up the files one.txt, two,txt and three.txt.
>>>>>
>>>>> Peter
>>>>>
>>>>> On 20 October 2015 at 12:28, Steve Cassidy 
>>>>> wrote:
>>>>>
>>>>>> Sorry, it wa

Re: [galaxy-dev] Tools that make datasets

[Steve Cassidy]

Ah, thankyou, yes, I can now get results by using patterns to match the
output.  I used your example but prepend 'simple' to the filename and then
searches for that with:


this solves the problem for the sample script but not generally since in
general I can't predict the filenames that will be generated - this is a
tool for downloading data from a repository which could be text, audio or
video data.

If I don't use the 'simple' prefix and omit the file extension I still get
my data but I also get three other files which are temporary scripts
generated by galaxy and placed in the working directory.  So, back to
trying to put things in a subdirectory. It turns out that the issue I was
having was as you pointed out earlier, the directory attribute to
discover_datasets doesn't allow variables, so I need to write to a fixed
directory name:


This now works!

I had thought that I'd need to use a unique directory name but since galaxy
runs each job in a separate directory, this isn't required.  My real tool
now works too after following the same pattern.

Thanks for your help.  I'll see if I can write this up in a blog post.

Steve



On 21 October 2015 at 00:06, Peter van Heusden  wrote:

> I poked around at your tool XML and the code a bit and the problem is
> directory="$job_name". Galaxy expects to collect files from the job's
> working directory - basically the current working directory the job runs
> in. The directory= argument doesn't have variables expanded as far as I can
> tell. In any event it is used in walk_over_extra_files() that is in
> lib/galaxy/tools/parameters/output_collect.py - if you look there you see
> that it is simply appended to the job's working directory.
>
> So if you use:
>
> 
>
> (note the < and > - this is effectively the regexp
> r"(?P.*)\.txt" with the < and > escaped out)
>
> And alter the code so that it just writes files to the current directory,
> then you'll pick up the files one.txt, two,txt and three.txt.
>
> Peter
>
> On 20 October 2015 at 12:28, Steve Cassidy 
> wrote:
>
>> Sorry, it was just an example of a tool that works - the file that it
>> writes out is put into that directory, so I assume that's where my files
>> should end up too.
>>
>> Steve
>>
>> On 20 October 2015 at 21:12, Peter van Heusden  wrote:
>>
>>> Sorry, I don't understand - what does the Upload File tool have to do
>>> with this?
>>>
>>> On 20 October 2015 at 11:49, Steve Cassidy 
>>> wrote:
>>>
>>>> Yes, I'm sure that's where the problem lies. Writing out to the current
>>>> directory doesn't work.  The files get written to
>>>> 'job_working_directory/000/1/' but if I run the Upload File tool the result
>>>> is placed in 'files/000/'.  I think I need to work out where to write the
>>>> files, I found some references to $__new_file_path__ but that doesn't seem
>>>> to help.
>>>>
>>>> Steve
>>>>
>>>>
>>>>
>>>> On 20 October 2015 at 19:57, Peter van Heusden  wrote:
>>>>
>>>>> I suspect that the problem might be in the  then.
>>>>> I'm not an export on this, but "__name_and_ext__" turns into the
>>>>> regexp r"(?P.*)\.(?P[^\.]+)?" in
>>>>> lib/galaxy/tools/parameters/output_collect.py, and is used by the
>>>>> DatasetCollector (line 358). This looks like it should match the filenames
>>>>> you're creating, but I'm not 100% sure how that code works. One thing I
>>>>> notice is the "directory" argument. If you write jobs to the current
>>>>> directory instead of "output_path" can you get it to work?
>>>>>
>>>>> Peter
>>>>>
>>>>> On 20 October 2015 at 09:52, Steve Cassidy 
>>>>> wrote:
>>>>>
>>>>>> Thanks Peter,
>>>>>>   I did see that proviso somewhere but no, refreshing doesn't help.
>>>>>>
>>>>>> That page was one of those that I referred to getting to this point.
>>>>>>
>>>>>> Steve
>>>>>>
>>>>>> On 20 October 2015 at 18:33, Peter van Heusden 
>>>>>> wrote:
>>>>>>
>>>>>>> Just a quick check - did you refresh your history to confirm that
>>>>>>> the dataset *is* empty? We had the same thing at SANBI but it turns out
>>>&g

Re: [galaxy-dev] Tools that make datasets

[Steve Cassidy]

Yes, I'm sure that's where the problem lies. Writing out to the current
directory doesn't work.  The files get written to
'job_working_directory/000/1/' but if I run the Upload File tool the result
is placed in 'files/000/'.  I think I need to work out where to write the
files, I found some references to $__new_file_path__ but that doesn't seem
to help.

Steve



On 20 October 2015 at 19:57, Peter van Heusden  wrote:

> I suspect that the problem might be in the  then. I'm
> not an export on this, but "__name_and_ext__" turns into the
> regexp r"(?P.*)\.(?P[^\.]+)?" in
> lib/galaxy/tools/parameters/output_collect.py, and is used by the
> DatasetCollector (line 358). This looks like it should match the filenames
> you're creating, but I'm not 100% sure how that code works. One thing I
> notice is the "directory" argument. If you write jobs to the current
> directory instead of "output_path" can you get it to work?
>
> Peter
>
> On 20 October 2015 at 09:52, Steve Cassidy 
> wrote:
>
>> Thanks Peter,
>>   I did see that proviso somewhere but no, refreshing doesn't help.
>>
>> That page was one of those that I referred to getting to this point.
>>
>> Steve
>>
>> On 20 October 2015 at 18:33, Peter van Heusden  wrote:
>>
>>> Just a quick check - did you refresh your history to confirm that the
>>> dataset *is* empty? We had the same thing at SANBI but it turns out that
>>> Galaxy creates an empty output collection and then only populates it
>>> sometime after job completion (this is a know UI bug).
>>>
>>> See:
>>> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/
>>>
>>> On 20 October 2015 at 08:48, Steve Cassidy 
>>> wrote:
>>>
>>>> Hi all,
>>>>   I'm trying to understand how to write a tool that generates a dataset
>>>> rather than a single output file.  I've tried following all of the examples
>>>> but I'm stuck, so I thought I would distil down the simplest example I
>>>> could write and ask for help here.
>>>>
>>>> So here's my example:
>>>>
>>>> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55
>>>>
>>>> it's a simple python script that writes three files to a directory
>>>> named for the single input parameter.
>>>>
>>>> I think one of the problems I'm having is knowing where to write the
>>>> output to. I've run this under planemo serve and the job runs, creating the
>>>> output directory within the 'job_working_directory/000/1/SampleDataset'
>>>> directory, however my dataset doesn't contain anything so clearly my
>>>> outputs directive isn't working:
>>>>
>>>> 
>>>> 
>>>> >>> directory="$job_name" />
>>>> 
>>>> 
>>>>
>>>> ($job_name is the name of the directory that is being written to,
>>>> SampleDataset in this case)
>>>>
>>>> Any help in getting this example working would be appreciated.
>>>>
>>>> Thanks,
>>>>
>>>> Steve
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Department of Computing, Macquarie University
>>>> http://web.science.mq.edu.au/~cassidy/
>>>>
>>>> ___
>>>> Please keep all replies on the list by using "reply all"
>>>> in your mail client.  To manage your subscriptions to this
>>>> and other Galaxy lists, please use the interface at:
>>>>   https://lists.galaxyproject.org/
>>>>
>>>> To search Galaxy mailing lists use the unified search at:
>>>>   http://galaxyproject.org/search/mailinglists/
>>>>
>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/mailinglists/
>>>
>>
>>
>>
>> --
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy/
>>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>



-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Steve Cassidy]

Thanks Peter,
  I did see that proviso somewhere but no, refreshing doesn't help.

That page was one of those that I referred to getting to this point.

Steve

On 20 October 2015 at 18:33, Peter van Heusden  wrote:

> Just a quick check - did you refresh your history to confirm that the
> dataset *is* empty? We had the same thing at SANBI but it turns out that
> Galaxy creates an empty output collection and then only populates it
> sometime after job completion (this is a know UI bug).
>
> See:
> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/
>
> On 20 October 2015 at 08:48, Steve Cassidy 
> wrote:
>
>> Hi all,
>>   I'm trying to understand how to write a tool that generates a dataset
>> rather than a single output file.  I've tried following all of the examples
>> but I'm stuck, so I thought I would distil down the simplest example I
>> could write and ask for help here.
>>
>> So here's my example:
>>
>> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55
>>
>> it's a simple python script that writes three files to a directory named
>> for the single input parameter.
>>
>> I think one of the problems I'm having is knowing where to write the
>> output to. I've run this under planemo serve and the job runs, creating the
>> output directory within the 'job_working_directory/000/1/SampleDataset'
>> directory, however my dataset doesn't contain anything so clearly my
>> outputs directive isn't working:
>>
>> 
>> 
>> > directory="$job_name" />
>> 
>> 
>>
>> ($job_name is the name of the directory that is being written to,
>> SampleDataset in this case)
>>
>> Any help in getting this example working would be appreciated.
>>
>> Thanks,
>>
>> Steve
>>
>>
>>
>>
>>
>> --
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy/
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>



-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Tools that make datasets

[Steve Cassidy]

Hi all,
  I'm trying to understand how to write a tool that generates a dataset
rather than a single output file.  I've tried following all of the examples
but I'm stuck, so I thought I would distil down the simplest example I
could write and ask for help here.

So here's my example:

https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55

it's a simple python script that writes three files to a directory named
for the single input parameter.

I think one of the problems I'm having is knowing where to write the output
to. I've run this under planemo serve and the job runs, creating the output
directory within the 'job_working_directory/000/1/SampleDataset' directory,
however my dataset doesn't contain anything so clearly my outputs directive
isn't working:







($job_name is the name of the directory that is being written to,
SampleDataset in this case)

Any help in getting this example working would be appreciated.

Thanks,

Steve





-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] planemo and tool dependencies

[Steve Cassidy]

Ok, thanks, I'll try to work that out.  Meanwhile I've got my tools working
in planemo test mode - it now seems to be using the installed python
libraries where earlier it was creating a fresh venv - I updated my galaxy
sources.

Thanks for your help.

Steve


On 4 September 2015 at 22:09, Christian Brenninkmeijer <
christian.brenninkmei...@manchester.ac.uk> wrote:

> Hi Steve,
> You would need to create a too_depencency and load it to a tool shed.
>
> Then install it from that tool shed to your galaxy.
>
> Unless someone who know galaxy better jumps and says there is another way!
>
> Christian
>
> PS. Either uses the testtoolshed or start your own.
>
> --
> *From:* Steve Cassidy [steve.cass...@mq.edu.au]
> *Sent:* Friday, September 04, 2015 12:29 PM
>
> *To:* Christian Brenninkmeijer
> *Cc:* galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] planemo and tool dependencies
>
> On 4 September 2015 at 21:00, Christian Brenninkmeijer <
> christian.brenninkmei...@manchester.ac.uk> wrote:
>
>> I wrote the tool because when I ran planemo it found the tool_depencecy
>> file but then logged that it could not find the actual tools.
>>
>> So
>> 1. Did you install the dependecies into the galaxy you are using to run
>> planemo?
>> If not my tool will not work until you do.
>>
>>
> I didn't do this - how do I do this?  Do I install the modules into the
> --tool_dependency_dir ?
>
>
>
>> 2. When you run the test does the log show it looked for but could not
>> find python_nltk_tools?
>> If not then the error is outside of the scope of the tool.
>>
>>
> Yes, it said:
>
> galaxy.tools.deps DEBUG 2015-09-04 20:55:32,054 Building dependency shell
> command for dependency 'python_nltk_tools'
> galaxy.tools.deps.resolvers.unlinked_tool_shed_packages DEBUG 2015-09-04
> 20:55:32,054 Unable to find dependency,'python_nltk_tools' '1.0' 'package'
> galaxy.tools.deps WARNING 2015-09-04 20:55:32,054 Failed to resolve
> dependency on 'python_nltk_tools', ignoring
>
> Thanks,
>
> Steve
>
>
>
>
>> Christian
>> --
>> *From:* Steve Cassidy [steve.cass...@mq.edu.au]
>> *Sent:* Friday, September 04, 2015 11:41 AM
>> *To:* Christian Brenninkmeijer
>> *Cc:* galaxy-dev@lists.galaxyproject.org
>> *Subject:* Re: [galaxy-dev] planemo and tool dependencies
>>
>> Thanks Christian,
>>   I'm not sure I understand how to make use of this.  My tool dependency
>> is described in an XML file which is in the same source directory as the
>> tool xml and python files.  The tool xml file contains:
>>
>> > version="1.0">python_nltk_tools
>>
>> and the tool_dependencies.xml file contains the definition:
>>
>> 
>> 
>> ...
>>
>> how do I tell planemo to tell galaxy that it should refer to this tool
>> dependency file to resolve the requirement? At the moment it's loading all
>> of the tools but doesn't seem to be concerned that the requirement isn't
>> met.
>>
>> Thanks for your help.
>>
>> Steve
>>
>>
>>
>> On 3 September 2015 at 18:06, Christian Brenninkmeijer <
>> christian.brenninkmei...@manchester.ac.uk> wrote:
>>
>>> Hi,
>>>
>>> I wrote an extension that will help planemo pull in tool dependencies.
>>>
>>> See
>>> https://github.com/galaxyproject/galaxy/blob/master/lib/galaxy/tools/deps/resolvers/unlinked_tool_shed_packages.py
>>>
>>> It is for when the dependency was installed into galaxy but where
>>> planemo is not using the database to find that tool.
>>>
>>> It works my helping planemo guess where the tool dependency may be.
>>>
>>> Hope that helps.
>>> Christian
>>> University of Manchester
>>> --
>>> *From:* galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on
>>> behalf of Steve Cassidy [steve.cass...@mq.edu.au]
>>> *Sent:* Thursday, September 03, 2015 9:00 AM
>>> *To:* galaxy-dev@lists.galaxyproject.org
>>> *Subject:* [galaxy-dev] planemo and tool dependencies
>>>
>>> Hi all,
>>>   I'm trying to test some tools under planemo and can't find out how to
>>> have it install tool dependencies while setting up the environment.
>>>
>>> The dependency is on a particular python module and we have a
>>> tool_dependencies.xml fil

Re: [galaxy-dev] planemo and tool dependencies

[Steve Cassidy]

On 4 September 2015 at 21:00, Christian Brenninkmeijer <
christian.brenninkmei...@manchester.ac.uk> wrote:

> I wrote the tool because when I ran planemo it found the tool_depencecy
> file but then logged that it could not find the actual tools.
>
> So
> 1. Did you install the dependecies into the galaxy you are using to run
> planemo?
> If not my tool will not work until you do.
>
>
I didn't do this - how do I do this?  Do I install the modules into the
--tool_dependency_dir ?



> 2. When you run the test does the log show it looked for but could not
> find python_nltk_tools?
> If not then the error is outside of the scope of the tool.
>
>
Yes, it said:

galaxy.tools.deps DEBUG 2015-09-04 20:55:32,054 Building dependency shell
command for dependency 'python_nltk_tools'
galaxy.tools.deps.resolvers.unlinked_tool_shed_packages DEBUG 2015-09-04
20:55:32,054 Unable to find dependency,'python_nltk_tools' '1.0' 'package'
galaxy.tools.deps WARNING 2015-09-04 20:55:32,054 Failed to resolve
dependency on 'python_nltk_tools', ignoring

Thanks,

Steve




> Christian
> --
> *From:* Steve Cassidy [steve.cass...@mq.edu.au]
> *Sent:* Friday, September 04, 2015 11:41 AM
> *To:* Christian Brenninkmeijer
> *Cc:* galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] planemo and tool dependencies
>
> Thanks Christian,
>   I'm not sure I understand how to make use of this.  My tool dependency
> is described in an XML file which is in the same source directory as the
> tool xml and python files.  The tool xml file contains:
>
>  version="1.0">python_nltk_tools
>
> and the tool_dependencies.xml file contains the definition:
>
> 
> 
> ...
>
> how do I tell planemo to tell galaxy that it should refer to this tool
> dependency file to resolve the requirement? At the moment it's loading all
> of the tools but doesn't seem to be concerned that the requirement isn't
> met.
>
> Thanks for your help.
>
> Steve
>
>
>
> On 3 September 2015 at 18:06, Christian Brenninkmeijer <
> christian.brenninkmei...@manchester.ac.uk> wrote:
>
>> Hi,
>>
>> I wrote an extension that will help planemo pull in tool dependencies.
>>
>> See
>> https://github.com/galaxyproject/galaxy/blob/master/lib/galaxy/tools/deps/resolvers/unlinked_tool_shed_packages.py
>>
>> It is for when the dependency was installed into galaxy but where planemo
>> is not using the database to find that tool.
>>
>> It works my helping planemo guess where the tool dependency may be.
>>
>> Hope that helps.
>> Christian
>> University of Manchester
>> --
>> *From:* galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on
>> behalf of Steve Cassidy [steve.cass...@mq.edu.au]
>> *Sent:* Thursday, September 03, 2015 9:00 AM
>> *To:* galaxy-dev@lists.galaxyproject.org
>> *Subject:* [galaxy-dev] planemo and tool dependencies
>>
>> Hi all,
>>   I'm trying to test some tools under planemo and can't find out how to
>> have it install tool dependencies while setting up the environment.
>>
>> The dependency is on a particular python module and we have a
>> tool_dependencies.xml file to describe it. I can't see a way to tell
>> planemo serve etc to refer to that file to get the dependencies.
>>
>> Any help appreciated.
>>
>> Steve
>>
>> --
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy/
>>
>
>
>
> --
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy/
>



-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] planemo and tool dependencies

[Steve Cassidy]

Thanks Christian,
  I'm not sure I understand how to make use of this.  My tool dependency is
described in an XML file which is in the same source directory as the tool
xml and python files.  The tool xml file contains:

python_nltk_tools

and the tool_dependencies.xml file contains the definition:



...

how do I tell planemo to tell galaxy that it should refer to this tool
dependency file to resolve the requirement? At the moment it's loading all
of the tools but doesn't seem to be concerned that the requirement isn't
met.

Thanks for your help.

Steve



On 3 September 2015 at 18:06, Christian Brenninkmeijer <
christian.brenninkmei...@manchester.ac.uk> wrote:

> Hi,
>
> I wrote an extension that will help planemo pull in tool dependencies.
>
> See
> https://github.com/galaxyproject/galaxy/blob/master/lib/galaxy/tools/deps/resolvers/unlinked_tool_shed_packages.py
>
> It is for when the dependency was installed into galaxy but where planemo
> is not using the database to find that tool.
>
> It works my helping planemo guess where the tool dependency may be.
>
> Hope that helps.
> Christian
> University of Manchester
> --
> *From:* galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf
> of Steve Cassidy [steve.cass...@mq.edu.au]
> *Sent:* Thursday, September 03, 2015 9:00 AM
> *To:* galaxy-dev@lists.galaxyproject.org
> *Subject:* [galaxy-dev] planemo and tool dependencies
>
> Hi all,
>   I'm trying to test some tools under planemo and can't find out how to
> have it install tool dependencies while setting up the environment.
>
> The dependency is on a particular python module and we have a
> tool_dependencies.xml file to describe it. I can't see a way to tell
> planemo serve etc to refer to that file to get the dependencies.
>
> Any help appreciated.
>
> Steve
>
> --
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy/
>



-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] planemo and tool dependencies

[Steve Cassidy]

Hi all,
  I'm trying to test some tools under planemo and can't find out how to
have it install tool dependencies while setting up the environment.

The dependency is on a particular python module and we have a
tool_dependencies.xml file to describe it. I can't see a way to tell
planemo serve etc to refer to that file to get the dependencies.

Any help appreciated.

Steve

-- 
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/