Re: [galaxy-dev] environment variables and paths for toolshed tools
On Tue, Feb 19, 2013 at 12:06 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Feb 19, 2013 at 11:39 AM, Greg Von Kuster g...@bx.psu.edu wrote: On Feb 19, 2013, at 5:10 AM, Peter Cock wrote: Yes, the complete directory hierarchy and content of the original repository revision is preserved when it installed into Galaxy. Based on a sample of one, it seems the installation process just unpacks the tool shed files (well, probably using hg rather than unpacking a tar-ball), and preserves their relative path structure. If so, I can just use relative paths to find a data file from the tool executable's own location on disk. (This is what I was hoping would be the case - I'm looking for explicit confirmation). This is correct, hg clone provides this behavior. i.e. For the example use case, if motif.py and motif.dat are in the same folder in the Tool Shed upload, they will be in the same folder as each other once installed. That way motif.py can easily locate the data file motif.dat by looking in the same folder as it itself is located. This is actually very simple (provided I can assume the local folder structure is maintained). Yes, this is correct. Excellent - that solves this use-case nicely. The tool I was talking about is actually a re-implementation of predictNLS from the Rost Lab, which uses regular expressions to look for Nuclear Localization Signals (NLS): https://rostlab.org/owiki/index.php/PredictNLS My Galaxy version is now available here: http://toolshed.g2.bx.psu.edu/view/peterjc/predictnls Touch wood that will automatically install (and comes with a minimal test case too). Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
On Sat, Feb 16, 2013 at 9:41 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Peter, Thanks for your thoughts on this. I've created the following Trello card for this enhancement. https://trello.com/card/galaxy-tool-enhancement-to-accommodate-repository-install-dir/506338ce32ae458f6d15e4b3/636 The priority for the tool shed in the area of tool dependencies like this has been the implementation of features that enable sharing a single dependency across multiple tools in separate repositories, so your scenario is not yet supported. This enhancement would actually be as much (or perhaps more) on the Galaxy end than the tool shed, I think, so there may be more resources available to get to it sooner than I can right now. Of course, someone wil get to it as soon as possible. Thanks again for all of your feedback and insight on this. Greg Von Kuster Hi Greg, I've read http://wiki.galaxyproject.org/InstallingRepositoriesToGalaxy which is targeted at Galaxy administrators rather than tools authors. It explains how a revision specific folder is assigned to each installed tool, but what (if anything) of the Tool's folder structure is preserved? e.g. One of the paths given is: .../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/emboss_datatypes/a89163f31369/emboss_datatypes/datatypes_conf.xml The prefix is the revision specific path for that tool for that Tool Shed, .../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/emboss_datatypes/a89163f31369/ Within that, it seems to preserve the emboss_datatypes folders. i.e. goto http://toolshed.g2.bx.psu.edu/view/devteam/emboss_datatypes and click browse repository tip files from the top left actions menu, it has just one file: emboss_datatypes/datatypes_conf.xml Based on a sample of one, it seems the installation process just unpacks the tool shed files (well, probably using hg rather than unpacking a tar-ball), and preserves their relative path structure. If so, I can just use relative paths to find a data file from the tool executable's own location on disk. (This is what I was hoping would be the case - I'm looking for explicit confirmation). i.e. For the example use case, if motif.py and motif.dat are in the same folder in the Tool Shed upload, they will be in the same folder as each other once installed. That way motif.py can easily locate the data file motif.dat by looking in the same folder as it itself is located. This is actually very simple (provided I can assume the local folder structure is maintained). (I think I was originally on the wrong track by assuming I should be using the tool-data folder, which is complicated by not knowing where that will existing on disk). Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hi Peter, see below... On Feb 19, 2013, at 5:10 AM, Peter Cock wrote: On Sat, Feb 16, 2013 at 9:41 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Peter, Thanks for your thoughts on this. I've created the following Trello card for this enhancement. https://trello.com/card/galaxy-tool-enhancement-to-accommodate-repository-install-dir/506338ce32ae458f6d15e4b3/636 The priority for the tool shed in the area of tool dependencies like this has been the implementation of features that enable sharing a single dependency across multiple tools in separate repositories, so your scenario is not yet supported. This enhancement would actually be as much (or perhaps more) on the Galaxy end than the tool shed, I think, so there may be more resources available to get to it sooner than I can right now. Of course, someone wil get to it as soon as possible. Thanks again for all of your feedback and insight on this. Greg Von Kuster Hi Greg, I've read http://wiki.galaxyproject.org/InstallingRepositoriesToGalaxy which is targeted at Galaxy administrators rather than tools authors. It explains how a revision specific folder is assigned to each installed tool, but what (if anything) of the Tool's folder structure is preserved? The installation process uses the mercurial API, so the process to install a specific repository changeset_revision ultimately includes the following 2 hg commands. hg clone repository_clone_url hg update -r changeset_revision This results in the structure of the installed repository being exactly the same of the specific revision from which the repository was cloned. e.g. One of the paths given is: .../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/emboss_datatypes/a89163f31369/emboss_datatypes/datatypes_conf.xml The prefix is the revision specific path for that tool for that Tool Shed, .../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/emboss_datatypes/a89163f31369/ Within that, it seems to preserve the emboss_datatypes folders. i.e. goto http://toolshed.g2.bx.psu.edu/view/devteam/emboss_datatypes and click browse repository tip files from the top left actions menu, it has just one file: emboss_datatypes/datatypes_conf.xml Yes, the complete directory hierarchy and content of the original repository revision is preserved when it installed into Galaxy. Based on a sample of one, it seems the installation process just unpacks the tool shed files (well, probably using hg rather than unpacking a tar-ball), and preserves their relative path structure. If so, I can just use relative paths to find a data file from the tool executable's own location on disk. (This is what I was hoping would be the case - I'm looking for explicit confirmation). This is correct, hg clone provides this behavior. i.e. For the example use case, if motif.py and motif.dat are in the same folder in the Tool Shed upload, they will be in the same folder as each other once installed. That way motif.py can easily locate the data file motif.dat by looking in the same folder as it itself is located. This is actually very simple (provided I can assume the local folder structure is maintained). Yes, this is correct. (I think I was originally on the wrong track by assuming I should be using the tool-data folder, which is complicated by not knowing where that will existing on disk). I think your request is still valid though and we will add this enhancement for the Galaxy tool component to handle $REPOSITORY_INSTALL_DIR in tool configs. In the meantime, it looks like you can work around this missing feature. Please let me know if you bump into issues. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
On Tue, Feb 19, 2013 at 11:39 AM, Greg Von Kuster g...@bx.psu.edu wrote: On Feb 19, 2013, at 5:10 AM, Peter Cock wrote: Yes, the complete directory hierarchy and content of the original repository revision is preserved when it installed into Galaxy. Based on a sample of one, it seems the installation process just unpacks the tool shed files (well, probably using hg rather than unpacking a tar-ball), and preserves their relative path structure. If so, I can just use relative paths to find a data file from the tool executable's own location on disk. (This is what I was hoping would be the case - I'm looking for explicit confirmation). This is correct, hg clone provides this behavior. i.e. For the example use case, if motif.py and motif.dat are in the same folder in the Tool Shed upload, they will be in the same folder as each other once installed. That way motif.py can easily locate the data file motif.dat by looking in the same folder as it itself is located. This is actually very simple (provided I can assume the local folder structure is maintained). Yes, this is correct. Excellent - that solves this use-case nicely. (I think I was originally on the wrong track by assuming I should be using the tool-data folder, which is complicated by not knowing where that will existing on disk). I think your request is still valid though and we will add this enhancement for the Galaxy tool component to handle $REPOSITORY_INSTALL_DIR in tool configs. In the meantime, it looks like you can work around this missing feature. Please let me know if you bump into issues. I can still see potential uses for $REPOSITORY_INSTALL_DIR both in the tool XML language, and as an environment variable - but I don't need that for the time being. Thanks Greg, all sorted for now, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hi Peter, Thanks for your thoughts on this. I've created the following Trello card for this enhancement. https://trello.com/card/galaxy-tool-enhancement-to-accommodate-repository-install-dir/506338ce32ae458f6d15e4b3/636 The priority for the tool shed in the area of tool dependencies like this has been the implementation of features that enable sharing a single dependency across multiple tools in separate repositories, so your scenario is not yet supported. This enhancement would actually be as much (or perhaps more) on the Galaxy end than the tool shed, I think, so there may be more resources available to get to it sooner than I can right now. Of course, someone wil get to it as soon as possible. Thanks again for all of your feedback and insight on this. Greg Von Kuster On Feb 15, 2013, at 12:07 PM, Peter Cock wrote: On Fri, Feb 15, 2013 at 4:57 PM, Greg Von Kuster g...@bx.psu.edu wrote: In this case, your motif.xml file can be configured with the following: command interpreter=python motif.py $fasta_file $tabular_file /command requirements requirement type=set_environmentMOTIF_DAT_PATH/requirement /requirements and your repository will need to include the following tool_dependencies.xml file: ?xml version=1.0? tool_dependency set_environment version=1.0 environment_variable name=MOTIF_DAT_PATH action=set_to$REPOSITORY_INSTALL_DIR/environment_variable /set_environment /tool_dependency That seems overly complicated. Would I not be able to assume that the motif.xml, motif.py, motif.dat etc are all in the same folder (then I can just use a relative path to find the data file - nice and easy). If not, how about simply defining $REPOSITORY_INSTALL_DIR within the XML language so I can do this: command interpreter=pythonmotif.py $fasta_file $tabular_file $REPOSITORY_INSTALL_DIR/motif.dat/command And/or defining $REPOSITORY_INSTALL_DIR as an environment variable which I can then use in the script (be it Python, Perl, shell, etc)? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hello Peter, please see my response at the end.
On Feb 15, 2013, at 6:47 AM, Peter Cock wrote:
On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
Are there any environment variables that are honored by toolshed installed
tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or
$GALAXY_HOME, then uploaded it the test toolshed, then installed it
automatically. Neither of these resolved to what I expected. I don't want
to hard-code the path in the xml tool file, but rather have a default
location for other executables and jar files. How should this best be done?
David Hoover
Helix Systems Staff
On Wed, Sep 12, 2012 at 6:15 PM, Greg Von Kuster g...@bx.psu.edu wrote:
Hello David,
This is not currently possible, but will be available in the next Galaxy
distribution release, currently scheduled for about 10 days from now. You
will include a tool_dependencies.xml file in your tool shed repository that
looks something like the following.
?xml version=1.0?
tool_dependency
set_environment version=1.0
environment_variable name=JAVA_JAR_PATH
action=set_to$INSTALL_DIR/environment_variable
/set_environment
/tool_dependency
Your tool will find the required files via the defined JAVA_JAR_PATH by
using a requirements tag set in the tool config, something like this:
requirements
requirement type=set_environmentJAVA_JAR_PATH /requirement
/requirements
The command tag in the tool config would be something like this:
command interpreter=python
python_wrapper.py $JAVA_JAR_PATH/some_file.jar $param1 $param2...
/command
I'm close to having this working, so if you are interested in testing before
the next Galaxy dist release, let me know and I'll tell you when the Galaxy
central repository has the new feature.
Thanks!
Greg Von Kuster
Hi guys,
Reading the rest of the thread from December, Greg's new code did
get checked in and should be working (I've not tried this yet).
However, I can't help feeling that defining the two environment variables
David suggested, $GALAXY_DATA_INDEX_DIR and/or $GALAXY_HOME,
would cover the majority of use cases and be far simpler for tool authors
to use. Am I overlooking something?
Usecase: Tool comes with a simple datafile (e.g. defining a search motif,
say motif.dat) which is used by a script (say motif.py) via a normal
Galaxy tool XML file (say motif.xml).
Perhaps I can just put my data file next to the script and XML file, in
which case it is easy for the script to locate? But I assumed that
Galaxy best practice would be to use the tool-data folder somehow...
Thanks,
Peter
The current implementation places your motif.dat data file in the repository
installation directory, and tools that are properly configured to locate
dependencies that are included in the repository contents will find it. This
approach allows for tool dependency discovery and easy maintenance when
uninstalling repositories as all contents are kept where they were installed.
Tools that are properly configured with requirement tags will find
dependencies because the associated env.sh file is sourced, providing the
location of all dependencies to the environment.
If your motif.dat file is one that would be manually altered by the Galaxy
administrator over time, perhaps automatically moving it to the
GALAXY_DATA_INDEX_DIR is justified. However, in this case, uninstalling the
repository would probably not uninstall the motif.dat file. Does this fit with
what you are thinking?
What kinds of files would require knowledge of the GALAXY_HOME directory?
Thanks Peter,
Greg Von Kuster
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
On Fri, Feb 15, 2013 at 4:05 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi guys, Reading the rest of the thread from December, Greg's new code did get checked in and should be working (I've not tried this yet). However, I can't help feeling that defining the two environment variables David suggested, $GALAXY_DATA_INDEX_DIR and/or $GALAXY_HOME, would cover the majority of use cases and be far simpler for tool authors to use. Am I overlooking something? Usecase: Tool comes with a simple datafile (e.g. defining a search motif, say motif.dat) which is used by a script (say motif.py) via a normal Galaxy tool XML file (say motif.xml). Perhaps I can just put my data file next to the script and XML file, in which case it is easy for the script to locate? But I assumed that Galaxy best practice would be to use the tool-data folder somehow... Thanks, Peter The current implementation places your motif.dat data file in the repository installation directory, and tools that are properly configured to locate dependencies that are included in the repository contents will find it. This approach allows for tool dependency discovery and easy maintenance when uninstalling repositories as all contents are kept where they were installed. Tools that are properly configured with requirement tags will find dependencies because the associated env.sh file is sourced, providing the location of all dependencies to the environment. Is there a documented example I should be reading? In this use case (without any explicit XML markup), can I assume/have the motif.dat file be installed in the same folder next to the motif.py script and tool defining motif.xml file? If your motif.dat file is one that would be manually altered by the Galaxy administrator over time, perhaps automatically moving it to the GALAXY_DATA_INDEX_DIR is justified. However, in this case, uninstalling the repository would probably not uninstall the motif.dat file. Does this fit with what you are thinking? In this case no, I would not expect the motif.dat file to be edited. Does this mean GALAXY_DATA_INDEX_DIR and the galaxy tool-data folder are intended for 'configuration' files the local admin may need to edit, rather than static tool specific data file? What kinds of files would require knowledge of the GALAXY_HOME directory? Knowing GALAXY_DATA_INDEX_DIR would probably cover most cases, so knowing GALAXY_HOME is probably not needed. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hi Peter, On Feb 15, 2013, at 11:14 AM, Peter Cock wrote: On Fri, Feb 15, 2013 at 4:05 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi guys, Reading the rest of the thread from December, Greg's new code did get checked in and should be working (I've not tried this yet). However, I can't help feeling that defining the two environment variables David suggested, $GALAXY_DATA_INDEX_DIR and/or $GALAXY_HOME, would cover the majority of use cases and be far simpler for tool authors to use. Am I overlooking something? Usecase: Tool comes with a simple datafile (e.g. defining a search motif, say motif.dat) which is used by a script (say motif.py) via a normal Galaxy tool XML file (say motif.xml). Perhaps I can just put my data file next to the script and XML file, in which case it is easy for the script to locate? But I assumed that Galaxy best practice would be to use the tool-data folder somehow... Thanks, Peter The current implementation places your motif.dat data file in the repository installation directory, and tools that are properly configured to locate dependencies that are included in the repository contents will find it. This approach allows for tool dependency discovery and easy maintenance when uninstalling repositories as all contents are kept where they were installed. Tools that are properly configured with requirement tags will find dependencies because the associated env.sh file is sourced, providing the location of all dependencies to the environment. Is there a documented example I should be reading? In this use case (without any explicit XML markup), can I assume/have the motif.dat file be installed in the same folder next to the motif.py script and tool defining motif.xml file? This scenario uses a tool dependency that is included in the repository which is discussed in the following section of the tool shed wiki. Using this approach a tool config (Cheetah template) would have to be configured with the proper requirement tag set and an associated command tag set that would ultimately have the path to the motif.py script and motif.dat file defined in the environment. Of course, this assumes the motify.py script is not the tool executable itself. http://wiki.galaxyproject.org/ToolShedToolFeatures#Finding_dependencies_included_in_an_installed_repository_at_tool_execution_time If your motif.dat file is one that would be manually altered by the Galaxy administrator over time, perhaps automatically moving it to the GALAXY_DATA_INDEX_DIR is justified. However, in this case, uninstalling the repository would probably not uninstall the motif.dat file. Does this fit with what you are thinking? In this case no, I would not expect the motif.dat file to be edited. Does this mean GALAXY_DATA_INDEX_DIR and the galaxy tool-data folder are intended for 'configuration' files the local admin may need to edit, rather than static tool specific data file? Not necessarily, but with the exception of sample .loc files included in the repository, repository contents should not generally be required to be moved outside of their installation directory. What kinds of files would require knowledge of the GALAXY_HOME directory? Knowing GALAXY_DATA_INDEX_DIR would probably cover most cases, so knowing GALAXY_HOME is probably not needed. I believe that the current implementation supports locating all tool dependencies, whether included in the repository or 3-rd party, so I'm not sure introducing GALAXY_DATA_INDEX_DIR to the repository installation process will be beneficial. I can be swayed, however, if I see justification for supporting it. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
On Fri, Feb 15, 2013 at 4:32 PM, Greg Von Kuster g...@bx.psu.edu wrote: Usecase: Tool comes with a simple datafile (e.g. defining a search motif, say motif.dat) which is used by a script (say motif.py) via a normal Galaxy tool XML file (say motif.xml). Perhaps I can just put my data file next to the script and XML file, in which case it is easy for the script to locate? But I assumed that Galaxy best practice would be to use the tool-data folder somehow... Thanks, Peter Is there a documented example I should be reading? In this use case (without any explicit XML markup), can I assume/have the motif.dat file be installed in the same folder next to the motif.py script and tool defining motif.xml file? This scenario uses a tool dependency that is included in the repository which is discussed in the following section of the tool shed wiki. Using this approach a tool config (Cheetah template) would have to be configured with the proper requirement tag set and an associated command tag set that would ultimately have the path to the motif.py script and motif.dat file defined in the environment. Of course, this assumes the motify.py script is not the tool executable itself. Actually for the example I am working on, the motif.py script would be the tool executable, called from motif.xml like this: command interpreter=pythonmotif.py $fasta_file $tabular_file/command i.e. A completely self contained Galaxy tool consisting of four files: motif.xml - Galaxy tool definition motif.txt - README file motif.py - Python script which is the tool motif.dat - Data file used by the Python script I am hoping that no further XML configuration files are needed in order to handle the data file. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
On Feb 15, 2013, at 11:45 AM, Peter Cock wrote: On Fri, Feb 15, 2013 at 4:32 PM, Greg Von Kuster g...@bx.psu.edu wrote: Usecase: Tool comes with a simple datafile (e.g. defining a search motif, say motif.dat) which is used by a script (say motif.py) via a normal Galaxy tool XML file (say motif.xml). Perhaps I can just put my data file next to the script and XML file, in which case it is easy for the script to locate? But I assumed that Galaxy best practice would be to use the tool-data folder somehow... Thanks, Peter Is there a documented example I should be reading? In this use case (without any explicit XML markup), can I assume/have the motif.dat file be installed in the same folder next to the motif.py script and tool defining motif.xml file? This scenario uses a tool dependency that is included in the repository which is discussed in the following section of the tool shed wiki. Using this approach a tool config (Cheetah template) would have to be configured with the proper requirement tag set and an associated command tag set that would ultimately have the path to the motif.py script and motif.dat file defined in the environment. Of course, this assumes the motify.py script is not the tool executable itself. Actually for the example I am working on, the motif.py script would be the tool executable, called from motif.xml like this: command interpreter=pythonmotif.py $fasta_file $tabular_file/command i.e. A completely self contained Galaxy tool consisting of four files: motif.xml - Galaxy tool definition motif.txt - README file motif.py - Python script which is the tool motif.dat - Data file used by the Python script I am hoping that no further XML configuration files are needed in order to handle the data file. Peter In this case, your motif.xml file can be configured with the following: command interpreter=python motif.py $fasta_file $tabular_file /command requirements requirement type=set_environmentMOTIF_DAT_PATH/requirement /requirements and your repository will need to include the following tool_dependencies.xml file: ?xml version=1.0? tool_dependency set_environment version=1.0 environment_variable name=MOTIF_DAT_PATH action=set_to$REPOSITORY_INSTALL_DIR/environment_variable /set_environment /tool_dependency ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
On Fri, Feb 15, 2013 at 4:57 PM, Greg Von Kuster g...@bx.psu.edu wrote: On Feb 15, 2013, at 11:45 AM, Peter Cock wrote: Actually for the example I am working on, the motif.py script would be the tool executable, called from motif.xml like this: command interpreter=pythonmotif.py $fasta_file $tabular_file/command i.e. A completely self contained Galaxy tool consisting of four files: motif.xml - Galaxy tool definition motif.txt - README file motif.py - Python script which is the tool motif.dat - Data file used by the Python script I am hoping that no further XML configuration files are needed in order to handle the data file. Peter In this case, your motif.xml file can be configured with the following: command interpreter=python motif.py $fasta_file $tabular_file /command requirements requirement type=set_environmentMOTIF_DAT_PATH/requirement /requirements and your repository will need to include the following tool_dependencies.xml file: ?xml version=1.0? tool_dependency set_environment version=1.0 environment_variable name=MOTIF_DAT_PATH action=set_to$REPOSITORY_INSTALL_DIR/environment_variable /set_environment /tool_dependency That seems overly complicated. Would I not be able to assume that the motif.xml, motif.py, motif.dat etc are all in the same folder (then I can just use a relative path to find the data file - nice and easy). If not, how about simply defining $REPOSITORY_INSTALL_DIR within the XML language so I can do this: command interpreter=pythonmotif.py $fasta_file $tabular_file $REPOSITORY_INSTALL_DIR/motif.dat/command And/or defining $REPOSITORY_INSTALL_DIR as an environment variable which I can then use in the script (be it Python, Perl, shell, etc)? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
in prepare_for_install
tool_dependencies
{'set_environment': [{'type': 'environment variable', 'name':
'JAVA_JAR_PATH'}], 'snpEff/3.0f': {'readme': '\n', 'version':
'3.0f', 'type': 'package', 'name': 'snpEff'}}
...
Module _admin_tool_shed_repository_common_mako:242 in
render_render_tool_dependency_section
name = requirements_dict[ 'name' ]
TypeError: list indices must be integers, not str
It appears that the requirements_dict for the set_environment tag is
enclosed in an array.
I haven't yet looked at the repository code to see how that might happen.
Pablo,
The http://sourceforge.net/projects/snpeff/files/ repository should
probably have versions packaged for download named like:
http://sourceforge.net/projects/snpeff/files/snpEff_3_0.tgz
$ tar ztvf snpEff_3_0.tgz
drwxrwxrwx 0 0 0 0 Aug 23 12:17 snpEff_3_0/
-rwxrwxrwx 0 0 014298767 Aug 23 12:17 snpEff_3_0/SnpSift.jar
...
That will allow versioning within galaxy, and successive versions of
tool_dependencies.xml with only need a change to the action tag:
download_by_url
Thanks,
JJ
I'm trying to test a repository that use in tool_depen
Message: 18
Date: Thu, 13 Sep 2012 11:46:32 -0400
From: Greg Von Kuster g...@bx.psu.edu
To: David Hoover hoove...@helix.nih.gov
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] environment variables and paths for toolshed
tools
Message-ID: f75d6320-3af8-4bb0-8f31-b0f44aa13...@bx.psu.edu
Content-Type: text/plain; charset=us-ascii
Hello David,
I've just committed change set 7656:6aadac8026cb to the Galaxy central
repository that provides the ability to do what you need here. It would be
great if you could try things out and let me know if you run into any
problems.
I've been working with nikhil-joshi's deseq_and_sam2counts repository in
the main Galaxy tool shed because his tools require this new feature as
well. This example should provide you with the information you'll need to
tweak your tool shed repository so that your tool dependencies are located
where they get installed rather than attempting to move them to
${GALAXY_DATA_INDEX_DIR} or some other location.
Here is the tool_dependency.xml file entry for locating a directory
referred to by an environment variable name R_SCRIPT_PATH.
?xml version=1.0?
tool_dependency
set_environment version=1.0
environment_variable name=R_SCRIPT_PATH
action=set_to$REPOSITORY_INSTALL_DIR/environment_variable
/set_environment
/tool_dependency
The set_enviroment tag, is still supported inside package tag sets, but
when defined at the xml root level, it will locate dependencies included in
the installed tool shed repository. Of course, the above
tool_dependencies.xml file could of course also include entries for tool
dependencies that are packages, For example:
?xml version=1.0?
tool_dependency
set_environment version=1.0
environment_variable name=R_SCRIPT_PATH
action=set_to$REPOSITORY_INSTALL_DIR/environment_variable
/set_environment
package name=R version=2.15.1
install version=1.0
actions
action
type=download_by_urlhttp://CRAN.R-project.org/src/base/R-2/R-2.15.1.tar.gz/action
action type=shell_command./configure
--prefix=$INSTALL_DIR/action
action type=shell_commandmake/action
action type=set_environment
environment_variable name=PATH
action=prepend_to$INSTALL_DIR/bin/environment_variable
/action
/actions
/install
readme
You need a FORTRAN compiler or perhaps f2c in addition to a C compiler to
build R.
/readme
/package
/tool_dependency
These tool dependency definitions are handled as described in the following
section of the tool shed wiki:
http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_third-party_tool_dependency_installation_and_compilation_with_installed_repositories
So in order for the dependencies to be handled when the repository is
installed, they must be defined in at least one of the requirement tag
sets in at least 1 tool config in the repository. So, the requirements
tag set in the deseq.xml tool config file will look something like this
(there are more requirement tags defined here than in the
tool_dependencies.xml fiel above, but you should get the idea).
requirements
requirement type=set_environmentR_SCRIPT_PATH/requirement
requirement type=package version=2.15.1R/requirement
requirement type=package
version=2.10Bioconductor/requirement
requirement type=package version=1.8.3DESeq/requirement
requirement type=package
version=1.24.0aroma.light/requirement
requirement type=package version=0.20-6lattice/requirement
/requirements
The command tag set in the deseq.xml tool config
Re: [galaxy-dev] environment variables and paths for toolshed tools
: Greg Von Kusterg...@bx.psu.edu
To: David Hooverhoove...@helix.nih.gov
Cc:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] environment variables and paths for toolshed
tools
Message-ID:f75d6320-3af8-4bb0-8f31-b0f44aa13...@bx.psu.edu
Content-Type: text/plain; charset=us-ascii
Hello David,
I've just committed change set 7656:6aadac8026cb to the Galaxy central
repository that provides the ability to do what you need here. It would be
great if you could try things out and let me know if you run into any problems.
I've been working with nikhil-joshi's deseq_and_sam2counts repository in the
main Galaxy tool shed because his tools require this new feature as well. This
example should provide you with the information you'll need to tweak your tool
shed repository so that your tool dependencies are located where they get
installed rather than attempting to move them to ${GALAXY_DATA_INDEX_DIR} or
some other location.
Here is the tool_dependency.xml file entry for locating a directory referred to
by an environment variable name R_SCRIPT_PATH.
?xml version=1.0?
tool_dependency
set_environment version=1.0
environment_variable name=R_SCRIPT_PATH
action=set_to$REPOSITORY_INSTALL_DIR/environment_variable
/set_environment
/tool_dependency
The set_enviroment tag, is still supported inside package tag sets, but
when defined at the xml root level, it will locate dependencies included in the installed
tool shed repository. Of course, the above tool_dependencies.xml file could of course also
include entries for tool dependencies that are packages, For example:
?xml version=1.0?
tool_dependency
set_environment version=1.0
environment_variable name=R_SCRIPT_PATH
action=set_to$REPOSITORY_INSTALL_DIR/environment_variable
/set_environment
package name=R version=2.15.1
install version=1.0
actions
action
type=download_by_urlhttp://CRAN.R-project.org/src/base/R-2/R-2.15.1.tar.gz/action
action type=shell_command./configure
--prefix=$INSTALL_DIR/action
action type=shell_commandmake/action
action type=set_environment
environment_variable name=PATH
action=prepend_to$INSTALL_DIR/bin/environment_variable
/action
/actions
/install
readme
You need a FORTRAN compiler or perhaps f2c in addition to a C compiler to build
R.
/readme
/package
/tool_dependency
These tool dependency definitions are handled as described in the following
section of the tool shed wiki:
http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_third-party_tool_dependency_installation_and_compilation_with_installed_repositories
So in order for the dependencies to be handled when the repository is installed, they must
be defined in at least one of the requirement tag sets in at least 1 tool config in
the repository. So, the requirements tag set in the deseq.xml tool config file
will look something like this (there are more requirement tags defined here than in the
tool_dependencies.xml fiel above, but you should get the idea).
requirements
requirement type=set_environmentR_SCRIPT_PATH/requirement
requirement type=package version=2.15.1R/requirement
requirement type=package version=2.10Bioconductor/requirement
requirement type=package version=1.8.3DESeq/requirement
requirement type=package version=1.24.0aroma.light/requirement
requirement type=package version=0.20-6lattice/requirement
/requirements
The command tag set in the deseq.xml tool config is also slightly altered.
Here is the way it looks before using this new approach:
command interpreter=python
stderr_wrapper.py Rscript ${GALAXY_DATA_INDEX_DIR}/deseq.R $counts $column_types
$comparison $top_table $diagnostic_html $diagnostic_html.files_path
$counts.name
/command
To use this new feature, the command string now uses the R_SCRIPT_PATH
environment variable ( notice the required backslash to escape the $ )
command interpreter=python
stderr_wrapper.py Rscript \$R_SCRIPT_PATH/deseq.R $counts $column_types $comparison $top_table
$diagnostic_html $diagnostic_html.files_path $counts.name
/command
When installed from the tool shed, a tool dependency object named R_SCRIPT_PATH
will be created and associated with the installed repository. The dependency
will have a pointer to the env.sh file that is created to set the value of the
R_SCRIPT_PATH environment variable.
Let me know if you bump into any issues in getting this working for your tools.
Thanks!
Greg Von Kuster
On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
Are there any environment variables that are honored by toolshed installed
tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or
$GALAXY_HOME, then uploaded it the test toolshed, then installed it
automatically. Neither of these resolved
Re: [galaxy-dev] environment variables and paths for toolshed tools
Greg, Pablo,
I'm using a SnpEffect repository 'snpeff_with_dep' in
http://testtoolshed.g2.bx.psu.edu/
to test using tool_dependencies.xml with the set_environment tag.
( You both write access, if you want to use this for correcting my errors or
debugging. )
I'm getting an error in prepare_for_install:
Modulegalaxy.web.controllers.admin_toolshed:*1174*in|prepare_for_install|
tool_dependencies||
|{'set_environment': [{'type': 'environment variable', 'name':
'JAVA_JAR_PATH'}], 'snpEff/3.0f': {'readme': '\n', 'version': '3.0f',
'type': 'package', 'name': 'snpEff'}}|
...
Module_admin_tool_shed_repository_common_mako:*242*in|render_render_tool_dependency_section|
|
http://localhost:18080/admin_toolshed/prepare_for_install?tool_shed_url=http://testtoolshed.g2.bx.psu.edu/repository_ids=d01ace51ab1282a5changeset_revisions=38b3a6b97bd1#name*=*requirements_dict*[*'name'*]*|
*TypeError: list indices must be integers, not str*
It appears that the requirements_dict for the set_environment tag is enclosed
in an array.
I haven't yet looked at the repository code to see how that might happen.
Pablo,
The http://sourceforge.net/projects/snpeff/files/ repository should probably
have versions packaged for download named like:
http://sourceforge.net/projects/snpeff/files/snpEff_3_0.tgz
$ tar ztvf snpEff_3_0.tgz
drwxrwxrwx 0 0 0 0 Aug 23 12:17 snpEff_3_0/
-rwxrwxrwx 0 0 014298767 Aug 23 12:17 snpEff_3_0/SnpSift.jar
...
That will allow versioning within galaxy, and successive versions of
tool_dependencies.xml with only need a change to the action tag: download_by_url
Thanks,
JJ
I'm trying to test a repository that use in tool_depen
Message: 18
Date: Thu, 13 Sep 2012 11:46:32 -0400
From: Greg Von Kusterg...@bx.psu.edu
To: David Hooverhoove...@helix.nih.gov
Cc:galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] environment variables and paths for toolshed
tools
Message-ID:f75d6320-3af8-4bb0-8f31-b0f44aa13...@bx.psu.edu
Content-Type: text/plain; charset=us-ascii
Hello David,
I've just committed change set 7656:6aadac8026cb to the Galaxy central
repository that provides the ability to do what you need here. It would be
great if you could try things out and let me know if you run into any problems.
I've been working with nikhil-joshi's deseq_and_sam2counts repository in the
main Galaxy tool shed because his tools require this new feature as well. This
example should provide you with the information you'll need to tweak your tool
shed repository so that your tool dependencies are located where they get
installed rather than attempting to move them to ${GALAXY_DATA_INDEX_DIR} or
some other location.
Here is the tool_dependency.xml file entry for locating a directory referred to
by an environment variable name R_SCRIPT_PATH.
?xml version=1.0?
tool_dependency
set_environment version=1.0
environment_variable name=R_SCRIPT_PATH
action=set_to$REPOSITORY_INSTALL_DIR/environment_variable
/set_environment
/tool_dependency
The set_enviroment tag, is still supported inside package tag sets, but
when defined at the xml root level, it will locate dependencies included in the installed
tool shed repository. Of course, the above tool_dependencies.xml file could of course also
include entries for tool dependencies that are packages, For example:
?xml version=1.0?
tool_dependency
set_environment version=1.0
environment_variable name=R_SCRIPT_PATH
action=set_to$REPOSITORY_INSTALL_DIR/environment_variable
/set_environment
package name=R version=2.15.1
install version=1.0
actions
action
type=download_by_urlhttp://CRAN.R-project.org/src/base/R-2/R-2.15.1.tar.gz/action
action type=shell_command./configure
--prefix=$INSTALL_DIR/action
action type=shell_commandmake/action
action type=set_environment
environment_variable name=PATH
action=prepend_to$INSTALL_DIR/bin/environment_variable
/action
/actions
/install
readme
You need a FORTRAN compiler or perhaps f2c in addition to a C compiler to build
R.
/readme
/package
/tool_dependency
These tool dependency definitions are handled as described in the following
section of the tool shed wiki:
http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_third-party_tool_dependency_installation_and_compilation_with_installed_repositories
So in order for the dependencies to be handled when the repository is installed, they must
be defined in at least one of the requirement tag sets in at least 1 tool config in
the repository. So, the requirements tag set in the deseq.xml tool config file
will look something like this (there are more requirement tags defined here than in the
tool_dependencies.xml fiel above, but you should get the idea).
requirements
requirement type
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hello David,
I've just committed change set 7656:6aadac8026cb to the Galaxy central
repository that provides the ability to do what you need here. It would be
great if you could try things out and let me know if you run into any problems.
I've been working with nikhil-joshi's deseq_and_sam2counts repository in the
main Galaxy tool shed because his tools require this new feature as well. This
example should provide you with the information you'll need to tweak your tool
shed repository so that your tool dependencies are located where they get
installed rather than attempting to move them to ${GALAXY_DATA_INDEX_DIR} or
some other location.
Here is the tool_dependency.xml file entry for locating a directory referred to
by an environment variable name R_SCRIPT_PATH.
?xml version=1.0?
tool_dependency
set_environment version=1.0
environment_variable name=R_SCRIPT_PATH
action=set_to$REPOSITORY_INSTALL_DIR/environment_variable
/set_environment
/tool_dependency
The set_enviroment tag, is still supported inside package tag sets, but
when defined at the xml root level, it will locate dependencies included in the
installed tool shed repository. Of course, the above tool_dependencies.xml
file could of course also include entries for tool dependencies that are
packages, For example:
?xml version=1.0?
tool_dependency
set_environment version=1.0
environment_variable name=R_SCRIPT_PATH
action=set_to$REPOSITORY_INSTALL_DIR/environment_variable
/set_environment
package name=R version=2.15.1
install version=1.0
actions
action
type=download_by_urlhttp://CRAN.R-project.org/src/base/R-2/R-2.15.1.tar.gz/action
action type=shell_command./configure
--prefix=$INSTALL_DIR/action
action type=shell_commandmake/action
action type=set_environment
environment_variable name=PATH
action=prepend_to$INSTALL_DIR/bin/environment_variable
/action
/actions
/install
readme
You need a FORTRAN compiler or perhaps f2c in addition to a C compiler to build
R.
/readme
/package
/tool_dependency
These tool dependency definitions are handled as described in the following
section of the tool shed wiki:
http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_third-party_tool_dependency_installation_and_compilation_with_installed_repositories
So in order for the dependencies to be handled when the repository is
installed, they must be defined in at least one of the requirement tag sets
in at least 1 tool config in the repository. So, the requirements tag set in
the deseq.xml tool config file will look something like this (there are more
requirement tags defined here than in the tool_dependencies.xml fiel above, but
you should get the idea).
requirements
requirement type=set_environmentR_SCRIPT_PATH/requirement
requirement type=package version=2.15.1R/requirement
requirement type=package version=2.10Bioconductor/requirement
requirement type=package version=1.8.3DESeq/requirement
requirement type=package version=1.24.0aroma.light/requirement
requirement type=package version=0.20-6lattice/requirement
/requirements
The command tag set in the deseq.xml tool config is also slightly altered.
Here is the way it looks before using this new approach:
command interpreter=python
stderr_wrapper.py Rscript ${GALAXY_DATA_INDEX_DIR}/deseq.R $counts
$column_types $comparison $top_table $diagnostic_html
$diagnostic_html.files_path $counts.name
/command
To use this new feature, the command string now uses the R_SCRIPT_PATH
environment variable ( notice the required backslash to escape the $ )
command interpreter=python
stderr_wrapper.py Rscript \$R_SCRIPT_PATH/deseq.R $counts $column_types
$comparison $top_table $diagnostic_html $diagnostic_html.files_path
$counts.name
/command
When installed from the tool shed, a tool dependency object named R_SCRIPT_PATH
will be created and associated with the installed repository. The dependency
will have a pointer to the env.sh file that is created to set the value of the
R_SCRIPT_PATH environment variable.
Let me know if you bump into any issues in getting this working for your tools.
Thanks!
Greg Von Kuster
On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
Are there any environment variables that are honored by toolshed installed
tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or
$GALAXY_HOME, then uploaded it the test toolshed, then installed it
automatically. Neither of these resolved to what I expected. I don't want
to hard-code the path in the xml tool file, but rather have a default
location for other executables and jar files. How should this best be done?
David Hoover
Helix Systems Staff
___
Please keep all replies on
[galaxy-dev] environment variables and paths for toolshed tools
Are there any environment variables that are honored by toolshed installed
tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or
$GALAXY_HOME, then uploaded it the test toolshed, then installed it
automatically. Neither of these resolved to what I expected. I don't want to
hard-code the path in the xml tool file, but rather have a default location for
other executables and jar files. How should this best be done?
David Hoover
Helix Systems Staff
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hello David,
This is not currently possible, but will be available in the next Galaxy
distribution release, currently scheduled for about 10 days from now. You will
include a tool_dependencies.xml file in your tool shed repository that looks
something like the following.
?xml version=1.0?
tool_dependency
set_environment version=1.0
environment_variable name=JAVA_JAR_PATH
action=set_to$INSTALL_DIR/environment_variable
/set_environment
/tool_dependency
Your tool will find the required files via the defined JAVA_JAR_PATH by using a
requirements tag set in the tool config, something like this:
requirements
requirement type=set_environmentJAVA_JAR_PATH /requirement
/requirements
The command tag in the tool config would be something like this:
command interpreter=python
python_wrapper.py $JAVA_JAR_PATH/some_file.jar $param1 $param2...
/command
I'm close to having this working, so if you are interested in testing before
the next Galaxy dist release, let me know and I'll tell you when the Galaxy
central repository has the new feature.
Thanks!
Greg Von Kuster
On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
Are there any environment variables that are honored by toolshed installed
tools? I tried creating a tool that uses ${GALAXY_DATA_INDEX_DIR} or
$GALAXY_HOME, then uploaded it the test toolshed, then installed it
automatically. Neither of these resolved to what I expected. I don't want
to hard-code the path in the xml tool file, but rather have a default
location for other executables and jar files. How should this best be done?
David Hoover
Helix Systems Staff
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] environment variables and paths for toolshed tools
Hi David,
With the exception of requiring versions strings in requirement tags in the
tool config in order to enable versioned tool dependency installation (which is
documented in the tool shed wiki at
http://wiki.g2.bx.psu.edu/Tool%20Shed#Automatic_third-party_tool_dependency_installation_and_compilation_with_installed_repositories),
this is the first change to the way Galaxy tool configs should be written.
I'll get a wiki put together for this new enhancement as soon as possible. I'm
hoping to have it as part of the next Galaxy dist release when the feature is
released.
Greg Von Kuster
On Sep 12, 2012, at 1:47 PM, David Hoover wrote:
On the topic of toolshed tools, is there any documented guidance about how to
create a tool for the toolshed? There is some syntactical help on the wiki,
but no comprehensive guide.
On Sep 12, 2012, at 1:41 PM, Greg Von Kuster wrote:
It'll be available in Galaxy central sooner than Galaxy dist. I'll send you
an email when I commit it.
On Sep 12, 2012, at 1:31 PM, David Hoover wrote:
Will it be available through galaxy-dist or galaxy-central? I have started
using the galaxy-central repository just to be on the bleeding edge.
On Sep 12, 2012, at 1:15 PM, Greg Von Kuster wrote:
Hello David,
This is not currently possible, but will be available in the next Galaxy
distribution release, currently scheduled for about 10 days from now. You
will include a tool_dependencies.xml file in your tool shed repository
that looks something like the following.
?xml version=1.0?
tool_dependency
set_environment version=1.0
environment_variable name=JAVA_JAR_PATH
action=set_to$INSTALL_DIR/environment_variable
/set_environment
/tool_dependency
Your tool will find the required files via the defined JAVA_JAR_PATH by
using a requirements tag set in the tool config, something like this:
requirements
requirement type=set_environmentJAVA_JAR_PATH /requirement
/requirements
The command tag in the tool config would be something like this:
command interpreter=python
python_wrapper.py $JAVA_JAR_PATH/some_file.jar $param1 $param2...
/command
I'm close to having this working, so if you are interested in testing
before the next Galaxy dist release, let me know and I'll tell you when
the Galaxy central repository has the new feature.
Thanks!
Greg Von Kuster
On Sep 12, 2012, at 1:00 PM, David Hoover wrote:
Are there any environment variables that are honored by toolshed
installed tools? I tried creating a tool that uses
${GALAXY_DATA_INDEX_DIR} or $GALAXY_HOME, then uploaded it the test
toolshed, then installed it automatically. Neither of these resolved to
what I expected. I don't want to hard-code the path in the xml tool
file, but rather have a default location for other executables and jar
files. How should this best be done?
David Hoover
Helix Systems Staff
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
