Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
Hi again, I've realised that my current Blast2GO install script would be a perfect example of where this tweak to fabric_util.py would help - when I wrote the tool_dependencies.xml I didn't appreciate that the download_by_url action was only allowed as the first action - currently it is ignored if listed later in the script (just as any unknown actions are currently ignored). http://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go/6e7694a0ae00 (I'm about to try and fix this tool_dependencies.xml file) Peter On Mon, May 13, 2013 at 10:34 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Could I suggest an enhancement to abort on undefined action types, rough patch below (untested)? This will help give a clear error if a new command is used but the Galaxy host is too old to implement it (which is exactly the situation we'd see right now on the main Tool Shed if the new download_file action is used). Regards, Peter $ hg diff diff -r 65a81aead95e lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py --- a/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py Sun May 12 11:52:55 2013 -0400 +++ b/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py Mon May 13 10:29:17 2013 +0100 @@ -152,6 +152,12 @@ url = action_dict[ 'url' ] filename = url.split( '/' )[ -1 ] common_util.url_download( current_dir, filename, url ) +else: +# This will happen if the tool_dependencies.xml is using a very new action: +tool_dependency.status = app.model.ToolDependency.installation_status.ERROR +tool_dependency.error_message = Undefined action type %r % action_type +return + def log_results( command, fabric_AttributeString, file_path ): ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
On Mon, May 13, 2013 at 8:35 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, May 13, 2013 at 11:42 AM, Peter Cock p.j.a.c...@googlemail.com wrote: ... This would explain the behaviour I am seeing with effectivet3 and other tools on the Tool Shed - despite the tool_dependencies.xml seeming to fail part way though, the tests are run anyway. (I would instead expect the Tool Shed to tell me the install failed, and not bother trying to run the tests, which are probably going to fail anyway). It looks like this is coming soon - including displaying something about the install failure reasons if I'm reading this correctly :) https://bitbucket.org/galaxy/galaxy-central/commits/ef2b705c7de3 I'm seeing the new install failures section on some tools now :) (I've just updated the tool concerned, so hopefully that install error will go away). Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
Hi Peter, I'm going to be enhancing this new feature further today, creating a new list Latest revision: installation errors, so this category will be extracted from the current failing tests category. At the same time I'll be adding the new feature allowing the owner of a repository to set the skip test flag. Greg Von Kuster On May 15, 2013, at 9:08 AM, Peter Cock wrote: On Mon, May 13, 2013 at 8:35 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Mon, May 13, 2013 at 11:42 AM, Peter Cock p.j.a.c...@googlemail.com wrote: ... This would explain the behaviour I am seeing with effectivet3 and other tools on the Tool Shed - despite the tool_dependencies.xml seeming to fail part way though, the tests are run anyway. (I would instead expect the Tool Shed to tell me the install failed, and not bother trying to run the tests, which are probably going to fail anyway). It looks like this is coming soon - including displaying something about the install failure reasons if I'm reading this correctly :) https://bitbucket.org/galaxy/galaxy-central/commits/ef2b705c7de3 I'm seeing the new install failures section on some tools now :) (I've just updated the tool concerned, so hopefully that install error will go away). Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
On Fri, May 10, 2013 at 8:11 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Dave, that was fast. Thanks a lot! Björn Thanks Dave, I see this also should fix handling of JAR files in download_by_url :) https://bitbucket.org/galaxy/galaxy-central/commits/b4733e42a2c98c42401bca5e0643fc4d4dcf2db6 Could I suggest an enhancement to abort on undefined action types, rough patch below (untested)? This will help give a clear error if a new command is used but the Galaxy host is too old to implement it (which is exactly the situation we'd see right now on the main Tool Shed if the new download_file action is used). Regards, Peter $ hg diff diff -r 65a81aead95e lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py --- a/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py Sun May 12 11:52:55 2013 -0400 +++ b/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py Mon May 13 10:29:17 2013 +0100 @@ -152,6 +152,12 @@ url = action_dict[ 'url' ] filename = url.split( '/' )[ -1 ] common_util.url_download( current_dir, filename, url ) +else: +# This will happen if the tool_dependencies.xml is using a very new action: +tool_dependency.status = app.model.ToolDependency.installation_status.ERROR +tool_dependency.error_message = Undefined action type %r % action_type +return + def log_results( command, fabric_AttributeString, file_path ): ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
On Mon, May 13, 2013 at 10:34 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Fri, May 10, 2013 at 8:11 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Dave, that was fast. Thanks a lot! Björn Thanks Dave, I see this also should fix handling of JAR files in download_by_url :) https://bitbucket.org/galaxy/galaxy-central/commits/b4733e42a2c98c42401bca5e0643fc4d4dcf2db6 Could I suggest an enhancement to abort on undefined action types, rough patch below (untested)? This will help give a clear error if a new command is used but the Galaxy host is too old to implement it (which is exactly the situation we'd see right now on the main Tool Shed if the new download_file action is used). Regards, Peter Another data point, with my old wget based tool_wrapper.xml file: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/b863c4bec840 From the test failure of the previous commit I suspected this, but this revision checks specifically for the 'module' subdirectory and confirms it is not being created: Effective T3 module folder not found: '/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpAzkY3r/effectiveT3/1.0.1/peterjc/effectivet3/b863c4bec840/module' I infer the 'install script' is failing silently after setting the environment variable but before or when creating the sub-directory $INSTALL_DIR/module, actions !-- Set environment variable so Python script knows where to look -- action type=set_environment environment_variable name=EFFECTIVET3 action=set_to$INSTALL_DIR/environment_variable /action !-- Main JAR file -- action type=shell_commandwget http://effectors.org/download/version/TTSS_GUI-1.0.1.jar/action action type=move_filesourceTTSS_GUI-1.0.1.jar/sourcedestination$INSTALL_DIR//destination/action !-- Three model JAR files -- action type=make_directory$INSTALL_DIR/module/action action type=shell_commandwget http://effectors.org/download/module/TTSS_ANIMAL-1.0.1.jar/action action type=move_filesourceTTSS_ANIMAL-1.0.1.jar/sourcedestination$INSTALL_DIR/module//destination/action action type=shell_commandwget http://effectors.org/download/module/TTSS_PLANT-1.0.1.jar/action action type=move_filesourceTTSS_PLANT-1.0.1.jar/sourcedestination$INSTALL_DIR/module//destination/action action type=shell_commandwget http://effectors.org/download/module/TTSS_STD-1.0.1.jar/action action type=move_filesourceTTSS_STD-1.0.1.jar/sourcedestination$INSTALL_DIR/module//destination/action /actions The first possible cause of failure is the set_environment action, but from the limited information from the failing test that worked. The second possible causes of failure is the wget call, perhaps that command line tool isn't even installed? However, from the limited information reported via the failing test I know the JAR file was downloaded (and moved into place). The third possible failure is the move_file action, and again I know from the limited information reported via the failing test that the TTSS_GUI-1.0.1.jar file was moved into place. My suspicion therefore falls on the make_directory action - my debug output from the failing test indicates the expected directory was not created. However, any exception within fabric_util.install_and_build_package should be caught by install_util.install_and_build_package_via_fabric and recorded in the tool_dependency object. Tracing up the call tree, I found this puzzling bit in the common_install_util.py file: https://bitbucket.org/galaxy/galaxy-central/src/65a81aead95e147f709ee3969d49766f35d6a2e2/lib/tool_shed/util/common_install_util.py?at=default def handle_tool_dependencies( app, tool_shed_repository, tool_dependencies_config, tool_dependencies ): ... if tool_dependency.can_install: try: tool_dependency = install_package( app, elem, tool_shed_repository, tool_dependencies=tool_dependencies ) except Exception, e: error_message = Error installing tool dependency %s version %s: %s % ( str( package_name ), str( package_version ), str( e ) ) log.debug( error_message ) if tool_dependency: tool_dependency.status = app.model.ToolDependency.installation_status.ERROR tool_dependency.error_message = error_message sa_session.add( tool_dependency ) sa_session.flush() if tool_dependency and tool_dependency.status in [ app.model.ToolDependency.installation_status.INSTALLED,
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
On Thu, May 9, 2013 at 4:39 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, For defining the 'install script' for some of my tool wrappers, I have mainly following http://wiki.galaxyproject.org/ToolShedToolFeatures and looking at one or two examples. I've got some of my tools to work, but not all. For instance, this one appears not to be downloading or moving JAR files correctly: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/5644c28cf965 ... Here my Python wrapper script has evidently read the $EFFECTIVET3 environment variable (defined in the tool_dependencies.xml to point to the tool's $INSTALL_DIR), but the JAR file isn't there: Effective T3 JAR file not found: /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpYR9FXD/effectiveT3/1.0.1/peterjc/effectivet3/5644c28cf965/TTSS_GUI-1.0.1.jar My previous revision set the $EFFECTIVET3 environment variable at the end of the tool_dependencies.xml file, and from the test failure it appeared not to be getting set. My hunch is something breaks part way through the installation but the Tool Shed isn't reporting this - or am I just not looking in the right place? It would make sense to show any installation issues under the Tool Dependencies entry on the main tool page. Related to this, I'd like to suggest another couple of assertion action types, used for checking if directories or files exist, which would abort the installation with a clear error if the check fails: action type=assert_dir_exists$INSTALL_DIR/module//action action type=assert_file_exists$INSTALL_DIR/module/TTSS_STD-1.0.1.jar/action An obvious application of these would be immediately after an action type=download_by_url line to sanity test the download (and decompression) worked as expected - and didn't for example just download an HTML error page for instance. (In this case I am worried that perhaps the download_by_url action has been overly enthusiastic and unzipped the JAR file, perhaps based on the mime-type?). Thanks, Peter Thanks for the input Björn and Greg about not being able to make multiple calls to download_by_url - I've tried tweaking both the effectivet3 and clinod tool_dependencies.xml files to use wget but they are still failing: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/c70df461aae5 http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/20e44218fa63 I believe that either I have a simple error in both install descriptions, leading to the install to 'work' but not put things where I expect them (which is my problem), or the install is really failing yet there is no feedback about this (which is a Galaxy Tool Shed problem). The idea of my new action types for checking if files and folders exist is to catch some types of install failure as early as possible - in this case verify the install put the relevant files where it was intended to. Could you direct me at the source code which handles running the tool_dependencies.xm install scripts? I'd like to look at what kind of error handling it does (e.g. I'd hope non-zero return codes from a shell_command action would be treated as an error), and how easy it would be to add the new actions myself. Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
Hi Peter, For defining the 'install script' for some of my tool wrappers, I have mainly following http://wiki.galaxyproject.org/ToolShedToolFeatures and looking at one or two examples. I've got some of my tools to work, but not all. For instance, this one appears not to be downloading or moving JAR files correctly: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/5644c28cf965 ... Here my Python wrapper script has evidently read the $EFFECTIVET3 environment variable (defined in the tool_dependencies.xml to point to the tool's $INSTALL_DIR), but the JAR file isn't there: Effective T3 JAR file not found: /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpYR9FXD/effectiveT3/1.0.1/peterjc/effectivet3/5644c28cf965/TTSS_GUI-1.0.1.jar My previous revision set the $EFFECTIVET3 environment variable at the end of the tool_dependencies.xml file, and from the test failure it appeared not to be getting set. My hunch is something breaks part way through the installation but the Tool Shed isn't reporting this - or am I just not looking in the right place? It would make sense to show any installation issues under the Tool Dependencies entry on the main tool page. Related to this, I'd like to suggest another couple of assertion action types, used for checking if directories or files exist, which would abort the installation with a clear error if the check fails: action type=assert_dir_exists$INSTALL_DIR/module//action action type=assert_file_exists$INSTALL_DIR/module/TTSS_STD-1.0.1.jar/action An obvious application of these would be immediately after an action type=download_by_url line to sanity test the download (and decompression) worked as expected - and didn't for example just download an HTML error page for instance. (In this case I am worried that perhaps the download_by_url action has been overly enthusiastic and unzipped the JAR file, perhaps based on the mime-type?). Thanks, Peter Thanks for the input Björn and Greg about not being able to make multiple calls to download_by_url - I've tried tweaking both the effectivet3 and clinod tool_dependencies.xml files to use wget but they are still failing: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/c70df461aae5 http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/20e44218fa63 I'm not sure, but can you try to fetch the first jar file with download_by_url. The last time I was struggling with that the download_by_url was required, if I remember correctly. I believe that either I have a simple error in both install descriptions, leading to the install to 'work' but not put things where I expect them (which is my problem), or the install is really failing yet there is no feedback about this (which is a Galaxy Tool Shed problem). The idea of my new action types for checking if files and folders exist is to catch some types of install failure as early as possible - in this case verify the install put the relevant files where it was intended to. Could you direct me at the source code which handles running the tool_dependencies.xm install scripts? I'd like to look at what kind of error handling it does (e.g. I'd hope non-zero return codes from a shell_command action would be treated as an error), and how easy it would be to add the new actions myself. Have a look at: lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py Hope that helps! Björn Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
Björn, Peter, I've created a trello card (https://trello.com/c/QY0Z7L7p) for this requirement, and will be working on enhancing that feature as my next priority. --Dave B. On 5/10/13 09:59:37.000, Björn Grüning wrote: Hi Peter, For defining the 'install script' for some of my tool wrappers, I have mainly following http://wiki.galaxyproject.org/ToolShedToolFeatures and looking at one or two examples. I've got some of my tools to work, but not all. For instance, this one appears not to be downloading or moving JAR files correctly: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/5644c28cf965 ... Here my Python wrapper script has evidently read the $EFFECTIVET3 environment variable (defined in the tool_dependencies.xml to point to the tool's $INSTALL_DIR), but the JAR file isn't there: Effective T3 JAR file not found: /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpYR9FXD/effectiveT3/1.0.1/peterjc/effectivet3/5644c28cf965/TTSS_GUI-1.0.1.jar My previous revision set the $EFFECTIVET3 environment variable at the end of the tool_dependencies.xml file, and from the test failure it appeared not to be getting set. My hunch is something breaks part way through the installation but the Tool Shed isn't reporting this - or am I just not looking in the right place? It would make sense to show any installation issues under the Tool Dependencies entry on the main tool page. Related to this, I'd like to suggest another couple of assertion action types, used for checking if directories or files exist, which would abort the installation with a clear error if the check fails: action type=assert_dir_exists$INSTALL_DIR/module//action action type=assert_file_exists$INSTALL_DIR/module/TTSS_STD-1.0.1.jar/action An obvious application of these would be immediately after an action type=download_by_url line to sanity test the download (and decompression) worked as expected - and didn't for example just download an HTML error page for instance. (In this case I am worried that perhaps the download_by_url action has been overly enthusiastic and unzipped the JAR file, perhaps based on the mime-type?). Thanks, Peter Thanks for the input Björn and Greg about not being able to make multiple calls to download_by_url - I've tried tweaking both the effectivet3 and clinod tool_dependencies.xml files to use wget but they are still failing: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/c70df461aae5 http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/20e44218fa63 I'm not sure, but can you try to fetch the first jar file with download_by_url. The last time I was struggling with that the download_by_url was required, if I remember correctly. I believe that either I have a simple error in both install descriptions, leading to the install to 'work' but not put things where I expect them (which is my problem), or the install is really failing yet there is no feedback about this (which is a Galaxy Tool Shed problem). The idea of my new action types for checking if files and folders exist is to catch some types of install failure as early as possible - in this case verify the install put the relevant files where it was intended to. Could you direct me at the source code which handles running the tool_dependencies.xm install scripts? I'd like to look at what kind of error handling it does (e.g. I'd hope non-zero return codes from a shell_command action would be treated as an error), and how easy it would be to add the new actions myself. Have a look at: lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py Hope that helps! Björn Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
On Fri, May 10, 2013 at 2:59 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Peter, I believe that either I have a simple error in both install descriptions, leading to the install to 'work' but not put things where I expect them (which is my problem), or the install is really failing yet there is no feedback about this (which is a Galaxy Tool Shed problem). The idea of my new action types for checking if files and folders exist is to catch some types of install failure as early as possible - in this case verify the install put the relevant files where it was intended to. Could you direct me at the source code which handles running the tool_dependencies.xm install scripts? I'd like to look at what kind of error handling it does (e.g. I'd hope non-zero return codes from a shell_command action would be treated as an error), and how easy it would be to add the new actions myself. Have a look at: lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py Hope that helps! Björn That is educational, thanks Björn :) The good news is that the shell_command handling code does check the return code and record an error if it failed as part of the tool_dependency object (status and error_message). The bad news is that in general install_and_build_package doesn't seem to catch any other errors - for instance a failing download, corrupt zip file, invalid path name, etc. Further bad news, it looks like by guess about the JAR files wrongly being unzipped was accurate - digging down the imports, the common_util.iszip function seems to just be doing this: import zipfile zipfile.is_zipfile(/opt/clinod/clinod-1.3.jar) True Therefore (right now) as I feared, we cannot use the download_by_url action with JAR files because it will unzip them. As suggested earlier, one fix would be an option for controlling any decompression (defaulting to automatic as now, but allowing it to be explicitly set to true or false?). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
Peter, Yes, jar files are just a directory of .class (and other) files packaged with the zip algorithm, which has its benefits and drawbacks, as you've discovered. In the new action type I'm working on, I do intend to include the option to extract a downloaded file, or leave it as-is, which will allow the download of jar files needed at build time or run time, as well as any other compressed file that should not be automatically extracted. --Dave B. On 5/10/13 11:13:36.000, Peter Cock wrote: On Fri, May 10, 2013 at 2:59 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Peter, I believe that either I have a simple error in both install descriptions, leading to the install to 'work' but not put things where I expect them (which is my problem), or the install is really failing yet there is no feedback about this (which is a Galaxy Tool Shed problem). The idea of my new action types for checking if files and folders exist is to catch some types of install failure as early as possible - in this case verify the install put the relevant files where it was intended to. Could you direct me at the source code which handles running the tool_dependencies.xm install scripts? I'd like to look at what kind of error handling it does (e.g. I'd hope non-zero return codes from a shell_command action would be treated as an error), and how easy it would be to add the new actions myself. Have a look at: lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py Hope that helps! Björn That is educational, thanks Björn :) The good news is that the shell_command handling code does check the return code and record an error if it failed as part of the tool_dependency object (status and error_message). The bad news is that in general install_and_build_package doesn't seem to catch any other errors - for instance a failing download, corrupt zip file, invalid path name, etc. Further bad news, it looks like by guess about the JAR files wrongly being unzipped was accurate - digging down the imports, the common_util.iszip function seems to just be doing this: import zipfile zipfile.is_zipfile(/opt/clinod/clinod-1.3.jar) True Therefore (right now) as I feared, we cannot use the download_by_url action with JAR files because it will unzip them. As suggested earlier, one fix would be an option for controlling any decompression (defaulting to automatic as now, but allowing it to be explicitly set to true or false?). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
Peter, As of 9696:b4733e42a2c9, Galaxy now supports an additional tool dependency installation action called download_file, which downloads a file from the specified URL and saves it to the current working directory, without extracting it. The format is: action type=download_file[url]/action For your effectivet3 repository, the recommended tool_dependencies.xml now looks like: ?xml version=1.0? tool_dependency package name=effectiveT3 version=1.0.1 install version=1.0 actions !-- Download jar files required by this tool. -- action type=download_filehttp://effectors.org/download/version/TTSS_GUI-1.0.1.jar/action action type=move_filesourceTTSS_GUI-1.0.1.jar/sourcedestination$INSTALL_DIR//destination/action action type=make_directory$INSTALL_DIR/module/action action type=download_filehttp://effectors.org/download/module/TTSS_ANIMAL-1.0.1.jar/action action type=download_filehttp://effectors.org/download/module/TTSS_PLANT-1.0.1.jar/action action type=download_filehttp://effectors.org/download/module/TTSS_STD-1.0.1.jar/action action type=move_filesourceTTSS_PLANT-1.0.1.jar/sourcedestination$INSTALL_DIR/module//destination/action action type=move_filesourceTTSS_STD-1.0.1.jar/sourcedestination$INSTALL_DIR/module//destination/action action type=move_filesourceTTSS_ANIMAL-1.0.1.jar/sourcedestination$INSTALL_DIR/module//destination/action !-- Set environment variable so Python script knows where to look -- action type=set_environment environment_variable name=EFFECTIVET3 action=set_to$INSTALL_DIR/environment_variable /action /actions /install readme Downloads effectiveT3 v1.0.1 and the three models from http://effectors.org/ /readme /package /tool_dependency An enhancement I intend to make on Monday is the option to specify a target path and filename, which will eliminate the need for the move_file steps you see in the above XML. As this is a new feature, if you discover any bugs or anomalous behavior, please feel free to let me know, and I'll get that resolved as soon as I can. --Dave B. On 5/10/13 11:13:36.000, Peter Cock wrote: On Fri, May 10, 2013 at 2:59 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Peter, I believe that either I have a simple error in both install descriptions, leading to the install to 'work' but not put things where I expect them (which is my problem), or the install is really failing yet there is no feedback about this (which is a Galaxy Tool Shed problem). The idea of my new action types for checking if files and folders exist is to catch some types of install failure as early as possible - in this case verify the install put the relevant files where it was intended to. Could you direct me at the source code which handles running the tool_dependencies.xm install scripts? I'd like to look at what kind of error handling it does (e.g. I'd hope non-zero return codes from a shell_command action would be treated as an error), and how easy it would be to add the new actions myself. Have a look at: lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py Hope that helps! Björn That is educational, thanks Björn :) The good news is that the shell_command handling code does check the return code and record an error if it failed as part of the tool_dependency object (status and error_message). The bad news is that in general install_and_build_package doesn't seem to catch any other errors - for instance a failing download, corrupt zip file, invalid path name, etc. Further bad news, it looks like by guess about the JAR files wrongly being unzipped was accurate - digging down the imports, the common_util.iszip function seems to just be doing this: import zipfile zipfile.is_zipfile(/opt/clinod/clinod-1.3.jar) True Therefore (right now) as I feared, we cannot use the download_by_url action with JAR files because it will unzip them. As suggested earlier, one fix would be an option for controlling any decompression (defaulting to automatic as now, but allowing it to be explicitly set to true or false?). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy
[galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
Hi all, For defining the 'install script' for some of my tool wrappers, I have mainly following http://wiki.galaxyproject.org/ToolShedToolFeatures and looking at one or two examples. I've got some of my tools to work, but not all. For instance, this one appears not to be downloading or moving JAR files correctly: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/5644c28cf965 ?xml version=1.0? tool_dependency package name=effectiveT3 version=1.0.1 install version=1.0 actions !-- Set environment variable so Python script knows where to look -- action type=set_environment environment_variable name=EFFECTIVET3 action=set_to$INSTALL_DIR/environment_variable /action !-- Main JAR file -- action type=download_by_urlhttp://effectors.org/download/version/TTSS_GUI-1.0.1.jar/action action type=move_filesourceTTSS_GUI-1.0.1.jar/sourcedestination$INSTALL_DIR//destination/action !-- Three model JAR files -- action type=make_directory$INSTALL_DIR/module/action action type=download_by_urlhttp://effectors.org/download/module/TTSS_ANIMAL-1.0.1.jar/action action type=move_filesourceTTSS_ANIMAL-1.0.1.jar/sourcedestination$INSTALL_DIR/module//destination/action action type=download_by_urlhttp://effectors.org/download/module/TTSS_PLANT-1.0.1.jar/action action type=move_filesourceTTSS_PLANT-1.0.1.jar/sourcedestination$INSTALL_DIR/module//destination/action action type=download_by_urlhttp://effectors.org/download/module/TTSS_STD-1.0.1.jar/action action type=move_filesourceTTSS_STD-1.0.1.jar/sourcedestination$INSTALL_DIR/module//destination/action /actions /install readme Downloads effectiveT3 v1.0.1 and the three models from http://effectors.org/ /readme /package /tool_dependency Here my Python wrapper script has evidently read the $EFFECTIVET3 environment variable (defined in the tool_dependencies.xml to point to the tool's $INSTALL_DIR), but the JAR file isn't there: Effective T3 JAR file not found: /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpYR9FXD/effectiveT3/1.0.1/peterjc/effectivet3/5644c28cf965/TTSS_GUI-1.0.1.jar My previous revision set the $EFFECTIVET3 environment variable at the end of the tool_dependencies.xml file, and from the test failure it appeared not to be getting set. My hunch is something breaks part way through the installation but the Tool Shed isn't reporting this - or am I just not looking in the right place? It would make sense to show any installation issues under the Tool Dependencies entry on the main tool page. Related to this, I'd like to suggest another couple of assertion action types, used for checking if directories or files exist, which would abort the installation with a clear error if the check fails: action type=assert_dir_exists$INSTALL_DIR/module//action action type=assert_file_exists$INSTALL_DIR/module/TTSS_STD-1.0.1.jar/action An obvious application of these would be immediately after an action type=download_by_url line to sanity test the download (and decompression) worked as expected - and didn't for example just download an HTML error page for instance. (In this case I am worried that perhaps the download_by_url action has been overly enthusiastic and unzipped the JAR file, perhaps based on the mime-type?). Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
Hi Peter, Hi all, For defining the 'install script' for some of my tool wrappers, I have mainly following http://wiki.galaxyproject.org/ToolShedToolFeatures and looking at one or two examples. I've got some of my tools to work, but not all. For instance, this one appears not to be downloading or moving JAR files correctly: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/5644c28cf965 ?xml version=1.0? tool_dependency package name=effectiveT3 version=1.0.1 install version=1.0 actions !-- Set environment variable so Python script knows where to look -- action type=set_environment environment_variable name=EFFECTIVET3 action=set_to$INSTALL_DIR/environment_variable /action !-- Main JAR file -- action type=download_by_urlhttp://effectors.org/download/version/TTSS_GUI-1.0.1.jar/action action type=move_filesourceTTSS_GUI-1.0.1.jar/sourcedestination$INSTALL_DIR//destination/action !-- Three model JAR files -- action type=make_directory$INSTALL_DIR/module/action action type=download_by_urlhttp://effectors.org/download/module/TTSS_ANIMAL-1.0.1.jar/action action type=move_filesourceTTSS_ANIMAL-1.0.1.jar/sourcedestination$INSTALL_DIR/module//destination/action action type=download_by_urlhttp://effectors.org/download/module/TTSS_PLANT-1.0.1.jar/action action type=move_filesourceTTSS_PLANT-1.0.1.jar/sourcedestination$INSTALL_DIR/module//destination/action action type=download_by_urlhttp://effectors.org/download/module/TTSS_STD-1.0.1.jar/action action type=move_filesourceTTSS_STD-1.0.1.jar/sourcedestination$INSTALL_DIR/module//destination/action /actions /install readme Downloads effectiveT3 v1.0.1 and the three models from http://effectors.org/ /readme /package /tool_dependency Afaik download_by_url is only supported as first action. To work around that limitation, until its fixed, I used wget to fetch any additional file. Ciao, Björn Here my Python wrapper script has evidently read the $EFFECTIVET3 environment variable (defined in the tool_dependencies.xml to point to the tool's $INSTALL_DIR), but the JAR file isn't there: Effective T3 JAR file not found: /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpYR9FXD/effectiveT3/1.0.1/peterjc/effectivet3/5644c28cf965/TTSS_GUI-1.0.1.jar My previous revision set the $EFFECTIVET3 environment variable at the end of the tool_dependencies.xml file, and from the test failure it appeared not to be getting set. My hunch is something breaks part way through the installation but the Tool Shed isn't reporting this - or am I just not looking in the right place? It would make sense to show any installation issues under the Tool Dependencies entry on the main tool page. Related to this, I'd like to suggest another couple of assertion action types, used for checking if directories or files exist, which would abort the installation with a clear error if the check fails: action type=assert_dir_exists$INSTALL_DIR/module//action action type=assert_file_exists$INSTALL_DIR/module/TTSS_STD-1.0.1.jar/action An obvious application of these would be immediately after an action type=download_by_url line to sanity test the download (and decompression) worked as expected - and didn't for example just download an HTML error page for instance. (In this case I am worried that perhaps the download_by_url action has been overly enthusiastic and unzipped the JAR file, perhaps based on the mime-type?). Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
On Thu, May 9, 2013 at 4:57 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Peter, Afaik download_by_url is only supported as first action. To work around that limitation, until its fixed, I used wget to fetch any additional file. Ciao, Björn Interesting tip - I don't see that on the wiki but it sounds believable. This would resolve by concerns about how this altered the current working directory since (assuming I understand correctly) after decompressing a tar ball, it changes into that new directory. I'll try the wget alternative - thanks! Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
Currently I feel that a tool dependency definition should allow for only a singe download of some item, and if that item requires other items. they should be defined as separate dependencies that are installed using the prior_installation_required=True attribute. Is there a case where this is not possible? If so, I'm willing to enable downloading more than 1 item per dependency package definition, but I think we have to be careful in how we go about this. In any case, wget should not be used yet as it requires wget to be on the installation server, and we have not yet decided what 3rd party utilities we can assume exist in that environment. If there is justification for allowing download_by_url multiple times in the same package dependency definition, then I would rather enhance the tool shed framework to support that. Greg Von Kuster On May 9, 2013, at 12:15 PM, Peter Cock wrote: On Thu, May 9, 2013 at 4:57 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Peter, Afaik download_by_url is only supported as first action. To work around that limitation, until its fixed, I used wget to fetch any additional file. Ciao, Björn Interesting tip - I don't see that on the wiki but it sounds believable. This would resolve by concerns about how this altered the current working directory since (assuming I understand correctly) after decompressing a tar ball, it changes into that new directory. I'll try the wget alternative - thanks! Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
Hi Greg and Peter, two examples from my side. One of my tools (osra) has 7 dependencies. A few of them, GraphicsMagick for example, makes sense to include as separate repo. But I feel that the other are too special. - openbabel (is an extra repo) - GraphicsMagick - potrace - gocr - tclap - ocrad - cuneiform-linux Osra, on its own, needs a patch to be applied to install with my parameters. I used wget to fetch these patch. Maybe patch-action can be created to download and patch a package. But other corner cases will arise ;) Cheers, Bjoern Currently I feel that a tool dependency definition should allow for only a singe download of some item, and if that item requires other items. they should be defined as separate dependencies that are installed using the prior_installation_required=True attribute. Is there a case where this is not possible? If so, I'm willing to enable downloading more than 1 item per dependency package definition, but I think we have to be careful in how we go about this. In any case, wget should not be used yet as it requires wget to be on the installation server, and we have not yet decided what 3rd party utilities we can assume exist in that environment. If there is justification for allowing download_by_url multiple times in the same package dependency definition, then I would rather enhance the tool shed framework to support that. Greg Von Kuster On May 9, 2013, at 12:15 PM, Peter Cock wrote: On Thu, May 9, 2013 at 4:57 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Peter, Afaik download_by_url is only supported as first action. To work around that limitation, until its fixed, I used wget to fetch any additional file. Ciao, Björn Interesting tip - I don't see that on the wiki but it sounds believable. This would resolve by concerns about how this altered the current working directory since (assuming I understand correctly) after decompressing a tar ball, it changes into that new directory. I'll try the wget alternative - thanks! Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
On Thu, May 9, 2013 at 6:05 PM, Greg Von Kuster g...@bx.psu.edu wrote: Currently I feel that a tool dependency definition should allow for only a singe download of some item, and if that item requires other items. they should be defined as separate dependencies that are installed using the prior_installation_required=True attribute. That seems too idealised: Not every tool is bundled up nicely for a clean install, often there are bits and pieces which must be fetched individually. In some cases this is a judgement call about when this should be a separate dependency. Is there a case where this is not possible? If so, I'm willing to enable downloading more than 1 item per dependency package definition, but I think we have to be careful in how we go about this. Yes, effectiveT3 comes as four JAR files - the core tool and three models: http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3 This is an example of a fairly common pattern where there is a core tool, plus one or more essential data files to be downloaded individually. In any case, wget should not be used yet as it requires wget to be on the installation server, and we have not yet decided what 3rd party utilities we can assume exist in that environment. If there is justification for allowing download_by_url multiple times in the same package dependency definition, then I would rather enhance the tool shed framework to support that. If you won't want us to use wget, then we will likely also need a download_by_url variant which DOES NOT decompress anything. I'm thinking of compressed data files which should stay compressed. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Handling of tool_dependencies.xml errors in Tool Shed testing
On Thu, May 9, 2013 at 6:21 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Greg and Peter, two examples from my side. One of my tools (osra) has 7 dependencies. A few of them, GraphicsMagick for example, makes sense to include as separate repo. But I feel that the other are too special. I made the same judgement call with clinod (command line NoD), which needs a single binary tool 'batchman' (either on the path or next to the clinod install) from a 3rd party package, the Stuttgart Neural Network Simulator (SNNS). http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/ I could have tried to work out how to download and install the whole of the SNNS package, but that in turn would trigger more dependencies. Instead I am just installing batchman by putting it in the clinod $INSTALL_DIR - this is far simpler and less likely to break. (If it turns out in six months time that some other Galaxy Tools also use bits of SNNS, then I'd reconsider this choice.) Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
