On Tue, Jun 2, 2015 at 4:14 AM, Loraine Guéguen
<loraine.gueg...@sb-roscoff.fr> wrote:
> Hi,
>
> I have investigated the testing possibilities within a Galaxy local instance
> and within a toolshed. I met some difficulties and I have 4 questions listed
> below.
>
>
> 1- Tests on tools installed in a local Galaxy instance (with run_tests.sh) :
> It seems that it is only possible to test the tools that have been manually
> installed (run_test.sh -id ...), not the tools that have been installed
> through a toolshed. Is that true ? When I try to launch the tests of a
> toolshed installed tool, I have the following error message :
>
> #########
>
> Failure: ValueError (No such test TestForTool_sartools_edger) ... ERROR
> ======================================================================
> ERROR: Failure: ValueError (No such test TestForTool_sartools_edger)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File
> "/projet/htdocs/galaxy-dev/galaxy-dist/eggs/nose-0.11.1-py2.7.egg/nose/failure.py",
> line 39, in runTest
> raise self.exc_class(self.exc_val)
> ValueError: No such test TestForTool_sartools_edger
>
> ----------------------------------------------------------------------
> Ran 1 test in 0.002s
> FAILED (errors=1)
./run_tests.sh -installed runs the tests of all installed tool shed
tools. It might be possible to combine that with -id - for a specific
tool but I am not sure. It may be possible to construct that test even
if -id doesn't work - but it is probably pretty hacky. Let me know if
-installed works for you but not when combined with -id and we can
work out the details.
>
> #########
>
>
> 2- Tests on tools installed in a local Galaxy instance (with run_tests.sh) :
> Is it possible to test tools which have repository or tool dependencies ? I
> have a tool (manually installed in galaxy-dist/tools/) with a dependence to
> a datatype installed from a toolshed. The datatype (no_unzip.zip) is defined
> in the <param> input tag (in <test>) with attribute "ftype". When running
> run_tests.sh, it seems that the dataype is not known :
>
> #########
>
> ======================================================================
> ERROR: test_tool_000001 (functional.test_toolbox.TestForTool_sartools_edger)
> SARTools edgeR ( sartools_edger ) > Test-2
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File
> "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line
> 268, in test_tool
> self.do_it( td )
> File
> "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line
> 37, in do_it
> stage_data_in_history( galaxy_interactor, testdef.test_data(),
> test_history, shed_tool_id )
> File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line
> 36, in stage_data_in_history
> upload_waits.append( galaxy_interactor.stage_data_async( test_data,
> history, shed_tool_id ) )
> File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line
> 182, in stage_data_async
> raise Exception(submit_response)
> Exception: {u'message': {u'type': u'error', u'data': {u'file_type': u"An
> invalid option was selected for file_type, u'no_unzip.zip', please verify.",
> u'files_metadata': [u"An invalid option was selected for file_type,
> u'no_unzip.zip', please verify."]}}}
>
> #########
It is possible it is the case that tool shed datatypes are not loaded
unless running with the -installed flag. I would either setup the
tests to run locally (manually add datatypes and tools) or run
installed tools with -installed.
>
>
> 3- Toolshed Install and Test framework (
> install_and_test_tool_shed_repositories.sh) : For my tool, the test fails if
> I don't explicitely defined the dbkey attribute to "unknown" (dbkey="?") in
> the <param> input tag (in <test>). The default dbkey value seems to be
> "hg17" (whereas hg17 is not defined on the server running my toolshed). So I
> have the following error message if I don't add dbkey="?" in <test> :
>
> #########
>
> ======================================================================
> ERROR: test_tool_000001
> (functional.test_toolbox.TestForTool_galaxy4loraine:9009/repos/lgueguen/sartools_1_0_2/sartools_edger/20150209)
> SARTools edgeR (
> galaxy4loraine:9009/repos/lgueguen/sartools_1_0_2/sartools_edger/20150209 )
>> Test-2
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File
> "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line
> 268, in test_tool
> self.do_it( td )
> File
> "/w/galaxy/galaxy4loraine/galaxy-dist/test/functional/test_toolbox.py", line
> 37, in do_it
> stage_data_in_history( galaxy_interactor, testdef.test_data(),
> test_history, shed_tool_id )
> File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line
> 36, in stage_data_in_history
> upload_waits.append( galaxy_interactor.stage_data_async( test_data,
> history, shed_tool_id ) )
> File "/w/galaxy/galaxy4loraine/galaxy-dist/test/base/interactor.py", line
> 182, in stage_data_async
> raise Exception(submit_response)
> Exception: {u'message': {u'type': u'error', u'data': {u'dbkey': u"An invalid
> option was selected for dbkey, u'hg17', please verify."}}}
>
> #########
Well that is frustrating. I am not aware of the internals of that
script or why that is happening - but my personal recommendation would
be not to use the install and test framework at this point - it seems
rather un-maintained. I recently added a subcommnd to planemo called
shed_test
(http://planemo.readthedocs.org/en/master/commands.html#shed-test-command)
- that will install and test a tool shed repository. It probably
requires modifying your tool shed development process to use planemo
(http://planemo.readthedocs.org/en/latest/publishing.html#shed).
>
>
> 4- Is it possible to launch the tests against an external server ? That
> would be very useful to be able run the tests on a production Galaxy
> instance in order to test the tools in real condition, within the whole
> environment.
This would be great - but as I understand it - it is not possible at
this time. The test framework needs local access to some aspects of
the Galaxy app. Last year I outlined a Trello card to add this option
back to Galaxy (https://trello.com/c/8K86l4Qk) - it was available
years ago.
>
>
>
> Thanks in advance for your answers.
>
> Loraine
>
>
>
>
> --
> Loraine Guéguen
>
> Plateforme ABiMS (Analyses and Bioinformatics for Marine Sciences)
> Service Informatique et Bioinformatique
> Station Biologique de Roscoff
> FR 2424 CNRS UPMC
> Place Georges Teissier
> CS 90074
> 29688 Roscoff cedex
> tel: ++33 (0)2 98 29 56 46 (interne : 415)
> http://abims.sb-roscoff.fr/
> http://www.sb-roscoff.fr/
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
