Thanks John, that's quite the detailed response. I agree with your reasoning, and I can see the progress in planemo -- the shed_test command is great and I'm using it to see why some of my tool tests fail. In this particular case there is nothing wrong with the framework though, as the planemo test will also fail if samtools is not in the path. I guess it's just because samtools is not installed on the machine that runs the testing framework. In the light of the proposed changes I understand if you don't want to touch this aspect anymore, and I'm happy to have samtools installed when doing the (planemo) testing!
Marius On 12 June 2015 at 17:37, John Chilton <jmchil...@gmail.com> wrote: > tl;dr - The devteam is planning to remove the display of those test > results from the tool shed UI and replace them with something better > hopefully by the end of 2015. > > Full Details: > > As Peter has documented very well over the last year, there are large > number of problems with the install and test framework right now - it > has been very consistently broken. I think the samtools problem > specifically has been reported before for instance. At a recent > devteam meeting - I think the devteam reached a reluctant consensus > that it is very important (testing installs and tools) but until such > a time that we feel we can devote the the engineering time to keep it > up and running consistently or that the framework improves to the > point where this engineering effort is minimal - it would be better to > just remove the display of these tests in the tool shed interface - > since they are making perfectly functional tools look broken. > > Planemo now implements a shed_test command > (http://planemo.readthedocs.org/en/latest/commands.html#shed-test-command) > that supports similar functionality to the install and test framework. > On the plus side it is much easier to run the tests through planemo > and the output is better and more flexible. It is not a complete > replacement right now however - since it doesn't hook into the tool > shed display at this point and isn't a service the Galaxy project is > providing. I think the team will do both of these - hopefully by > sometime this Fall - and then we will have a framework for installing > and testing tools that makes it trivial to run the tools from the > command-line or use within a tool shed display. I have created a > Planemo milestone to track some of these developments - > https://github.com/galaxyproject/planemo/milestones/Planemo-as-a-Service. > > This may make people unhappy - it certainly doesn't make me happy - > but a year worth of brokenness does need to be addressed. So I would > encourage everyone to please respond with comments if you feel > strongly one way or the other - maybe we can make a new thread for > that and a more formal announcement of this change. > > -John > > On Thu, Jun 11, 2015 at 9:43 AM, Marius van den Beek > <m.vandenb...@gmail.com> wrote: > > Hi list, > > > > I’ve been trying to write functional tests for a tool that produces BAM > > files as output. The tests run fine when using planemo locally, > > but not on the testtoolshed or on machines that do not have samtools > > installed. > > Apparently galaxy automatically tries to convert BAM to SAM to compare > > the output line by line, and so this fails. Surprisingly this is also > true > > when > > comparing by size. > > This is the error: > > > > Tool id: bowtieForSmallRNA > > Tool version: bowtieForSmallRNA > > Test: test_tool_000000 > > ( > functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/bowtieForSmallRNA/1.1.0 > ) > > Stderr: > > > > Traceback: > > Traceback (most recent call last): > > File > > > "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", > > line 270, in test_tool > > self.do_it( td ) > > File > > > "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py", > > line 67, in do_it > > raise e > > JobOutputsError: History item different than expected, difference (using > > diff): > > ( > > /tmp/shed_tools/ > testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/316124e85b8d/msp_sr_bowtie/test-data/output.bam > > v. > > > /tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpjxtteB/tmpQMbAfNoutput.bam > > ) > > Converting local (test-data) bam to sam failed: (cmd=[samtools view -h -o > > > "/tmp/buildslave/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/repositories_with_tools/tmp/tmpjxtteB/local_bam_converted_to_sam_8Ym0af.sam" > > "/tmp/shed_tools/ > testtoolshed.g2.bx.psu.edu/repos/drosofff/msp_sr_bowtie/316124e85b8d/msp_sr_bowtie/test-data/output.bam > "], > > stdout=[], stderr=[/bin/sh: 1: samtools: not found > > ]) > > > > Should samtools perhaps be added to the list of tools required for > running > > galaxy? > > > > Best, > > Marius > > > > > > ___________________________________________________________ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ >
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
