[galaxy-dev] Keeping Galaxy up to date
Hello, I try to keep our Galaxy instance up to date. Very often, some tools disappear and other do not work anymore. This breaks some users workflows. I may miss something. Is there a best update pratice to keep a Galaxy instance and the tool-sheds fully functionnal ? Regards, Thomas -- Thomas Bellembois, Network and System Administrator, IGFL (France) http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Keeping Galaxy up to date
Hi Thomas, We had this topic at the GCC this year, especially within the Galaxy Admins BoF. The truth is (please correct me anyone if this is not the case anymore!) that you are touching an unresolved point. Keeping track of tools, settings etc. across the versions is not given. People have their 'home-brew' solutions for this, e.g. keeping a (maybe virtual) machine in spare in order to invest several hours up to some days to test (with or without participation of the users) applicability to local constraints/histories/workflow/tools. The more testing is automated and the less users (or non-default pieces) the faster the update procedure can be. But there are also instances which did not receive updates for months or even years. On our side we will be facing reality within the next two weeks as we really need the update. Although preconditions are pretty good (few users, not that deep modifications, high grade of automation) we expect some issues. happy to have a virtualized environment... To wrap up my answer in a nutshell: no, there is no best practice guide and no migration tool, but every single contribution is warmly welcome :). It would be *very* interesting how updates are handled at Galaxy Main...? Cheers, Sebastian (very interested in further feedback) Hello, I try to keep our Galaxy instance up to date. Very often, some tools disappear and other do not work anymore. This breaks some users workflows. I may miss something. Is there a best update pratice to keep a Galaxy instance and the tool-sheds fully functionnal ? Regards, Thomas -- Thomas Bellembois, Network and System Administrator, IGFL (France) http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Sebastian Schaaf, M.Sc. Bioinformatics Faculty Coordinator NGS Infrastructure Chair of Biometry and Bioinformatics Department of Medical Informatics, Biometry and Epidemiology (IBE) University of Munich Marchioninistr. 15, K U1 (postal) Marchioninistr. 17, U 006 (office) D-81377 Munich (Germany) Tel: +49 89 2180-78178 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Keeping Galaxy up to date
Hi Sebastian, Thanks a lot for this detailed answer. I will think about a custom recipe and try to contribute to this topic. Thomas -- Thomas Bellembois, Network and System Administrator, IGFL (France) http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) On 09/09/2014 12:55, Sebastian Schaaf wrote: Hi Thomas, We had this topic at the GCC this year, especially within the Galaxy Admins BoF. The truth is (please correct me anyone if this is not the case anymore!) that you are touching an unresolved point. Keeping track of tools, settings etc. across the versions is not given. People have their 'home-brew' solutions for this, e.g. keeping a (maybe virtual) machine in spare in order to invest several hours up to some days to test (with or without participation of the users) applicability to local constraints/histories/workflow/tools. The more testing is automated and the less users (or non-default pieces) the faster the update procedure can be. But there are also instances which did not receive updates for months or even years. On our side we will be facing reality within the next two weeks as we really need the update. Although preconditions are pretty good (few users, not that deep modifications, high grade of automation) we expect some issues. happy to have a virtualized environment... To wrap up my answer in a nutshell: no, there is no best practice guide and no migration tool, but every single contribution is warmly welcome :). It would be *very* interesting how updates are handled at Galaxy Main...? Cheers, Sebastian (very interested in further feedback) Hello, I try to keep our Galaxy instance up to date. Very often, some tools disappear and other do not work anymore. This breaks some users workflows. I may miss something. Is there a best update pratice to keep a Galaxy instance and the tool-sheds fully functionnal ? Regards, Thomas -- Thomas Bellembois, Network and System Administrator, IGFL (France) http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Keeping Galaxy up to date
Hello Thomas, After public debate on Trello, there have been a couple tools that have indeed disappeared completely but nearly all tools that have gone away have been migrated to the tool shed and can be recovered with the migration scripts that are included with each new release (https://wiki.galaxyproject.org/MigratingToolsFromGalaxyDistribution). Are you running those scripts and still having those problems or were you not aware of these scripts? My understanding is that if you are running these scripts workflows and tool re-runs should continue to work. (Am I wrong about this?). If particular tools don't disappear but no longer work - this sounds like a bug and you should let the list or Trello know so that the devteam can attempt to fix the problem or at least be aware it exists. Hopefully at some point soon nearly all of the tools will be migrated out the core release and the devteam will provide some mechanism to quickly deploy complete say NGS packages all at once to new releases. Once everything is in the tool shed then hopefully these problems will go away. (There will be new problems of course - but hopefully updates will easier). -John On Tue, Sep 9, 2014 at 5:49 AM, Thomas Bellembois thomas.bellemb...@ens-lyon.fr wrote: Hello, I try to keep our Galaxy instance up to date. Very often, some tools disappear and other do not work anymore. This breaks some users workflows. I may miss something. Is there a best update pratice to keep a Galaxy instance and the tool-sheds fully functionnal ? Regards, Thomas -- Thomas Bellembois, Network and System Administrator, IGFL (France) http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] ToolShed test failure: NotFound: cannot find 'ucsc_display_sites' while searching for 'APP.config.ucsc_display_sites'
Hi all, I'm wondering why my samtools_depad repository tests have failed, and since I have not changed this recently presume this is due to a Galaxy change or general TestToolShed problem not specific to my tool: https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_depad Tests that failed Tool id: samtools_depad Tool version: samtools_depad Test: test_tool_00 (functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/samtools_depad/samtools_depad/0.0.1) Stderr: Traceback: Traceback (most recent call last): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 114, in test_tool self.do_it( td ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 35, in do_it self._verify_outputs( testdef, test_history, shed_tool_id, data_list, galaxy_interactor ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 75, in _verify_outputs galaxy_interactor.verify_output( history, output_data, output_testdef=output_testdef, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py, line 82, in verify_output self._verify_metadata( history_id, hid, attributes ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py, line 103, in _verify_metadata raise Exception( msg ) Exception: Dataset metadata verification for [file_ext] failed, expected [bam] but found [None]. Traceback (most recent call last): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/web/framework/decorators.py, line 243, in decorator rval = func( self, trans, *args, **kwargs) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/webapps/galaxy/api/history_contents.py, line 188, in show return self.__show_dataset( trans, id, **kwd ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/webapps/galaxy/api/history_contents.py, line 214, in __show_dataset hda_dict[ 'display_apps' ] = self.get_display_apps( trans, hda ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/web/base/controller.py, line 855, in get_display_apps for display_app in hda.get_display_applications( trans ).itervalues(): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/model/__init__.py, line 1754, in get_display_applications return self.datatype.get_display_applications_by_dataset( self, trans ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/datatypes/data.py, line 445, in get_display_applications_by_dataset value = value.filter_by_dataset( dataset, trans ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/datatypes/display_applications/application.py, line 200, in filter_by_dataset if link_value.filter_by_dataset( data, trans ): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/datatypes/display_applications/application.py, line 78, in filter_by_dataset if fill_template( filter_elem.text, context = context ) != filter_elem.get( 'value', 'True' ): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/util/template.py, line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg/Cheetah/Template.py, line 1004, in __str__ return getattr(self, mainMethName)() File cheetah_DynamicallyCompiledCheetahTemplate_1410263883_33_43576.py, line 82, in respond NotFound: cannot find 'ucsc_display_sites' while searching for 'APP.config.ucsc_display_sites' requests.packages.urllib3.connectionpool: DEBUG: GET /api/histories/993bad2fe35335db/contents/7fbe67cfae825002?key=edc04240db9605fb7edc7bab44d3404c HTTP/1.1 500 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 requests.packages.urllib3.connectionpool: DEBUG: GET /api/histories/993bad2fe35335db/contents/7fbe67cfae825002/provenance?key=edc04240db9605fb7edc7bab44d3404c HTTP/1.1 200 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 requests.packages.urllib3.connectionpool: DEBUG: GET /api/histories/993bad2fe35335db/contents/7fbe67cfae825002/provenance?key=edc04240db9605fb7edc7bab44d3404c HTTP/1.1 200 None Any thoughts? Thanks,
Re: [galaxy-dev] Output Excel file
Bjorn, that did the trick, thank you for the suggestion. Onwards! Patrick On Thu, Sep 4, 2014 at 2:41 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Patrick, a nice workaround is the following: Please create your file.xls file, but not in the path galaxy will provide, use a temporary path, for example in the current working dir in your wrapper. Than after your program has finished, just move the *.xls file to the galaxy output path. Done. Like this. myprogram.r -i $input -o foo.xls -p 4; mv foo.xls $output; Cheers, Bjoern Am 04.09.2014 um 23:27 schrieb Patrick Leyshock: Hello, I am trying to write a Galaxy tool that will output an Excel file. Currently the tool wrapper calls an R script, which uses the xlsx package to read and write to and from Excel files. After being invoked by Galaxy, the script is able to successfully read an input Excel file: suppressPackageStartupMessages(library(xlsx, quietly=TRUE)); raw.data - read.xlsx(commandArgs(trailingOnly=TRUE)[1]), sheetName=input_records); The script then does its work on the data just read in. Then, when the work is done, I'd like to output the results to an Excel file. Here's where I run into a problem. I first tried to output the result like this: write.xlsx(processed.data, file=commandArgs(trailingOnly=TRUE)[2]); but write.xlsx throws an error. Looks like .xls and .xlsx are the only legal file extensions for the write.xlsx() function. Inspecting commandArgs(trailingOnly=TRUE)[2] shows that Galaxy provided a filename extension of .dat. I tried a workaround using this: write.xlsx(processed.data, file=paste(commandArgs(trailingOnly=TRUE)[2]), .xls, sep=); The write.xlsx function no longer throws an error (since the file name supplied as a parameter has an acceptable file extension) but now Galaxy won't display the result in the History. If I look in Galaxy's database I see two files there corresponding to my tool's output. Supposing that Galaxy assigned the result the name dataset_87, then looking in the Galaxy database I see: dataset_87.dat dataset_87.dat.xls dataset_87.dat is empty but visible to Galaxy (and so displayed as an empty dataset in the History window). dataset_87.dat.xls has the results I want but isn't displayed by Galaxy in the History window. There might be R libraries that can write xls or xlsx files without requiring a .xls or .xlsx file extension. That's a path I'm willing to explore. That said, is there a configuration option I can set that'll let me continue to use write.xlsx()? I've been working from the suggestions made on this helpful thread: http://lists.bx.psu.edu/pipermail/galaxy-dev/2011-December/007807.html so the relevant entries in datatypes_conf.xml are: datatype extension=xls type=galaxy.datatypes.binary.Xls display_in_upload=true / datatype extension=xlsx type=galaxy.datatypes.binary.Xlsx display_in_upload=true / and the relevant additions to binary.py are: class Xls(Binary): '''Class describing an Excel 2003 (xls) file''' file_ext='xls' class Xlsx(Binary): '''Class describing an Excel 2007 (xlsx) file''' file_ext='xlsx' Any suggestions appreciated. Thanks, Patrick ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Keeping Galaxy up to date
Hello John, I had forgotten that these migration scripts exist. It should work better now. :) Regards, Thomas -- Thomas Bellembois, Network and System Administrator, IGFL (France) http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) On 09/09/2014 15:05, John Chilton wrote: Hello Thomas, After public debate on Trello, there have been a couple tools that have indeed disappeared completely but nearly all tools that have gone away have been migrated to the tool shed and can be recovered with the migration scripts that are included with each new release (https://wiki.galaxyproject.org/MigratingToolsFromGalaxyDistribution). Are you running those scripts and still having those problems or were you not aware of these scripts? My understanding is that if you are running these scripts workflows and tool re-runs should continue to work. (Am I wrong about this?). If particular tools don't disappear but no longer work - this sounds like a bug and you should let the list or Trello know so that the devteam can attempt to fix the problem or at least be aware it exists. Hopefully at some point soon nearly all of the tools will be migrated out the core release and the devteam will provide some mechanism to quickly deploy complete say NGS packages all at once to new releases. Once everything is in the tool shed then hopefully these problems will go away. (There will be new problems of course - but hopefully updates will easier). -John On Tue, Sep 9, 2014 at 5:49 AM, Thomas Bellembois thomas.bellemb...@ens-lyon.fr wrote: Hello, I try to keep our Galaxy instance up to date. Very often, some tools disappear and other do not work anymore. This breaks some users workflows. I may miss something. Is there a best update pratice to keep a Galaxy instance and the tool-sheds fully functionnal ? Regards, Thomas -- Thomas Bellembois, Network and System Administrator, IGFL (France) http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] How to escape % characters in tool parameters?
Hi folks, I'm working on a tool that queries a database based on user input, where this user input is a text parameter. The wildcard for the database is the percent (%) character. The user input, which is ultimately translated into the query, should be able to contain a % character. For example, one input field lets the user indicate which cancer cohort he or she is interested in, in a table of available cohorts, and the user should be able to give input like 'TCGA% to indicate all TCGA cohorts. The problem I'm having is that when % is given in the input field, it's translated (to X) by the time it reaches my tool's python script. I haven't been able to figure out the right way to escape the input so that % is not translated. So, how do I escape this? Thanks! Melissa ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] How to escape % characters in tool parameters?
Funny - I just exchanged an e-mail with someone about this tool this morning, the Galaxy-P dbbuilder tool escapes exactly this santizing. Here is the relevant portion: param name=url value= type=text label=URL (http, ftp) sanitizer valid add value=%/ /valid /sanitizer /param The full tool is on the tool shed. https://toolshed.g2.bx.psu.edu/repos/galaxyp/dbbuilder Hope this helps. -John On Tue, Sep 9, 2014 at 10:08 PM, Melissa Cline cl...@soe.ucsc.edu wrote: Hi folks, I'm working on a tool that queries a database based on user input, where this user input is a text parameter. The wildcard for the database is the percent (%) character. The user input, which is ultimately translated into the query, should be able to contain a % character. For example, one input field lets the user indicate which cancer cohort he or she is interested in, in a table of available cohorts, and the user should be able to give input like 'TCGA% to indicate all TCGA cohorts. The problem I'm having is that when % is given in the input field, it's translated (to X) by the time it reaches my tool's python script. I haven't been able to figure out the right way to escape the input so that % is not translated. So, how do I escape this? Thanks! Melissa ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] How to escape % characters in tool parameters?
Awesome! Thank you, John - I'd actually glanced at the sanitizer and wondered if it was what I needed, but having a working example makes all the difference! On Tue, Sep 9, 2014 at 7:26 PM, John Chilton jmchil...@gmail.com wrote: Funny - I just exchanged an e-mail with someone about this tool this morning, the Galaxy-P dbbuilder tool escapes exactly this santizing. Here is the relevant portion: param name=url value= type=text label=URL (http, ftp) sanitizer valid add value=%/ /valid /sanitizer /param The full tool is on the tool shed. https://toolshed.g2.bx.psu.edu/repos/galaxyp/dbbuilder Hope this helps. -John On Tue, Sep 9, 2014 at 10:08 PM, Melissa Cline cl...@soe.ucsc.edu wrote: Hi folks, I'm working on a tool that queries a database based on user input, where this user input is a text parameter. The wildcard for the database is the percent (%) character. The user input, which is ultimately translated into the query, should be able to contain a % character. For example, one input field lets the user indicate which cancer cohort he or she is interested in, in a table of available cohorts, and the user should be able to give input like 'TCGA% to indicate all TCGA cohorts. The problem I'm having is that when % is given in the input field, it's translated (to X) by the time it reaches my tool's python script. I haven't been able to figure out the right way to escape the input so that % is not translated. So, how do I escape this? Thanks! Melissa ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
