[galaxy-dev] job_working_directory: no subdirectories created
Hi Galaxy-devs, I recently ran into odd behavior when jobs were submitted at the same time to our cluster (jobs wouldn't return correctly, e.g. ). I changed "cleanup_job" in galaxy.ini to "never" to see what was going on in my job_working_directory (/scratch/galaxy/job_working/). It looks like subdirectories are no longer being created in my parent job_working_directory (e.g. the tool_script.sh along with the outputs are always placed and executed in /scratch/galaxy/job_working). Isn't there usually a bunch of subdirectories created (e.g. 000/XXX, etc...) I suspect my issues arise because script execution is happening in the parent directory rather than in temporary ones that get created within the parent directory. Could this be the issue and if so does anyone know how this could have happened or how I can fix this? I've also noticed that my environment variables are pointing to the parent directory rather than a subdirectory: SGE_CWD_PATH=/scratch/galaxy/job_working As an example, the contents of /scratch/galaxy/job_working look like this (note that there are no usual subdirectories like 000/XXX, etc...): [galaxy@galaxy job_working]$ ls | more __instrument_core_epoch_end __instrument_core_epoch_start __instrument_core_galaxy_slots conda-env conda-metadata-env galaxy_20771.e galaxy_20771.ec galaxy_20771.o galaxy_20771.sh galaxy_20772.e galaxy_20772.ec galaxy_20772.o galaxy_20772.sh metadata_in_HistoryDatasetAssociation_46704_NHpqc0 metadata_in_HistoryDatasetAssociation_46705_nUT573 metadata_kwds_HistoryDatasetAssociation_46704_5IADnM metadata_kwds_HistoryDatasetAssociation_46705__z5cTO metadata_out_HistoryDatasetAssociation_46704_G8phbW metadata_out_HistoryDatasetAssociation_46705_qTOxoE metadata_override_HistoryDatasetAssociation_46704_v0f_Wl metadata_override_HistoryDatasetAssociation_46705_8pzLX_ metadata_results_HistoryDatasetAssociation_46704_fSFQsP metadata_results_HistoryDatasetAssociation_46705_r4vUOP set_metadata_2A6wuD.py set_metadata_2Dasxn.py tool_script.sh working Any help would be greatly appreciated!! ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] toolshed and testtoolshed
Hi Matthias, update of the MTS is triggerd. It can take a while, so many tools. In fact galaxyp uses the same mechanism as IUC, but the openms-tools are so many that the TS upload was cancelled by an Travis timeout. Sorry for the inconvenience. Tomorrow everything will be updated, Bjoern On 23.08.2017 15:28, Dave Bouvier wrote: > Matthias, > > - There is a regular sync for repositories owned or maintained by the > IUC, but I don't think galaxyp has one. Your best bet would be to ask > Björn (CC'd here) to update it in the main toolshed. > - No, the hashes are specific to the toolshed, so unfortunately that > solution is not a solution. > - You guess correctly, the test toolshed is not recommended to rely on > for tools. > > - > Dave Bouvier > http://galaxyproject.org > http://usegalaxy.org > > On 08/23/2017 09:20 AM, Matthias Bernt wrote: >> Dear dev-list, >> >> Bjoern Gruening was so nice to provide a fix for one of the galaxy >> tools (openms_metaprosip). It seems that the fix found its way into >> the testtoolshed, but not the toolshed. >> >> So I was curious how the connection between these two tool sources works. >> >> - Is there a regular sync interval? Or does someone need to push a >> button to get a new version to the toolshed? >> - Are the versions (and the hashes) of the tools that find their way >> to the main toolshed identical? Then I could just switch the >> tool_shed_url in my tool_list.yml file when the desired version or a >> newer one appears in the main toolshed. >> - I guess its not a good idea (in general) to provide tools from the >> testtoolshed in a prodiction instance? >> >> Best, >> Matthias >> >> ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] Conda dependency install problems for DESeq2 tool
Hello Dave Thanks for the quick response, that seems to have fixed it - the r-getopt conda package installs without a problem and the readline.so.6/missing symbol error has gone away. (DESeq2 is failing with an R error now, but I suspect that's a separate problem.) Thanks again, Best wishes Peter On 23/08/17 15:28, Dave Bouvier wrote: Peter, We recently resolved a similar issue on usegalaxy.org, part of which involved rebuilding some of deseq2's dependencies for R version 3.3.1, which is the currently preferred version for bioconda R packages. You should be able to get the right versions and builds of deseq2 and its dependencies with the following command: conda_/bin/conda create --name __bioconductor-deseq2@1.14.1 --override-channels -c iuc -c bioconda -c conda-forge -c defaults -c r bioconductor-deseq2=1.14.1 r-base=3.3.1 This will force installation of R 3.3.1 bioconda builds of r-getopt and other dependencies, which were rebuilt with conda-build > 2.0 a few months ago. - Dave Bouvier http://galaxyproject.org http://usegalaxy.org On 08/23/2017 08:38 AM, Peter Briggs wrote: Dear devs I'm experiencing problems with the conda dependencies for the DESeq2 tool (https://toolshed.g2.bx.psu.edu/repository?repository_id=1f158f7565dc70f9) installed into our production instance. When this tool is run the only output is: /PATH/TO/job_working_directory/025/25029/conda-env/lib/R/bin/exec/R: symbol lookup error: /PATH/TO/job_working_directory/025/25029/conda-env/lib/R/bin /exec/../../lib/../../libreadline.so.6: undefined symbol: PC I've tried manually reinstalling the two dependencies (bioconductor-deseq2@1.14.2 and r-getopt@1.20.0) by moving the relevant directories from 'tool_dependencies/_conda/envs' and then doing e.g. conda_/bin/conda create -n __r-getopt@1.20.0 --override-channels -c iuc -c bioconda -c conda-forge -c defaults -c r r-getopt=1.20.0 to see if this fixes the problem. However while it completes okay for the bioconductor-deseq2 package, for r-getopt I get the following failure: PaddingError: Placeholder of length '80' too short in package r::r-base-3.2.2-0. The package must be rebuilt with conda-build > 2.0. I've tried doing 'conda_/bin/conda clean -pt' and retrying but I get the same error. Any help to fix this is greatly appreciated, many thanks Best wishes Peter Ps conda version is 4.2.13 and Galaxy is release_17.05; I'm trying to rebuild the conda environments manually because I couldn't see anything within the Galaxy UI which would let me do it there. -- Peter Briggs peter.bri...@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482 ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] Conda dependency install problems for DESeq2 tool
Peter, We recently resolved a similar issue on usegalaxy.org, part of which involved rebuilding some of deseq2's dependencies for R version 3.3.1, which is the currently preferred version for bioconda R packages. You should be able to get the right versions and builds of deseq2 and its dependencies with the following command: conda_/bin/conda create --name __bioconductor-deseq2@1.14.1 --override-channels -c iuc -c bioconda -c conda-forge -c defaults -c r bioconductor-deseq2=1.14.1 r-base=3.3.1 This will force installation of R 3.3.1 bioconda builds of r-getopt and other dependencies, which were rebuilt with conda-build > 2.0 a few months ago. - Dave Bouvier http://galaxyproject.org http://usegalaxy.org On 08/23/2017 08:38 AM, Peter Briggs wrote: Dear devs I'm experiencing problems with the conda dependencies for the DESeq2 tool (https://toolshed.g2.bx.psu.edu/repository?repository_id=1f158f7565dc70f9) installed into our production instance. When this tool is run the only output is: /PATH/TO/job_working_directory/025/25029/conda-env/lib/R/bin/exec/R: symbol lookup error: /PATH/TO/job_working_directory/025/25029/conda-env/lib/R/bin /exec/../../lib/../../libreadline.so.6: undefined symbol: PC I've tried manually reinstalling the two dependencies (bioconductor-deseq2@1.14.2 and r-getopt@1.20.0) by moving the relevant directories from 'tool_dependencies/_conda/envs' and then doing e.g. conda_/bin/conda create -n __r-getopt@1.20.0 --override-channels -c iuc -c bioconda -c conda-forge -c defaults -c r r-getopt=1.20.0 to see if this fixes the problem. However while it completes okay for the bioconductor-deseq2 package, for r-getopt I get the following failure: PaddingError: Placeholder of length '80' too short in package r::r-base-3.2.2-0. The package must be rebuilt with conda-build > 2.0. I've tried doing 'conda_/bin/conda clean -pt' and retrying but I get the same error. Any help to fix this is greatly appreciated, many thanks Best wishes Peter Ps conda version is 4.2.13 and Galaxy is release_17.05; I'm trying to rebuild the conda environments manually because I couldn't see anything within the Galaxy UI which would let me do it there. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] toolshed and testtoolshed
Matthias, - There is a regular sync for repositories owned or maintained by the IUC, but I don't think galaxyp has one. Your best bet would be to ask Björn (CC'd here) to update it in the main toolshed. - No, the hashes are specific to the toolshed, so unfortunately that solution is not a solution. - You guess correctly, the test toolshed is not recommended to rely on for tools. - Dave Bouvier http://galaxyproject.org http://usegalaxy.org On 08/23/2017 09:20 AM, Matthias Bernt wrote: Dear dev-list, Bjoern Gruening was so nice to provide a fix for one of the galaxy tools (openms_metaprosip). It seems that the fix found its way into the testtoolshed, but not the toolshed. So I was curious how the connection between these two tool sources works. - Is there a regular sync interval? Or does someone need to push a button to get a new version to the toolshed? - Are the versions (and the hashes) of the tools that find their way to the main toolshed identical? Then I could just switch the tool_shed_url in my tool_list.yml file when the desired version or a newer one appears in the main toolshed. - I guess its not a good idea (in general) to provide tools from the testtoolshed in a prodiction instance? Best, Matthias ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] toolshed and testtoolshed
Dear dev-list, Bjoern Gruening was so nice to provide a fix for one of the galaxy tools (openms_metaprosip). It seems that the fix found its way into the testtoolshed, but not the toolshed. So I was curious how the connection between these two tool sources works. - Is there a regular sync interval? Or does someone need to push a button to get a new version to the toolshed? - Are the versions (and the hashes) of the tools that find their way to the main toolshed identical? Then I could just switch the tool_shed_url in my tool_list.yml file when the desired version or a newer one appears in the main toolshed. - I guess its not a good idea (in general) to provide tools from the testtoolshed in a prodiction instance? Best, Matthias -- --- Matthias Bernt Bioinformatics Service Molekulare Systembiologie (MOLSYB) Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ Helmholtz Centre for Environmental Research GmbH - UFZ Permoserstraße 15, 04318 Leipzig, Germany Phone +49 341 235 482296, m.be...@ufz.de, www.ufz.de Sitz der Gesellschaft/Registered Office: Leipzig Registergericht/Registration Office: Amtsgericht Leipzig Handelsregister Nr./Trade Register Nr.: B 4703 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig Wilfried Kraus Wissenschaftlicher Geschäftsführer/Scientific Managing Director: Prof. Dr. Dr. h.c. Georg Teutsch Administrative Geschäftsführerin/ Administrative Managing Director: Prof. Dr. Heike Graßmann --- ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] Conda dependency install problems for DESeq2 tool
Dear devs I'm experiencing problems with the conda dependencies for the DESeq2 tool (https://toolshed.g2.bx.psu.edu/repository?repository_id=1f158f7565dc70f9) installed into our production instance. When this tool is run the only output is: /PATH/TO/job_working_directory/025/25029/conda-env/lib/R/bin/exec/R: symbol lookup error: /PATH/TO/job_working_directory/025/25029/conda-env/lib/R/bin /exec/../../lib/../../libreadline.so.6: undefined symbol: PC I've tried manually reinstalling the two dependencies (bioconductor-deseq2@1.14.2 and r-getopt@1.20.0) by moving the relevant directories from 'tool_dependencies/_conda/envs' and then doing e.g. conda_/bin/conda create -n __r-getopt@1.20.0 --override-channels -c iuc -c bioconda -c conda-forge -c defaults -c r r-getopt=1.20.0 to see if this fixes the problem. However while it completes okay for the bioconductor-deseq2 package, for r-getopt I get the following failure: PaddingError: Placeholder of length '80' too short in package r::r-base-3.2.2-0. The package must be rebuilt with conda-build > 2.0. I've tried doing 'conda_/bin/conda clean -pt' and retrying but I get the same error. Any help to fix this is greatly appreciated, many thanks Best wishes Peter Ps conda version is 4.2.13 and Galaxy is release_17.05; I'm trying to rebuild the conda environments manually because I couldn't see anything within the Galaxy UI which would let me do it there. -- Peter Briggs peter.bri...@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482 ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
