[galaxy-dev] ulimit problems after update
Hi, I updated to the latest galaxy distribution (after one year). And now every job fails with: /home/imba/solexa/.profile.sh: line 118: ulimit: virtual memory: cannot modify limit: Operation not permitted The limit is ridiculously high: ulimit -v 6000 Its just to prevent some badly programmed in house galaxy tools to crash the server. I think the problem happens after the set_metadata stage. Any advice? thank you very much, ido ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Jobs stuck in new state - Data Library datasets to blame?
did you check if the metadata on the input was set correctly/at all? This happens sometimes in our galaxy instance. Metadata is not set correctly, and jobs run after metadata is set by hand. No reupload necessary. best, ido On 06 Nov 2014, at 17:13, Lance Parsons lpars...@princeton.edu wrote: I'v run into this same issue again (just with some other Data Library datasets). This time, there are a few users involved with quite a few stuck jobs. Does anyone have any advice on pushing these jobs through? Maybe even a pointer to the relevant code? I'm running latest_2014.08.11. Thanks in advance. Lance Lance Parsons wrote: Thanks, that was the first thing I checked. However, restarting the handler didn't help. Downloading the offending data and re-uploading as a new data set and then rerunning using the new dataset as input did work. Also, all other jobs continued to run fine. Lance Kandalaft, Iyad wrote: I’ve had jobs get stuck in the new state when one of the handler servers crashes. If you have dedicated handlers, check to make sure they are still running. Restart the handler to see if the jobs get resumed automatically. Iyad Kandalaft From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Aaron Petkau Sent: Wednesday, October 01, 2014 5:32 PM To: Lance Parsons Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Jobs stuck in new state - Data Library datasets to blame? Are you attempting to upload datasets to a Data Library, and then copy to a history and run jobs on them right away? I've run into issues before where if I attempt to run a job on a dataset in a library before it is finished being uploaded and processed, then the job gets stuck in a queued state and never executes. Aaron On Wed, Oct 1, 2014 at 2:51 PM, Lance Parsons lpars...@princeton.edu wrote: Recently, I updated our Galaxy instance to use two processes (one for web, the other as a job handler). This has been working well, except in a few cases. I've noticed that a number of jobs get stuck in the new status. In a number of cases, I've resolved the issue by downloading and uploading one of the input files and rerunning the job using the newly uploaded file. In at least one of these cases, the offending input file was one that was copied from a Data Library. Can anyone point me to something to look for in the database, etc. that would cause a job to think a dataset was not ready for use as a job input? I'd very much like to fix these datasets since having to re-upload data libraries would be very tedious. Thanks in advance. -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University -- Lance Parsons - Scientific Programmer 134 Carl C. Icahn Laboratory Lewis-Sigler Institute for Integrative Genomics Princeton University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] A proposed modules extension for toolshed wrappers
Hi, has anything changed in galaxy in this regard? Any way to modify an environment before a tools is run? I now have a tool relying on R-devel and bioconductor devel, both of which I can load in a module. The tool comes from the toolshed with xml like: command interpreter=Rscript … I don't want to hack around in the tool itself, but simply load the necessary R-version. thank you very much, ido On Sep 13, 2013, at 3:23 AM, Guest, Simon simon.gu...@agresearch.co.nz wrote: Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions. Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool. We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load. So instead of the tool_dependencies.xml fragment: tool_dependency package name=bwa version=0.6.2 install version=1.0 actions action type=download_by_urlhttp://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2/action action type=shell_commandmake/action action type=move_file sourcebwa/source destination$INSTALL_DIR/bin/destination /action action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install /package /tool_dependency We have something like this (NB: element and attribute names are for illustrative purposes only): tool_dependency package name=bwa version=0.6.2 use_native actions action type=module_loadbwa/0.6.2/action /actions /use_native /package /tool_dependency This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one. We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example). What do you think? cheers, Simon PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-) === Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. === ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] A proposed modules extension for toolshed wrappers
Dear Björn, maybe I could change the tool, which is not mine, and which I don't want to maintain to use a specific R-version that is already available on our cluster and which I can put into my path with module load R/3.1.0-devel ( http://modules.sourceforge.net/) there even is a requirements in the wrapper which is not fulfilled (these versions are not available at the moment) and still it installed without problems. requirements requirement type=R-module version=3.5.27edgeR/requirement requirement type=R-module version=3.18.13limma/requirement /requirements This requirements tags looks also rather inflexible to me. With an additional level of user configurable indirection it would be possible to make the tools fit to different infrastructures without having to use binaries provided by somebody else, taking up space for just one tool etc... Currently for jobs that are not run locally there is a general environment_setup_file. There could be one optional environment_setup_file for every job (or destination). In the end I created yet another wrapper. best, ido On Apr 25, 2014, at 2:34 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Ido, I do not get your question in all detail, but it is possible to define a tool_dependencies.xml with a specific R version, + R libraries and use only that specific version from your tool with requirement tags. For an example please see various R tools from: https://github.com/bgruening/galaxytools Cheers, Bjoern Am 25.04.2014 14:25, schrieb Ido Tamir: Hi, has anything changed in galaxy in this regard? Any way to modify an environment before a tools is run? I now have a tool relying on R-devel and bioconductor devel, both of which I can load in a module. The tool comes from the toolshed with xml like: command interpreter=Rscript … I don't want to hack around in the tool itself, but simply load the necessary R-version. thank you very much, ido On Sep 13, 2013, at 3:23 AM, Guest, Simon simon.gu...@agresearch.co.nz wrote: Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions. Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool. We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load. So instead of the tool_dependencies.xml fragment: tool_dependency package name=bwa version=0.6.2 install version=1.0 actions action type=download_by_urlhttp://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2/action action type=shell_commandmake/action action type=move_file sourcebwa/source destination$INSTALL_DIR/bin/destination /action action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install /package /tool_dependency We have something like this (NB: element and attribute names are for illustrative purposes only): tool_dependency package name=bwa version=0.6.2 use_native actions action type=module_loadbwa/0.6.2/action /actions /use_native /package /tool_dependency This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with the existing download-and-make one. We might want a bit of flexibility on what actions are supported (in case we want to support Software Collections, for example). What do you think? cheers, Simon PS: In case it wasn't already clear, solving this problem well is quite important to us here at AgResearch. ;-) === Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify
Re: [galaxy-dev] A proposed modules extension for toolshed wrappers
On Apr 25, 2014, at 3:55 PM, John Chilton jmchil...@gmail.com wrote: Additionally (perhaps more pertinently), I have an open pull request (created just this week) to allow setting environment variables on a per destination basis - this is sort of what you are getting at and might be a better way to go - https://bitbucket.org/galaxy/galaxy-central/pull-request/378/allow-specification-of-environment. With this modification you could hack up the path to reflect the changes made by the module load. That is a bit of hack but would work for this case. That said I think you make a good point about the environment file, I will modify the new env tags to allow a file attribute and just source that file. This will give you your per destination environment files - hopefully this is satisfactory. That sounds great! best, ido On Fri, Apr 25, 2014 at 8:22 AM, Ido Tamir ta...@imp.ac.at wrote: Dear Björn, maybe I could change the tool, which is not mine, and which I don't want to maintain to use a specific R-version that is already available on our cluster and which I can put into my path with module load R/3.1.0-devel ( http://modules.sourceforge.net/) there even is a requirements in the wrapper which is not fulfilled (these versions are not available at the moment) and still it installed without problems. requirements requirement type=R-module version=3.5.27edgeR/requirement requirement type=R-module version=3.18.13limma/requirement /requirements This requirements tags looks also rather inflexible to me. With an additional level of user configurable indirection it would be possible to make the tools fit to different infrastructures without having to use binaries provided by somebody else, taking up space for just one tool etc... Currently for jobs that are not run locally there is a general environment_setup_file. There could be one optional environment_setup_file for every job (or destination). In the end I created yet another wrapper. best, ido On Apr 25, 2014, at 2:34 PM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Ido, I do not get your question in all detail, but it is possible to define a tool_dependencies.xml with a specific R version, + R libraries and use only that specific version from your tool with requirement tags. For an example please see various R tools from: https://github.com/bgruening/galaxytools Cheers, Bjoern Am 25.04.2014 14:25, schrieb Ido Tamir: Hi, has anything changed in galaxy in this regard? Any way to modify an environment before a tools is run? I now have a tool relying on R-devel and bioconductor devel, both of which I can load in a module. The tool comes from the toolshed with xml like: command interpreter=Rscript … I don't want to hack around in the tool itself, but simply load the necessary R-version. thank you very much, ido On Sep 13, 2013, at 3:23 AM, Guest, Simon simon.gu...@agresearch.co.nz wrote: Just been reading a bit more about the Galaxy packaging system. Here's a slight modification to what I was suggesting that might fit in a bit better. Apologies for not being more familiar with the existing system before proposing extensions. Recall that my goal is to support using a system-installed (native) package, at a defined version, which I aim to achieve by loading the appropriate environment module before running a tool. We still have tool_dependencies.xml defining a package at a particular version, but rather than download and build the source code, there's just a directive that says how to pick up the correct program version at runtime, e.g. which environment module to load. So instead of the tool_dependencies.xml fragment: tool_dependency package name=bwa version=0.6.2 install version=1.0 actions action type=download_by_urlhttp://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2/action action type=shell_commandmake/action action type=move_file sourcebwa/source destination$INSTALL_DIR/bin/destination /action action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install /package /tool_dependency We have something like this (NB: element and attribute names are for illustrative purposes only): tool_dependency package name=bwa version=0.6.2 use_native actions action type=module_loadbwa/0.6.2/action /actions /use_native /package /tool_dependency This causes the right thing (module load bwa/0.6.2) to be stuck into the dependencies env.sh file when this package is installed from the toolshed. We could call this toolshed tool native_package_bwa_0_6_2, to avoid
[galaxy-dev] select both chromosome size and mappable genome size
Hi, my tool needs both a file with chromosome lengths and the total mappable genome size. I would like to hardcode the mappable genome size per build somewhere. However the user should have to use only one drop down to select both. How can I achieve this? e.g. currently: param name=FAI type=select label=Select chromsome sizes options from_file=chrsize.loc column name=name index=0/ column name=value index=1/ /options /param with chrsize.loc: mm9 /groups/csf-ngs/wfsys/galaxy/tool-data/shared/ucsc/chrom/mm9.len mm10/groups/csf-ngs/wfsys/galaxy/tool-data/shared/ucsc/chrom/mm10.len hg19/groups/csf-ngs/wfsys/galaxy/tool-data/shared/ucsc/chrom/hg19.len dm3 /groups/csf-ngs/wfsys/galaxy/tool-data/shared/ucsc/chrom/dm3.len Currently I multiply the genome size from the fai file by 0.8 to get the mappable size within my tool, but I would like to use more exact numbers per build. thank you very much, ido ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] jobs stuck in new state
In my case it was uncompleted metadata in one of the input files. (but maybe it was not new state but something else?) HTH, ido On Mar 26, 2014, at 5:25 PM, David Hoover hoove...@helix.nih.gov wrote: I have many jobs stuck in the 'new' state on our local Galaxy instance. The jobs can't be stopped using the Admin-Manage jobs tool. First, does anyone know why a job would get stuck in the 'new' state for weeks? I have cleaned things up by manually setting their states to 'error' in the MySQL database. Is there a better way of dealing with 'new' jobs? BTW, our Galaxy instance was updated about two weeks ago. Wondering, David Hoover Helix Systems Staff ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] universe_wsgi.ini question for Cistrome
I have CEAS as a tool. I wrapped the standalone CEAS version. I tried adding a binary datatype for the annotation db but even with the binary datatype upload somehow did not work (IIRC galaxy tried to convert it to ascii). I then put the database as a selectable parameter in a dropdown list, because users anyway would not create and upload their own annotation db. best, ido On Feb 27, 2014, at 7:13 AM, ruiwang.sz ruiwang...@gmail.com wrote: Hi Guys, Are there anyone who is using Cistrome? I tried to merge Cistrome's setting into our own galaxy instance. A new entry is # Path to the static library files for assembly, ceaslib, chromLen, conservation, liftOver and MAT-lib # Then in other tool configuration xml file, we can use 'from galaxy import config' then '$config.Configuration().cistrome_static_library_path' # default would be in tool-data/ folder cistrome_static_library_path = /home/bioinfoadmin/app/cistrome-apps/cistrome_library However, I got error when I tried CEAS: Enrichment on chromosome and annotation: * Dataset 23: CEAS: Enrichment on chromosome and annotation on data 16 and data 13 The Galaxy framework encountered the following error while attempting to run the tool: Traceback (most recent call last): File /home/bioinfoadmin/app/galaxy-dist/lib/galaxy/jobs/runners/__init__.py, line 121, in prepare_job job_wrapper.prepare() File /home/bioinfoadmin/app/galaxy-dist/lib/galaxy/jobs/__init__.py, line 707, in prepare config_filenames = self.tool.build_config_files( param_dict, self.working_directory ) File /home/bioinfoadmin/app/galaxy-dist/lib/galaxy/tools/__init__.py, line 2609, in build_config_files f.write( fill_template( template_text, context=param_dict ) ) File /home/bioinfoadmin/app/galaxy-dist/lib/galaxy/util/template.py, line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File /home/bioinfoadmin/app/galaxy-dist/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg/Cheetah/Template.py, line 1004, in __str__ return getattr(self, mainMethName)() File cheetah_DynamicallyCompiledCheetahTemplate_1391037616_15_43258.py, line 274, in respond NotFound: cannot find 'cistrome_static_library_path' while searching for '__app__.config.cistrome_static_library_path' Tool execution generated the following error message: failure preparing job I'm quite confused since '__app__.config.cistrome_static_library_path' should be defined? or right now __app__ is dropped as Galaxy evolves? Besides, what does the comment mean by # Then in other tool configuration xml file, we can use 'from galaxy import config' then '$config.Configuration().cistrome_static_library_path' Does that mean that we should not be using __app__.config...but how to use this? I tried to insert 'from galaxy import config' into the xml file but only got error... I'll appreciate any input. Thanks, Rui ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Security vulnerability in Galaxy filtering tools
This seems to happen often e.g. http://wiki.galaxyproject.org/DevNewsBriefs/2012_10_23#Compute_Tool_Security_Fix a) are there general guidelines in the wiki on how to avoid these problems when creating tools? b) is there a way to check automatically if all input fields are correctly escaped in a tool? A search for security in the wiki brings up: • Admin/Data Libraries/Library Security 0.0k - rev: 1 (current) last modified: 2013-01-02 23:54:33 • Admin/DataLibraries/LibrarySecurity 19.2k - rev: 4 (current) last modified: 2013-01-03 00:12:36 • HelpOnConfiguration/SecurityPolicy 1.9k - rev: 1 (current) last modified: 0 • Learn/Security Features 7.0k - rev: 3 (current) last modified: 2011-09-13 16:52:08 • News/2013_04_08_Galaxy_Security_Release 1.3k - rev: 3 (current) last modified: 2013-04-08 16:56:41 escape does not bring up anything. thank you very much, ido On Nov 5, 2013, at 12:45 AM, Nate Coraor n...@bx.psu.edu wrote: A security vulnerability was recently discovered by John Chilton with Galaxy's Filter data on any column using simple expressions and Filter on ambiguities in polymorphism datasets tools that can allow for arbitrary execution of code on the command line. The fix for these tools has been committed to the Galaxy source. The timing of this commit coincides with the next Galaxy stable release (which has also been pushed out today). To apply the fix and simultaneously update to the new Galaxy stable release, ensure you are on the stable branch and upgrade to the latest changeset: % hg branch stable % hg pull -u For Galaxy installations that administrators are not yet ready to upgrade to the latest release, there are three workarounds. First, for Galaxy installations running on a relatively new version of the stable release (e.g. release_2013.08.12), Galaxy can be updated to the specific changeset that that contains the fix. This will include all of the stable (non-feature) commits that have been accumulated since the 8/12 release plus any new features included with (and prior to) the 8/12 release, but without all of the new features included in the 11/4 release. Ensure you are on the stable branch and then upgrade to the specific changeset: % hg pull -u -r e094c73fed4d Second, the patch can be downloaded and applied manually: % wget -o security.patch https://bitbucket.org/galaxy/galaxy-central/commits/e094c73fed4dc66b589932edb83412cb8b827cd3raw/ and then: % hg patch security.patch or: % patch -p1 security.patch Third, the tools can be completely disabled by removing them from the tool configuration file (by default, tool_conf.xml) and restarting all Galaxy server processes. The relevant lines in tool_conf.xml are: tool file=stats/dna_filtering.xml / tool file=stats/filtering.xml / The full 11/4 Galaxy Distribution News Brief will be available later today and will contain details of changes since the last release. --nate Galaxy Team ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Security vulnerability in Galaxy filtering tools
On Nov 5, 2013, at 6:28 PM, Nate Coraor n...@bx.psu.edu wrote: Hi Ido, Thanks for the feedback. Replies below. On Nov 5, 2013, at 9:54 AM, Ido Tamir wrote: This seems to happen often e.g. http://wiki.galaxyproject.org/DevNewsBriefs/2012_10_23#Compute_Tool_Security_Fix I'm not sure I'd agree that it's often - we've had 4 or 5 vulnerabilities over the life of the project. 2 allowed arbitrary code execution, the others were less severe. But these were written by experts, not by people like me, that don't know what the galaxy framework really does/does not do with the input, so I guess I make many more mistakes. a) are there general guidelines in the wiki on how to avoid these problems when creating tools? The guidelines for writing a Galaxy tool are no different from best practices for writing secure code. In specific for this vulnerability, execution of user input should be handled with extreme care, and this tool had some gaps in its input validation and sanitization. For what it's worth, the filter tool (on which the other vulnerable tools were based) is one of the few tools surviving from the very early days of Galaxy, and would not be implemented the same way if written today. I think it would be nice to have a small outline on the wiki of what galaxy does with the input data and how it could affect a tool. What sanitisation is there by default so I don't have to worry about it, but what could happen I if I don't care to check/remove sanitise ' | or ..., maybe with examples. b) is there a way to check automatically if all input fields are correctly escaped in a tool? I am not sure how Galaxy could do this. Galaxy sanitizes the command line so that input fields passed to a tool as command line arguments cannot be crafted to exploit the shell's parsing rules. Thats good best, ido What the tool itself does with its inputs are out of Galaxy's control. --nate A search for security in the wiki brings up: • Admin/Data Libraries/Library Security 0.0k - rev: 1 (current) last modified: 2013-01-02 23:54:33 • Admin/DataLibraries/LibrarySecurity 19.2k - rev: 4 (current) last modified: 2013-01-03 00:12:36 • HelpOnConfiguration/SecurityPolicy 1.9k - rev: 1 (current) last modified: 0 • Learn/Security Features 7.0k - rev: 3 (current) last modified: 2011-09-13 16:52:08 • News/2013_04_08_Galaxy_Security_Release 1.3k - rev: 3 (current) last modified: 2013-04-08 16:56:41 escape does not bring up anything. thank you very much, ido On Nov 5, 2013, at 12:45 AM, Nate Coraor n...@bx.psu.edu wrote: A security vulnerability was recently discovered by John Chilton with Galaxy's Filter data on any column using simple expressions and Filter on ambiguities in polymorphism datasets tools that can allow for arbitrary execution of code on the command line. The fix for these tools has been committed to the Galaxy source. The timing of this commit coincides with the next Galaxy stable release (which has also been pushed out today). To apply the fix and simultaneously update to the new Galaxy stable release, ensure you are on the stable branch and upgrade to the latest changeset: % hg branch stable % hg pull -u For Galaxy installations that administrators are not yet ready to upgrade to the latest release, there are three workarounds. First, for Galaxy installations running on a relatively new version of the stable release (e.g. release_2013.08.12), Galaxy can be updated to the specific changeset that that contains the fix. This will include all of the stable (non-feature) commits that have been accumulated since the 8/12 release plus any new features included with (and prior to) the 8/12 release, but without all of the new features included in the 11/4 release. Ensure you are on the stable branch and then upgrade to the specific changeset: % hg pull -u -r e094c73fed4d Second, the patch can be downloaded and applied manually: % wget -o security.patch https://bitbucket.org/galaxy/galaxy-central/commits/e094c73fed4dc66b589932edb83412cb8b827cd3raw/ and then: % hg patch security.patch or: % patch -p1 security.patch Third, the tools can be completely disabled by removing them from the tool configuration file (by default, tool_conf.xml) and restarting all Galaxy server processes. The relevant lines in tool_conf.xml are: tool file=stats/dna_filtering.xml / tool file=stats/filtering.xml / The full 11/4 Galaxy Distribution News Brief will be available later today and will contain details of changes since the last release. --nate Galaxy Team ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Yes this tar is broken at least on OSX. Other people have the same issue: http://code.google.com/p/libarchive/issues/detail?id=299 On Sep 20, 2013, at 10:41 AM, Bjoern Gruening bjoern.gruen...@gmail.com wrote: Hi Carlos, Hi Peter and Carlos, On Mon, Sep 16, 2013 at 8:57 PM, Carlos Borroto carlos.borr...@gmail.com wrote: I did an extra test. Started with a clean 'galaxy-dist'. This time both repositories fail with the same error. I guess before something was cached for the repository with version 0.1.4. I used biopython repository as a guide to write my tool dependency definition: http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61 I can confirm biopython repository is failing to install for me with exactly the same error. I wonder if a recently addition in the test toolshed broke the treatment of prior_installation_required. Thanks, Carlos Could be - note that currently Biopython isn't currently installing properly on the Test Tool Shed due to ALTAS failing (a requirement of NumPy which is a requirement of Biopython). Dave and Bjoern are I think looking at this already... I can't do much I tested it again and for me its working fine on my computers I have at hand ... sorry. In case it helps, this is how the INSTALLATION.log file ends on OS X 10.8.4: $ tail -n 3 ~/src/shed_tools_dependencies.central/atlas/3.10.1/iuc/package_atlas_3_10/3508de0ebae1/INSTALLATION.log x ATLAS/tune/lapack/lanbsrch.c tar: Error exit delayed from previous errors. # This is the relevant part I can find in Galaxy's log: [localhost] local: tar xfvj atlas3.10.1.tar.bz2 Warning: local() encountered an error (return code 1) while executing 'tar xfvj atlas3.10.1.tar.bz2' After noticing this I got what I'm guessing is the original file from sourceforge: http://downloads.sourceforge.net/project/math-atlas/Stable/3.10.1/atlas3.10.1.tar.bz2 I can confirm that trying to untar this file fails with the exact same error. However, on Ubuntu 13.04 untaring this file works just fine. That is new to me. How can that happen? Anyone with an OS-X can confirm that? On Ubuntu 13.04 the error I see is: STDERR It appears you have cpu throttling enabled, which makes timings unreliable and an ATLAS install nonsensical. Aborting. See ATLAS/INSTALL.txt for further information # Björn, you said cpu throttling can be easily disable on Ubuntu. Can you comment how? Do I need to disable it completely, or should the install script do it when installing? ATLAS (once untared ;-)) needs cpu-throttling to be disabled to optimizes its library. If it is not disabled ATLAS compilation will fail. For OS-X I found that: http://apple.stackexchange.com/questions/41045/how-can-i-disable-cpu-throttling-and-cpu-disabling Sorry, I never touched a OS-X. Nevertheless, If its not disabled, it is supposed to fail silently and downstream packages will not be affected. But if its crashing during untarring, I can't do much. What I can do is to repack the tarball and host it elsewhere. What brings me to: http://gmod.827538.n3.nabble.com/RFC-Storing-of-tarballs-and-patches-for-tool-dependencies-to-enable-reproducibility-td4036591.html Bad news for a Friday morning, sorry :( Bjoern Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
tar xvfj atlas3.11.11.tar.bz2 shows no errors on OSX and creates one ATLAS folder. best, ido On Sep 20, 2013, at 11:17 AM, Bjoern Gruening bjoern.gruen...@gmail.com wrote: Hi Ido and Carlos, can you check if that tarball is working? http://downloads.sourceforge.net/project/math-atlas/Developer%20%28unstable%29/3.11.11/atlas3.11.11.tar.bz2 The chance is low, but if its working for you I will consider to create a new version for it. Thanks, Bjoern Yes this tar is broken at least on OSX. Other people have the same issue: http://code.google.com/p/libarchive/issues/detail?id=299 On Sep 20, 2013, at 10:41 AM, Bjoern Gruening bjoern.gruen...@gmail.com wrote: Hi Carlos, Hi Peter and Carlos, On Mon, Sep 16, 2013 at 8:57 PM, Carlos Borroto carlos.borr...@gmail.com wrote: I did an extra test. Started with a clean 'galaxy-dist'. This time both repositories fail with the same error. I guess before something was cached for the repository with version 0.1.4. I used biopython repository as a guide to write my tool dependency definition: http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61 I can confirm biopython repository is failing to install for me with exactly the same error. I wonder if a recently addition in the test toolshed broke the treatment of prior_installation_required. Thanks, Carlos Could be - note that currently Biopython isn't currently installing properly on the Test Tool Shed due to ALTAS failing (a requirement of NumPy which is a requirement of Biopython). Dave and Bjoern are I think looking at this already... I can't do much I tested it again and for me its working fine on my computers I have at hand ... sorry. In case it helps, this is how the INSTALLATION.log file ends on OS X 10.8.4: $ tail -n 3 ~/src/shed_tools_dependencies.central/atlas/3.10.1/iuc/package_atlas_3_10/3508de0ebae1/INSTALLATION.log x ATLAS/tune/lapack/lanbsrch.c tar: Error exit delayed from previous errors. # This is the relevant part I can find in Galaxy's log: [localhost] local: tar xfvj atlas3.10.1.tar.bz2 Warning: local() encountered an error (return code 1) while executing 'tar xfvj atlas3.10.1.tar.bz2' After noticing this I got what I'm guessing is the original file from sourceforge: http://downloads.sourceforge.net/project/math-atlas/Stable/3.10.1/atlas3.10.1.tar.bz2 I can confirm that trying to untar this file fails with the exact same error. However, on Ubuntu 13.04 untaring this file works just fine. That is new to me. How can that happen? Anyone with an OS-X can confirm that? On Ubuntu 13.04 the error I see is: STDERR It appears you have cpu throttling enabled, which makes timings unreliable and an ATLAS install nonsensical. Aborting. See ATLAS/INSTALL.txt for further information # Björn, you said cpu throttling can be easily disable on Ubuntu. Can you comment how? Do I need to disable it completely, or should the install script do it when installing? ATLAS (once untared ;-)) needs cpu-throttling to be disabled to optimizes its library. If it is not disabled ATLAS compilation will fail. For OS-X I found that: http://apple.stackexchange.com/questions/41045/how-can-i-disable-cpu-throttling-and-cpu-disabling Sorry, I never touched a OS-X. Nevertheless, If its not disabled, it is supposed to fail silently and downstream packages will not be affected. But if its crashing during untarring, I can't do much. What I can do is to repack the tarball and host it elsewhere. What brings me to: http://gmod.827538.n3.nabble.com/RFC-Storing-of-tarballs-and-patches-for-tool-dependencies-to-enable-reproducibility-td4036591.html Bad news for a Friday morning, sorry :( Bjoern Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To
Re: [galaxy-dev] Towards Galaxy Linux (not) ? [was [RFC] Storing of tarballs and patches for tool_dependencies to enable reproducibility]
If I might chime in, I am a bit worried about all the automatic installation going on in galaxy, and it seems that the trend is to enhance this. A small R or python script calling into well known libraries that come from well known repositories (bioconductor etc… ) I can check. (Of course I install too much stuff from github, bioconductor etc… without checking). I'm not sure it is comparable to a entire Linux distribution, its more like an Appstore, like pypi, bioconductor or gems, and yes that is The app stores are checked by Apple or google for malicious code, the apps are sandboxed. There are many eyes for python, bioconductor packages and gems because much more people interact with them directly compared to galaxy-tools. Sorry maybe I was misleading. I only want a central storage for binaries/tarballs where the source can not be trusted for long term. 'long term' and 'trusted' needs to be defined in such a discussion here. I do not think we should copy python packages that are stored in pypi. We should make it easy as possible to install them in our repository. If you do not trust pypi, we can offer a mirror. Some goes for gems. Trusted for me means I trust the source not having dangerous code. I trust pypi more than some mirror, bioconductor base packages from more than some freshly published package that few people have used, tools from galaxy core developers more than from tool-shed etc… I know this is not the type of trust you were talking about. best, ido ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] tools installed from a toolshed repository displayed in an arbitrary order
IIRC then you can change the order of tools manually by copying around the individual tools in shed_tool_conf.xml In my case it builds the integrated_tool_panel.xml in reverse order within the section. I still have labels between the section defined in the tool_conf.xml. But no subsections. best, ido On Sep 16, 2013, at 6:40 AM, yulia.arzha...@csiro.au wrote: Hello guys, After porting our Galaxy tools into a tool shed repository the tool sections got lost. I know there is a possibility to assign a repository to a section inside the tools panel when the repository is getting installed. However, we would still like to have sections inside the repository – that makes navigating the tools much neater. Also, the way shed_tool_conf.xml is populated with tools during repository installation seems arbitrary to me. Is there any way to control this, to have repository tools sorted/displayed in a certain order in Tools panel, in the very least, alphabetically? I’ve got many tools in my repository (and no, it is not possible to divide them into several repositories), and the way they are sorted or divided into sections makes navigating the tools much easier for a user. Thanks, Yulia ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy Release Cycle Length
Why the dislike for quick turnover? Could somebody present the arguments for people not having been at the BOF? People don't have to upgrade - unless its breaking changes that e.g. disable the possibility to download from the public toolshed which forced me to upgrade. The alternative to frequent releases are not better tested and documented releases. I think its impossible for the galaxy team to test on all the diverse configurations galaxy is deployed on. Then you really have accumulated many bugs, feedback from users comes in at once creating coordination need for developers going all at once into bugfixing mode for features they have worked on months ago etc… Then the fixes for these bugs also come only with the next release which is still many months away etc…. This makes it necessary to split development into major releases (say once a year) and minor, bugfix, releases (1-2 months after the major) - where is the gain for the users? I would not even split things between breaking changes and minor changes. I think this slows down development of the platform and what concerns people most, the tools, are developed independently of the platform and one can upgrade them any time. To give an example, the job_config is now much better than before and its good that I did not have to wait months from its development to deployment at our site. Now small additional features like setting the number of threads dynamically are suggested, and then I would have to wait again many,many months until the next release. Upgrading galaxy was o.k. and while its unfortunate to have to learn new settings and remember the old ones from sometimes not so good (but improving!) documentation, there is no way around it. I think I will now upgrade more often, because it went so well. I like release early, release often. best, ido On Aug 20, 2013, at 8:36 PM, Dave Clements cleme...@galaxyproject.org wrote: Hello all, At one of the GCC2013 Birds of a Feather sessions the group was very clear that they would like to see less frequent releases of Galaxy. We're currently aiming to do a release every 2 months and have been pretty successful at making that target. In the past, we have tried doing releases more often and less often. Is there a sweet spot for the time between releases? Please reply to the group. We are interested in a discussion. Thanks, Dave C -- http://galaxyproject.org/GCC2013 http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://wiki.galaxyproject.org/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] shall us specify which tool run in cluster?
On Jul 31, 2013, at 8:52 AM, shenwiyn shenw...@gmail.com wrote: Hi Thon Deboer , I am a newer in Galaxy.I installed my Galaxy with Torque2.5.0 ,and Galaxy uses the pbs_modoule to interface with TORQUE.But I have some question of the job_conf.xml : 1.)In your job_conf.xml ,you use regularjobs,longjobs,shortjobs...to run different jobs ,how our Galaxy know which tool belongs to regularjobs or longjobs.And what is the meaning of nativeSpecification? by specifying, as Thon did, in tools the id of the tool and its destination which are the settings. the nativeSpecification allows you to set additional parameters that are passed with the call. e.g. -pe smp 4 tells the grid engine to use the parallel environment smp with 4 cores. 2.)Shall us use toolscollection of tool id=bwa_wrapper destination=multicorejobs4/to specify bwa ?Does it mean the bwa belong to multicorejobs4,and run in cluster? exactly 3.)Does every tool need us to specify which job it belong to? I saw http://wiki.galaxyproject.org/Admin/Config/Jobs about this,but I am not sure above.Could you help me please? fortunately there is a default shenwiyn From: Thon Deboer Date: 2013-07-18 14:31 To: galaxy-dev Subject: [galaxy-dev] Jobs remain in queue until restart Hi, I have noticed that from time to time, the job queue seems to be “stuck” and can only be unstuck by restarting galaxy. The jobs seem to be in the queue state and the python job handler processes are hardly ticking over and the cluster is empty. When I restart, the startup procedure realizes all jobs are in the a “new state” and it then assigns a jobhandler after which the jobs start fine…. Any ideas? Torque Thon P.S I am using the june version of galaxy and I DO set limits on my users in job_conf.xml as so: (Maybe it is related? Before it went into dormant mode, this user had started lots of jobs and may have hit the limit, but I assumed this limit was the number of running jobs at one time, right?) ?xml version=1.0? job_conf plugins workers=4 !-- workers is the number of threads for the runner's work queue. The default from plugins is used if not defined for a plugin. -- plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner workers=2/ plugin id=drmaa type=runner load=galaxy.jobs.runners.drmaa:DRMAAJobRunner workers=8/ plugin id=cli type=runner load=galaxy.jobs.runners.cli:ShellJobRunner workers=2/ /plugins handlers default=handlers !-- Additional job handlers - the id should match the name of a [server:id] in universe_wsgi.ini. -- handler id=handler0 tags=handlers/ handler id=handler1 tags=handlers/ handler id=handler2 tags=handlers/ handler id=handler3 tags=handlers/ !-- handler id=handler10 tags=handlers/ handler id=handler11 tags=handlers/ handler id=handler12 tags=handlers/ handler id=handler13 tags=handlers/ -- /handlers destinations default=regularjobs !-- Destinations define details about remote resources and how jobs should be executed on those remote resources. -- destination id=local runner=local/ destination id=regularjobs runner=drmaa tags=cluster !-- These are the parameters for qsub, such as queue etc. -- param id=nativeSpecification-V -q long.q -pe smp 1/param /destination destination id=longjobs runner=drmaa tags=cluster,long_jobs !-- These are the parameters for qsub, such as queue etc. -- param id=nativeSpecification-V -q long.q -pe smp 1/param /destination destination id=shortjobs runner=drmaa tags=cluster,short_jobs !-- These are the parameters for qsub, such as queue etc. -- param id=nativeSpecification-V -q short.q -pe smp 1/param /destination destination id=multicorejobs4 runner=drmaa tags=cluster,multicore_jobs !-- These are the parameters for qsub, such as queue etc. -- param id=nativeSpecification-V -q long.q -pe smp 4/param /destination !-- destination id=real_user_cluster runner=drmaa param id=galaxy_external_runjob_scriptscripts/drmaa_external_runner.py/param param id=galaxy_external_killjob_scriptscripts/drmaa_external_killer.py/param param id=galaxy_external_chown_scriptscripts/external_chown_script.py/param /destination -- destination id=dynamic runner=dynamic !-- A destination that represents a method in the dynamic runner. -- param id=typepython/param param id=functioninteractiveOrCluster/param /destination /destinations tools !-- Tools can be configured to use specific destinations or
[galaxy-dev] job_conf.xml workers
Hi, what is the relationship between workers defined in universe and job_conf.xml? e.g.: [server:handler0] use = egg:Paste#http port = 9092 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 plugins workers=4 !-- workers is the number of threads for the runner's work queue. The default from plugins is used if not defined for a plugin. -- plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner/ plugin id=drmaa type=runner load=galaxy.jobs.runners.drmaa:DRMAAJobRunner workers=10/ /plugins a) do I have to specify the sum of the workers in job_conf.xml in universe? e.g. 4 local, 10 drmaa = 14 threadpool_workers per handler? Or is universe overridden by job_conf.xml? b) How many concurrent jobs can I then expect to have concurrently? 14 per handler? thank you very much, ido ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] job_conf.xml workers
Ah yes, this was local_job_queue_workers and cluster_job_queue_workers in universe.ini thank you very much, ido On Jul 24, 2013, at 5:10 PM, James Taylor ja...@jamestaylor.org wrote: use_threadpool and threadpool_workers specify the number of threads for handling web requests and have nothing to do with job running. -- James Taylor, Assistant Professor, Biology/CS, Emory University On Wed, Jul 24, 2013 at 11:02 AM, Ido Tamir ta...@imp.ac.at wrote: Hi, what is the relationship between workers defined in universe and job_conf.xml? e.g.: [server:handler0] use = egg:Paste#http port = 9092 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 plugins workers=4 !-- workers is the number of threads for the runner's work queue. The default from plugins is used if not defined for a plugin. -- plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner/ plugin id=drmaa type=runner load=galaxy.jobs.runners.drmaa:DRMAAJobRunner workers=10/ /plugins a) do I have to specify the sum of the workers in job_conf.xml in universe? e.g. 4 local, 10 drmaa = 14 threadpool_workers per handler? Or is universe overridden by job_conf.xml? b) How many concurrent jobs can I then expect to have concurrently? 14 per handler? thank you very much, ido ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] job_conf.xml questions
Hi, I work myself through the job_conf.xml and have a question: a) In your advanced example you have: destination id=remote_cluster runner=drmaa tags=longjobs/ destination id=real_user_cluster runner=drmaa !-- TODO: The real user options should maybe not be considered runner params. -- param id=galaxy_external_runjob_scriptscripts/drmaa_external_runner.py/param param id=galaxy_external_killjob_scriptscripts/drmaa_external_killer.py/param param id=galaxy_external_chown_scriptscripts/external_chown_script.py/param /destination Does this mean that remote_cluster jobs can not be killed unless I add the 3 scripts to this destination? Or does this have to do with the run jobs as real user that I don't need currently. I am using the galaxy user for all jobs for the moment? To phrase this differently: Do I need these three scripts? b) Can I set a wall time limit for only for my local runners? like: destination id=local runner=local/ param id=Resource_Listwalltime=72:00:00/param /destination When I set it in limits, will it work for local runners and then I have to override it in every destination? thank you very much, ido ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] no metadata for tools in toolshed
Hi, I grabbed some xml files from Hannons website for his fastx toolkit (improved over galaxys default version). Then I created a repository, uploaded the xml files, but the toolshed complains that it could not find the tools for creating metadata in the xml files. Could somebody please tell me what the problem is with these files? They look like valid tool.xml files to me. http://toolshed.g2.bx.psu.edu/view/idot/fastx_toolkit2 http://i...@toolshed.g2.bx.psu.edu/repos/idot/fastx_toolkit2 thank you very much, ido ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] deleted_new kills handlers
No the problem is still there, but we have not updated yet since beginning of december. maybe this is related: http://gmod.827538.n3.nabble.com/segfault-in-libdrmaa-gt-galaxy-front-end-failure-td4027349.html best, ido On Jan 21, 2013, at 5:45 AM, Derrick Lin wrote: Hi Ido, I just found you post, and I think we are having the similar issue as your. (I posted it in the mailing yesterday, you can find it for the detail of my problem). I am wondering if you are able to fix your problem? Cheers, Derrick On Tue, Dec 11, 2012 at 12:13 AM, Ido Tamir ta...@imp.ac.at wrote: Dear galaxy maintainers, we have the problem that killing jobs running on drmaa before they run leading to a deleted_new state, kills our handlers (sometimes?). The job gets sent to the handler, but the handler never acknowledges the jobs (local and drmaa) and all submitted jobs afterwards stay in submitted state. Normally submission works fine, running also. Is such a problem known to anybody else? in my universe_wsgi.ini: enable_job_recovery = False I don't know which other settings could affect this. Could it be an old drmaa library, maybe because the SGE was updated after galaxy installation? Other ideas? thank you very much, ido ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job handler keeps crashing
Is it possible to backport this onto the latest distribution? Yes, I'm lazy, but there are also others that are still updating within the next weeks and will have problems without them being aware of this fix. best, ido On Jan 21, 2013, at 9:50 PM, Nate Coraor wrote: Hi all, The commit[1] that fixes this is not in the January 11 distribution. It'll be part of the next distribution. --nate [1] https://bitbucket.org/galaxy/galaxy-central/commits/c015b82b3944f967e2c859d5552c00e3e38a2da0 On Jan 21, 2013, at 3:10 PM, Anthonius deBoer wrote: I have seen this same issue exactly. Python just dies without any errors in the log. Using the latest galaxy-dist Sent from my iPhone On Jan 20, 2013, at 8:35 PM, Derrick Lin klin...@gmail.com wrote: Update to the 11 Jan 2013 dist does not help with this issue. :( I checked the database and have the look at the job entries that handler0 tried to stop then shutdown: | 3088 | 2013-01-03 14:25:38 | 2013-01-03 14:27:05 |531 | toolshed.g2.bx.psu.edu/repos/kevyin/homer/homer_findPeaks/0.1.2 | 0.1.2| deleted_new | Job output deleted by user before job completed. | NULL | NULL | NULL| NULL | NULL | NULL | 1659 | drmaa://-V -j n -R y -q intel.q/ | NULL | NULL | 76 |0 | NULL| NULL | handler0 | NULL | | 3091 | 2013-01-04 10:52:19 | 2013-01-07 09:14:34 |531 | toolshed.g2.bx.psu.edu/repos/kevyin/homer/homer_findPeaks/0.1.2 | 0.1.2| deleted_new | Job output deleted by user before job completed. | NULL | NULL | NULL| NULL | NULL | NULL | 1659 | drmaa://-V -j n -R y -q intel.q/ | NULL | NULL | 76 |0 | NULL| NULL | handler0 | NULL | | 3093 | 2013-01-07 22:02:21 | 2013-01-07 22:16:27 |531 | toolshed.g2.bx.psu.edu/repos/kevyin/homer/homer_pos2bed/1.0.0 | 1.0.0| deleted_new | Job output deleted by user before job completed. | NULL | NULL | NULL| NULL | NULL | NULL | 1749 | drmaa://-V -j n -R y -q intel.q/ | NULL | NULL | 76 |0 | NULL| NULL | handler0 | NULL | So basically the job table has several of these entries what assigned to handler0 and marked as deleted_new. When the handler0 is up, it starts stopping these jobs, after the first job has been stopped, handler0 went crash and died. But that job was then marked as deleted. I think if I manually change the job state from deleted_new to deleted in the db, the handler0 will become fine. I am just concerned about how these jobs were created (like assigned to a handler but marked as deleted_new). Cheers, D On Mon, Jan 21, 2013 at 1:49 PM, Derrick Lin klin...@gmail.com wrote: I had a close look at the code in galaxy-dist / lib / galaxy / jobs / handler.py galaxy-dist / lib / galaxy / jobs / runners / drmaa.py and found that stopping deleted and deleted_new seems to be normal routine for the job handler. Could not find any exception that caused the shutdown. I do notice in the galaxy-dist on bitbucket, there is one commit with comment Fix shutdown on python = 2.6.2 by calling setDaemon when creating threads (these are still..., it seems to be relevant? I will do the update to 11 Jan release and see if it fixes the issue. D On Fri, Jan 18, 2013 at 4:03 PM, Derrick Lin klin...@gmail.com wrote: Hi guys, We have updated our galaxy to 20 Dec 2012 release. Recently we found that some submitted jobs could not start (stay gray forever). We found that it was caused by the job manager sent jobs to a handler (handler0) whose python process crashed and died. From the handler log we found the last messages right before the crash: galaxy.jobs.handler DEBUG 2013-01-18 15:00:34,481 Stopping job 3032: galaxy.jobs.handler DEBUG 2013-01-18 15:00:34,481 stopping job 3032 in drmaa runner We restarted the galaxy, handler0 is up for few seconds then died again with the same error messages except the job number moved to the next one. We observed that the jobs it was trying to stop are all previous jobs whose status is either deleted or deleted_new. We have never seen this in the past, so wondering if there is bugs in the new release? Cheers, Derrick ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists,
Re: [galaxy-dev] In which Encryption form does Galaxy store password?
Its hashed (sha1) and you can not reverse the hash. Storing passwords in a reversible manner is bad practice. http://galaxy-central.readthedocs.org/en/latest/_modules/galaxy/util/hash_util.html You could maybe authenticate your users from the other program with the same method against the hashed passwords. best, ido On Jan 7, 2013, at 12:43 PM, Sachit Adhikari wrote: Hi, In which encryption galaxy store the password? It's really urgent. I want to decode the password and use in the other program. Thanks ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] let galaxy use more hardware resouses
If you did what the links posted previously suggested (inc. parallelization of tools, mutliple server processes), there is nothing more you can do. Most of the tools galaxy just calls, so its the tool that you have to change. Often genomic tools are also IO bound, so a faster storage system could lead to better CPU utilization - if this is the bottleneck. HTH, ido On Dec 10, 2012, at 9:51 AM, 泽 蔡 wrote: Hi all, I run a local instance of galaxy. Today, I take a look at the system monitor, I found the galaxy just use 4% of a CPU and a little bit of the memory of the PC. But the PC has twelve CPUs and 126G memory. So, how to configure the galaxy to use more hardware resouces to calculate quickly? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] binary data types get corrupted on upload
The specification does not mention zip, gzip or bz2 [1] and I am pretty confident, that it does not in addition compresses its content with zip etc.. It starts with a header SQLite format 3\000 which is visible in my file with less. ido [1]http://www.sqlite.org/fileformat.html On Dec 7, 2012, at 3:50 PM, John Chilton wrote: Hi Ido, Is .sqlite a common compressed file type (i.e. zip, gzip, or bz2) just with a different extension? If it is, I think upload.py is always going to try to decompress it. That that has long been a behavior I would like to be able to disable on a per file type basis. If this is the problem and the galaxy team agrees to accept a patch for address the issue, I would be happy to try to implement the fix. -John On Fri, Dec 7, 2012 at 8:39 AM, Ido Tamir ta...@imp.ac.at wrote: from http://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes I thought that adding subclass=True in datatypes_conf.xml would be enough to register my new type, but it seems like I stil have to add an entry in binary.py for it to be recognized. But its still corrupted. thank you very much, ido -- Tool: Upload File Name: hg19.sqlite Created:Dec 07, 2012 Filesize: 11.1 MB Dbkey: hg19 Format: sqlite ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] from shed to shed
Hi, how do I get tools from one local toolshed (or the main galaxy toolshed) into another tool-shed? Is it possible to clone the content of a complete tool-shed - all repositories at once into another tool-shed? Can I then import all the tools at once into a local galaxy instance? We have a group with their own tool-shed. Unfortunately, I accessed it directly without a proxy on its port which leads to a problem described in issue 825 in galaxy-central. I want to a) clone the tool-shed as it is into my own tool-shed (rsync?) b) selectively or bulk install these tools into my galaxy server. thank you very much, ido ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
