[galaxy-dev] Enhancement: selecting input
Hi all, What if the input dataset selection for tools could be selected via clicking a dataset (or even more) directly in the history, instead of selecting from that good ol' dropdown box? It is just that I have to concatenate the contents of 24 files... I agree, in bash the command is typed below two seconds. But we're not (and my data is not) in bash, we have this wonderful Galaxy interface. Anybody triggered to develop this enhancement ? ;-) Cheers, Joachim. PS: I would have posted on Trello, but I am not allowed to do so. I understand that this is the way to propose enhancements. -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Trackster error
Hi all, I want to view two interval-type datasets in trackster on genome hg19 together with one BAM dataset. The interval datasets give an error (while the BAM is displayed nicely): [ti_index_core] bug in BGZF: 80048ed5001b 80048ed5 Thanks for any input, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Trackster and gff file with multiple chromosome annotations
Here are the links to get the gff and the related genome files: http://genomeportal.jgi-psf.org/Crypa2/download/Cparasiticav2.GeneCatalog20091217.gff.gz http://genomeportal.jgi-psf.org/Crypa2/download/Cryphonectria_parasiticav2.nuclearAssembly.unmasked.gz Whatever the file type I set for the gff file (gff3, gff or gtf), I get the transcript_id error: Traceback (most recent call last): File /home/pgtgal/galaxy-dist/lib/galaxy/datatypes/converters/interval_to_fli.py, line 91, in main() File /home/pgtgal/galaxy-dist/lib/galaxy/datatypes/converters/interval_to_fli.py, line 30, in main for feature in read_unordered_gtf( open( in_fname, 'r' ) ): File /home/pgtgal/galaxy-dist/lib/galaxy/datatypes/util/gff_util.py, line 375, in read_unordered_gtf transcript_id = line_attrs[ 'transcript_id' ] KeyError: 'transcript_id' If I fix the transcript_id problem, I get the other error: Traceback (most recent call last): File ~/galaxy-dist/lib/galaxy/datatypes/converters/interval_to_fli.py, line 91, in main() File ~/galaxy-dist/lib/galaxy/datatypes/converters/interval_to_fli.py, line 30, in main for feature in read_unordered_gtf( open( in_fname, 'r' ) ): File ~/galaxy-dist/lib/galaxy/datatypes/util/gff_util.py, line 389, in read_unordered_gtf feature = GFFFeature( None, intervals=intervals ) File ~/galaxy-dist/lib/galaxy/datatypes/util/gff_util.py, line 65, in __init__ ( interval.chrom, self.chrom ) ) ValueError: interval chrom does not match self chrom: scaffold_10 != scaffold_10 Is the gff file not correct? PS : I use the galaxy changeset : 7828:b5bda7a5c345 Yec'han Yec'han LAIZET Ingenieur Plateforme Genome Transcriptome Tel: 05 57 12 27 75 _ INRA-UMR BIOGECO 1202 Equipe Genetique 69 route d'Arcachon 33612 CESTAS Le 23/10/2012 18:37, Jeremy Goecks a écrit : Yes, you should be able to use a single GFF for the complete genome. This error stems from the same issue as before, namely that Galaxy is treating your GFF file as GTF. If you think your GFF is well formatted and there is an issue with Galaxy's handling of GFF, please send me your GFF and I'll take a look. Best, J. On Oct 23, 2012, at 9:24 AM, Yec'han Laizet wrote: Hello, Is it possible to load a unique gff file with the annotations of several chromosomes for my custom build in one step (one gff file)? With the current version of galaxy, it seems that I can load a gff file referring only to one chromosome. That's pretty tedious to load 43 gff files separatly for my custom build... If I try, I get this error: Traceback (most recent call last): File ~/galaxy-dist/lib/galaxy/datatypes/converters/interval_to_fli.py, line 91, in main() File ~/galaxy-dist/lib/galaxy/datatypes/converters/interval_to_fli.py, line 30, in main for feature in read_unordered_gtf( open( in_fname, 'r' ) ): File ~/galaxy-dist/lib/galaxy/datatypes/util/gff_util.py, line 389, in read_unordered_gtf feature = GFFFeature( None, intervals=intervals ) File ~/galaxy-dist/lib/galaxy/datatypes/util/gff_util.py, line 65, in __init__ ( interval.chrom, self.chrom ) ) ValueError: interval chrom does not match self chrom: SAGS2 != SAGS1 Thanks Yec'han Yec'han LAIZET Ingenieur Plateforme Genome Transcriptome Tel: 05 57 12 27 75 _ INRA-UMR BIOGECO 1202 Equipe Genetique 69 route d'Arcachon 33612 CESTAS ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Enhancement: selecting input
Not exactly what you asked for but it would allow you to build tools to solve this problem and I never pass up an opportunity to sell my pull requests: https://bitbucket.org/galaxy/galaxy-central/pull-request/76/multi-input-data-tool-parameter-fixes This would allow one to put a multiple=true on input data parameters. Then a multi-selection widget is rendered instead of a drop down list. We have addressed rerunning tools, interaction with the workflow editor, workflow imports and exports, etc -John John Chilton Senior Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 Cell: 612-226-9223 Bitbucket: https://bitbucket.org/jmchilton Github: https://github.com/jmchilton Web: http://jmchilton.net On Oct 24, 2012 2:38 AM, Joachim Jacob joachim.ja...@vib.be wrote: Hi all, What if the input dataset selection for tools could be selected via clicking a dataset (or even more) directly in the history, instead of selecting from that good ol' dropdown box? It is just that I have to concatenate the contents of 24 files... I agree, in bash the command is typed below two seconds. But we're not (and my data is not) in bash, we have this wonderful Galaxy interface. Anybody triggered to develop this enhancement ? ;-) Cheers, Joachim. PS: I would have posted on Trello, but I am not allowed to do so. I understand that this is the way to propose enhancements. -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib __**_ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Enhancement: selecting input
On Oct 24, 2012, at 3:36 AM, Joachim Jacob joachim.ja...@vib.be wrote: PS: I would have posted on Trello, but I am not allowed to do so. I understand that this is the way to propose enhancements. For adding cards, anyone can use the form at http://galaxyproject.org/trello. A new card will be created on the Inbox list of the trello board with the title/description entered in the form. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Enhancement: selecting input
Thanks for this Trello. Looks nice and powerful! it would be great to have more Trello lists, like: enhancements requests, Currently working on, ... etc. I love to help sorting the trello lists, but we cannot drag en drop the cards around. A last comment ( ^_^ ) it is rather cumbersome to start voting on Trello. Cheers, Joachim On Wed 24 Oct 2012 01:48:51 PM CEST, Dannon Baker wrote: On Oct 24, 2012, at 3:36 AM, Joachim Jacob joachim.ja...@vib.be wrote: PS: I would have posted on Trello, but I am not allowed to do so. I understand that this is the way to propose enhancements. For adding cards, anyone can use the form at http://galaxyproject.org/trello. A new card will be created on the Inbox list of the trello board with the title/description entered in the form. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Trackster error
Can you please share the datasets that are generating this errors? Thanks, J. On Oct 24, 2012, at 4:19 AM, Joachim Jacob wrote: Hi all, I want to view two interval-type datasets in trackster on genome hg19 together with one BAM dataset. The interval datasets give an error (while the BAM is displayed nicely): [ti_index_core] bug in BGZF: 80048ed5001b 80048ed5 Thanks for any input, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] JBrowse direct export to Galaxy
Passing it back to Jeremy, there is also a Javascript wrapper around the tools API which might help: https://bitbucket.org/galaxy/galaxy-central/src/tip/static/scripts/mvc/tools.js Yes, the Galaxy team is working on a JavaScript binding to the API, and one of the areas that we've done some work on is running tools. It's probably a little heavyweight if all your want to do is push datasets into Galaxy—HTTP posts to the API will work just fine—but it's useful if you want real integration between Galaxy and your Web application (JBrowse or otherwise), such as the ability to work with histories/datasets/tools/visualizations on the fly. J. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Enhancement: selecting input
Hi John, +1 Will it be accepted in galaxy-central? This looks very useful to me. Cheers, Joachim Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib On 10/24/2012 01:41 PM, John Chilton wrote: Not exactly what you asked for but it would allow you to build tools to solve this problem and I never pass up an opportunity to sell my pull requests: https://bitbucket.org/galaxy/galaxy-central/pull-request/76/multi-input-data-tool-parameter-fixes This would allow one to put a multiple=true on input data parameters. Then a multi-selection widget is rendered instead of a drop down list. We have addressed rerunning tools, interaction with the workflow editor, workflow imports and exports, etc -John John Chilton Senior Software Developer University of Minnesota Supercomputing Institute Office: 612-625-0917 tel:612-625-0917 Cell: 612-226-9223 tel:612-226-9223 Bitbucket: https://bitbucket.org/jmchilton Github: https://github.com/jmchilton Web: http://jmchilton.net http://jmchilton.net/ On Oct 24, 2012 2:38 AM, Joachim Jacob joachim.ja...@vib.be mailto:joachim.ja...@vib.be wrote: Hi all, What if the input dataset selection for tools could be selected via clicking a dataset (or even more) directly in the history, instead of selecting from that good ol' dropdown box? It is just that I have to concatenate the contents of 24 files... I agree, in bash the command is typed below two seconds. But we're not (and my data is not) in bash, we have this wonderful Galaxy interface. Anybody triggered to develop this enhancement ? ;-) Cheers, Joachim. PS: I would have posted on Trello, but I am not allowed to do so. I understand that this is the way to propose enhancements. -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 tel:%2B32%209%20244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] High CPU usage of paster.py python process (daemon)
Hello, Recently I started 2 local galaxy servers on Linux PC for internal use. I noticed that the main WSGI process uses a lot of CPU time - see the output of ps command below: $ ps aux | head -1 ps aux | grep wsgi [1] 5293 USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND galaxy26685 59.6 0.3 457332 125224 ? Sl Sep14 34447:36 python ./scripts/paster.py serve universe_wsgi.ini --daemon galaxy19346 38.9 1.9 1098388 634772 ? Sl Oct17 4451:59 python ./scripts/paster.py serve universe_wsgi.ini --daemon From this output, the first galaxy process used 59% of the CPU time in the recent month, it is 34447 minutes! The second process used 39% in the recent days. In the same month a tomcat web application used 10 minutes of CPU time, which is 0.0%. I have to say, that galaxy was accessed very, very seldom, a few times per week. Also the top command returns similar results: both galaxy server use more than 90% of CPU time, even if nobody is accessing galaxy. Is this normal? Do you have the same %CPU in ps command? Can this be configured/minimized somehow? Why is the CPU usage so high, if nobody is accessing galaxy? Is this a feature of WSGI? Sebastian ** Sebastian Sośnik FQS Poland 30-538 Krakow, ul. Parkowa 11 URL: http://www.fqs.pl http://www.fqs.pl Tel. (+48 12) 429 43 45 Mobile (+48) 608500161 Fax. (+48 12) 429 61 24 ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Manage Jobs: This Galaxy instance is not the job manager.
On Oct 20, 2012, at 10:36 AM, Christos Kannas wrote: Hi all, We have a locally installed Galaxy server that runs only a set of tools we are developing. It has been setup to run with 3 Web processes , 1 Job Manager and 4 Job Handlers. This setup is beeing run behind an Apache web server and we created a config file similar to the one found @ http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application%20Scaling. The problem we have is that Job Manager is not accessible, and we get error messages like the one below: This Galaxy instance (web1) is not the job manager (manager). If using multiple servers, please directly access the job manager instance to manage jobs. We do have the following RewriteRule: RewriteRule ^/admin/jobs(.*) http://localhost:8079/admin/jobs$1 [P] but it doesn't seem to work. Can you help me fix it? Hi Christos, Does the manager application actually see requests sent to this URL (in its logs)? --nate Regards, -- Christos Kannas Researcher Ph.D Student e-Health Laboratory kannas.chris...@ucy.ac.cy kannas.chris...@cs.ucy.ac.cy chriskan...@gmail.com Mob: (+357) 99530608 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Manage Jobs: This Galaxy instance is not the job manager.
Hi Nate, I do not see anything in the manager.log that seems as a HTTP request. All Galaxy processes are set to listen from localhost. When I change this to 0.0.0.0 I can access manager process by requesting http directly to its listening port. To me it seems that this RewriteRule is somehow being skipped. Hmm... Update: I have found the solution. The *[P]* in *RewriteRule *makes the rule to push it to Proxy if it matches, and stops parsing any rules after the rule that have matched. So if you write: *# Job Manager* *RewriteRule ^/admin/jobs(.*) http://localhost:8079/admin/jobs$1 [P]* * * *# Balancer* *RewriteRule ^(.*) balancer://galaxy$1 [P]* instead of: *# Balancer* *RewriteRule ^(.*) balancer://galaxy$1 [P]* * * *# Job Manager* *RewriteRule ^/admin/jobs(.*) http://localhost:8079/admin/jobs$1 [P]* It works, because for every request *RewriteRule ^(.*) balancer://galaxy$1 [P]* has a valid match but* RewriteRule ^/admin/jobs(.*) http://localhost:8079/admin/jobs$1 [P]* has a match only on /admin/jobs. So you have to also point this in http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application%20Scaling . Regards, Christos Kannas On Wed, Oct 24, 2012 at 5:33 PM, Nate Coraor n...@bx.psu.edu wrote: On Oct 20, 2012, at 10:36 AM, Christos Kannas wrote: Hi all, We have a locally installed Galaxy server that runs only a set of tools we are developing. It has been setup to run with 3 Web processes , 1 Job Manager and 4 Job Handlers. This setup is beeing run behind an Apache web server and we created a config file similar to the one found @ http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application%20Scaling . The problem we have is that Job Manager is not accessible, and we get error messages like the one below: This Galaxy instance (web1) is not the job manager (manager). If using multiple servers, please directly access the job manager instance to manage jobs. We do have the following RewriteRule: RewriteRule ^/admin/jobs(.*) http://localhost:8079/admin/jobs$1 [P] but it doesn't seem to work. Can you help me fix it? Hi Christos, Does the manager application actually see requests sent to this URL (in its logs)? --nate Regards, -- Christos Kannas Researcher Ph.D Student e-Health Laboratory kannas.chris...@ucy.ac.cy kannas.chris...@cs.ucy.ac.cy chriskan...@gmail.com Mob: (+357) 99530608 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Christos Kannas Researcher Ph.D Student e-Health Laboratory http://www.medinfo.cs.ucy.ac.cy/ kannas.chris...@ucy.ac.cy kannas.chris...@cs.ucy.ac.cy chriskan...@gmail.com Mob: (+357) 99530608 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy Admin Tools
On Oct 23, 2012, at 12:07 PM, lenta...@jimmy.harvard.edu wrote: Hi, We have some common admin-tasks that we would like to implement on our instance, but before doing so, we were wondering what Galaxy has already baked it. 1. Is the Galaxy team planning to implement job_walltime for the local runner? Currently, job_walltime is only implemented for the pbs runner--we believe that it would be a useful feature regardless of which runner one is using. Hi Len, I've implemented this in changeset bb365c782b64. 2. Is there a way to limit the number of active jobs a user can run at one time?--perhaps a record in universe_wsgi.ini (e.g. max_jobs) could be helpful. Yes, see the user_job_limit parameter in universe_wsgi.ini. --nate 3. This is a bug report--but on our version of Galaxy, rev 949e4f5fa03a, the sort-by-last login (on the manage users page), doesn't sort correctly. We are interested in deleting inactive users, and sorting by last-login is one way of doing this, but I was wondering if others out there have a better solution. Thanks! Len Taing Cistrome Development Team ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] The blastdb_p.loc sample is wrong.
Hello Peter, Thanks very much for this - I'm sure many will find it very helpful. I've applied your patch to changeset revision 8096:0c56502c7fd7, which will be included in the next Galaxy release currently scheduled for next Friday (not today's currently scheduled release). Thanks again, Greg Von Kuster On Oct 22, 2012, at 7:03 AM, Peter Cock wrote: FAO the Galaxy dev team, I've tested a patch (at end of email) which issues a warning if loading a loc file with inconsistent numbers of tabs. In the case of blastdb_p.loc this would result in showing the warning three times, since currently Galaxy appears to reload a *.loc file for each tool using it. (If the error happens to be on the first list, then the warning is triggered for all the following lines in the file - because they won't have the same number of fields as the first line.) Could someone apply this to the trunk as is, or tell me if you would prefer it as a pull request on bitbucket? Furthermore, would a more invasive change to treat this as an error condition be acceptable? Thanks, Peter On Sun, Oct 21, 2012 at 5:53 AM, Fields, Christopher J cjfie...@illinois.edu wrote: Yes, have to agree there, an error would be more informative. chris On Oct 20, 2012, at 2:38 PM, Shane Sturrock sh...@biomatters.com wrote: That would certainly have saved me a fair bit of time since the only indication I got was an empty set of quotes in the error log. I always feel like tabs aren't the best separators for these sorts of files anyway since they are usually invisible but since that isn't likely to change, making it more obvious what the problem is would be the best approach. Shane On 21/10/2012, at 1:44 AM, Peter Cock wrote: Another idea would be for Galaxy to issue a clear error if a loc file has an inconsistent number of fields/tabs per line. Peter Suggested patch: $ hg diff lib/galaxy/tools/parameters/dynamic_options.py diff -r c05a680cbc80 lib/galaxy/tools/parameters/dynamic_options.py --- a/lib/galaxy/tools/parameters/dynamic_options.py Fri Oct 19 15:56:51 2012 +0100 +++ b/lib/galaxy/tools/parameters/dynamic_options.py Mon Oct 22 11:59:04 2012 +0100 @@ -471,6 +471,7 @@ def parse_file_fields( self, reader ): rval = [] +field_count = None for line in reader: if line.startswith( '#' ) or ( self.line_startswith and not line.startswith( self.line_startswith ) ): continue @@ -478,6 +479,17 @@ if line: fields = line.split( self.separator ) if self.largest_index len( fields ): +if not field_count: +field_count = len(fields) +elif field_count != len(fields): +try: +name = reader.name +except AttributeError: +name = a configuration file +#Perhaps this should be an error, but even a warning is useful +log.warn(Inconsistent number of fields (%i vs %i) in %s using + separator %r, check line: %r \ + % (field_count, len(fields), name, self.separator, line)) rval.append( fields ) return rval ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Jobs are slow to start on my galaxy instance. Used to be much faster
On Oct 23, 2012, at 12:55 PM, Anthonius deBoer wrote: I created the index but that did not change anything... I had sent James Taylor the schema as he requested, but I never heard anything so I guess he must not have found anything weird... I am waiting to move my Galaxy server off to a three server configuration, so I am hoping the problem will not return, although your issue does not bode well for this issue... Are the Galaxy server processes using a large amount of CPU or memory? Unfortunately, you may need to add a bit of debugging to the job code to see exactly where the delay is taking place. --nate Regards, Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeb...@me.com LinkedIn Profile On Oct 23, 2012, at 4:14 AM, Hans-Rudolf Hotz h...@fmi.ch wrote: Hi Thon Sorry for re-activating an old e-mail thread. I am just curious, whether Nate's suggestion solved your problem? After our upgrade (from the March 12th changeset 6799:40f1816d6857 to the October 5th changeset 7824:b5bda7a5c345) we see long delays in the jobs going from queued to running. For our server, switching from set_metadata_externally = False to set_metadata_externally = True did not speed up anything. Regards, Hans-Rudolf On 09/19/2012 04:27 PM, Nate Coraor wrote: On Sep 18, 2012, at 6:31 PM, Anthonius deBoer wrote: Hi, Jobs that I start on my in-house Galaxy instance now take up to 3-4 min to go from queued to Running, even though there is nothing much going on on the galaxy server... I have been running this instance since June and use a relatively new version of Galaxy-central (Last update, 22-Aug changeset: 7535:bf6517b2b336) I have noticed that my jobs table in the galaxy Postgres database contains about 60,000 jobs... Could that be the culprit? Does it slow to the complete database to see if there are any jobs that need to run? Could I purge the jobs and related tables to speed up the submissions? Hi Thon, It's not really possible to remove things from the database since there are a lot of interdependencies. The first thing to check would be whether set_metadata_externally = True in your Galaxy config. --nate Thanks, Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] The blastdb_p.loc sample is wrong.
On Wed, Oct 24, 2012 at 4:46 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Peter, Thanks very much for this - I'm sure many will find it very helpful. I've applied your patch to changeset revision 8096:0c56502c7fd7, which will be included in the next Galaxy release currently scheduled for next Friday (not today's currently scheduled release). Thanks again, Greg Von Kuster Excellent, and thanks for taking care of the line wrapping in my email (does the mailing list object to patches as attachments?). Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy, Apache, WSGI and mod_wsgi
Hi, Paul I would like to know if mod_wsgi has been considered for the deployment of Galaxy at all, and whether anyone has any positive or negative experiences with it. I can only speak for the second part of your question: I've had some experience with Apache 2 + mod_wsgi, but within a Django stack. There was a somewhat complex set up/configuration (not horrible, just tedious) - that may have been particular to our situation and/or Django. It might be easier in your Galaxy situation. After set up, I found it pretty easy to work with: - Very rarely had to deal with bugs/workarounds or modification in general. - His/Their documentation for the mod is excellent - which is an often understated positive. - I found no problems with logging, debugging, or the mod 'getting in the way' of normal Apache features. - If I recall correctly, there was a good user base out there (~2-3 years ago). Carl On Mon, Oct 22, 2012 at 9:26 AM, Paul Boddie paul.bod...@biotek.uio.nowrote: Hello, I have recently had the opportunity to look at the deployment of Galaxy together with Apache, and I saw that the recommendation is to run the Galaxy Web server behind Apache with the latter acting as a proxy: http://wiki.g2.bx.psu.edu/**Admin/Config/Apache%20Proxyhttp://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy Other than convenience - one can just put Apache in front of an existing server - is there any particular reason for doing things this way? It seems that Galaxy uses the built-in Web server provided by the Paste framework, which in turn is based on the Python standard library BaseHTTPServer framework, and although paste.httpserver seems to add capabilities to the underlying framework, each such server must still be constrained to running in a single process. I imagine that this then leads to the use of load balancing as described on the following page: http://wiki.g2.bx.psu.edu/**Admin/Config/Performance/Web%** 20Application%20Scalinghttp://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application%20Scaling Given that Apache is an acceptable part of a large-scale solution, I would like to know if mod_wsgi has been considered for the deployment of Galaxy at all, and whether anyone has any positive or negative experiences with it. It seems to me that mod_rewrite is often something that should only be brought into play where other, typically more elegant, solutions cannot be used. Many Python-based Web applications have mod_wsgi as a recommended deployment option once their users look beyond CGI, and I wondered why this isn't the case with Galaxy. Paul __**_ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy, Apache, WSGI and mod_wsgi
On 24/10/12 18:09, Carl Eberhard wrote: I would like to know if mod_wsgi has been considered for the deployment of Galaxy at all, and whether anyone has any positive or negative experiences with it. I can only speak for the second part of your question: I've had some experience with Apache 2 + mod_wsgi, but within a Django stack. There was a somewhat complex set up/configuration (not horrible, just tedious) - that may have been particular to our situation and/or Django. It might be easier in your Galaxy situation. I've administered applications on various mod_wsgi sites, but I've never had to set it up. It can't really be worse than mod_python, though, can it? After set up, I found it pretty easy to work with: - Very rarely had to deal with bugs/workarounds or modification in general. - His/Their documentation for the mod is excellent - which is an often understated positive. - I found no problems with logging, debugging, or the mod 'getting in the way' of normal Apache features. - If I recall correctly, there was a good user base out there (~2-3 years ago). I think it's even more usable than it was 2-3 years ago, which may also have been when I first had to use it. I just wondered whether there were any issues preventing its use with Galaxy. Having a persistent Galaxy-only Web server process, as the Paste-based server seems to be, might inadvertently encourage people to store things in memory (class and module globals, for example) that would mysteriously go away (and cause errors) under different deployment mechanisms, although I would imagine that doing load balancing of several Paste-based servers might provoke similar problems and so they would be known to the community. It just seemed rather involved to have to manage mod_rewrite rules or use mod_proxy when an integrated solution exists and is widely used. The only really practical reason I can think of, given that the Paste-based server is probably not well regarded for performance, is that mod_wsgi isn't packaged for certain operating system distributions like Red Hat Enterprise Linux, but having just checked, that actually isn't the case for RHEL 6: mod_wsgi.x86_64 3.2-1.el6 rhel-x86_64-server-6 Maybe the answer is that people either don't know or don't really care. :-) Paul ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Handling MS Excel output from Galaxy tool
Hello We have some tools that wrap an in-house program which generates an Excel spreadsheet as one of its outputs, and wanted Galaxy to associate the correct mime type so that the user's browser knows how to handle it on view/download. The advice on the wiki at http://wiki.g2.bx.psu.edu/Admin/FAQ#Configuring_Galaxy_so_a_tool_can_properly_generate_an_Excel_file appears to be out of date, so instead as a workaround I've edited datatypes_conf.xml and replaced the existing xls type: datatype extension=xls type=galaxy.datatypes.data:Data mimetype=application/vnd.ms-excel / This seems to work (in that clicking the eye icon now givens me a download dialogue rather than a server error message). However I'm wondering: - Would type=galaxy.datatypes.binary:Binary be a better choice? - What are the consequences of replacing the original xls datatype? - More generally, is there a way to keep the locally defined datatypes separate from the main datatypes_conf.xml file? Thanks for any advice or comments ( apologies if I've missed anything obvious on the list or on the wiki) Best wishes, Peter -- Peter Briggs peter.bri...@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy, Apache, WSGI and mod_wsgi
The Galaxy application does store quite a bit in memory (not as globals though). This doesn't preclude running under mod_wsgi, but it will work best in a configuration that uses a small number of long running processes with multiple threads. Basically, we run nginx proxying paste on the main Galaxy, so that is the configuration that is most well tested, and that is what we recommend (and can help support most easily). Other configurations might work just fine. I've used Galaxy with various other wsgi servers with issues. The performance of Paste's http server isn't really an issue since all requests that actually to paste require database access, which is typically orders of magnitude slower than Paste#http's overhead. -- jt On Wed, Oct 24, 2012 at 12:23 PM, Paul Boddie paul.bod...@biotek.uio.no wrote: Having a persistent Galaxy-only Web server process, as the Paste-based server seems to be, might inadvertently encourage people to store things in memory (class and module globals, for example) that would mysteriously go away (and cause errors) under different deployment mechanisms, although I would imagine that doing load balancing of several Paste-based servers might provoke similar problems and so they would be known to the community. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] JBrowse direct export to Galaxy
Do the APIs produce the proper CORS headers? Access-Control-Allow-Origin:
*, and responding to OPTIONS requests would be enough. Since this is an
API protected by a key, this doesn't have any real security implications to
just have hard-coded in.
Rob
On Oct 23, 2012 9:59 PM, Brad Chapman chapm...@50mail.com wrote:
Erik and Jeremy;
I am a JBrowse Dev hoping to add the ability to export data directly
from JBrowse (JavaScript) to Galaxy
The API could be used for this.
Specifically, you could do an upload for the user from a URL via the
tools API. I know that Brad Chapman (cc'd) has done this successfully
recently. Brad, can you share the parameters used to do this?
Definitely, you want to do a POST to:
/api/tools?key=YOURAPIKEY
with a JSON payload of:
{tool_id: upload1,
history_id: identifier of history to use,
params: {file_type: vcf,
dbkey: hg19,
files_0|url_paste:
http://jbrowse-server.org/path/to/file.vcf;,
files_0|NAME: file_name_for_history.vcf}}
This is all new since the last galaxy-dist release, so you'd need a
recent galaxy-central server.
Passing it back to Jeremy, there is also a Javascript wrapper around the
tools API which might help:
https://bitbucket.org/galaxy/galaxy-central/src/tip/static/scripts/mvc/tools.js
but I haven't used it myself. Hope this helps,
Brad
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Cannot install ncbi_blast_plus or other tools from Tool Shed: HTTP 500 error
The galaxy webserver can solve this by allowing cross-domain resources from whatever toolsheds have been configured. http://en.wikipedia.org/wiki/Cross-origin_resource_sharing -Joel ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] GMAP fails to install from mail toolshed
Hello Carlos, The cause of the server error is a tool shed issue which I've corrected, so you should now be able to install the gmap repository. I've installed the repository locally and it seems to be ok (althoug I haven't executed contained tools), but it's in a strange state in the tool shed probably because all of it's contents were deleted except the tool_dependencies.xml file in the repository tip: Changeset 6adc485b6dc0 added: tool_dependencies.xml removed: README gmap.xml gmap_build.xml gsnap.xml iit_store.xml lib/galaxy/datatypes/gmap.py snpindex.xml tool-data/datatypes_conf.xml tool-data/gmap_indices.loc.sample You can contact the repository owner via the Contact repository owner option in the Repository Actions pop-up menu when viewing the repository. Greg Von Kuster On Oct 24, 2012, at 12:27 PM, Carlos Borroto wrote: Hi, I get a HTTPError: HTTP Error 500: Internal Server Error(see full error trace below) when trying to install GMAP from the main toolshed. Is there a way to report this to the tool authors? Thanks, Carlos URL: http://galaxy.bfx.brel.local/admin_toolshed/prepare_for_install?tool_shed_url=http://toolshed.g2.bx.psu.edu/repository_ids=4131098bea459833changeset_revisions=561503a442f0 File '/local/opt/galaxy/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/local/opt/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/local/opt/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) File '/local/opt/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/recursive.py', line 80 in __call__ return self.application(environ, start_response) File '/local/opt/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/remoteuser.py', line 91 in __call__ return self.app( environ, start_response ) File '/local/opt/galaxy/galaxy-dist/eggs/Paste-1.6-py2.6.egg/paste/httpexceptions.py', line 632 in __call__ return self.application(environ, start_response) File '/local/opt/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line 160 in __call__ body = method( trans, **kwargs ) File '/local/opt/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 205 in decorator return func( self, trans, *args, **kwargs ) File '/local/opt/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py', line 1178 in prepare_for_install response = urllib2.urlopen( url ) File '/usr/lib64/python2.6/urllib2.py', line 126 in urlopen return _opener.open(url, data, timeout) File '/usr/lib64/python2.6/urllib2.py', line 397 in open response = meth(req, response) File '/usr/lib64/python2.6/urllib2.py', line 510 in http_response 'http', request, response, code, msg, hdrs) File '/usr/lib64/python2.6/urllib2.py', line 435 in error return self._call_chain(*args) File '/usr/lib64/python2.6/urllib2.py', line 369 in _call_chain result = func(*args) File '/usr/lib64/python2.6/urllib2.py', line 518 in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] trying to set up local toolshed
Hi David, Setting the following in your community_wsgi.ini file should enable remote authentication in the tool shed. Features like this that are available in Galaxy itself are usually also available in the tool shed (and function the same way) since the code base supporting the features is shared across both applications. Let us know if you bump into problems with this, but please keep all communication on the dev list. # User authentication can be delegated to an upstream proxy server (usually # Apache). The upstream proxy should set a REMOTE_USER header in the request. # Enabling remote user disables regular logins. For more information, see: # http://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy use_remote_user = True Thanks! Greg Von Kuster On Oct 24, 2012, at 1:53 PM, David Hoover wrote: One more question. Is there any hope of having external authentication for the local toolshed? Thanks for your (ongoing) support! David ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Trackster error
The error you saw was generated by tabix (which is used for fast random access to interval files), so it's possible that you hit tabix's upper bound. You might consider following up with the tabix folks to see what the issue is. As you noted, your dataset is definitely better formatted as a wiggle/bigwig file. Best, J. On Oct 24, 2012, at 10:14 AM, Joachim Jacob wrote: It appears the my interval file is too big, and actually in the wrong format: I think it would be better in wig (or other) format since I have only one bp intervals. Anyway, I have pasted the first 500 000 lines of the output of a liftover file (which is in interval format). https://dl.dropbox.com/u/18352887/Galaxy48-%5BSelect_first_on_data_23%5D.interval As you can see, every interval is only one bp long... Thanks Joachim On 10/24/2012 03:27 PM, Jeremy Goecks wrote: Can you please share the datasets that are generating this errors? Thanks, J. On Oct 24, 2012, at 4:19 AM, Joachim Jacob wrote: Hi all, I want to view two interval-type datasets in trackster on genome hg19 together with one BAM dataset. The interval datasets give an error (while the BAM is displayed nicely): [ti_index_core] bug in BGZF: 80048ed5001b 80048ed5 Thanks for any input, Joachim -- Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Trackster error
FYI, trackster supports BigWig directly (no conversion or additional indexing) and quite efficiently, so you will likely have the most luck with that. -- jt On Wed, Oct 24, 2012 at 3:09 PM, Jeremy Goecks jeremy.goe...@emory.edu wrote: bigwig file. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] How to get a shedtool to run locally
On Oct 19, 2012, at 9:35 AM, Carlos Borroto carlos.borr...@gmail.com wrote: On Fri, Oct 19, 2012 at 12:00 AM, Fields, Christopher J cjfie...@illinois.edu wrote: My local one is set up like this and it works (for augustus): toolshed.g2.bx.psu.edu/repos/bjoern-gruening/augustus/augustus/0.1 = pbs:///default/-l ncpus=1,mem=8gb,walltime=24:00:00/ The above is the path in within the shed repo. chris Hi Chris, This was quite helpful, although I don't know what you mean by path in within the shed repo. Apologies, not enough caffeine. But you seemed to work it out, so it all came together in the end :) The tool I was trying to specify a runner for is ncbi_blastn_wrapper form the ncbi_blast_plus repository. This is the past to the tool under the local shed_tools directory: shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/9d5beacae92b/ncbi_blast_plus/tools/ncbi_blast_plus/ncbi_blastn_wrapper.xml I did however, now that I see what is the overall look of IDs in these cases, find a way to come up with the correct ID. I put a job to run and looked in Admin/Manage jobs, the column Tool displays the correct ID of the tool running. In my case it ended being: toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastn_wrapper/0.0.13 Thanks a lot for giving me a good hint of what to look for!, Carlos It would be nice to have a shorter alias for these, but maybe that's not possible at the moment. chris ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Importing files into Libraries
Hi Niel, I'm assuming the data uploaded to a library on a galaxy cloudman instance. In that case, all the data that's uploaded to a data library (or a history) goes on an nfs-exported file system and is thus available to all the worker nodes. Cheers, Enis On Wed, Oct 24, 2012 at 3:51 PM, neil.burd...@csiro.au wrote: Hi, If we upload data and then import the data into libraries, how is that data then accessed when running a tool in the cloud. For example if we have hundreds of template files that we need to analyse the data, and we have a tool that is instantiated on a number of nodes in the clous, is the data in the libraries copied to each node? Or is the data access via nfs, symbolic link or some other method? Thanks Neil ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Importing files into Libraries
Thanks for the information. Much appreciated Neil From: Enis Afgan [mailto:eaf...@emory.edu] Sent: Thursday, 25 October 2012 7:38 AM To: Burdett, Neil (ICT Centre, Herston - RBWH) Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Importing files into Libraries Hi Niel, I'm assuming the data uploaded to a library on a galaxy cloudman instance. In that case, all the data that's uploaded to a data library (or a history) goes on an nfs-exported file system and is thus available to all the worker nodes. Cheers, Enis On Wed, Oct 24, 2012 at 3:51 PM, neil.burd...@csiro.aumailto:neil.burd...@csiro.au wrote: Hi, If we upload data and then import the data into libraries, how is that data then accessed when running a tool in the cloud. For example if we have hundreds of template files that we need to analyse the data, and we have a tool that is instantiated on a number of nodes in the clous, is the data in the libraries copied to each node? Or is the data access via nfs, symbolic link or some other method? Thanks Neil ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] FTP upload not working
Hi, I am having an issue withuploading data via FTP. I get the following error: Response:530 Sorry, the maximum number of clients (3) for this user are already connected. Error: Critical error Error: Could not connect to server Is this because I try to upload too many files in parallel. Should I upload only one at a time. Cheers Oliver -- ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Data library displays (OperationError) no such column error
Hi all, After updating my galaxy-dist to the latest release (23/Oct) I get the following error message when I try to access one of my data libraries. Error attempting to display contents of library (testlibrary): (OperationalError) no such column: True u'SELECT dataset_permissions.id AS dataset_permissions_id, dataset_permissions.create_time AS dataset_permissions_create_time, dataset_permissions.update_time AS dataset_permissions_update_time, dataset_permissions.action AS dataset_permissions_action, dataset_permissions.dataset_id AS dataset_permissions_dataset_id, dataset_permissions.role_id AS dataset_permissions_role_id XnFROM dataset_permissions XnWHERE True AND dataset_permissions.action = ?' ['access']. I was getting the same error message as the previous post (http://gmod.827538.n3.nabble.com/quot-No-module-named-controllers-library-common-quot-error-on-fresh-Galaxy-install-from-4-Oct-2012-td4025537.html) before the update and it’s now replaced with the above. I’ve also tried cloning a fresh galaxy-dist but I’m still receiving the same error message. The library exists and files are present because I can see them when I go to http://localhost:8080/api/libraries/X ... I just cannot access it via the web interface. Has anyone experienced a similar situation, or have any ideas? Many Thanks, Xin-Yi ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
