[galaxy-dev] How to get the import of BAM files to create its index on the cluster
Hi,I am looking how to improve the import of BAM files into galaxy.I often use the Upload tool in the Data Libraries to upload a number of BAM files into galaxy (after recombining several readgroups into a BAM file for the library, which I unfortunately have to do OUTSIDE of galaxy since it cannot take an arbitrary set of BAM files in a workflow)...The UPLOAD tool seems to be correctly submitted to the cluster as you would expect, but then it seems that SAMTOOLS is run on each of the BAM files to create a index file (even though the directory already has one) on the server that is running galaxy, i.e. locally!Is there a way to move the SAMTOOLS index creation into the upload tool so that it can be run in parallel on the cluster?ThanksThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Why does my trackster complain about not being able to display BED files?
Hi,I am using the latest version of galaxy at my local installation and I try to use trackster to display a simple BED file, but it invariably complains about not having a converter for this datatype and it tells me to check mydatatypes_conf.xml which I did (not sure what I should be looking for, but this is the section on BED files...Am I missing a converter? And why would a BED file even needed to be converted?ThanksThon datatype extension="bed" type="galaxy.datatypes.interval:Bed" display_in_upload="true" converter file="bed_to_gff_converter.xml" target_datatype="gff"/ converter file="interval_to_coverage.xml" target_datatype="coverage"/ converter file="bed_to_bgzip_converter.xml" target_datatype="bgzip"/ converter file="bed_to_tabix_converter.xml" target_datatype="tabix" depends_on="bgzip"/ converter file="bed_to_summary_tree_converter.xml" target_datatype="summary_tree"/ converter file="bed_to_fli_converter.xml" target_datatype="fli"/ !-- display file="ucsc/interval_as_bed.xml" / -- !-- display file="genetrack.xml" / -- display file="igb/bed.xml" / /datatype datatype extension="bedgraph" type="galaxy.datatypes.interval:BedGraph" display_in_upload="true" converter file="bedgraph_to_bigwig_converter.xml" target_datatype="bigwig"/ /datatype datatype extension="bedstrict" type="galaxy.datatypes.interval:BedStrict" / !-- datatype extension="bed6" type="galaxy.datatypes.interval:Bed6" converter file="bed_to_genetrack_converter.xml" target_datatype="genetrack"/ /datatype -- datatype extension="bed12" type="galaxy.datatypes.interval:Bed12" / datatype extension="len" type="galaxy.datatypes.chrominfo:ChromInfo" display_in_upload="true" converter file="len_to_linecount.xml" target_datatype="linecount" / /datatype datatype extension="bigbed" type="galaxy.datatypes.binary:BigBed" mimetype="application/octet-stream" dis___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Why does my trackster complain about not being able to display BED files?
Yep...$ bedtools --versionbedtools v2.17.0ThonOn Aug 21, 2013, at 01:40 PM, sam guerler aysam.guer...@gmail.com wrote:Hi Anthonius,Do you have the BEDTools installed and its directory added to your PATH environment variable? Trackster uses compressed file formats to reduce the amount of data transfers.I hope this helps,SamOn Wed, Aug 21, 2013 at 4:11 PM, Anthonius deBoer thondeb...@me.com wrote:Hi,I am using the latest version of galaxy at my local installation and I try to use trackster to display a simple BED file, but it invariably complains about not having a converter for this datatype and it tells me to check mydatatypes_conf.xml which I did (not sure what I should be looking for, but this is the section on BED files...Am I missing a converter? And why would a BED file even needed to be converted?ThanksThon datatype extension="bed" type="galaxy.datatypes.interval:Bed" display_in_upload="true" converter file="bed_to_gff_converter.xml" target_datatype="gff"/ converter file="interval_to_coverage.xml" target_datatype="coverage"/ converter file="bed_to_bgzip_converter.xml" target_datatype="bgzip"/ converter file="bed_to_tabix_converter.xml" target_datatype="tabix" depends_on="bgzip"/ converter file="bed_to_summary_tree_converter.xml" target_datatype="summary_tree"/ converter file="bed_to_fli_converter.xml" target_datatype="fli"/ !-- display file="ucsc/interval_as_bed.xml" / -- !-- display file="genetrack.xml" / -- display file="igb/bed.xml" / /datatype datatype extension="bedgraph" type="galaxy.datatypes.interval:BedGraph" display_in_upload="true" converter file="bedgraph_to_bigwig_converter.xml" target_datatype="bigwig"/ /datatype datatype extension="bedstrict" type="galaxy.datatypes.interval:BedStrict" / !-- datatype extension="bed6" type="galaxy.datatypes.interval:Bed6" converter file="bed_to_genetrack_converter.xml" target_datatype="genetrack"/ /datatype -- datatype extension="bed12" type="galaxy.datatypes.interval:Bed12" / datatype extension="len" type="galaxy.datatypes.chrominfo:ChromInfo" display_in_upload="true" converter file="len_to_linecount.xml" target_datatype="linecount" / /datatype datatype extension="bigbed" type="galaxy.datatypes.binary:BigBed" mimetype="application/octet-stream" dis___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Jobs remain in queue until restart
I don't think it's a memory issue (but what made you say that?) since each process is hardly using any memory, although VIRT memory in top is showing 2.7GB per python process, RES is only ever going to 250MB and I have a 16GB machine (although SWAP is only 4GB but not using any of the swap either, so don't think memory is the issueI AM running this on a VM machine, but the physical machine is not doing much either...I'll run profile on it to see what is causing the massive load issue...ThonOn Aug 14, 2013, at 08:51 AM, Nate Coraor n...@bx.psu.edu wrote:On Aug 2, 2013, at 1:06 PM, Thon de Boer wrote: I did some more investigation of this issueI do notice that my 4 core, 8 slot VM machine has a load of 32, with only my 4 handler processes running (Plus my web server), but not even getting more than 10% of the CPU each.There seems to be some process in my handlers that takes an incredible amount of resources, even though TOP is not showing that (Show below)Has anyone have any idea how to figure out where the bottleneck is?Is there a way to turn on more detailed logging perhaps to see what each process is doing?My IT guy suggested there may be some “context Switching” going on due to the many threads that are running (I use a threadpool of 7 for each server), but not sure how to address that issue… Hi Thon, It looks like it's probably the memory use - if you restart the Galaxy processes, do you see any change? --nate Anyone?top - 10:00:53 up 37 days, 19:29, 8 users, load average: 32.10, 32.10, 32.09Tasks: 181 total, 1 running, 180 sleeping, 0 stopped, 0 zombieCpu(s): 4.8%us, 2.5%sy, 0.0%ni, 92.5%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%stMem: 16334504k total, 16164084k used, 170420k free, 127720k buffersSwap: 4194296k total, 15228k used, 4179068k free, 2460252k cachedPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND7190 svcgalax 20 0 2721m 284m 5976 S 9.9 1.8 142:53.84 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler3 --pid-file=handler3.pid --log-file=handler3.log --daemon7183 svcgalax 20 0 2720m 286m 5984 S 6.4 1.8 135:52.63 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler2 --pid-file=handler2.pid --log-file=handler2.log --daemon7175 svcgalax 20 0 2720m 287m 5976 S 5.6 1.8 117:59.40 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler1 --pid-file=handler1.pid --log-file=handler1.log --daemon7166 svcgalax 20 0 3442m 2.7g 4884 S 4.6 17.5 74:31.66 python ./scripts/paster.py serve universe_wsgi.ini --server-name=web0 --pid-file=web0.pid --log-file=web0.log --daemon7172 svcgalax 20 0 2720m 294m 5984 S 4.0 1.8 133:17.19 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler0 --pid-file=handler0.pid --log-file=handler0.log --daemon1564 root 20 0 291m 13m 7552 S 0.3 0.1 1:49.65 /usr/sbin/httpd7890 svcgalax 20 0 17216 1456 1036 S 0.3 0.0 2:15.73 top10682 apache 20 0 297m 11m 3516 S 0.3 0.1 0:02.23 /usr/sbin/httpd11224 apache 20 0 295m 11m 3236 S 0.3 0.1 0:00.29 /usr/sbin/httpd11263 svcgalax 20 0 17248 1460 1036 R 0.3 0.0 0:00.06 top1 root 20 0 21320 1040 784 S 0.0 0.0 0:00.95 /sbin/init2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 [kthreadd]3 root RT 0 0 0 0 S 0.0 0.0 0:06.35 [migration/0]Regards,ThonThon deBoer Ph.D., Bioinformatics GuruCalifornia, USA |p: +1 (650) 799-6839 |m: thondeb...@me.comFrom: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Thon DeboerSent: Wednesday, July 17, 2013 11:31 PMTo: galaxy-dev@lists.bx.psu.eduSubject: [galaxy-dev] Jobs remain in queue until restartHi,I have noticed that from time to time, the job queue seems to be “stuck” and can only be unstuck by restarting galaxy.The jobs seem to be in the queue state and the python job handler processes are hardly ticking over and the cluster is empty.When I restart, the startup procedure realizes all jobs are in the a “new state” and it then assigns a jobhandler after which the jobs start fine….Any ideas?ThonP.S I am using the june version of galaxy and I DO set limits on my users in job_conf.xml as so: (Maybe it is related? Before it went into dormant mode, this user had started lots of jobs and may have hit the limit, but I assumed this limit was the number of running jobs at one time, right?)?xml version="1.0"?job_confplugins workers="4"!-- "workers" is the number of threads for the runner's work queue.The default from plugins is used if not defined for a plugin.--plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/plugin id="drmaa" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" workers="8"/plugin id="cli" type="runner" load="galaxy.jobs.runners.cli:ShellJobRunner" workers="2"//pluginshandlers default="handlers"!-- Additional job handlers - the id should match the name of a[server:id] in universe_wsgi.ini.--handler id="handler0" tags="handlers"/handler id="handler1" tags="handlers"/handler id="handler2" tags="handlers"/handler id="handler3" tags="handlers"/!--
[galaxy-dev] Is there a way to re-run a failed job and check the re-associated checkbox from the API?
Hi,I was very excited to see the re-run option and associate with paused jobs option working very well..I was hoping to be able to re-run a job like this from the API, but cannot seem to find any API calls that corresponds to this here:http://galaxy-dist.readthedocs.org/en/latest/lib/galaxy.webapps.galaxy.api.htmlIs this not implemented yet, or just not documented?ThanksThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Getting FATAL: sorry, too many clients already in postgres database
Hi,I am getting the error "FATAL: sorry, too many clients already" in in my local Galaxy instance in the POSTGRES database log.I have set max_connections = 1000 in already in "postgresql.conf" and STILL get this error...I was running a Flowcell with about 96 samples and a workflow with about 50 steps each, but I never had a problem before when max_connections was set to 100 and did not change the database...I do have a cluster that is much faster so i guess there may be several connections going to the database, but maybe there is some leakage in the galaxy code that is not closing the connections?It really screws up the galaxy instance and jobs are just in limbo and never finsihed (in the database)...Any ideas?I adjusteddatabase_engine_option_pool_size = 20database_engine_option_max_overflow = 40already but maybe I set those too high?What is a normal setting for max_connections in galaxy production servers?Thanks,Thon___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Bypassing the proxy server when using the API, without breaking the automatic update of the history in the web interface
Hi,I am trying desperately to setup my GALAXY system so that if you enter the website, you need to authenticate yourself using the proxy setup...I also would like to be able to BYPASS the proxy server when an API call is made...In the example proxy setting I have setup in the following way:ReWriteRule ^(/api/histories/.*) http://localhost:8080$1 [P,L]ReWriteRule ^(/api/.*) http://localhost:8081$1 [P,L]RewriteRule ^(.*) http://localhost:8080$1 [P]I then setup a PROXY setting in the apache galaxy.d in /etc/httpd/conf.d/galaxy.d to require authentication when a user is re-directed to the localhost:8080 and allow the calls to localhost:8081 to pass unhindered...This works brilliantly EXCEPT for when you want to make an API call that involves some historiesThe call fails, since /api/histories calls are routed through the proxy server and require authentication, which I want to avoid...It seems that histories are "special" so they have their own line in the configuration file above...I tried to changeReWriteRule ^(/api/histories/.*) http://localhost:8080$1 [P,L]toReWriteRule ^(/api/histories/.*) http://localhost:8081$1 [L]Which DOES work, but it also breaks the automatic update of the history in the webbrowser...I have to reload the page manually to update the history...Is there anyway I can get the automatic update to work again, but don't require autentication?ThanksThon___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] How to create BAM indices on cluster upon upload
Hi,I am uploading a lot of data to our galaxy system, and it seems that the index creation is always happening on the server instance of the galaxy server and NOT on the cluster.This seems to be particularly true for BAM files, where the index creation is being done on the galaxy server and it is done SERIALLY at thatThis means that uploading 100s of BAM files can sometimes take hour upon hours...I have a cluster with 500 cores so I was wondering how I can get GALAXY to create these indices on the cluster in a parallel fashion, in order to speed up the processIt seems that the UPLOAD process IS kicked off on the server, but then the index creation for BAM files is done afterwards on the galaxy server itself...Any help would be welcome...Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Is there a way through the API to know when a file has completed the upload process?
Hi,I am trying to use the API to upload a file and since the upload can take a long time (see my previous post) I need to check if the upload has completed so I know I can start a workflow (or can I start a workflow even when the file is still in the "queued" state?)...I have successfully used the examples from this page:http://galaxy-dist.readthedocs.org/en/latest/lib/galaxy.webapps.galaxy.api.htmland can easily upload a file, but I cannot seem to learn the state of the file...The upload process I used, returned a bunch of attributes, such as:{'ldda_id': '049d723a33542847', 'misc_blurb': None, 'name': '7_101_1_ACTTGA_R1.fastq', 'data_type': 'fastq', 'file_name': '/mnt/ngs/analysis/tdeboer/PROJECTS/STAGING/SEQUENCER07/130207_SN7001380_0101_AD1H7GACXX/fastq/7_101_1_ACTTGA_R1.fastq', 'uploaded_by': 'tdeb...@genomichealth.com', 'metadata_sequences': None, 'template_data': {}, 'genome_build': 'hg19', 'model_class': 'LibraryDataset', 'misc_info': None, 'file_size': 6410080411, 'metadata_data_lines': None, 'message': '', 'id': '049d723a33542847', 'date_uploaded': '2013-05-22T00:50:50.474318', 'metadata_dbkey': 'hg19'}But none of those tell me if the upload has succeeded or not...Is there a way to retrieve the state of an uploaded file once I have uploaded a file through the API?thanksThon___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Resubmit a job if the node fails
I have not seen any reply to this question from last year, so I wanted to re-up it again...I also run into this issue quite often and with the recent introduction of the (half completed?) feature of the notion of Paused-Jobs it seems we are getting very close to this working...I know I can re-run a failed job, but I don't think restarting a paused job that is reliant on that first job "knows" to wiat for the new job to finish, does it?But I suspect that is what the intendedfunctionalityis, since otherwise paused jobs are not very usefulI am still using the Feb-8 version of Galaxy, but don't think I saw anything in the April version that addresses this issue, right?Maybe it would be useful to make one particular error state ("Job did not return any result from the cluster" or something of that sort that I see if a cluster node fails) make Galaxy simply re-submit the job (with a fixed number of tries ofcourse, 3 seems a decent number) and keep on going, rather than immediately make the job go into error state...Thanks,ThonOn Apr 19, 2012, at 09:39 AM, zhengqiu cai caizhq2...@yahoo.com.cn wrote:Hi, Can Galaxy resubmit a job if the node where the job is running fails? I know sge can do that by using qsub -r. It should be very useful if Galaxy can do that. Thank you, Cai ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Example for DRMAA in new jobs_conf.xml example file incorrect. It's DRMAAJobRunner not DRMAARunner
It seems that the examples in the jobs_conf.xml file incorrectly defines the DRMAA runner asgalaxy.jobs.runners.drmaa:DRMAARunnerwhich should begalaxy.jobs.runners.drmaa:DRMAAJobRunnerIt's really unhelpful if a major change to the way things are run contains these kinds of issues/bugs in the examples, since there is not much on the wikki pages on what are the pre-defined runners..I had to dig down in the code to figure out the DRMAA runners name...Hope this helps others getting DRMAA to run in Galaxy after this major update...Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Is there a way to determine if a tool was run from the UI or from a Workflow step?
Hi,I would like to allow interactive use of the Galaxy UI to be run in a different "destination" from a tool that is run from a workflow...I am looking into setting up the dynamic running method in jobs_conf.xml and a special dynamic function I am defining, but not too sure how I can determine (if this is even possible) if a tool is run as part of a Workflow step or if it was run interactivelyAny pointers?Thanks ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] cleanup script fails on dataset_instance None
Hi,I have some problems with the cleanup scripts...I seem to have a dataset_instance that is "None" and if it tries to delete that you obviously get an error. (See below..I added the print statement, but the line is the same)...I wonder how I could get a dataset with a None definition and how can I clean up the database for this?Thanks,Thonsh purge_libraries.shTraceback (most recent call last): File "./scripts/cleanup_datasets/cleanup_datasets.py", line 527, in module if __name__ == "__main__": main() File "./scripts/cleanup_datasets/cleanup_datasets.py", line 122, in main purge_libraries( app, cutoff_time, options.remove_from_disk, info_only = options.info_only, force_retry = options.force_retry ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 217, in purge_libraries _purge_folder( library.root_folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 500, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 500, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 500, in _purge_folder _purge_folder( sub_folder, app, remove_from_disk, info_only = info_only ) File "./scripts/cleanup_datasets/cleanup_datasets.py", line 498, in _purge_folder _purge_dataset_instance( ldda, app, remove_from_disk, info_only = info_only ) #mark a DatasetInstance as deleted, clear associated files, and mark the Dataset as deleted if it is deletable File "./scripts/cleanup_datasets/cleanup_datasets.py", line 376, in _purge_dataset_instance print('dataset_instance.id: {0}'.format(dataset_instance.id))AttributeError: 'NoneType' object has no attribute 'id'(galaxy_env)[svcgalaxy@srv151 cleanup_datasets]$ sh purge_libraries.sh___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Submitting jobs as a real user without using chown, please
I cannot run chown even as a sudo command...Same error...What is root squashing? I am reading on the internet that it is very common not to allow users to change the ownership of files...ThonOn Feb 20, 2013, at 05:52 AM, Nate Coraor n...@bx.psu.edu wrote:On Feb 19, 2013, at 11:02 PM, Thon de Boer wrote: Hi,I am trying to setup my galaxy system to allow jobs to be submitted as the real user, since people want to keep an eye on their job on the cluster sometimes and they have no ideas which ones are theirs…I tried the approach on the wiki here:http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster?highlight=%28submit%29%7C%28jobs%29%7C%28as%29%7C%28user%29#Submitting_Jobs_as_the_Real_Userbut unfortunately, the CHOWN command is not allowed, not even as a sudo user… Probably has to do with the fact that we run our cluster from an isilon system, which I assume is pretty typical…The job was actually successfully submitted as the intended user, so that part works, but if we can just get it to work without having to rely on chown that would be awesome…Can someone point me in the right direction? Hi Thon, If you run the command from the command line, what results do you get? /usr/bin/sudo -E scripts/external_chown_script.py /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/148 svcgalaxy 1 Note that external_chown_script.py can be modified as necessary to allow you to change ownership in whatever way is appropriate for your site. Since it should just be an NFS mount, as long as root squashing is not enabled and your svcgalaxy user has sudo permission to run this script, it should succeed. --nate Here’s the error…galaxy.jobs.runners.local DEBUG 2013-02-19 19:35:31,524 execution of external set_meta for job 148 finished galaxy.jobs DEBUG 2013-02-19 19:35:31,576 (148) Changing ownership of working directory with: /usr/bin/sudo -E scripts/external_chown_script.py /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/148 svcgalaxy 1 galaxy.jobs ERROR 2013-02-19 19:35:31,653 (148) Failed to change ownership of /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/148, failing Traceback (most recent call last): File "/mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__.py", line 343, in finish self.reclaim_ownership() File "/mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__.py", line 916, in reclaim_ownership self._change_ownership( self.galaxy_system_pwent[0], str( self.galaxy_system_pwent[3] ) ) File "/mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__.py", line 902, in _change_ownership assert p.returncode == 0 AssertionError galaxy.jobs DEBUG 2013-02-19 19:35:31,722 fail(): Moved /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/148/galaxy_dataset_332.dat to /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/files/000/dataset_332.dat galaxy.datatypes.metadata DEBUG 2013-02-19 19:35:31,924 Cleaning up external metadata filesThanksThon ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Submitting jobs as a real user without using chown, please
Ah...Found what root squashing is and yes, that is turned on our isilon system...So out of luck I take it? We need to chown? we cannot fake the submission name in another way ;)ThanksThonOn Feb 20, 2013, at 10:32 AM, Anthonius deBoer thondeb...@me.com wrote:I cannot run chown even as a sudo command...Same error...What is root squashing? I am reading on the internet that it is very common not to allow users to change the ownership of files...ThonOn Feb 20, 2013, at 05:52 AM, Nate Coraor n...@bx.psu.edu wrote:On Feb 19, 2013, at 11:02 PM, Thon de Boer wrote: Hi,I am trying to setup my galaxy system to allow jobs to be submitted as the real user, since people want to keep an eye on their job on the cluster sometimes and they have no ideas which ones are theirs…I tried the approach on the wiki here:http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster?highlight=%28submit%29%7C%28jobs%29%7C%28as%29%7C%28user%29#Submitting_Jobs_as_the_Real_Userbut unfortunately, the CHOWN command is not allowed, not even as a sudo user… Probably has to do with the fact that we run our cluster from an isilon system, which I assume is pretty typical…The job was actually successfully submitted as the intended user, so that part works, but if we can just get it to work without having to rely on chown that would be awesome…Can someone point me in the right direction? Hi Thon, If you run the command from the command line, what results do you get? /usr/bin/sudo -E scripts/external_chown_script.py /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/148 svcgalaxy 1 Note that external_chown_script.py can be modified as necessary to allow you to change ownership in whatever way is appropriate for your site. Since it should just be an NFS mount, as long as root squashing is not enabled and your svcgalaxy user has sudo permission to run this script, it should succeed. --nate Here’s the error…galaxy.jobs.runners.local DEBUG 2013-02-19 19:35:31,524 execution of external set_meta for job 148 finished galaxy.jobs DEBUG 2013-02-19 19:35:31,576 (148) Changing ownership of working directory with: /usr/bin/sudo -E scripts/external_chown_script.py /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/148 svcgalaxy 1 galaxy.jobs ERROR 2013-02-19 19:35:31,653 (148) Failed to change ownership of /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/148, failing Traceback (most recent call last): File "/mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__.py", line 343, in finish self.reclaim_ownership() File "/mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__.py", line 916, in reclaim_ownership self._change_ownership( self.galaxy_system_pwent[0], str( self.galaxy_system_pwent[3] ) ) File "/mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__.py", line 902, in _change_ownership assert p.returncode == 0 AssertionError galaxy.jobs DEBUG 2013-02-19 19:35:31,722 fail(): Moved /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/148/galaxy_dataset_332.dat to /mnt/ngs/analysis/svcgalaxy/galaxy-test/database/files/000/dataset_332.dat galaxy.datatypes.metadata DEBUG 2013-02-19 19:35:31,924 Cleaning up external metadata filesThanksThon ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] History does not automatically update anymore
Hi,I have noticed that in the latest version of galaxy-dist, my history does not automatically update anymore.I have to press the refresh circle to have it update...I did change my proxy setup a little, so it would no longer require API calls to be authenticated, while access through the frontend would still require that so it may have something to do with that, but just wanted to check if others see this issue...I will send out my proxy settings in a later email, since it may help people that are struggling with the same...Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Setting up a proxy with authentication while waiving this for API calls
Hi,I have been struggling with setting up my galaxy system to allow it to use the LDAP authentication for users that come in through the frontend, while not wanting to have to use authentication when user want to access through the APII found some solution and I thought I'd share it with the world so that people may learn from this and maybe also tell me if I am doing something wrong, or worse, insecure!I setup two web servers in the universe file at localhost:8080 and localhost:8081.For the one at 8080 i require authentication, for the one at 8081 I do not in the settings for apache proxy server shown belowThe I setup a ReWriteRule for anything addressed to /api/ and route that to the one at 8081, that does not require authentication, while everything else goes through 8080 which does...It seems to work nicely, although I have a nagging feeling this leaves open some backdoors somewhere, but please tell me if I do...codeDirectory /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static Options FollowSymLinks Order Allow,Deny Allow from all/DirectoryProxy http://localhost:8080AuthType BasicAuthName "GALAXY@GHI. Please log in with your Windows account"AuthBasicProvider ldapAuthLDAPURL "ldap://__YOUR_LDAP_SERVER/ou=Users - Employees,dc=genomichealth,dc=com?sAMAccountName?sub?(objectClass=*)"AuthzLDAPAuthoritative offAuthLDAPBindDN __USERNAME__AuthLDAPBindPassword __PASSWORDRequire valid-userRequestHeader set REMOTE_USER %{AUTHENTICATE_SAMACCOUNTNAME}e/Proxy#Extended the timeout since some GALAXY actions can take a long timeTimeOut 6RewriteEngine onReWriteRule ^(/api/.*) http://localhost:8081$1 [L]RewriteRule ^/static/style/(.*) /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/june_2007_style/blue/$1 [L]RewriteRule ^/static/scripts/(.*) /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/scripts/packed/$1 [L]RewriteRule ^/static/(.*) /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/$1 [L]RewriteRule ^/favicon.ico /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/favicon.ico [L]RewriteRule ^/robots.txt /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/robots.txt [L]RewriteRule ^/admin/jobs(.*) http://localhost:8079/admin/jobs$1 [P]RewriteRule ^(.*) http://localhost:8080$1 [P]/codeThon___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Setting up a proxy with authentication while waiving this for API calls
There was a small, but important error in the proxy configuration...I had to pass the /api/ part to a proxy again with a [P] directive, not just the [L].So the correct proxy definition is:Directory /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static Options FollowSymLinks Order Allow,Deny Allow from all/DirectoryProxy http://localhost:8080AuthType BasicAuthName "GALAXY@GHI. Please log in with your Windows account"AuthBasicProvider ldapAuthLDAPURL "ldap://__YOUR_LDAP_SERVER/ou=Users - Employees,dc=genomichealth,dc=com?sAMAccountName?sub?(objectClass=*)"AuthzLDAPAuthoritative offAuthLDAPBindDN __USERNAME__AuthLDAPBindPassword __PASSWORDRequire valid-userRequestHeader set REMOTE_USER %{AUTHENTICATE_SAMACCOUNTNAME}e/Proxy#Extended the timeout since some GALAXY actions can take a long timeTimeOut 6RewriteEngine onReWriteRule ^(/api/.*) http://localhost:8081$1 [P,L]RewriteRule ^/static/style/(.*) /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/june_2007_style/blue/$1 [L]RewriteRule ^/static/scripts/(.*) /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/scripts/packed/$1 [L]RewriteRule ^/static/(.*) /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/$1 [L]RewriteRule ^/favicon.ico /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/favicon.ico [L]RewriteRule ^/robots.txt /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/robots.txt [L]RewriteRule ^/admin/jobs(.*) http://localhost:8079/admin/jobs$1 [P]RewriteRule ^(.*) http://localhost:8080$1 [P]On Feb 14, 2013, at 02:41 PM, Anthonius deBoer thondeb...@me.com wrote:Hi,I have been struggling with setting up my galaxy system to allow it to use the LDAP authentication for users that come in through the frontend, while not wanting to have to use authentication when user want to access through the APII found some solution and I thought I'd share it with the world so that people may learn from this and maybe also tell me if I am doing something wrong, or worse, insecure!I setup two web servers in the universe file at localhost:8080 and localhost:8081.For the one at 8080 i require authentication, for the one at 8081 I do not in the settings for apache proxy server shown belowThe I setup a ReWriteRule for anything addressed to /api/ and route that to the one at 8081, that does not require authentication, while everything else goes through 8080 which does...It seems to work nicely, although I have a nagging feeling this leaves open some backdoors somewhere, but please tell me if I do...codeDirectory /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static Options FollowSymLinks Order Allow,Deny Allow from all/DirectoryProxy http://localhost:8080AuthType BasicAuthName "GALAXY@GHI. Please log in with your Windows account"AuthBasicProvider ldapAuthLDAPURL "ldap://__YOUR_LDAP_SERVER/ou=Users - Employees,dc=genomichealth,dc=com?sAMAccountName?sub?(objectClass=*)"AuthzLDAPAuthoritative offAuthLDAPBindDN __USERNAME__AuthLDAPBindPassword __PASSWORDRequire valid-userRequestHeader set REMOTE_USER %{AUTHENTICATE_SAMACCOUNTNAME}e/Proxy#Extended the timeout since some GALAXY actions can take a long timeTimeOut 6RewriteEngine onReWriteRule ^(/api/.*) http://localhost:8081$1 [L]RewriteRule ^/static/style/(.*) /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/june_2007_style/blue/$1 [L]RewriteRule ^/static/scripts/(.*) /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/scripts/packed/$1 [L]RewriteRule ^/static/(.*) /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/$1 [L]RewriteRule ^/favicon.ico /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/favicon.ico [L]RewriteRule ^/robots.txt /mnt/ngs/analysis/svcgalaxy/galaxy-dist/static/robots.txt [L]RewriteRule ^/admin/jobs(.*) http://localhost:8079/admin/jobs$1 [P]RewriteRule ^(.*) http://localhost:8080$1 [P]/codeThon___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] jobs don't get killed when histories deleted?
Hi,I am continue to struggle with the proccess of killing jobs that run on the SGE cluster when deleting histories.I would assume that jobs get removed from the queue when a history they are part of is deleted but in most cases, the jobs stay on the cluster and are not killedI also never have been able to get the kill jobs in the admin section to work (using the 8-Feb version)...It just sites there, returns and the jobs are still there on the cluster and still there in the databaseIs there anything in the database I can do to really kick out those jobs?ThanksThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Circster save just hangs
I'm getting this in the console...SharedContentScript::onBackgroundPageMessage: init {"oktaDomain":{},"settings":{"pluginFormFillCaptureEnabled":false},"matchedFullCredSitesNoPw":[]} log.js:1Uncaught TypeError: Cannot call method 'get' of undefined circster:6Uncaught TypeError: Cannot call method 'set' of undefined circster:25Failed to load resource: the server responded with a status of 500 (Internal Server Error) http://srv151/api/datasets/251653b1722255b8?data_type=genome_dataFailed to load resource: the server responded with a status of 500 (Internal Server Error) http://srv151/api/datasets/0ee5c74dd2bafd2c?data_type=genome_dataFailed to load resource: the server responded with a status of 500 (Internal Server Error) http://srv151/api/datasets/afa56ed7242d6156?data_type=genome_dataUncaught ReferenceError: view is not defined On Feb 13, 2013, at 06:00 AM, Jeremy Goecks jeremy.goe...@emory.edu wrote:I will try to reproduce. In the meantime, can you please check the _javascript_ console and forward along any errors that you see? Thanks, J. On Feb 12, 2013, at 11:15 PM, Anthonius deBoer wrote: Hi,I had created a trackster vizualization that had not finished indexing yet, and I decided to change it into a circster visualization. I then added a few more BAM files and tried to save the visualization and now it just hangs there...no errors in the logs so far, but no saving of the visualization either...Thon ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:http://lists.bx.psu.edu/ ___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Reloading a tools configuration does not seem to actually work
Would this problem go away if I only used ONE webserver in my proxy setting and not use a balancer?I may not need it and if it means I could not use the toolshed effectively, it may be worth it not to use balancing..What i really need is the manager being balanced, since submitting a workflow with 100's of BAM files can take hours just to get started...ANy change we can run multiple managers?ThanksOn Feb 13, 2013, at 07:59 AM, Nate Coraor n...@bx.psu.edu wrote:On Feb 7, 2013, at 2:43 PM, Anthonius deBoer wrote: That's very unfortunate... I have a ton of tools and I guess now I have to create a package for them in a local toolshed to update them in a running galaxy server?In any case...The toolshed installation also does not work for me...I still have to restart galaxy, even after using the toolshed approach to install a tool...It either does not show up at all or give a bunch of errors, about not being able to find the tool...Is this also related to the fact I have two webservers and am behind a proxy server as well? Hi Thon, Essentially yes, there is no way for one web process to communicate with others that it has installed a tool. We'd like to allow for this sort of notification via a message queue, but we don't have a proper message queue in Galaxy right now. --nate ThonOn Feb 07, 2013, at 05:29 AM, Dannon Baker dannonba...@me.com wrote:Unfortunately not, and with the migration of tools to the toolshed installation mechanism I don't imagine this will be addressed (at least by the team) anytime soon. If you wanted you could probably write a script that would reload a specified tool in each of the separate web processes, or just implement a complete rolling restart of your web processes to avoid service disruption while still loading the tool updates.-Dannon On Feb 6, 2013, at 8:40 PM, Anthonius deBoer thondeb...@me.com wrote: I am indeed using multiple web processes and I guess I am talking about the "old" admine tool reloader... Is there any other way to do this for your own tools that you just manually place in tools etc.? Thon On Feb 05, 2013, at 06:22 PM, Dannon Baker dannonba...@me.com wrote: Are you using multiple web processes, and are you referring to the old admin tool reloader or the toolshed reloading interface? -Dannon On Feb 5, 2013, at 9:13 PM, Anthonius deBoer thondeb...@me.com wrote: Hi,I find that reloading a tool's configuration file does not really work.First, you have to click the reload buttow twice to actually have it update the VERSION number (so it does read something)...But when I try to run my tool, the old bug is still there...I am using proxy server so something may still be cached, but I have to restart my server for it actually to pick up the changes...Any ideas?Thon___Please keep all replies on the list by using "reply all"in your mail client. To manage your subscriptions to thisand other Galaxy lists, please use the interface at:http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Exception seen in update_manager.py; global name 'suc' is not defined
Hi,I see this in the log file...this really "suc"ks :)ThonException in thread Thread-1:Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/mnt/ngs/analysis/svcgalaxy/galaxy-dist/lib/galaxy/tool_shed/update_manager.py", line 28, in __restarter if self.check_for_update( repository ): File "/mnt/ngs/analysis/svcgalaxy/galaxy-dist/lib/galaxy/tool_shed/update_manager.py", line 37, in check_for_update tool_shed_url = suc.get_url_from_repository_tool_shed( self.app, repository )NameError: global name 'suc' is not defined___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Reloading a tools configuration does not seem to actually work
Interesting!But maybe it is not the manager that is the bottleneck, but which process is the one that takes the inputs from the webpage (or in the API) and prepares all the files that are produced by the workflows?I guess it is the MAIN server?THAT is the one that is the bottleneck I think then...Are there any plans to make that process parallel?What I am facing, is that I have 100 FASTQ pairs or so, for a single flowcell, I can start the analysis of that set from the UI, but it will just crank through them and takes about 2-4 minutues for each pair to be processed, so with 100 pairs or so, you are looking 3-4 hours of an hourglass before control is given back to the user...It would be ideal if we would farm out the parsing of the web input to the cluster, so the creation of the histories and datasets is no longer the bottleneckHow do others deal with large sequencing projects on this list?Are there any groups that routinely do 100-200 pairs of FASTQ files from within galaxy?I would like to hear their experience...ThanksThonOn Feb 13, 2013, at 01:44 PM, Nate Coraor n...@bx.psu.edu wrote:On Feb 13, 2013, at 4:12 PM, Anthonius deBoer wrote: Would this problem go away if I only used ONE webserver in my proxy setting and not use a balancer? I may not need it and if it means I could not use the toolshed effectively, it may be worth it not to use balancing.. No, because your manager/handlers will still need to be restarted to pick up the new tools. The only way you can have it work without restarting is to have everything in a single process. What i really need is the manager being balanced, since submitting a workflow with 100's of BAM files can take hours just to get started...ANy change we can run multiple managers? I'm surprised the manager is the holdup here since all it does is assign a handler. Regardless, in the development branch (which will be the next stable/release branch), the manager has been killed, and handlers are assigned directly by the web processes. --nate ThanksOn Feb 13, 2013, at 07:59 AM, Nate Coraor n...@bx.psu.edu wrote:On Feb 7, 2013, at 2:43 PM, Anthonius deBoer wrote: That's very unfortunate... I have a ton of tools and I guess now I have to create a package for them in a local toolshed to update them in a running galaxy server? In any case...The toolshed installation also does not work for me...I still have to restart galaxy, even after using the toolshed approach to install a tool...It either does not show up at all or give a bunch of errors, about not being able to find the tool... Is this also related to the fact I have two webservers and am behind a proxy server as well?Hi Thon,Essentially yes, there is no way for one web process to communicate with others that it has installed a tool. We'd like to allow for this sort of notification via a message queue, but we don't have a proper message queue in Galaxy right now.--nateThon On Feb 07, 2013, at 05:29 AM, Dannon Baker dannonba...@me.com wrote: Unfortunately not, and with the migration of tools to the toolshed installation mechanism I don't imagine this will be addressed (at least by the team) anytime soon. If you wanted you could probably write a script that would reload a specified tool in each of the separate web processes, or just implement a complete rolling restart of your web processes to avoid service disruption while still loading the tool updates. -Dannon On Feb 6, 2013, at 8:40 PM, Anthonius deBoer thondeb...@me.com wrote: I am indeed using multiple web processes and I guess I am talking about the "old" admine tool reloader...Is there any other way to do this for your own tools that you just manually place in tools etc.?ThonOn Feb 05, 2013, at 06:22 PM, Dannon Baker dannonba...@me.com wrote:Are you using multiple web processes, and are you referring to the old admin tool reloader or the toolshed reloading interface?-DannonOn Feb 5, 2013, at 9:13 PM, Anthonius deBoer thondeb...@me.com wrote: Hi, I find that reloading a tool's configuration file does not really work. First, you have to click the reload buttow twice to actually have it update the VERSION number (so it does read something)... But when I try to run my tool, the old bug is still there... I am using proxy server so something may still be cached, but I have to restart my server for it actually to pick up the changes... Any ideas? Thon ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in yo
[galaxy-dev] Why are the search options in datasets no longer using AND logic?
Hi,One very cool feature was to be able to search quickly for datasets using the search feature.You could continue to add terms until your search results in the correct set...It was basically doing an AND search for the terms you entered.This feature has seem to have disappeared?Now a search term replaces your previous term so you can no longer pair your search down and there does not seem to be any support for using AND or Is this a bug or a new feature?ThanksThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Circster save just hangs
Hi,I had created a trackster vizualization that had not finished indexing yet, and I decided to change it into a circster visualization.I then added a few more BAM files and tried to save the visualization and now it just hangs there...no errors in the logs so far, but no saving of the visualization either...Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] hg clone link on news brief incorrect
I think the hg clone link on the news brief is incorrect:it states: hgclonehttps://bitbucket.org/galaxy-dist#stableProbably should behgclonehttps://bitbucket.org/galaxy/galaxy-dist#stableThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Reloading a tools configuration does not seem to actually work
That's very unfortunate...I have a ton of tools and I guess now I have to create a package for them in a local toolshed to update them in a running galaxy server?In any case...The toolshed installation also does not work for me...I still have to restart galaxy, even after using the toolshed approach to install a tool...It either does not show up at all or give a bunch of errors, about not being able to find the tool...Is this also related to the fact I have two webservers and am behind a proxy server as well?ThonOn Feb 07, 2013, at 05:29 AM, Dannon Baker dannonba...@me.com wrote:Unfortunately not, and with the migration of tools to the toolshed installation mechanism I don't imagine this will be addressed (at least by the team) anytime soon. If you wanted you could probably write a script that would reload a specified tool in each of the separate web processes, or just implement a complete rolling restart of your web processes to avoid service disruption while still loading the tool updates. -Dannon On Feb 6, 2013, at 8:40 PM, Anthonius deBoer thondeb...@me.com wrote: I am indeed using multiple web processes and I guess I am talking about the "old" admine tool reloader... Is there any other way to do this for your own tools that you just manually place in tools etc.?ThonOn Feb 05, 2013, at 06:22 PM, Dannon Baker dannonba...@me.com wrote:Are you using multiple web processes, and are you referring to the old admin tool reloader or the toolshed reloading interface?-DannonOn Feb 5, 2013, at 9:13 PM, Anthonius deBoer thondeb...@me.com wrote: Hi, I find that reloading a tool's configuration file does not really work. First, you have to click the reload buttow twice to actually have it update the VERSION number (so it does read something)... But when I try to run my tool, the old bug is still there... I am using proxy server so something may still be cached, but I have to restart my server for it actually to pick up the changes... Any ideas? Thon ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Reloading a tools configuration does not seem to actually work
I am indeed using multiple web processes and I guess I am talking about the "old" admine tool reloader...Is there any other way to do this for your own tools that you just manually place in tools etc.?ThonOn Feb 05, 2013, at 06:22 PM, Dannon Baker dannonba...@me.com wrote:Are you using multiple web processes, and are you referring to the old admin tool reloader or the toolshed reloading interface? -Dannon On Feb 5, 2013, at 9:13 PM, Anthonius deBoer thondeb...@me.com wrote: Hi,I find that reloading a tool's configuration file does not really work. First, you have to click the reload buttow twice to actually have it update the VERSION number (so it does read something)... But when I try to run my tool, the old bug is still there...I am using proxy server so something may still be cached, but I have to restart my server for it actually to pick up the changes...Any ideas?Thon ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] How to use the API when using a proxy server
Hi Carl,I am using the submit and get functions from the scripts/api/common.pyI actually found that I just need to add authentication to the calls, so I added thiscodedef authenticate(url, u, p): auth_handler = urllib2.HTTPBasicAuthHandler() auth_handler.add_password(realm='GALAXY@GHI. Please log in with your Windows account', uri=url, user=u, passwd=p) opener = urllib2.build_opener(auth_handler) # ...and install it globally so it can be used with urlopen. urllib2.install_opener(opener)def make_url( api_key, url, args=None ): # Adds the API Key to the URL if it's not already there. if args is None:args = [] argsep = '' if '?' not in url:argsep = '?' if '?key=' not in url and 'key=' not in url:args.insert( 0, ( 'key', api_key ) ) return url + argsep + ''.join( [ '='.join( t ) for t in args ] )def get( api_key, url, user, pw ): # Do the actual GET. url = "" api_key, url ) authenticate(url, user, pw) try:return simplejson.loads( urllib2.urlopen( url ).read() ) except simplejson.decoder.JSONDecodeError, e:print "URL did not return JSON data"sys.exit(1).../codeThis will authenticate for each call and it seems to work nicely (if you add the authenticate step to each of the functions, like I did here for getThonOn Jan 31, 2013, at 11:44 AM, Carl Eberhard carlfeberh...@gmail.com wrote:Hello Anthonius,Can you elaborate on how you're invoking the API? How are you calling it?Thanks,CarlOn Wed, Jan 30, 2013 at 6:30 PM, Anthonius deBoer thondeb...@me.com wrote:Hi,I am trying to use the API with my galaxy server running behind a proxy server.How do I pass username/password information to the API so it can actually run?If I use the regular API calls I get a access denied...ThanksThon___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] How do I allow a large group of users to RUN my workflow without them having to make a local copy
Hi,I am trying to build some workflows for people that are part of a group and I want to share that workflow with them, so that they can run the workflow without having to make a local copy for them self.It seems I can only share a workflow in that manner with an individual, and not with a group...Am I reading that correctly?It seems strange not to be able to share workflows with groups if you can share them with individual users..I also have to know the exact username (email) for each user, rather than being able to choose it from a list...Is there any reason for this or is just underdeveloped?ThanksThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] iterating over files
Nope...Still only through the API can you access multiple, related files in a workflow...There is some ugly hack where you have to select the related files, and you need to hack some of the code, but if you are willing to do some editing of the source code, I believe there issomethingin this mailing list...I tried to look it up, but could no longer find it but maybe you have better luck...It is something about allowing for multiple inputsGood luck...(I can give you some api code if you need it, but I think I already sent it out to this list some time ago, so may also be in the archives...Look for me...ThonOn Jan 31, 2013, at 04:05 PM, mark.r...@syngenta.com wrote:HelloI’ve seen posts from a couple of years ago asking about galaxy’s ability to iterate over multiple files. The reply was that the developers were working on it. Have they succeeded and if so how can it now be done.ThanksMark This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited.___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Still seeing this CRITICAL message in the logs when its trying to delete the working directories
Hi,I continue to see this error in the logs,every timea jobs finishes:galaxy.objectstore CRITICAL 2013-01-30 11:42:14,161 /mnt/ngs/analysis/svcgalaxy/DATA/job_working_directory/000/327 deleteerror [Errno 39] Directory not empty: '/mnt/ngs/analysis/svcgalaxy/DATA/job_working_directory/000/327'When I check the directory, it isof courseempty:$ls -la /mnt/ngs/analysis/svcgalaxy/DATA/job_working_directory/000/327total 68drwxrwsr-x 2 svcgalaxy Domain Linux Users 0 Jan 30 11:42 .drwxrwsr-x 283 svcgalaxy Domain Linux Users 5793 Jan 30 11:45 ..Any ideas why this keeps happening?I end up with thousands of directories in the working directories which I need to clean out manually now...ThanksThon___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] How to use the API when using a proxy server
Hi,I am trying to use the API with my galaxy server running behind a proxy server.How do I pass username/password information to the API so it can actually run?If I use the regular API calls I get a access denied...ThanksThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Job handler keeps crashing
I have seen this same issue exactly. Python just dies without any errors in the log. Using the latest galaxy-dist Sent from my iPhone On Jan 20, 2013, at 8:35 PM, Derrick Lin klin...@gmail.com wrote: Update to the 11 Jan 2013 dist does not help with this issue. :( I checked the database and have the look at the job entries that handler0 tried to stop then shutdown: | 3088 | 2013-01-03 14:25:38 | 2013-01-03 14:27:05 |531 | toolshed.g2.bx.psu.edu/repos/kevyin/homer/homer_findPeaks/0.1.2| 0.1.2| deleted_new | Job output deleted by user before job completed. | NULL | NULL | NULL| NULL | NULL | NULL | 1659 | drmaa://-V -j n -R y -q intel.q/ | NULL | NULL | 76 |0 | NULL| NULL | handler0 | NULL | | 3091 | 2013-01-04 10:52:19 | 2013-01-07 09:14:34 |531 | toolshed.g2.bx.psu.edu/repos/kevyin/homer/homer_findPeaks/0.1.2| 0.1.2| deleted_new | Job output deleted by user before job completed. | NULL | NULL | NULL| NULL | NULL | NULL | 1659 | drmaa://-V -j n -R y -q intel.q/ | NULL | NULL | 76 |0 | NULL| NULL | handler0 | NULL | | 3093 | 2013-01-07 22:02:21 | 2013-01-07 22:16:27 |531 | toolshed.g2.bx.psu.edu/repos/kevyin/homer/homer_pos2bed/1.0.0 | 1.0.0| deleted_new | Job output deleted by user before job completed. | NULL | NULL | NULL| NULL | NULL | NULL | 1749 | drmaa://-V -j n -R y -q intel.q/ | NULL | NULL | 76 |0 | NULL| NULL | handler0 | NULL | So basically the job table has several of these entries what assigned to handler0 and marked as deleted_new. When the handler0 is up, it starts stopping these jobs, after the first job has been stopped, handler0 went crash and died. But that job was then marked as deleted. I think if I manually change the job state from deleted_new to deleted in the db, the handler0 will become fine. I am just concerned about how these jobs were created (like assigned to a handler but marked as deleted_new). Cheers, D On Mon, Jan 21, 2013 at 1:49 PM, Derrick Lin klin...@gmail.com wrote: I had a close look at the code in galaxy-dist / lib / galaxy / jobs / handler.py galaxy-dist / lib / galaxy / jobs / runners / drmaa.py and found that stopping deleted and deleted_new seems to be normal routine for the job handler. Could not find any exception that caused the shutdown. I do notice in the galaxy-dist on bitbucket, there is one commit with comment Fix shutdown on python = 2.6.2 by calling setDaemon when creating threads (these are still..., it seems to be relevant? I will do the update to 11 Jan release and see if it fixes the issue. D On Fri, Jan 18, 2013 at 4:03 PM, Derrick Lin klin...@gmail.com wrote: Hi guys, We have updated our galaxy to 20 Dec 2012 release. Recently we found that some submitted jobs could not start (stay gray forever). We found that it was caused by the job manager sent jobs to a handler (handler0) whose python process crashed and died. From the handler log we found the last messages right before the crash: galaxy.jobs.handler DEBUG 2013-01-18 15:00:34,481 Stopping job 3032: galaxy.jobs.handler DEBUG 2013-01-18 15:00:34,481 stopping job 3032 in drmaa runner We restarted the galaxy, handler0 is up for few seconds then died again with the same error messages except the job number moved to the next one. We observed that the jobs it was trying to stop are all previous jobs whose status is either deleted or deleted_new. We have never seen this in the past, so wondering if there is bugs in the new release? Cheers, Derrick ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] NameError in update_manager.py
Hi,I am seeing this in the log files when I start up...It seems to identify a name error in the file update_manager.pyI am not sure if it is related, but some of my manager processes are just dying without any errors in the log...The processes are just killed (The PID is just gone, so run.sh --stop_daemon tells me they are not valid)I did notice that python library is /usr/lib64/python2.6/threading.py, but I am using a virtual_env installation so I would have assumed it would have been using the lib64 stuff in my virtual_env environment, but then again, there is no file threading.py in that library so I guess it is not making a complete copy of all the files...In any case, my galaxy system is very unstable, so any help would be appreciated...Thongalaxy.tool_shed.update_manager INFO 2013-01-19 12:19:02,167 Update manager restarter starting up...Exception in thread Thread-1:Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib64/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "/mnt/ngs/analysis/svcgalaxy/galaxy-dist/lib/galaxy/tool_shed/update_manager.py", line 25, in __restarter .filter( and_( self.app.model.ToolShedRepository.table.c.update_available == False,NameError: global name 'and_' is not definedgalaxy.datatypes.registry DEBUG 2013-01-19 12:19:02,175 Loaded display application 'ucsc_bam' for datatype 'bam', i___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Error trying to remove trackster visualization in latest galaxy-dist
Hi,Tried to remove a trackster visualization in the latest galaxy-dist and got this error:Error Traceback:View as: Interactive|Text|XML(full)⇝NameError: global name 'util' is not definedURL:http://srv151/visualization/list?f-sharing=Allsort=-update_timef-tags=Alloperation=Deletef-title=Allid=f2db41e1fa331b3eModuleweberror.evalexception.middleware:364inrespond viewapp_iter=self.application(environ,detect_start_response)Modulepaste.debug.prints:98in__call__ viewenviron,self.app)Modulepaste.wsgilib:539inintercept_output viewapp_iter=application(environ,replacement_start_response)Modulepaste.recursive:80in__call__ viewreturnself.application(environ,start_response)Modulegalaxy.web.framework.middleware.remoteuser:91in__call__ viewreturnself.app(environ,start_response)Modulepaste.httpexceptions:632in__call__ viewreturnself.application(environ,start_response)Modulegalaxy.web.framework.base:160in__call__ viewbody=method(trans,**kwargs)Modulegalaxy.web.framework:94indecorator viewreturnfunc(self,trans,*args,**kwargs)Modulegalaxy.webapps.galaxy.controllers.visualization:320inlist viewids=util.listify(kwargs['id'])NameError: global name 'util' is not definedExtra FeaturesDisplay the lines of code near each part of the tracebackShow a debug prompt to allow you to directly debug the code at the tracebackRe-GET Page___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Can't see history settings, reload etc. after recent update
Hi,I just updated to the latest galaxy-dist, but now my history pane is not showing the options and no reload etc.Attached a picture, not sure it will come through...Any ideas? Now my galaxy is totally broken...I was hoping not to have to worry about it as much by switching to the galaxy-dist,from galaxy-central.Hope I can revert the changesThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Can't see history settings, reload etc. after recent update
Never mind...I was pointing to some older galaxy-central scripts in my proxy setup...It's all good now :)ThonOn Jan 14, 2013, at 08:03 PM, Anthonius deBoer thondeb...@me.com wrote:Hi,I just updated to the latest galaxy-dist, but now my history pane is not showing the options and no reload etc.Attached a picture, not sure it will come through...Any ideas? Now my galaxy is totally broken...I was hoping not to have to worry about it as much by switching to the galaxy-dist,from galaxy-central.Hope I can revert the changesThonCapture.JPG___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Problem in Manage data Library
I have a similar problem in the latest galaxy-dist version from Dec 3rd.When I run BWA or BOWTIE2 (but it probably is not limited to those) the resulting SAM/BAM file is shown to be empty and none of the metadata has been updated.When I do a manual "Auto-detect" in the "Edit Attributes" section of the dataset, it correctly detects the size and other metadata of the dataset...It seems to me that the set metadata is not running after a dataset is produced.I tried bothset_metadata_externally = True andset_metadata_externally = False but this did not make a difference...Any ideas?Thanks,ThonOn Dec 06, 2012, at 04:03 AM, Sachit Adhikari sachit.techner...@gmail.com wrote:We have found that, in Galaxy, when we add dataset to a library under Manage data libraries, Galaxy sometimes detects the type of the file and sometimes does not. What I have noticed is that it detects the type of the file when we select "Copy file into Galaxy" under "Copy data into Galaxy?" option and does not detect it when we select "Link to files ..." Is this a known issue?Regards,Sachit___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] pysam egg shows possible binary incompatibility
Hi,I installed a fresh copy of galaxy-dist on my VM machines and I am seeing this as the first line of the log files./mnt/ngs/analysis/svcgalaxy/galaxy-dist/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg/pysam/__init__.py:1: RuntimeWarning: __builtin__.file size changed, may indicate binary incompatibility from csamtools import *I am also seeing some issues with the metadata update of SAM files which I suspect is related to this error...What does this incompatibility message mean and is this the cause of the failed auto update of the metadata (If I run the update metadata manually, it runs fine)Thanks,Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Tool runners don't listen to settings in [galaxy:tool_runners]
Hi,In my fresh copy of galaxy-dist I am having a problem with tools being run only with the default tool runner and not with the assigned tool runner I created in the [galaxy:tool_runners] section...This is the setting for bwa_wrapper in my universe_wsgi.ini:bwa_wrapper = drmaa://-V -q long.q -pe smp 4/and this is what the job admin page shows:It's in a different queue and does not use the smp setting I ask for...(It's the default setting)any ideas?7X26 minutes agotoolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/bwa_wrapper/1.2.3runningpython /mnt/ngs/analysis/svcgalaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/ffa8aaa14f7c/bwa_wrappers/bwa_wrapper.py --threads="4" --fileSource=indexed --ref="/mnt/ngs/analysis/svcgalaxy/DATA/galaxy-indexes/bwa-indexes/human_g1k_v37_decoy.fasta" --do_not_build_index --input1=/home/tdeboer/P/MUT190-MAIN/fastq-2_205/2_205_1_AAACAT_R1.fastq --input2=/home/tdeboer/P/MUT190-MAIN/fastq-2_205/2_205_1_AAACAT_R2.fastq --output=/mnt/ngs/analysis/svcgalaxy/DATA/files/000/dataset_776.dat --genAlignType=paired --params=pre_set --suppressHeader=falsedrmaa://-V -q all.q -pe smp 1/4440144___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Tool runners don't listen to settings in [galaxy:tool_runners]
Hi Ross,I was indeed expecting something like that and when I tried what you suggestyed, it worked :)you need to use the complete path to the tool (including version number??!?!) in the tool-runner definition to make it be able to be set in the universe_wsgi.ini...So indeed the toolshed move of BWA breaks the individual tool_runner settings and makes for one ugly config file now...But at least we found a workaround...ThanksThonOn Dec 05, 2012, at 01:52 PM, Ross ross.laza...@gmail.com wrote:Hi, ThonI'm guessing here: but now that bwa is kept in the toolshed, the tool name appears to betoolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/bwa_wrapper/1.2.3according to your job runner entry - so I wonder if that whole ugly string now needs to be in the tool_runners section to over-ride the default rather thant "bwa_wrapper" ?This might be an unforseen consequence of the way the toolshed adapts tool names to keep versions under control. Fixing this might be a little tricky since one goal is to keep multiple tool versions separate - but we will all want all versions of bwa_wrapper to use the same runner flags?On Thu, Dec 6, 2012 at 8:40 AM, Anthonius deBoer thondeb...@me.com wrote:Hi,In my fresh copy of galaxy-dist I am having a problem with tools being run only with the default tool runner and not with the assigned tool runner I created in the [galaxy:tool_runners] section...This is the setting for bwa_wrapper in my universe_wsgi.ini:bwa_wrapper = drmaa://-V -q long.q -pe smp 4/and this is what the job admin page shows:It's in a different queue and does not use the smp setting I ask for...(It's the default setting)any ideas?7X26 minutes agotoolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/bwa_wrapper/1.2.3runningpython /mnt/ngs/analysis/svcgalaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/ffa8aaa14f7c/bwa_wrappers/bwa_wrapper.py --threads="4" --fileSource=indexed --ref="/mnt/ngs/analysis/svcgalaxy/DATA/galaxy-indexes/bwa-indexes/human_g1k_v37_decoy.fasta" --do_not_build_index --input1=/home/tdeboer/P/MUT190-MAIN/fastq-2_205/2_205_1_AAACAT_R1.fastq --input2=/home/tdeboer/P/MUT190-MAIN/fastq-2_205/2_205_1_AAACAT_R2.fastq --output=/mnt/ngs/analysis/svcgalaxy/DATA/files/000/dataset_776.dat --genAlignType=paired --params=pre_set --suppressHeader=falsedrmaa://-V -q all.q -pe smp 1/4440144___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Running galaxy on multiple, physical servers
Hi,I am trying to figure out how to run galaxy distributed over several physical servers, with different IP addresses...The wiki on scaling and load balancing only speaks of balancing on the same physical server, but does not address how to put a handful of managers on a different physical server...Any ideas how to do this?I guess I can just copy the .ini file, define managers in that and try to start it on that server, but I am hesitant until I know this can work at all...Thankshttp://wiki.galaxyproject.org/Admin/Config/Performance/Web%20Application%20ScalingThon! ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Key error in latest build of galaxy-central
Got this when trying to install GMAP from the tool shed...Any ideas?Error Traceback:View as: Interactive|Text|XML(full)⇝KeyError: 'includes_repository_dependencies'URL:http://srv151/admin_toolshed/prepare_for_install?tool_shed_url=http://toolshed.g2.bx.psu.edu/repository_ids=4131098bea459833changeset_revisions=561503a442f0Moduleweberror.evalexception.middleware:364inrespond viewapp_iter=self.application(environ,detect_start_response)Modulepaste.debug.prints:98in__call__ viewenviron,self.app)Modulepaste.wsgilib:539inintercept_output viewapp_iter=application(environ,replacement_start_response)Modulepaste.recursive:80in__call__ viewreturnself.application(environ,start_response)Modulegalaxy.web.framework.middleware.remoteuser:91in__call__ viewreturnself.app(environ,start_response)Modulepaste.httpexceptions:632in__call__ viewreturnself.application(environ,start_response)Modulegalaxy.web.framework.base:160in__call__ viewbody=method(trans,**kwargs)Modulegalaxy.web.framework:206indecorator viewreturnfunc(self,trans,*args,**kwargs)Modulegalaxy.webapps.galaxy.controllers.admin_toolshed:1070inprepare_for_install viewincludes_repository_dependencies=util.string_as_bool(repo_information_dict['includes_repository_dependencies'])KeyError: 'includes_repository_dependencies'___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Multiple inputs in a workflow (variable number of inputs) ?
Thanks JOhn for the tip... I take it it was disabled since it is still in development? I tried it, but you have to manually match the pairs it looks like to me? In my API way of running paired end, I actively search for the matching read 2 for every read1 I get (_R1 and _R2 suffix)...We need to be able to match the filenames for files somehow (I know galaxy eschews the ideas of actual names for the output it creates, so probably not something galaxy would want to implement, but for automated workflows were I routinely get flowcells with 256 FASTQ PAIRS, manually matching which pairs should be matched, is not a viable option Using the names of files is the only option and maybe using a regular expression to match files would be the most generic implementation (but it would rely on names of the files being informative and not 2: FASTQ Groomer on data 1 etc. Regards, Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeb...@me.com LinkedIn Profile On Nov 12, 2012, at 6:29 AM, John Chilton chil...@msi.umn.edu wrote: The general case is hard and it was brought up repeatedly at the Galaxy community conference. One path might be this work (https://bitbucket.org/galaxyp/galaxy-central-homogeneous-composite-datatypes) on defining datasets that represent homongenous collections of files, but there are no easy answers currently. As for the comment about paired-end sequence workflows, such workflows should actually be doable now in Galaxy (since pull request 75). All you need to do is comment out line 40 of templates/workflow/run.mako and then you should be good to go. -John On Sat, Nov 10, 2012 at 4:04 PM, Anthonius deBoer thondeb...@me.com wrote: Very likely you have to use the API for these kinds of workflows. The UI for Galaxy is very limited in workflows that deviate from the One Input paradigm. Even for the most common workflow (paired end DNA sequencing of multiple samples ) you have to resort to using the API...Puzzling... Regards, Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeb...@me.com LinkedIn Profile On Nov 8, 2012, at 5:13 AM, LUO Yufei yufei@versailles.inra.fr wrote: Hello, The number of inputs is not fixed for my workflow, maybe I have 6 inputs, maybe more than that, the number of inputs depends on what data I have for the analysis. As I know, in a workflow, all inputs should be pre-defined...that is to say, the number of inputs is known before running one workflow. Do you have any idea for this problem? How can I do to make a workflow with the variable number of inputs? In advance, thanks for the answer, Yufei ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Multiple inputs in a workflow (variable number of inputs) ?
Very likely you have to use the API for these kinds of workflows. The UI for Galaxy is very limited in workflows that deviate from the One Input paradigm. Even for the most common workflow (paired end DNA sequencing of multiple samples ) you have to resort to using the API...Puzzling... Regards, Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeb...@me.com LinkedIn Profile On Nov 8, 2012, at 5:13 AM, LUO Yufei yufei@versailles.inra.fr wrote: Hello, The number of inputs is not fixed for my workflow, maybe I have 6 inputs, maybe more than that, the number of inputs depends on what data I have for the analysis. As I know, in a workflow, all inputs should be pre-defined...that is to say, the number of inputs is known before running one workflow. Do you have any idea for this problem? How can I do to make a workflow with the variable number of inputs? In advance, thanks for the answer, Yufei ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Always seem to see this [Errno 39] Directory not empty error when a step fails
Hi,I always see this error pop up in the logs when one of the steps failed...It seems to have a problem deleting the working directory since it claims they are not empty.Ofcourse when I check, the directory is empty, but I am left with a gazilion empty working directories, alhtough no problems with the installation otherwise...Using the latest pullAny ideas?Thanks,Thongalaxy.datatypes.metadata DEBUG 2012-11-08 15:43:29,532 Cleaning up external metadata filesgalaxy.objectstore CRITICAL 2012-11-08 15:43:29,621 /mnt/ngs/analysis/svcgalaxy/DATA/job_working_directory/000/105 delete error [Errno 39] Directory not empty: '/mnt/ngs/analysis/svcgalaxy/DATA/job_working_directory/000/105'ls -la /mnt/ngs/analysis/svcgalaxy/DATA/job_working_directory/000/105total 36Kdrwxrwsr-x 2 svcgalaxy 0 Nov 8 15:42 .drwxrwsr-x 30 svcgalaxy 565 Nov 8 15:42 ..(galaxy_env)[svcgalaxy@srv151 g]$___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Syntax error (!) in latest pull?!
I am running under virtualenv and running python 2.6 but I am suspecting that it is not passed on to my cluster that I am running under 2.6But thanks for pointing me in this direction...On Nov 07, 2012, at 01:08 AM, Geert Vandeweyer geert.vandewey...@ua.ac.be wrote:We recently ran into that issue as well. It seems that the used finally: syntax is valid from python 2.5, and we were using python 2.4. Loading a more recent version of python helped. Best, Geert Vandeweyer On 11/07/2012 04:54 AM, Scott McManus wrote:Well, lib/galaxy/datatypes/registry.py hasn't changed in over two months. Out of curiousity, I tried downloading a fresh copy of galaxy-central (without any initial configuration or database, either) and didn't encounter the same problem. I even tried messing with that line of code and didn't have any luck reproducing the same traceback. What version of python are you using? -Scott Got this in the latest pull of galaxy central...Ideas?Traceback (most recent call last): File "./scripts/set_metadata.py", line 24, in ? import galaxy.model.mapping #need to load this before we unpickle, in order to setup properties assigned by the mappers File "/mnt/ngs/analysis/svcgalaxy/galaxy-central/lib/galaxy/model/__init__.py", line 13, in ? import galaxy.datatypes.registry File "/mnt/ngs/analysis/svcgalaxy/galaxy-central/lib/galaxy/datatypes/registry.py", line 145 finally: ^ SyntaxError: invalid syntax ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Syntax error (!) in latest pull?!
Got this in the latest pull of galaxy central...Ideas?Traceback (most recent call last): File "./scripts/set_metadata.py", line 24, in ? import galaxy.model.mapping #need to load this before we unpickle, in order to setup properties assigned by the mappers File "/mnt/ngs/analysis/svcgalaxy/galaxy-central/lib/galaxy/model/__init__.py", line 13, in ? import galaxy.datatypes.registry File "/mnt/ngs/analysis/svcgalaxy/galaxy-central/lib/galaxy/datatypes/registry.py", line 145 finally: ^ SyntaxError: invalid syntax ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] How to run galaxy workflows in batch
Hi Geert,I did something similar but two things I got stumped on and maybe you found a way to do this through the API.1) I would like to put some of my results into a datalibrary. But i can't find the API call for that. Do you know if this is possible? I asked repeatedly here on the dev list but no one seems to know if this is possible...2) I am trying to pass my workflow some parameters that are defined in the workflow with ther ${NameOfVarible} notation.It works great in the UI, but I cannot seem to figure out how to pass parameters to a workflow...Any ideas there?Bedankt!ThonOn Nov 02, 2012, at 05:39 AM, Geert Vandeweyer geert.vandewey...@ua.ac.be wrote:Hi all, After some fiddling, I've put together an api-based workflow runner. It allows you to start workflows on batches of paired-end data from the commandline, and if set in the galaxy-config, output results to a different user (making history sharing/cloning obsolete from adminuser to end-user). Needed input (which workflow, which history, which input datafiles (from history/library) are selected on runtime by fetching the the options using the api. I've also put some notes together on the following page, regarding the api-syntax: http://tinyurl.com/cv6zmz2 It's using urllib to post the sumbission to a remote galaxy-server, so you don't need access to the galaxy-installation to run ./submit.py or ./workflow_execute.py. It's a stand-alone script, with no dependencies on the galaxy-provided code. I hope this helps some people getting started with the galaxy-api :-). Best, GeertOn 07/18/2012 02:38 PM, Sascha Kastens wrote:Hi Dev-Team,are you planning or maybe working on an update which enables the possiblilty to run workflows in batch mode with paired end data?Cheers,Sascha___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/ --
Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: geert.vandewe...@ua.ac.be
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726 ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Output file naming
and if you use something like;label="#echo os.path.splitext ( str ( $input.name ) ) [ 0 ] #-BWA.bam"You can change "SomeFile.fastq" into "SomeFile-BWA.bam".This allows you to remove the extension (.fastq in this case), insert some extra text after the original name and set the right extension (Not needed for Galaxy, but we are all used to seeing those labels)...I do this for all the tools in galaxy (bit of a pain, but worth it) so they produce more intelligent names than "12: X on data 10" etc.ThonOn Oct 26, 2012, at 08:12 AM, Joachim Jacob joachim.ja...@vib.be wrote:Hi, You can use label="$input.name" Cheers, Joachim Joachim Jacob, PhD Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib On 10/26/2012 04:38 PM, Darin wrote: Hello, I'm new to Galaxy, and I've been asked to tweak some tools so that output file names are related to input file names. For example, an input file of myData.abc might lead to an output file named myData_afterMyTool.xyz. I think the user is really just interested in the labels showing up in the history panel, so that gives me some flexibility. In my development environment, I've changed one tool's xml file's output tag to this: outputs data name="out_file1" format="bed" label="${input1}_afterTool.xyz"/ /outputs ... and that seems to work, but it's using an internal file name, instead of the label for the input file. So - does anyone know if there's an easy way to get the label from the input file into the output label, instead of the path/dataset_n.dat file format for this name? If there's a better way to approach this, I'd love any suggestions on that, too. Thanks, Darin ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] How to move a history item to a data library through the API
So I take it this is currently not possible with the API?I know I could dump the files to the filesystem and re-load it with the load into library from disk step in the API, but it seems slightly wasteful :)ThanksThonOn Oct 22, 2012, at 06:29 PM, Anthonius deBoer thondeb...@me.com wrote:Hi,I am trying to automatically move certain outputs from my workflow into a particular folder of a particular library THROUGH THE API.I can find how to do the reverse, but can't find an example of how to do this...Is there an equivalent of passing the history ID, like you can pass the Library ID using data['from_ld_id'].I tried the 'from_hd_id' but there is nothing in the lib/galaxy/web/api*.py that looks for that, so I guess that would not work.And looking in library_content.py it is not obvious either...Thanks!Regards,Thon de Boer, Ph.D.Bioinformatics Guru+1-650-799-6839thondeb...@me.comLinkedIn Profile___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Jobs are slow to start on my galaxy instance. Used to be much faster
I created the index but that did not change anything... I had sent James Taylor the schema as he requested, but I never heard anything so I guess he must not have found anything weird... I am waiting to move my Galaxy server off to a three server configuration, so I am hoping the problem will not return, although your issue does not bode well for this issue... Regards, Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeb...@me.com LinkedIn Profile On Oct 23, 2012, at 4:14 AM, Hans-Rudolf Hotz h...@fmi.ch wrote: Hi Thon Sorry for re-activating an old e-mail thread. I am just curious, whether Nate's suggestion solved your problem? After our upgrade (from the March 12th changeset 6799:40f1816d6857 to the October 5th changeset 7824:b5bda7a5c345) we see long delays in the jobs going from queued to running. For our server, switching from set_metadata_externally = False to set_metadata_externally = True did not speed up anything. Regards, Hans-Rudolf On 09/19/2012 04:27 PM, Nate Coraor wrote: On Sep 18, 2012, at 6:31 PM, Anthonius deBoer wrote: Hi, Jobs that I start on my in-house Galaxy instance now take up to 3-4 min to go from queued to Running, even though there is nothing much going on on the galaxy server... I have been running this instance since June and use a relatively new version of Galaxy-central (Last update, 22-Aug changeset: 7535:bf6517b2b336) I have noticed that my jobs table in the galaxy Postgres database contains about 60,000 jobs... Could that be the culprit? Does it slow to the complete database to see if there are any jobs that need to run? Could I purge the jobs and related tables to speed up the submissions? Hi Thon, It's not really possible to remove things from the database since there are a lot of interdependencies. The first thing to check would be whether set_metadata_externally = True in your Galaxy config. --nate Thanks, Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Source code documentation
The API allows you to do some of that... If you pass it the ID of the object (input.id) you can do all kinds of requests with the API. Look in the scripts/api folder of your local Galaxy instance... NOTE: The API seems to be a bit of a stepchild, since there is no good documentation and it seems to be undeveloped to some extent. For instance, the biggest issues is that you cannot pass a workflow any parameters, only inputs and outputs... So caveat emptor! Regards, Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeb...@me.com LinkedIn Profile On Oct 22, 2012, at 5:26 PM, alex.khassa...@csiro.au wrote: Hi, I wonder if there’s some kind of documentation (reference) for the Galaxy source? At the moment I have a couple of questions for example. 1. How can I get the dataset object (in my Python wrapper) given the dataset name? 2. How can I access the job parameters (enered in the UI or ‘hidden’) in the Python code? In general, when I have this kind of questions, where do I look? -Alex ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] How to move a history item to a data library through the API
Hi, I am trying to automatically move certain outputs from my workflow into a particular folder of a particular library THROUGH THE API. I can find how to do the reverse, but can't find an example of how to do this... Is there an equivalent of passing the history ID, like you can pass the Library ID using data['from_ld_id']. I tried the 'from_hd_id' but there is nothing in the lib/galaxy/web/api*.py that looks for that, so I guess that would not work. And looking in library_content.py it is not obvious either... Thanks! Regards, Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeb...@me.com LinkedIn Profile ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Adding metadata to datasets
Thanks, but that's too free form. Leaving it up to the user to define the keys. I would like to add defined keys that users can fill in in some form. Regards, Thon === Thon Deboer | Bioinformatics Guru thondeb...@me.com | 650-799-6839 On Oct 12, 2012, at 12:49 PM, Jeremy Goecks jeremy.goe...@emory.edu wrote: Is there an easy way to add meta data to a dataset without having to resort to coding it into the dataset definition? I am looking for things beyond tags and just a simple key value pair association would be enough... Tags support key-value pairs using this syntax: -- key:value OR key=value -- Best, J. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Adding metadata to datasets
Hi,Is there an easy way to add meta data to a dataset without having to resort to coding it into the dataset definition?I am looking for things beyond tags and just a simple key value pair association would be enough...I cannot see anything that is obvious, but I feel something must be there...ThanksThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] No module named controllers.library_common error on fresh Galaxy install from 4-Oct-2012
I gave up on figuring out why jobs don't get started for 15 minutes and started a new instance of galaxy, but when I tried to create a new data library, I got this error...Error attempting to display contents of library (TEST DATA): No module named controllers.library_common.Any ideas?Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Where does Galaxy get its TMP directory from?
Hi,I am running into some out-of-disk space errors with some of my jobs and this was because the /tmp space is running out of space.I want to use a different temporary directory but I can't get GALAXY to pick up my settings...I use this in the .ini file# Temporary files are stored in this directory.new_file_path = /mnt/ngs/analysis/tdeboer/galaxy-data/database2/tmpI set my TMPDIR environment variable to something similar, but when I check what my python script is using as its temp dir I get this:print "Temp dir: %s" % tempfile.gettempdir()Temp dir: /tmp/3642962.1.all.qIt seems that the SGE engine overrides the settings in GALAXY, no?So where do I set the tmp directory to be something different then /tmp?ThanksThon___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Tool migration errors
I'm getting this error when I try to run the tool migration:sh ./scripts/migrate_tools/0003_tools.sh install_dependenciesNo handlers could be found for logger "docutils"Repositories will be installed into configured tool_path location ../shed_toolsAdding new row (or updating an existing row) for repository 'freebayes' in the tool_shed_repository table.Traceback (most recent call last): File "./scripts/migrate_tools/migrate_tools.py", line 21, in module app = MigrateToolsApplication( sys.argv[ 1 ] ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/migrate/common.py", line 150, in __init__ install_dependencies=install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 37, in __init__ self.install_repository( repository_elem, install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 258, in install_repository install_dependencies=install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 174, in handle_repository_contents tool_dependencies=tool_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/util/shed_util.py", line 1231, in handle_tool_dependencies tool_dependency = install_package( app, elem, tool_shed_repository, tool_dependencies=tool_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/tool_dependencies/install_util.py", line 52, in install_package install_dir = get_tool_dependency_install_dir( app, tool_shed_repository, package_name, package_version ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/tool_dependencies/install_util.py", line 42, in get_tool_dependency_install_dir repository.installed_changeset_revision ) ) File "/home/tdeboer/code/galaxy_env/lib/python2.6/posixpath.py", line 67, in join elif path == '' or path.endswith('/'):AttributeError: 'NoneType' object has no attribute 'endswith'Any ideas?ThanksThon___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Tool migration errors
My Galaxy version is the most recent, but I had cloned this from an earlier version and have made lots of changes over the months...I'm a little confused by the fact that the tool_shed tools seem to be defined BOTH by the various XML files AND the databaseI have on occasions manually edited the XML file which I guess screws up the database...it is bad practice to have two different places that can easily get out of sync I would think...Either stick with the XML paradigm OR move everything into a database...Doing both is bound to get out of syncI don't really want to dive into the database since I don't know the schema and not that good at SQL to begin with, but maybe a script that would sync stuff would be an idea...ThanksOn Jul 11, 2012, at 12:29 PM, Greg Von Kuster g...@bx.psu.edu wrote:Hi Thon, What revision of Galaxy are you running? Have you made multiple attempts at the migration? Are you running a postgres database? How many records do you have in your tool_dependency table in your database, and what is the status of each record? Have you made any changes to the Galaxy code base in your environment? Any additional information that you can send will help track down the problem in your environment. Thanks! Greg Von Kuster On Jul 11, 2012, at 2:46 PM, Anthonius deBoer wrote: I'm getting this error when I try to run the tool migration:sh ./scripts/migrate_tools/0003_tools.sh install_dependencies No handlers could be found for logger "docutils" Repositories will be installed into configured tool_path location ../shed_tools Adding new row (or updating an existing row) for repository 'freebayes' in the tool_shed_repository table. Traceback (most recent call last): File "./scripts/migrate_tools/migrate_tools.py", line 21, in module app = MigrateToolsApplication( sys.argv[ 1 ] ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/migrate/common.py", line 150, in __init__ install_dependencies=install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 37, in __init__ self.install_repository( repository_elem, install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 258, in install_repository install_dependencies=install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 174, in handle_repository_contents tool_dependencies=tool_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/util/shed_util.py", line 1231, in handle_tool_dependencies tool_dependency = install_package( app, elem, tool_shed_repository, tool_dependencies=tool_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/tool_dependencies/install_util.py", line 52, in install_package install_dir = get_tool_dependency_install_dir( app, tool_shed_repository, package_name, package_version ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/tool_dependencies/install_util.py", line 42, in get_tool_dependency_install_dir repository.installed_changeset_revision ) ) File "/home/tdeboer/code/galaxy_env/lib/python2.6/posixpath.py", line 67, in join elif path == '' or path.endswith('/'): AttributeError: 'NoneType' object has no attribute 'endswith'Any ideas?ThanksThon ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:http://lists.bx.psu.edu/ ___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Tool migration errors
If I drop the following two tables, would those be reconstructed again correctly if I had the XML files in order?tool_version_associationandtool_versionThanksThonOn Jul 11, 2012, at 12:29 PM, Greg Von Kuster g...@bx.psu.edu wrote:Hi Thon, What revision of Galaxy are you running? Have you made multiple attempts at the migration? Are you running a postgres database? How many records do you have in your tool_dependency table in your database, and what is the status of each record? Have you made any changes to the Galaxy code base in your environment? Any additional information that you can send will help track down the problem in your environment. Thanks! Greg Von Kuster On Jul 11, 2012, at 2:46 PM, Anthonius deBoer wrote: I'm getting this error when I try to run the tool migration:sh ./scripts/migrate_tools/0003_tools.sh install_dependencies No handlers could be found for logger "docutils" Repositories will be installed into configured tool_path location ../shed_tools Adding new row (or updating an existing row) for repository 'freebayes' in the tool_shed_repository table. Traceback (most recent call last): File "./scripts/migrate_tools/migrate_tools.py", line 21, in module app = MigrateToolsApplication( sys.argv[ 1 ] ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/migrate/common.py", line 150, in __init__ install_dependencies=install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 37, in __init__ self.install_repository( repository_elem, install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 258, in install_repository install_dependencies=install_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/install_manager.py", line 174, in handle_repository_contents tool_dependencies=tool_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/util/shed_util.py", line 1231, in handle_tool_dependencies tool_dependency = install_package( app, elem, tool_shed_repository, tool_dependencies=tool_dependencies ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/tool_dependencies/install_util.py", line 52, in install_package install_dir = get_tool_dependency_install_dir( app, tool_shed_repository, package_name, package_version ) File "/home/tdeboer/code/galaxy-copy/lib/galaxy/tool_shed/tool_dependencies/install_util.py", line 42, in get_tool_dependency_install_dir repository.installed_changeset_revision ) ) File "/home/tdeboer/code/galaxy_env/lib/python2.6/posixpath.py", line 67, in join elif path == '' or path.endswith('/'): AttributeError: 'NoneType' object has no attribute 'endswith'Any ideas?ThanksThon ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:http://lists.bx.psu.edu/ ___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] Where are the datatype associations stored?
Hi,I had some more bad luck trying to install the GMAP stuff from the TOOLSHEd but revision 10 is broken since it does not allow you to select a section and just stores the programs in the main tree...But when I tried to remove the tool I got the following error for every run, since the sniffers where still looking for a module called GMAP...Which table should I nuke to get rid of this association? Where are the datatype associations stored?There does not seem to be an XML file for this, so I hope i can just remove some lines from the postgres database table but there are no obvious table names I can find...WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:IntervalAnnotation' to sniff_order: No module named g! map WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:SpliceSiteAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:IntronAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:SNPAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:IntervalAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:SpliceSiteAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:IntronAnnotation' to sniff_order: No module named gmapWARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:SNPAnnotation' to sniff_order: No module named gmapThanks ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] History items get REMOVED when using the API to run workflows
WOW...I guess I know how you guys that designed the API think :)Small other question...Is it possible (yet) to pass a parameter to the API yet, that is NOT an input step?Things that get defined in the workflow step such as {Analysis_Run} that get filled in at runtime?Now it reverts to "X__oc__Analysis_Run__cc__" but I would like to change this at runtime through the API...ThanksThonOn Jul 05, 2012, at 02:34 PM, Dannon Baker dannonba...@me.com wrote:On Jul 5, 2012, at 5:33 PM, Thon Deboer wrote: data['no_add_to_history']="True" ? Should do it.
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] Has anyone implemented PINDEL as a tool in their GALAXY
Hi,I am about to embark of embedding PINDEL into GALAXY since it is not available in the TOOLSHED.Before I do, I thought I check if someone has already done this and could send me their XML so I don't have to start from scratch...Thanks!Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] History items get REMOVED when using the API to run workflows
OK...I tried this and it no longer removes the entries from the history...Thanks!I do see that each of the new histories I create still have a reference to some of the other files I pick up from the library (such as dbSNP) but those entries in the history have the number "None" in stead of the usual 1,2 3 etc. that each file in a history hasI guess that is an artifact and it does not really bother me, although it makes it hard to run a tool again that uses one of those files later manually, but since that is not the point of running through the API it is fine for now, but it just looks scary...ThonOn Jul 05, 2012, at 02:34 PM, Dannon Baker dannonba...@me.com wrote:On Jul 5, 2012, at 5:33 PM, Thon Deboer wrote: data['no_add_to_history']="True" ? Should do it. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Galaxy cleaning up after it self after jobs fail
I am running galaxy on a system that normally runs all jobs on the cluster, but some jobs need to be run locally...But when I restart galaxy when these jobs are running, those jobs fail (and I am fine with that)...But then when I check the logs, I see a lot of this kind of errors and the word CRITICAL scares me a little...It complains about certain directories not being empty (although they look empty to me when I look at them)But if it needs to clean up should it not expect stuff to be in those working directories? And why is this a critical error and why don't those directories get removed anyway...It clutters my database and I fear this was the reason for weird behaviour I saw earlier, since it may re-use some of these directories after a while ?galaxy.objectstore CRITICAL 2012-07-05 16:57:24,964 /mnt/ngs/analysis/tdeboer/galaxy-data/database2/job_working_directory/003/3350 delete error [Errno 39] Directory not empty: '/mnt/ngs/analysis/tdeboer/galaxy-data/database2/job_working_directory/003/3350'ThanksThon___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Tophat Troubles
The cluster was busy I think, so you just have to wait for the queue to clear...On Jun 25, 2012, at 02:25 PM, Kristen Roop kristen.r...@gmail.com wrote:Hello,I am having an issue on the Galaxy Main server.I had reached my maximum disk space this morning. So I cleaned out some items until my space was down to 61%. I then tried to run Tophat for RNA seq and the tool is doing nothing. I know the datasets aren't too large because I would have received an error message. The job simply goes into my queue as gray and says waiting to run. Any ideas on what is going on?Thanks,Kristen Roop___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Are there any database test scripts?
Hi,I am still having serious issues with running galaxy workflows from the API on my installation...I am getting random files on which the workflows are run, rather than the files I select in my script.I am suspecting that my database may be corrupt...Is there any test script that can test my theory?I am running postgres and the database may have become corrupted, although running the same WF from the UI has no problems, so I still suspect the API...ThanksThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] API calls randomly selecting files upon submission
Hi,I have a very bizarre problem with Galaxy.I have to run a workflow through the API, since Galaxy cannot deal with many pair-end FASTQ files...In the past I had no issues with this and it always run as expected.But in the last few days when I select 2 FASTQ files to be run in with the API, invariably Galaxy chooses a completely different set of files to run the workflow on...I double checked that I find the right file throug the API and that the ID's are correct and that those ID's are indeed pointing to the right files, but once I pass the MAP to Galaxy, it randomly chooses some file...Not sure how to test this further..(Using the latest pull from galaxy_central, database @101)Here's some logs from my tool (not going to help a lot I think)Getting GALAXY configuration settings...Found FASTQ library: ONCOMED_TESTFound Workflow: MUT190-1 PE DNA_SEQ - Trimmomatic-BWA-REALIGN-DEDUP-VCF (GRCh37-DECOY)Found history prefix: ONCOMED-BAKEOFFFound Galaxy URL: http://srp106:8080/api[OK]--- DISPLAY.py output for the two files I want to run...But different files were run in the WF ---Getting FASTQ files from the library...{'/2_181_3_CTTGTA_R1.fastq': '11049d6355c1fae6', '/2_181_3_CTTGTA_R2.fastq': '2172873b78ef3de8'}[OK] Found 2 FASTQ files---Retrieving workflow step definitions...TARGETS (GATK) - ONCOMED/Baits and Targets/SS_OM_v1_targetsOM.gatk-interval[OK]BAITS - ONCOMED/Baits and Targets/SS_OM_v1_baitsOM.interval[OK]COSMIC Dataset - Annotations/CosmicMutantExport_v56_15.vcf[OK]Mills_and_1000G_gold_standard - Annotations/Annotations used in GATK-1.5/Mills_and_1000G_gold_standard.indels.b37.sites.vcf[OK]Adapters File - Annotations/Adapters.fasta[OK]TARGETS - ONCOMED/Baits and Targets/SS_OM_v1_targetsOM.interval[OK]dbSNP VCF File - Annotations/Annotations used in GATK-1.5/dbsnp_135.b37.vcf[OK]TARGETS400 (GATK) - ONCOMED/Baits and Targets/SS_OM_v1_targetsOM_PLUS400.gatk-interval[OK]---Preparing workflow for :/2_181_3_CTTGTA_R1.fastqResponse{'outputs': ['685cc0daf312e1e2', '5faa831fd7938831', '31b4c60b45b51900', '55dec7d4d8038078', '8b4bd251f4acfdb2', '991643c8bf045bdf', '45509d1f3c223d75', '69ee8e65832b0a48', '1d42f64460defe7a', 'c58ddab89b2bb805', '8bfc64bb47770912', 'f204dc7bb5ecfc2e', 'cde9aa04cfe839d1', '6362db4d5959d831', 'c01d9ed9facb9b70', '8a6baaaff54e1b05', '3842cee51a54bc8a', 'a1c8e68fee33b3f4', '7780674a8a538878', '32baf3117f18f4a9', 'd81e438f0fefa4fe', '37291cf3d6fd439a', '9b0ffea8ff1d4477', '8be5b6d3aeb85faf', 'b7c76befdaa37432', 'ffcf2e98e3761238', '900232bec5f8fcbe', 'acb2c56b985cec44', '7f15aaa8ca46161f', 'bb8f19f6facd93af', 'f9c244896d5b57dc', '8512f4fc5aed6ea7', '3f52c44d599b9716', '55806ff905fd6286', '0cb88461cff7d031', 'a273ec073a5ae9a6', 'b4fecb3654146377', '9931773a693bd74d', 'da97567b301b6e88', 'db19f117b9ba1f38', '6e9cefcf946aa2eb', '32b7a1d98d5ba938', '950b17b0eeebc8ce', 'c46ef25bdd0b128c', 'd2d71b57d04fdf56', '99ad472ce159d2b4', 'c2a0592a7bfde32d', 'cd476a668e76e2fd', '610abd782fd87335', 'f65cfa41776c8092', '33a09f65ad37eefb', '6a4013fe7480be2e', '0161b00ac62dd980', '177fef443e7d000a', '093d5543780988ae', '603db5a5490d658a', '5e639ad0ecdbbfaa', '601406f579c6', '318bdcc79e1698f5', '9e0a3f75c768292a', '4a913b7f86293e50', '0edd411e46492187', '270afa41507f14e5', '4b75e21d1ac0d2ad', '1816e3ba5ffa42d6', '71b643cde9fb710c', '62d1d0058327a5da', '133576f2dda5c2b4', '450c3dc15a3aa0a0', '5e747c660af21e40', '7db0323336d28301', '9878576367d5f544', 'b150a412957324a5', 'e97867c104e59e02', 'c8e9faf4b8a01e67', '5d194649d99822eb'], 'history': '31d04ac4a1e13563'}--Member Information--ldda_id: ccfcc222604b2175misc_blurb: 14.3 Gbname: 2_181_3_CTTGTA_R2.fastqdata_type: fastqfile_name: /mnt/ngs/data/ProcessedData/Casava_builds/SEQUENCER02/120608_SN402_0181_AD143FACXX/fastq/2_181_3_CTTGTA_R2.fastquploaded_by: tdeb...@genomichealth.commetadata_sequences: Nonetemplate_data: {}genome_build: hg19-decoymodel_class: LibraryDatasetmisc_info: uploaded fastq filefile_size: 15361395563metadata_data_lines: Nonemessage:id: 2172873b78ef3de8date_uploaded: 2012-06-21T17:37:20.383314metadata_dbkey: hg19-decoy(galaxy_env)[tdeboer@srp106 api]$ myDisplay.py libraries/fb85969571388350/contents/11049d6355c1fae6Member Information--ldda_id: 6768e90755e11e63misc_blurb: 14.3 Gbname: 2_181_3_CTTGTA_R1.fastqdata_type: fastqfile_name: /mnt/ngs/data/ProcessedData/Casava_builds/SEQUENCER02/120608_SN402_0181_AD143FACXX/fastq/2_181_3_CTTGTA_R1.fastquploaded_by: tdeb...@genomichealth.commetadata_sequences: Nonetemplate_data: {}genome_build: hg19-decoymodel_class: LibraryDatasetmisc_info: uploaded fastq filefile_size: 15361395563metadata_data_lines: Nonemessage:id: 11049d6355c1fae6date_uploaded: 2012-06-21T17:37:19.122245metadata_dbkey:
Re: [galaxy-dev] Is there a way to dynamically assign Queues in Galaxy
Hi Jeremy,Thanks for the info...But it's not clear from your message if I could use this Key/Value approach at the moment to distinguish if a job should be run in the fast or in the slow queue.I would like to add a parameter to every tool that would have it determine if it should be in the fast queue or in the slow queue...It would be checked for interactive jobs and if someone created a workflow with this tool, they could turn it off and it would run in the slow/high memory queueCould I add this today and what would the XML look like or are you saying it only works for the trackster example you gave...ThanksThonOn Jun 01, 2012, at 05:44 AM, Jeremy Goecks jeremy.goe...@emory.edu wrote: Is there a way for a tool to sometimes be placed in the fast queue and sometimes in the long queue?Not through Galaxy as far as I know. Yes, this is possible using job parameterization. From universe.ini.sample: -- # Per-tool job handler and runner overrides. Parameters can be included to define multiple # runners per tool. E.g. to run Cufflinks jobs initiated from Trackster # differently than standard Cufflinks jobs: # # cufflinks = local:/// # cufflinks[source@trackster] = local:/// -- This approach is definitely a beta feature, but the idea is that any set of key@value parameters should be able to be used to direct jobs to different queues as needed. Job parameterization is done in only one place right now, the tracks.py controller in rerun_tool The idea is that jobs run via Trackster are short, so they get a different queue: -- subset_job, subset_job_outputs = tool.execute( trans, incoming=tool_params, history=target_history, job_params={ "source" : "trackster" } ) --Right now I'd like to be able to allocate jobs to different queues based on the input data size (and thus the expected compute time and resource needed), but that is rather complicated. e.g. If you have a low memory queue and a high memory query. To make this work, you'd want to modify the execute() method in the DefaultToolAction class (/lib/galaxy/tools/actions/__init__.py) to add job parameters based on either tool parameters and/or input dataset size. You might even want different queues according to the user (e.g. one group might have paid for part of the cluster and get priority access). This could also be done in the same location as trans.user will give you the user running the tool/job. Best, J. ___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Is there a way to dynamically assign Queues in Galaxy
Could you give me a pointer where in the codebase I should start looking to implement this, please?I'd be happy to share this with the community if it is useful for othersThonOn Jun 01, 2012, at 10:53 AM, Jeremy Goecks jeremy.goe...@emory.edu wrote:No, this is not configurable via XML; you'd have to extend the Galaxy codebase to implement this behavior.Best,J.On Jun 1, 2012, at 1:50 PM, Anthonius deBoer wrote:Hi Jeremy,Thanks for the info...But it's not clear from your message if I could use this Key/Value approach at the moment to distinguish if a job should be run in the fast or in the slow queue.I would like to add a parameter to every tool that would have it determine if it should be in the fast queue or in the slow queue...It would be checked for interactive jobs and if someone created a workflow with this tool, they could turn it off and it would run in the slow/high memory queueCould I add this today and what would the XML look like or are you saying it only works for the trackster example you gave...ThanksThonOn Jun 01, 2012, at 05:44 AM, Jeremy Goecks jeremy.goe...@emory.edu wrote: Is there a way for a tool to sometimes be placed in the fast queue and sometimes in the long queue?Not through Galaxy as far as I know. Yes, this is possible using job parameterization. From universe.ini.sample: -- # Per-tool job handler and runner overrides. Parameters can be included to define multiple # runners per tool. E.g. to run Cufflinks jobs initiated from Trackster # differently than standard Cufflinks jobs: # # cufflinks = local:/// # cufflinks[source@trackster] = local:/// -- This approach is definitely a beta feature, but the idea is that any set of key@value parameters should be able to be used to direct jobs to different queues as needed. Job parameterization is done in only one place right now, the tracks.py controller in rerun_tool The idea is that jobs run via Trackster are short, so they get a different queue: -- subset_job, subset_job_outputs = tool.execute( trans, incoming=tool_params, history=target_history, job_params={ "source" : "trackster" } ) --Right now I'd like to be able to allocate jobs to different queues based on the input data size (and thus the expected compute time and resource needed), but that is rather complicated. e.g. If you have a low memory queue and a high memory query. To make this work, you'd want to modify the execute() method in the DefaultToolAction class (/lib/galaxy/tools/actions/__init__.py) to add job parameters based on either tool parameters and/or input dataset size. You might even want different queues according to the user (e.g. one group might have paid for part of the cluster and get priority access). This could also be done in the same location as trans.user will give you the user running the tool/job. Best, J. ___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Is there a way to dynamically assign Queues in Galaxy
I would be very interested in this piece of code...Would I be wrong in assuming if I did the pull of that version I would be messing up my Galaxy system and I need to manually resolve it?I would not want to re-introduce many of the bugs etc. so could I not just pull in whatever code you changed? Could you indicate which python modules you changed so I can pull in just those?ThanksThonOn Jun 01, 2012, at 11:18 AM, John Chilton chil0...@umn.edu wrote:I posted a message about this half a year ago and issued a pull request (which is now conflicted). I developed some code which was not a turn key solution but I thought it was a good first step and it was very expressive. It would allow you to do stuff like inspect input sizes or parameter values and dynamically change queues or queue parameters based on these (using raw python to express these rules). galaxy-dev@lists.bx.psu.edu/msg03010.html'http://www.mail-archive.com/galaxy-dev@lists.bx.psu.edu/msg03010.html https://bitbucket.org/galaxy/galaxy-central/pull-request/12/dynamic-job-runners You are not the first person to follow up with requests like this, I would be happy to update the pull request if it were going to be acted on. -John On Fri, Jun 1, 2012 at 1:02 PM, Anthonius deBoer thondeb...@me.com wrote: Could you give me a pointer where in the codebase I should start looking to implement this, please? I'd be happy to share this with the community if it is useful for others Thon On Jun 01, 2012, at 10:53 AM, Jeremy Goecks jeremy.goe...@emory.edu wrote: No, this is not configurable via XML; you'd have to extend the Galaxy codebase to implement this behavior. Best, J. On Jun 1, 2012, at 1:50 PM, Anthonius deBoer wrote: Hi Jeremy, Thanks for the info...But it's not clear from your message if I could use this Key/Value approach at the moment to distinguish if a job should be run in the fast or in the slow queue. I would like to add a parameter to every tool that would have it determine if it should be in the fast queue or in the slow queue... It would be checked for interactive jobs and if someone created a workflow with this tool, they could turn it off and it would run in the slow/high memory queue Could I add this today and what would the XML look like or are you saying it only works for the trackster example you gave... Thanks Thon On Jun 01, 2012, at 05:44 AM, Jeremy Goecks jeremy.goe...@emory.edu wrote: Is there a way for a tool to sometimes be placed in the fast queue and sometimes in the long queue? Not through Galaxy as far as I know. Yes, this is possible using job parameterization. From universe.ini.sample: -- # Per-tool job handler and runner overrides. Parameters can be included to define multiple # runners per tool. E.g. to run Cufflinks jobs initiated from Trackster # differently than standard Cufflinks jobs: # # cufflinks = local:/// # cufflinks[source@trackster] = local:/// -- This approach is definitely a beta feature, but the idea is that any set of key@value parameters should be able to be used to direct jobs to different queues as needed. Job parameterization is done in only one place right now, the tracks.py controller in rerun_tool The idea is that jobs run via Trackster are short, so they get a different queue: -- subset_job, subset_job_outputs = tool.execute( trans, incoming=tool_params, history=target_history, job_params={ "source" : "trackster" } ) --Right now I'd like to be able to allocate jobs to different queues based on the input data size (and thus the expected compute time and resource needed), but that is rather complicated. e.g. If you have a low memory queue and a high memory query. To make this work, you'd want to modify the execute() method in the DefaultToolAction class (/lib/galaxy/tools/actions/__init__.py) to add job parameters based on either tool parameters and/or input dataset size. You might even want different queues according to the user (e.g. one group might have paid for part of the cluster and get priority access). This could also be done in the same location as trans.user will give you the user running the tool/job. Best, J. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] Is there a way to dynamically assign Queues in Galaxy
Hi,I often find my self having to battle with other users of the cluster, even for the one-off jobs that someone runs interactively...We have setup Fast/short queues and Long queues...Is there a way for a tool to sometimes be placed in the fast queue and sometimes in the long queue?I know I can setup a tool in the universe file and add a -pe option there...But is there a way to dynamically assign jobs sometimes to one queue and sometimes to another?Even an option in the UI would work, since I could make the short queue the default for interactive work and people could turn it off in workflows that will run for a long time or will run with many samplesThanksThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Meme gives me UnicodeDecodeError. Is this an error from parsing the OUTPUT?
Hi,I am trying to run MEME and I got this error below when running it.Is this an error complaining about the non-ASCII nature of the OUTPUT or of the program (or input?).ThanksThonError Traceback:View as: Interactive | Text | XML (full)⇝ UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 25: ordinal not in range(128)URL: http://srp106:8080/dataset/errors?id=176856Module weberror.evalexception.middleware:364 in respond view app_iter = self.application(environ, detect_start_response)Module paste.debug.prints:98 in __call__ view environ, self.app)Module paste.wsgilib:539 in intercept_output view app_iter = application(environ, replacement_start_response)Module paste.recursive:80 in __call__ view return self.application(environ, start_response)Module paste.httpexceptions:632 in __call__ view return self.application(environ, start_response)Module galaxy.web.framework.base:160 in __call__ view body = method( trans, **kwargs )Module galaxy.web.controllers.dataset:160 in errors view return trans.fill_template( "dataset/errors.mako", hda=hda )Module galaxy.web.framework:733 in fill_template view return self.fill_template_mako( filename, **kwargs )Module galaxy.web.framework:744 in fill_template_mako view return template.render( **data )Module mako.template:296 in render view return runtime._render(self, self.callable_, args, data)Module mako.runtime:660 in _render view **_kwargs_for_callable(callable_, data))Module mako.runtime:692 in _render_context view _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)Module mako.runtime:718 in _exec_template view callable_(context, *args, **kwargs)Module dataset_errors_mako:74 in render_body view __M_writer(filters.html_escape(unicode(job.stdout )))UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 25: ordinal not in range(128)___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Uncaught exception when submitting jobs
Hi,All of a sudden I am getting these errors when I submit a job in my Galaxy system.I can't recall I changed anything and I checked the universe file but nothing obvious there...Any ideas?ThanksThongalaxy.jobs.runners.drmaa ERROR 2012-05-29 09:40:46,873 Uncaught exception queueing jobTraceback (most recent call last): File "/home/tdeboer/code/galaxy-central/lib/galaxy/jobs/runners/drmaa.py", line 133, in run_next self.queue_job( obj ) File "/home/tdeboer/code/galaxy-central/lib/galaxy/jobs/runners/drmaa.py", line 213, in queue_job job_id = self.ds.runJob(jt) File "/home/tdeboer/code/galaxy-central/eggs/drmaa-0.4b3-py2.6.egg/drmaa/__init__.py", line 331, in runJob _h.c(_w.drmaa_run_job, jid, _ct.sizeof(jid), jobTemplate) File "/home/tdeboer/code/galaxy-central/eggs/drmaa-0.4b3-py2.6.egg/drmaa/helpers.py", line 213, in c return f(*(args + (error_buffer, sizeof(error_buffer File "/home/tdeboer/code/galaxy-central/eggs/drmaa-0.4b3-py2.6.egg/drmaa/errors.py", line 90, in error_check raise _ERRORS[code-1]("code %s: %s" % (code, error_buffer.value))DeniedByDrmException: code 17: ERROR! invalid option argument ""___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Batch limit on Wokflows
Unfortunately this is one of the most glaring problems with the UI based workflow engine that is Galaxy (That, and no modular workflows)...You cannot easily pair up more than one set of paired end reads from the UI, you need to use the API to run pairs of FASTQ files like that...I have written a script that will do this through the API..Excerpt below...You won't be able to run this as-is, but it gives you some idea what hoops you have to jump through to do this..The section INPUT FILE DEFINITION is the meat of the script, in that I simply cycle through all the files in a Library and match up the files myself...Ugly but it worksThoncode#!/usr/bin/env python# MUT190-1.py## Version 2.0.0# Created: 5-Apr-2012# By Thon de Boer, GHI# Last Update: 23-Apr-2012# By Thon de Boer, GHI## This program will run step MUT190-1; the alignment of FASTQ file with BWA# It is based on the generic GALAXY Workflow execution engine,# but has hardcoded defaults, seen below## Version 1.0.1: Added get_folder_id# Version 2.0.0: Made the config file an option (and the only option)#"""Execute a specifc workflow on a specific set of history items.It is created for running the GATK pipeline on a selected set of paired-end readsInput to the tool is the name of a History and it will extract all the paired-end FASTQ files from this."""import os, sys, optparse, shutil, subprocess, tempfile, fileinput, psycopg2, ConfigParsersys.path.insert( 0, os.path.dirname( __file__ ) )sys.path.insert( 0, "/home/tdeboer/g/scripts/api" )from common import *from ThonsModules import *def main(): #Parse Command Line parser = optparse.OptionParser() parser.add_option( '-c', '--config', dest='config', type='string', help='the configuration file containing all the settings for this workflow' ) parser.add_option( '-t', '--test', action='', dest='test', help='Do not submit, but test the settings', default=False) (options, args) = parser.parse_args() if not options.config:print "Error: No config file proivided. Please provide a configuration file with the -c,--config flag."sys.exit(1) config = ConfigParser.RawConfigParser() try:config.read(options.config)GALAXY_LIBRARY_FASTQ = config.get('GALAXY', 'GALAXY_LIBRARY_FASTQ').strip('"')GALAXY_WORKFLOW_DEFAULT=config.get('GALAXY', 'GALAXY_WORKFLOW_DEFAULT').strip('"')GALAXY_OUTHISTORY_DEFAULT=config.get('GALAXY', 'GALAXY_OUTHISTORY_DEFAULT').strip('"')galaxyURL=config.get('GALAXY', 'GALAXY_URL').strip('"')myKey=config.get('GALAXY', 'MYKEY').strip('"') except Exception, e:print 'Error: %s' % esys.exit(1) data = ""> #Find the files that go with the history or library try:historyPrefix = GALAXY_OUTHISTORY_DEFAULT + '-'files = {}lib_id = get_library(myKey, galaxyURL, GALAXY_LIBRARY_FASTQ)files = get_files_from_library(myKey, galaxyURL, GALAXY_LIBRARY_FASTQ, lib_id)if files == {}: print 'Error: Did not get any files from the library' sys.exit(1)workflow_id = get_workflow(myKey, galaxyURL, GALAXY_WORKFLOW_DEFAULT)data['workflow_id'] = workflow_id except Exception, e:print 'Error: %s' % esys.exit(1) data['ds_map'] ={} #Define the standard library datasets try:##### WORKFLOW STEP DEFINITION#####Format for these lines should be followed...Only change the 'BAITS' and 'MUT190/Baits and Targets/SS_Mut_v2_baits190.bed' textdata['ds_map'][get_workflow_step(myKey, galaxyURL, workflow_id, 'BAITS')] = get_library_file(myKey, galaxyURL, 'MUT190/Baits and Targets/SS_Mut_v2_baitsMUT190.bed')data['ds_map'][get_workflow_step(myKey, galaxyURL, workflow_id, 'TARGETS')] = get_library_file(myKey, galaxyURL, 'MUT190/Baits and Targets/SS_Mut_v2_targetsMUT190_PLUS60.interval')data['ds_map'][get_workflow_step(myKey, galaxyURL, workflow_id, 'TARGETS (GATK)')] = get_library_file(myKey, galaxyURL, 'MUT190/Baits and Targets/SS_Mut_v2_targetsMUT190_PLUS400.gatk-interval')data['ds_map'][get_workflow_step(myKey, galaxyURL, workflow_id, 'dbSNP VCF File')] = get_library_file(myKey, galaxyURL, 'Annotations/Annotations used in GATK-1.5/dbsnp_135.b37.vcf')data['ds_map'][get_workflow_step(myKey, galaxyURL, workflow_id, 'Mills_and_1000G_gold_standard')] = get_library_file(myKey, galaxyURL, 'Annotations/Annotations used in GATK-1.5/Mills_and_1000G_gold_standard.indels.b37.sites.vcf') except Exception, e:print "Error: Problem with finding all the input steps for this workflow: %s" % esys.exit(1) ### # # INPUT FILE DEFINITION # These are the files that are cycled over # ### for f in files:if '_R1.fastq' in f: f2 =
[galaxy-dev] Is there a way to turn of the pretty display for VCF files
Hi,Is there a way to turn of the special pretty print version for VCF files since it is not working correctly?I now have to download every VCF file I want to look at rather than being able to see the first MB of the text file it is...Where is this pretty printer and how do I turn it off?ThanksThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Is there a way to turn of the pretty display for VCF files
I'm running the latest galaxy central verson that I pulled down today...On May 29, 2012, at 03:59 PM, Dannon Baker dannonba...@me.com wrote:What Galaxy revision are you running? Given the new information I'd guess it's an erroneous console.log that was removed in bb7f51fb545d. On May 29, 2012, at 6:49 PM, Anthonius deBoer wrote: Interestingly when I open the Developer Tools in Chrome, my VCF file is actually loading...As soon as I close the developer tool, it stops loading the the page again (and it resumes loading when I re-open the Developer tools...)Really funky...Here's theVCF file...On my computer it only shows up to line5 112175639 rs121913332 C T 3206.18 PASS AC=3;AF=0.0174;AN=172;COSMIC;DB;DP=15405;Dels=0.00;EFF=DOWNSTREAM(MODIFIERAPC|protein_coding|CODING|ENST0502371|),DOWNSTREAM(MODIFIERAPC|protein_coding|CODING|ENST0504915|),DOWNSTREAM(MODIFIERAPC|protein_coding|CODING|ENST0507379|),DOWNSTREAM(MODIFIERAPC|protein_coding|CODING|ENST0512211|),DOWNSTREAM(MODIFIERAPC|protein_coding|CODING|ENST0514164|),STOP_GAINED(HIGH|NONSENSE|Cga/Tga|R1450*|APC|protein_coding|CODING|ENST0257430|exon_5_112173250_112181936),STOP_GAINED(HIGH|NONSENSE|Cga/Tga|R1450*|APC|protein_coding|CODING|ENST0457016|exon_5_112173250_112181936),STOP_GAINED(HIGH|NONSENSE|Cga/Tga|R1450*|APC|protein_coding|CODING|ENST0508376|exon_5_112173250_112181753),TRANSCRIPT(MODIFIERCTC-554D6.1|processed_transcript|NON_CODING|ENST0520401|),UTR_3_PRIME(MODIFIERAPC|protein_coding|CODING|ENST0508624|);GC=45.64;HRun=0;MQ0=0;_OND_=0.01;set=variant5-variant49-variant10 On May 29, 2012, at 03:39 PM, Dannon Baker dannonba...@me.com wrote:Hmm. Chrome on OSX looks good to me, please do send over the VCF file and I can take a look.Do you see any _javascript_ errors in the browser console? On May 29, 2012, at 6:31 PM, Anthonius deBoer wrote: Hi Dannon, I could share the VCF file, but I think it is dependent on the browser you use... I used the same file on FireFox on Linux and there it loaded the complete VCF file (Still don't see the "Show All" option) but on my chrome on Windows 7 it only shows the first 5 lines or so of the genotypes and then nothing...The stuff that IS shown looks correctly formatted etc. it just does not show the rest of the file which it normally does Thon On May 29, 2012, at 03:01 PM, Dannon Baker dannonba...@me.com wrote: There isn't a universe toggle for the tabular display. The VCF datatype inherits the pretty printing from the base tabular datatype, and if you'd like to disable it one way would be to override the display_data method in VCF using the raw Data display_data method. That said, I'd rather just fix your problem. Looking at a few VCF files I have, they look fine. Can you share a VCF file with me that doesn't display properly? -Dannon On May 29, 2012, at 5:50 PM, Anthonius deBoer wrote: Hi,Is there a way to turn of the special pretty print version for VCF files since it is not working correctly?I now have to download every VCF file I want to look at rather than being able to see the first MB of the text file it is...Where is this pretty printer and how do I turn it off?ThanksThon___Please keep all replies on the list by using "reply all"in your mail client. To manage your subscriptions to thisand other Galaxy lists, please use the interface at:http://lists.bx.psu.edu/ CalledVariants-Select.zip ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] specifying nodes/processors with PBS
Yes, you can set these things for each tool independently.Check the file"universe_wsgi.ini"and check the section:[galaxy:tool_runners]There you can set the -pe options for GSUB in the same way you can do on the command line...Here's an example of my setup:gatk_variant_select = drmaa://-V -pe smp 1gatk_validate_variants = drmaa://-V -pe smp 1far = drmaa://-V -pe smp 1velveth = drmaa://-V -pe mpi 4velvetg = drmaa://-V -pe mpi 8I set the -pe mpi option to 8 for VELVETH/G and to 1 for GATK for instanceThonOn May 23, 2012, at 12:47 PM, "J. Greenbaum" jgb...@liai.org wrote:Hi,I've successfully configured a local galaxy instance to submit jobs to our local cluster, but have a question regarding setting PBS options such as the number of nodes and processors to use. I've followed the guide here:http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster?highlight=%28pbs%29Is there any way to do this in a tool-specific manner. For example, for running tophat to do a mapping run, I would want it to use 24 CPUs, but for simply converting a file type I would want only 1 CPU requested. Is there a way to specify this somewhere?Thanks,Jason--Jason Greenbaum, Ph.D.Manager, Bioinformatics Core | jgb...@liai.orgLa Jolla Institute for Allergy and Immunology___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Using workflows as components in workflows?
I would like to reiterate this point in as far as this single feature is preventing us from implementing large workflows in our organization in Galaxy...It is just not feasible to re-do all workflow steps each time in complex studies...Is there any way we can speed up the implementation of this issue?Thanks,ThonOn May 18, 2012, at 05:44 AM, Jeremy Goecks jeremy.goe...@emory.edu wrote:Nested workflows aren't yet possible, but they're definitely something we'd like to do in the future. Best, J. On May 17, 2012, at 9:48 AM, Peter van Heusden wrote: Hi thereIs there a way in Galaxy to use workflows as components in workflows? I've got two datasets that start as Genbank files and are processed (features extracted, translated to AA) by a small workflow, and then BLASTed against one another. Ideally I'd like to modularise the initial steps as a workflow so that I can say input data goes through this workflow and then the result is fed into either the query or the subject slot of blastp. So is this possible?Thanks!Peter___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Preview of large files seems not to finish
Hi,I noticed in the latest installment of Galaxy-Central that there is a pretty view option for VCF files, in that it tries to show the data in nice columns.But for my VCF files it does not seem to finish loading the first MB of the file..It also no longer shows the SHOW ALL at the top, so now I have to load the file out of my browser, which is a pain...Is this just a bug or a feature?Thanks,Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Possible bug: Tags in a workflow gets duplicated hundreds of times
Hi,I have run into an issue with a tag being duplicated hundreds of times each time I open, edit or clone a workflow.I have tried to manually remove them, but each time the tag gets inserted somehow and duplicated each time I edit the workflow.By now there are hundreds of copies of the tags at it makes loading and running the workflow very slow.Is there a way to remove a tag from the system somehow? I am willing to dig into the database with some SQL statement but not sure where to start...ThanksThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Trackster cannot display INTERVAL or GATK-INTERVAL files???
It seems that trackster does not know how to display INTERVAL files?Is that true? Surely there is an easy way to support those kinds of simple files without having to convert them?Am I missing something?I also don't seem to be able to convert an interval file into a BED file?Thanks,Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Trackster cannot display INTERVAL or GATK-INTERVAL files???
Oh wait..It's in the info section...Weird place for a conversion tool but it is there :)On Mar 19, 2012, at 12:02 PM, Anthonius deBoer thondeb...@me.com wrote:It seems that trackster does not know how to display INTERVAL files?Is that true? Surely there is an easy way to support those kinds of simple files without having to convert them?Am I missing something?I also don't seem to be able to convert an interval file into a BED file?Thanks,Thon___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Error when installing toolshed tool
I'm getting this error when I try to install the NGS Java toolkit toolshed...Version information for the tools included in the java_genomics_toolkit repository is missing. Reset all of this repository's metadata in the tool shed, then set the installed tool versions from the installed repository's Repository Actions menu. Installed 1 repository and all tools were loaded into tool panel section NGS: Toolkit:Installed repositories: java_genomics_toolkit.Any idea why the version number stuff is not correctly inserted in the database?Bug?ThanksThon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error 500 when trying to execute a workflow with the API
Huh? I have many string inputs to some particular workflow steps...Those are parameters etc that need to be entered or chosen from a dropdown..How would we deal with those kinds of entries into a workflow in the API? Or do all of them have to be pre-set?ThanksThonOn Feb 21, 2012, at 04:21 PM, Dannon Baker dannonba...@me.com wrote:There is no string input. If the string you're looking for is, say, the name of a dataset what you'd do is make separate calls to the history and workflow API. For example, if you had a history "My important history", and a dataset "My input dataset", and no knowledge of the id's (as you shouldn't' have to have, externally), it'd be three separate calls. One to list histories, then you would programmatically grab the one you want ("My important history") and the ID associated with it. Then you'd make a call to grab the contents of that history, from which you could extract the id of the dataset you had the name of. And from there you know how to execute a workflow with that id (hda, in this case). -Dannon On Feb 21, 2012, at 7:13 PM, thondeb...@me.com wrote: One more question... How do I pass a string value to one of the workflow input steps? Is there an equivalent of ldda and hda, maybe "str"..So 38=str="Some value"Where could I find a list of the expected dataset sources?ThanksThonOn Feb 21, 2012, at 03:46 PM, Dannon Baker dannonba...@me.com wrote:Hi Thon,You have the right idea about what's going wrong here. Galaxy is trying to pull up a library dataset with the ID you specify, but it doesn't exist.In this context, src refers to the 'type' of dataset input id, more specifically whether it's from a history or library. 'hda' indicates that the dataset is from a history, and 'ldda' is what you'll most likely use in the case of a library dataset. If you browse the history and library API functionality, you'll see other methods for explicitly grabbing these ids.My hunch is that you have an id from a history and that you should swap ldda to hda and give it another shot. Definitely let me know if you run into more issues, though, and I'll help figure out what's going on.-DannonOn Feb 21, 2012, at 6:22 PM, thondeb...@me.com wrote: Hi, I tried to run a workflow with the API, but get an Error 500 when I try to run the WF...The paster.log shows the following error... $ workflow_execute.py 92cc01ed93dc0f0fc91e3ded35497c0a http://srp106:8080/api/workflows ebfb8f50c6abde6d 'TEST the API' '1=ldda=7c5ebce002fc9d5c' Paster.log galaxy.web.framework ERROR 2012-02-21 14:36:33,067 Uncaught exception in exposed API method: Traceback (most recent call last): File "/home/tdeboer/code/galaxy-central/lib/galaxy/web/framework/__init__.py", line 145, in decorator return simplejson.dumps( func( self, trans, *args, **kwargs ), indent=4, sort_keys=True ) File "/home/tdeboer/code/galaxy-central/lib/galaxy/web/api/workflows.py", line 123, in create hda = ldda.to_history_dataset_association(history, add_to_history=add_to_history) AttributeError: 'NoneType' object has no attribute 'to_history_dataset_association' 172.16.108.6 - - [21/Feb/2012:14:36:32 -0700] "POST /api/workflows?key=92cc01ed93dc0f0fc91e3ded35497c0a HTTP/1.1" 500 - "-" "Python-urllib/2.6" Any ideas? Also...I had a hard time finding out what I should use for the dataset source parameter "src" in "step=src="" and just tried ldda, but would hope there is a little info on what this hda and ldda is? Thanks, Thon ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/One ___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Extracting the metrics files from a composite file such as Picard
Hi Ross,Thanks for the reply...It is indeed as I feared...I'll probably refactor the code so the user is able to get the actual output files.ThonOn Feb 20, 2012, at 01:29 PM, Ross ross.laza...@gmail.com wrote:Hi, Thon,On Tue, Feb 21, 2012 at 6:47 AM, Thon Deboer thondeb...@me.com wrote:Hi,Is there a way to extract the individual files from a composite file, such as the HTML files created by the picard tools?The decision to hide multiple outputs in a single history html object has this as a downside to the benefits of less cluttered histories.The results you want can be manually extracted in the usual ways - eg pasting the relevant html page url into an upload box.. but that doesn't solve your challenge of automating the process for a very large number of datasets.One possible solution is to add some complexity to each of the relevant tool forms to allow the user to specifically nominate outputs to be returned as individual new history datasets.It is not a huge task but it's not as far as I'm aware, high on the list of priorities for the team.If you are motivated sufficiently to fix this so it can do what you need, contributions of code to improve Galaxy tools are always very welcome ?I would like to take the metrics files and use them further down in some workflow, but I only get an HTML file...While these HTML files are nice for quickly looking at the results of one or two files, it becomes a problem if you have 88 samples like I do...I hate to have to re-write all the picard tools to produce actual files, but maybe there is something I am missing about composite datatype files?The wiki only explains how to CREATE them, not how to use those files downstreamRegards,Thon de Boer, Ph.D.Bioinformatics Guru+1-650-799-6839thondeb...@me.comLinkedIn Profile___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/-- Ross Lazarus MBBS MPH;Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Running workflows on mulitple paired-end data sets
Hi,Is there an intelligent way to run a workflow that starts with more than one pair of paired-end data sets in a workflow?I can run multiple workflows when there is a SINGLE input file, but in the case of many paired end workflows, you need to provide two paired samples and just adding another input workflow step does not work, since you can only select a single file in the second input workflow set.I realize that there is an issue with the cycling through the datasets since they need to be paired-up and it would be very easy to mess this up if you allow two (or more) input data sets, but since paired end data is going to be very common, I would assume a special paired end input workflow step that is intelligent and can pair up the input datasets before handing them off to the workflow would be very useful.I can think of workarounds ofcourse, such as creating an interlaced file containing both pairs and then just de-interlacing them in the workflow, but that is kludgy and will result in a lot of data duplication for no reason...I have to run 88 samples which will soon grow to over 200 samples, so running each step manually is not really an option and I would hate to have to program the workflow steps myself...Any ideas?ThanksThon___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Problem with GMAP datatypes
I'll give it a try and let you know if I run into any more problemsThanksThonOn Feb 10, 2012, at 12:00 PM, Greg Von Kuster g...@bx.psu.edu wrote:Thon, I believe this is fixed in change set 6711:62fc9e053835, which is now available in out central repo. However, I do not have access to any data of the required type, so if you still encounter problems, could you send me some data that I can use to reproduce the behavior? Thanks! Greg On Feb 9, 2012, at 9:04 PM, Greg Von Kuster wrote: Hi Thon,I've been working on fixes which I believe may be related to this. When uploading a file, the proprietary datatypes / sniffers (datatypes included in installed tool shed repositories) are not correctly handled by the upload process. Also, when running jobs, if setting metadata externally, the same issue occurs. I'm very close to a fix for this - I just need to polish up the changes. I'm planning to have the fix committed tomorrow - I'll let you know.Thanks for reporting this.Greg On Feb 9, 2012, at 5:32 PM, Anthonius deBoer wrote:I tried to install the GMAP from the main toolshed, but I am getting this error when I actually try to run the GSNAP/GMAP toolsIt seems that there is a problem with the definition of the datatypes...When I check the paster.log it does complain a little, when loading GMAP stuff, but it seems just a problem with the markup of the help textAny ideas? Error in galaxy when running GSNAP WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmapdb': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmapsnpindex': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'iit': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'splicesites.iit': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'introns.iit': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'snps.iit': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmap_annotation': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmap_splicesites': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmap_introns': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmap_snps': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:IntervalAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:SpliceSiteAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:IntronAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:SNPAnnotation' to sniff_order: No module named gmapPASTER.LOG ==galaxy.util.shed_util DEBUG 2012-02-09 12:51:43,862 Installing repository 'gmap' galaxy.util.shed_util DEBUG 2012-02-09 12:51:43,914 Cloning http://toolshed.g2.bx.psu.edu/repos/jjohnson/gmap galaxy.util.shed_util DEBUG 2012-02-09 12:51:46,834 Updating cloned repository to revision "93911bac43da" docutils WARNING 2012-02-09 12:51:47,128 string:10: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.galaxy.util.shed_util DEBUG 2012-02-09 12:51:47,190 Adding new row (or updating an existing row) for repository 'gmap' in the tool_shed_repository table. docutils WARNING 2012-02-09 12:51:47,238 string:10: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.docutils WARNING 2012-02-09 12:51:47,332 string:10: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.galaxy.tools DEBUG 2012-02-09 12:51:47,391 Reloading section: GMAP galaxy.tools DEBUG 2012-02-09 12:51:47,413 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap_iit_store/2.0.0, version: 2.0.0. docutils WARNING 2012-02-09 12:51:47,445 string:10: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.galaxy.tools DEBUG 2012-02-09 12:51:47,462 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gsnap/2.0.1, version: 2.0.1. galaxy.tools DEBUG 2012-02-09 12:51:47,482 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap_snpindex/2.0.0, version: 2.0.0. galaxy.tools DEBUG 2012-02-09 12:51:47,501 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap_build/2.0.0, version: 2.0.0. galaxy.tools DEBUG 2012-0
[galaxy-dev] Problem with GMAP datatypes
I tried to install the GMAP from the main toolshed, but I am getting this error when I actually try to run the GSNAP/GMAP toolsIt seems that there is a problem with the definition of the datatypes...When I check the paster.log it does complain a little, when loading GMAP stuff, but it seems just a problem with the markup of the help textAny ideas?Error in galaxy when running GSNAPWARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmapdb': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmapsnpindex': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'iit': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'splicesites.iit': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'introns.iit': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'snps.iit': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmap_annotation': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmap_splicesites': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmap_introns': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'gmap_snps': 'module' object has no attribute 'gmap' WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:IntervalAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:SpliceSiteAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:IntronAnnotation' to sniff_order: No module named gmap WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.gmap:SNPAnnotation' to sniff_order: No module named gmapPASTER.LOG==galaxy.util.shed_util DEBUG 2012-02-09 12:51:43,862 Installing repository 'gmap'galaxy.util.shed_util DEBUG 2012-02-09 12:51:43,914 Cloning http://toolshed.g2.bx.psu.edu/repos/jjohnson/gmapgalaxy.util.shed_util DEBUG 2012-02-09 12:51:46,834 Updating cloned repository to revision "93911bac43da"docutils WARNING 2012-02-09 12:51:47,128 string:10: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.galaxy.util.shed_util DEBUG 2012-02-09 12:51:47,190 Adding new row (or updating an existing row) for repository 'gmap' in the tool_shed_repository table.docutils WARNING 2012-02-09 12:51:47,238 string:10: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.docutils WARNING 2012-02-09 12:51:47,332 string:10: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.galaxy.tools DEBUG 2012-02-09 12:51:47,391 Reloading section: GMAPgalaxy.tools DEBUG 2012-02-09 12:51:47,413 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap_iit_store/2.0.0, version: 2.0.0.docutils WARNING 2012-02-09 12:51:47,445 string:10: (WARNING/2) Explicit markup ends without a blank line; unexpected unindent.galaxy.tools DEBUG 2012-02-09 12:51:47,462 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gsnap/2.0.1, version: 2.0.1.galaxy.tools DEBUG 2012-02-09 12:51:47,482 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap_snpindex/2.0.0, version: 2.0.0.galaxy.tools DEBUG 2012-02-09 12:51:47,501 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap_build/2.0.0, version: 2.0.0.galaxy.tools DEBUG 2012-02-09 12:51:47,534 Loaded tool id: toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/gmap/2.0.1, version: 2.0.1.galaxy.datatypes.registry DEBUG 2012-02-09 12:51:49,247 Loading datatypes from /mnt/ngs/analysis/tdeboer/galaxy-data/shed_tools/toolshed.g2.bx.psu.edu/repos/jjohnson/gmap/93911bac43da/gmap/tool-data/datatypes_conf.xmlgalaxy.datatypes.registry DEBUG 2012-02-09 12:51:49,248 Loaded sniffer for datatype 'galaxy.datatypes.gmap:IntervalAnnotation'galaxy.datatypes.registry DEBUG 2012-02-09 12:51:49,248 Loaded sniffer for datatype 'galaxy.datatypes.gmap:SpliceSiteAnnotation'galaxy.datatypes.registry DEBUG 2012-02-09 12:51:49,248 Loaded sniffer for datatype 'galaxy.datatypes.gmap:IntronAnnotation'galaxy.datatypes.registry DEBUG 2012-02-09 12:51:49,248 Loaded sniffer for datatype 'galaxy.datatypes.gmap:SNPAnnotation'172.17.50.220 - - [09/Feb/2012:12:51:43 -0700] "POST
Re: [galaxy-dev] Problem with latest database upgrade
Thanks for fixing it so fast! Regards, Thon Thon de Boer, Ph.D Bioinformatics Guru T: +1.650.799.6839 | E-mail: thondeb...@me.com http://www.linkedin.com/pub/thon-de-boer/1/1ba/a5b IMPORTANT NOTICE: This email message is legally privileged, confidential and is for the use of the individual or entity to whom it is addressed. If you have received this email message by error, please immediately notify us by email and delete the message. Thank you. On Feb 7, 2012, at 8:11 AM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Thon, The fix for this issue in now available in the central repo in change set revision 271ccd09bc23. Even though you have a working version of Galaxy, you should update your instance to at least this revision, and then do the following as soon as possible. %sh manage_db.sh downgrade 90 91 - 90... done %sh manage_db.sh upgrade 90 - 91... The problem was caused because you are running postgres version 9 and the original version of my migration script did not handle hex bytea for that version of postgres. Thanks for finding this problem! Greg On Feb 6, 2012, at 10:08 PM, Anthonius deBoer wrote: Oh, and whatever you did in the little patch, it worked...At least I have a working version of galaxy again and the database seems to be all intact.. Thanks! Thon On Feb 06, 2012, at 03:52 PM, Greg Von Kuster g...@bx.psu.edu wrote: Can you replace lines 75 - 86 of the file ~/lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py with the following for loop and send me your paster log? Thanks! for row in result: tool_shed_repository_id = row[0] try: repository_metadata = from_json_string( str( row[1] ) ) # Create a new row in the tool table for each tool included in repository. We will NOT # handle tool_version_associaions because we do not have the information we need to do so. tools = repository_metadata.get( 'tools', [] ) for tool_dict in tools: cmd = INSERT INTO tool_version VALUES (%s, %s, %s, '%s', %s) % \ ( nextval( 'tool_version' ), localtimestamp(), localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id ) db_session.execute( cmd ) count += 1 except Exception, e: print Repository metadata: , str( row[1] ) print Exception setting tool versions for tools in tool_shed_repository with id: , str( tool_shed_repository_id ) print str( e ) The lines of code you are replacing look like this: for row in result: if row[1]: tool_shed_repository_id = row[0] repository_metadata = from_json_string( str( row[1] ) ) # Create a new row in the tool table for each tool included in repository. We will NOT # handle tool_version_associaions because we do not have the information we need to do so. tools = repository_metadata.get( 'tools', [] ) for tool_dict in tools: cmd = INSERT INTO tool_version VALUES (%s, %s, %s, '%s', %s) % \ ( nextval( 'tool_version' ), localtimestamp(), localtimestamp(), tool_dict[ 'guid' ], tool_shed_repository_id ) db_session.execute( cmd ) count += 1 On Feb 6, 2012, at 5:23 PM, Anthonius deBoer wrote: I manually dropped the tool_version and tool_association table in the database and I am left with the JSON error as shown below: 90 - 91... Migration script to create the tool_version and tool_version_association tables and drop the tool_id_guid_map table. Traceback (most recent call last): File ./scripts/manage_db.py, line 63, in module main( repository=repo, url=db_url ) File /home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py, line 150, in main ret = command_func(**kwargs) File /home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py, line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File /home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py, line 349, in _migrate schema.runchange(ver, change, changeset.step) File /home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py, line 184, in runchange change.run(self.engine, step) File /home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py, line 101, in run func() File lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py, line 78, in upgrade repository_metadata = from_json_string( str( row[1] ) ) File /home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2
Re: [galaxy-dev] Problem with latest database upgrade
Hi Greg,I tried the downgrade/upgrade path and got the exact same message I sent before.I am using a postgres database that is running on local host.I did indeed install some toolshed repositories before I tried to upgrade.I also tried to downgrade to 89 and then upgrade, with the exact same message.ThonOn Feb 06, 2012, at 10:51 AM, Greg Von Kuster g...@bx.psu.edu wrote:Hello Thon,Attempting to run the migration script twice to the same version always poses problems. Can you downgrade successfully as follows?sh manage_db.sh downgrade 90If so, downgrade, then upgrade again and send the paster log from the clean upgrade if you see a problem. Also:What database are you using? Have you installed any tool shed repositories into your local Galaxy instance?Thanks,GregOn Feb 6, 2012, at 1:32 PM, Thon Deboer wrote:Hi,I updated to the latest galaxy-central version but got this error when trying to upgrade...Any ideas? It seems the table had already been created (this was the second time I run upgrade) but the JSON error was one that showed up the first time...)ThanksThonsh manage_db.sh upgrade90 - 91...Migration script to create the tool_version and tool_version_association tables and drop the tool_id_guid_map table.0091_add_tool_version_tables DEBUG 2012-02-06 10:23:39,847 Creating tool_version table failed: (ProgrammingError) relation "tool_version" already exists'\nCREATE TABLE tool_version (\n\tid SERIAL NOT NULL, \n\tcreate_time TIMESTAMP WITHOUT TIME ZONE, \n\tupdate_time TIMESTAMP WITHOUT TIME ZONE, \n\ttool_id VARCHAR(255), \n\ttool_shed_repository_id INTEGER, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_shed_repository_id) REFERENCES tool_shed_repository (id)\n)\n\n' {}0091_add_tool_version_tables DEBUG 2012-02-06 10:23:39,847 Creating tool_version table failed: (ProgrammingError) relation "tool_version" already exists'\nCREATE TABLE tool_version (\n\tid SERIAL NOT NULL, \n\tcreate_time TIMESTAMP WITHOUT TIME ZONE, \n\tupdate_time TIMESTAMP WITHOUT TIME ZONE, \n\ttool_id VARCHAR(255), \n\ttool_shed_repository_id INTEGER, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_shed_repository_id) REFERENCES tool_shed_repository (id)\n)\n\n' {}0091_add_tool_version_tables DEBUG 2012-02-06 10:23:39,863 Creating tool_version_association table failed: (ProgrammingError) relation "tool_version_association" already exists'\nCREATE TABLE tool_version_association (\n\tid SERIAL NOT NULL, \n\ttool_id INTEGER NOT NULL, \n\tparent_id INTEGER NOT NULL, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_id) REFERENCES tool_version (id), \n\t FOREIGN KEY(parent_id) REFERENCES tool_version (id)\n)\n\n' {}0091_add_tool_version_tables DEBUG 2012-02-06 10:23:39,863 Creating tool_version_association table failed: (ProgrammingError) relation "tool_version_association" already exists'\nCREATE TABLE tool_version_association (\n\tid SERIAL NOT NULL, \n\ttool_id INTEGER NOT NULL, \n\tparent_id INTEGER NOT NULL, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_id) REFERENCES tool_version (id), \n\t FOREIGN KEY(parent_id) REFERENCES tool_version (id)\n)\n\n' {}Traceback (most recent call last): File "./scripts/manage_db.py", line 63, in module main( repository=repo, url="" ) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py", line 78, in upgrade repository_metadata = from_json_string( str( row[1] ) ) File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/__init__.py", line 384, in loads File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/decoder.py", line 402, in decode File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/decoder.py", line 420, in raw_decodesimplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)Regards,Thon de Boer, Ph.D.Bioinformatics Guru+1-650-799-6839thondeb...@me.comLinkedIn Profile___Please keep all replies on the list by using "reply all"in your mail client. To manage your subscriptions to thisand other Galaxy lists, please use the interface at:
Re: [galaxy-dev] Problem with latest database upgrade
I manually dropped the tool_version and tool_association table in the database and I am left with the JSON error as shown below:90 - 91...Migration script to create the tool_version and tool_version_association tables and drop the tool_id_guid_map table.Traceback (most recent call last): File "./scripts/manage_db.py", line 63, in module main( repository=repo, url="" ) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/shell.py", line 150, in main ret = command_func(**kwargs) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 221, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/api.py", line 349, in _migrate schema.runchange(ver, change, changeset.step) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/schema.py", line 184, in runchange change.run(self.engine, step) File "/home/tdeboer/code/galaxy-central/eggs/sqlalchemy_migrate-0.5.4-py2.6.egg/migrate/versioning/script/py.py", line 101, in run func() File "lib/galaxy/model/migrate/versions/0091_add_tool_version_tables.py", line 78, in upgrade repository_metadata = from_json_string( str( row[1] ) ) File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/__init__.py", line 384, in loads File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/decoder.py", line 402, in decode File "/home/tdeboer/code/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs2.egg/simplejson/decoder.py", line 420, in raw_decodesimplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)On Feb 06, 2012, at 02:08 PM, Anthonius deBoer thondeb...@me.com wrote:Hi Greg,I tried the downgrade/upgrade path and got the exact same message I sent before.I am using a postgres database that is running on local host.I did indeed install some toolshed repositories before I tried to upgrade.I also tried to downgrade to 89 and then upgrade, with the exact same message.ThonOn Feb 06, 2012, at 10:51 AM, Greg Von Kuster g...@bx.psu.edu wrote:Hello Thon,Attempting to run the migration script twice to the same version always poses problems. Can you downgrade successfully as follows?sh manage_db.sh downgrade 90If so, downgrade, then upgrade again and send the paster log from the clean upgrade if you see a problem. Also:What database are you using? Have you installed any tool shed repositories into your local Galaxy instance?Thanks,GregOn Feb 6, 2012, at 1:32 PM, Thon Deboer wrote:Hi,I updated to the latest galaxy-central version but got this error when trying to upgrade...Any ideas? It seems the table had already been created (this was the second time I run upgrade) but the JSON error was one that showed up the first time...)ThanksThonsh manage_db.sh upgrade90 - 91...Migration script to create the tool_version and tool_version_association tables and drop the tool_id_guid_map table.0091_add_tool_version_tables DEBUG 2012-02-06 10:23:39,847 Creating tool_version table failed: (ProgrammingError) relation "tool_version" already exists'\nCREATE TABLE tool_version (\n\tid SERIAL NOT NULL, \n\tcreate_time TIMESTAMP WITHOUT TIME ZONE, \n\tupdate_time TIMESTAMP WITHOUT TIME ZONE, \n\ttool_id VARCHAR(255), \n\ttool_shed_repository_id INTEGER, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_shed_repository_id) REFERENCES tool_shed_repository (id)\n)\n\n' {}0091_add_tool_version_tables DEBUG 2012-02-06 10:23:39,847 Creating tool_version table failed: (ProgrammingError) relation "tool_version" already exists'\nCREATE TABLE tool_version (\n\tid SERIAL NOT NULL, \n\tcreate_time TIMESTAMP WITHOUT TIME ZONE, \n\tupdate_time TIMESTAMP WITHOUT TIME ZONE, \n\ttool_id VARCHAR(255), \n\ttool_shed_repository_id INTEGER, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_shed_repository_id) REFERENCES tool_shed_repository (id)\n)\n\n' {}0091_add_tool_version_tables DEBUG 2012-02-06 10:23:39,863 Creating tool_version_association table failed: (ProgrammingError) relation "tool_version_association" already exists'\nCREATE TABLE tool_version_association (\n\tid SERIAL NOT NULL, \n\ttool_id INTEGER NOT NULL, \n\tparent_id INTEGER NOT NULL, \n\tPRIMARY KEY (id), \n\t FOREIGN KEY(tool_id) REFERENCES tool_version (id), \n\t FOREIGN KEY(parent_id) REFERENCES tool_version (id)\n)\n\n' {}0091_add_tool_version_tables DEBUG 2012-02-06 10:23:39,863 Creating tool_version_association table failed: (ProgrammingError) relation "tool_version_association" already exists'\nCREATE TABLE tool_version_association (\n\tid SERIAL NOT NULL, \n\ttool_id INTEGER NOT NULL, \n\tparent_id INTEGER N
Re: [galaxy-dev] software installs: PATH vs env.sh
All tools need to be in the path of the user running galaxy. Regards, Thon Thon de Boer, Ph.D Bioinformatics Guru T: +1.650.799.6839 | E-mail: thondeb...@me.com http://www.linkedin.com/pub/thon-de-boer/1/1ba/a5b IMPORTANT NOTICE: This email message is legally privileged, confidential and is for the use of the individual or entity to whom it is addressed. If you have received this email message by error, please immediately notify us by email and delete the message. Thank you. On Jan 21, 2012, at 3:50 PM, Andrew Warren anwar...@vbi.vt.edu wrote: Hello, We recently transitioned from a CloudMan instance of galaxy to our own cluster and started having problems with calls to tools from within other tools. For example when Tophat calls bowtie-inspect its not finding the executable. To fix this I listed bowtie in the requirements section of the tophat wrapper like so: tool id=tophat name=Tophat for Illumina version=1.5.0 descriptionFind splice junctions using RNA-seq data/description version_commandtophat --version/version_command requirements requirement type=packagetophat/requirement requirement type='package'bowtie/requirement requirement type=packagesamtools/requirement /requirements Now I am wondering, is it generally expected that all tools used by galaxy will have their executables on the user galaxy's PATH? Is the above a good solution? Or is there something else likely amiss with our galaxy setup? I think we recently pulled updates for some major tool_shed release but I haven't been able to determine if any of the tools listed above were affected by that. Wish I were in Český Krumlov asking this question. Missed the registration deadline...doh. Thanks, Andrew Warren ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Toolshed tools are not run
Hi,I am trying to install some tools from the main toolshed, and the installation went OK for the tool "tabToHtml (really something that galaxy should do out of the box...Show tab data in actual tables, rather than a dump of text).But when I try to run the tool, it is just marked as queued only and never actually runs...Anyone know what this could be?Using the latest galaxy-central on a local installation with an SGE cluster.Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Toolshed tools are not run
Thanks, but I don't think this is a problem with the tool itself, but with galaxy.I think it just does not run the toolshed tool...ThonOn Jan 11, 2012, at 01:04 PM, Greg Von Kuster g...@bx.psu.edu wrote:Hello Thon, In case you don't get an answer from the community, you can send your question to the repository owner by selecting the option from the "Repository Options" pop-up menu in the upper right corner when viewing the repository in the tool shed. repository_actions.tiff Greg Von Kuster On Jan 11, 2012, at 3:46 PM, Anthonius deBoer wrote: Hi,I am trying to install some tools from the main toolshed, and the installation went OK for the tool "tabToHtml (really something that galaxy should do out of the box...Show tab data in actual tables, rather than a dump of text).But when I try to run the tool, it is just marked as queued only and never actually runs...Anyone know what this could be?Using the latest galaxy-central on a local installation with an SGE cluster.Thon ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:http://lists.bx.psu.edu/ Greg Von Kuster Galaxy Development Team g...@bx.psu.edu I___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Getting global name 'copy_sample_loc_file' is not defined error when trying to install tool
Im trying to install a tool from the toolshed that showed me this error. NameError: global name 'copy_sample_loc_file' is not definedI know the tool is trying to install the loc file...Does this not work yet?Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
