Re: [galaxy-dev] Error trying to add tools from tool shed respository
Ok, so that confirms that you definitely have local customizations we'll want to make sure to keep. What I'd do before we go any further is this: `hg diff -p local_changes.patch` This will save all of your current changes to a patch, should we have to recreate them (though I don't expect this). `hg ci -m Local changes to cleanup_datasets and various tools` Which will commit your local changes to your local repository. At this point, hopefully, you can just 'hg update', possibly needing to merge as well. For IRC: Just join the Freenode network shown in XChat, and then Join Channel should pop up a box you can put #galaxyproject in On Jan 29, 2013, at 1:52 PM, Waldron, Michael H mwald...@email.unc.edu wrote: Here is hg st output: $ hg st M scripts/cleanup_datasets/delete_datasets.sh M scripts/cleanup_datasets/delete_datasets_main.sh M scripts/cleanup_datasets/delete_userless_histories.sh M scripts/cleanup_datasets/delete_userless_histories_main.sh M scripts/cleanup_datasets/purge_datasets.sh M scripts/cleanup_datasets/purge_datasets_main.sh M scripts/cleanup_datasets/purge_folders.sh M scripts/cleanup_datasets/purge_folders_main.sh M scripts/cleanup_datasets/purge_histories.sh M scripts/cleanup_datasets/purge_histories_main.sh M scripts/cleanup_datasets/purge_libraries.sh M scripts/cleanup_datasets/purge_libraries_main.sh M scripts/cleanup_datasets/update_metadata.sh M tools/peak_calling/macs_wrapper.xml M tools/plotting/boxplot.xml M tools/sr_mapping/bfast_wrapper.xml M tools/sr_mapping/bowtie2_wrapper.xml M tools/sr_mapping/bowtie_color_wrapper.xml M tools/sr_mapping/bowtie_wrapper.xml M tools/sr_mapping/bwa_color_wrapper.xml M tools/sr_mapping/bwa_wrapper.xml ? contrib/galaxy-init ? galaxy_cleanup.log ? galaxy_cleanup.sh ? migrated_tools_conf.xml.110712 ? paster.pid ? scripts/cleanup_datasets/delete_userless_histories.log ? scripts/cleanup_datasets/purge_datasets.log ? scripts/cleanup_datasets/purge_folders.log ? scripts/cleanup_datasets/purge_histories.log ? scripts/cleanup_datasets/purge_libraries.log ? static/UNC_galaxy_mirror_housekeeping_notes.pdf ? tool-data/bwa_index.loc.sample ? tool-data/bwa_index_color.loc.sample ? tool-data/tool_data_table_conf.xml.sample ? tool_conf.xml.110212 ? tool_conf.xml.new ? tool_data_table_conf.xml.030112 ? tools/plotting/boxplot.xml.012512 ? typescript The hg diff output is quite long. It turns out I didn't connect to the IRC channel. I have XChat IRC client, but not familiar with using it. How to I connect to the irc.freenode.net server to access the #galaxyproject channel? Mike Waldron Systems Specialist ITS - Research Computing Center University of North Carolina at Chapel Hill From: Dannon Baker [dannonba...@me.com] Sent: Tuesday, January 29, 2013 1:48 PM To: Waldron, Michael H Cc: Paul Boddie; James Taylor; Greg Von Kuster; galaxy-...@bx.psu.edu Dev Subject: Re: [galaxy-dev] Error trying to add tools from tool shed respository I don't know why I thought we'd already checked this, but -- 'hg st' will tell you which files have been modified in your local repository, and 'hg diff' will tell you exactly what changes have taken place I'm not sure of the exact state of your repository at the moment, so this *might* display all changes from the pull as well, which wouldn't be particularly useful. Worth trying, though. Don't see you in #galaxyproject yet, sure it's working? On Jan 29, 2013, at 1:42 PM, Waldron, Michael H mwald...@email.unc.edu wrote: The hg update --check command reports the following: $ hg update --check abort: uncommitted local changes I just connected to the #galaxyproject IRC channel. Mike Waldron Systems Specialist ITS - Research Computing Center University of North Carolina at Chapel Hill From: Dannon Baker [dannonba...@me.com] Sent: Tuesday, January 29, 2013 1:37 PM To: Waldron, Michael H Cc: Paul Boddie; James Taylor; Greg Von Kuster; galaxy-...@bx.psu.edu Subject: Re: [galaxy-dev] Error trying to add tools from tool shed respository So that's interesting. My best guess was that this was caused by swapping from galaxy-dist to galaxy-central in the situation where galaxy-dist has had changesets applied directly and then ported to galaxy-central out of order, combined with your oder version of mercurial. That parent, however, would have been before that happened. Anyway, at this point, does `hg update --check` actually work? And, lastly, If you want -- hop on IRC (irc.freenode.net #galaxyproject, http://wiki.galaxyproject.org/Get%20Involved#IRC_Channel) and it might be faster to iterate through what we need to do to get your repository back in shape. On Jan 29, 2013, at 1:32 PM, Waldron, Michael H mwald...@email.unc.edu wrote: Forgot to also include the hg parents output. $ hg parents
Re: [galaxy-dev] Uploading large files as AdminŠ Still slow!
What probably happened here is that due to the filesize, the browser upload failed but this went undetected. The good news is that you *can* tell galaxy to copy directly, or you could even use the files exactly where they are without any copy. What you want to do is enable this section in your universe_wsgi.ini, and then restart galaxy, after which you'll be able to just paste in the local path and galaxy will handle the rest. # Add an option to the admin library upload tool allowing admins to paste # filesystem paths to files and directories in a box, and these paths will be # added to a library. Set to True to enable. Please note the security # implication that this will give Galaxy Admins access to anything your Galaxy # user has access to. #allow_library_path_paste = False -Dannon On Jan 28, 2013, at 8:21 AM, Jackie Lighten jackie.ligh...@dal.ca wrote: Hi, I am trying to upload 2x 2.5GB files to my data libraries (in Admin on a local instance) but it has so far been running over night and not completed. How can I speed this up? Can I upload compressed files? Can I not just copy these files directly in to $galaxy-dist/database/files/000 ? Thanks, Jack ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy with EMBOSS 5 or EMBOSS 6 ?
Hey Simon, You're right -- the EMBOSS version supported by the Galaxy wrappers is currently EMBOSS 5. We do have a Tool Dependency page here: http://wiki.galaxyproject.org/Admin/Tools/Tool%20Dependencies, but unfortunately while it lists versions for many tools it doesn't cover them all. I'll update the EMBOSS entry. As far as updating the galaxy EMBOSS wrappers to use a newer version, I'm not aware of any plans to do this currently (though someone could certainly do this if so inclined). -Dannon On Jan 28, 2013, at 3:10 PM, Guest, Simon simon.gu...@agresearch.co.nz wrote: Dear Galaxians, I am troubleshooting a problem with running EMBOSS infoseq on our local galaxy installation. We are running EMBOSS 6.5. There is a mismatch between the options Galaxy is passing to infoseq (-version), and what infoseq wants (-seqversion). This breaks the tool. While this would be easy to fix in this instance, I wonder if I should be using EMBOSS 5 (which is *really* old, mind you) instead of EMBOSS 6. I suspect with EMBOSS 6 there may be a whole host of similar changes to be made. Can anyone advise? If it turns out that EMBOSS 5 is actually required, perhaps someone could update the relevant info on the Galaxy dependencies page, or at least point me to the place where it says this. cheers, Simon Simon Guest Senior UNIX Technical Consultant AgResearch === Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. === ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] minimum AWS resources for TopHat?
Hey Andy, Financially, it's probably best to start small cluster-wise. What I'd probably recommend for your particular project would be using a single m1.xlarge instance as the head node, seeing how that goes, and then adding workers as you find it useful. Should you find that it isn't enough, it's trivially easy using cloudman to shut the cluster down and restart with a much larger AMI. Regarding extra nodes -- Given that you're a single user, in serial pipelines and workflows where each job depends on the previous one it isn't useful to have extra instances at all and you'd only waste money. If your analysis can be done in parallel, however, say you have multiple samples all requiring the same basic preliminary steps, then extra nodes can definitely help get the work done much faster. You could also use cloudman's autoscaling to handle this; it would automatically scale up the cluster (while adhering to your min/max parameters) as necessary to process jobs as fast as possible while trimming any idle nodes to prevent waste. Lastly, depending on the analysis you need to do, you may find you need a high memory instance. In this case (given your m1.xlarge head node) you can either restart your instance using the larger node, or even simpler disable job running on the head node in the interface and add a high memory worker instance temporarily to handle the special demand. Let me know if there's anything else I can do to help! -Dannon On Jan 23, 2013, at 3:40 PM, Andrew Norman anorma...@gmail.com wrote: Hi all I'd like to use AWS EC2 cluster to run Galaxy TopHat to analyze the RNA seq data for my project. I'm trying to price out the resources that I'll need to do this, but I don't have any experience setting up clusters, virtual or real, so I'd like to get the insight of someone who has done this. I have studied the wiki page dedicated to this topic (http://wiki.galaxyproject.org/CloudMan/AWS/CapacityPlanning) but I don't know how many worker nodes I'll need, and I will just be doing my personal analysis with this cluster (only one TopHat analysis at a time). Can anyone help me out with this? Thanks! Andy ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy Main cloudlaunch stall
Cloudlaunch on the main public instance is fully supported and should work fine -- I use it regularly for launching instances without issue. There was a brief EC2 API outage yesterday (on the Amazon end) that caused intermittent errors to all users of the API (including cloudlaunch), but that should be resolved now. Creating security groups and selecting (or creating, if necessary) an access key should be possible -- since it's not even getting that far I'm wondering if there isn't a Javascript error or something preventing the form submission. What browser are you using, and do you see any errors? -Dannon On Jan 17, 2013, at 5:11 PM, Matthew Shirley mshir...@jhmi.edu wrote: I've searched the list to see if anyone specifically mentions not being able to use the cloud launch feature on the main public instance, but no one seems to address it. I have no trouble launching Galaxy-cloudman using an AMI and pasting in my own user data. However, I can't successfully use the cloud launch page to do the same. I've tried letting it create a new security group as well as using a pre-existing security group. Also, I've then deleted all security groups to see if it would create one, which it did not. Similarly, I tried creating a new access key pair, with no further success. I make it through the process of entering the key ID and secret key, it pulls existing security group information into the drop-down selector, and I give a cluster name and password and submit the form. Then it just stalls forever. No new security groups and no EC2 instances launched. What is the state of this tool? Thanks. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] Workflows
Geert, this is great stuff! One small correction -- the API is enabled by default as of revision 7022:8376ad08ae41 (April 2012). -Dannon On Jan 16, 2013, at 1:17 AM, Geert Vandeweyer geertvandewe...@gmail.com wrote: Hi, I've put together some exempels for activating and using the api. You can find them here: http://geertvandeweyer.zymichost.com/index.php?page=readid=29 Best regards, Geert vandeweyer Op 16-jan.-2013 om 02:52 heeft neil.burd...@csiro.au het volgende geschreven: Hi Sorry if these questions are obvious but I just don’t know how to find the answers. I’m trying to get one of the API examples to work in http://wiki.galaxyproject.org/Learn/API/Examples . I’ve got my API key but how do I get/find the workflow id (f2db41e1fa331b3e in the examples). I’ve created workflow but don’t know how to access this key. Also in the first example it states galaxy_url/api/workflows However I don’t have the “api/workflows” directory structure in my galaxy-dist, do I need to create this? Thanks Neil ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] galaxy using EC2 spot pricing
While it isn't bulletproof yet (restarting jobs that fail due to spot instance reclaims, etc), Cloudman (usegalaxy.org/cloud) does currently have support for running worker nodes as spot instances. When you go to add nodes, just click Use Spot Instances, and add your spot price. For obvious reasons, you wouldn't want the head node of your cluster to be a spot instance, use a normal one for that. -Dannon On Jan 18, 2013, at 11:58 AM, Andrew Norman anorma...@gmail.com wrote: Does anyone have any experience running Galaxy using Amazon's EC2 spot pricing? It seems like that could save a lot of money as compared to on-demand pricing, and it appears that there are ways to mix on-demand and spot instances within one processing cluster: http://aws.amazon.com/ec2/spot-and-science/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy Main cloudlaunch stall
Should definitely be stable enough to use in a classroom and that it isn't working for you is definitely unexpected; it's the only way I've launched instances for many months now. I'm digging through logs to see if I can find any indication of why your particular requests are having issues. On Jan 18, 2013, at 12:33 PM, Matthew Shirley mshir...@jhmi.edu wrote: Thanks, Quang. That's not exactly what I'm looking for. My motivation is to teach an entire class how to start a Galaxy cluster on their own, with minimal knowledge of AWS or systems administration. The cloudlaunch feature on the main psx site seems to fit this purpose exactly, but does not work reliably for me. I've just tried using Chrome 24.0 with no success. I'm using no browser plugins, and this is a clean installation with javascript enabled. The cloudlaunch page just sits there and tells me Launch Pending, please be patient. and I have - it's been over an hour. Any help figuring this out is appreciated. If this feature is just not mature enough to use in a classroom, that would be an acceptable answer as well. On Jan 18, 2013, at 11:40 AM, Quang Trinh quang.tr...@gmail.com wrote: Hi Matthew, Would you like to try out what we put together for modENCODE DCC? Please see the README file in the docs folder at https://github.com/modENCODE-DCC/Galaxy Thanks, Q On Fri, Jan 18, 2013 at 10:17 AM, Dannon Baker dannonba...@me.com wrote: Cloudlaunch on the main public instance is fully supported and should work fine -- I use it regularly for launching instances without issue. There was a brief EC2 API outage yesterday (on the Amazon end) that caused intermittent errors to all users of the API (including cloudlaunch), but that should be resolved now. Creating security groups and selecting (or creating, if necessary) an access key should be possible -- since it's not even getting that far I'm wondering if there isn't a Javascript error or something preventing the form submission. What browser are you using, and do you see any errors? -Dannon On Jan 17, 2013, at 5:11 PM, Matthew Shirley mshir...@jhmi.edu wrote: I've searched the list to see if anyone specifically mentions not being able to use the cloud launch feature on the main public instance, but no one seems to address it. I have no trouble launching Galaxy-cloudman using an AMI and pasting in my own user data. However, I can't successfully use the cloud launch page to do the same. I've tried letting it create a new security group as well as using a pre-existing security group. Also, I've then deleted all security groups to see if it would create one, which it did not. Similarly, I tried creating a new access key pair, with no further success. I make it through the process of entering the key ID and secret key, it pulls existing security group information into the drop-down selector, and I give a cluster name and password and submit the form. Then it just stalls forever. No new security groups and no EC2 instances launched. What is the state of this tool? Thanks. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Issues with Trello, and enhancement request
Thanks for pointing this out. I 'enhanced' the trello reporting last friday to optionally @mention submitters as a method for both claiming a submission and easier notifications, but it looks like something has gone awry and I'm looking into it now. -Dannon On Jan 14, 2013, at 9:58 AM, Joachim Jacob joachim.ja...@vib.be wrote: Hi all, I cannot post to Trello today. Actually, this problem exists from last friday already. My enhancement request (which was not yet posted on Trello - at least not in a way I could find it...): Some files I upload have a first line containing the column names. Sometimes I need to first strip of that line, before continuing analyses. Afterward, I need to paste that line back on top. Perhaps it would be more efficient to have an option set in the 'Edit attributes' of a dataset: column names from first line? Or something similar. The first line would disappear, and the content would be set as metadata on the tabular file. Thanks, Joachim -- Joachim Jacob Rijvisschestraat 120, 9052 Zwijnaarde Tel: +32 9 244.66.34 Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] macs error for chipseq data
There's an issue with the MACS installation on the current cloud tools volume (which will be fixed with the next volume update coming soon). For existing instances, you can get MACS working correctly by executing the following two commands (which change the default version of macs used) after ssh'ing in to your instance: sudo su galaxy ln -sfn /mnt/galaxyTools/tools/macs/1.3.7.1 /mnt/galaxyTools/tools/macs/default And that's it, no need to restart galaxy or anything. -Dannon On Jan 11, 2013, at 10:59 AM, Jennifer Jackson j...@bx.psu.edu wrote: Hi Kathryn, I am going to post this back to the galaxy-dev list and cc Dannon so that he or or one of the developers more experienced with troubleshooting cloud issues can help. My guess is that there is a path problem (covered in the first wiki link) but by no means is that the only possibility. Thanks! Jen Galaxy team On 1/11/13 7:38 AM, Sun, Wenping [USA] wrote: Hi Jen, Sorry that I missed some part of your email. It is not for local galaxy install. I am using the cloudman from aws and it seemed to have everything for having the input files and reference genome chose. The job was shown on the right panel and then the error message coming out. It seems to me that MACS was installed…. What is the issue? Thank you very much, Kathryn On 1/11/13 6:55 AM, Sun, Wenping [USA] wrote: Hi Jennifer, Thanks for the information on MACS tools. Yes, I forgot to mention that I used bowtie mapped files as the input. Thank you very much, Kathryn From: Jennifer Jackson [mailto:j...@bx.psu.edu] Sent: Friday, January 11, 2013 3:48 AM To: Sun, Wenping [USA] Cc: galaxy-u...@bx.psu.edu Subject: [External] Re: [galaxy-user] macs error for chipseq data Hello Kathryn, This is occurring on your local Galaxy install? Have you installed the actual MACS tool and set up the proper configuration paths? Details for how to do this are in these wikis: http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies http://wiki.galaxyproject.org/Admin/Tools/Tool%20Dependencies Apart from that, I should let you know that prior to running MACS, groomed fastq datasets first need to be mapped. An example can be found in our Using Galaxy paper, protocol #3. Links to the paper and supplemental materials (including a screencast walk-through) are available here: https://main.g2.bx.psu.edu/u/galaxyproject/p/using-galaxy-2012 If you need more help with your local install, directing questions to the galaxy-...@bx.psu.edu mailing list would be best, Thanks! Jen Galaxy team On 1/10/13 1:14 PM, Sun, Wenping [USA] wrote: Dear galaxy users, I’ve encountered the error while running macs on galaxy for chipseq data. I used fastq groomed files as input and had the following error An error occurred running this job: /bin/sh: macs: not found Anybody can kindly provide some hints? Thanks, Kathryn ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Hillman-Jackson Galaxy Support and Training http://galaxyproject.org -- Jennifer Hillman-Jackson Galaxy Support and Training http://galaxyproject.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy December 20, 2012 Distribution News Brief
No -- we're reorganizing the storage approach the cloud uses and will not be updating for this distribution release. Updates to the cloud deployment will most likely resume with the next distribution. Until then, you can always update your individual cloud instances to any revision you'd like. On Dec 22, 2012, at 8:59 AM, Scooter Willis hwil...@scripps.edu wrote: Is the cloud man Amazon EC2 instance updated with the latest? From: Jennifer Jackson j...@bx.psu.edu Date: Thursday, December 20, 2012 8:49 PM To: Galaxy Announce galaxy-annou...@bx.psu.edu, Galaxy User galaxy-u...@bx.psu.edu, Galaxy Dev galaxy-...@bx.psu.edu Subject: [galaxy-dev] Galaxy December 20, 2012 Distribution News Brief Galaxy Dec 20, 2012 Distribution News Brief Complete News Brief Highlights: • Required Metadata Reset for instances running Tool Shed repositories. Be sure to learn how. • Simplified tool config: all tools (including tool shed repos) can now have handlers and runners assigned based on their short id. • New GFF/GFF3 annotation support for Cuffcompare/merge/diff. • More Updated tools: TopHat2, Bowtie2. • Multiple pull-requests contributed by community (thanks!): #83, #85, #90, #91. • Issue tracking help is now covered in detailed in our new wiki: Using the Galaxy Issue Board • GCC2013 Training Day Topic Nominations are now open through January 11th • Plus more Tool Shed updates, Bug Fixes, and improvements to Trackster, UI, Libraries, and Histories. getgalaxy.org bitbucket.org/galaxy/galaxy-dist galaxy-dist.readthedocs.org new: $ hg clone http://www.bx.psu.edu/hg/galaxy galaxy-dist upgrade: $ hg pull -u -r 36ad8aa7b922 Best wishes for a safe and happy holiday, The Galaxy Team -- Jennifer Jackson http://galaxyproject.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy December 20, 2012 Distribution News Brief
The cloud automated update (through the admin UI) won't pull updated any updated binaries -- that's up to you just like for a local install, and you could still do it by ssh'ing in. It will update all galaxy tool wrappers (the .xml) that come with the distribution, but that's not the binary version (which I think you're asking about, correct me if I'm wrong). And, the next release will probably be in early-mid January. On Dec 22, 2012, at 9:07 AM, Scooter Willis hwil...@scripps.edu wrote: If I do the update will that also pull the latest versions of TopHat2 and Bowtie2 as an example or just the Galaxy software? Any guess on when the next distribution will release occur? On 12/22/12 9:04 AM, Dannon Baker dannonba...@me.com wrote: No -- we're reorganizing the storage approach the cloud uses and will not be updating for this distribution release. Updates to the cloud deployment will most likely resume with the next distribution. Until then, you can always update your individual cloud instances to any revision you'd like. On Dec 22, 2012, at 8:59 AM, Scooter Willis hwil...@scripps.edu wrote: Is the cloud man Amazon EC2 instance updated with the latest? From: Jennifer Jackson j...@bx.psu.edu Date: Thursday, December 20, 2012 8:49 PM To: Galaxy Announce galaxy-annou...@bx.psu.edu, Galaxy User galaxy-u...@bx.psu.edu, Galaxy Dev galaxy-...@bx.psu.edu Subject: [galaxy-dev] Galaxy December 20, 2012 Distribution News Brief Galaxy Dec 20, 2012 Distribution News Brief Complete News Brief Highlights: € Required Metadata Reset for instances running Tool Shed repositories. Be sure to learn how. € Simplified tool config: all tools (including tool shed repos) can now have handlers and runners assigned based on their short id. € New GFF/GFF3 annotation support for Cuffcompare/merge/diff. € More Updated tools: TopHat2, Bowtie2. € Multiple pull-requests contributed by community (thanks!): #83, #85, #90, #91. € Issue tracking help is now covered in detailed in our new wiki: Using the Galaxy Issue Board € GCC2013 Training Day Topic Nominations are now open through January 11th € Plus more Tool Shed updates, Bug Fixes, and improvements to Trackster, UI, Libraries, and Histories. getgalaxy.org bitbucket.org/galaxy/galaxy-dist galaxy-dist.readthedocs.org new: $ hg clone http://www.bx.psu.edu/hg/galaxy galaxy-dist upgrade: $ hg pull -u -r 36ad8aa7b922 Best wishes for a safe and happy holiday, The Galaxy Team -- Jennifer Jackson http://galaxyproject.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy December 20, 2012 Distribution News Brief
The toolshed dependency installation capabilities will fill this exact role -- for examples see how the blast+ suite or some of the other tools already migrated out of the codebase have been configured. When you install these repositories from the toolshed you have the option to automatically download, build, and install dependencies along with the wrappers. On Dec 22, 2012, at 9:19 AM, Scooter Willis hwil...@scripps.edu wrote: Hoping for the ease of apt-get for installing the binaries. Would be great if you could make that a requirement for installing specific version of binaries that align with the XML mapping in Galaxy. Would go a long way to make this easy and at the same time minimize the support requirements on why something is not working. On 12/22/12 9:11 AM, Dannon Baker dannonba...@me.com wrote: The cloud automated update (through the admin UI) won't pull updated any updated binaries -- that's up to you just like for a local install, and you could still do it by ssh'ing in. It will update all galaxy tool wrappers (the .xml) that come with the distribution, but that's not the binary version (which I think you're asking about, correct me if I'm wrong). And, the next release will probably be in early-mid January. On Dec 22, 2012, at 9:07 AM, Scooter Willis hwil...@scripps.edu wrote: If I do the update will that also pull the latest versions of TopHat2 and Bowtie2 as an example or just the Galaxy software? Any guess on when the next distribution will release occur? On 12/22/12 9:04 AM, Dannon Baker dannonba...@me.com wrote: No -- we're reorganizing the storage approach the cloud uses and will not be updating for this distribution release. Updates to the cloud deployment will most likely resume with the next distribution. Until then, you can always update your individual cloud instances to any revision you'd like. On Dec 22, 2012, at 8:59 AM, Scooter Willis hwil...@scripps.edu wrote: Is the cloud man Amazon EC2 instance updated with the latest? From: Jennifer Jackson j...@bx.psu.edu Date: Thursday, December 20, 2012 8:49 PM To: Galaxy Announce galaxy-annou...@bx.psu.edu, Galaxy User galaxy-u...@bx.psu.edu, Galaxy Dev galaxy-...@bx.psu.edu Subject: [galaxy-dev] Galaxy December 20, 2012 Distribution News Brief Galaxy Dec 20, 2012 Distribution News Brief Complete News Brief Highlights: € Required Metadata Reset for instances running Tool Shed repositories. Be sure to learn how. € Simplified tool config: all tools (including tool shed repos) can now have handlers and runners assigned based on their short id. € New GFF/GFF3 annotation support for Cuffcompare/merge/diff. € More Updated tools: TopHat2, Bowtie2. € Multiple pull-requests contributed by community (thanks!): #83, #85, #90, #91. € Issue tracking help is now covered in detailed in our new wiki: Using the Galaxy Issue Board € GCC2013 Training Day Topic Nominations are now open through January 11th € Plus more Tool Shed updates, Bug Fixes, and improvements to Trackster, UI, Libraries, and Histories. getgalaxy.org bitbucket.org/galaxy/galaxy-dist galaxy-dist.readthedocs.org new: $ hg clone http://www.bx.psu.edu/hg/galaxy galaxy-dist upgrade: $ hg pull -u -r 36ad8aa7b922 Best wishes for a safe and happy holiday, The Galaxy Team -- Jennifer Jackson http://galaxyproject.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] installing BWA from toolshed
My guess what's going on here is that you're still logged in as the ubuntu user. `sudo su galaxy` and give it another shot. -Dannon On Dec 22, 2012, at 1:51 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Quang, I didn't realize you were running on the cloud, which is not my area of expertise - my instructions were for a local Galaxy instance. I'll let others with more knowledge of the cloud environment answer your question. Please keep all replies on the mail list to enable optimal response to questions and issues. Greg Von Kuster On Dec 22, 2012, at 10:45 AM, Quang Trinh wrote: Hi Greg, Quick question: I am running off Amazon EC2 and when I do the upgrade, I get the following error: Not trusting file /mnt/galaxyTools/galaxy-central/.hg/hgrc from untrusted user galaxy, group galaxy Not trusting file /mnt/galaxyTools/galaxy-central/.hg/hgrc from untrusted user galaxy, group galaxy abort: repository default not found! How do I get around this? THanks, On 12/22/12, Quang Trinh quang.tr...@gmail.com wrote: Thanks Greg, will follow the instructions from the news brief. Q On Sat, Dec 22, 2012 at 10:14 AM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Quang, You'll need to upgrade your Galaxy instance to the distribution release that went out on December 20. Here is the link to the news brief for the December 20 release. http://wiki.galaxyproject.org/DevNewsBriefs/2012_12_20 From this point forward, all communication between Galaxy and the tool shed will be backward compatible to this December 20 release, so new tool shed features introduced in the future will function for any Galaxy instance running the December 20 release or newer. Greg Von Kuster On Dec 22, 2012, at 9:53 AM, Quang Trinh wrote: Hi, Trying to install BWA from toolshed just now and I get these error messages ( see below ). Installing BWA from toolshed works for me before. Thanks Q Server Error URL: http://54.242.71.44/admin_toolshed/prepare_for_install?tool_shed_url=http://toolshed.g2.bx.psu.edu/repository_ids=25bdd3c7f0d57b06changeset_revisions=ffa8aaa14f7c Module paste.exceptions.errormiddleware:143 in __call__ app_iter = self.application(environ, start_response) Module paste.debug.prints:98 in __call__ environ, self.app) Module paste.wsgilib:539 in intercept_output app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:632 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ body = method( trans, **kwargs ) Module galaxy.web.framework:206 in decorator return func( self, trans, *args, **kwargs ) Module galaxy.webapps.galaxy.controllers.admin_toolshed:1199 in prepare_for_install description, repository_clone_url, changeset_revision, ctx_rev, repository_owner, tool_dependencies = repo_info_tuple ValueError: too many values to unpack ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] help configuring tools in Galaxy
What specific errors are you seeing? Some tools have external dependencies that need to be installed. -Dannon On Dec 18, 2012, at 11:29 AM, Tilahun Abebe tilahun.ab...@uni.edu wrote: Hi, We installed a local galaxy a couple of days ago following the basic installation instruction (http://wiki.galaxyproject.org/Admin/Get%20Galaxy). The installation went well, all the tools are listed on the left. I can also upload data to Galaxy. However, an error occurs when I try to run RNA-seq analysis. I am thinking the tools are not configured properly. Can you give me tips to configure NGS tools? An example file with setup syntax will help as well. Thanks. Tilahun ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] cannot Delete Permanently
Check the setting 'allow_user_dataset_purge' in your universe_wsgi.ini -- this is false by default to prevent errors, but changing that should allow users of your instance to purge datasets permanently. -Dannon On Dec 14, 2012, at 12:18 PM, Fenglou Mao feng...@gmail.com wrote: I installed a local galaxy server, did some calculation. Now I want to clean datasets for some space. I tried Delete Permanently on both datasets and histories, none of them worked. The datasets and histories will be marked as deleted, but it will not be really deleted even I am using Delete Permanently. They will not be really removed, and space are not released. Sometimes it will give me a message this feature is not enabled, sometimes it won't. Any one know how to enable Delete Permanently feature? Thanks, Fenglou ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Impossible to add a (new) tool in a workflow (in edition mode)
The paster.log is just the file for the output you've copied below when running daemonized -- if you're running in the foreground it just gets sent there instead of to the file. So, no obvious errors there. Could you check for javascript console errors? And, if you'd like, feel free to send the tool xml to me directly and I'll try to see if I can reproduce the issue. -Dannon On Dec 11, 2012, at 10:47 AM, Sarah Maman sarah.ma...@toulouse.inra.fr wrote: Thanks Danon, But I have no error generated in my error log file (logs of run.sh): [11/Dec/2012:16:39:09 +0200] GET /galaxy/workflow HTTP/1.1 200 - http://URL/galaxy/root; Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0 IP - - [11/Dec/2012:16:39:11 +0200] GET /galaxy/workflow/editor?id=7085c9c37b8829fc HTTP/1.1 200 - http://URL/galaxy/workflow; Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0 IP - - [11/Dec/2012:16:39:12 +0200] GET /galaxy/workflow/get_datatypes?_=1355240375711 HTTP/1.1 200 - http://URL/galaxy/workflow/editor?id=7085c9c37b8829fc; Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0 IP - - [11/Dec/2012:16:39:12 +0200] GET /galaxy/workflow/load_workflow?id=7085c9c37b8829fc_=1355240375756 HTTP/1.1 200 - http://URL/galaxy/workflow/editor?id=7085c9c37b8829fc; Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0 IP - - [11/Dec/2012:16:39:17 +0200] GET /galaxy/workflow/get_new_module_info?type=tooltool_id=BWA_miRNA_=true HTTP/1.1 200 - http://URL/galaxy/workflow/editor?id=7085c9c37b8829fc; Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0 Others actions are listed by not my click on this tool's name. Is paster.log file a default file of Galaxy? Where could I find this file ? Sarah Dannon Baker a écrit : Do you see any errors in Galaxy's paster.log, or in the javascript console? -Dannon On Dec 11, 2012, at 8:40 AM, Sarah Maman sarah.ma...@toulouse.inra.fr wrote: Hello, For one of my tools that I have added to my local instance of Galaxy, I can not add this tool in the editing workflow. When I click on the tool name in the tools windows on the left, it does not appear in the editing workflow. Do you have any idea? Thank you in advance, Sarah Maman ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Installation issue on EC2
On Dec 10, 2012, at 6:17 PM, Fabiano Lucchese fabiano.lucch...@hds.com wrote: I appreciate your effort to help me, but it looks like my AWS account has some serious hidden issues going on. I completely wiped out CloudMan/Galaxy instances from my EC2 environment as well as their volumes, and waited a couple of hours for the instances to disappear from the instances list. After that, I repeated the whole process twice, trying to create a Galaxy cluster with 10 and then 15 Gb of storage space, but the result was equally frustrating with some minor differences. PS: Dannon, one thing that intrigues me is how the web form manages to find out the names of the previous clusters that I tried to instantiate. Where does it get this information from if all the respective instances have been terminated and wiped out? Did you reuse the same cluster name for either of these? This would explain conflicting settings - there's more to a cluster than just the running instances and persistent volumes. That form retrieves those listings from the S3 buckets in your account. Each cluster has its own S3 bucket -- you can identify them with the yourCluster.clusterName files in the listing. These contain lots of information about your galaxy clusters (references to volumes, universe settings, etc.), and if you're attempting to eliminate a cluster completely (you never want to restart it and don't want *anything* preserved), you should delete the buckets referring to them as well. When you ask Cloudman to terminate and remove a cluster permanently, it removes all of this for you, and I'd recommend always using the interface to do this and not doing it manually. Definitely let me know what else I can do to help. If you have a running instance you'd like for me to look at directly I'd be happy to do so -- maybe this is indeed some weird issue that we can work around better. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Cannot see data when selecting Display Data in Browser
Hey Dave, What revision are you running locally? And, just to confirm, in galaxy the file is recognized as a 'tabular' file type? -Dannon On Dec 7, 2012, at 1:25 PM, Dave Walton dave.wal...@jax.org wrote: I'm seeing some weird behavior in our local galaxy instance and am wondering if anyone has seen anything similar. I have a custom tool that I've written that generates a tab-delimted file. When I click on the item name in the history, I can see the contents in the preview window. However, when I click on the eye (Display Data in Browser), it comes up empty. Same thing if I download the file. However, if I go to the database directory to examine the file on disk, the content is exactly what I would expect. The permissions on the file are galaxy user/group and the file has permissions of 644. I don't see any reason why I shouldn't be able to view or download this file through galaxy. Any ideas? Thanks, Dave --- Dave Walton Manager Scientific Computing The Jackson Laboratory Bar Harbor, Maine dave.wal...@jax.org The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] workflow input param issue
This should be resolved in changeset 1ac27213bafb in galaxy-central. Thanks for pointing this out! -Dannon On Dec 3, 2012, at 11:37 AM, Marc Logghe marc.log...@ablynx.com wrote: Hi, The conf of the parameters in question looks like this: param name=project1 type=select label=Project options from_data_table=my_projects filter type=unique_value name=unique column=0/ /options /param param name=target1 type=select label=Target multiple=true options from_data_table=my_projects column name=value index=2/ column name=name index=2/ filter type=param_value ref=project1 name=target column=0/ /options help Select 1 or more targets if you would like to restrict your data set /help /param The idea is that if a project is chosen from the project1 dropdown list, the target1 parameter is updated. This seems to work fine as a standalone tool. As soon as the tool is wrapped into a workflow, this gets broken: if project1 is chosen from the list, a server call is made but the response seems to reset the form fields. Both project1 and target1. Thanks, Marc From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Marc Logghe Sent: Monday, December 03, 2012 11:46 AM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] workflow input param issue Hi, I have a workflow that basically needs a select parameter as input. 2 steps in the workflow actually need the very same input. I don’t think there is a (easy) way to let the user only input the parameter once and that it is passed to both steps. Anyhow, currently - as a workaround and not very user friendly - the user needs to input the very same parameter twice, for each step where that parameter is required. The first issue however, is that as soon as the first parameter is set, the second is set as well apparently (they have the same name, that could explain) which is fine, but not to the chosen one, eg. both are kind of reset to default. No errors or something, simply reset, which makes it impossible to enter the parameter. THIS E-MAIL MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE. If the reader of this E-mail message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately at abl...@ablynx.com. Thank you for your co-operation. NANOBODY and NANOCLONE are registered trademarks of Ablynx N.V. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Can't find new pause / resume functionality in Dec 2012 Galaxy
This isn't (at least at first) Pause/Resume as you might be expecting - where you could manually pause a currently running job and continue it later. What we're doing at least in the first pass is using 'Paused' as an internal state that jobs only go into in two scenarios: 1) User quota is full - all jobs waiting to run will be paused instead of waiting in the queue forever. After clearing their quota, paused jobs can be resumed. 2) A parent job (say, in a workflow) fails, then dependent jobs will be paused instead of having the cascaded error. You'll be able to rerun the failed parent and (if it succeeds) have paused jobs resume, instead of having to re-queue everything all over again. This isn't quite fully implemented yet, but will be finished in the near future. Other than being able to effectively resume workflows and potentially do other things with this framework, the changes should also reduce the load on Galaxy's job running system. -Dannon On Dec 3, 2012, at 7:31 PM, Greg Edwards gedwar...@gmail.com wrote: Hi all, I was keen to try the new Pause / Resume functionality mentuioned in http://wiki.galaxyproject.org/News/2012_12_03_DistributionNewsBrief but can't see it anywhere, in Galaxy or searches. It's not on the Test site at https://test.g2.bx.psu.edu/ or production at https://main.g2.bx.psu.edu/ Can you point me to where I can try it ? Thanks, -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Move files between histories?
Yes. If you click on the little sprocket icon in your history panel and go to Copy Datasets, you'll be able to do this. -Dannon On Nov 20, 2012, at 2:52 PM, Thyssen, Gregory - ARS gregory.thys...@ars.usda.gov wrote: Hello Is it possible to move a file from one history to another? I have generated BAM files in separate histories that now must be merged. How can I do this without downloading and then uploading the files? Thanks, Gregory Thyssen, PhD Molecular Biologist Cotton Fiber Bioscience USDA-ARS-Southern Regional Research Center 1100 Robert E Lee Blvd New Orleans, LA 70124 gregory.thys...@ars.usda.gov 504-286-4280 This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Direct access the fastq files stored in galaxy server local storage
This is actually possible using data libraries. What you'd want to do is upload by filepath (described in the wiki page Brad linked, heading Upload files from filesystem paths) and check the No box under Copy data into Galaxy. -Dannon On Nov 20, 2012, at 4:13 PM, Langhorst, Brad langho...@neb.com wrote: I don't think this is possible. Galaxy is designed to import data files and track their relationships and disk usage internally. You can import lots of files at once though.. see: http://wiki.galaxyproject.org/Admin/Data%20Libraries/Uploading%20Library%20Files Brad On Nov 20, 2012, at 3:49 PM, Fenglou Mao feng...@gmail.com wrote: We have a local galaxy server. I know I can get data by ftp, upload directly from browser, or by a URL. If I have some fastq files in the Galaxy server local storage, is it possible to let Galaxy use it directly? Thanks, Fenglou ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Brad Langhorst langho...@neb.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Can't edit galaxy workflow
This changeset is only in galaxy-central, at the moment, and not galaxy-dist. You have a few options here: Pull directly from -central to the revision indicated (though this means you will be updating your instance to less thoroughly tested galaxy-central code): hg pull -u -r 5013377e0bf7 http://bitbucket.org/galaxy/galaxy-central/ Or, probably preferable to that, you could apply just the individual changeset as a patch (which will merge in silently when you do update -dist). Simply execute the following within your galaxy-dist directory: curl https://bitbucket.org/galaxy/galaxy-central/changeset/5013377e0bf7/raw modulefix.patch hg patch --no-commit modulefix.patch -Dannon On Nov 14, 2012, at 8:04 AM, Sanjarbek Hudaiberdiev hudai...@icgeb.org wrote: Thanks a lot Dannon, for your reply. But hg keeps saying following when I try to pull: abort: unknown revision '5013377e0bf7'! Is there any other way of manually getting it? (Sorry, I'm not an expert on those new SCMs) Sanjar. On 11/13/2012 06:33 PM, Dannon Baker wrote: Sanjar, This is fixed as of 5013377e0bf7. This may not be in the next distribution, but will be in the one after that. Of course, you can manually pull the change from galaxy-central at any time. -Dannon On Nov 13, 2012, at 9:45 AM, Sanjarbek Hudaiberdiev hudai...@icgeb.org wrote: I tried to reply to similar posts, but couldn't figure out how to do it. So posting again: Galaxy giving this error when editing workflow, just after creating workflow from existing history: URL: http://localhost/galaxy/workflow/load_workflow?id=df7a1f0c02a5b08e_=1352817462148 Module weberror.evalexception.middleware:364 in respond app_iter = self.application(environ, detect_start_response) Module paste.debug.prints:98 in __call__ environ, self.app) Module paste.wsgilib:539 in intercept_output app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:632 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ body = method( trans, **kwargs ) Module galaxy.web.framework:73 in decorator return simplejson.dumps( func( self, trans, *args, **kwargs ) ) Module galaxy.webapps.galaxy.controllers.workflow:733 in load_workflow 'tooltip': module.get_tooltip( static_path=url_for( '/static' ) ), Module galaxy.workflow.modules:258 in get_tooltip return self.tool.help.render( static_path=static_path ) AttributeError: 'NoneType' object has no attribute 'render' Could anyone help me to solve this problem? Thanks, Sanjar. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Warning message
It looks like you have conflicting blastxml entries. Edit your datatypes_conf.xml to remove any references to blastxml (the toolshed manages datatypes separately), restart galaxy, and you should be good to go. -Dannon On Nov 13, 2012, at 9:19 AM, rolandomantil...@gmail.com wrote: Sent from my Verizon Wireless BlackBerry From: Rolando Mantilla rolandomantil...@gmail.com Date: Mon, 12 Nov 2012 19:31:20 -0800 To: galaxy-u...@lists.bx.psu.edu Subject: Warning message I'm having issues with the FASTQ_Groomer. What I have done it first I downloaded an SRA file created by an Ion torrent sequencer from the NCBI site. Then used the fastq-dump app from the NCBI site to covert the .sra file to .fastq file. When I uploaded the data into galaxy it recognized it as a fastq(as it should) but when I try to run the FASTQ groomer I get the message and warnings below. I also have already downloaded the the blast_datatypes tool from the tool_shed. I truly don't know what the issue is, any help An error occurred running this job: Groomed 12376 sanger reads into sanger reads. WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'blastxml': 'module' object has no attribute 'BlastXml' WARNING:galaxy.datatypes.registry:Overriding conflicting datatype with extension 'blastxml', using datatype from /mnt/galaxyData/tmp/tmpdP_cZ7. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Can't edit galaxy workflow
Richard, This is fixed as of 5013377e0bf7. This may not be in the next distribution, but will be in the one after that. Of course, you can manually pull the change at any time. -Dannon On Nov 9, 2012, at 4:20 PM, Richard Park rp...@bu.edu wrote: Hi Guys, I updated to the latest galaxy distribution today and I cant seem to edit existing workflows. I've seen others with this error from a Sept 07 distribution (http://dev.list.galaxyproject.org/Can-t-edit-Galaxy-Workflow-ElementInterface-instance-has-no-attribute-render-td4656393.html), but Jeremy Goecks mentioned that this might of been fixed in the Sept 20 distribution. Any ideas on how to fix this? Thanks, Richard Server error URL: http://127.0.0.1:8080/workflow/load_workflow?id=f597429621d6eb2b_=1352495509531 Module weberror.evalexception.middleware:364 in respond app_iter = self.application(environ, detect_start_response) Module paste.debug.prints:98 in __call__ environ, self.app) Module paste.wsgilib:539 in intercept_output app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:632 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ body = method( trans, **kwargs ) Module galaxy.web.framework:73 in decorator return simplejson.dumps( func( self, trans, *args, **kwargs ) ) Module galaxy.webapps.galaxy.controllers.workflow:733 in load_workflow 'tooltip': module.get_tooltip( static_path=url_for( '/static' ) ), Module galaxy.workflow.modules:258 in get_tooltip return self.tool.help.render( static_path=static_path ) AttributeError: 'NoneType' object has no attribute 'render' [] ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Can't edit galaxy workflow
Sanjar, This is fixed as of 5013377e0bf7. This may not be in the next distribution, but will be in the one after that. Of course, you can manually pull the change from galaxy-central at any time. -Dannon On Nov 13, 2012, at 9:45 AM, Sanjarbek Hudaiberdiev hudai...@icgeb.org wrote: I tried to reply to similar posts, but couldn't figure out how to do it. So posting again: Galaxy giving this error when editing workflow, just after creating workflow from existing history: URL: http://localhost/galaxy/workflow/load_workflow?id=df7a1f0c02a5b08e_=1352817462148 Module weberror.evalexception.middleware:364 in respond app_iter = self.application(environ, detect_start_response) Module paste.debug.prints:98 in __call__ environ, self.app) Module paste.wsgilib:539 in intercept_output app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:632 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ body = method( trans, **kwargs ) Module galaxy.web.framework:73 in decorator return simplejson.dumps( func( self, trans, *args, **kwargs ) ) Module galaxy.webapps.galaxy.controllers.workflow:733 in load_workflow 'tooltip': module.get_tooltip( static_path=url_for( '/static' ) ), Module galaxy.workflow.modules:258 in get_tooltip return self.tool.help.render( static_path=static_path ) AttributeError: 'NoneType' object has no attribute 'render' Could anyone help me to solve this problem? Thanks, Sanjar. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] galaxy/cloudman failure handling
On Nov 12, 2012, at 10:23 AM, Jorrit Boekel jorrit.boe...@scilifelab.se wrote: I was therefore looking for fault tolerance mechanisms in the galaxy project, which I seem to remember existed. Somehow I can't find anything about it right now though. I've tested a little bit, and it seems that as soon as one reboots instances or manually kills a job or task, the whole job is deleted and set to error state. I am not that knowledgeable in cluster computing, so I don't really know what handles what here, but this would be an ideal starting point to learn something about SGE and queue handling. Is there any mechanism in place that deals with node failure, network problems, etc? If not, would it be hard to implement? You're correct in that currently jobs will be set to error and need to be automatically rerun by the Galaxy user. There isn't anything in place for automatic retry after spot instance failure, but this is definitely something we plan to implement in the near term - a generalized retry and resume mechanism will be useful for both cloud and local instances. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Problems with latest Galaxy 8167:b0c916349070
Unfortunately the cloud instance upgrade path requires some manual intervention here due to tool migrations. SSH in to your instance, edit /mnt/galaxyTools/galaxy-central/datatypes_conf.xml removing any references to BlastXML. Save the file, restart galaxy, and you should be good to go. -Dannon On Nov 8, 2012, at 10:40 AM, Scooter Willis hwil...@scripps.edu wrote: Running a CloudMan created instance on Amazon. Updated using cloudman from http://bitbucket.org/galaxy/galaxy-central to version 8167:b0c916349070 Tried to run bowtie and get the following error. Searched the message archives an someone else reported the same problem but thought it was related to a new version of cufflinks. Same problem also exists when running abyss. Given the nature of the error message not even sure what the actual path should be. Also for the abyss install from toolshed the toolshed.g2.bx.psu.edu directory doesn't exist or the six directories underneath. However Abyss was/is installed at the following location galaxy@ip-10-110-143-10:/mnt/galaxyTools/tools/abyss/1.2.5/bin$ ls ABYSS abyss-adjtodot abyss-joindist abyss-tofastq DistanceEst MergePaths PathConsensus SimpleGraph abyss2ace abyss-fac ABYSS-P AdjListKAligner Overlap PathOverlap abyss2afg abyss-fixmate abyss-peConsensus MergeContigs ParseAligns PopBubbles Would be great if someone could give some guidance on how to resolve the problems. I understand if I need to do something under the covers but the expectation is that the one click install/updates should not then require major command line work that isn't documented. Thanks Scooter Bowtie Stdout Sequence file aligned. ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ucsc/publicbuilds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ensembl/builds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ncbi/builds.txt' Stderr WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'blastxml': 'module' object has no attribute 'BlastXml' WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.xml:BlastXml' to sniff_order: 'module' object has no attribute 'BlastXml' Using toolshed installed Abyss and get the same error in Stdout. Stdout ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ucsc/publicbuilds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ensembl/builds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ncbi/builds.txt' Stderr COMMAND FAILURE: /mnt/galaxyTools/shed_tools/toolshed.g2.bx.psu.edu/repos/msjeon/abyss_tool/ce99bd1f/abyss_tool/abyss_toolsuite-92636934a189/abyss/abyss_wrapper.sh: line 9: ABYSS: command not found WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'blastxml': 'module' object has no attribute 'BlastXml' WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.xml:BlastXml' to sniff_order: 'module' object has no attribute 'BlastXml' Looking at the util directory based on the expected path of the read build file galaxy@ip-10-110-143-10:/mnt/galaxyTools/galaxy-central/lib/galaxy/util$ ls aliaspickler.py docutils_ext heartbeat.pyjson.pyc odict.py streamball.py aliaspickler.pyc docutils_template.txt inflection.py lrucache.py odict.pyc streamball.pyc backports expressions.py inflection.pyc lrucache.pyc sanitize_html.py template.py bunch.py expressions.pyc__init__.py memdump.py sanitize_html.pyc template.pyc bunch.pyc hash_util.py __init__.pycnone_like.py shed_util.py topsort.py debugging.py hash_util.pyc json.py none_like.pyc shed_util.pyc topsort.pyc ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the
Re: [galaxy-dev] Problems with latest Galaxy 8167:b0c916349070
Neither abyss wrapper in the toolshed installs binaries (you'd see a tool_dependencies.xml in the repository), that's left up to the end user. You might contact the authors of the wrappers via the toolshed to see if they were going to add dependency management and had a newer version, but otherwise you'll have to install binaries on your own. The cloud volumes do have a version of the Abyss binaries preinstalled, but it's certainly possible to add a newer version to your particular instance and persist the changes to the tool volume (see the cloud admin panel). On Nov 8, 2012, at 11:20 AM, Scooter Willis hwil...@scripps.edu wrote: Ok that fixed Bowtie. Any thoughts on expected state of installing Abyss from toolshed. It indicates no requirements so my expectation is that installing from toolshed should install the appropriate abyss package in a location the tool is expecting to find it. In looking at the toolshed looks like two wrappers have been configured. One by msjeon and the other edward-kirton. I did the install for msjeon. Will try the other. Is the abyss install located at /mnt/galaxyTools/tools/abyss/1.2.5/bin from prior testing and should be deleted. Current release version of abyss as of May 30 2012 is Abyss 1.3.4 On 11/8/12 11:00 AM, Dannon Baker dannonba...@me.com wrote: Unfortunately the cloud instance upgrade path requires some manual intervention here due to tool migrations. SSH in to your instance, edit /mnt/galaxyTools/galaxy-central/datatypes_conf.xml removing any references to BlastXML. Save the file, restart galaxy, and you should be good to go. -Dannon On Nov 8, 2012, at 10:40 AM, Scooter Willis hwil...@scripps.edu wrote: Running a CloudMan created instance on Amazon. Updated using cloudman from http://bitbucket.org/galaxy/galaxy-central to version 8167:b0c916349070 Tried to run bowtie and get the following error. Searched the message archives an someone else reported the same problem but thought it was related to a new version of cufflinks. Same problem also exists when running abyss. Given the nature of the error message not even sure what the actual path should be. Also for the abyss install from toolshed the toolshed.g2.bx.psu.edu directory doesn't exist or the six directories underneath. However Abyss was/is installed at the following location galaxy@ip-10-110-143-10:/mnt/galaxyTools/tools/abyss/1.2.5/bin$ ls ABYSS abyss-adjtodot abyss-joindist abyss-tofastq DistanceEst MergePaths PathConsensus SimpleGraph abyss2ace abyss-fac ABYSS-P AdjListKAligner Overlap PathOverlap abyss2afg abyss-fixmate abyss-peConsensus MergeContigs ParseAligns PopBubbles Would be great if someone could give some guidance on how to resolve the problems. I understand if I need to do something under the covers but the expectation is that the one click install/updates should not then require major command line work that isn't documented. Thanks Scooter Bowtie Stdout Sequence file aligned. ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shar ed/ucsc/publicbuilds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shar ed/ensembl/builds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shar ed/ncbi/builds.txt' Stderr WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'blastxml': 'module' object has no attribute 'BlastXml' WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.xml:BlastXml' to sniff_order: 'module' object has no attribute 'BlastXml' Using toolshed installed Abyss and get the same error in Stdout. Stdout ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shar ed/ucsc/publicbuilds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shar ed/ensembl/builds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shar ed/ncbi/builds.txt' Stderr COMMAND FAILURE: /mnt/galaxyTools/shed_tools/toolshed.g2.bx.psu.edu/repos/msjeon/abyss_to ol/ce99bd1f/abyss_tool/abyss_toolsuite-92636934a189/abyss/abyss_wrap per.sh: line 9: ABYSS: command not found WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'blastxml': 'module' object has no attribute 'BlastXml' WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.xml:BlastXml' to sniff_order
Re: [galaxy-dev] Problems with latest Galaxy 8167:b0c916349070
You do not need to restart or add/remove worker nodes, the master's tool and data volumes are shared via NFS. The easiest way to make this work for you in the cloud is probably going to be installing the version of the Abyss binaries you'd like to use manually in /mnt/galaxyTools/abyss/version/ using galaxy's dependency management (http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies) and then, after it has been installed, modify the abyss wrapper to include a requirement type=packageabyss/requirement tag, and finally persist changes to galaxyTools (which will also restart Galaxy, reloading the tools). -Dannon On Nov 8, 2012, at 12:09 PM, Scooter Willis hwil...@scripps.edu wrote: Dannon If I install abyss to the galaxy tools volume and use cloud admin to do a snapshot I assume I then need to remove and then add compute nodes? From the EC2 management console I see two 15GiB EBS volumes from the same snapshot and I have a master and one compute node running. The other approach is that the compute nodes have a NFS mount back into the galaxy_tools volume. Trying to make sure I understand the need to restart the compute nodes after an install of third party tools. From looking at the abyss wrapper it is expecting ABYSS to be found on the path via a .sh file that gets called. Where would I update the path to a particular installed tool such that it will be picked up for each compute node that gets started? I couldn't find any examples of doing a third party install with discussion about how that install would persist to future compute nodes. The missing element for me at the moment is where to make path related changes that will be picked up by compute nodes. I could hard code the path in the .sh file used by the wrapper. Just trying to figure out how to use the software as designed. Thanks Scooter On 11/8/12 11:49 AM, Dannon Baker dannonba...@me.com wrote: Neither abyss wrapper in the toolshed installs binaries (you'd see a tool_dependencies.xml in the repository), that's left up to the end user. You might contact the authors of the wrappers via the toolshed to see if they were going to add dependency management and had a newer version, but otherwise you'll have to install binaries on your own. The cloud volumes do have a version of the Abyss binaries preinstalled, but it's certainly possible to add a newer version to your particular instance and persist the changes to the tool volume (see the cloud admin panel). On Nov 8, 2012, at 11:20 AM, Scooter Willis hwil...@scripps.edu wrote: Ok that fixed Bowtie. Any thoughts on expected state of installing Abyss from toolshed. It indicates no requirements so my expectation is that installing from toolshed should install the appropriate abyss package in a location the tool is expecting to find it. In looking at the toolshed looks like two wrappers have been configured. One by msjeon and the other edward-kirton. I did the install for msjeon. Will try the other. Is the abyss install located at /mnt/galaxyTools/tools/abyss/1.2.5/bin from prior testing and should be deleted. Current release version of abyss as of May 30 2012 is Abyss 1.3.4 On 11/8/12 11:00 AM, Dannon Baker dannonba...@me.com wrote: Unfortunately the cloud instance upgrade path requires some manual intervention here due to tool migrations. SSH in to your instance, edit /mnt/galaxyTools/galaxy-central/datatypes_conf.xml removing any references to BlastXML. Save the file, restart galaxy, and you should be good to go. -Dannon On Nov 8, 2012, at 10:40 AM, Scooter Willis hwil...@scripps.edu wrote: Running a CloudMan created instance on Amazon. Updated using cloudman from http://bitbucket.org/galaxy/galaxy-central to version 8167:b0c916349070 Tried to run bowtie and get the following error. Searched the message archives an someone else reported the same problem but thought it was related to a new version of cufflinks. Same problem also exists when running abyss. Given the nature of the error message not even sure what the actual path should be. Also for the abyss install from toolshed the toolshed.g2.bx.psu.edu directory doesn't exist or the six directories underneath. However Abyss was/is installed at the following location galaxy@ip-10-110-143-10:/mnt/galaxyTools/tools/abyss/1.2.5/bin$ ls ABYSS abyss-adjtodot abyss-joindist abyss-tofastq DistanceEst MergePaths PathConsensus SimpleGraph abyss2ace abyss-fac ABYSS-P AdjListKAligner Overlap PathOverlap abyss2afg abyss-fixmate abyss-peConsensus MergeContigs ParseAligns PopBubbles Would be great if someone could give some guidance on how to resolve the problems. I understand if I need to do something under the covers but the expectation is that the one click install/updates should not then require major command line work that isn't documented. Thanks Scooter
Re: [galaxy-dev] Problems with latest Galaxy 8167:b0c916349070
Glad to help. The first step would be to check and see if those files actually exist (and they should). If those messages are occurring on stdout and not stderr though, they shouldn't be causing failed jobs and something else might be wrong. -Dannon On Nov 8, 2012, at 1:27 PM, Scooter Willis hwil...@scripps.edu wrote: Thanks! That was the explanation I was looking for and the link for how to Do you have any suggestions for fixing the error that are still occurring after running Bowtie? Sequence file aligned. ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ ucsc/publicbuilds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ ensembl/builds.txt' ERROR: Unable to read builds file: [Errno 2] No such file or directory: '/mnt/galaxyTools/galaxy-central/lib/galaxy/util/../../../tool-data/shared/ ncbi/builds.txt' On 11/8/12 12:44 PM, Dannon Baker dannonba...@me.com wrote: You do not need to restart or add/remove worker nodes, the master's tool and data volumes are shared via NFS. The easiest way to make this work for you in the cloud is probably going to be installing the version of the Abyss binaries you'd like to use manually in /mnt/galaxyTools/abyss/version/ using galaxy's dependency management (http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies) and then, after it has been installed, modify the abyss wrapper to include a requirement type=packageabyss/requirement tag, and finally persist changes to galaxyTools (which will also restart Galaxy, reloading the tools). -Dannon On Nov 8, 2012, at 12:09 PM, Scooter Willis hwil...@scripps.edu wrote: Dannon If I install abyss to the galaxy tools volume and use cloud admin to do a snapshot I assume I then need to remove and then add compute nodes? From the EC2 management console I see two 15GiB EBS volumes from the same snapshot and I have a master and one compute node running. The other approach is that the compute nodes have a NFS mount back into the galaxy_tools volume. Trying to make sure I understand the need to restart the compute nodes after an install of third party tools. From looking at the abyss wrapper it is expecting ABYSS to be found on the path via a .sh file that gets called. Where would I update the path to a particular installed tool such that it will be picked up for each compute node that gets started? I couldn't find any examples of doing a third party install with discussion about how that install would persist to future compute nodes. The missing element for me at the moment is where to make path related changes that will be picked up by compute nodes. I could hard code the path in the .sh file used by the wrapper. Just trying to figure out how to use the software as designed. Thanks Scooter On 11/8/12 11:49 AM, Dannon Baker dannonba...@me.com wrote: Neither abyss wrapper in the toolshed installs binaries (you'd see a tool_dependencies.xml in the repository), that's left up to the end user. You might contact the authors of the wrappers via the toolshed to see if they were going to add dependency management and had a newer version, but otherwise you'll have to install binaries on your own. The cloud volumes do have a version of the Abyss binaries preinstalled, but it's certainly possible to add a newer version to your particular instance and persist the changes to the tool volume (see the cloud admin panel). On Nov 8, 2012, at 11:20 AM, Scooter Willis hwil...@scripps.edu wrote: Ok that fixed Bowtie. Any thoughts on expected state of installing Abyss from toolshed. It indicates no requirements so my expectation is that installing from toolshed should install the appropriate abyss package in a location the tool is expecting to find it. In looking at the toolshed looks like two wrappers have been configured. One by msjeon and the other edward-kirton. I did the install for msjeon. Will try the other. Is the abyss install located at /mnt/galaxyTools/tools/abyss/1.2.5/bin from prior testing and should be deleted. Current release version of abyss as of May 30 2012 is Abyss 1.3.4 On 11/8/12 11:00 AM, Dannon Baker dannonba...@me.com wrote: Unfortunately the cloud instance upgrade path requires some manual intervention here due to tool migrations. SSH in to your instance, edit /mnt/galaxyTools/galaxy-central/datatypes_conf.xml removing any references to BlastXML. Save the file, restart galaxy, and you should be good to go. -Dannon On Nov 8, 2012, at 10:40 AM, Scooter Willis hwil...@scripps.edu wrote: Running a CloudMan created instance on Amazon. Updated using cloudman from http://bitbucket.org/galaxy/galaxy-central to version 8167:b0c916349070 Tried to run bowtie
Re: [galaxy-dev] Problems with latest Galaxy 8167:b0c916349070
,droYak1,droYak2,dp3,dp2,dm1,canFam1,danRer5,canFam2,danRer3,danRer2,ornAna1,ci2,ci1,tetNig1,bosTau1,bosTau3,bosTau2,equCab1,oryLat1,droAna2,droEre1,ponAbe2,rheMac2,sacCer1,droPer1,droSim1,monDom1,cb1,dm2,droSec1,strPur1,droVir2,droVir1,strPur2,sc1,xenTro1,droGri1,xenTro2,cb3,gasAcu1,caePb1,anoGam1,fr2,fr1,hg15,hg16,hg17,hg18,felCat3,apiMel2,monDom4,apiMel1,ce2,hg19 #Harvested from http://archaea.ucsc.edu/cgi-bin/das/dsn archaea http://archaea.ucsc.edu/cgi-bin/hgTracks? alkaEhrl_MLHE_1,shewW318,idioLoih_L2TR,sulSol1,erwiCaro_ATROSEPTICA,symbTher_IAM14863,moorTher_ATCC39073,therFusc_YX,methHung1,bradJapo,therElon,shewPutrCN32,pediPent_ATCC25745,mariMari_MCS10,nanEqu1,baciSubt,chlaTrac,magnMagn_AMB_1,chroViol,ralsSola,acidCryp_JF_5,erytLito_HTCC2594,desuVulg_HILDENBOROUG,pyrAer1,sulfToko1,shewANA3,paraSp_UWE25,geobKaus_HTA426,rhizEtli_CFN_42,uncuMeth_RCI,candBloc_FLORIDANUS,deinRadi,yersPest_CO92,saccEryt_NRRL_2338,rhodRHA1,candCars_RUDDII,burkMall_ATCC23344,eschColi_O157H7,burk383,psycIngr_37,rhodSpha_2_4_1,wolbEndo_OF_DROSOPHIL,burkViet_G4,propAcne_KPA171202,enteFaec_V583,campJeju_81_176,acidJS42,heliPylo_26695,pseuHalo_TAC125,chroSale_DSM3043,methVann1,archFulg1,neisMeni_Z2491_1,fusoNucl,vermEise_EF01_2,anabVari_ATCC29413,tropWhip_TW08_27,heliHepa,acinSp_ADP1,anapMarg_ST_MARIES,natrPhar1,haheChej_KCTC_2396,therPetr_RKU_1,neisGono_FA1090_1,colwPsyc_34H,desuPsyc_LSV54,hyphNept_ATCC15444,vibrChol1,deinGeot_DSM11300,strePyog_M1_GAS,franCcI3,salmTyph,metaSedu,lactSali_UCC118,trepPall,neisMeni_MC58_1,syntWolf_GOETTINGEN,flavJohn_UW101,methBoon1,haemSomn_129PT,shewLoihPV4,igniHosp1,haemInfl_KW20,haloHalo_SL1,ferrAcid1,sphiAlas_RB2256,candPela_UBIQUE_HTCC1,caldSacc_DSM8903,aerPer1,lactPlan,carbHydr_Z_2901,therTher_HB8,vibrVuln_YJ016_1,rhodPalu_CGA009,acidCell_11B,siliPome_DSS_3,therVolc1,haloWals1,rubrXyla_DSM9941,shewAmaz,nocaJS61,vibrVuln_CMCP6_1,sinoMeli,ureaUrea,baciHalo,bartHens_HOUSTON_1,nitrWino_NB_255,hypeButy1,methBurt2,polaJS66,mesoLoti,methMari_C7,caulCres,neisMeni_FAM18_1,acidBact_ELLIN345,caldMaqu1,salmEnte_PARATYPI_ATC,glucOxyd_621H,cytoHutc_ATCC33406,nitrEuro,therMari,coxiBurn,woliSucc,heliPylo_HPAG1,mesoFlor_L1,pyrHor1,methAeol1,procMari_CCMP1375,pyroArse1,oenoOeni_PSU_1,alcaBork_SK2,wiggBrev,actiPleu_L20,lactLact,methJann1,paraDeni_PD1222,borrBurg,pyroIsla1,orieTsut_BORYONG,shewMR4,methKand1,methCaps_BATH,onioYell_PHYTOPLASMA,bordBron,cenaSymb1,burkCeno_HI2424,franTula_TULARENSIS,pyrFur2,mariAqua_VT8,heliPylo_J99,psycArct_273_4,vibrChol_MO10_1,vibrPara1,rickBell_RML369_C,metAce1,buchSp,ehrlRumi_WELGEVONDEN,methLabrZ_1,chlaPneu_CWL029,thioCrun_XCL_2,pyroCali1,chloTepi_TLS,stapAure_MU50,novoArom_DSM12444,magnMC1,zymoMobi_ZM4,salmTyph_TY2,chloChlo_CAD3,azoaSp_EBN1,therTher_HB27,bifiLong,picrTorr1,listInno,bdelBact,gramFors_KT0803,sulfAcid1,geobTher_NG80_2,peloCarb,ralsEutr_JMP134,mannSucc_MBEL55E,syneSp_WH8102,methTherPT1,clavMich_NCPPB_382,therAcid1,syntAcid_SB,porpGing_W83,therNeut0,leifXyli_XYLI_CTCB0,shewFrig,photProf_SS9,thioDeni_ATCC25259,methMaze1,desuRedu_MI_1,burkThai_E264,campFetu_82_40,blocFlor,jannCCS1,nitrMult_ATCC25196,streCoel,soliUsit_ELLIN6076,pastMult,saliRube_DSM13855,methTher1,nostSp,shigFlex_2A,saccDegr_2_40,oceaIhey,dehaEthe_195,rhodRubr_ATCC11170,arthFB24,shewMR7,pireSp,anaeDeha_2CP_C,haloVolc1,dichNodo_VCS1703A,tricEryt_IMS101,mycoGeni,thioDeni_ATCC33889,methSmit1,geobUran_RF4,shewDeni,halMar1,desuHafn_Y51,methStad1,granBeth_CGDNIH1,therPend1,legiPneu_PHILADELPHIA,vibrChol_O395_1,nitrOcea_ATCC19707,campJeju_RM1221,methPetr_PM1,heliAcin_SHEEBA,eschColi_APEC_O1,peloTher_SI,haloHalo1,syntFuma_MPOB,xyleFast,gloeViol,leucMese_ATCC8293,bactThet_VPI_5482,xantCamp,sodaGlos_MORSITANS,geobSulf,roseDeni_OCH_114,coryEffi_YS_314,brucMeli,mycoTube_H37RV,vibrFisc_ES114_1,pyrAby1,burkXeno_LB400,polyQLWP,stapMari1,peloLute_DSM273,burkCeno_AU_1054,shewBalt,nocaFarc_IFM10152,ente638,mculMari1,saliTrop_CNB_440,neorSenn_MIYAYAMA,aquiAeol,dechArom_RCB,myxoXant_DK_1622,burkPseu_1106A,burkCepa_AMMD,methMari_C5_1,azorCaul2,methFlag_KT,leptInte,eschColi_K12,synePCC6,baumCica_HOMALODISCA,methBark1,pseuAeru,geobMeta_GS15,eschColi_CFT073,photLumi,metMar1,hermArse,campJeju,therKoda1,aeroHydr_ATCC7966,baciAnth_AMES,shewOnei,therTeng,lawsIntr_PHE_MN1_00 #Harvested from http://genome-test.cse.ucsc.edu/cgi-bin/das/dsn On 11/8/12 1:55 PM, Dannon Baker dannonba...@me.com wrote: That path should resolve to /mnt/galaxyTools/galaxy-central/tool-data/shared/ucsc/ (or ensembl, ncbi, etc) Can you tell me the output of 'hg tip' 'hg st' and 'hg diff' from the root /mnt/galaxyTools/galaxy-central/ ? -Dannon On Nov 8, 2012, at 1:41 PM, Scooter Willis hwil...@scripps.edu wrote: Errors are being output to stdout The files don't exist and complicated by the /../../ where I don't know which subdirectory to follow. Attached screen shot of job failing giving the ERROR message in the web gui. I also get the same errors when attempting to run Abys which doesn't run because it isn't installed
Re: [galaxy-dev] missing BWA tools on AWS AMI
Hi Quang, This is indeed temporary. You can get things working in the interim by adding BWA via the toolshed. -Dannon On Nov 7, 2012, at 2:12 PM, Quang Trinh quang.tr...@gmail.com wrote: Hi dev, I launched an instance of Galaxy on Amazon ( AMI ami-da58aab3 ) this morning and noticed bwa is no longer available. Is this temporary or if we want to use bwa, we have to add the bwa ourselves? Thanks, Q ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Dataset upload fail
On Nov 6, 2012, at 11:25 PM, Vladimir Yamshchikov yaxi...@gmail.com wrote: Error attempting to display contents of library (SC datasets): (OperationalError) no such column: True u'SELECT dataset_permissions.id AS dataset_permissions_id, dataset_permissions.create_time AS dataset_permissions_create_time, dataset_permissions.update_time AS dataset_permissions_update_time, dataset_permissions.action AS dataset_permissions_action, dataset_permissions.dataset_id AS dataset_permissions_dataset_id, dataset_permissions.role_id AS dataset_permissions_role_id XnFROM dataset_permissions XnWHERE True AND dataset_permissions.action = ?' ['access']. This issue has been fixed as of changeset 8116:ecd131b136d0 and will be available in the next distribution release. Why reporting is disabled from within this instance of Galaxy? KInda silly - there is suggestion to report the error to the developers' team and after long typing it says Disabled. Having encountered that joke before I wisely copied the message content before sending it over (that is trashing in fact), so I can send using outside email. I'm not sure exactly what is causing your IOError (hopefully someone else has ideas and can chime in), but error reporting requires some configuration on the part of the local admin -- see your universe_wsgi.ini, specifically the Mail and notification section. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Think I found a bug
Hi Juan, Thanks for reporting this, it is indeed a bug. The fix below isn't quite correct (if there is an external metadata job, we do actually want to terminate it) but I'll take care of it. For reporting bugs in the future, certainly feel free to message this list or you can also file an issue using http://galaxyproject.org/trello Thanks! -Dannon On Nov 6, 2012, at 11:54 AM, Juan González-Vallinas juanramongvalli...@gmail.com wrote: Hi galaxy devs, We are in the final stages of deploying a new Galaxy instance. We downloaded the galaxy-dist version a month ago aprox. We realized that, when launching a job and deleting it in the middle of the execution (hitting the Delete button in the history bar) will output the following error in the logger: galaxy.jobs.handler DEBUG 2012-11-05 15:56:33,537 Stopping job 45: galaxy.jobs.handler DEBUG 2012-11-05 15:56:33,537 stopping job 45 in local runner galaxy.jobs.handler ERROR 2012-11-05 15:56:33,538 Exception in monitor_step Traceback (most recent call last): File /usr/local/www/galaxy-dist/lib/galaxy/jobs/handler.py, line 298, in monitor self.monitor_step() File /usr/local/www/galaxy-dist/lib/galaxy/jobs/handler.py, line 341, in monitor_step self.dispatcher.stop( job ) File /usr/local/www/galaxy-dist/lib/galaxy/jobs/handler.py, line 438, in stop self.job_runners[runner_name].stop_job( job ) File /usr/local/www/galaxy-dist/lib/galaxy/jobs/runners/local.py, line 180, in stop_job if job.get_external_output_metadata(): AttributeError: 'Job' object has no attribute 'get_external_output_metadata' By copying the definition of get_external_output_data from the Task class into the Job class, we fixed the problem and now the jobs are terminated successfully. This is the code I added: class Job( object ): ... def get_external_output_metadata( self ): return None ... In this file: https://bitbucket.org/galaxy/galaxy-central/src/9d6a61f060d359a0289b3163990f6a8ba122d253/lib/galaxy/__init__.py?at=default Do you think this fix is correct, or is it a potential problem? Also, sorry I sent you through this channel, I couldn't find the way of reporting this on BitBucket. Regards, -- Juan González-Vallinas PhD Student Regulatory Genomics Group Research Unit in Biomedical Informatics (GRIB) Universitat Pompeu Fabra (UPF) Barcelona Spain (Currently @ UPenn) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Stop/Start/Reboot/Terminate
Galaxy Cloudman does not support Stop/Start through the AWS interface, this is known to cause problems and should be avoided. The persistence design allows for complete termination and restart -- the issue with your startup zone can be worked around currently by launching through the AWS console (following the instructions at usegalaxy.org/cloud) instead of cloudlaunch. That said, I'm currently updating cloudlaunch to support zone detection and launch, to avoid any issues with that moving forward, but this won't be available on main until monday, most likely. That out of the way, the issue you're experiencing right now is likely caused by an error with cm_boot.py causing duplicated nginx max_client_body_size directives. This problem has been fixed on the back end and won't happen with any new clusters going forward, but for existing clusters experiencing the problem you can probably fix it in in a few short steps: ssh in to the instance edit /opt/galaxy/pkg/nginx/conf/nginx.conf, removing any redundant max_client_body_size directives. There should be exactly one in the file. download a copy of the newest https://s3.amazonaws.com/cloudman/cm_boot.py (save it to your desktop or something) In the AWS console, go to S3 and find your cluster's bucket (it'll have a file called your cluster name.clusterName, for easy identification. Now upload the new cm_boot.py you just saved, replacing the one currently in the bucket. Once the modified file is in your bucket, simply restart the instance via the AWS console and everything *should* come up fine. Sorry for any inconvenience! -Dannon On Nov 1, 2012, at 10:49 AM, Scooter Willis hwil...@scripps.edu wrote: Last night I used the Amazon console to stop my working instance. Today started up the instance using amazon console. Waited appropriate time, using new assigned public ip address and no response. Also did a reboot and no response. I can ssh to the instance but do not know what to check for errors. Should you be able to stop/start and instance in amazon console and have it work correctly? Trying to avoid the power down option using the galaxy web interface since I had the problem with new instances being started in a different availability zone where the EBS volume was located. Looks like I will be leaving the master instance running and contributing to amazon profit margins! ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Amazon
For this instance, you'll need to restart using the old method for launching via the console, specifying the zone 1b. Detection of the zone volumes are in for existing clusters and specifying those for launch is on the short list of things coming up for cloud launch. On Oct 31, 2012, at 10:50 AM, Scooter Willis hwil...@scripps.edu wrote: Tried it again and same error message. The volume was originally created in us-east-1b and newly created instances are being started in us-east-1a. Shouldn't the availability zone be set to us-east-1b when the instance is requested or that info stored in the properties file in the S3 bucket? Any suggestions? From: Scooter Willis hwil...@scripps.edu Date: Wednesday, October 31, 2012 10:32 AM To: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu Subject: Amazon Started up a cluster on Amazon using the Launch a Galaxy Cloud Instance and got the following message. Since I don't have any control over where the instances are run not sure how I can control this. The last 4 or 5 times I have started up an existing instance has worked with no problem. Messages (CRITICAL messages cannot be dismissed.) [CRITICAL] Volume 'vol-f882ca85' is located in the wrong availability zone for this instance. You MUST terminate this instance and start a new one in zone 'us-east-1a'. (2012-10-31 14:25:20) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Column alignment in tabular display
Hey Peter,
I must have missed this the first time through, I like the change below and can
apply it.
-Dannon
On Oct 30, 2012, at 8:26 AM, Peter Cock p.j.a.c...@googlemail.com wrote:
Hi all,
The issue I raised below about column alignment in the display of
tabular files still affects the current latest release - would you like me
to turn my little proposed patch into a bitbucket pull request?
Thanks,
Peter
-- Forwarded message --
From: Peter Cock p.j.a.c...@googlemail.com
Date: Mon, Jun 18, 2012 at 4:25 PM
Subject: Column alignment in tabular display
To: galaxy-dev@lists.bx.psu.edu
Hello all,
I'm curious about the logic used for column alignment when displaying
tabular files (which for me was the headline feature of the last
Galaxy Release - I'm very pleased to have this added at last).
https://bitbucket.org/galaxy/galaxy-central/issue/553
Quoting from templates/dataset/tabular_chunked.mako
function renderCell(cell_contents, index, colspan){
if (colspan !== undefined){
return $('td').attr('colspan',
colspan).addClass('stringalign').text(cell_contents);
}
else if (DATASET_TYPES[index] == 'str'){
/* Left align all str columns, right align the rest */
return $('td').addClass('stringalign').text(cell_contents);;
}
else{
return $('td').text(cell_contents);
}
}
It seems to me that the logic should be right align all numeric
columns, left align the rest. At very least, columns of type 'list'
should also be left aligned. Otherwise you can easily have a situation
where on clicking the eye you get an apparently empty dataset
(unless you scroll right or down).
This can easily happen with FASTA to tabular (default settings), where
the descriptions have commas in them (common with annotation) thus the
first column is considered to be a list (and the second column type
str for the sequence). If some of the later sequences in the file have
very long descriptions, right aligning the description column makes
the dataset appear empty.
I can provide a screen shot if it would help to explain, but here is
my proposed patch.
Regards,
Peter
--
$ hg diff templates/dataset/tabular_chunked.mako
diff -r 8657ada96fca templates/dataset/tabular_chunked.mako
--- a/templates/dataset/tabular_chunked.makoFri Jun 15 16:05:48 2012 +0100
+++ b/templates/dataset/tabular_chunked.makoMon Jun 18 16:23:44 2012 +0100
@@ -17,8 +17,8 @@
if (colspan !== undefined){
return $('td').attr('colspan',
colspan).addClass('stringalign').text(cell_contents);
}
-else if (DATASET_TYPES[index] == 'str'){
-/* Left align all str columns, right align the rest */
+else if (DATASET_TYPES[index] == 'str' ||
DATASET_TYPES[index] == 'list'){
+/* Left align all str and list columns, right align the rest
*/
return $('td').addClass('stringalign').text(cell_contents);;
}
else{
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] MrBayes wrapper - set parameters through Galaxy?
On Oct 29, 2012, at 4:40 PM, Oleksandr Moskalenko o...@hpc.ufl.edu wrote: This is an interesting project. I'm glad to see more people working on phylogenetics related wrappers and workflows. I wonder if we could get a Phylogenetics category added to the main Galaxy Toolshed, so we could put all the relevant wrappers in there to make them easier to find considering the apparent duplication of efforts. Good idea, the main toolshed now has a Phylogenetics category. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Enhancement: selecting input
On Oct 24, 2012, at 3:36 AM, Joachim Jacob joachim.ja...@vib.be wrote: PS: I would have posted on Trello, but I am not allowed to do so. I understand that this is the way to propose enhancements. For adding cards, anyone can use the form at http://galaxyproject.org/trello. A new card will be created on the Inbox list of the trello board with the title/description entered in the form. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] How to report issues (link on the website requires Galaxy team membership)
On Oct 22, 2012, at 3:52 AM, David van Enckevort david.van.enckev...@nbic.nl wrote: On the main galaxy page there is a link 'Report Issue' to the bitbucket issue tracker, however since a few weeks it is not possible to view or report issues anymore since it requires membership of the project. The wiki needs to be updated, thanks for pointing this out. It actually has conflicting instructions, both saying to, and not to, report issues on bitbucket. This raises two questions: 1. How should issues be reported? 2. How can we see which issues have been reported by others, and most importantly, whether they have been addressed by the Galaxy Team. I think it is very important for any group that maintains a Galaxy instance to have insight in the known issues and fixes, and I hope this information is still available somewhere. We now have a public facing Trello board accessible from galaxyproject.org/trello (direct link to the board is https://trello.com/board/galaxy-development-inbox/50686d0302dfa79d13d90c45). To add new cards simply fill out the form at galaxyproject.org/trello (use markdown for formatting, if desired), but you can comment on and vote for issues directly on the board. All existing bitbucket issues were imported and, though they still need to be sorted out, comments/voting on them are more than welcome. Our hope is that this issue tracker will be a more effective community interface than the bitbucket issues were, with the voting features and simpler visibility. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Failed to run display.py
On Oct 21, 2012, at 6:11 AM, Tom Hait sth...@gmail.com wrote: so I tried to change the URL to: http://localhost:8080, http://127.0.0.1:8080 ... It also didn't work. Any Ideas about could go wrong? First thing I'd check would be to verify that your galaxy server is currently running, and on localhost:8080. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Renaming in Workflows
The renaming input selector uses # instead of $ to allow combinations with
workflow parameters.
So, in your case, #{input} should work. There are also options (basename,
upper, lower) that you can use to format the text. So, #{input | upper} would
use the input name but ensure that it was all uppercase.
-Dannon
On Sep 26, 2012, at 8:23 AM, Kohler Manuel manuel.koh...@bsse.ethz.ch wrote:
Hi,
I have workflows where I would like to rename the item at each step. The
reason for this to have a proper naming.
What I did is to use the renaming option in the workflow editor. I
specified the 'New output name:' to ${input}
This does not work.
Is there any variable which works or is there a easy way in doing this?
Any help or hint is appreciated.
Cheers
Manuel
--
Manuel Kohler
Center for Information Sciences and Databases (C-ISD)
Department of Biosystems Science Engineering (D-BSSE)
ETH Zurich, Maulbeerstrasse (1078, 1.02), CH-4058 Basel, +41 61 387 3132
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Renaming in Workflows
Ahh, I see what's probably going on. The matching uses the tool's actual input
name (which isn't exposed, and isn't always 'input') as opposed to the label
that you'll see in the tool form and elsewhere. Which tool is this for? If
you look at the tool xml file you should see the real input name and can try
that.
On Sep 26, 2012, at 10:32 AM, Kohler Manuel manuel.koh...@bsse.ethz.ch wrote:
Hi Dannon,
thanks for the fast reply.
I changed it to #{input}. But all the data names are still empty. Is there
anything I am missing?
BTW: I am on r7720
Cheers
Manuel
On 26. September 2012 4:10 PM, Dannon Baker dannonba...@me.com wrote:
The renaming input selector uses # instead of $ to allow combinations
with workflow parameters.
So, in your case, #{input} should work. There are also options
(basename, upper, lower) that you can use to format the text. So,
#{input | upper} would use the input name but ensure that it was all
uppercase.
-Dannon
On Sep 26, 2012, at 8:23 AM, Kohler Manuel manuel.koh...@bsse.ethz.ch
wrote:
Hi,
I have workflows where I would like to rename the item at each step. The
reason for this to have a proper naming.
What I did is to use the renaming option in the workflow editor. I
specified the 'New output name:' to ${input}
This does not work.
Is there any variable which works or is there a easy way in doing this?
Any help or hint is appreciated.
Cheers
Manuel
--
Manuel Kohler
Center for Information Sciences and Databases (C-ISD)
Department of Biosystems Science Engineering (D-BSSE)
ETH Zurich, Maulbeerstrasse (1078, 1.02), CH-4058 Basel, +41 61 387 3132
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Screen shot 2012-09-26 at 4.31.34 PM.png
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Instantiate Galaxy on AWS
Kourosh, My first guess would be a misconfiguration of the security groups or the like -- are you launching the instance using the regular AWS console? If so, it might be worth it to try using Galaxy's Cloud Launch at https://main.g2.bx.psu.edu/cloudlaunch which will help you format any userData correctly and will automatically configure security groups and the like. -Dannon On Sep 25, 2012, at 11:06 AM, Ravvaz, Kourosh rav...@gmail.com wrote: Dear Sir/Madam, I just started to instantiate a Galaxy instance on the AWS. I have followed your instructions on the following page: http://wiki.g2.bx.psu.edu/CloudMan http://wiki.g2.bx.psu.edu/CloudMan/AWS/GettingStarted At the end of step 2, I faced a problem which resulted in the following error: The Galaxy Cloud Console is not running. It may take a few moments to start after your EC2 instance has become available, or you may not have provided the correct user data when configuring and launching the instance from the AWS Management Console. Please refer to the documentation for more information. If you did enter the correct user data, you may be able to reboot this EC2 instance to solve the problem. So, I rebooted the instance and then since the problem was not resolved I terminated the instance and created another instance. I did so a few times. However, it did not help. I have reviewed my AWS credentials and also the master instance (AMI: ami-da58aab3, Name: 861460482541/galaxy-cloudman-2011-03-22). Every thing looks fine. Would you please help me fix this? Regards, Kourosh ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] The user creation and login script can be injected with executable javascript in Galaxy
Hanfei, I'd be happy to take a look at the report and share it with the rest of the team if you'd like to send it directly to me. Regarding SSL, this is definitely something that you can set up for your own instance, see the documentation for configuring proxies on the wiki http://wiki.g2.bx.psu.edu/Admin/Config/Performance/nginx%20Proxy. Thanks! -Dannon On Sep 24, 2012, at 12:01 AM, Hanfei Sun ad9...@gmail.com wrote: Hello Galaxy-team, A galaxy instance is being hold on our server. But last week, an expert in security makes some tests on our server. He warned us that the user creation and login script can be injected with executable javascript in Galaxy, which may make our server vulnerable. He gives us a report of 3 pages (other issues including Non-SSL Password and cookie of Galaxy). We don't know whether it's serious and whether we need to fix these issues immediately. Is Galaxy going to update for issues? Or we need to modify them ourselves? Any suggestion is appreciated. Thanks! -- Hanfei Sun Sent with Sparrow ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Blastxml error on Galaxy Cloudman
The cloud admin interface's automatic update mechanism doesn't currently support toolshed updates due to the interaction required. At this point, if you're updating a cloud instance you'll need to do it manually. SSH in, switch to the galaxy user, navigate to /mnt/galaxyTools/galaxy-central, and perform an hg pull and update as usual, following all of the instructions printed to the command line (including adding tool_config_file, etc.) We'll definitely streamline this going forward, but for now if you want to update you have to do it manually. Dannon On Aug 29, 2012, at 7:01 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Mohammad, Don't forget to CC the mailing list ;) It sounds like something has gone wrong with the update... Peter On Wednesday, August 29, 2012, Mohammad Heydarian wrote: Hi Peter, Thanks for your response. I am not using any of the Blast tools, I have only tried using the filter and sort and text manipulation tools and gotten this error. Could this be fixed by re-updating my instance of Galaxy Cloudman? Or some similar fix? I am not able to install tools to my instance of Galaxy Cloudman, the error I get is: The tool_config_file setting in universe_wsgi.ini must include at least one shed tool configuration file name with a toolbox tag that includes a tool_path attribute value which is a directory relative to the Galaxy installation directory in order to automatically install tools from a Galaxy tool shed (e.g., the file name shed_tool_conf.xml whose toolbox tag is toolbox tool_path=../shed_tools). Cheers, Mo Heydarian PhD candidate The Johns Hopkins School of Medicine Department of Biological Chemistry 725 Wolfe Street 414 Hunterian Baltimore, MD 21205 On Wed, Aug 29, 2012 at 3:24 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi The BLAST+ tools and the definition of the blastxml format were moved to the tool shed. You should have seen a migration message during the update... If you are not using these tools, perhaps you have a legacy reference to the blastxml file which should be removed? If you are using BLAST+ they and the blastxml format need to be installed from the ToolShed, Peter On Wednesday, August 29, 2012, Mohammad Heydarian wrote: Hello, I recently updated my instance of Galaxy Cloudman (using Amazon AWS) and I have since not been able to run any jobs. I get the following error: WARNING:galaxy.datatypes.registry:Error loading datatype with extension 'blastxml': 'module' object has no attribute 'BlastXml' WARNING:galaxy.datatypes.registry:Error appending sniffer for datatype 'galaxy.datatypes.xml:BlastXml' to sniff_order: 'module' object has no attribute 'BlastXml' Any help would greatly be appreciated. Cheers, Mo Heydarian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] DataTypes
We have a wiki page describing the addition of both simple and complex datatypes, here's the link: http://wiki.g2.bx.psu.edu/Admin/Datatypes/Adding%20Datatypes Let me know if this isn't sufficient, and I can try to help. -Dannon On Aug 28, 2012, at 3:56 PM, Alfredo Guilherme Silva Souza alfredo.bioinformat...@gmail.com wrote: Good Afternoon. I need to create a new data type in the galaxy, because I need to import a file in the format. Pdb. Someone already had to create a new kind of data? Can you help? Thank you. -- Alfredo Guilherme ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Update Galaxy instance with Mercurial error
Hi Makis, This should work to get you an update-able galaxy instance going forward, without having to manually migrate anything. Please note that this method will clobber any existing changes to code, so if you've modified any of the core galaxy you'll need to patch the new files. All of your data and database settings will remain intact (but still do steps 1 and 2). 1) Back up everything, just in case 2) Make sure you did step #1 2) cd into the root galaxy directory and `hg init` 3) create a file, galaxy_root/.hg/hgrc with the following two lines: [paths] default = https://bitbucket.org/galaxy/galaxy-dist 4) And again from the galaxy root `hg pull` 5) Finally `hg update -C` And you should be good to go. -Dannon On Jul 31, 2012, at 6:05 AM, Makis Ladoukakis makis4e...@hotmail.com wrote: Dear Galaxy users, I am running a Galaxy instance in my institution's server and I want to keep it up to date with the latest release. Nevertheless when i try hg incomingi get the following error: abort: There is no Mercurial repository here (.hg not found)! Most probably the administrator of the server has downloaded the Galaxy-dist directory without the use of Mercurial (I was suspicious of that all along as the directory of the Galaxy installation is named galaxy_dist and not galaxy-dist). Is there another way to update my instance? Thank you, Makis Ladoukakis ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error in loading tools.
The problem here is that the public toolshed interface has been updated more recently than the galaxy install on that cloud instance. You should be good to go if you update galaxy (possible through the admin interface at your_ec2_instance/cloud/admin/) to the latest version. -Dannon On Jul 31, 2012, at 12:11 PM, Mohammad Heydarian mheydar...@gmail.com wrote: Hi Galaxy Community! Hope all is well. I've been trying to install some tools in my Galaxy Cloudman instance (using image: ami-da58aab3 ) from the main toolshed. When I try to install to local Galaxy I get the error: Not Found The resource could not be found. No action for /admin_toolshed/prepare_for_install I have tried to load tools on a couple of different Cloudman instances and tried to load a variety of tools and end up with the same error. Any help would be fantastic. Cheers, Mo Heydarian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] can not login to workflows after postgresql setup
For mail configuration, see the Mail and Notification section in your universe_wsgi.ini -- all you need to do is specify an SMTP server that Galaxy can use. Regarding the invalid password, keep in mind that this would be your previously registered password (from when you were using the sqlite database originally), and not the one you used when you reregistered with a postgres database. -Dannon On Jul 6, 2012, at 8:26 AM, j.w.f.van_der_heij...@lumc.nl wrote: Hi, Any ideas yet on how to configure my mail in galaxy? Kind regards Jaap From: galaxy-dev-boun...@lists.bx.psu.edu [galaxy-dev-boun...@lists.bx.psu.edu] on behalf of j.w.f.van_der_heij...@lumc.nl [j.w.f.van_der_heij...@lumc.nl] Sent: Friday, July 06, 2012 11:13 AM To: jeremy.goe...@emory.edu Cc: bernd.ja...@pasteur.fr; galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] can not login to workflows after postgresql setup Hi all, I tried to reconnect with the SQLite database but this time when I try to log in to use workflows it says that my password is invalid. When I try to reset it with the forgotten password option I get this message again: Mail is not configured for this Galaxy instance. Please contact an administrator. What can I do? Jaap From: Jeremy Goecks [jeremy.goe...@emory.edu] Sent: Wednesday, July 04, 2012 11:38 PM To: Heijden, J.W.F. van der (HG) Cc: bernd.ja...@pasteur.fr; galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] can not login to workflows after postgresql setup I did manage to log in now after the postgres installation by re-registering. However my workflow is gone. Is there a file stored in galaxy by default which I can reupload to retrieve my workflow?. Moving to a new database will make all your previous data inaccessible as it's no longer in the database that Galaxy is using, so this behavior is expected. You could: (a) reconnect Galaxy to your SQLite database and export (download) your workflow and then (b) change Galaxy to use your PostGres database and import (upload) your workflow. Also it seems that I am no longer able to upload files. It's only a 300kb test file but it seems to be uploading forever. The database used should not affect uploading. Are you still having problems? Best, J. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] History items get REMOVED when using the API to run workflows
I see the bug you're running into. As a temporary solution, executing workflows via the API with 'no_add_to_history' in the payload should work as expected. I'll have a permanent fix out shortly. -Dannon On Jul 5, 2012, at 5:12 PM, Thon Deboer wrote: Hi, I am continueing to struggle with the API I have created a workflow execution engine that submits a workflow using the examples in the API (scripts/api). When I run the WF on data in the LIBRARIES everything works fine. But when I choose a HISTORY as my source of data, the WF executes fine, but the history I use is emptied of all the files in there! This is very bizarreAny ideas what is going on here? I know not a lot of people are using the API, but it is the only way I can run paired end WF on more than a couple of samples... Regards, Thon de Boer, Ph.D. Bioinformatics Guru +1-650-799-6839 thondeb...@me.com LinkedIn Profile ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] History items get REMOVED when using the API to run workflows
On Jul 5, 2012, at 5:33 PM, Thon Deboer wrote: data['no_add_to_history']=True ? Should do it. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Looking for recommendations: How to run galaxy workflows in batch
In your workflows, are you using Input Dataset steps? Galaxy uses these steps to know how to map datasets to do special things like this. If you're not currently using them, just open the workflow editor and add input dataset steps (it's at the very bottom of the tool list) connected to the tool inputs at the highest level of the workflow, and you'll see the multiple dataset flagging when you go to run it next time. -Dannon On Jul 4, 2012, at 3:19 AM, Bernd Jagla wrote: Dannon Baker dannonbaker@... writes: Hi Dave, Yes, galaxy's standard run-workflow dialog has a feature where you can select multiple datasets as input for a single Input Dataset step. To do this, click the icon referenced by the tooltip in the screenshot below to select multiple files. All parameters remain static between executions except for the single input dataset that gets modified for each run, and that only one input dataset can be set to multiple files in this fashion. -Dannon Dannon, what if I don't have this icon??? How can I enable this? Where is this documented? Thanks, Bernd ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Server Error after fresh installation of Galaxy
It looks like what happened is that you've updated the instance that was
originally connected to this database from galaxy-central at some point
recently. The current tip of galaxy-central is at database version 103.
The first two options that come to mind are:
1) Hook a fresh galaxy-central clone up to this database and `manage_db.sh
downgrade 97` which should set your database back to the version appropriate
for galaxy-dist.
2) Swap to galaxy-central instead of -dist
Taking a brief look at the migrations, #1 looks fairly safe, but as always --
back up the database first.
-Dannon
On Jul 3, 2012, at 1:03 PM, Iry Witham wrote:
I am still experiencing issues getting my galaxy server to start since the
upgrade. The logs are all getting this type of error:
galaxy@galaxy2:/hpcdata/galaxy-test/galaxy-setup/galaxy-dist cat runner0.log
python path is:
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/numpy-1.6.0-py2.6-linux-x86_64-ucs4.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/boto-2.2.2-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/mercurial-2.1.2-py2.6-linux-x86_64-ucs4.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/Whoosh-0.3.18-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/pycrypto-2.0.1-py2.6-linux-x86_64-ucs4.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/python_lzo-1.08_2.03_static-py2.6-linux-x86_64-ucs4.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/bx_python-0.7.1_7b95ff194725-py2.6-linux-x86_64-ucs4.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/amqplib-0.6.1-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/pexpect-2.4-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/SQLAlchem!
y-0.5.6_dev_r6498-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/Babel-0.9.4-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/MarkupSafe-0.12-py2.6-linux-x86_64-ucs4.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/Mako-0.4.1-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/WebHelpers-0.2-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs4.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/wchartype-0.1-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/elementtree-1.2.6_20050316-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/docutils-0.7-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/WebOb-0.8.5-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/Routes-1.12.3-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/PasteDe!
ploy-1.3.3-py2.6.egg, /hpcdata/galaxy-test/galaxy-setup/galaxy!
-dist/eg
gs/PasteScript-1.7.3-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/Paste-1.6-py2.6.egg,
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib, /usr/lib/python26.zip,
/usr/lib64/python2.6, /usr/lib64/python2.6/plat-linux2,
/usr/lib64/python2.6/lib-tk, /usr/lib64/python2.6/lib-old,
/usr/lib64/python2.6/lib-dynload, /usr/lib64/python2.6/site-packages/Numeric,
/usr/lib64/python2.6/site-packages/gtk-2.0,
/usr/local/lib64/python2.6/site-packages, /usr/lib64/python2.6/site-packages
galaxy.model.migrate.check DEBUG 2012-07-03 12:35:03,580 psycopg2 egg
successfully loaded for postgres dialect
Traceback (most recent call last):
File
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/galaxy/web/buildapp.py,
line 82, in app_factory
app = UniverseApplication( global_conf = global_conf, **kwargs )
File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/galaxy/app.py,
line 35, in __init__
create_or_verify_database( db_url, kwargs.get( 'global_conf', {} ).get(
'__file__', None ), self.config.database_engine_options, app=self )
File
/hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/galaxy/model/migrate/check.py,
line 103, in create_or_verify_database
% ( db_schema.version, migrate_repository.versions.latest, config_arg ) )
Exception: Your database has version '101' but this code expects version
'97'. Please backup your database and then migrate the schema by running 'sh
manage_db.sh -c ./universe_wsgi.runner.ini upgrade'.
Removing PID file runner0.pid
I have not dropped my database and built it fresh. However, I have attempted
the recommended 'sh manage_db.sh -c ./universe_wsgi.webapp.ini upgrade' for
both the runner and web app ini files with the following error:
galaxy@galaxy2:/hpcdata/galaxy-test/galaxy-setup/galaxy-dist sh manage_db.sh
-c ./universe_wsgi.webapp.ini upgrade
Traceback (most recent call last):
File ./scripts/manage_db.py, line 63, in module
main( repository=repo, url=db_url )
File
Re: [galaxy-dev] Workflow API (runtime modification of tool parameters)
Hi Richard, that'd be great! The easiest thing for us is a bitbucket pull
request on a clean fork (with your changes applied) of galaxy-central.
-Dannon
On Jun 21, 2012, at 4:54 PM, Richard Park wrote:
Hi Dannon,
Is there a guide or format you need for submitting pull requests? I'd
like to contribute my API enhancements specifically for importing,
creating, deleting workflows. I also had a delete library api call,
but I think that was also added to the galaxy code recently.
thanks,
Richard
On Tue, Apr 10, 2012 at 3:45 PM, Dannon Baker dannonba...@me.com wrote:
Richard,
For specifying or changing species, you may want to inspect the input via
the API (api/histories/id/contents/id/) and snag the genome_build
attribute that way. Or am I missing the issue you're running into?
As an aside, I'd love to look at the runtime parameter specification changes
you've made for potential inclusion. This is a feature I've wanted to add
for some time.
Thanks!
Dannon
On Apr 4, 2012, at 4:17 PM, Richard Park wrote:
Hi Dannon,
Thank for the pointers from before, I have a version of galaxy that is able
to change runtime parameters through the API when running a workflow.
Format: param=tool name=tool parameter=value
# example execution: (currently changing 2 parameters: 1) bowtie,
suppressHeader parameter 2) Chip-seq peakcalling, aligner change
python workflow_execute.py api_key http://localhost:8080/api/workflows
workflow_galaxy_id 'Test API' '69=ld=a799d38679e985db'
'70=ld=33b43b4e7093c91f' 'param=peakcalling_spp=aligner=arachne'
'param=bowtie_wrapper=suppressHeader=True'
However, I was wondering if you had any pointers on how to deal w/ changing
species or chromosome lengths?
# example workflow: bowtie tool
tool_id: bowtie_wrapper,
tool_state: {\suppressHeader\: False,
\refGenomeSource\: \{\\\genomeSource\\\: \\\indexed\\\,
\\\index\\\: \\\dm3\\\, \\\__current_case__\\\: 0}\, \__page__\:
0, \chromInfo\:
/data/home/galaxy/galaxy-dist/tool-data/shared/ucsc/chrom/dm3.len,
\singlePaired\: \{\\\sInput1\\\: null, \\\sParams\\\:
{\\\sMismatchSeed\\\: \\\2\\\, \\\sUnmappedFile\\\: \\\False\\\,
\\\sTrimH\\\: \\\0\\\, \\\sTrimL\\\: \\\0\\\,
\\\sSuppressAlign\\\: \\\1\\\, \\\sSettingsType\\\: \\\full\\\,
\\\sSeed\\\: \\\-1\\\, \\\sMismatchQual\\\: \\\70\\\,
\\\sMaqSoapAlign\\\: \\\-1\\\, \\\sAlignLimit\\\: \\\-1\\\,
\\\sTryHard\\\: \\\noTryHard\\\, \\\sRounding\\\: \\\round\\\,
\\\__current_case__\\\: 1, \\\sSkip\\\: \\\0\\\, \\\sBestOption\\\:
{\\\snMaxBacktracks\\\: \\\125\\\, \\\sBest\\\: \\\noBest\\\,
\\\__current_case__\\\: 0}, \\\sAllValAligns\\\:
\\\noAllValAligns\\\, \!
\\sOffrate\\\: \\\-1\\\, \\\sSeedLen\\\: \\\28\\\, \\\sValAlign\\\:
\\\1\\\, \\\sMaxFile\\\: \\\False\\\}, \\\sPaired\\\: \\\single\\\,
\\\__current_case__\\\: 0}\},
tool_version: 1.1.2,
type: tool,
user_outputs: []
Is there a way to configure bowtie in galaxy to use the species
automatically associated with input file? I am currently having troubles
dealing w/ parameters that are defined in sub dictionaries defined in the
tool state. Any general approaches would be appreciated.
Thanks!
Richard
On Mon, Dec 5, 2011 at 6:32 PM, Dannon Baker dannonba...@me.com wrote:
Richard,
You're correct in that currently the workflow API affords no method for
runtime modification of tool parameters, other than inputs. Depending on
your needs, it might be feasible to have a few static workflows that you
reuse often via the workflow API. If that isn't the case, and you think
it'd be likely that you'll want to, say, programmatically modify
parameters at runtime based on input data heuristics, then the API does
need to be extended. The first place to look at would be
lib/galaxy/web/controllers/workflow.py (run() method), to see the general
approach to running workflows and providing parameters in the existing
infrastructure. Then, see the API version at
lib/galaxy/web/api/workflows.py (create() method) for a comparison.
Let me know if you need any help with this at all. Once it's done, if
you're willing, we'd certainly like to merge the changes back in and
include the functionality in the base galaxy distribution.
-Dannon
On Dec 5, 2011, at 5:10 PM, Richard Park wrote:
Hello,
I was wondering what would be the best way to extend Galaxy's API
functionality to allow for runtime modification of tool parameters?
I have successfully been able to run workflows programmatically using
the API, following the basic steps in:
scripts/api/execute_workflow.py.
scripts/api/example_watch_folder.py
However, it is unclear to me, what would be the best way to run
workflows through API with specific parameters at various steps.
Should I generate new workflows for every workflow that requires
different parameters and upload this to galaxy
Re: [galaxy-dev] Job splitters + SQLIte = OperationalError?
On Jun 14, 2012, at 11:37 AM, Peter Cock wrote: My hunch is that all the child-jobs are created and added to the queue and some of this is happening in parallel leading to contention over the SQLite database. Does this sound likely? Yep, this is exactly what's happening. It isn't just the job splitting code either, this happens in other circumstances as well. Is this something that could be more robust in Galaxy, or should I simply switch to a real database (SQLite or PostreSQL) for these tests? I've never taken a serious look at what it'd take to prevent this happening, but I use (and recommend) Postgres for any of hte parallelism work. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Loss of sub-task order with parallelism
On Jun 14, 2012, at 12:48 PM, Peter Cock wrote: In a separate example with 33 sub-tasks, there were two of these inversions, while in yet another example with 33 sub-tasks there was a trio submitted out of order. This non-deterministic behavior is a little surprising, but in itself not an immediate problem. You're correct in that submission order shouldn't matter at all, but I'll take a look and see if I can come up with an explanation for why. In what appears to be a separate (and more concerning) loss of order, after merging the output file order appears randomized. I would expect the output from task_0, then task_1, ..., finally task_16. I haven't yet worked out what order I am getting, but it isn't this, and neither is it the order from the SGE job numbers (e.g. correct bar one pair switched round). This would be happening in the merge. It looks like changeset c959d32f2405 might be the culprit for this -- it doesn't explicitly reorder by task number in the merge method, which would lead to (I'm guessing) an alphanumeric sort. I'll test and fix this. [*] P.S. I would like to see an upper bound on the sleep_time in method run_job, say half an hour? Otherwise with a group of long running jobs it seems Galaxy may end up waiting a very long time between checks for their completion since it just doubles the wait at each point. I had sometimes noticed a delay between the sub-jobs finishing according to the cluster and Galaxy doing anything about merging it - this is probably why. This sleep time should currently cap at 8 seconds. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] CloudMan: Command not found
You can probably modfiy /home/galaxy/.sge_request to make it work for now, but for portability of tools I'd still recommend the requirements tag. Changes to .sge_request will not be persisted after shutdown, so you'll have to redo it with each new cluster. On Jun 13, 2012, at 9:25 PM, Jose Navas wrote: Hi Dannon, thank you for your help. I've tried the requirements tag option with the 'env.sh' script. I have added ~100 XML files... there isn't any option to modify the environment variables directly? If not I will modify all my XML files thanks! Jose Subject: Re: [galaxy-dev] CloudMan: Command not found From: dannonba...@me.com Date: Wed, 13 Jun 2012 21:05:02 -0400 CC: galaxy-dev@lists.bx.psu.edu To: josenavasmol...@hotmail.com Instead of modifying the environment variables directly, I've found it easier to use Galaxy's built in tool dependency framework. All you'd have to do is add a requirements type=package tag to your custom tools to specify the package directory that needs to be checked at runtime. See http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies, specifically the Managed Tool Dependencies section. Many of the other tools on the cloud volume use the 'env.sh' method discussed there. For example, blast has the 'blast' package requirement, and in the /mnt/galaxyTools/blast/default directory you'll find an 'env.sh' file that configures the environment at runtime. -Dannon On Jun 13, 2012, at 8:49 PM, Jose Navas wrote: Hi Galaxy Team, I'm modifying my Galaxy CloudMan instance by adding custom tools. I've installed my tools under the /mnt/galaxyTools/tools folder and I've modified the .bashrc files from the sgeadmin and galaxy users to add the needed paths to the PATH and PYTHONPATH variables. When I'm in Galaxy and I try to launch one of my tools, it fails and shows the 'Command not found' error. Where I should add the paths to make Galaxy now where are my executables? Whan I log into the isntance through ssh and I use the galaxy user, it knows where are my executables. What I'm missing? Thanks, Jose ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Is there a way to turn of the pretty display for VCF files
Hmm. Chrome on OSX looks good to me, please do send over the VCF file and I can take a look. Do you see any javascript errors in the browser console? On May 29, 2012, at 6:31 PM, Anthonius deBoer wrote: Hi Dannon, I could share the VCF file, but I think it is dependent on the browser you use... I used the same file on FireFox on Linux and there it loaded the complete VCF file (Still don't see the Show All option) but on my chrome on Windows 7 it only shows the first 5 lines or so of the genotypes and then nothing...The stuff that IS shown looks correctly formatted etc. it just does not show the rest of the file which it normally does Thon On May 29, 2012, at 03:01 PM, Dannon Baker dannonba...@me.com wrote: There isn't a universe toggle for the tabular display. The VCF datatype inherits the pretty printing from the base tabular datatype, and if you'd like to disable it one way would be to override the display_data method in VCF using the raw Data display_data method. That said, I'd rather just fix your problem. Looking at a few VCF files I have, they look fine. Can you share a VCF file with me that doesn't display properly? -Dannon On May 29, 2012, at 5:50 PM, Anthonius deBoer wrote: Hi, Is there a way to turn of the special pretty print version for VCF files since it is not working correctly? I now have to download every VCF file I want to look at rather than being able to see the first MB of the text file it is... Where is this pretty printer and how do I turn it off? Thanks Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Is there a way to turn of the pretty display for VCF files
What Galaxy revision are you running? Given the new information I'd guess it's an erroneous console.log that was removed in bb7f51fb545d. On May 29, 2012, at 6:49 PM, Anthonius deBoer wrote: Interestingly when I open the Developer Tools in Chrome, my VCF file is actually loading...As soon as I close the developer tool, it stops loading the the page again (and it resumes loading when I re-open the Developer tools...) Really funky... Here's theVCF file...On my computer it only shows up to line 5 112175639 rs121913332 C T 3206.18 PASS AC=3;AF=0.0174;AN=172;COSMIC;DB;DP=15405;Dels=0.00;EFF=DOWNSTREAM(MODIFIERAPC|protein_coding|CODING|ENST0502371|),DOWNSTREAM(MODIFIERAPC|protein_coding|CODING|ENST0504915|),DOWNSTREAM(MODIFIERAPC|protein_coding|CODING|ENST0507379|),DOWNSTREAM(MODIFIERAPC|protein_coding|CODING|ENST0512211|),DOWNSTREAM(MODIFIERAPC|protein_coding|CODING|ENST0514164|),STOP_GAINED(HIGH|NONSENSE|Cga/Tga|R1450*|APC|protein_coding|CODING|ENST0257430|exon_5_112173250_112181936),STOP_GAINED(HIGH|NONSENSE|Cga/Tga|R1450*|APC|protein_coding|CODING|ENST0457016|exon_5_112173250_112181936),STOP_GAINED(HIGH|NONSENSE|Cga/Tga|R1450*|APC|protein_coding|CODING|ENST0508376|exon_5_112173250_112181753),TRANSCRIPT(MODIFIERCTC-554D6.1|processed_transcript|NON_CODING|ENST0520401|),UTR_3_PRIME(MODIFIERAPC|protein_coding|CODING|ENST0508624|);GC=45.64;HRun=0;MQ0=0;OND=0.01;set=variant5-variant49-variant10 On May 29, 2012, at 03:39 PM, Dannon Baker dannonba...@me.com wrote: Hmm. Chrome on OSX looks good to me, please do send over the VCF file and I can take a look. Do you see any javascript errors in the browser console? On May 29, 2012, at 6:31 PM, Anthonius deBoer wrote: Hi Dannon, I could share the VCF file, but I think it is dependent on the browser you use... I used the same file on FireFox on Linux and there it loaded the complete VCF file (Still don't see the Show All option) but on my chrome on Windows 7 it only shows the first 5 lines or so of the genotypes and then nothing...The stuff that IS shown looks correctly formatted etc. it just does not show the rest of the file which it normally does Thon On May 29, 2012, at 03:01 PM, Dannon Baker dannonba...@me.com wrote: There isn't a universe toggle for the tabular display. The VCF datatype inherits the pretty printing from the base tabular datatype, and if you'd like to disable it one way would be to override the display_data method in VCF using the raw Data display_data method. That said, I'd rather just fix your problem. Looking at a few VCF files I have, they look fine. Can you share a VCF file with me that doesn't display properly? -Dannon On May 29, 2012, at 5:50 PM, Anthonius deBoer wrote: Hi, Is there a way to turn of the special pretty print version for VCF files since it is not working correctly? I now have to download every VCF file I want to look at rather than being able to see the first MB of the text file it is... Where is this pretty printer and how do I turn it off? Thanks Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ CalledVariants-Select.zip ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy not killing split cluster jobs
On a related point, I've noticed sometimes one child job from a split task can fail, yet the rest of the child jobs continue to run on the cluster wasting CPU time. As soon as one child job dies (assuming there are no plans for attempting a retry), I would like the parent task to kill all the other children, and fail itself. I suppose you could merge the output of any children which did finish... but it would be simpler not to bother. Right now, yes, this would make sense- I'll see about adding it. Ultimately we want to build in a mechanism for retrying child tasks that fail due to cluster errors, etc, so it isn't necessary to rerun the entire job. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Problems saving a cloned workflow
Liisa, Are there any errors in your paster.log or javascript console? What revision are you running? -Dannon On May 3, 2012, at 2:28 PM, Liisa Koski wrote: Hello, I cloned a workflow on my local Galaxy installation, renamed it, made some edits and pressed save. It has been saving now for about 3 hours. It only has 12 steps. Any suggestions? Thanks in advance, Liisa ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Full path through API display.py
Yep, that's the right place to put it. And you'll never hear me suggest someone *not* send in a pull request, so go for it :) -Dannon On May 2, 2012, at 3:08 PM, Carlos Borroto wrote: Thanks Dannon. I was actually in a hurry so I poke around and found what I think is the right place to tie this in. In function get_api_value in lib/galaxy/model/__init__.py line 1159: rval = dict( id = hda.id, model_class = self.__class__.__name__, name = hda.name, deleted = hda.deleted, visible = hda.visible, state = hda.state, + file_name = hda.file_name, file_size = int( hda.get_size() ), data_type = hda.ext, genome_build = hda.dbkey, misc_info = hda.info, misc_blurb = hda.blurb ) Would it be helpful if I submit a pull request for this? Cause I was wondering if for changes so simple as this one, a pull request from a third party introduces more overhead than help. On Tue, May 1, 2012 at 3:05 PM, Dannon Baker dannonba...@me.com wrote: Sure, good idea. I'll tie it in. -Dannon On May 1, 2012, at 3:03 PM, Carlos Borroto wrote: Hi, Recently Full Path display was added as an option. I was wondering if this information could also be available when accessing a dataset information through the API. Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy not killing split cluster jobs
I'll take care of it. Thanks for reminding me about the TODO! On May 1, 2012, at 10:03 AM, Dannon Baker dannonba...@me.com wrote: On May 1, 2012, at 9:51 AM, Peter Cock wrote: I'm a little confused about tasks.py vs drmaa.py but that TODO comment looks pertinent. Is that the problem here? The runner in tasks.py is what executes the primary job, splitting and creating the tasks. The tasks themselves are actually injected back into the regular job queue and run as normal jobs with the usual runners (in your case drmaa). And, yes, it should be fairly straightforward to add, but this just hasn't been implemented yet. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Full path through API display.py
Sure, good idea. I'll tie it in. -Dannon On May 1, 2012, at 3:03 PM, Carlos Borroto wrote: Hi, Recently Full Path display was added as an option. I was wondering if this information could also be available when accessing a dataset information through the API. Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy version / Mercurial
Hi Sarah, You should be able to do this with the -r option of the clone command, so: `hg clone -r b258de1e6cea https://bitbucket.org/galaxy/galaxy-dist`. -Dannon On Apr 25, 2012, at 3:08 AM, Sarah Maman wrote: Hello, I would like to get source code with Mercurial of this tarballs : galaxy-dist.b258de1e6cea.tar.gz, and not the latest source. Is it possible to specify b258de1e6cea version with hg clone command ? The aim is to update my copy of Galaxy wich have been locally installed with a tarball in order to use, in the future, Mercurial and hg pull to update Galaxy version. Thanks a lot, Sarah Maman From: Sarah Maman sarah.ma...@toulouse.inra.fr Subject: Galaxy version Date: April 24, 2012 10:23:01 AM EDT To: Nate Coraor n...@bx.psu.edu, galaxy-dev-requ...@lists.bx.psu.edu Hello, I would like to get source code with Mercurial of this tarballs : galaxy-dist.b258de1e6cea.tar.gz, and not the latest source. Is it possible to specify b258de1e6cea version with hg clone command ? The aim is to update my copy of Galaxy wich have been locally installed with a tarball in order to use, in the future, Mercurial and hg pull to update Galaxy version. Thanks a lot, Sarah Maman ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Cloudman- problem with galaxy update
Hi Dave, The problem here is that the galaxy update failed to merge a change to run.sh because of minor customizations it has. We'll have a long term fix out for this in the next week, but for now what you can do is ssh in to your instance and update run.sh yourself prior to restarting galaxy. All you need to do is add 'migrated_tools_conf.xml.sample' to the SAMPLES in /mnt/galaxyTools/galaxy-central/run.sh, execute `sh run.sh --run-daemon` (or restart galaxy again from the admin page) and you should be good to go. -Dannon On Apr 24, 2012, at 4:36 PM, Dave Lin wrote: Dear Cloudman Team, I created a fresh galaxy instance using AMI: galaxy-cloudman-2011-03-22 (ami-da58aab3) In case it matters, instance type = High-Memory Double Extra Large I was trying to use the Cloudman Admin Console to update Galaxy. (http://bitbucket.org/galaxy/galaxy-central) but galaxy ran into an issue during the update process. Log message copied below: Any suggestions? Thanks in advance, Dave /mnt/galaxyTools/galaxy-central/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg/pysam/__init__.py:1: RuntimeWarning: __builtin__.file size changed, may indicate binary incompatibility from csamtools import * python path is: /mnt/galaxyTools/galaxy-central/eggs/numpy-1.6.0-py2.6-linux-x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.6-linux-x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/boto-2.2.2-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/mercurial-2.1.2-py2.6-linux-x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/Whoosh-0.3.18-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/pycrypto-2.0.1-py2.6-linux-x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/python_lzo-1.08_2.03_static-py2.6-linux-x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/bx_python-0.7.1_7b95ff194725-py2.6-linux-x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/amqplib-0.6.1-py2.6.egg, /mnt/galaxyTools/galaxy-central/egg! s/pexpect-2.4-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/Babel-0.9.4-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/MarkupSafe-0.12-py2.6-linux-x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/Mako-0.4.1-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/WebHelpers-0.2-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux-x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/wchartype-0.1-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/elementtree-1.2.6_20050316-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/docutils-0.7-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/WebOb-0.8.5-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/Routes-1.12.3-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg, /mnt/gala! xyTools/galaxy-central/eggs/Paste-1.6-py2.6.egg, /mnt/galaxyTo! ols/gala xy-central/lib, /usr/lib/python2.6/, /usr/lib/python2.6/plat-linux2, /usr/lib/python2.6/lib-tk, /usr/lib/python2.6/lib-old, /usr/lib/python2.6/lib-dynload Traceback (most recent call last): File /mnt/galaxyTools/galaxy-central/lib/galaxy/web/buildapp.py, line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File /mnt/galaxyTools/galaxy-central/lib/galaxy/app.py, line 25, in __init__ self.config.check() File /mnt/galaxyTools/galaxy-central/lib/galaxy/config.py, line 290, in check raise ConfigurationError(File not found: %s % path ) ConfigurationError: File not found: ./migrated_tools_conf.xml ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] More meaningful dataset names/easier method of identifying?
In changeset 7013:dae7eefe2f71 I added the full file path to the dataset View Details page. Galaxy administrators will always see this, and if you set expose_dataset_path to True in your universe_wsgi.ini, users will see it as well. Hopefully that's what you're looking for, but let me know if I've misunderstood what you're after and I can take another look. -Dannon On Apr 24, 2012, at 4:41 PM, Josh Nielsen wrote: Hello, For a while now with the Galaxy mirror that we have I have found on many occasions a need to identify which dataset_*.dat files on the file system (in the [galaxy_dist]/database/files/000/ directory) belong to which user, and even for the same user to distinguish between their various datasets. Files directly uploaded by the user will have a Galaxy job dataset file name which match - like a Galaxy job name of data 18 (for example) which actually is reflective of the file name 'dataset_18.dat' on the file system. However any analysis on that file thereafter that produces another dataset does not give you a clue of the corresponding file name. For example, a Clip on data 18 run some time later may be called 'dataset_44.dat' on the filesystem, and a Map with Bowtie on data 18 that runs on the clipped 'dataset_44.dat' may produce an output file of 'dataset_53.dat'. When debugging failed jobs, and after the user has rerun them for the umpteenth time, there may be dozens of identical or near-identical files to weed through, and the generic naming scheme is not helpful even though it is sequential (also not easy to keep track of/match up unless you are watching the file writes in the directory live). The current implementation makes sense for internal usage and the code that uses it, but it is difficult for a human to distinguish which files match the jobs in Galaxy. It would be useful to have more meaningful dataset file names or an easier way to identify them (a record that matches the internal and external names) for administrative maintenance reasons so that I can delete files, or possibly even export those .dat files to a network share where our users can perform manual analysis on them. Could anyone point me to where in the code I could look to make the dataset names more meaningful? Or perhaps I should request of the Galaxy developers (as a feature) a way for the users themselves to see under the metadata name of their job (like Map with Bowtie on data 18) in the right side pane the *actual* corresponding file and location on the file system path to it (dataset_53.dat, for example). Or if not for users at least something for Administrators. Even a database that has four columns for the internal/filesystem dataset name, the job metadata name, the Galaxy job number (that the user sees), and the user that the dataset belong! s to, would be helpful. A lot of our users are heavy into informatics though and would probably prefer that the user be able to see that information. Does anyone have any suggestions or thoughts about this? Thanks, Josh Nielsen ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy pbs scripts and job_working_directory files
Josh, Check out the cleanup_job setting in universe_wsgi.ini(and included below). It sounds like 'cleanup_job = onsuccess' is exactly what you're looking for. -Dannon # Clean up various bits of jobs left on the filesystem after completion. These # bits include the job working directory, external metadata temporary files, # and DRM stdout and stderr files (if using a DRM). Possible values are: # always, onsuccess, never #cleanup_job = always On Apr 24, 2012, at 5:00 PM, Josh Nielsen wrote: Hello again, As I mentioned in a recent post I often find need to debug jobs running from our local Galaxy mirror and I often have the need to look at the script data files that the job is trying to use in order to figure out what is causing a problem. The directories containing those file are in '[galaxy_dist]/database/pbs/' and '[galaxy_dist]/database/job_working_directory/' for me. Each job that is run gets a corresponding .sh file in the pbs/ directory (like 344.sh) which will have the entire sequence of bash commands to execute the job with and also a call to a wrapper script somewhere in the middle normally. That script information is very useful, but the problem is that when a job fails (often within the first 30 seconds of running it) the script is deleted and there is no trace of it left in the directory. The same with the output or job data files in job_working_directory/. I have had to suffice with using the technique of coordinating with the user when to (re)run their failed job and then quickly within the 30 second window do a cp -R script_I_care_about.sh copy_of_script.sh command, so that when the script is deleted I have a copy that I can examine. The same goes with copying the job_working_directory/ files. I know that it would get very cluttered in those directories if they were not automatically cleaned/deleted but I find those files essential for debugging. Is there a way to force Galaxy to retain those files (optionally) for debugging purposes? Maybe make a new option in the universe.ini file for that purpose that can be set for people who want it? Thanks, Josh Nielsen ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Hidden output files
Hi Frank, This should be resolved as of changeset 7057:08fbfeaaf3e1. Thanks! -Dannon On Mar 27, 2012, at 4:37 AM, Frank Sørensen wrote: Hi Dannon, Unfortunately our server is (still) behind a firewall, so I can't share anything, but I hope you can reproduce the error from the following description. If I make a vanilla test workflow say: Input Dataset - Map with BWA for Illumina (single end) - SAM-to-BAM - flagstat and save it. Then all outputs will show if I run it, even if none of the snowflakes are checked which is not logical. It would be better if the output visibility was checked by default, and the user should actively de-select the outputs that should be hidden, but that's another matter. If I edit the workflow that I just created, and select the output from flagstat to be visible, then the two other (BWA and SAM-to-BAM) will be hidden when I run it, which they should be. So far so good... If I edit the workflow again and select the remaining outputs (BWA and SAM-to-BAM) to be visible and save the workflow, then only BWA will be visible after run, thats a problem, SAM-to-BAM should also be visible since that's what I selected. If I want the output from SAM-to-BAM to be visible, I have to edit the workflow again, and de-select visibility of the output from SAM-to-BAM, save the workflow, and re-edit the workflow and select visibility of the output from SAM-to-BAM and save the workflow. It seems that selection/de-selection of output visibility only works under some circumstances and not others. I can conclude the state of the output visibility on the existing or missing post-hide-action, which is apparent if I select run workflow, and press expand all on the Running workflow screen, so I don't actually have to run the workflow, which is a great help. By tweaking my workflows using this editing / re-editing method, it is actually possible to get the visibility of all outputs into the state that I desire, but it is quite cumbersome on larger workflows. Regards - Frank ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Exporting worflow to local instal
Hi Danny, For moving workflows from one instance to another you'll want to click on Download or Export in the workflow context menu and use the URL for Importing to Another Galaxy. It looks something like this: (note the for_direct_import, that's how you'll know you have the right link type). https://main.g2.bx.psu.edu/workflow/for_direct_import?id=5214085ac1d2a9af And, the duplicate headers issue should be resolved on our main server now, please let us know if you find this isn't the case. -Dannon On Mar 14, 2012, at 12:13 PM, Muehlschlegel, Jochen D.,M.D. wrote: Hi, I would like to download/export my workflows from the Galaxy main to a local install at my institution. When I use the URL method, I get the following error message: Data at 'http://main.g2.bx.psu.edu/u/muehlsch12/w/imported-tophat---cuffdiff-paired-end-fastq ' does not appear to be a Galaxy workflow Message: No JSON object could be decoded: line 1 column 0 (char 0) When I want to download a workflow to my local machine, I get a server error (Duplicate headers received from server). The local install does not recognize publicly published workflows, so that step won't work either. Thanks for your help, Danny The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Hidden output files
On Apr 18, 2012, at 8:44 AM, Frank Sørensen wrote: - Should I pull from galaxy-central to get this update (7057:08fbfeaaf3e1)? - If so, could you please tell me how I do that? Sure, you can pull from galaxy-central by issuing a manual pull with source from your galaxy-dist directory: hg pull -u https://bitbucket.org/galaxy/galaxy-central - Do you know how I permanently change repository from galaxy-dist to galaxy central, without loosing my tools and modifications? If you want to permanently change to galaxy-central, you'll need to manually edit the file galaxydirectory/.hg/hgrc. It should look something like this: [paths] default = https://bitbucket.org/galaxy/galaxy-dist Just change galaxy-dist to galaxy-central, save it, and you're good to go. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] database engine pool size
On Mar 26, 2012, at 12:18 PM, Shantanu Pavgi wrote:
{{{
TimeoutError: QueuePool limit of size 40 overflow 50 reached, connection
timed out, timeout 30
}}}
These limits are not reached for regular (non-workflow) galaxy jobs. Any help
on optimum values for these settings or performance tuning for workflow runs
would be appreciated.
Hi Shantanu,
Two things could help here:
1. Enable database_engine_option_strategy = threadlocal in your
universe_wsgi.ini
2. Run additional Galaxy processes:
http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application%20Scaling
-Dannon
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Private key file format issue (for connecting EC2 instance with EBS volume)
Mo, I just tested one of my amazon keys with PuTTYgen and the default settings and it worked. Can you verify the contents of the .pem file you're using for me? On windows I'd open it in http://notepad-plus-plus.org/ or a similar utility to see it correctly. The file should look something like this, with both start and end lines. -BEGIN RSA PRIVATE KEY- about 20 lines of random characters -END RSA PRIVATE KEY- And, lastly, the amazon documentation at http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-06-26/ says to make sure the file ends with a newline character or it won't work with PuTTYgen, so that's something to look for. -Dannon On Apr 18, 2012, at 3:29 PM, Mohammad Heydarian wrote: Hello, I am trying to connect an EC2 instance to an EBS volume. I have attached the EBS volume to the EC2 instance (running Galaxy). I now need to mount the EBS volume to the EC2 instance. To do this I will use Putty to make the connection. But first I have to convert the private key from .pem format to .ppk format. I am trying to use Puttygen for the conversion, but the .pem file is not recognized and I get this error message: Could't load private key (key file does not begin with OpenSSH key header) Does anyone have experience with this? BTW, I am using windows. Thanks! Cheers, Mo Heydarian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Workflow import error
Robert, It sounds like you're experiencing two separate problems to me. Let's isolating the problem to just the FastQC tool install, first, and move on from there. There is a FastQC tool in your toolbar, and execution results in the error you describe in the previous email of 'Fastq failed. Error executing FastQC. /bin/sh: fastqc command not found'. Correct? If this is the case, can you verify that you do have fastqc installed and accessible on the path of the galaxy user? And, how did you add the tool to galaxy- via toolshed or did you use the FastQC wrapper included with the primary Galaxy distribution? -Dannon On Apr 16, 2012, at 6:23 PM, Greg Von Kuster wrote: Hi Robert, I've cc'd the galaxy-dev list on this reply since the workflow components are primarily supported by other Galaxy development team members and I'm not optimally suited to handle this issue. Please keep responses on the galaxy-dev mail list so others that are interested can follow the entire thread. You should receive a response on this soon. Thanks for sending your messages, Greg Von Kuster On Apr 16, 2012, at 5:53 PM, Robert Chase wrote: Hi Greg, When I upload my workload ga file to galaxy, I get a screen saying that some of the tools in the workflow are not available in this instance of galaxy, specifically fastqc. I downloaded fastqc into my tools directory and restarted galaxy, but when I tried to re-upload the workflow it complained again that fastqc wasn't available. Fastqc does in fact load in the tools sidebar, but when I try to run it I get the following: An error occurred running this job: Fastq failed. Error executing FastQC. /bin/sh: fastqc: command not found -Rob On Mon, Apr 16, 2012 at 10:21 AM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Rob, When you import a workflow into a Galaxy instance, the instance must have all tools required by the workflow available in its tool panel. I've only seen this error when the workflow requires a tool that is not available, so can you let us know if this is the case? Thanks, Greg Von Kuster On Apr 16, 2012, at 10:07 AM, Robert Chase wrote: Hi Greg, After I modified the modules.py file that bug went away, but now I'm getting a new error: Module galaxy.web.base.controller:420 in get_stored_workflow_steps view module = module_factory.from_workflow_step( trans, step ) #Check if tool was upgraded step.upgrade_messages = module.check_and_update_state() # Any connected input needs to have value DummyDataset (these # are not persisted so we need to do it every time) step.upgrade_messages = module.check_and_update_state() AttributeError: 'NoneType' object has no attribute 'check_and_update_state' -Rob On Mon, Apr 16, 2012 at 9:19 AM, Robert Chase re...@channing.harvard.edu wrote: Thanks Greg, I'll try to merge the change into my version. -Rob On Mon, Apr 16, 2012 at 9:18 AM, Greg Von Kuster g...@bx.psu.edu wrote: Hello Robert, Thanks for reporting this issue - it has been fixed in change set 7038:1fdcce63a06f, which is currently available only in our central repository. It will be available in the dist in the next update. Greg Von Kuster On Apr 13, 2012, at 5:22 PM, Robert Chase wrote: Hello All, I'm pretty sure I found a bug in the main galaxy distribution. The following code diff shows the introduction of the line tool_version = self.__get_tool_version( trans, tool_id ) but if you look at the function definition containing that line, you see def from_workflow_step( Class, trans, step ): which does not half a self in it. Then we get an error. https://bitbucket.org/galaxy/galaxy-dist/diff/lib/galaxy/workflow/modules.py?diff2=e6464387ed3fdiff1=c60760713d26 -Rob ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Traceback on original execution
On Apr 10, 2012, at 2:42 PM, Daniel Patrick Sullivan wrote: Is this somthing that somebody could possibly commment on? Should I just try to install a more recent version of python? Thank-you so much for your help and guidance. Hi Dan, This past October we did officially deprecate python2.4, and this particular error aside you may run into further issues trying to run galaxy under 2.4. If you're able to install a newer version of python, I would recommend doing so. Prior to python2.5, you could not have 'finally' and 'except' clauses in the same 'try' statement, and that's the issue you're seeing here. -Dannon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Workflow API (runtime modification of tool parameters)
Richard,
For specifying or changing species, you may want to inspect the input via the
API (api/histories/id/contents/id/) and snag the genome_build attribute
that way. Or am I missing the issue you're running into?
As an aside, I'd love to look at the runtime parameter specification changes
you've made for potential inclusion. This is a feature I've wanted to add for
some time.
Thanks!
Dannon
On Apr 4, 2012, at 4:17 PM, Richard Park wrote:
Hi Dannon,
Thank for the pointers from before, I have a version of galaxy that is able
to change runtime parameters through the API when running a workflow.
Format: param=tool name=tool parameter=value
# example execution: (currently changing 2 parameters: 1) bowtie,
suppressHeader parameter 2) Chip-seq peakcalling, aligner change
python workflow_execute.py api_key http://localhost:8080/api/workflows
workflow_galaxy_id 'Test API' '69=ld=a799d38679e985db'
'70=ld=33b43b4e7093c91f' 'param=peakcalling_spp=aligner=arachne'
'param=bowtie_wrapper=suppressHeader=True'
However, I was wondering if you had any pointers on how to deal w/ changing
species or chromosome lengths?
# example workflow: bowtie tool
tool_id: bowtie_wrapper,
tool_state: {\suppressHeader\: False,
\refGenomeSource\: \{\\\genomeSource\\\: \\\indexed\\\, \\\index\\\:
\\\dm3\\\, \\\__current_case__\\\: 0}\, \__page__\: 0, \chromInfo\:
/data/home/galaxy/galaxy-dist/tool-data/shared/ucsc/chrom/dm3.len,
\singlePaired\: \{\\\sInput1\\\: null, \\\sParams\\\:
{\\\sMismatchSeed\\\: \\\2\\\, \\\sUnmappedFile\\\: \\\False\\\,
\\\sTrimH\\\: \\\0\\\, \\\sTrimL\\\: \\\0\\\, \\\sSuppressAlign\\\:
\\\1\\\, \\\sSettingsType\\\: \\\full\\\, \\\sSeed\\\: \\\-1\\\,
\\\sMismatchQual\\\: \\\70\\\, \\\sMaqSoapAlign\\\: \\\-1\\\,
\\\sAlignLimit\\\: \\\-1\\\, \\\sTryHard\\\: \\\noTryHard\\\,
\\\sRounding\\\: \\\round\\\, \\\__current_case__\\\: 1, \\\sSkip\\\:
\\\0\\\, \\\sBestOption\\\: {\\\snMaxBacktracks\\\: \\\125\\\,
\\\sBest\\\: \\\noBest\\\, \\\__current_case__\\\: 0},
\\\sAllValAligns\\\: \\\noAllValAligns\\\, \\\!
sOffrate\\\: \\\-1\\\, \\\sSeedLen\\\: \\\28\\\, \\\sValAlign\\\:
\\\1\\\, \\\sMaxFile\\\: \\\False\\\}, \\\sPaired\\\: \\\single\\\,
\\\__current_case__\\\: 0}\},
tool_version: 1.1.2,
type: tool,
user_outputs: []
Is there a way to configure bowtie in galaxy to use the species automatically
associated with input file? I am currently having troubles dealing w/
parameters that are defined in sub dictionaries defined in the tool state.
Any general approaches would be appreciated.
Thanks!
Richard
On Mon, Dec 5, 2011 at 6:32 PM, Dannon Baker dannonba...@me.com wrote:
Richard,
You're correct in that currently the workflow API affords no method for
runtime modification of tool parameters, other than inputs. Depending on
your needs, it might be feasible to have a few static workflows that you
reuse often via the workflow API. If that isn't the case, and you think
it'd be likely that you'll want to, say, programmatically modify parameters
at runtime based on input data heuristics, then the API does need to be
extended. The first place to look at would be
lib/galaxy/web/controllers/workflow.py (run() method), to see the general
approach to running workflows and providing parameters in the existing
infrastructure. Then, see the API version at
lib/galaxy/web/api/workflows.py (create() method) for a comparison.
Let me know if you need any help with this at all. Once it's done, if
you're willing, we'd certainly like to merge the changes back in and
include the functionality in the base galaxy distribution.
-Dannon
On Dec 5, 2011, at 5:10 PM, Richard Park wrote:
Hello,
I was wondering what would be the best way to extend Galaxy's API
functionality to allow for runtime modification of tool parameters?
I have successfully been able to run workflows programmatically using
the API, following the basic steps in:
scripts/api/execute_workflow.py.
scripts/api/example_watch_folder.py
However, it is unclear to me, what would be the best way to run
workflows through API with specific parameters at various steps.
Should I generate new workflows for every workflow that requires
different parameters and upload this to galaxy? Or would it be better
to extend the API to allow for runtime parameter configuration? If
this is the case, any pointers on how to extend this would be greatly
appreciated.
Thank you,
Richard Park
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies
Re: [galaxy-dev] Hidden output files
Could you share an offending workflow with me? Or is it any workflow? I've not seen this behavior before, but it definitely shouldn't be random. So in your situation the same workflow run multiple times will not consistently hide the same datasets? -Dannon On Mar 23, 2012, at 6:05 AM, Frank Sørensen wrote: Hi, I seem to have some problems with the visibility of tool outputs in my workflows. When I flag any given output as visible, Galaxy sometimes decides to hide it in the resulting history anyway. I can't find any systematic behavior though. It seems to be rather arbitrary weather Galaxy decides that the file should be hidden or not. I would estimate the random hiding to occur on about 50% of my outputs. Has anyone seen this before? Regards - Frank -- Frank Sørensen, B.Sc., Programmer Molecular Diagnostic Laboratory (MDL) Molekylær Medicinsk Afdeling (MOMA) Århus Universitetshospital Skejby, Brendstrupgårdsvej, 8200 Århus N Tlf. +45 7845 5363 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] HTML output with Javascript
Assuming everything is generated correctly for the html report, what you're probably running into is Galaxy's html sanitization. In your Galaxy instance's universe_wsgi.ini you can set the following option (to false) to disable this: # Sanitize All HTML # By default, all tool output served as 'text/html' will be sanitized # thoroughly. This can be disabled if you have special tools that require # unaltered output. #sanitize_all_html = True -Dannon On Mar 22, 2012, at 7:35 PM, Jose Navas wrote: Hi Galaxy developers, I have a tool that generates an html output which executes some javascript functions that are defined in a *.js file. I put this file under the $output1.files_path but when I display the html file in the galaxy browser, it doesn't executes the Javascript. What can I do to execute Javascripts in the Galaxy browser? Thanks, Jose ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] HTML output with Javascript
It's in the [app:main] section. See the universe_wsgi.ini.sample file for the exact position-- this file will always be up-to-date with the initial configuration options that might have shown up as you've updated your galaxy instance. -Dannon On Mar 22, 2012, at 7:50 PM, Jose Navas wrote: Hi Dannon, Thank you for your quick answer. I search in my Galaxy instance's universe_wsgi.ini and I didn't find this option... Where can I add this option? Thanks, Jose Subject: Re: [galaxy-dev] HTML output with Javascript From: dannonba...@me.com Date: Thu, 22 Mar 2012 19:44:05 -0400 CC: galaxy-dev@lists.bx.psu.edu To: josenavasmol...@hotmail.com Assuming everything is generated correctly for the html report, what you're probably running into is Galaxy's html sanitization. In your Galaxy instance's universe_wsgi.ini you can set the following option (to false) to disable this: # Sanitize All HTML # By default, all tool output served as 'text/html' will be sanitized # thoroughly. This can be disabled if you have special tools that require # unaltered output. #sanitize_all_html = True -Dannon On Mar 22, 2012, at 7:35 PM, Jose Navas wrote: Hi Galaxy developers, I have a tool that generates an html output which executes some javascript functions that are defined in a *.js file. I put this file under the $output1.files_path but when I display the html file in the galaxy browser, it doesn't executes the Javascript. What can I do to execute Javascripts in the Galaxy browser? Thanks, Jose ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Can't clone workflows
Liisa, I'm not able to reproduce this locally with a fresh galaxy-dist. Is there anything unique about your workflows or configuration here? And, this might be a long shot, but do you frequently use tags with your workflows? There was an issue that I've fixed with this recently that would cause a significant hang when cloning. -Dannon On Feb 29, 2012, at 4:03 PM, Liisa Koski wrote: Hello, I have lost the ability to clone workflows in my local installation of Galaxy (the latest galaxy-dist). When I try...it just hangs...and hangs... Any help would be much appreciated. Thanks, Liisa ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Possible bug: Tags in a workflow gets duplicated hundreds of times
Hi Thon, Thanks for reporting this. I see what the problem is here at least for the clone duplication, and I've committed a fix in 6833:e8e361707865 that will affect all workflows going forward. Unfortunately, there isn't a complete solution for fixing the extra tags. The problem was that any time you cloned a workflow, the mapping that retrieved the set of tags that should have been copied was actually returning *all* of the user's tags. That means that even if the source workflow had no tags, all workflow tags owned by the user would get copied to the newly created workflow. Because of this, just removing duplicate tags (while an obvious step in the right direction) won't be enough and you'll still have tags where they shouldn't be. I'll add the duplicate removal code as a migration, but in the short term if there are a few individual workflows that you need to fix (by simply removing all tags) the easiest thing to do on your end would be to manually remove the stored_workflow_tag_associations for those particular workflows, using something like this sql statement: delete from stored_workflow_tag_associations where stored_workflow_id = 'enter stored_workflow id to fix' -Dannon On Mar 19, 2012, at 1:47 PM, Anthonius deBoer wrote: Hi, I have run into an issue with a tag being duplicated hundreds of times each time I open, edit or clone a workflow. I have tried to manually remove them, but each time the tag gets inserted somehow and duplicated each time I edit the workflow. By now there are hundreds of copies of the tags at it makes loading and running the workflow very slow. Is there a way to remove a tag from the system somehow? I am willing to dig into the database with some SQL statement but not sure where to start... Thanks Thon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Looking where to edit web view
Ed, The top bar is defined in templates/webapps/galaxy/base_panels.mako. To create a new masthead tab, you would have to edit that file in addition to creating the relevant controller methods for the functionality you wish to add. -Dannon On Mar 20, 2012, at 8:32 PM, Edward Hills wrote: Hello Galaxy team, I've recently begun building a resource usage monitor / predictor for Galaxy as part of my Computer Science Honours degree and I was wanting to at some later stage in my project, once the actual resource monitor is built, to incorporate it to the actual Galaxy web interface. have been looking through your code and have found it somewhat difficult to track down where exactly i would go about doing this. I would like to ahve a separate tab up in the top nav bar and then be able to display resource information about Galaxy when its running (processes, I/O, RAM, CPU etc.) Are you able to point me in the right direction for this? Thankyou, Ed ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Reducing costs in Cloud Galaxy
Just one extra thought on this-- If you leave your instance up all the time it may be worth looking into having a reserved micro instance up as the front end (cheap, or free, with your intro tier) with SGE submission disabled. Then, enable autoscaling(max 1) of m1.large/xlarge instances. -Dannon On Mar 19, 2012, at 7:20 PM, Dave Clements wrote: Hi Enis, Greg, I've taken stuff from my this email, and previous conversations with Enis and put it in the wiki: http://wiki.g2.bx.psu.edu/Admin/Cloud/CapacityPlanning Please feel free to update/correct/enhance. Dave C. On Mon, Mar 19, 2012 at 2:58 PM, Enis Afgan eaf...@emory.edu wrote: Greg, Regarding the performance of different types of instances, I came across this and thought you might potentially find it useful: http://cloudharmony.com/benchmarks Enis On Mon, Mar 19, 2012 at 7:49 PM, Greg Edwards gedwar...@gmail.com wrote: Enis, Thanks. Will try that re the storage. Greg E On Mon, Mar 19, 2012 at 4:49 PM, Enis Afgan eaf...@emory.edu wrote: Hi Greg, On Mon, Mar 19, 2012 at 11:01 AM, Greg Edwards gedwar...@gmail.com wrote: Hi, I've got an implementation of some proteomics tools going well in Galaxy on AWS EC2 under Cloudman. Thanks for the help along the way. I need to drive the costs down a bit. I'm using an m1.large AMI and it's costing about $180 - $200 / month. This is about 55% storage and 45% instance costs. That's peanuts in some senses but for now we need to get it down so that it comes out of petty cash for the department, while the case is proven for it's use. I have a few questions and would appreciate ny insights .. 1. AWS has just released an m1.medium and m1.small instance type, which are 1/2 and 1/4 the cost of m1.large. http://aws.amazon.com/ec2/instance-types/ http://aws.amazon.com/ec2/pricing/ I tried the m1.small and m1.medium with the latest Cloudman AMI galaxy-cloudman-2011-03-22 (ami-da58aab3) All seemed to install ok, but the Tools took up tp 30 minutes to start execution on m1.medium, and never started on m1.small. m1.medium only added about 15% to run times compared with m1.large, can't say for m1.small. t1.micro does run (and for free in my Free Tier first year) but blows execution times out by a factor of about 3 which is too much. Has anyone tried these new Instance Types ? (m1.small/medium) I have no real experience with these instance types yet either so maybe someone else can chime in on this? 2. The vast majority of the storage costs are fro the Gemome databases in the 700GB /mnt/galaxyIndices, which I don't need. Can this be reduced to the bare essentials ? You can do this manually: 1. Start a new Galaxy cluster (ie, one you can easily delete later) 2. ssh into the master instance and delete whatever genomes you don't need/want (these are all located under /mnt/galaxyIndices) 3. Create a new EBS volume of size that'll fit whatever's left on the original volume, attach it and mount it 4. Copy over the data from the original volume to the new one while keeping the directory structure the same (rsync is probably the best tool for this) 5. Unmount detach the new volume; create a snapshot from it 6. For the cluster you want to keep around (while it is terminated), edit persistent_data.yaml in it's bucket on S3 and replace the existing snap ID for the galaxyIndices with the snapshot ID you got in the previous step 7. Start that cluster and you should have a file system from the new snapshot mounted. 8. Terminate delete the cluster you created in step 1 If you don't want to have to do this the first time around on your custom cluster, you can first try it with another temporary cluster and make sure it all works as expected and then move on to the real cluster. Best, Enis Using m1.small/medium and getting rid of the 700GB would being my costs down to say $50 / month which is ok. Thanks ! Greg E -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Greg Edwards, Port Jackson Bioinformatics gedwar...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- http://galaxyproject.org/GCC2012 http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://galaxyproject.org/wiki/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your
Re: [galaxy-dev] postgres
Yes, see the table 'job'. You may or may not find this helpful, but here's a big database schema from our wiki: http://wiki.g2.bx.psu.edu/DataModel?action=AttachFiledo=viewtarget=galaxy_schema.png -Dannon On Mar 9, 2012, at 8:00 AM, christin weinberg wrote: Hi, thanks for the answer. I have one more question. Is there one main table (in the database), which creates one key for one job and this key is allocated over other tables? Thank you! Best, Christin Am 08.03.2012 16:40, schrieb Björn Grüning: Hi Christin, I would like to clarify whether we have understood something about the database correctly. Is is right that all calculations with galaxy are stored as a link in the database. And the data are stored somewhereelse? Yes, kind of :) Is it possible to store the data only in the database? No and its advisable to do so. Galaxy operates on files and they can be huge. It does not make much sense to store such data in a database. Best, Bjoern For some advice, I am very grateful. Best regards, Christin -- Christin Weinberg Institute for Applied Computer Science (IACS) FH Stralsund - University of Applied Sciences Phone: +49 3831 456948 E-Mail : christin.weinb...@fh-stralsund.de ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy SNPEff Display Bug (2nd Independent Report)
This was addressed in 6788:e58a87c91bc4. The reason for the initial change that's causing these display issues was to eliminate potential XSS vulnerabilities. There's now a configuration option (sanitize_all_html, which is True by default) for local instances where you can disable the extra html sanitization. -Dannon On Mar 8, 2012, at 10:33 AM, John David Osborne wrote: Hi Pablo, About 3 weeks ago Praveen mentioned a bug in SNPEff such that it no longer correctly displays the HTML report in galaxy – we have the exact same error. Is there a workaround for this yet? I found his original report here: http://article.gmane.org/gmane.science.biology.galaxy.user/2418 The report looks great when generated from the command line or even if the files from galaxy are copied over to a local machine and displayed. However in galaxy it appears that the supplementary data file (for example dataset_41.dat.genes.txt) from the original dataset_41.dat is not accessible to the main html file although I can copy it off the server. Perhaps the data file it is being called from elsewhere at this is a path related issue? The html looks for the data file in the same directory which I think is quite reasonable. I'm not sure which galaxy update broke this, but it is definitely broken and I'm guessing any efforts to display html reports in galaxy that involve other supplementary data files may be broken too…. -John ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Using complete workflow in another workflow
This is definitely something we're looking to implement in Galaxy. I started working on a proof of concept a while back for boxing workflows up as tools, but I have not had a chance to finish it yet. What Thon suggests in terms of a simple copy/paste is interesting, and probably a far simpler first step that I'll look into. I could imagine a configurable set of 'component' workflows that you could click and have the steps plopped into your existing workflow. -Dannon On Mar 6, 2012, at 10:25 AM, Ann Black wrote: Hello - I would also like the +1 this feature. I currently use the clone feature of a workflow to make a copy and modify, but if I change the original workflow, I think have to remember which workflows I cloned into and hand modify them as well. If we had the ability to re-use workflows, then we could develop modules that could be re-used in larger workflows. Thanks much for the consideration, Ann -- Message: 3 Date: Thu, 01 Mar 2012 20:14:25 + (GMT) From: thondeb...@me.com To: Galaxy-dev galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Using complete workflow in another workflow Message-ID: 7d9b3791-1822-e517-b5b9-e1ba1579b...@me.com Content-Type: text/plain; charset=utf-8; Format=flowed Hi, How far along are we about thinking about being able to re-use a complete workflow as a workflow step in another workflow? This would really allow us to modularize certain aspects of the analyses and would allow us to re-use a workflow in another. Barring that, a simple copy/paste from one workflow into another would also really help... Any plans in that direction? Thanks Thon? -- next part -- An HTML attachment was scrubbed... URL: http://lists.bx.psu.edu/pipermail/galaxy-dev/attachments/20120301/20f0e47 3/attachment-0001.html * ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Gal-Cloudman AMI Fails
John, I'm still working on figuring out who created the AMI you're running across (it isn't under the main Galaxy cloud account, see the '072133624695' in the name string), but for future reference we'll always keep the recommended AMI to use listed on usegalaxy.org/cloud. At this point, it's still ami-da58aab3 (861460482541/galaxy-cloudman-2011-03-22). -Dannon On Mar 6, 2012, at 10:47 PM, John Major wrote: Hello GalFolks- I saw a new galaxy-cloudman AMI appear ( 072133624695/galaxy-cloudman-2012-02-26 ) and was interested in giving it a whirl. But when I tried to launch it got the following error: The requested instance type's architecture (x86_64) does not match the architecture in the manifest for aki-407d9529 (i386) Is this a dev-AMI? Or should I be expecting this to run? Thanks- John ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy strips CSS from HTML files
A tool whitelist is an interesting idea that deserves more thought, but for now I'm going to add an option to the galaxy configuration to easily enable or disable the extended html filtering entirely. It'll be enabled by default, but this should make it easier for the administrators of local Galaxy instances in the case where you have custom tools that need to do fancy things and have other security and access controls in place. -Dannon On Mar 2, 2012, at 4:05 AM, Ayton Meintjes wrote: This breaks some of the tools we're developing, although in our case it's harder to fix because it's Javascript we're inserting. I understand the security concerns though. Any advice on a more secure way to allow particular content? Perhaps a whitelist of allowed scripts? On Wed, Feb 1, 2012 at 23:58, Cory Spencer cspen...@sprocket.org wrote: On 2012-02-01, at 1:33 PM, Dannon Baker wrote: With Galaxy's toolbox at hand you could generate invalid HTML from plain text components. A simple example, but consider the following: Upload one plain text file with the content: script Change the type of this dataset to html and there's your attack. If you tried to upload this, we'd interpret it as malicious HTML and discard it. As separate datasets, it's impossible to tell. Given Galaxy's powerful text manipulation tools you could write just about whatever you wanted using Galaxy itself and get it in the system as a (seemingly) valid tool-generated dataset. Now, with the outbound sanitation on any dataset served as text/html it doesn't matter and it gets handled prior to serving. Okay, I follow you there. That's a good example, thank you! Another option we discussed would be to trust all tool generated HTML, disallow changing the datatype of anything *to* html, and so on, but that approach comes with its own problems. In the case of the tool we're working on, this option is probably what would have worked best. If anything, would it be possible to make this sort of sanitization controllable via a configuration file option? I'm rather hesitant to put in a disable option for a security feature, though you're more than welcome to pop those two lines out of your instance. I think the best path forward is probably relaxing the filter a bit, the initial pass was somewhat draconian. Would relaxing the filter to allow style content to pass through work for your needs? Yes, we've already commented it out for the time being. :) Relaxing the filter would be a good improvement so far as we're concerned. I'd be happy to keep in contact with you during the process so that we can find the happy middle ground between security and usability. Thanks again! Cory ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Computational Biology Group University of Cape Town South Africa ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [API] allow_user_impersonation
Hi L-A, This exists. See your universe_wsgi for the following lines: # Optional list of email addresses of API users who can make calls on behalf of # other users #api_allow_run_as = None And then with any api call from a user in the allow list above you can add an extra parameter to the request, 'run_as' with the value being the encoded user ID you wish to impersonate. Good luck! Dannon On Feb 28, 2012, at 10:18 AM, Louise-Amélie Schmitt wrote: Hello everyone! I just wanted to know if the user impersonation will be available through the API someday :) It could be veeery useful for triggering automatic QA on the data without having to share the resulting histories afterwards. Best regards, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Issues importing a shared workflow
How did the user share the workflow with you, via link or directly with your user? And which import option are you using? Does Clone in the workflow context menu work? -Dannon On Feb 24, 2012, at 3:25 PM, Iry Witham wrote: I have a Galaxy user who has created a workflow from a history and share it with me. However, when I attempt to view and import the workflow I get the following error: The owner of this workflow has disabled imports via this link. You can return to the previous page How do I fix this issue? Are there settings that need to be modified on the users side? Thanks Iry Witham Scientific Applications Administrator Scientific Computing Group Computational Sciences Dept. The Jackson Laboratory 600 Main Street Bar Harbor, ME 04609 Phone: 207-288-6744 email: iry.wit...@jax.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?
Peter has it right in that we need to do this internally to ensure functionality across a range of job runners. A side benefit is that it gives us direct access to the tasks so that we can eventually do interesting things with scheduling, resubmission, feedback, etc. If the overhead looks to be a performance issue I could see having an override that would allow pushing task scheduling to the underlying cluster, but that functionality would come later. -Dannon On Feb 20, 2012, at 3:13 AM, Peter Cock wrote: On Mon, Feb 20, 2012 at 8:08 AM, Bram Slabbinck br...@psb.vib-ugent.be wrote: Hi Dannon, If I may further elaborate on this issue, I would like to mention that this kind of functionality is also supported by the Sun Grid Engine in the form of 'array jobs'. With this functionality you can execute a job multiple times in an independent way, only differing for instance in the parameter settings. From your description below, it seems similar to the Galaxy parallelism tag. Is there or do you foresee any implementation of this SGE functionality through the drmaa interface in Galaxy? If not, is there anybody who has achieved this through some custom coding? We would be highly interested in this. thanks Bram I was wondering about why Galaxy submits N separate jobs to SGE after splitting (identical bar their working directory). I'm not sure if all the other cluster back ends supported can do this, but basic dependencies is possible using SGE. That means the cluster could take care of scheduling the split jobs, the N processing jobs, and the final merge job (i.e. three stages where for example it won't do the merge till all the N processing jobs are finished). My hunch is Galaxy is doing a lot of this 'housekeeping' internally in order to remain flexible regarding the cluster back end. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?
On Feb 16, 2012, at 5:15 AM, Peter Cock wrote: On Wed, Feb 15, 2012 at 6:07 PM, Dannon Baker dannonba...@me.com wrote: Main still runs these jobs in the standard non-split fashion, and as a resource that is occasionally saturated (and thus doesn't necessarily have extra resources to parallelize to) will probably continue doing so as long as there's significant overhead involved in splitting the files. Fancy scheduling could minimize the issue, but as it is during heavy load you would actually have lower total throughput due to the splitting overhead. Because the splitting (currently) happens on the main server? No, because the splitting process is work which has to happen somewhere. Ignoring possible benefits from things that haven't been implemented yet, in a situation where your cluster is saturated with work you are unable to take advantage of the parallelism and splitting files apart is only adding more work, reducing total job throughput. That splitting always happens on the head node is not ideal, and needs to be configurable. I have a fork somewhere that attempts to address this but it needs work. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?
Very cool, I'll check it out! The addition of the JSON files is indeed very new and was likely unfinished with respect to the base splitter. -Dannon On Feb 16, 2012, at 1:24 PM, Peter Cock wrote: On Thu, Feb 16, 2012 at 4:28 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Dan, I think I need a little more advice - what is the role of the script scripts/extract_dataset_part.py and the JSON files created when splitting FASTQ files in lib/galaxy/datatypes/sequence.py, and then used by the class' process_split_file method? Why is there no JSON file created by the base data class in lib/galaxy/datatypes/data.py and no method process_split_file? Is the JSON thing part of a partial and unfinished rewrite of the splitter code? On the assumption that not all splitters bother with the JSON, I am trying a little hack to scripts/extract_dataset_part.py to abort silently if there is no JSON file: https://bitbucket.org/peterjc/galaxy-central/changeset/ebe94a2c25c3 This seems to be working with my current attempt at a FASTA splitter (not checked in yes, only partly implemented and tested). I've checked in my FASTA splitting, which now seems to be working OK with my BLAST tests. So far this only does splitting into chunks of the requested number of sequences, rather than the option to split the whole file into a given number of pieces. https://bitbucket.org/peterjc/galaxy-central/changeset/416c961c0da9 I also need to look at merging multiple BLAST XML outputs, but this is looking promising. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] While executing tool galaxy giving this error
Hi,
This is not a tool from our distribution. Please send questions about local
instances, core galaxy development, or your own tool development to the
galaxy-dev@lists.bx.psu.edu mailing list and *not* this galaxy-user mailing
list.
That said, if you could provide more information about what's going on here
(structure of getsitesnotoverlap_tf, etc) I might be able to give a few
pointers.
Thanks!
-Dannon
On Feb 15, 2012, at 3:28 AM, La Chi wrote:
hi,
will you please help me how can i fix this error'
thanks
Traceback (most recent call last):
File /galaxy-dist/tools/ova/getsitesnotoverlap_tf.py, line 60, in ?
if __name__ == __main__: __main__()
File /galaxy-dist/tools/ova/getsitesnotoverlap_tf.py, line 50, in __main__
result_set = connection.execute( select * from getsitesnotoverlap_tf (
+ userid + , + tfid + , + bpover + , + ch + , + sf +) )
File
/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/engine/base.py,
line 824, in execute
return Connection.executors[c](self, object, multiparams, params)
File
/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/engine/base.py,
line 888, in _execute_text
return self.__execute_context(context)
File
/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/engine/base.py,
line 896, in __execute_context
self._cursor_execute(context.cursor, context.statement,
context.parameters[0], context=context)
File
/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/engine/base.py,
line 950, in _cursor_execute
self._handle_dbapi_exception(e, statement, parameters, cursor, context)
File
/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.4.egg/sqlalchemy/engine/base.py,
line 931, in _handle_dbapi_exception
raise exc.DBAPIError.instance(statement, parameters, e,
connection_invalidated=is_disconnect)
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column chr1 does not
exist
LINE 1: select * from getsitesnotoverlap_tf (6,149,-1000,chr1,tf)
^
'select * from getsitesnotoverlap_tf (6,149,-1000,chr1,tf)' {}
___
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org. Please keep all replies on the list by
using reply all in your mail client. For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists,
please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?
It's definitely an experimental feature at this point, and there's no wiki, but basic support for breaking jobs into tasks does exist. It needs a lot more work and can go in a few different directions to make it better, but check out the wrappers with parallelism defined, and enable use_tasked_jobs in your universe_wsgi.ini and restart. That's all it should take from a fresh galaxy install to get, iirc, at least BWA and a few other tools working. If you want a super trivial example to play with, change the tool .xml for text tool like change case to have parallelism method=basic/parallelism and give that a shot. If you decide to try this out, do keep in mind that this feature is not at all complete and while there's a long list of things we still want to experiment with along these lines suggestions (and especially contributions) are absolutely welcome. -Dannon On Feb 15, 2012, at 11:36 AM, Peter Cock wrote: Hi all, The comments on this issue suggest that the Galaxy team is/were working on splitting large jobs over multiple nodes/CPUs: https://bitbucket.org/galaxy/galaxy-central/issue/79/split-large-jobs Is there any relevant page on the wiki I should be aware of? Specifically I am hoping for a general framework where one of the tool inputs can be marked as embarrassingly parallel meaning it can be subdivided easily (e.g. multiple sequences in FASTA or FASTQ format, multiple annotations in BED format, multiple lines in tabular format) and the outputs can all be easily combined (e.g. by concatenation in the same order as the input was split). Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Splitting large jobs over multiple nodes/CPUs?
Are those four tools being used on Galaxy Main already with this basic parallelism in place?Main still runs these jobs in the standard non-split fashion, and as a resource that is occasionally saturated (and thus doesn't necessarily have extra resources to parallelize to) willprobablycontinue doing so as long as there's significant overhead involved in splitting the files. Fancy scheduling could minimize the issue, but as it is during heavy load you would actually have lower total throughput due to the splitting overhead.Looking at the code in lib/galaxy/jobs/splitters/basic.py its comments suggest it only works on tools with one input and one output file (although that seems a bit fuzzy as you could be using BWA with a FASTA history item as the reference - would that fail?).I haven't tried it, but probably. I see also interesting things in lib/galaxy/jobs/splitters/multi.py Is that even more experimental? It looks like it could be used to say BWA's read file was to be split, but the reference file shared.Yes. Regarding the merging of the out, I see there is a default merge method in lib/galaxy/datatypes/data.py which just concatenates the files. I am surprised at that - it seems like a very bad idea in general - consider many binary files, or XML. Why not put this as the default for text and subclasses thereof?I can't think of a better reasonable default behavior for "Data", though you're obviously right that each datatype subclass will need to define particular behaviors for merging files.OK then, I hope to have a play with this shortly.Good luck, let me know how it goes, and again - contributions are certainly welcome :)-Dannon___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy workflow data
Hi Paul, Thanks for this suggestion, it would definitely make sense to create a reference to the actual workflow inputs in the new history. I'll see what I can do! -Dannon On Feb 14, 2012, at 11:05 AM, Paul Gordon wrote: Hi all, I have noticed that when I run a workflow with output to a new history, the data sets used as input to the workflow do not become part of the new history. Unfortunately, if a job fails in the workflow, the rerun interface for the job can't access the original data…it needs to be manually imported. While this is a minor annoyance, it becomes quite the pain if you ran the workflow for a bunch of input files. Can I put in a request that input data be imported automatically into workflow run histories, or have I missed something in the workflow run options? Cheers, Paul ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
