[galaxy-dev] PATH and local jobs

[Ryan Davis]

Hi,

 

We have a local server running Galaxy and a majority of the tools work
correctly (bowtie, BWA, samtools and etc.).  I am having issues with some of
the FASTX-Toolkit tools.  I am starting Galaxy with an init script on Centos
6 and the tools are in my $PATH and work at the command line, just not in
Galaxy.  The wiki suggest to "set $PATH in the startup file or use." 

It leaves me hanging and don't know what the other option is.  Can someone
share with me their workaround?

 

Here is the wiki page:
http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies

 

Here is the section I am hoping will help:

 


Local Jobs


Changes to $PATH can be persisted by setting them in your shell's startup
file(s).  This typically means ~/.bash_profile for bash, but please see the

bash documentation on startup files or the INVOCATION section of the bash(1)
man page to understand the intricacies of how that file is read.  Of
particular importance, if you are starting Galaxy in a method other than
manual invocation from a shell prompt (with sh run.sh) such as with an init
script, it is likely that your startup file will not be read.  In this
instance, you should set $PATH in the startup file or use.

 

 

Thanks!

Ryan

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problem with installing MAF cached dataset

[Daniel Blankenberg]

Hi Leon,

Can you see if there is any info reported reported by running the MAF 
consistency checking script at scripts/tools/maf/check_loc_file.py 
path/to/maf_index.loc?

You'll need to make sure that required dependencies (e.g. galaxy_root/lib and 
galaxy_root/eggs) are included in your PYTHONPATH.


As far as cutting down a MAF to only contain the set of desired species and 
depending upon your needs, you can use the tools included with Galaxy, or 
alternatively the maf_thread_for_species.py script in bx-python.



Thanks for using Galaxy,


Dan


On Sep 11, 2013, at 2:10 PM, Leon Mei wrote:

> Hi Dan,
> 
> Here are the output:
> ==
> hmei@ubuntu:~$ ls -lah /mnt/galaxyIndices/alignments/maf/
> total 236G
> drwxrwsr-x 2 postgres postgres 4.0K Sep 10 00:08 .
> drwxrwsr-x 4 postgres postgres 4.0K Sep  9 16:27 ..
> -rw-rw-r-- 1 postgres postgres  13G Oct 31  2009 chr10.maf
> -rw-rw-r-- 1 postgres postgres 189M Sep  9 17:29 chr10.maf.index
> -rw-rw-r-- 1 postgres postgres 189M Sep  8 21:35 chr11_gl000202_random.maf
> -rw-rw-r-- 1 postgres postgres  12G Oct 31  2009 chr11.maf
> -rw-rw-r-- 1 postgres postgres 184M Sep  9 18:07 chr11.maf.index
> -rw-rw-r-- 1 postgres postgres  12G Oct 31  2009 chr12.maf
> -rw-rw-r-- 1 postgres postgres 186M Sep  9 18:31 chr12.maf.index
> -rw-rw-r-- 1 postgres postgres 8.7G Oct 31  2009 chr13.maf
> -rw-rw-r-- 1 postgres postgres 135M Sep  9 18:49 chr13.maf.index
> -rw-rw-r-- 1 postgres postgres 8.3G Oct 31  2009 chr14.maf
> -rw-rw-r-- 1 postgres postgres 125M Sep  9 19:05 chr14.maf.index
> -rw-rw-r-- 1 postgres postgres 7.3G Oct 31  2009 chr15.maf
> -rw-rw-r-- 1 postgres postgres 112M Sep  9 19:19 chr15.maf.index
> -rw-rw-r-- 1 postgres postgres 7.1G Oct 31  2009 chr16.maf
> -rw-rw-r-- 1 postgres postgres 108M Sep  9 19:32 chr16.maf.index
> -rw-rw-r-- 1 postgres postgres  39M Oct 31  2009 chr17_ctg5_hap1.maf
> -rw-rw-r-- 1 postgres postgres 456K Sep  9 19:45 chr17_ctg5_hap1.maf.index
> -rw-rw-r-- 1 postgres postgres 753K Oct 31  2009 chr17_gl000203_random.maf
> -rw-rw-r-- 1 postgres postgres  45K Sep  9 19:45 
> chr17_gl000203_random.maf.index
> -rw-rw-r-- 1 postgres postgres 5.1M Oct 31  2009 chr17_gl000204_random.maf
> -rw-rw-r-- 1 postgres postgres 101K Sep  9 19:45 
> chr17_gl000204_random.maf.index
> -rw-rw-r-- 1 postgres postgres 1.6M Oct 31  2009 chr17_gl000205_random.maf
> -rw-rw-r-- 1 postgres postgres  33K Sep  9 19:45 
> chr17_gl000205_random.maf.index
> -rw-rw-r-- 1 postgres postgres 968K Oct 31  2009 chr17_gl000206_random.maf
> -rw-rw-r-- 1 postgres postgres  24K Sep  9 19:45 
> chr17_gl000206_random.maf.index
> -rw-rw-r-- 1 postgres postgres 7.3G Oct 31  2009 chr17.maf
> -rw-rw-r-- 1 postgres postgres 112M Sep  9 19:45 chr17.maf.index
> -rw-rw-r-- 1 postgres postgres 265K Oct 31  2009 chr18_gl000207_random.maf
> -rw-rw-r-- 1 postgres postgres 3.5K Sep  9 19:59 
> chr18_gl000207_random.maf.index
> -rw-rw-r-- 1 postgres postgres 7.0G Oct 31  2009 chr18.maf
> -rw-rw-r-- 1 postgres postgres 106M Sep  9 19:59 chr18.maf.index
> -rw-rw-r-- 1 postgres postgres 4.2K Oct 31  2009 chr19_gl000208_random.maf
> -rw-rw-r-- 1 postgres postgres  193 Sep  9 20:05 
> chr19_gl000208_random.maf.index
> -rw-rw-r-- 1 postgres postgres 2.6M Oct 31  2009 chr19_gl000209_random.maf
> -rw-rw-r-- 1 postgres postgres  54K Sep  9 20:05 
> chr19_gl000209_random.maf.index
> -rw-rw-r-- 1 postgres postgres 3.5G Oct 31  2009 chr19.maf
> -rw-rw-r-- 1 postgres postgres  48M Sep  9 20:05 chr19.maf.index
> -rw-rw-r-- 1 postgres postgres 4.1M Oct 31  2009 chr1_gl000191_random.maf
> -rw-rw-r-- 1 postgres postgres 108K Sep  9 20:05 
> chr1_gl000191_random.maf.index
> -rw-rw-r-- 1 postgres postgres  22M Oct 31  2009 chr1_gl000192_random.maf
> -rw-rw-r-- 1 postgres postgres 376K Sep  9 20:06 
> chr1_gl000192_random.maf.index
> -rw-rw-r-- 1 postgres postgres  21G Oct 31  2009 chr1.maf
> -rw-rw-r-- 1 postgres postgres 330M Sep  9 17:13 chr1.maf.index
> -rw-rw-r-- 1 postgres postgres 5.6G Oct 31  2009 chr20.maf
> -rw-rw-r-- 1 postgres postgres  83M Sep  9 20:59 chr20.maf.index
> -rw-rw-r-- 1 postgres postgres 342K Oct 31  2009 chr21_gl000210_random.maf
> -rw-rw-r-- 1 postgres postgres  31K Sep  9 21:05 
> chr21_gl000210_random.maf.index
> -rw-rw-r-- 1 postgres postgres 2.9G Oct 31  2009 chr21.maf
> -rw-rw-r-- 1 postgres postgres  38M Sep  9 21:05 chr21.maf.index
> -rw-rw-r-- 1 postgres postgres 2.8G Oct 31  2009 chr22.maf
> -rw-rw-r-- 1 postgres postgres  37M Sep  9 21:10 chr22.maf.index
> -rw-rw-r-- 1 postgres postgres  23G Oct 31  2009 chr2.maf
> -rw-rw-r-- 1 postgres postgres 355M Sep  9 20:49 chr2.maf.index
> -rw-rw-r-- 1 postgres postgres  19G Oct 31  2009 chr3.maf
> -rw-rw-r-- 1 postgres postgres 291M Sep  9 21:45 chr3.maf.index
> -rw-rw-r-- 1 postgres postgres 9.1M Oct 31  2009 chr4_ctg9_hap1.maf
> -rw-rw-r-- 1 postgres postgres 150K Sep  9 22:16 chr4_ctg9_hap1.maf.index
> -rw-rw-r-- 1 postgres postgres 4.8M Oct 31  2009 chr4_gl000193_random.maf
> -rw-rw-r-- 1 postgres postgres 101K Sep  9 22:16 
> chr4_

Re: [galaxy-dev] Stable Galaxy for local install

[cngbgal...@gmail.com]

Hi all,
What I want to do in a hurry is to fix the problem of "Select" in "Filter and 
Sort" .It is a step very important for me to Sort SAM file for Cufflinks.Thank 
you very much for all of your help.




cngbgalaxy

From: Adam Brenner
Date: 2013-09-12 00:41
To: cngbgalaxy
CC: joachim.jacob; galaxy-dev@lists.bx.psu.edu; galaxy-...@bx.psu.edu
Subject: Re: [galaxy-dev] Stable Galaxy for local install
Galaxy development occurs in two repositories, galaxy-central and
galaxy-dist. Changes are stabilized in galaxy-central and then pushed
to galaxy-dist roughly once a month.[1]

So

$ hg clone https://bitbucket.org/galaxy/galaxy-dist/
$ cd galaxy-dist
$ hg update stable

[1]: http://wiki.galaxyproject.org/Admin/Get%20Galaxy

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

Research Computing Support
Office of Information Technology
http://www.oit.uci.edu/rcs/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
aebre...@uci.edu


On Wed, Sep 11, 2013 at 8:42 AM, cngbgalaxy  wrote:
> Hi Joachim,
> Thank you very much for your help.
> Actually I find some problems about such versions of Galaxy in
> https://bitbucket.org/galaxy/galaxy-dist/downloads .
> We can not view the data in history by clicking the eyes icon on the
> c42567f43aa7 version.The "Select" in "Filter and Sort" fail to work on the
> 1ae95b3aa98d (release_2013.08.12 ) version and the 524f246ca853
> (release_2013.06.03 ),though I don't know whether it is a bug or not.
> As a result,I feel confused about which version of Galaxy to install and
> avoid the above problems.
>
> 
> cngbgalaxy
>
> From: Joachim Jacob | VIB |
> Date: 2013-09-11 22:38
> To: test galaxy
> CC: galaxy-dev; galaxy-dev
> Subject: Re: [galaxy-dev] Stable Galaxy for local install
> The latest code from the 'stable' branch of Galaxy is preferred. See the
> news brief accompanying the update (release_2013.08.12) on
> http://wiki.galaxyproject.org/DevNewsBriefs/2013_08_12
>
> You need mercurial (http://mercurial.selenic.com/) for installing
> Galaxy. Report bugs at https://trello.com/b/75c1kASa/galaxy-development
>
> Cheers
> Joachim
>
> Joachim Jacob
> Contact details: http://www.bits.vib.be/index.php/about/80-team
>
>
> On 09/11/2013 03:55 AM, test galaxy wrote:
>> Hi everyone,
>> I want to install my local Galaxy with less bugs,and I don't know
>> which version is more stable and work well.Could anyone give same
>> advise?Thank you very much.
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>http://galaxyproject.org/search/mailinglists/
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problem with installing MAF cached dataset

[Leon Mei]

Hi Dan,

Here are the output:
==
hmei@ubuntu:~$ ls -lah /mnt/galaxyIndices/alignments/maf/
total 236G
drwxrwsr-x 2 postgres postgres 4.0K Sep 10 00:08 .
drwxrwsr-x 4 postgres postgres 4.0K Sep  9 16:27 ..
-rw-rw-r-- 1 postgres postgres  13G Oct 31  2009 chr10.maf
-rw-rw-r-- 1 postgres postgres 189M Sep  9 17:29 chr10.maf.index
-rw-rw-r-- 1 postgres postgres 189M Sep  8 21:35 chr11_gl000202_random.maf
-rw-rw-r-- 1 postgres postgres  12G Oct 31  2009 chr11.maf
-rw-rw-r-- 1 postgres postgres 184M Sep  9 18:07 chr11.maf.index
-rw-rw-r-- 1 postgres postgres  12G Oct 31  2009 chr12.maf
-rw-rw-r-- 1 postgres postgres 186M Sep  9 18:31 chr12.maf.index
-rw-rw-r-- 1 postgres postgres 8.7G Oct 31  2009 chr13.maf
-rw-rw-r-- 1 postgres postgres 135M Sep  9 18:49 chr13.maf.index
-rw-rw-r-- 1 postgres postgres 8.3G Oct 31  2009 chr14.maf
-rw-rw-r-- 1 postgres postgres 125M Sep  9 19:05 chr14.maf.index
-rw-rw-r-- 1 postgres postgres 7.3G Oct 31  2009 chr15.maf
-rw-rw-r-- 1 postgres postgres 112M Sep  9 19:19 chr15.maf.index
-rw-rw-r-- 1 postgres postgres 7.1G Oct 31  2009 chr16.maf
-rw-rw-r-- 1 postgres postgres 108M Sep  9 19:32 chr16.maf.index
-rw-rw-r-- 1 postgres postgres  39M Oct 31  2009 chr17_ctg5_hap1.maf
-rw-rw-r-- 1 postgres postgres 456K Sep  9 19:45 chr17_ctg5_hap1.maf.index
-rw-rw-r-- 1 postgres postgres 753K Oct 31  2009 chr17_gl000203_random.maf
-rw-rw-r-- 1 postgres postgres  45K Sep  9 19:45 chr17_gl000203_random.maf.index
-rw-rw-r-- 1 postgres postgres 5.1M Oct 31  2009 chr17_gl000204_random.maf
-rw-rw-r-- 1 postgres postgres 101K Sep  9 19:45 chr17_gl000204_random.maf.index
-rw-rw-r-- 1 postgres postgres 1.6M Oct 31  2009 chr17_gl000205_random.maf
-rw-rw-r-- 1 postgres postgres  33K Sep  9 19:45 chr17_gl000205_random.maf.index
-rw-rw-r-- 1 postgres postgres 968K Oct 31  2009 chr17_gl000206_random.maf
-rw-rw-r-- 1 postgres postgres  24K Sep  9 19:45 chr17_gl000206_random.maf.index
-rw-rw-r-- 1 postgres postgres 7.3G Oct 31  2009 chr17.maf
-rw-rw-r-- 1 postgres postgres 112M Sep  9 19:45 chr17.maf.index
-rw-rw-r-- 1 postgres postgres 265K Oct 31  2009 chr18_gl000207_random.maf
-rw-rw-r-- 1 postgres postgres 3.5K Sep  9 19:59 chr18_gl000207_random.maf.index
-rw-rw-r-- 1 postgres postgres 7.0G Oct 31  2009 chr18.maf
-rw-rw-r-- 1 postgres postgres 106M Sep  9 19:59 chr18.maf.index
-rw-rw-r-- 1 postgres postgres 4.2K Oct 31  2009 chr19_gl000208_random.maf
-rw-rw-r-- 1 postgres postgres  193 Sep  9 20:05 chr19_gl000208_random.maf.index
-rw-rw-r-- 1 postgres postgres 2.6M Oct 31  2009 chr19_gl000209_random.maf
-rw-rw-r-- 1 postgres postgres  54K Sep  9 20:05 chr19_gl000209_random.maf.index
-rw-rw-r-- 1 postgres postgres 3.5G Oct 31  2009 chr19.maf
-rw-rw-r-- 1 postgres postgres  48M Sep  9 20:05 chr19.maf.index
-rw-rw-r-- 1 postgres postgres 4.1M Oct 31  2009 chr1_gl000191_random.maf
-rw-rw-r-- 1 postgres postgres 108K Sep  9 20:05 chr1_gl000191_random.maf.index
-rw-rw-r-- 1 postgres postgres  22M Oct 31  2009 chr1_gl000192_random.maf
-rw-rw-r-- 1 postgres postgres 376K Sep  9 20:06 chr1_gl000192_random.maf.index
-rw-rw-r-- 1 postgres postgres  21G Oct 31  2009 chr1.maf
-rw-rw-r-- 1 postgres postgres 330M Sep  9 17:13 chr1.maf.index
-rw-rw-r-- 1 postgres postgres 5.6G Oct 31  2009 chr20.maf
-rw-rw-r-- 1 postgres postgres  83M Sep  9 20:59 chr20.maf.index
-rw-rw-r-- 1 postgres postgres 342K Oct 31  2009 chr21_gl000210_random.maf
-rw-rw-r-- 1 postgres postgres  31K Sep  9 21:05 chr21_gl000210_random.maf.index
-rw-rw-r-- 1 postgres postgres 2.9G Oct 31  2009 chr21.maf
-rw-rw-r-- 1 postgres postgres  38M Sep  9 21:05 chr21.maf.index
-rw-rw-r-- 1 postgres postgres 2.8G Oct 31  2009 chr22.maf
-rw-rw-r-- 1 postgres postgres  37M Sep  9 21:10 chr22.maf.index
-rw-rw-r-- 1 postgres postgres  23G Oct 31  2009 chr2.maf
-rw-rw-r-- 1 postgres postgres 355M Sep  9 20:49 chr2.maf.index
-rw-rw-r-- 1 postgres postgres  19G Oct 31  2009 chr3.maf
-rw-rw-r-- 1 postgres postgres 291M Sep  9 21:45 chr3.maf.index
-rw-rw-r-- 1 postgres postgres 9.1M Oct 31  2009 chr4_ctg9_hap1.maf
-rw-rw-r-- 1 postgres postgres 150K Sep  9 22:16 chr4_ctg9_hap1.maf.index
-rw-rw-r-- 1 postgres postgres 4.8M Oct 31  2009 chr4_gl000193_random.maf
-rw-rw-r-- 1 postgres postgres 101K Sep  9 22:16 chr4_gl000193_random.maf.index
-rw-rw-r-- 1 postgres postgres 2.2M Oct 31  2009 chr4_gl000194_random.maf
-rw-rw-r-- 1 postgres postgres 112K Sep  9 22:16 chr4_gl000194_random.maf.index
-rw-rw-r-- 1 postgres postgres  17G Oct 31  2009 chr4.maf
-rw-rw-r-- 1 postgres postgres 257M Sep  9 22:16 chr4.maf.index
-rw-rw-r-- 1 postgres postgres  17G Oct 31  2009 chr5.maf
-rw-rw-r-- 1 postgres postgres 255M Sep  9 22:47 chr5.maf.index
-rw-rw-r-- 1 postgres postgres  44M Oct 31  2009 chr6_apd_hap1.maf
-rw-rw-r-- 1 postgres postgres 496K Sep  9 23:18 chr6_apd_hap1.maf.index
-rw-rw-r-- 1 postgres postgres  96M Oct 31  2009 chr6_cox_hap2.maf
-rw-rw-r-- 1 postgres postgres 1.1M Sep  9 23:18 chr6_cox_hap2.maf.index
-rw-rw-r-- 1 postgres postgres  86M Oct 31  2009 chr6_dbb_hap3.maf

Re: [galaxy-dev] PATH and local jobs

[Guest, Simon]

> From: galaxy-dev-bounces ... On Behalf Of Adam Brenner
> On our HPC setup, we use the modules system but its the same issue you are
> running into.

Another vote for the excellent modules system from us at AgResearch.  I think 
it's the only sane way to manage PATH, etc, when you have multiple versions of 
tools installed.
http://modules.sourceforge.net/
I know I've said this before here.  Tell all your friends. ;-)
 
> A number of methods exists for setting up the correct path, for you, I
> believe the easiest would be one of the two:
> 
> 
> 1) In your init script: source (or setup) PATHS as you have them in your
> .bash_profile.
> /etc/init.d/galaxy
> 
> OR
> 
> 2) Within Galaxy, edit universe_wsgi.ini and look for:
> environment_setup_file
> and create a file that will setup the correct $PATH for you.


We use (2), which is better than (1), as it will continue to work when you 
start running your jobs on a cluster, whereas the PATH you set in (1) will not 
be picked up on the cluster (as far as I remember from our early experience).

cheers,
Simon


===
Attention: The information contained in this message and/or attachments
from AgResearch Limited is intended only for the persons or entities
to which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipients is prohibited by AgResearch
Limited. If you have received this message in error, please notify the
sender immediately.
===

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Creating Galaxy Admin / Local Install Mailing List - Updates?

[Adam Brenner]

Howdy,

I know this has been discussed before, but I can not find the email
threads on this...but what was the status on creating a galaxy admin /
local install mailing list?

The Devel list is very chatty and while my filters do a good job, I
think it will be easier for others to find help on their own if it was
on a separate mailing list It would be easier to search and look
back at previous threads.

However, the added benefit of having the devel list is that we get
feedback from Galaxy Developers

As the saying godes: Just food for thought,
-Adam

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

Research Computing Support
Office of Information Technology
http://www.oit.uci.edu/rcs/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
aebre...@uci.edu
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Stable Galaxy for local install

[Adam Brenner]

> What I want to do in a hurry is to fix the problem of "Select" in "Filter
> and Sort" .It is a step very important for me to Sort SAM file for
> Cufflinks.Thank you very much for all of your help.

Per my last email:.

$ hg clone https://bitbucket.org/galaxy/galaxy-dist/
$ cd galaxy-dist
$ hg update stable


If you are running into issues, can you be more specific...provide
error message? Steps on how to reproduce your issue?

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

Research Computing Support
Office of Information Technology
http://www.oit.uci.edu/rcs/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
aebre...@uci.edu


On Wed, Sep 11, 2013 at 6:58 PM, cngbgal...@gmail.com
 wrote:
> Hi all,
> What I want to do in a hurry is to fix the problem of "Select" in "Filter
> and Sort" .It is a step very important for me to Sort SAM file for
> Cufflinks.Thank you very much for all of your help.
>
> 
> cngbgalaxy
>
> From: Adam Brenner
> Date: 2013-09-12 00:41
> To: cngbgalaxy
> CC: joachim.jacob; galaxy-dev@lists.bx.psu.edu; galaxy-...@bx.psu.edu
> Subject: Re: [galaxy-dev] Stable Galaxy for local install
> Galaxy development occurs in two repositories, galaxy-central and
> galaxy-dist. Changes are stabilized in galaxy-central and then pushed
> to galaxy-dist roughly once a month.[1]
>
> So
>
> $ hg clone https://bitbucket.org/galaxy/galaxy-dist/
> $ cd galaxy-dist
> $ hg update stable
>
> [1]: http://wiki.galaxyproject.org/Admin/Get%20Galaxy
>
> --
> Adam Brenner
> Computer Science, Undergraduate Student
> Donald Bren School of Information and Computer Sciences
>
> Research Computing Support
> Office of Information Technology
> http://www.oit.uci.edu/rcs/
>
> University of California, Irvine
> www.ics.uci.edu/~aebrenne/
> aebre...@uci.edu
>
>
> On Wed, Sep 11, 2013 at 8:42 AM, cngbgalaxy  wrote:
>> Hi Joachim,
>> Thank you very much for your help.
>> Actually I find some problems about such versions of Galaxy in
>> https://bitbucket.org/galaxy/galaxy-dist/downloads .
>> We can not view the data in history by clicking the eyes icon on the
>> c42567f43aa7 version.The "Select" in "Filter and Sort" fail to work on the
>> 1ae95b3aa98d (release_2013.08.12 ) version and the 524f246ca853
>> (release_2013.06.03 ),though I don't know whether it is a bug or not.
>> As a result,I feel confused about which version of Galaxy to install and
>> avoid the above problems.
>>
>> 
>> cngbgalaxy
>>
>> From: Joachim Jacob | VIB |
>> Date: 2013-09-11 22:38
>> To: test galaxy
>> CC: galaxy-dev; galaxy-dev
>> Subject: Re: [galaxy-dev] Stable Galaxy for local install
>> The latest code from the 'stable' branch of Galaxy is preferred. See the
>> news brief accompanying the update (release_2013.08.12) on
>> http://wiki.galaxyproject.org/DevNewsBriefs/2013_08_12
>>
>> You need mercurial (http://mercurial.selenic.com/) for installing
>> Galaxy. Report bugs at https://trello.com/b/75c1kASa/galaxy-development
>>
>> Cheers
>> Joachim
>>
>> Joachim Jacob
>> Contact details: http://www.bits.vib.be/index.php/about/80-team
>>
>>
>> On 09/11/2013 03:55 AM, test galaxy wrote:
>>> Hi everyone,
>>> I want to install my local Galaxy with less bugs,and I don't know
>>> which version is more stable and work well.Could anyone give same
>>> advise?Thank you very much.
>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>http://lists.bx.psu.edu/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>http://galaxyproject.org/search/mailinglists/
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Stable Galaxy for local install

[Adam Brenner]

Galaxy development occurs in two repositories, galaxy-central and
galaxy-dist. Changes are stabilized in galaxy-central and then pushed
to galaxy-dist roughly once a month.[1]

So

$ hg clone https://bitbucket.org/galaxy/galaxy-dist/
$ cd galaxy-dist
$ hg update stable

[1]: http://wiki.galaxyproject.org/Admin/Get%20Galaxy

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

Research Computing Support
Office of Information Technology
http://www.oit.uci.edu/rcs/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
aebre...@uci.edu


On Wed, Sep 11, 2013 at 8:42 AM, cngbgalaxy  wrote:
> Hi Joachim,
> Thank you very much for your help.
> Actually I find some problems about such versions of Galaxy in
> https://bitbucket.org/galaxy/galaxy-dist/downloads .
> We can not view the data in history by clicking the eyes icon on the
> c42567f43aa7 version.The "Select" in "Filter and Sort" fail to work on the
> 1ae95b3aa98d (release_2013.08.12 ) version and the 524f246ca853
> (release_2013.06.03 ),though I don't know whether it is a bug or not.
> As a result,I feel confused about which version of Galaxy to install and
> avoid the above problems.
>
> 
> cngbgalaxy
>
> From: Joachim Jacob | VIB |
> Date: 2013-09-11 22:38
> To: test galaxy
> CC: galaxy-dev; galaxy-dev
> Subject: Re: [galaxy-dev] Stable Galaxy for local install
> The latest code from the 'stable' branch of Galaxy is preferred. See the
> news brief accompanying the update (release_2013.08.12) on
> http://wiki.galaxyproject.org/DevNewsBriefs/2013_08_12
>
> You need mercurial (http://mercurial.selenic.com/) for installing
> Galaxy. Report bugs at https://trello.com/b/75c1kASa/galaxy-development
>
> Cheers
> Joachim
>
> Joachim Jacob
> Contact details: http://www.bits.vib.be/index.php/about/80-team
>
>
> On 09/11/2013 03:55 AM, test galaxy wrote:
>> Hi everyone,
>> I want to install my local Galaxy with less bugs,and I don't know
>> which version is more stable and work well.Could anyone give same
>> advise?Thank you very much.
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>http://galaxyproject.org/search/mailinglists/
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Deseq2 wrapper question

[ruiwang.sz]

Hey Guys,

It's great to receive so many responses! Thank you all for the
information...
and let me check out each of the masterpiece. :-)

Best,
Rui


On Wed, Sep 11, 2013 at 1:56 AM, Peter Cock wrote:

> Wow - that makes at least five Deseq2 wrappers for Galaxy
> available or in progress :(
>
> Bjoern's wrapper on the (test) tool shed,
> http://testtoolshed.g2.bx.psu.edu/view/bgruening/deseq2
>
> Ross' combined wrapper for edgeR, DESeq2 and voom in one tool:
> http://testtoolshed.g2.bx.psu.edu/view/fubar/differential_count_models
>
> Joachim's wrapper on the Tool Shed
> http://toolshed.bits.vib.be/view/joachim/deseq2
>
> Vipin's which is soon to be released to the (Test?) Tool Shed,
> https://github.com/ratschlab/oqtans_tools/tree/master/DESeq2
>
> And Rui's is working on one too.
>
> This does seem like duplicated effort & a source of confusion
> for end users and Galaxy administrators (a problem not unique
> to deseq2, but affecting many Galaxy wrappers).
>
> I appreciate there will be different needs, and one wrapper
> may not suit all, but I would prefer if the default behaviour
> for Galaxy tool wrapper authors was to collaborate on one
> good wrapper rather than writing competing ones.
>
> When I started work on a new wrapper I tried to announce
> this on the mailing list to find out if anyone else was already
> tackling the same tool - and that seemed to work quite well.
> Of course, the volume of emails on galaxy-dev has grown
> quite a lot over the last few years so that may not be as
> effective, but the archives should be searchable.
>
> Perhaps we need to improve communication in some way?
> One option would be to encourage greater use of the Test
> Tool Shed for works in progress to give them visibility?
> You could even have the Tool Shed itself require a search
> step before creating a new repository to avoid accidental
> duplication of effort? Or maybe a wiki page of wrappers
> in progress?
>
> (And maybe we should split this into a new thread)
>
> Regards,
>
> Peter
>
>
> On Wed, Sep 11, 2013 at 8:59 AM, Joachim Jacob | VIB |
>  wrote:
> > Hi Rui,
> >
> > I should provide a 'sample metadata table', a tabular file with one
> column
> > the names of your samples, the other column the associated metadata (e.g.
> > type).
> >
> > This is how I've done it in my DESeq2 wrapper. :-) Feel free to check the
> > interface at http://toolshed.bits.vib.be/view/joachim/deseq2
> >
> >
> > Cheers,
> > Joachim
> >
> > Joachim Jacob
> > Contact details: http://www.bits.vib.be/index.php/about/80-team
> >
> >
> >
> > On 09/11/2013 01:15 AM, ruiwang.sz wrote:
> >>
> >> Hi All,
> >>
> >> we are making a wrapper for deseq2, and there is a step where
> >> control/experimental conditions
> >> need to be determined. For example, it would be a column like the
> >> following in the input file
> >>
> >> Type
> >> CTC
> >> CTC
> >> LM
> >> LM
> >> PT
> >> PT
> >>
> >> and we want the select list to contain
> >>
> >> CTC
> >> LM
> >> PT
> >>
> >> In other words, we want to filter out the 'Type' and have only distinct
> >> ones in the select list. We could
> >> make it in two steps, generate a intermediate file for this but it would
> >> be nice that we could directly
> >> retrieve the distinct types to make the list. I took a look at the tool
> >> config wiki, but didn't see anything
> >> (maybe because it was a quick scan). Does anyone have any tip on this?
> >> we'll really appreciate.
> >>
> >> Thanks,
> >> Rui
> >>
> >>
> >> ___
> >> Please keep all replies on the list by using "reply all"
> >> in your mail client.  To manage your subscriptions to this
> >> and other Galaxy lists, please use the interface at:
> >>http://lists.bx.psu.edu/
> >>
> >> To search Galaxy mailing lists use the unified search at:
> >>http://galaxyproject.org/search/mailinglists/
> >
> >
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >  http://lists.bx.psu.edu/
> >
> > To search Galaxy mailing lists use the unified search at:
> >  http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Deseq2 wrapper question

[ruiwang.sz]

Hi Joachim,

Thanks for the info! but I could not register on your instance...and thus
could not check your wrapper. :-( Please help!

Thanks,
Rui


On Wed, Sep 11, 2013 at 12:59 AM, Joachim Jacob | VIB | <
joachim.ja...@vib.be> wrote:

> Hi Rui,
>
> I should provide a 'sample metadata table', a tabular file with one column
> the names of your samples, the other column the associated metadata (e.g.
> type).
>
> This is how I've done it in my DESeq2 wrapper. :-) Feel free to check the
> interface at 
> http://toolshed.bits.vib.be/**view/joachim/deseq2
>
>
> Cheers,
> Joachim
>
> Joachim Jacob
> Contact details: 
> http://www.bits.vib.be/index.**php/about/80-team
>
>
>
> On 09/11/2013 01:15 AM, ruiwang.sz wrote:
>
>> Hi All,
>>
>> we are making a wrapper for deseq2, and there is a step where
>> control/experimental conditions
>> need to be determined. For example, it would be a column like the
>> following in the input file
>>
>> Type
>> CTC
>> CTC
>> LM
>> LM
>> PT
>> PT
>>
>> and we want the select list to contain
>>
>> CTC
>> LM
>> PT
>>
>> In other words, we want to filter out the 'Type' and have only distinct
>> ones in the select list. We could
>> make it in two steps, generate a intermediate file for this but it would
>> be nice that we could directly
>> retrieve the distinct types to make the list. I took a look at the tool
>> config wiki, but didn't see anything
>> (maybe because it was a quick scan). Does anyone have any tip on this?
>> we'll really appreciate.
>>
>> Thanks,
>> Rui
>>
>>
>> __**_
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>
>> http://galaxyproject.org/**search/mailinglists/
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Stable Galaxy for local install

[test galaxy]

Hi everyone,
I want to install my local Galaxy with less bugs,and I don't know which
version is more stable and work well.Could anyone give same advise?Thank
you very much.
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] 'DECLARE CURSOR ... FOR UPDATE/SHARE is not supported'

[Nate Coraor]

On Sep 11, 2013, at 9:16 AM, hogart wrote:

> Hi Nate,
> Yes, db schema is old, but the upgrading (manage_db.sh upgrade) to the 
> current version resulted to the error, as mentioned above. I don't remember 
> the release number of the previous version of Galaxy, but it was installed in 
> the March this year. A didn't restored the dump yet, now I am playinig with 
> the newly created database. In my newbie point of view, it seems like, that 
> dropping of the old database (drop database galaxy; as superuser) was 
> incomplete, - is it possible?
> Thanks for letting me know, I will try to upgrade the PostgeSQL (the current 
> version is default version of CentOS 5.6).


Sorry, I missed the bit where you were starting with an empty database and not 
reloading your dump of the old one.

It does seem possible that your database is not empty on startup, as the 
username column should not already exist at the time that this migration script 
runs.  When you drop and then create the database, Galaxy should create all the 
necessary tables and run through all of the migration scripts the first time 
that you start it.  Is it doing this?  If it thinks that the database is 
already at revision 13, I suspect the database is not empty when it starts up.  
Check your database_connection string in universe_wsgi.ini and make sure it is 
pointed at the correct database.

--nate
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Deseq2 wrapper question

[Jeremy Goecks]

> STAR aligner 

I have a partial STAR wrapper that may be useful: 
https://bitbucket.org/jgoecks/jeremys-code/src/d7096c132657420f08b540c6954b947f648aac80/galaxy/tool-wrappers/star.xml?at=default

Whether you use this code or not, I do have one request: please put the output 
in BAM rather than SAM format in order to save disk space.

Best,
J.___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Galaxy job status not synchronize with pbs

[Tsung-Ying Tsai]

Hi,

I found that there is a problem when pbs job terminate unexpectedly (job killed 
manually / meet walltime),
but galaxy shows the job still working.

And there are some error messages from log file below.

galaxy.jobs.runners.pbs DEBUG 2013-09-10 18:15:57,841 (3811/762.localhost) PBS 
job state changed from R to C
galaxy.jobs.runners.pbs ERROR 2013-09-10 18:15:57,841 (3811/762.localhost) PBS 
job failed: Unknown error: -10
galaxy.jobs.runners ERROR 2013-09-10 18:15:57,841 (unknown) Unhandled exception 
calling fail_job
Traceback (most recent call last):
  File 
"/mnt/RAID1/mirror/galaxy/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", 
line 58, in run_next
method(arg)
  File "/mnt/RAID1/mirror/galaxy/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", 
line 560, in fail_job
if pbs_job_state.stop_job:
AttributeError: 'AsynchronousJobState' object has no attribute 'stop_job'

It seems that galaxy had received job status from pbs, but pbs.py can't handle 
message correctly.
Is there any solution to solve this problem?

Thanks,
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] ZFS storage recommendations

[Joachim Jacob | VIB |]

Thanks Nate. Compression is currently with lzjb. Will do some testing 
with dd.


Joachim

Joachim Jacob
Contact details: http://www.bits.vib.be/index.php/about/80-team




On Wed 11 Sep 2013 04:03:55 PM CEST, Nate Coraor wrote:

Hi all,

I'm a big fan of ZFS, we have long used it behind Galaxy Main.  Some of our 
older servers are (still) Solaris, and the newest is FreeBSD.

I've lately been using SmartOS for virtualization and while it has a drawback 
as a fileserver (since currently the nfs server can only run in the global 
zone, which is not ideal on SmartOS), there are other illumos derivatives that 
would probably be great for this task (e.g. OmniOS).  Native ZFS in the OS in 
which it is developed is a win for me, especially when you are serving via 
simple NFS.  For more complex network filesystems, Linux is probably preferable.

I considered a separate ZIL and L2ARC for the latest ZFS server, but DTrace 
revealed that I probably would not see much of a performance benefit with our 
usage patterns.  The memory usage you're seeing is to be expected - it will 
pretty much consume whatever is available for caching, but it's available to be 
freed if needed for something else.

I wouldn't suggest rsync for performance testing.  I typically do things like 
timed writing of blocks read from /dev/zero using dd, so that the source 
filesystem and checksumming algorithm can be taken out of the equation.  And 
dedup/compression will of course cause a significant write penalty.  If you can 
suffer the decreased space optimization, lzjb performs significantly better 
than gzip.  gzip-1 is a nice compromise between the default gzip level and 
lzjb, as well.

--nate

On Sep 11, 2013, at 3:45 AM, Joachim Jacob | VIB | wrote:


Thank you all for the reactions.

Some details about my current ZFS and Galaxy setup:

- Galaxy runs as a single virtual machine, with currently 20 cores, 80GB RAM. 
Will be 32 cores and about 160GB RAM soon.
- The postgres database is on the virtual machine itself.
- The 'files' and 'job_working_dir' are on an NFS exported directory, hosted on 
the host machine of the guest.
- The NFS exported directory is an raidz1 dataset.
- The raidsz1 runs on 7 550GB SAS disks, which are (unfortunately) controlled 
by a RAID hardware controller, but passed as RAID0 (JBOD not available). So 
raidz1 runs on 7 RAID0 disks (with settings in the hardware controller PERC 
H700: no read ahead, write through, 8 KB stripe size, disk cache policy 
enabled).
- Compression and deduplication is enabled.
- The directory on which the zfs dataset is mounted, is exported using the 
native linux NFS daemon to the Galaxy virtual machine. The 'zfs sharenfs' did 
not work (ownerships not set correctly - perhaps need some more investigation, 
but I found several times reports about sharenfs option in ZFS in linux is not 
behaving well...).

The numbers:
- my initial files database (ext4 on RAID5) is 3.0TB in size. On ZFS, with 
compression and deduplication, this database is *1.8TB *(-40%).
- Did not yet provide a SLOG to host the ZIL and a L2ARC, since I have a 
cleared picture about the performance I can get. Would you advise preference 
over ZIL on a SLOG, or go for a SSD to host the L2ARC.
- The cost for this performance of ZFS in storage is RAM: currently 
continuously using*284GB RAM* for ZFS!
- The write and read speed is from the Galaxy VM over NFS is *~40MB/s and 
~100MB/s* (tested by simply copying over rsync - I still need to check the 
presentation and scripts of Anne Black-Ziegelbein). This is a 66% decrease in 
previously achieved write and read speed (ext4 on hardware RAID5), but I feel 
that the benefits (deduplication, backing up via snapshots, data integrity,) 
outweigh this IO performance.

(I am setting this ZFS up on a new server (well, actually 2 years old now, has 
served on another project well))

Currently our Galaxy uses this zfs with success!

For your interest, my settings on the 'galaxydb' zfs tank below. (I was 
wondering if here some more wizardry can be applied).


[root@r910bits ~]# *zfs get all tank/galaxydb*
NAME   PROPERTY  VALUE  SOURCE
tank/galaxydb  type  filesystem -
tank/galaxydb  creation  Mon Sep  9 12:44 2013  -
tank/galaxydb  used  1.81T  -
tank/galaxydb  available 1.66T  -
tank/galaxydb  referenced1.81T  -
tank/galaxydb  compressratio 1.66x  -
tank/galaxydb  mounted   yes-
tank/galaxydb  quota none default
tank/galaxydb  reservation   none default
tank/galaxydb  recordsize128K default
tank/galaxydb  mountpoint/mnt/galaxydb  local
tank/galaxydb  sharenfs  rw=@galaxy local
tank/galaxydb  checksum  on default
tank/galaxydb  compression   lzjb 

[galaxy-dev] Stable Galaxy to install.

[cngbgal...@gmail.com]

Hi everyone,
I want to install my local Galaxy,and I don't know which version is more stable 
and work well.Could anyone give same advise?Thank you very much. 




cngbgal...@gmail.com___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Difficulty dispatching toolshed tool jobs via job_conf.xml

[Guest, Simon]

> From: Nate Coraor [mailto:n...@bx.psu.edu]

> This should work, it looks like there may be a bug with handling tool shed
> IDs when determining job destinations. You are actually supposed to be
> able to use any of:
> 
>   shed_host/repos/owner/repo/tool_id/version
>   shed_host/repos/owner/repo/tool_id
>   tool_id
> 
> I'll take a look at this as soon as possible.  One thing you might try in
> the short term is using the percent encoded tool id in the  tag in
> job_conf.xml:

Hi Nate,

Using the percent encoded form was the first thing I tried (before emailing the 
list).  It didn't make any difference.  I think there's another reason it's not 
picking it up.  Thanks for looking into this.

cheers,
Simon


===
Attention: The information contained in this message and/or attachments
from AgResearch Limited is intended only for the persons or entities
to which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipients is prohibited by AgResearch
Limited. If you have received this message in error, please notify the
sender immediately.
===

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Deseq2 wrapper question

[Ross]

Vipin,
Search works pretty well on the toolshed - I'd recommend a search for (eg)
'star' before writing yet another rnastar toolshed wrapper and dependency
package? There's an installable fubar package_rnastar and an rnastar
wrapper based on Jeremy's code mentioned below - but working AFAIK - at
least it installs and works fine here and on the GVL.



On Thu, Sep 12, 2013 at 1:49 AM, Jeremy Goecks wrote:

> STAR aligner
>
>
> I have a partial STAR wrapper that may be useful:
> https://bitbucket.org/jgoecks/jeremys-code/src/d7096c132657420f08b540c6954b947f648aac80/galaxy/tool-wrappers/star.xml?at=default
>
> Whether you use this code or not, I do have one request: please put the
> output in BAM rather than SAM format in order to save disk space.
>
> Best,
> J.
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] tool shed can't install via admin user

[Dave Bouvier]

Adhemar,

Thank you for reporting this issue, it has been resolved in 
10562:48191107fd2d.


   --Dave B.

On 09/11/2013 12:10 PM, Adhemar wrote:

Hello,
I thought it might be a problem related to authentication and decided to
run it in a brand new install on my desktop.
Same problem: HTTP Error 500
Could you please verify this?
Thanks,
Adhemar


  Error Traceback:

View as: Interactive

  |
Text

  |
XML
(full)


⇝|HTTPError: HTTP Error 500: Internal Server Error|

URL:
http://127.0.0.1:8080/admin_toolshed/prepare_for_install?tool_shed_url=http://toolshed.g2.bx.psu.edu/&repository_ids=c45d6c51a4fcfc6c&changeset_revisions=bf1c3f9f8282

Module weberror.evalexception.middleware:*364* in
|respond|view

|>>
||app_iter*=*self||*.*||application||*(*||environ||*,*detect_start_response||*)*|

Module paste.recursive:*84* in
|__call__|view

|>>
||*return*self||*.*||application||*(*||environ||*,*start_response||*)*|

Module paste.httpexceptions:*633* in
|__call__|view

|>>
||*return*self||*.*||application||*(*||environ||*,*start_response||*)*|

Module galaxy.web.framework.base:*132* in
|__call__|view

Re: [galaxy-dev] Visualisation of VCF files in trackster

[Jeremy Goecks]

> My attempts to visualise a vcf file in trackster fail. I can visualise 
> the corresponding SAM/BAM files fine. I can visualise the same file on 
> Galaxy main too, but not in my local install.

To be clear, you can visualize the VCF dataaset on main but not locally? Can 
you visualize SAM/BAM datasets locally?

One guess is that you may need to install and add tabix and bgzip to your path.

J.
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] PATH and local jobs

[Adam Brenner]

Ryan,

On our HPC setup, we use the modules system but its the same issue you are
running into.

A number of methods exists for setting up the correct path, for you, I
believe the easiest would be one of the two:


1) In your init script: source (or setup) PATHS as you have them in your
.bash_profile.
/etc/init.d/galaxy

OR

2) Within Galaxy, edit universe_wsgi.ini and look for:
environment_setup_file
and create a file that will setup the correct $PATH for you.


You can also invoke the Tool Dependency options for each specific tool you
load as shown in[1].

[1]: http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

Research Computing Support
Office of Information Technology
http://www.oit.uci.edu/rcs/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
aebre...@uci.edu


On Wed, Sep 11, 2013 at 10:18 AM, Ryan Davis  wrote:

> Hi,
>
> ** **
>
> We have a local server running Galaxy and a majority of the tools work
> correctly (bowtie, BWA, samtools and etc…).  I am having issues with some
> of the FASTX-Toolkit tools.  I am starting Galaxy with an init script on
> Centos 6 and the tools are in my $PATH and work at the command line, just
> not in Galaxy.  The wiki suggest to “set $PATH in the startup file or
> use.” 
>
> It leaves me hanging and don’t know what the other option is.  Can someone
> share with me their workaround?
>
> ** **
>
> Here is the wiki page:
> http://wiki.galaxyproject.org/Admin/Config/Tool%20Dependencies
>
> ** **
>
> Here is the section I am hoping will help:
>
> ** **
> Local Jobs
>
> Changes to $PATH can be persisted by setting them in your shell's startup
> file(s).  This typically means ~/.bash_profile for bash, but please see
> the bash documentation on startup 
> files
>  or the *INVOCATION* section of the bash(1) man page to understand the
> intricacies of how that file is read.  Of particular importance, if you are
> starting Galaxy in a method other than manual invocation from a shell
> prompt (with sh run.sh) such as with an init script, it is likely that
> your startup file will not be read.  In this instance, you should set
> $PATH in the startup file or use.
>
> ** **
>
> ** **
>
> Thanks!
>
> Ryan
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] HOWTO for making new toolshed package

[Greg Von Kuster]

Hello Pieter,

Please send questions like this to the galaxy-...@bx.psu.edu or 
galaxy-dev@lists.bx.psu.edu mailing list rather than individual email addresses 
to ensure optimal, timely responses.

The Tool Shed wiki contains all of the information you need:

http://wiki.galaxyproject.org/Tool%20Shed

This page provides a nice overview of repository features:

http://wiki.galaxyproject.org/ToolShedRepositoryFeatures

See additional specific pages below for each item you've listed:


On Sep 11, 2013, at 11:52 AM, "Lukasse, Pieter"  wrote:

> Hi Greg,
>  
> I want to make a toolshed package with the following contents:

The above "package" is actually a mercurial repository in the tool shed.

> · A number of tools

The following pages will provide all of the information you'll need for 
building Galaxy tools for the Tool Shed.

http://wiki.galaxyproject.org/ToolShedToolFeatures

> · Some workflows

The following pages provide information for enabling workflow sharing via the 
Tool Shed.

http://wiki.galaxyproject.org/ToolShedWorkflowSharing


> · Datatypes

Datatype features in the Tool Shed are described in the following pages:

http://wiki.galaxyproject.org/ToolShedDatatypesFeatures


> · Installation configurations for installing external tools like R, 
> Gnuplot and X!Tandem

The above information is document in the "ToolShedToolFeatures" section of the 
wiki, specifically in the following pages:

http://wiki.galaxyproject.org/ToolShedToolFeatures#Automatic_third-party_tool_dependency_installation_and_compilation_with_installed_repositories


> · Dependencies configurations to other packages

If by "packages", you are referring to other tool shed repositories, then the 
documentation describing defining these relationships is located in the 
following pages:

http://wiki.galaxyproject.org/DefiningRepositoryDependencies


>  
> Is there a HOWTO wiki page where I can find exactly which files I should add 
> to the .zip file and how they should be named? Or maybe an example package 
> which I could use?
>  
> Regards,
>  
> Pieter Lukasse
> 
> Wageningen UR, Plant Research International
> 
> Departments of Bioscience and Bioinformatics
> 
> Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB, 
> Wageningen, the Netherlands
> 
> +31-317480891; skype: pieter.lukasse.wur
> 
> http://www.pri.wur.nl
> 
>  

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] workflow_execute.py limitation?

[Hakeem Almabrazi]

Hi,

Has anyone use the API workflow_execute.py to run workflows with a lot of 
steps?  The reason why I am asking is that I keep getting some errors when I 
ran workflows that are long (i.e ~ 50 steps).  The same workflow could be 
executed successfully from the GUI.  The first Few steps (16 steps) get 
scheduled in the created history but the rest don't and then after while error 
out.

Here are the error messages I keep getting.

HTTP Error 500: Internal Server Error
500 Internal Server Error
The server has either erred or is incapable of performing
the requested operation.
*
And here is some error I capture in the log file>

  File "/usr/local/galaxy/galaxy-dist/lib/galaxy/tools/__init__.py", line 2126, 
in execute
return self.tool_action.execute( self, trans, incoming=incoming, 
set_output_hid=set_output_hid, history=history, **kwargs )
  File "/usr/local/galaxy/galaxy-dist/lib/galaxy/tools/actions/__init__.py", 
line 362, in execute
history.add_dataset( data, set_hid = set_output_hid )
  File "/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py", 
line 1124, in __getattr__
if key not in self: raise AttributeError, key
AttributeError: add_dataset


I am not sure if this is related to some kind of waiting processing time and 
since it does not get a reply then it errors out.

I would really appreciate any kind of help.  I might have to break my workflow 
into smaller ones if there is no other way.

Regards
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problem with installing MAF cached dataset

[Daniel Blankenberg]

Hi Leon,

What is the output of "ls -lah /mnt/galaxyIndices/alignments/maf/"?

Also, you may what to use additional sub-directory structure for your file 
locations in case you want to add more alignments later, e.g., 
/mnt/galaxyIndices/hg19/maf/10_way_multiz/ or similiar

Also, note that although you are only indexing (for extraction) for the 
specified listed species, the alignments will contain all of the species 
present in the original file (you list 46way multiZ as the source).


Thanks for using Galaxy,

Dan


On Sep 10, 2013, at 6:44 PM, Leon Mei wrote:

> Dear colleagues,
> 
> I am having some trouble installing MAF datasets at our server. When I
> run "Extract MAF blocks given a set of genomic intervals", I got this
> error message:
> 
> "Fatal Error: The MAF source specified (10_WAY_MULTIZ_hg19) appears to
> be invalid."
> 
> Below is what I did by following the wiki page:
> http://wiki.galaxyproject.org/Admin/Datatypes/Add%20MAFs
> 
> 1, download all mafs for hg19 from
> http://hgdownload.cse.ucsc.edu/goldenPath/hg19/multiz46way/maf/
> 
> 2, index them using "maf_build_index.py --species=hg19,panTro2,
> gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2
> $file". Our user requested primates only.
> 
> 3, Here is my maf_index.loc file:
> ==
> 10-way multiZ (hg19)10_WAY_MULTIZ_hg19
> hg19,panTro2,gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2
>  hg19,panTro2,gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2
>   /mnt/galaxyIndices/alignments/maf/chr1.maf,/mnt/galaxyIndices/alignments/m
>   
> af/chr10.maf,/mnt/galaxyIndices/alignments/maf/chr11.maf,/mnt/galaxyIndices/alignments/maf/chr11_gl000202_random.maf,/mnt/galaxyIndices/ali
>   
> gnments/maf/chr12.maf,/mnt/galaxyIndices/alignments/maf/chr13.maf,/mnt/galaxyIndices/alignments/maf/chr14.maf,/mnt/galaxyIndices/alignments
>   
> /maf/chr15.maf,/mnt/galaxyIndices/alignments/maf/chr16.maf,/mnt/galaxyIndices/alignments/maf/chr17.maf,/mnt/galaxyIndices/alignments/maf/ch
>   
> r17_ctg5_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000203_random.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000204_random.maf,/
>   
> mnt/galaxyIndices/alignments/maf/chr17_gl000205_random.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000206_random.maf,/mnt/galaxyIndices/a
>   
> lignments/maf/chr18.maf,/mnt/galaxyIndices/alignments/maf/chr18_gl000207_random.maf,/mnt/galaxyIndices/alignments/maf/chr19.maf,/mnt/galaxy
>   
> Indices/alignments/maf/chr19_gl000208_random.maf,/mnt/galaxyIndices/alignments/maf/chr19_gl000209_random.maf,/mnt/galaxyIndices/alignments/
>   
> maf/chr1_gl000191_random.maf,/mnt/galaxyIndices/alignments/maf/chr1_gl000192_random.maf,/mnt/galaxyIndices/alignments/maf/chr2.maf,/mnt/gal
>   
> axyIndices/alignments/maf/chr20.maf,/mnt/galaxyIndices/alignments/maf/chr21.maf,/mnt/galaxyIndices/alignments/maf/chr21_gl000210_random.maf
>   
> ,/mnt/galaxyIndices/alignments/maf/chr22.maf,/mnt/galaxyIndices/alignments/maf/chr3.maf,/mnt/galaxyIndices/alignments/maf/chr4.maf,/mnt/gal
>   
> axyIndices/alignments/maf/chr4_ctg9_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr4_gl000193_random.maf,/mnt/galaxyIndices/alignments/maf/c
>   
> hr4_gl000194_random.maf,/mnt/galaxyIndices/alignments/maf/chr5.maf,/mnt/galaxyIndices/alignments/maf/chr6.maf,/mnt/galaxyIndices/alignments
>   
> /maf/chr6_apd_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr6_cox_hap2.maf,/mnt/galaxyIndices/alignments/maf/chr6_dbb_hap3.maf,/mnt/galaxyI
>   
> ndices/alignments/maf/chr6_mann_hap4.maf,/mnt/galaxyIndices/alignments/maf/chr6_mcf_hap5.maf,/mnt/galaxyIndices/alignments/maf/chr6_qbl_hap
>   
> 6.maf,/mnt/galaxyIndices/alignments/maf/chr6_ssto_hap7.maf,/mnt/galaxyIndices/alignments/maf/chr7.maf,/mnt/galaxyIndices/alignments/maf/chr
>   
> 7_gl000195_random.maf,/mnt/galaxyIndices/alignments/maf/chr8.maf,/mnt/galaxyIndices/alignments/maf/chr8_gl000196_random.maf,/mnt/galaxyIndi
>   ces/alignments/maf/chr8_gl000197_random.maf
> ==
> 
> Any hint on which step is wrong or missing is highly appreciate!
> 
> Thanks,
> Leon
> 
> 
> 
> 
> 
> -- 
> Hailiang (Leon) Mei
> Netherlands Bioinformatics Center
> BioAssist NGS Taskforce
> - http://ngs.nbic.nl
> Skype: leon_meiMobile: +31 6 41709231
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Stable Galaxy for local install

[cngbgalaxy]

Hi Joachim,
Thank you very much for your help.
Actually I find some problems about such versions of Galaxy in 
https://bitbucket.org/galaxy/galaxy-dist/downloads .
We can not view the data in history by clicking the eyes icon on the 
c42567f43aa7 version.The "Select" in "Filter and Sort" fail to work on the 
1ae95b3aa98d (release_2013.08.12 ) version and the 524f246ca853 
(release_2013.06.03 ),though I don't know whether it is a bug or not.
As a result,I feel confused about which version of Galaxy to install and avoid 
the above problems.




cngbgalaxy

From: Joachim Jacob | VIB |
Date: 2013-09-11 22:38
To: test galaxy
CC: galaxy-dev; galaxy-dev
Subject: Re: [galaxy-dev] Stable Galaxy for local install
The latest code from the 'stable' branch of Galaxy is preferred. See the 
news brief accompanying the update (release_2013.08.12) on
http://wiki.galaxyproject.org/DevNewsBriefs/2013_08_12

You need mercurial (http://mercurial.selenic.com/) for installing 
Galaxy. Report bugs at https://trello.com/b/75c1kASa/galaxy-development

Cheers
Joachim

Joachim Jacob
Contact details: http://www.bits.vib.be/index.php/about/80-team


On 09/11/2013 03:55 AM, test galaxy wrote:
> Hi everyone,
> I want to install my local Galaxy with less bugs,and I don't know 
> which version is more stable and work well.Could anyone give same 
> advise?Thank you very much.
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>http://galaxyproject.org/search/mailinglists/___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Proper way of using bioblend.galaxy.workflows.run_workflow() {'': {'id': , 'src': '[ldda, ld, hda]'}

[Carlos Borroto]

Hi,

I'm rewriting my galaxy-api-tools project to use bioblend. I can't
believe I missed it the first time around.

I'm a little confused reading about LibraryDatasetDatasetAssociation
(ldda), LibraryDataset (ld) and HistoryDatasetAssociation (hda). I
found something in the wiki[1] but is not helping enough.

I guess my question is, let's say I use any of
bioblend.galaxy.libraries.upload_*() fucntions[2] to upload a file to
a data library, the json response for that is something that looks
like this:
[{'url': '/api/libraries/ca4f6513e2849358/contents/e70b6e37a29da1c1',
'id': 'e70b6e37a29da1c1', 'name': '1.fastq'}]

Now let's say I use bioblend.galaxy.tools.upload_file()[2] and upload
a file to a history, the json response for that looks like:
{'outputs': [{'misc_blurb': None, 'peek': '', 'uuid': None, 'hda_ldda': 'hda',
'data_type': 'auto', 'deleted': False, 'history_id':
'6eafbc3efd020327', 'purged': False, 'state': 'queued', 'name':
'1.fastq', 'visible': True, 'genome_build': '?', 'model_class':
'HistoryDatasetAssociation', 'hid': 5, 'file_size': 0,
'metadata_data_lines': None, 'file_ext': 'auto', 'id':
'c3e7863ed8b6cf9e', 'misc_info': None, 'metadata_dbkey': '?'}]}

Which 'src' setting should I use for
bioblend.galaxy.workflows.run_workflow()'s[2] {'': {'id':
, 'src': '[ldda, ld, hda]'}?

If I have to guess {'': {'id': 'e70b6e37a29da1c1', 'src':
'ldda'} for the first one and {'': {'id': 'c3e7863ed8b6cf9e',
'src': 'hda'} for the second one. That leaves out 'ld', when do I have
to use that one?

Finally and as a comment. I think the inconsistency on what to expect
after delivering a payload to the API makes it hard to use these
responses. See this issue[3] I submitted to bioblend but it seems is
actually Galaxy who is at fault.

[1]http://wiki.galaxyproject.org/Admin/Config/Performance/Purge%20Histories%20and%20Datasets
[2]http://bioblend.readthedocs.org/en/latest/api_docs/galaxy/all.html
[3]https://github.com/afgane/bioblend/issues/40

Thanks,
Carlos
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] 'DECLARE CURSOR ... FOR UPDATE/SHARE is not supported'

[hogart]

Actually, the example in my previous letter wasn't correct - I have tried
to run run.sh and manage_db.sh scripts several times after creation of
novel db, and the copy-paste is the result of the last attempt. So, indeed,
the db galaxy was created in the first running, while warning messages from
the last running. But now I drop the db again, and the issue is reproduced
right after creation of the fresh db:

[galaxy@cluster galaxy-dist]$ sh run.sh
<~! several messages are omitted !~>
migrate.versioning.script.base DEBUG 2013-09-11 18:33:49,426 Loading script
lib/galaxy/model/migrate/versions/0113_update_migrate_tools_table.py...
migrate.versioning.script.base DEBUG 2013-09-11 18:33:49,426 Script
lib/galaxy/model/migrate/versions/0113_update_migrate_tools_table.py loaded
successfully
migrate.versioning.script.base DEBUG 2013-09-11 18:33:49,426 Loading script
lib/galaxy/model/migrate/versions/0114_update_migrate_tools_table_again.py...
migrate.versioning.script.base DEBUG 2013-09-11 18:33:49,426 Script
lib/galaxy/model/migrate/versions/0114_update_migrate_tools_table_again.py
loaded successfully
migrate.versioning.script.base DEBUG 2013-09-11 18:33:49,426 Loading script
lib/galaxy/model/migrate/versions/0115_longer_user_password_field.py...
migrate.versioning.script.base DEBUG 2013-09-11 18:33:49,426 Script
lib/galaxy/model/migrate/versions/0115_longer_user_password_field.py loaded
successfully
migrate.versioning.repository DEBUG 2013-09-11 18:33:49,427 Repository
lib/galaxy/model/migrate loaded successfully
migrate.versioning.repository DEBUG 2013-09-11 18:33:49,427 Config:
OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'),
('repository_id', 'Galaxy'), ('version_table', 'migrate_version'),
('required_dbs', '[]')]))])
galaxy.model.migrate.check DEBUG 2013-09-11 18:33:49,433 psycopg2 egg
successfully loaded for postgres dialect
/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/url.py:105:
SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed
from 'postgres' to 'postgresql'. The new URL format is
postgresql[+driver]://:@/
galaxy.model.migrate.check INFO 2013-09-11 18:33:49,447 No database,
initializing
galaxy.model.migrate.check INFO 2013-09-11 18:33:49,477 Migrating 0 -> 1...
galaxy.model.migrate.check INFO 2013-09-11 18:33:49,712
galaxy.model.migrate.check INFO 2013-09-11 18:33:49,712 Migrating 1 -> 2...
galaxy.model.migrate.check INFO 2013-09-11 18:33:50,013
galaxy.model.migrate.check INFO 2013-09-11 18:33:50,014 Migrating 2 -> 3...
galaxy.model.migrate.check INFO 2013-09-11 18:33:50,803
galaxy.model.migrate.check INFO 2013-09-11 18:33:50,803 Migrating 3 -> 4...
galaxy.model.migrate.check INFO 2013-09-11 18:33:51,502
galaxy.model.migrate.check INFO 2013-09-11 18:33:51,502 Migrating 4 -> 5...
galaxy.model.custom_types DEBUG 2013-09-11 18:33:51,556 Fixing a
discrepancy concerning deleted shared history items.
galaxy.model.custom_types DEBUG 2013-09-11 18:33:51,583 0 items affected,
and restored.
galaxy.model.custom_types DEBUG 2013-09-11 18:33:51,583 Time elapsed:
0.0264229774475
galaxy.model.custom_types DEBUG 2013-09-11 18:33:51,583 Fixing a
discrepancy concerning cleaning up deleted history items shared before HDAs.
galaxy.model.custom_types DEBUG 2013-09-11 18:33:51,586 0 items affected,
and restored.
galaxy.model.custom_types DEBUG 2013-09-11 18:33:51,586 Time elapsed:
0.00265407562256
galaxy.model.migrate.check INFO 2013-09-11 18:33:51,589
galaxy.model.migrate.check INFO 2013-09-11 18:33:51,589 Migrating 5 -> 6...
galaxy.model.migrate.check INFO 2013-09-11 18:33:52,334

galaxy.model.migrate.check INFO 2013-09-11 18:33:52,334 This migration
script changes certain values in the history_dataset_association.extension
galaxy.model.migrate.check INFO 2013-09-11 18:33:52,334 column,
specifically 'qual' is chaged to be 'qual454'.
galaxy.model.migrate.check INFO 2013-09-11 18:33:52,334

galaxy.model.migrate.check INFO 2013-09-11 18:33:52,334
galaxy.model.migrate.check INFO 2013-09-11 18:33:52,335 Migrating 6 -> 7...
galaxy.model.migrate.check INFO 2013-09-11 18:33:53,053

galaxy.model.migrate.check INFO 2013-09-11 18:33:53,053 This migration
script creates the new history_user_share_association table, and adds
galaxy.model.migrate.check INFO 2013-09-11 18:33:53,053 a new boolean type
column to the history table.  This provides support for sharing
galaxy.model.migrate.check INFO 2013-09-11 18:33:53,053 histories in the
same way that workflows are shared.
galaxy.model.migrate.check INFO 2013-09-11 18:33:53,053

galaxy.model.migrate.check INFO 2013-09-11 18:33:53,053
galaxy.model.migrate.check INFO 2013-09-11 18:33:53,054 Migrating 7 -> 8...
galaxy.model.migrate.check INFO 2013-09-11 18:33:53,969

galaxy.model.migrate.check INFO 201

Re: [galaxy-dev] Stable Galaxy for local install

[Joachim Jacob | VIB |]

The latest code from the 'stable' branch of Galaxy is preferred. See the 
news brief accompanying the update (release_2013.08.12) on

http://wiki.galaxyproject.org/DevNewsBriefs/2013_08_12

You need mercurial (http://mercurial.selenic.com/) for installing 
Galaxy. Report bugs at https://trello.com/b/75c1kASa/galaxy-development


Cheers
Joachim

Joachim Jacob
Contact details: http://www.bits.vib.be/index.php/about/80-team


On 09/11/2013 03:55 AM, test galaxy wrote:

Hi everyone,
I want to install my local Galaxy with less bugs,and I don't know 
which version is more stable and work well.Could anyone give same 
advise?Thank you very much.



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Deseq2 wrapper question

[Vipin TS]

Yes I am agreeing on all statements.
Let me finish my package and will add in one of the toolshed by today
evening. I also willing to test other wrappers to improve the additional
parameter settings.

Tool wrappers Work In Progress from my side:
DEXSeq
edgeR
Scripture
STAR aligner
SAFT multi mapper resolution

This will be adding soon to the toolshed.

Thanks to all, Vipin

On Wed, Sep 11, 2013 at 6:08 AM, Bjoern Gruening
wrote:

> Am Mittwoch, den 11.09.2013, 10:21 +0100 schrieb Peter Cock:
> > On Wed, Sep 11, 2013 at 10:12 AM, Joachim Jacob | VIB |
> >  wrote:
> > > It is not necessarily bad. If we all test each others tools
> (dependencies,
> > > interface, output,...) we might get to the best of all worlds :-)  (I
> have
> > > planned to test Björns wrapper)
> > >
> > > One of the points of differentiation is also how 'integrated' a tool
> needs
> > > to be: e.g. Ross' tool for diff expression (combining a complete
> pipelin),
> > > and my tool just on DE with DESeq2. I like to keep things granular on
> the
> > > level of the tools.
> >
> > That is a good reason for different wrappers, one for the tool itself,
> > and one for a larger pipeline using the same tool. And both can
> > declare a binary package dependency via the Tool Shed using
> > the binary dependency package Bjoern is working on, which I
> > assume will be released on the Test & Main Tool Shed under
> > the shared IUC account once ready:
> >
> > http://testtoolshed.g2.bx.psu.edu/view/bgruening/package_deseq2_1_0_17
>
>
> Yes that is right, at the moment I have patches for the toolshed in
> preparation for a unified and easy installation of R dependencies, once
> that is done it will be under the IUC account.
>
> I don't see the need for 5 different wrappers. Ross approach deserves an
> own wrapper I think. I thought a lot about if we should use it, since it
> can compare 3 different approaches for DE, but it comes with some
> complexity. We spend some time in trying to get the UI correct and
> investigated some possibilities. Currently we have a few guys testing it
> and giving comments to our UI idea. If the ownershop is a problem, we
> probably can move the wrappers to a different account. But please lets
> merge the code base and start communicating about the UI.
>
> Thanks,
> Bjoern
>
> > Regards,
> >
> > Peter
> >
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >   http://lists.bx.psu.edu/
> >
> > To search Galaxy mailing lists use the unified search at:
> >   http://galaxyproject.org/search/mailinglists/
>
>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] ZFS storage recommendations

[Nate Coraor]

Hi all,

I'm a big fan of ZFS, we have long used it behind Galaxy Main.  Some of our 
older servers are (still) Solaris, and the newest is FreeBSD.

I've lately been using SmartOS for virtualization and while it has a drawback 
as a fileserver (since currently the nfs server can only run in the global 
zone, which is not ideal on SmartOS), there are other illumos derivatives that 
would probably be great for this task (e.g. OmniOS).  Native ZFS in the OS in 
which it is developed is a win for me, especially when you are serving via 
simple NFS.  For more complex network filesystems, Linux is probably preferable.

I considered a separate ZIL and L2ARC for the latest ZFS server, but DTrace 
revealed that I probably would not see much of a performance benefit with our 
usage patterns.  The memory usage you're seeing is to be expected - it will 
pretty much consume whatever is available for caching, but it's available to be 
freed if needed for something else.

I wouldn't suggest rsync for performance testing.  I typically do things like 
timed writing of blocks read from /dev/zero using dd, so that the source 
filesystem and checksumming algorithm can be taken out of the equation.  And 
dedup/compression will of course cause a significant write penalty.  If you can 
suffer the decreased space optimization, lzjb performs significantly better 
than gzip.  gzip-1 is a nice compromise between the default gzip level and 
lzjb, as well.

--nate

On Sep 11, 2013, at 3:45 AM, Joachim Jacob | VIB | wrote:

> Thank you all for the reactions.
> 
> Some details about my current ZFS and Galaxy setup:
> 
> - Galaxy runs as a single virtual machine, with currently 20 cores, 80GB RAM. 
> Will be 32 cores and about 160GB RAM soon.
> - The postgres database is on the virtual machine itself.
> - The 'files' and 'job_working_dir' are on an NFS exported directory, hosted 
> on the host machine of the guest.
> - The NFS exported directory is an raidz1 dataset.
> - The raidsz1 runs on 7 550GB SAS disks, which are (unfortunately) controlled 
> by a RAID hardware controller, but passed as RAID0 (JBOD not available). So 
> raidz1 runs on 7 RAID0 disks (with settings in the hardware controller PERC 
> H700: no read ahead, write through, 8 KB stripe size, disk cache policy 
> enabled).
> - Compression and deduplication is enabled.
> - The directory on which the zfs dataset is mounted, is exported using the 
> native linux NFS daemon to the Galaxy virtual machine. The 'zfs sharenfs' did 
> not work (ownerships not set correctly - perhaps need some more 
> investigation, but I found several times reports about sharenfs option in ZFS 
> in linux is not behaving well...).
> 
> The numbers:
> - my initial files database (ext4 on RAID5) is 3.0TB in size. On ZFS, with 
> compression and deduplication, this database is *1.8TB *(-40%).
> - Did not yet provide a SLOG to host the ZIL and a L2ARC, since I have a 
> cleared picture about the performance I can get. Would you advise preference 
> over ZIL on a SLOG, or go for a SSD to host the L2ARC.
> - The cost for this performance of ZFS in storage is RAM: currently 
> continuously using*284GB RAM* for ZFS!
> - The write and read speed is from the Galaxy VM over NFS is *~40MB/s and 
> ~100MB/s* (tested by simply copying over rsync - I still need to check the 
> presentation and scripts of Anne Black-Ziegelbein). This is a 66% decrease in 
> previously achieved write and read speed (ext4 on hardware RAID5), but I feel 
> that the benefits (deduplication, backing up via snapshots, data integrity,) 
> outweigh this IO performance.
> 
> (I am setting this ZFS up on a new server (well, actually 2 years old now, 
> has served on another project well))
> 
> Currently our Galaxy uses this zfs with success!
> 
> For your interest, my settings on the 'galaxydb' zfs tank below. (I was 
> wondering if here some more wizardry can be applied).
> 
> 
> [root@r910bits ~]# *zfs get all tank/galaxydb*
> NAME   PROPERTY  VALUE  SOURCE
> tank/galaxydb  type  filesystem -
> tank/galaxydb  creation  Mon Sep  9 12:44 2013  -
> tank/galaxydb  used  1.81T  -
> tank/galaxydb  available 1.66T  -
> tank/galaxydb  referenced1.81T  -
> tank/galaxydb  compressratio 1.66x  -
> tank/galaxydb  mounted   yes-
> tank/galaxydb  quota none default
> tank/galaxydb  reservation   none default
> tank/galaxydb  recordsize128K default
> tank/galaxydb  mountpoint/mnt/galaxydb  local
> tank/galaxydb  sharenfs  rw=@galaxy local
> tank/galaxydb  checksum  on default
> tank/galaxydb  compression   lzjb   local
> tank/galaxydb  atime on default
> tank/galaxydb  devices   on def

Re: [galaxy-dev] Difficulty dispatching toolshed tool jobs via job_conf.xml

[Nate Coraor]

Hi Simon,

This should work, it looks like there may be a bug with handling tool shed IDs 
when determining job destinations. You are actually supposed to be able to use 
any of:

  shed_host/repos/owner/repo/tool_id/version
  shed_host/repos/owner/repo/tool_id
  tool_id

I'll take a look at this as soon as possible.  One thing you might try in the 
short term is using the percent encoded tool id in the  tag in 
job_conf.xml:

  
toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS%3A%20infoseq46/5.0.0

--nate

On Sep 10, 2013, at 10:35 PM, Guest, Simon wrote:

> I am having trouble getting a toolshed tool to be dispatched to the 
> destination I list in the job_conf.xml file.
>  
> My job_conf.xml file looks like this:
>  
> 
> 
> 
>  load="galaxy.jobs.runners.local:LocalJobRunner" workers="20"/>
>  load="galaxy.jobs.runners.condor:CondorJobRunner" />
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  
> 
>  
> 
>  
> ... stuff omitted ...
>  
>  id="toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS:infoseq46/5.0.0"
> destination="local"/>
> 
> 
>  
>  
> So, you can see I have a default destination of condor, but I'm trying to run 
> my toolshed EMBOSS infoseq tool on local.  However, it is stubbornly running 
> on condor.
>  
> In lib/galaxy/tools/__init__.py:1132, I see this comment which got me 
> wondering:
> # In the toolshed context, there is no job config.
>  
> Is it possible to define tool destinations for toolshed tools?  Are there 
> some gotchas that I should know about?  Any other ideas why my job is 
> ignoring the config in job_conf.xml?  (By the way, I can change say the 
> upload1 tool to run on Condor by setting its destination in that file, so it 
> is doing something.)  The other thing I saw in the source code is stuff about 
> old_id and toolshed guids.  Do I need to understand this stuff?
>  
> The paster.log contains the following when I submit the infoseq job:
>  
> 147.158.130.216 - - [11/Sep/2013:13:53:34 +1300] "GET 
> /tool_runner?tool_id=toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS%3A%20infoseq46/5.0.0
>  HTTP/1.1" 200 - "http://galaxy-dev.agresearch.co.nz/"; "Mozilla/5.0 (X11; 
> Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0"
> 147.158.130.216 - - [11/Sep/2013:13:53:40 +1300] "POST /tool_runner/index 
> HTTP/1.1" 200 - 
> "http://galaxy-dev.agresearch.co.nz/tool_runner?tool_id=toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS%3A%20infoseq46/5.0.0";
>  "Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0"
> galaxy.jobs DEBUG 2013-09-11 13:53:40,886 (92) Working directory for job is: 
> /home/galaxy-dev/galaxy/database/job_working_directory/000/92
> galaxy.tools DEBUG 2013-09-11 13:53:40,886 Tool::get_job_destination: 
> {'runner': 'condor', 'legacy': False, 'params': {}, 'tags': None, 'url': 
> None, 'converted': False, 'id': 'condor'}.
> galaxy.jobs.handler DEBUG 2013-09-11 13:53:40,894 (92) Dispatching to condor 
> runner
>  
> (I added the debug output for Tool::get_job_destination to see what was going 
> on.)
>  
> Any ideas?
>  
> cheers,
> Simon
> 
> ===
> Attention: The information contained in this message and/or attachments
> from AgResearch Limited is intended only for the persons or entities
> to which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipients is prohibited by AgResearch
> Limited. If you have received this message in error, please notify the
> sender immediately.
> ===
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Visualisation of VCF files in trackster

[Ulf Schaefer]

Dear all

My attempts to visualise a vcf file in trackster fail. I can visualise 
the corresponding SAM/BAM files fine. I can visualise the same file on 
Galaxy main too, but not in my local install.

I receive the following error message:

"Trackster Error
Input error: found interval with block-counts not matching starts/sizes 
on line.
sort: write failed: standard output: Broken pipe
sort: write error
needLargeMem: trying to allocate 0 bytes (limit: 1000)"

The vcf format is in version 4.1. I am creating it from the sorted BAM 
file with samtools mpileup in the most basic way.

What am I missing?

Thanks a lot for your help
Ulf

**
The information contained in the EMail and any attachments is confidential and 
intended solely and for the attention and use of the named addressee(s). It may 
not be disclosed to any other person without the express authority of Public 
Health England, or the intended recipient, or both. If you are not the intended 
recipient, you must not disclose, copy, distribute or retain this message or 
any part of it. This footnote also confirms that this EMail has been swept for 
computer viruses by Symantec.Cloud, but please re-sweep any attachments before 
opening or saving. http://www.gov.uk/PHE
**

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] 'DECLARE CURSOR ... FOR UPDATE/SHARE is not supported'

[hogart]

Hi Nate,
Yes, db schema is old, but the upgrading (manage_db.sh upgrade) to the
current version resulted to the error, as mentioned above. I don't remember
the release number of the previous version of Galaxy, but it was installed
in the March this year. A didn't restored the dump yet, now I am playinig
with the newly created database. In my newbie point of view, it seems like,
that dropping of the old database (drop database galaxy; as superuser) was
incomplete, - is it possible?
Thanks for letting me know, I will try to upgrade the PostgeSQL (the
current version is default version of CentOS 5.6).

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] 'DECLARE CURSOR ... FOR UPDATE/SHARE is not supported'

[Nate Coraor]

Hi Sergei,

Your current Galaxy database migration revision (13) is extremely old.  How old 
was your previous Galaxy installation, and are you certain that the dump you 
created was complete, and fully restored?

To solve the original 'DECLARE CURSOR ... FOR UPDATE' problem, you'd need to 
update your version of PostgreSQL.  8.1.x is also very old, 9.1 or 9.2 are 
recommended.

--nate

On Sep 11, 2013, at 7:58 AM, hogart wrote:

> So,
> 
> I decided to completely drop the old galaxy psql database (I have a dump of 
> it) - 
> hogart# drop database galaxy;
> 
> and recreate it again -
> [galaxy@cluster ~]$ creatdb
> 
> After the launching of the Galaxy it is terminated with the message with 
> requesting of upgrade the database, from 13 to 115. But the upgrading of 
> database scheme is also failed:
> [galaxy@cluster galaxy-dist]$ sh manage_db.sh upgrade
> /export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/url.py:105:
>  SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed 
> from 'postgres' to 'postgresql'. The new URL format is 
> postgresql[+driver]://:@/
> 13 -> 14... 
> 
> Migration script to add support for "Pages".
>   1) Creates Page and PageRevision tables
>   2) Adds username column to User table
> 
> Traceback (most recent call last):
>   File "./scripts/manage_db.py", line 64, in 
> main( repository=repo, url=db_url )
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/shell.py",
>  line 207, in main
> ret = command_func(**kwargs)
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/api.py",
>  line 186, in upgrade
> return _migrate(url, repository, version, upgrade=True, err=err, **opts)
>   File "", line 2, in _migrate
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/util/__init__.py",
>  line 159, in with_engine
> return f(*a, **kw)
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/api.py",
>  line 366, in _migrate
> schema.runchange(ver, change, changeset.step)
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/schema.py",
>  line 91, in runchange
> change.run(self.engine, step)
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/script/py.py",
>  line 145, in run
> script_func(engine)
>   File "lib/galaxy/model/migrate/versions/0014_pages.py", line 55, in upgrade
> col.create( User_table, index_name='ix_user_username', 
> unique_name='username' )
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/schema.py",
>  line 528, in create
> engine._run_visitor(visitorcallable, self, connection, **kwargs)
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
>  line 2302, in _run_visitor
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
>  line 1972, in _run_visitor
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/ansisql.py",
>  line 53, in traverse_single
> ret = super(AlterTableVisitor, self).traverse_single(elem)
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/sql/visitors.py",
>  line 106, in traverse_single
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/ansisql.py",
>  line 101, in visit_column
> self.execute()
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/ansisql.py",
>  line 42, in execute
> return self.connection.execute(self.buffer.getvalue())
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
>  line 1449, in execute
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
>  line 1628, in _execute_text
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
>  line 1698, in _execute_context
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
>  line 1691, in _execute_context
>   File 
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/default.py",
>  line 331, in do_execute
> sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "username" of 
> relation "galaxy_user" already exists
>  '\nALTER TABLE galaxy_user ADD username VARCHAR(255)' {}
> 
> Since, it was past release of Galaxy (release_2013.06.03), I pulled the 
> lat

Re: [galaxy-dev] Scaling and hardware requirements

[Gerald Bothe]

Can I put in a similar question on top of this: How much resources do you need 
for re-sequencing of a mammalian genome (assembly and variant detection), one 
job at a time? E.g. how much RAM  etc. if I want the re-sequencing SAM file of 
a 30-fold coverage be done in 48 hours?
 
Gerald

Gerald Bothe
32 Plum Hill Road
East Lyme, CT 06333
(860) 451 8776


>
> From: Nikos Sidiropoulos 
>To: Peter Cock  
>Cc: ""  
>Sent: Wednesday, September 11, 2013 8:19 AM
>Subject: Re: [galaxy-dev] Scaling and hardware requirements
>  
>
>
>Hi Peter
>
>
>It's going to be one big machine, running both Galaxy server and the jobs. 
>It's going to be a multi-process configuration. If that idea is terribly bad 
>please let me know so I can give back the feedback.  
>
>
>De novo assembly can also be for the human/mouse genome. 
>
>
>Bests,
>Nikos
>
>
>
>2013/9/11 Peter Cock 
>
>On Wed, Sep 11, 2013 at 1:03 PM, Nikos Sidiropoulos
>> wrote:
>>> Hi all,
>>>
>>> I have a couple of questions regarding a server setup dedicated on Galaxy.
>>>
>>> The idea is to buy a 64 core 256GB RAM server. From my experience I believe
>>> that Galaxy will be able to scale up to 64 cpu's but I would like some more
>>> feedback on this. Also, is 4GB RAM per CPU core enough for NGS data?
>>> (including de-novo assembly)
>>>
>>> Bests,
>>> Nikos
>>
>>Hi Nikos,
>>
>>Is this going to be one server both for running Galaxy (which
>>needs fairly low resources) and running jobs for Galaxy,
>>like de novo assemblies (which need high resources)?
>>
>>i.e. You have one big machine only, no cluster?
>>
>>For de novo assembly the RAM per core/CPU isn't important,
>>it is the total RAM on the machine. How much RAM you
>>need depends on which assembler you use, the organism
>>(both size and also complexity) and the volume of data.
>>
>>What you've described should be fine for bacterial assemblies
>>and smaller eukaryotes - beyond that you'll need to give
>>more details.
>>
>>Peter
>>
> 
>___
>Please keep all replies on the list by using "reply all"
>in your mail client.  To manage your subscriptions to this
>and other Galaxy lists, please use the interface at:
>  http://lists.bx.psu.edu/
>
>To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
>
>___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Scaling and hardware requirements

[Nikos Sidiropoulos]

I'm not sure how easy it would be to setup your Galaxy to only
> allow one de novo assembly at a time - which would seem a
> sensible precaution given you may have multiple users (or the
> same user) trying to run assemblies in parallel.


I guess I could dedicate a handler to run this specific tool and in order
to run it again, the first job will have to be completed.

Thank you for all the help and suggestions!
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Scaling and hardware requirements

[Peter Cock]

On Wed, Sep 11, 2013 at 1:19 PM, Nikos Sidiropoulos
 wrote:
> Hi Peter
>
> It's going to be one big machine, running both Galaxy server and the jobs.
> It's going to be a multi-process configuration. If that idea is terribly bad
> please let me know so I can give back the feedback.

It should be OK for a small number of users, but at some point if
usage increases you will have to switch to a multi-machine setup
(or just a single even bigger machine). If you do have access to
a cluster, you can choose which tools are run locally and which
are run on the cluster.

Don't forget to budget for enough disk storage as well - Galaxy
usage tends to generate a lot of intermediate files which your
users may not routinely delete from their histories.

> De novo assembly can also be for the human/mouse genome.
>

You may not have enough RAM, however I have no personal
experience doing de novo assemblies of mouse/human to guide
you here. Someone else on the list should be able to help, or
since this is a general question try searching online, e.g. at
seqanswers.com

I'm not sure how easy it would be to setup your Galaxy to only
allow one de novo assembly at a time - which would seem a
sensible precaution given you may have multiple users (or the
same user) trying to run assemblies in parallel.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] 'DECLARE CURSOR ... FOR UPDATE/SHARE is not supported'

[hogart]

[galaxy@cluster galaxy-dist]$ hg heads
changeset:   10416:97d020901403
branch:  stable
tag: tip
user:Carl Eberhard 
date:Mon Sep 09 11:20:40 2013 -0400
summary: Fix to dbkey select in library upload when 'unspecified' is
not in dbkey list

changeset:   10411:c42567f43aa7
user:greg
date:Mon Aug 19 13:19:56 2013 -0400
summary: Filter invalid objects when generating the list of
repository_dependencies objects that are associated with a tool shed
repository installed into Galaxy.



2013/9/11 hogart 

> So,
>
> I decided to completely drop the old galaxy psql database (I have a dump
> of it) -
> hogart# drop database galaxy;
>
> and recreate it again -
> [galaxy@cluster ~]$ creatdb
>
> After the launching of the Galaxy it is terminated with the message with
> requesting of upgrade the database, from 13 to 115. But the upgrading of
> database scheme is also failed:
> [galaxy@cluster galaxy-dist]$ sh manage_db.sh upgrade
> /export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/url.py:105:
> SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed
> from 'postgres' to 'postgresql'. The new URL format is
> postgresql[+driver]://:@/
> 13 -> 14...
>
> Migration script to add support for "Pages".
>   1) Creates Page and PageRevision tables
>   2) Adds username column to User table
>
> Traceback (most recent call last):
>   File "./scripts/manage_db.py", line 64, in 
> main( repository=repo, url=db_url )
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/shell.py",
> line 207, in main
> ret = command_func(**kwargs)
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/api.py",
> line 186, in upgrade
> return _migrate(url, repository, version, upgrade=True, err=err,
> **opts)
>   File "", line 2, in _migrate
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/util/__init__.py",
> line 159, in with_engine
> return f(*a, **kw)
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/api.py",
> line 366, in _migrate
> schema.runchange(ver, change, changeset.step)
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/schema.py",
> line 91, in runchange
> change.run(self.engine, step)
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/script/py.py",
> line 145, in run
> script_func(engine)
>   File "lib/galaxy/model/migrate/versions/0014_pages.py", line 55, in
> upgrade
> col.create( User_table, index_name='ix_user_username',
> unique_name='username' )
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/schema.py",
> line 528, in create
> engine._run_visitor(visitorcallable, self, connection, **kwargs)
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
> line 2302, in _run_visitor
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
> line 1972, in _run_visitor
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/ansisql.py",
> line 53, in traverse_single
> ret = super(AlterTableVisitor, self).traverse_single(elem)
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/sql/visitors.py",
> line 106, in traverse_single
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/ansisql.py",
> line 101, in visit_column
> self.execute()
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/ansisql.py",
> line 42, in execute
> return self.connection.execute(self.buffer.getvalue())
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
> line 1449, in execute
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
> line 1628, in _execute_text
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
> line 1698, in _execute_context
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
> line 1691, in _execute_context
>   File
> "/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/default.py",
> line 331, in do_execute
> sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "username" of
> relation "galaxy_user" already exists
>  '\nALTER TABLE galaxy_user ADD username VARCHAR(255)' {}
>
> Since

Re: [galaxy-dev] Scaling and hardware requirements

[Nikos Sidiropoulos]

Hi Peter

It's going to be one big machine, running both Galaxy server and the jobs.
It's going to be a multi-process configuration. If that idea is terribly
bad please let me know so I can give back the feedback.

De novo assembly can also be for the human/mouse genome.

Bests,
Nikos


2013/9/11 Peter Cock 

> On Wed, Sep 11, 2013 at 1:03 PM, Nikos Sidiropoulos
>  wrote:
> > Hi all,
> >
> > I have a couple of questions regarding a server setup dedicated on
> Galaxy.
> >
> > The idea is to buy a 64 core 256GB RAM server. From my experience I
> believe
> > that Galaxy will be able to scale up to 64 cpu's but I would like some
> more
> > feedback on this. Also, is 4GB RAM per CPU core enough for NGS data?
> > (including de-novo assembly)
> >
> > Bests,
> > Nikos
>
> Hi Nikos,
>
> Is this going to be one server both for running Galaxy (which
> needs fairly low resources) and running jobs for Galaxy,
> like de novo assemblies (which need high resources)?
>
> i.e. You have one big machine only, no cluster?
>
> For de novo assembly the RAM per core/CPU isn't important,
> it is the total RAM on the machine. How much RAM you
> need depends on which assembler you use, the organism
> (both size and also complexity) and the volume of data.
>
> What you've described should be fine for bacterial assemblies
> and smaller eukaryotes - beyond that you'll need to give
> more details.
>
> Peter
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Scaling and hardware requirements

[Peter Cock]

On Wed, Sep 11, 2013 at 1:03 PM, Nikos Sidiropoulos
 wrote:
> Hi all,
>
> I have a couple of questions regarding a server setup dedicated on Galaxy.
>
> The idea is to buy a 64 core 256GB RAM server. From my experience I believe
> that Galaxy will be able to scale up to 64 cpu's but I would like some more
> feedback on this. Also, is 4GB RAM per CPU core enough for NGS data?
> (including de-novo assembly)
>
> Bests,
> Nikos

Hi Nikos,

Is this going to be one server both for running Galaxy (which
needs fairly low resources) and running jobs for Galaxy,
like de novo assemblies (which need high resources)?

i.e. You have one big machine only, no cluster?

For de novo assembly the RAM per core/CPU isn't important,
it is the total RAM on the machine. How much RAM you
need depends on which assembler you use, the organism
(both size and also complexity) and the volume of data.

What you've described should be fine for bacterial assemblies
and smaller eukaryotes - beyond that you'll need to give
more details.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Scaling and hardware requirements

[Nikos Sidiropoulos]

Hi all,

I have a couple of questions regarding a server setup dedicated on Galaxy.

The idea is to buy a 64 core 256GB RAM server. From my experience I believe
that Galaxy will be able to scale up to 64 cpu's but I would like some more
feedback on this. Also, is 4GB RAM per CPU core enough for NGS data?
(including de-novo assembly)

Bests,
Nikos
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] 'DECLARE CURSOR ... FOR UPDATE/SHARE is not supported'

[hogart]

So,

I decided to completely drop the old galaxy psql database (I have a dump of
it) -
hogart# drop database galaxy;

and recreate it again -
[galaxy@cluster ~]$ creatdb

After the launching of the Galaxy it is terminated with the message with
requesting of upgrade the database, from 13 to 115. But the upgrading of
database scheme is also failed:
[galaxy@cluster galaxy-dist]$ sh manage_db.sh upgrade
/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/url.py:105:
SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed
from 'postgres' to 'postgresql'. The new URL format is
postgresql[+driver]://:@/
13 -> 14...

Migration script to add support for "Pages".
  1) Creates Page and PageRevision tables
  2) Adds username column to User table

Traceback (most recent call last):
  File "./scripts/manage_db.py", line 64, in 
main( repository=repo, url=db_url )
  File
"/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/shell.py",
line 207, in main
ret = command_func(**kwargs)
  File
"/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/api.py",
line 186, in upgrade
return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File "", line 2, in _migrate
  File
"/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/util/__init__.py",
line 159, in with_engine
return f(*a, **kw)
  File
"/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/api.py",
line 366, in _migrate
schema.runchange(ver, change, changeset.step)
  File
"/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/schema.py",
line 91, in runchange
change.run(self.engine, step)
  File
"/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/script/py.py",
line 145, in run
script_func(engine)
  File "lib/galaxy/model/migrate/versions/0014_pages.py", line 55, in
upgrade
col.create( User_table, index_name='ix_user_username',
unique_name='username' )
  File
"/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/schema.py",
line 528, in create
engine._run_visitor(visitorcallable, self, connection, **kwargs)
  File
"/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
line 2302, in _run_visitor
  File
"/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
line 1972, in _run_visitor
  File
"/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/ansisql.py",
line 53, in traverse_single
ret = super(AlterTableVisitor, self).traverse_single(elem)
  File
"/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/sql/visitors.py",
line 106, in traverse_single
  File
"/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/ansisql.py",
line 101, in visit_column
self.execute()
  File
"/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/ansisql.py",
line 42, in execute
return self.connection.execute(self.buffer.getvalue())
  File
"/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
line 1449, in execute
  File
"/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
line 1628, in _execute_text
  File
"/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
line 1698, in _execute_context
  File
"/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py",
line 1691, in _execute_context
  File
"/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/default.py",
line 331, in do_execute
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column "username" of
relation "galaxy_user" already exists
 '\nALTER TABLE galaxy_user ADD username VARCHAR(255)' {}

Since, it was past release of Galaxy (release_2013.06.03), I pulled the
latest one and repeated the procedure, but the result was the same:

[galaxy@cluster galaxy-dist]$ sh run.sh
...
Exception: Your database has version '13' but this code expects version
'115'.  Please backup your database and then migrate the schema by running
'sh manage_db.sh upgrade'.
[galaxy@cluster galaxy-dist]$ sh manage_db.sh upgrade
/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/url.py:105:
SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed
from 'postgres' to 'postgresql'. The new URL format is
postgresql[+driver]://:@/
13 -> 14...

Migration script to add support for "Pages".
  1) Creates Page a

Re: [galaxy-dev] phylib file type

[Ulf Schaefer]

Dear Peter

Thank you for your email.

Yes. This is the file type I mean. Sorry for the typo.

Any chance there is a pretty class with appropriate sniff functions etc. 
somewhere? Currently it looks a little bit like a hack on my side.

Thanks a lot for your help
Ulf

On 11/09/13 10:16, Peter Cock wrote:
> On Wed, Sep 11, 2013 at 9:46 AM, Ulf Schaefer  wrote:
>> Hello all
>>
>> Has anyone successfully integrated the phylib file type into Galaxy in
>> their local install? It's a standard file type for multiple sequence
>> alignment and I could not possibly be the only one who want to support
>> it in a local install.
>
> Do you mean phylip with a p at the end? If so try this:
> http://toolshed.g2.bx.psu.edu/view/devteam/emboss_datatypes
> http://emboss.sourceforge.net/docs/themes/SequenceFormats.html
>
> Note however there are both strict and relaxed interpretations
> of the PHYLIP standard (the original version imposed taxon
> name limits), and also both interlaced/interleaved and
> sequential forms.
>
> The EMBOSS repository has "phylipnon" for non-interleaved
> (i.e. sequential) PHYLIP format , and "phylip" for either.
>
> Peter
>

**
The information contained in the EMail and any attachments is confidential and 
intended solely and for the attention and use of the named addressee(s). It may 
not be disclosed to any other person without the express authority of Public 
Health England, or the intended recipient, or both. If you are not the intended 
recipient, you must not disclose, copy, distribute or retain this message or 
any part of it. This footnote also confirms that this EMail has been swept for 
computer viruses by Symantec.Cloud, but please re-sweep any attachments before 
opening or saving. http://www.gov.uk/PHE
**

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Problem with installing MAF cached dataset

[Leon Mei]

Dear colleagues,

I am having some trouble installing MAF datasets at our server. When I
run "Extract MAF blocks given a set of genomic intervals", I got this
error message:

"Fatal Error: The MAF source specified (10_WAY_MULTIZ_hg19) appears to
be invalid."

Below is what I did by following the wiki page:
http://wiki.galaxyproject.org/Admin/Datatypes/Add%20MAFs

1, download all mafs for hg19 from
http://hgdownload.cse.ucsc.edu/goldenPath/hg19/multiz46way/maf/

2, index them using "maf_build_index.py --species=hg19,panTro2,
gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2
$file". Our user requested primates only.

3, Here is my maf_index.loc file:
==
10-way multiZ (hg19)10_WAY_MULTIZ_hg19
hg19,panTro2,gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2
  hg19,panTro2,gorGor1,ponAbe2,papHam1,calJac1,tarSyr1,micMur1,otoGar1,rheMac2
   /mnt/galaxyIndices/alignments/maf/chr1.maf,/mnt/galaxyIndices/alignments/m
   
af/chr10.maf,/mnt/galaxyIndices/alignments/maf/chr11.maf,/mnt/galaxyIndices/alignments/maf/chr11_gl000202_random.maf,/mnt/galaxyIndices/ali
   
gnments/maf/chr12.maf,/mnt/galaxyIndices/alignments/maf/chr13.maf,/mnt/galaxyIndices/alignments/maf/chr14.maf,/mnt/galaxyIndices/alignments
   
/maf/chr15.maf,/mnt/galaxyIndices/alignments/maf/chr16.maf,/mnt/galaxyIndices/alignments/maf/chr17.maf,/mnt/galaxyIndices/alignments/maf/ch
   
r17_ctg5_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000203_random.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000204_random.maf,/
   
mnt/galaxyIndices/alignments/maf/chr17_gl000205_random.maf,/mnt/galaxyIndices/alignments/maf/chr17_gl000206_random.maf,/mnt/galaxyIndices/a
   
lignments/maf/chr18.maf,/mnt/galaxyIndices/alignments/maf/chr18_gl000207_random.maf,/mnt/galaxyIndices/alignments/maf/chr19.maf,/mnt/galaxy
   
Indices/alignments/maf/chr19_gl000208_random.maf,/mnt/galaxyIndices/alignments/maf/chr19_gl000209_random.maf,/mnt/galaxyIndices/alignments/
   
maf/chr1_gl000191_random.maf,/mnt/galaxyIndices/alignments/maf/chr1_gl000192_random.maf,/mnt/galaxyIndices/alignments/maf/chr2.maf,/mnt/gal
   
axyIndices/alignments/maf/chr20.maf,/mnt/galaxyIndices/alignments/maf/chr21.maf,/mnt/galaxyIndices/alignments/maf/chr21_gl000210_random.maf
   
,/mnt/galaxyIndices/alignments/maf/chr22.maf,/mnt/galaxyIndices/alignments/maf/chr3.maf,/mnt/galaxyIndices/alignments/maf/chr4.maf,/mnt/gal
   
axyIndices/alignments/maf/chr4_ctg9_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr4_gl000193_random.maf,/mnt/galaxyIndices/alignments/maf/c
   
hr4_gl000194_random.maf,/mnt/galaxyIndices/alignments/maf/chr5.maf,/mnt/galaxyIndices/alignments/maf/chr6.maf,/mnt/galaxyIndices/alignments
   
/maf/chr6_apd_hap1.maf,/mnt/galaxyIndices/alignments/maf/chr6_cox_hap2.maf,/mnt/galaxyIndices/alignments/maf/chr6_dbb_hap3.maf,/mnt/galaxyI
   
ndices/alignments/maf/chr6_mann_hap4.maf,/mnt/galaxyIndices/alignments/maf/chr6_mcf_hap5.maf,/mnt/galaxyIndices/alignments/maf/chr6_qbl_hap
   
6.maf,/mnt/galaxyIndices/alignments/maf/chr6_ssto_hap7.maf,/mnt/galaxyIndices/alignments/maf/chr7.maf,/mnt/galaxyIndices/alignments/maf/chr
   
7_gl000195_random.maf,/mnt/galaxyIndices/alignments/maf/chr8.maf,/mnt/galaxyIndices/alignments/maf/chr8_gl000196_random.maf,/mnt/galaxyIndi
   ces/alignments/maf/chr8_gl000197_random.maf
==

Any hint on which step is wrong or missing is highly appreciate!

Thanks,
Leon





-- 
Hailiang (Leon) Mei
Netherlands Bioinformatics Center
BioAssist NGS Taskforce
 - http://ngs.nbic.nl
Skype: leon_meiMobile: +31 6 41709231
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Deseq2 wrapper question

[Bjoern Gruening]

Am Mittwoch, den 11.09.2013, 10:21 +0100 schrieb Peter Cock:
> On Wed, Sep 11, 2013 at 10:12 AM, Joachim Jacob | VIB |
>  wrote:
> > It is not necessarily bad. If we all test each others tools (dependencies,
> > interface, output,...) we might get to the best of all worlds :-)  (I have
> > planned to test Björns wrapper)
> >
> > One of the points of differentiation is also how 'integrated' a tool needs
> > to be: e.g. Ross' tool for diff expression (combining a complete pipelin),
> > and my tool just on DE with DESeq2. I like to keep things granular on the
> > level of the tools.
> 
> That is a good reason for different wrappers, one for the tool itself,
> and one for a larger pipeline using the same tool. And both can
> declare a binary package dependency via the Tool Shed using
> the binary dependency package Bjoern is working on, which I
> assume will be released on the Test & Main Tool Shed under
> the shared IUC account once ready:
> 
> http://testtoolshed.g2.bx.psu.edu/view/bgruening/package_deseq2_1_0_17


Yes that is right, at the moment I have patches for the toolshed in
preparation for a unified and easy installation of R dependencies, once
that is done it will be under the IUC account. 

I don't see the need for 5 different wrappers. Ross approach deserves an
own wrapper I think. I thought a lot about if we should use it, since it
can compare 3 different approaches for DE, but it comes with some
complexity. We spend some time in trying to get the UI correct and
investigated some possibilities. Currently we have a few guys testing it
and giving comments to our UI idea. If the ownershop is a problem, we
probably can move the wrappers to a different account. But please lets
merge the code base and start communicating about the UI. 

Thanks,
Bjoern 

> Regards,
> 
> Peter
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] 回复: Can not find hg_g1k_v37.fa.

[shenwiyn]

Hi,
Fortunately,I get the human_g1k_v37.fasta.gz in 
ftp://ftp.ncbi.nih.gov/1000genomes/ftp/technical/reference/ .




shenwiyn

发件人： shenwiyn
发送时间： 2013-09-05 10:03
收件人： galaxy-dev
主题： Can not find hg_g1k_v37.fa.
Hi everyone,
I want to use the hg_g1k_v37.fa on my local Galaxy,but I can not find this 
data.When I use the command : 
rsync -avzP 
rsync://datacache.g2.bx.psu.edu/indexes/hg_g1k_v37/seq/hg_g1k_v37.fa 
the error occurred:
receiving incremental file list
rsync: link_stat "/hg_g1k_v37/seq/hg_g1k_v37.fa" (in indexes) failed: No such 
file or directory (2)

sent 4 bytes  received 6 bytes  4.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 
23) at main.c(1532) [receiver=3.0.6]
Please any suggestions would be most welcome. 
Thanks.



shenwiyn___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Conditional and validator: refresh_on_change without validation

[Kahlke Tim]

Hi

I'm using the  tag for simple sanity checks of a tool input. But I 
realized that any conditional in the form will automatically call 
refresh_on_change which also validates all fields and prints errror messages.

So questioin: is there a more galaxy-like way of checking input parameters? I 
am aware of the possibility to do sanity checks through custom code 
(http://wiki.galaxyproject.org/Admin/Tools/Custom%20Code) but the page states 
it should only be done if there is no other way.

Regards,
Tim
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Deseq2 wrapper question

[Peter Cock]

On Wed, Sep 11, 2013 at 10:12 AM, Joachim Jacob | VIB |
 wrote:
> It is not necessarily bad. If we all test each others tools (dependencies,
> interface, output,...) we might get to the best of all worlds :-)  (I have
> planned to test Björns wrapper)
>
> One of the points of differentiation is also how 'integrated' a tool needs
> to be: e.g. Ross' tool for diff expression (combining a complete pipelin),
> and my tool just on DE with DESeq2. I like to keep things granular on the
> level of the tools.

That is a good reason for different wrappers, one for the tool itself,
and one for a larger pipeline using the same tool. And both can
declare a binary package dependency via the Tool Shed using
the binary dependency package Bjoern is working on, which I
assume will be released on the Test & Main Tool Shed under
the shared IUC account once ready:

http://testtoolshed.g2.bx.psu.edu/view/bgruening/package_deseq2_1_0_17

Regards,

Peter

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] phylib file type

[Peter Cock]

On Wed, Sep 11, 2013 at 9:46 AM, Ulf Schaefer  wrote:
> Hello all
>
> Has anyone successfully integrated the phylib file type into Galaxy in
> their local install? It's a standard file type for multiple sequence
> alignment and I could not possibly be the only one who want to support
> it in a local install.

Do you mean phylip with a p at the end? If so try this:
http://toolshed.g2.bx.psu.edu/view/devteam/emboss_datatypes
http://emboss.sourceforge.net/docs/themes/SequenceFormats.html

Note however there are both strict and relaxed interpretations
of the PHYLIP standard (the original version imposed taxon
name limits), and also both interlaced/interleaved and
sequential forms.

The EMBOSS repository has "phylipnon" for non-interleaved
(i.e. sequential) PHYLIP format , and "phylip" for either.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] How to configure and set up the Circos on my local Galaxy?

[沈维燕]

Hi Jeremy Goecks,
Thank you very much for your help.My problem has been solved successfully
and the *visualization of Circster works on my local Galaxy.
*


2013/9/3 Jeremy Goecks 

> Please take a look at this page for setting up visualizations:
> http://wiki.galaxyproject.org/Visualization%20Setup
>
> Let us know if you have any questions.
>
> Best,
> J.
>
> On Sep 1, 2013, at 9:37 PM, shenwiyn wrote:
>
> Hi ,
> I found that the *visualization of Circos works greatly in Galaxy
> main,but failed on my local Galaxy.I want to find out some information
> about  configuring and setting up the Circos on local Galaxy,I saw nothing
> in
> http://wiki.galaxyproject.org/New%20Visualization%20Page#Circos_plot_.28aka_Circster.29
>  .Please any suggestions would be most welcome.*
>
> --
> shenwiyn
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Deseq2 wrapper question

[Joachim Jacob | VIB |]

It is not necessarily bad. If we all test each others tools 
(dependencies, interface, output,...) we might get to the best of all 
worlds :-)  (I have planned to test Björns wrapper)


One of the points of differentiation is also how 'integrated' a tool 
needs to be: e.g. Ross' tool for diff expression (combining a complete 
pipelin), and my tool just on DE with DESeq2. I like to keep things 
granular on the level of the tools.



Cheers,
Joachim

Joachim Jacob.
Contact details: http://www.bits.vib.be/index.php/about/80-team


On 09/11/2013 10:56 AM, Peter Cock wrote:

Wow - that makes at least five Deseq2 wrappers for Galaxy
available or in progress :(

Bjoern's wrapper on the (test) tool shed,
http://testtoolshed.g2.bx.psu.edu/view/bgruening/deseq2

Ross' combined wrapper for edgeR, DESeq2 and voom in one tool:
http://testtoolshed.g2.bx.psu.edu/view/fubar/differential_count_models

Joachim's wrapper on the Tool Shed
http://toolshed.bits.vib.be/view/joachim/deseq2

Vipin's which is soon to be released to the (Test?) Tool Shed,
https://github.com/ratschlab/oqtans_tools/tree/master/DESeq2

And Rui's is working on one too.

This does seem like duplicated effort & a source of confusion
for end users and Galaxy administrators (a problem not unique
to deseq2, but affecting many Galaxy wrappers).

I appreciate there will be different needs, and one wrapper
may not suit all, but I would prefer if the default behaviour
for Galaxy tool wrapper authors was to collaborate on one
good wrapper rather than writing competing ones.

When I started work on a new wrapper I tried to announce
this on the mailing list to find out if anyone else was already
tackling the same tool - and that seemed to work quite well.
Of course, the volume of emails on galaxy-dev has grown
quite a lot over the last few years so that may not be as
effective, but the archives should be searchable.

Perhaps we need to improve communication in some way?
One option would be to encourage greater use of the Test
Tool Shed for works in progress to give them visibility?
You could even have the Tool Shed itself require a search
step before creating a new repository to avoid accidental
duplication of effort? Or maybe a wiki page of wrappers
in progress?

(And maybe we should split this into a new thread)

Regards,

Peter


On Wed, Sep 11, 2013 at 8:59 AM, Joachim Jacob | VIB |
 wrote:

Hi Rui,

I should provide a 'sample metadata table', a tabular file with one column
the names of your samples, the other column the associated metadata (e.g.
type).

This is how I've done it in my DESeq2 wrapper. :-) Feel free to check the
interface at http://toolshed.bits.vib.be/view/joachim/deseq2


Cheers,
Joachim

Joachim Jacob
Contact details: http://www.bits.vib.be/index.php/about/80-team



On 09/11/2013 01:15 AM, ruiwang.sz wrote:

Hi All,

we are making a wrapper for deseq2, and there is a step where
control/experimental conditions
need to be determined. For example, it would be a column like the
following in the input file

Type
CTC
CTC
LM
LM
PT
PT

and we want the select list to contain

CTC
LM
PT

In other words, we want to filter out the 'Type' and have only distinct
ones in the select list. We could
make it in two steps, generate a intermediate file for this but it would
be nice that we could directly
retrieve the distinct types to make the list. I took a look at the tool
config wiki, but didn't see anything
(maybe because it was a quick scan). Does anyone have any tip on this?
we'll really appreciate.

Thanks,
Rui


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Mail is not configured for this Galaxy instance.

[shenwiyn]

Hi Hans-Rudolf,
Apologies for the delay.Thank you very nuch for your help.My problem solved 
successfully after 'smtp_server' settings in my "universe_wsgi.ini" file.





shenwiyn

From: Hans-Rudolf Hotz
Date: 2013-07-31 23:40
To: shenwiyn
CC: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Mail is not configured for this Galaxy instance.
Hi Shenwiyn

You need to specify a SMTP server. Have a look at the 'smtp_server' 
settings in your "universe_wsgi.ini" file.


Regards, Hans-Rudolf




On 07/31/2013 04:19 PM, shenwiyn wrote:
> Hi everyone,
> In my local Galaxy,when I want to reset my Login password,some error
> occurre
> :"Mail is not configured for this Galaxy instance. Please contact an 
> administrator.
> "Could anyone tell me some imformation about how to configure the mail
> for my Galaxy instance?
> Thank you very much.
> 
> shenwiyn
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>http://galaxyproject.org/search/mailinglists/
>___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Deseq2 wrapper question

[Peter Cock]

Wow - that makes at least five Deseq2 wrappers for Galaxy
available or in progress :(

Bjoern's wrapper on the (test) tool shed,
http://testtoolshed.g2.bx.psu.edu/view/bgruening/deseq2

Ross' combined wrapper for edgeR, DESeq2 and voom in one tool:
http://testtoolshed.g2.bx.psu.edu/view/fubar/differential_count_models

Joachim's wrapper on the Tool Shed
http://toolshed.bits.vib.be/view/joachim/deseq2

Vipin's which is soon to be released to the (Test?) Tool Shed,
https://github.com/ratschlab/oqtans_tools/tree/master/DESeq2

And Rui's is working on one too.

This does seem like duplicated effort & a source of confusion
for end users and Galaxy administrators (a problem not unique
to deseq2, but affecting many Galaxy wrappers).

I appreciate there will be different needs, and one wrapper
may not suit all, but I would prefer if the default behaviour
for Galaxy tool wrapper authors was to collaborate on one
good wrapper rather than writing competing ones.

When I started work on a new wrapper I tried to announce
this on the mailing list to find out if anyone else was already
tackling the same tool - and that seemed to work quite well.
Of course, the volume of emails on galaxy-dev has grown
quite a lot over the last few years so that may not be as
effective, but the archives should be searchable.

Perhaps we need to improve communication in some way?
One option would be to encourage greater use of the Test
Tool Shed for works in progress to give them visibility?
You could even have the Tool Shed itself require a search
step before creating a new repository to avoid accidental
duplication of effort? Or maybe a wiki page of wrappers
in progress?

(And maybe we should split this into a new thread)

Regards,

Peter


On Wed, Sep 11, 2013 at 8:59 AM, Joachim Jacob | VIB |
 wrote:
> Hi Rui,
>
> I should provide a 'sample metadata table', a tabular file with one column
> the names of your samples, the other column the associated metadata (e.g.
> type).
>
> This is how I've done it in my DESeq2 wrapper. :-) Feel free to check the
> interface at http://toolshed.bits.vib.be/view/joachim/deseq2
>
>
> Cheers,
> Joachim
>
> Joachim Jacob
> Contact details: http://www.bits.vib.be/index.php/about/80-team
>
>
>
> On 09/11/2013 01:15 AM, ruiwang.sz wrote:
>>
>> Hi All,
>>
>> we are making a wrapper for deseq2, and there is a step where
>> control/experimental conditions
>> need to be determined. For example, it would be a column like the
>> following in the input file
>>
>> Type
>> CTC
>> CTC
>> LM
>> LM
>> PT
>> PT
>>
>> and we want the select list to contain
>>
>> CTC
>> LM
>> PT
>>
>> In other words, we want to filter out the 'Type' and have only distinct
>> ones in the select list. We could
>> make it in two steps, generate a intermediate file for this but it would
>> be nice that we could directly
>> retrieve the distinct types to make the list. I took a look at the tool
>> config wiki, but didn't see anything
>> (maybe because it was a quick scan). Does anyone have any tip on this?
>> we'll really appreciate.
>>
>> Thanks,
>> Rui
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>http://galaxyproject.org/search/mailinglists/
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Bismark error

[Nikos Sidiropoulos]

Hi again

You were right. Restarting all processes solved the problem and Bismark in
now up and running.

Bests,
Nikos


2013/9/4 Nate Coraor 

> On Sep 4, 2013, at 10:08 AM, Nikos Sidiropoulos wrote:
>
> > Hi Nate
> >
> > Yes, it is a multiprocess setup. But is it because we have more than one
> web-servers, handlers or both?
> >
> > Because if it's just the web-servers I could just scale it down to one
> since it doesn't seem necessary with our number of users. It's would be
> nice not to have to restart even for the tool-shed tools.
>
> The problem in this case is that one of the web server processes loaded
> the shed tool but none of the handlers did.  Scaling back to a single web
> process won't solve the problem since the handler(s) will still need to
> load the tool.  Unfortunately you'll have to restart all processes (except,
> technically, the one web process which happened to install the tool).  The
> only way to avoid the restart is to have a single process for all tasks
> (web serving and job handling), which comes at a performance penalty.
>
> --nate
>
> >
> > Bests,
> > Nikos
> >
> >
> > 2013/9/4 Nate Coraor 
> > On Sep 4, 2013, at 9:17 AM, Nikos Sidiropoulos wrote:
> >
> > > Hi Bjørn
> > >
> > > > that does not look like a bismark error. Is it happen with other
> tools
> > > > as well?
> > >
> > > No, I have only experienced it with Bismark.
> > >
> > > > Not sure, sorry. But you should migrate to the new job configuration,
> > > > better sooner than later.
> > >
> > > I am starting to get that idea. There are jobs still running on the
> server. When they're done I'll migrate it and get back to you if the
> problem is fixed or not.
> >
> > Hi Nikos,
> >
> > The old-style configuration is still supported for now.  I suspect this
> may be a problem of the tool not loading in the handler application.  Are
> you running a multiprocess Galaxy setup?  If so, you will need to restart
> all Galaxy processes after installing tools from the tool shed.
> >
> > --nate
> >
> > >
> > > Bests,
> > > Nikos
> > > ___
> > > Please keep all replies on the list by using "reply all"
> > > in your mail client.  To manage your subscriptions to this
> > > and other Galaxy lists, please use the interface at:
> > >  http://lists.bx.psu.edu/
> > >
> > > To search Galaxy mailing lists use the unified search at:
> > >  http://galaxyproject.org/search/mailinglists/
> >
> >
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] phylib file type

[Ulf Schaefer]

Hello all

Has anyone successfully integrated the phylib file type into Galaxy in 
their local install? It's a standard file type for multiple sequence 
alignment and I could not possibly be the only one who want to support 
it in a local install.

I saw the instructions on the wiki for adding new file types. I would 
guess that in addition to the changes in the datatypes_conf.xml I would 
have to derive a new class in galaxy.datatypes.sequence from the 
existing Alignment class.

Is there a commit where this is already included or will I actually have 
to code it myself? How can I find out?

Thanks a lot
Ulf

PS.: hg tip
changeset:   10201:ebe87051fadf
tag: tip
parent:  10199:8bf64d933704
user:Dannon Baker 
date:Tue Jul 02 10:48:31 2013 -0400
summary: Fix two more downgrade invocations to accept the 
migrate_engine parameter

**
The information contained in the EMail and any attachments is confidential and 
intended solely and for the attention and use of the named addressee(s). It may 
not be disclosed to any other person without the express authority of Public 
Health England, or the intended recipient, or both. If you are not the intended 
recipient, you must not disclose, copy, distribute or retain this message or 
any part of it. This footnote also confirms that this EMail has been swept for 
computer viruses by Symantec.Cloud, but please re-sweep any attachments before 
opening or saving. http://www.gov.uk/PHE
**

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Deseq2 wrapper question

[Joachim Jacob | VIB |]

Hi Rui,

I should provide a 'sample metadata table', a tabular file with one 
column the names of your samples, the other column the associated 
metadata (e.g. type).


This is how I've done it in my DESeq2 wrapper. :-) Feel free to check 
the interface at http://toolshed.bits.vib.be/view/joachim/deseq2



Cheers,
Joachim

Joachim Jacob
Contact details: http://www.bits.vib.be/index.php/about/80-team


On 09/11/2013 01:15 AM, ruiwang.sz wrote:

Hi All,

we are making a wrapper for deseq2, and there is a step where 
control/experimental conditions
need to be determined. For example, it would be a column like the 
following in the input file


Type
CTC
CTC
LM
LM
PT
PT

and we want the select list to contain

CTC
LM
PT

In other words, we want to filter out the 'Type' and have only 
distinct ones in the select list. We could
make it in two steps, generate a intermediate file for this but it 
would be nice that we could directly
retrieve the distinct types to make the list. I took a look at the 
tool config wiki, but didn't see anything
(maybe because it was a quick scan). Does anyone have any tip on this? 
we'll really appreciate.


Thanks,
Rui


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] ZFS storage recommendations

[Joachim Jacob | VIB |]

Thank you all for the reactions.

Some details about my current ZFS and Galaxy setup:

- Galaxy runs as a single virtual machine, with currently 20 cores, 80GB 
RAM. Will be 32 cores and about 160GB RAM soon.

- The postgres database is on the virtual machine itself.
- The 'files' and 'job_working_dir' are on an NFS exported directory, 
hosted on the host machine of the guest.

- The NFS exported directory is an raidz1 dataset.
- The raidsz1 runs on 7 550GB SAS disks, which are (unfortunately) 
controlled by a RAID hardware controller, but passed as RAID0 (JBOD not 
available). So raidz1 runs on 7 RAID0 disks (with settings in the 
hardware controller PERC H700: no read ahead, write through, 8 KB stripe 
size, disk cache policy enabled).

- Compression and deduplication is enabled.
- The directory on which the zfs dataset is mounted, is exported using 
the native linux NFS daemon to the Galaxy virtual machine. The 'zfs 
sharenfs' did not work (ownerships not set correctly - perhaps need some 
more investigation, but I found several times reports about sharenfs 
option in ZFS in linux is not behaving well...).


The numbers:
- my initial files database (ext4 on RAID5) is 3.0TB in size. On ZFS, 
with compression and deduplication, this database is *1.8TB *(-40%).
- Did not yet provide a SLOG to host the ZIL and a L2ARC, since I have a 
cleared picture about the performance I can get. Would you advise 
preference over ZIL on a SLOG, or go for a SSD to host the L2ARC.
- The cost for this performance of ZFS in storage is RAM: currently 
continuously using*284GB RAM* for ZFS!
- The write and read speed is from the Galaxy VM over NFS is *~40MB/s 
and ~100MB/s* (tested by simply copying over rsync - I still need to 
check the presentation and scripts of Anne Black-Ziegelbein). This is a 
66% decrease in previously achieved write and read speed (ext4 on 
hardware RAID5), but I feel that the benefits (deduplication, backing up 
via snapshots, data integrity,) outweigh this IO performance.


(I am setting this ZFS up on a new server (well, actually 2 years old 
now, has served on another project well))


Currently our Galaxy uses this zfs with success!

For your interest, my settings on the 'galaxydb' zfs tank below. (I was 
wondering if here some more wizardry can be applied).



[root@r910bits ~]# *zfs get all tank/galaxydb*
NAME   PROPERTY  VALUE  SOURCE
tank/galaxydb  type  filesystem -
tank/galaxydb  creation  Mon Sep  9 12:44 2013  -
tank/galaxydb  used  1.81T  -
tank/galaxydb  available 1.66T  -
tank/galaxydb  referenced1.81T  -
tank/galaxydb  compressratio 1.66x  -
tank/galaxydb  mounted   yes-
tank/galaxydb  quota none default
tank/galaxydb  reservation   none default
tank/galaxydb  recordsize128K default
tank/galaxydb  mountpoint/mnt/galaxydb  local
tank/galaxydb  sharenfs  rw=@galaxy local
tank/galaxydb  checksum  on default
tank/galaxydb  compression   lzjb   local
tank/galaxydb  atime on default
tank/galaxydb  devices   on default
tank/galaxydb  exec  on default
tank/galaxydb  setuidon default
tank/galaxydb  readonly  off default
tank/galaxydb  zoned off default
tank/galaxydb  snapdir   hidden default
tank/galaxydb  aclinheritrestricted default
tank/galaxydb  canmount  on default
tank/galaxydb  xattr on default
tank/galaxydb  copies1 default
tank/galaxydb  version   5  -
tank/galaxydb  utf8only  off-
tank/galaxydb  normalization none   -
tank/galaxydb  casesensitivity   sensitive  -
tank/galaxydb  vscan off default
tank/galaxydb  nbmandoff default
tank/galaxydb  sharesmb  off default
tank/galaxydb  refquota  none default
tank/galaxydb  refreservationnone default
tank/galaxydb  primarycache  all default
tank/galaxydb  secondarycacheall default
tank/galaxydb  usedbysnapshots   0  -
tank/galaxydb  usedbydataset 1.81T  -
tank/galaxydb  usedbychildren0  -
tank/galaxydb  usedbyrefreservation  0  -
tank/galaxydb  logbias   latency default
tank/galaxydb  dedup on local
tank/galaxydb  mlslabel  none default
tank/galaxydb  sync  standard default
tank/galaxydb  refcompressratio  1.66x  -
tank/galaxydb  written   1.81T  -
ta

Re: [galaxy-dev] Looking for brave testers. New barcode splitter outputing splitted datasets directly to the history.

[Jennifer Jackson]

Hi Carlos,

Nice this is being worked on.

Just wanted to mention a quick tip for loading barcode splitter output: 
1- right click on the output name & copy the link
2- open 'Get Data -> Upload File' tool, and paste link into box. Execute.
3- Repeat for all.

Data will load as dataset, no downloading/uploading required. I find that 
pasting all the links into a temp text file, then copy/paste all at once into 
the Upload tool speeds this up - as it then loads all in one go.

Not perfect, but maybe helpful for now.

Best,
Jen
Galaxy team

On Sep 10, 2013, at 11:49 AM, Carlos Borroto  wrote:

> Cool, I got a tweet about this tool from @GalaxyProject[1].
> 
> To further explain what I'm trying to accomplish here, as I realized
> not everybody might know what using "Multiple Output Files" and
> specifically "Number of Output datasets cannot be determined until
> tool run"[2] entails.
> 
> The current Barcode Splitter available on Galaxy Main and based on
> FASTX-toolkit by Assaf Gordon, makes all output files accessible
> through HTML links. This is not very convenient, as if you want to
> use, and you probably do, these outputs in a downstream analysis
> inside Galaxy, your only solution right now is to download the linked
> files in the HTML output and manually re-import then into Galaxy. The
> tool I wrote includes the option of writing the output files(splitted
> FASTA or FASTQ files) with a naming convention that can be used with
> Galaxy's "Multiple Output Files" so all results files are
> automatically added to your history. I believe you still can't easily
> use this tool upstream in a workflow. As I far as I can tell tools
> without a known number of outputs can't be used upstream in workflows.
> I do think you can accomplish some automation using the API, although
> I haven't tested this yet.
> 
> [1]https://twitter.com/galaxyproject/status/377497531745595392
> [2]http://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#Number_of_Output_datasets_cannot_be_determined_until_tool_run
> 
> Best,
> Carlos
> 
> On Fri, Sep 6, 2013 at 1:58 PM, Carlos Borroto  
> wrote:
>> Hi,
>> 
>> I was wondering if I could get someone to test this new barcode
>> splitter I wrote. The main reason for me to duplicate the already
>> great fastx-toolkit based splitter, is so I can use galaxy's multiple
>> output capabilities.
>> 
>> You can find this tool in the testtoolshed for now(after some more
>> testing I will moved to the main toolshed):
>> http://testtoolshed.g2.bx.psu.edu/view/cjav/split_by_barcode
>> 
>> Hopefully I got Galaxy's tool dependency system right(it works on my
>> box, not that this says much) and installing this tool should be quite
>> easy.
>> 
>> I have to say big thanks to Biopython and this[1] anonymous soul for
>> making it quite easy to write the actual code doing the heavy lifting.
>> 
>> [1]https://gist.github.com/dgrtwo/3725741
>> 
>> Cheers,
>> Carlos
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/