Re: [galaxy-dev] Public toolshed giving "internal server error"

[Björn Grüning]

Hi Peter,

can you try again? I'm able to browse the ToolShed and reset metatada
for example.

Cheers,
Bjoern

Am 19.11.2014 um 10:36 schrieb Peter Briggs:
> Hello
> 
> I'm trying to make a new repository on the public toolshed at
> https://toolshed.g2.bx.psu.edu/
> but I keep getting the "internal server error" page.
> 
> I was also unable to log out, or even to see the front page when trying
> to access it from a different browser.
> 
> Is anyone else having this problem?
> 
> Thanks & best wishes
> 
> Peter
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] tools version (eg: cufflinks) and bowtie indices file

[Björn Grüning]

Hi Elizabeth,

Am 09.11.2014 um 20:26 schrieb Elizabeth Chia:
> Hi there
> 
> 1. Why is it that even after installing cufflinks 2.1.1 from the toolshed,
> it doesn't get reflected when I clicked on the tool from the side panel? It
> still shows 0.0.5 and 0.0.7. Think it's the same with others too (eg:
> bowtie)

The Galaxy tool versions do not correspond to the cufflinks program
version. I think 0.0.7 is the most recent version.

> 2. I've modified the bowtie_indices.loc file under tool-data but the
> entries doesn't get reflected on the Map with Bowtie for Illumina (select a
> reference genome). I did the same for the bowtie2_indices.loc file and this
> one got reflected under Bowtie2.

Have you installed data_managers?
Please have also a look at:
https://wiki.galaxyproject.org/Admin/DataIntegration

Hope this helps a little bit,
Bjoern

> Been a little frustrating. Can anyone please advise?
> 
> Many thanks.
> 
> Best regards
> liz
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tool shed environment dependency - HOW TO do this?

[Björn Grüning]

I agree with Nicola here! :)

Am 06.11.2014 um 17:48 schrieb Nicola Soranzo:
> Hi Pieter,
> you should probably add:
> 
> 
>owner="pieterlukasse">
> 
>   
> 
> 
> before using the R_ROOT_DIR environment variable.
> 
> Best,
> Nicola
> 
> Il 2014-11-06 11:28 Lukasse, Pieter ha scritto:
>> Hi Bjoern,
>>
>> I tried your suggestion, but it didn't work. So now I have split up
>> the R part from the Bioconductor part (blue and yellow below,
>> respectively). But the problem is that the  part executes
>> before the  part is finished and the $R_ROOT_DIR reference
>> also doesn't seem to work as I get the following error : " /bin/sh: 1:
>> /bin/Rscript: not found " . This R_ROOT_DIR is set by the 
>> script in my other tool_dependency.xml but is not visible in the
>> tool_dependency.xml below. But maybe I misunderstood what you meant by
>> "You can now access R_ROOT_DIR from any other tool_dependency file
>> ..."?
>>
>> 
>>
>> 
>>
>>  
>>
>>  > name="prims_metabolomics_dependencies" owner="pieterlukasse"
>> prior_installation_required="True"
>> toolshed="http://testtoolshed.g2.bx.psu.edu"; />
>>
>>  
>>
>>  
>>
>>  
>>
>>  wget -P $INSTALL_DIR
>>
>> http://testtoolshed.g2.bx.psu.edu/repos/pieterlukasse/prims_metabolomics/raw-file/tip/INSTALL.r
>>
>>
>>
>>  $R_ROOT_DIR/bin/Rscript
>> $INSTALL_DIR/INSTALL.r
>>
>>  
>>
>>  
>>
>>  
>>
>>  
>>
>>  This dependency:
>>
>>  Ensures R 3.1.1 installation is triggered (via dependency).
>>
>>  Ensures Bioconductor 3.0 and package metaMS, multtest and snow are
>> installed.
>>
>>  
>>
>> 
>>
>> Thanks,
>>
>> Pieter
>>
>> -Original Message-
>>  From: galaxy-dev-boun...@lists.bx.psu.edu
>> [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Lukasse,
>> Pieter
>>  Sent: donderdag 6 november 2014 10:02
>>  To: 'Björn Grüning'; galaxy-dev@lists.bx.psu.edu; Dave Bouvier
>>  Subject: Re: [galaxy-dev] Tool shed environment dependency - HOW TO
>> do this?
>>
>> Hi Bjoern,
>>
>> Thanks for this reply. I will have to try referring to the R_ROOT_DIR
>> from another tool_dependency. If it doesn't work I'll come back ;)
>>
>> Regards,
>>
>> Pieter
>>
>> -Original Message-
>>
>> From: Björn Grüning [mailto:bjoern.gruen...@gmail.com [1]]
>>
>> Sent: donderdag 30 oktober 2014 11:25
>>
>> To: galaxy-dev@lists.bx.psu.edu [2]; Lukasse, Pieter; Dave Bouvier
>>
>> Subject: Re: [galaxy-dev] Tool shed environment dependency - HOW TO do
>> this?
>>
>> Hi Pieter,
>>
>> Am 30.10.2014 um 11:15 schrieb Lukasse, Pieter:
>>
>>> Hi,
>>
>>>
>>
>>> I have a Bioconductor package which depends on R 3.1.1 , so I think
>> I cannot use the "setup_r_environment" trick.
>>
>> Why? Do you need a new release? The latest current release in the Tool
>> Shed is R 3.1.0. If this is not enough we need to poke Dave to build a
>> new package. But this way would be the preferred way.
>>
>>> I already got a tool_dependency.xml working that installs R 3.1.1
>> and the necessary Bioconductor packages using bioclite method (see
>> attachment). Now my question is:
>>
>>>
>>
>>>
>>
>>> * I would like to split up this into two steps as I don't want to
>> trigger the compilation of new R environment every time I when I need
>> to just update the
>>
>>> Bioconductor packagethe question is: how to do such things in
>> general? How can I access the INSTALL_DIR of another tool from within
>> another tool_dependency.xml? If I can do this, then my problem is
>> solved.
>>
>> If you really want to build your R packages by your own. Have a look
>> at
>>
>> https://github.com/bgruening/galaxytools/blob/master/packages/package_r_3_0_3/tool_dependencies.xml
>>
>> [3]
>>
>> You will find an exhaustive installation instruction how to build R
>> properly. Also note that we export a variable called R_ROOT_DIR that
>> is set to $INSTALL_DIR. You can now access R_ROOT_DIR from any other
>> tool_dependency file ...
>>
>> Hope this helps,
>>
>> Bjoern
>>
>> P.S. Do you mind asking this question on biostar aga

Re: [galaxy-dev] Unable to add R package dependency to tool_dependencies.xml

[Björn Grüning]

Hi Bruno,

it is currently not possible to specify dependencies from other Tool
Sheds. So you can only depend on packages that are in the same Tool Shed
as your tool.

Is this your main issue?

Greg has some nice blog posts about the Tool Shed and how to set up a
local test Tool Shed with other packages, like the one from foobar.

http://gregvonkuster.org/

Cheers,
Bjoern


> I'm attempting to create a tool dependency definition file for ExomeCNV,
> which notably depends on R (I'm trying to use the package_r_3_1_1
> repository). However, I can't seem to set up the dependency properly. See
> attached for my tool_dependencies.xml. Also, see here 
> for the error message when I attempt to upload the tool_dependencies.xml
> file to the Tool Shed.
> 
> One of the possible issues is that the toolshed repository tag attribute
> for specifying an external Tool Shed isn't being considered. This is being
> done in a local Tool Shed installation that doesn't include
> the package_r_3_1_1 repository. Therefore, I must refer to the Test Tool
> Shed. This is inspired by a tool definition
> 
> written by Björn and the  tag description
> .
> 
> Best regards,
> Bruno
> ​
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tool shed environment dependency - HOW TO do this?

[Björn Grüning]

Hi Pieter,

Am 30.10.2014 um 11:15 schrieb Lukasse, Pieter:
> Hi,
> 
> I have a Bioconductor package which depends on R 3.1.1 , so I think I cannot 
> use the "setup_r_environment" trick.

Why? Do you need a new release? The latest current release in the Tool
Shed is R 3.1.0. If this is not enough we need to poke Dave to build a
new package. But this way would be the preferred way.

> I already got a tool_dependency.xml working that installs R 3.1.1 and the 
> necessary Bioconductor packages using bioclite method (see attachment). Now 
> my question is:
> 
> 
> * I would like to split up this into two steps as I don't want to 
> trigger the compilation of new R environment every time I when I need to just 
> update the 
>Bioconductor packagethe question is: how to do such things in general? How 
>can I access the INSTALL_DIR of another tool from within another 
>tool_dependency.xml? If I can do this, then my problem is solved.

If you really want to build your R packages by your own. Have a look at
https://github.com/bgruening/galaxytools/blob/master/packages/package_r_3_0_3/tool_dependencies.xml

You will find an exhaustive installation instruction how to build R
properly. Also note that we export a variable called R_ROOT_DIR that is
set to $INSTALL_DIR. You can now access R_ROOT_DIR from any other
tool_dependency file ...

Hope this helps,
Bjoern

P.S. Do you mind asking this question on biostar again, I think this is
a very nice question that can be of interest to a lot more people.



> Thanks!
> 
> Pieter Lukasse
> Wageningen UR, Plant Research International
> Department of Bioinformatics (Bioscience)
> Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB,
> Wageningen, the Netherlands
> T: +31-317481122;
> M: +31-628189540;
> skype: pieter.lukasse.wur
> http://www.pri.wur.nl
> 
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Dependencies not working on toolshed?

[Björn Grüning]

Hi Pieter,

think more about it from the actual name. tool_dependencies.xml for your
tools (wrappers) and repository_dependencies.xml for your repository.
Thinks that are only indirectly or not at all needed by your wrapper.

Glad it's now working for you!
Bjoern

Am 24.10.2014 um 23:17 schrieb Lukasse, Pieter:
> Hi Bjoern, John,
> 
> thanks, this worked! 
> 
> So: I have to add a repository dependency to my tool_dependencies.xml file 
> and NOT to my repository_dependencies.xmlI must say it is somewhat 
> confusing. Somehow I had this idea that tool_dependencies.xml was for 
> external tools But I'm happy it finally works!
> 
> Thanks again,
> 
> Pieter.
> 
> -Original Message-
> From: Björn Grüning [mailto:bjoern.gruen...@gmail.com] 
> Sent: vrijdag 24 oktober 2014 16:23
> To: Lukasse, Pieter; galaxy-dev@lists.bx.psu.edu
> Subject: Re: [galaxy-dev] Dependencies not working on toolshed?
> 
> Pieter,
> 
> please try to add your dependencies to a tool_dependencies.xml file not the 
> repository_dependencies.xml file.
> 
> repository_dependencies.xml is for datatypes and for workflow dependencies, 
> such things that need to be there, but that are not referenced from the tool.
> 
> Hope this helps,
> Bjoern
> 
> Am 24.10.2014 um 16:18 schrieb Lukasse, Pieter:
>> Hi ,
>>
>> I am trying to get a dependency to work in practice (when running a tool 
>> after the installation). But I keep getting the following message when 
>> running the tool:
>>
>>
>>   [sshexec] galaxy.tools.deps DEBUG 2014-10-24 16:04:15,514 Building 
>> dependency shell command for dependency 'R_bioc_metams'
>>   [sshexec] galaxy.tools.deps WARNING 2014-10-24 16:04:15,514 Failed 
>> to resolve dependency on 'R_bioc_metams', ignoring
>>
>>
>> This is the tool requirements section:
>>
>>
>>   > version="3.1.1">R_bioc_metams
>>
>>
>> This is the repository_dependencies.xml :
>>
>> 
>>   http://testtoolshed.g2.bx.psu.edu";
>>   name="prims_metabolomics_dependencies" owner="pieterlukasse" 
>> changeset_revision="71356c62e5cd" /> 
>>
>> This is the package name of the package that is being referred:
>>
>> 
>>  
>>   
>>   
>> 
>>
>> I can't find the problem. Any ideas? I have updated to the latest Galaxy 
>> version (today) and removed both items (with option to remove files) and 
>> installed them again. Am I facing a bug here?
>>
>> Thanks,
>>
>> Pieter Lukasse
>> Wageningen UR, Plant Research International Department of 
>> Bioinformatics (Bioscience) Wageningen Campus, Building 107, 
>> Droevendaalsesteeg 1, 6708 PB, Wageningen, the Netherlands
>> T: +31-317481122;
>> M: +31-628189540;
>> skype: pieter.lukasse.wur
>> http://www.pri.wur.nl<http://www.pri.wur.nl/>
>>
>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this and other 
>> Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Dependencies not working on toolshed?

[Björn Grüning]

Pieter,

please try to add your dependencies to a tool_dependencies.xml file not
the repository_dependencies.xml file.

repository_dependencies.xml is for datatypes and for workflow
dependencies, such things that need to be there, but that are not
referenced from the tool.

Hope this helps,
Bjoern

Am 24.10.2014 um 16:18 schrieb Lukasse, Pieter:
> Hi ,
> 
> I am trying to get a dependency to work in practice (when running a tool 
> after the installation). But I keep getting the following message when 
> running the tool:
> 
> 
>   [sshexec] galaxy.tools.deps DEBUG 2014-10-24 16:04:15,514 Building 
> dependency shell command for dependency 'R_bioc_metams'
>   [sshexec] galaxy.tools.deps WARNING 2014-10-24 16:04:15,514 Failed to 
> resolve dependency on 'R_bioc_metams', ignoring
> 
> 
> This is the tool requirements section:
> 
>
>version="3.1.1">R_bioc_metams
>
> 
> This is the repository_dependencies.xml :
> 
> 
>   http://testtoolshed.g2.bx.psu.edu";
>   name="prims_metabolomics_dependencies" owner="pieterlukasse" 
> changeset_revision="71356c62e5cd" />
> 
> 
> This is the package name of the package that is being referred:
> 
> 
>  
>   
>   
> 
> 
> I can't find the problem. Any ideas? I have updated to the latest Galaxy 
> version (today) and removed both items (with option to remove files) and 
> installed them again. Am I facing a bug here?
> 
> Thanks,
> 
> Pieter Lukasse
> Wageningen UR, Plant Research International
> Department of Bioinformatics (Bioscience)
> Wageningen Campus, Building 107, Droevendaalsesteeg 1, 6708 PB,
> Wageningen, the Netherlands
> T: +31-317481122;
> M: +31-628189540;
> skype: pieter.lukasse.wur
> http://www.pri.wur.nl
> 
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Uploading workflow to toolshed

[Björn Grüning]

Hi!

Am 21.10.2014 um 12:39 schrieb Mikel Egaña Aranguren:
> Hi;
> 
> We are developing a Metagenomics tool suite (along the lines of Mothur) and
> we would like to include some example workflows. What is the best practice
> to upload workflows to the tool shed?
> 
> - As part of the tool suite.
> - In a separate repository with only the workflows.

As a separate repository ending with _workflow and containing a
repository dependency file, with all needed tools and packages :)

Let me know when these tools are available, this is very interesting!
Bjoern

> Thanks
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Uploading workflow to toolshed

[Björn Grüning]

Hi!

Am 21.10.2014 um 12:39 schrieb Mikel Egaña Aranguren:
> Hi;
> 
> We are developing a Metagenomics tool suite (along the lines of Mothur) and
> we would like to include some example workflows. What is the best practice
> to upload workflows to the tool shed?
> 
> - As part of the tool suite.
> - In a separate repository with only the workflows.

As a separate repository ending with _workflow and containing a
repository dependency file, with all needed tools and packages :)

Let me know when these tools are available, this is very interesting!
Bjoern

> Thanks
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] error running cuffdiff

[Björn Grüning]

Hi Elizabeth,

can you please let us know which parameters you have used?

Thanks,
Bjoern

Am 18.10.2014 um 14:33 schrieb Elizabeth Chia:
> 
>> Hi there
>>
>>
>> I'm running into this error with cuffdiff:
>>
>>
>>
>>>
>>> 
>>> I'm running galaxy locally on a cluster (1 head+3 compute nodes) with MySQL.
>>> Here's my hg summary:
>>>
>>>
>>>

 ​
 Any clues?

 I also keep getting the a similar 149 Internal Server error (with 
 different Key Error) thrown when trying to install some tools. But 
 strangely, after leaving it for awhile, it went back to normal. So I'm not 
 too sure what's causing it.

 Should I be updating galaxy to the latest stable one?

 Please let me know if you require any other info.

 Thanks!

 liz 


 ___
 Please keep all replies on the list by using "reply all"
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] question about GALAXY_SLOTS

[Björn Grüning]

Hi Wolfgang,

\${GALAXY_SLOTS:-4} in this case will default to 4 if you have not
configured it in job_conf.xml. Otherwise yes, it will be 1.

Ciao,
Bjoern

Am 17.10.2014 um 00:05 schrieb Wolfgang Maier:
> Hi,
> 
> this is just to make sure: the GALAXY_SLOTS environmental variable set
> by Galaxy when running tools will always be a number >= 1 with 1 being
> the default if nothing else is configured in the job runner settings ?
> 
> Correct ?
> 
> Thanks,
> Wolfgang
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] snpeff tool for Galaxy extra_files_path

[Björn Grüning]

Hi John,

glad to see this gets some attention!

Am 15.10.2014 um 19:05 schrieb John Chilton:
> Hey JJ,
> 
> Opened a pull request to stable with my best guess at the right to
> proceed and hopefully a best practice recommendation we can all get
> behind. Do you want to try it out and let me know if it fixes snpeff?
> (It does fix the velvet datatypes you contributed to Galaxy.)
> 
> https://bitbucket.org/galaxy/galaxy-central/pull-request/532/fix-for-datatypes-consuming-output-extra/diff
> 
> Dan, Bjoern - does this make sense - can we move forward with this
> approach ($input.extra_files_path for inputs and $output.files_path
> for outputs) as the best practices for how to reference these
> directories.

I'm not sure why we need this distinction? Can we not simply choose one
for both, inputs and outputs?
Otherwise we need to explain it very well, why this is needed and I
would vote to rename it to reflect that files_path can be only used by
$outputs ...

Salve,
Bjoern

> -John
> 
> On Wed, Oct 15, 2014 at 11:44 AM, Jim Johnson  wrote:
>> I agree with you about the inadvisable use of:   input.dataset.*.
>>
>> I'm looking at:
>>
>> lib/galaxy/model/__init__.py
>> class Dataset( object ):
>> ...
>> def __init__( self, id=None, state=None, external_filename=None,
>> extra_files_path=None, file_size=None, purgable=True, uuid=None ):
>>...
>>self._extra_files_path = extra_files_path
>>...
>> @property
>> def extra_files_path( self ):
>> return self.object_store.get_filename( self, dir_only=True,
>> extra_dir=self._extra_files_path or "dataset_%d_files" % self.id )
>>
>> I'm trying to see when self._extra_files_path gets set. Otherwise, would
>> this return the path relative to the current file location of dataset?
>>
>>
>>
>>
>> On 10/15/14, 9:36 AM, John Chilton wrote:
>>>
>>> Okay - so this is what broke things:
>>>
>>>
>>> https://bitbucket.org/galaxy/galaxy-central/commits/d781366bc120787e201b73a4dd99b56282169d86
>>>
>>> My feeling with the commit was that wrappers and tools should never be
>>> explicitly accessing paths explicitly through input.dataset.*. I think
>>> this would circumvent options like outputs_to_working_directory and
>>> break remote job execution through Pulsar. It also breaks the object
>>> store abstraction I think - which is why I made the change for Bjoern
>>> I guess.
>>>
>>> I did not (and this was stupid on my part) realize that datatype code
>>> would be running on the remote host and accessing these model
>>> properties directly outside the abstractions setup by the wrappers
>>> supplied to cheetah code and so they have become out of sync as of
>>> that commit.
>>>
>>> I am thinking somehow changing what the datatype code gets is the
>>> right approach and not fixing things by circumvent the wrapper and
>>> accessing properties directly on the dataset. Since you will find that
>>> doing this breaks things for Bjoern object store and could probably
>>> never run on usegalaxy.org say for the same reason.
>>>
>>> Too many different competing deployment options all being incompatible
>>> with each other :(.
>>>
>>> Will keep thinking about this and respond again.
>>>
>>> -John
>>>
>>> On Wed, Oct 15, 2014 at 9:39 AM, John Chilton  wrote:

 JJ,

 Arg this is a mess. I am very sorry about this - I still don't
 understand extra_files_path versus files_path myself. There are open
 questions on Peter's blast repo and no one ever followed up on my
 object store questions about this with Bjoern's issues a couple
 release cycles ago. We need to get these to work - write documetation
 explicitly declaring best practices we can all agree on and then write
 some tests to ensure things don't break in the future.

 When you say your tools broke recently - can you say for certain which
 release broke these - the August14, October14, something older?

 I'll try to do some more research and get back to you.

 -John

 On Tue, Oct 14, 2014 at 6:04 AM, Jim Johnson  wrote:
>
> Andrew,
>
> Thanks for investigating this. I changed the subject and sent to the
> galaxy
> dev list.
>
> I've had a number of tools quit working recently.   Particularly tools
> that
> inspect the extra_files_path when setting metadata, Defuse, Rsem,
> SnpEff.
>
> I think there was a change in the galaxy framework:
> The extra_files_path when referenced from an input or output in the
> cheetah
> template sections of the tool config xml will be relative to the job
> working
> directly rather than the files location.
>   I've just changed a few of my tools on my server yesterday
> from: .extra_files_path
> to:   .dataset.extra_files_path
> and they now work again.
>
> Dan or John, is that the right way to handle this?
>   Thanks,
>
> JJ
>
>
>
> On 10/13/14, 9:29 PM, Andrew Lonie wrote:
>

Re: [galaxy-dev] Is the "new tool repositories summary" in the monthly newsletter useful?

[Björn Grüning]

Hi Dave,

on IRC we had a small discussion how we can make the progress easier for
you to collect this information. The idea was to create automatically
Trello cards for IUC and devteam tools and maybe a summary page in the
Tool Shed.

Cheers,
Bjoern

Am 15.10.2014 um 06:52 schrieb Dave Clements:
> Hello all,
> 
> Well, the results are unanimous: Keep it; it's worth the time invested
> 
> Where to put it was far from unanimous, so we'll experiment with putting it
> in both the monthly newsletter and dev news briefs.
> 
> Thanks all,
> 
> Dave C.
> 
> On Wed, Oct 8, 2014 at 9:42 AM, Dave Clements 
> wrote:
> 
>> Hi Peter, all,
>>
>> I've added post in both places as an option.
>>
>> So far we only have two responses ...
>>
>> Thanks,
>>
>> Dave C
>>
>>
>>
>> On Wed, Oct 8, 2014 at 1:16 AM, Peter Cock 
>> wrote:
>>
>>> On Wed, Oct 8, 2014 at 12:49 AM, Dave Clements
>>>  wrote:
 Hi All,

 The October Galaxy newsletter went out a week ago.  Buried at the
>>> bottom is
 this

 36 new ToolShed repos
>>>
>>> -->
>>> https://wiki.galaxyproject.org/GalaxyUpdates/2014_10#ToolShed_Contributions
>>>
 which lists repositories that have been published in the Galaxy Project
 ToolShed in the previous month.

 I have two questions about this:

 1. How useful is this summary?

 Compiling it is a manual process and it's kind of mind-numbing.  Most
>>> months
 it takes around 2 hours (I think).
>>>
>>> I find it moderately useful, so if most Galaxy Admins think the same, it
>>> probably is overall a good time investment.
>>>
 2. If we keep the summary, should we put it in the Dev News Briefs
>>> instead?

 I'm kinda thinking this summary is a better match for the Dev News
>>> Briefs
 (every release), then it is for the general newsletter (every month).
>>>
>>> I would suggest both (easy if it is just a link, a tiny bit of copy and
>>> paste
>>> if not), but that wasn't an option on the Google form.
>>>
>>> Peter
>>>
>>
>>
>>
>> --
>> http://galaxyproject.org/
>> http://getgalaxy.org/
>> http://usegalaxy.org/
>> https://wiki.galaxyproject.org/
>>
> 
> 
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Cloudman indices installation/configuration

[Björn Grüning]

Jumping in ;)

Hi Iry,

please have a look at our small readme file under:
https://github.com/galaxyproject/tools-iuc/tree/master/tools/gatk2

The problem is that we are not allowed to ship the jar file. So we came 
up with the idea to provide you with empty env.sh file you need to edit.


Sorry for the inconvenience,
Bjoern

Am 08.10.2014 um 00:09 schrieb Enis Afgan:

Hi Iry,
Yeah, I see what you mean about that *env.sh* file not being in the GATK2
repo the readme states so. I'm not sure what's exactly supposed to be in
that file for GATK2 in particular so perhaps one of the wrapper authors can
jump in. For majority of tools, you'd just need something like this in
there: 
*PATH=/mnt/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/gatk2/8bcc13094767/gatk2/bin:$PATH
<http://toolshed.g2.bx.psu.edu/repos/iuc/gatk2/8bcc13094767/gatk2/bin:$PATH>*
and
have placed the binaries for the tool in that directory.

If nobody else jumps in, I'll poke around more in the coming days.

On Tue, Oct 7, 2014 at 11:58 AM, Iry Witham  wrote:


  Hi Enis,

  Thanks for that information.  Now I am getting an error with the
Unified_Genotyper failing to locate the GenomeAnalysisTK.jar.  I discovered
that gatk2 needs to be downloaded and installed.  I have done that, but
can't seem to figure out where the env.sh file reference below exists.  Can
you point me to the correct proximity of that file?  Or do I need to create
the file and if so where?

  Thanks,
Iry

 Galaxy wrapper for GATK2

This wrapper is copyright 2013 by Björn Grüning, Jim Johnson & the Galaxy
Team.

The Genome Analysis Toolkit or GATK is a software package developed at the
Broad Institute to analyse next-generation resequencing data. The toolkit
offers a wide variety of tools, with a primary focus on variant discovery
and genotyping as well as strong emphasis on data quality assurance. Its
robust architecture, powerful processing engine and high-performance
computing features make it capable of taking on projects of any size.

http://www.broadinstitute.org/gatk
http://www.broadinstitute.org/gatk/about/citing-gatk

GATK is Free for academics, and fee for commercial use. Please study the
GATK licensing website:
http://www.broadinstitute.org/gatk/about/#licensing
   Installation

The recommended installation is by means of the toolshed
<http://toolshed.g2.bx.psu.edu/view/iuc/gatk2>.

Galaxy should be able to install samtools dependencies automatically for
you. GATK2, and its new licence model, does not allow us to distribute the
GATK binaries. As a consequence you need to install GATK2 by your own,
please see the GATK website for more information:

http://www.broadinstitute.org/gatk/download

Once you have installed GATK2, you need to edit the env.sh files that are
installed together with the wrappers. You must edit the GATK2_PATH
environment variable in the file:


/environment_settings/GATK2_PATH/iuc/gatk2//env.sh

to point to the folder where you have installed GATK2.

Optionally, you may also want to edit the GATK2_SITE_OPTIONS environment
variable in the file:


/environment_settings/GATK2_SITE_OPTIONS/iuc/gatk2//env.sh

to deactivate the 'call home feature' of GATK with something like:

GATK2_SITE_OPTIONS='-et NO_ET -K /data/gatk2_key_file'

GATK2_SITE_OPTIONS can be also used to insert other specific options into
every GATK2 wrapper at runtime, without changing the actual wrapper.

Read more about the "Phone Home" problem at:
http://www.broadinstitute.org/gatk/guide/article?id=1250

Optionally, you may also want to add some commands to be executed before
GATK (e.g. to load modules) to the file:

/gatk2/default/env.sh

Finally, you should fill in additional information about your genomes and
annotations in the gatk2_picard_index.loc and gatk2_annotations.txt. You
can find them in the tool-data/ Galaxy directory.

   From: Enis Afgan 
Date: Saturday, October 4, 2014 6:10 AM

To: Iry Witham 
Cc: "galaxy-dev@lists.bx.psu.edu" 
Subject: Re: [galaxy-dev] Cloudman indices installation/configuration

   Hi Iry,
Try adding the following to your
*/mnt/galaxy/galaxy-app/tool_data_table_conf.xml*, populating the
referenced files (tool-data/gatk2_picard_index.loc and
tool-data/gatk2_annotations.txt) as desired and restarting Galaxy:

  
 
 value, dbkey, name, path
 
 
 
 
 value, name, gatk_value, tools_valid_for
 
 

  Hope this gets you going. Let us know if it doesn't,
Enis

On Fri, Oct 3, 2014 at 1:36 PM, Iry Witham  wrote:


  It looks like I need to generate the dict file for the mm10 reference
as well as add the reference to the srma_index.loc.  My question is where
do these need to exist?  Do they belong in the repo directory structure or
or in the primary tool-data directory?  The hg19.fa, hg19.fa.fia, hg19.dict
as well as these same files for the mm9 GRCh37. However, the .dict does not
exist for mm10.  Even

Re: [galaxy-dev] tool for STAR RNA-seq aligner

[Björn Grüning]

Hi,

Am 24.09.2014 um 22:39 schrieb David Hoover:

Why didn't I see these before?  Hmm, I thought I had searched both toolsheds...

I was kind of hoping someone had tackled this in a different way.  It would be 
nice if there was a composite datatype for the reference genome.  It is 
important for users to generate their own personal genome references, rather 
than rely on shared, admin-installed indices.  And you're correct, we'd need at 
least 5-10 separate genome references for each organism, depending on read 
length and annotation GTF.


Yes :( For me this was to cumbersome and I decided to wait until the 
TopHat successor will be released :)



Back to wheel reinvention.

BTW, can you tell me which standardly installed tools use composite datatypes?  
It's always easier to build thing from comparison, rather than from scratch.


Sure. Have a look at the Galaxy wrappers from Peter: 
https://github.com/peterjc/galaxy_blast


Galaxy Datatypes are composite.
Cheers,
Bjoern


David


David Hoover, PhD
Helix Systems Staff

On Sep 24, 2014, at 4:14 PM, Björn Grüning  wrote:


Hi David,

yes there is inital code in the https://testtoolshed.g2.bx.psu.edu/. I think 
Ross has done some work on it.
The main problem with Star is that is needs special indices (and a lot of it) 
and it would be great to offer data managers for it.

Cheers,
Bjoern

Am 24.09.2014 um 22:05 schrieb David Hoover:

Hi,

I am developing a tool for STAR (https://code.google.com/p/rna-star/), and I 
realize I may be reinventing another wheel.  Has anyone else created a tool for 
STAR?  There's nothing else in the toolsheds for it yet.

David


David Hoover, PhD
Helix Systems Staff
SCB/DCSS/CIT/NIH
301-435-2986
http://helix.nih.gov





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] tool for STAR RNA-seq aligner

[Björn Grüning]

Hi David,

yes there is inital code in the https://testtoolshed.g2.bx.psu.edu/. I 
think Ross has done some work on it.
The main problem with Star is that is needs special indices (and a lot 
of it) and it would be great to offer data managers for it.


Cheers,
Bjoern

Am 24.09.2014 um 22:05 schrieb David Hoover:

Hi,

I am developing a tool for STAR (https://code.google.com/p/rna-star/), and I 
realize I may be reinventing another wheel.  Has anyone else created a tool for 
STAR?  There's nothing else in the toolsheds for it yet.

David


David Hoover, PhD
Helix Systems Staff
SCB/DCSS/CIT/NIH
301-435-2986
http://helix.nih.gov





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Question about uploading a large application to the tool-shed

[Björn Grüning]

Hi George,

this really seems to big. What kind of data do you have included? Do you 
know the Galaxy Data Managers? It seems to me that you should decouple 
your wrappers from the data and create a data_manager to handle it.


Cheers,
Bjoern

Am 17.09.2014 um 22:59 schrieb George Weingart:

Hello,

I have an application that is working perfectly in our local Galaxy (Not
using tool-shed)
I would like to upload it to the tool-shed (I have uploaded other
applications to the tool-shed before)
The size of the directory (//usr/local/galaxy-dist/tools/metaphlan2)   is
2.1 GB and after I tar it it is 1.1 GB.
The question:
Is it too big to be uploaded to the tool-shed?
Thanks!
George Weingart



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tool Errors

[Björn Grüning]

You can check the stderr and stdout in the information (small i Icon) 
site on every dataset. Also can you see what the log is saying? It 
should provide you the actuall commandline call, which is produced by 
the wrapper.


Cheers,
Bjoern

Am 12.09.2014 um 16:33 schrieb Calvin Morrison:

Which me thinks is weird, because my code, when given the -v flag will at
least put out some info to stdout, or worse case, some errors to stderr,
but those are empty too!

On 12 September 2014 10:31, Björn Grüning  wrote:


Hi,

as far as I know, it just means your tool did not produced any output, so
your results is empty. Nothing more. This is no error, or? The box is not
red?

Cheers,
Bjoern

Am 12.09.2014 um 16:28 schrieb Calvin Morrison:


Hi guys,

I am trying to write a tool with a conditional and for whatever reason the
results, well aren't any results. It just says 'no seek', which totally
bewildering. Does anyone know what this means?

Thank you,
Calvin

my xml file:


Classify with Quikr


# if $qdb.dbtype == "user"
quikr -v -k 0 -s ${qdb.dbname} -i ${input} -o ${output}
# else
quikr -v -k 0 -s "${qdb.dbname}${qdb.dbsize}".mat.gz -i ${input} -o
${output}
# end if



  
  
  
Pretrained
databases
Custom trained quikr database
  
  


  5

  
  

  Green Genes
91%
  Green Genes
94%
  RDP Trained
Set


  5
  6
  7
  8

  
  
  


  







___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tool Errors

[Björn Grüning]

Hi,

as far as I know, it just means your tool did not produced any output, 
so your results is empty. Nothing more. This is no error, or? The box is 
not red?


Cheers,
Bjoern

Am 12.09.2014 um 16:28 schrieb Calvin Morrison:

Hi guys,

I am trying to write a tool with a conditional and for whatever reason the
results, well aren't any results. It just says 'no seek', which totally
bewildering. Does anyone know what this means?

Thank you,
Calvin

my xml file:


   Classify with Quikr
   

   # if $qdb.dbtype == "user"
   quikr -v -k 0 -s ${qdb.dbname} -i ${input} -o ${output}
   # else
   quikr -v -k 0 -s "${qdb.dbname}${qdb.dbsize}".mat.gz -i ${input} -o
${output}
   # end if

   
   
 
 
 
   Pretrained
databases
   Custom trained quikr database
 
 
   
   
 5
   
 
 
   
 Green Genes 91%
 Green Genes 94%
 RDP Trained Set
   
   
 5
 6
 7
 8
   
 
 
 
   
   
 
   
   
   




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Incorrect revision getting referenced in a complex tool dependency - why?

[Björn Grüning]

Hi Melissa,

a few remarks to your wrappers, hopefully it will fix a few issues.

* If you put your python files and your java files next to your wrapper 
you do not need to install them. I think you can remove the 
tool_dependency file and it should work.
* if you stick to this approach you should rename JAVA_JAR_PATH to 
XENA_JAR_PATH or something like this
* do you really need the dependencies in the other repositories? 
Dependencies are used to access binaries or libaries from other 
installations, as far as I could see you are accessing a local port or? 
So no dependency
* Question: do you need the server running all the time, or can you that 
it during the tool execution (find, import)? We need a proper mechanism 
to support such things like servers (on some ports). You will here have 
the issue that multiple users can't start a server at once. We need to 
get devteam help here. Would you be so kind and create a trello card for it?




Okay, I really need debugging ideas on this one.

I have three repositories I'm developing under the test toolshed.  They're
named start_xena, xena_import and xena_find_datasets (they're all under
visualization).  start_xena contains a simple tool dependency to a package
named installXena.  xena_import and xena_find_datasets contain complex
dependencies to installXena.  I've installed these tools on two computers.
  On one, everything works great - amazingly well, in fact!  Kudos,
Development Team!  On the other computer, the env.sh files associated with
the complex dependencies keep referencing the wrong version of
start_xena/installXena.

Here are the gory details.

The start_xena tool is installed in
testtoolshed.g2.bx.psu.edu/repos/melissacline/start_xena/82755b0ee5a5/start_xena.
  I'm a bit confused about the revision part of the path, because the latest
revision is different (75c7d80df9c1), and I have uninstalled and
re-installed the tool since its last update.  Its package, installXena, is
installed under tool_dependencies at
tool_dependencies/installXena/1.0/melissacline/start_xena/82755b0ee5a5/.
  That directory contains all the files I'd expect to be there, and its
env.sh file contains all the environment variables I'd intended to set.
So far, so good.

xena_import is installed on my computer at
testtoolshed.g2.bx.psu.edu/repos/melissacline/xena_import/dc42b6bbc22b/xena_import.
  Its tool_dependencies.xml reference installXena under start_xena as:





   

 http://testtoolshed.g2.bx.psu.edu";
name="start_xena" owner="melissacline" changeset_revision="82755b0ee5a5"/>

   



(Aside: is it necessary to have the changeset_revision in there?  I'm a bit
worried about the maintenance task of updating it for all of my dependent
tools every time I update start_xena, but as I'll explain, it's not clear
that this revision info is getting used).

The tool dependency dir for xena_import is at
tool_dependencies/installXena/1.0/melissacline/xena_import/dc42b6bbc22b/.
  Its env.sh reads:

if [ -f
/Users/melissacline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh
] ; then .
/Users/melissacline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh
; fi

Notice that the wrong revision number is in there.  The 06 revision was an
older one.  Needless to say, the files and environment variables I need
aren't in the 06 directory.

Where does this incorrect revision number come from?  I've deleted all the
06 directories, repeatedly.  I've deleted and reinstalled the tools, and
verified that after deletion, the directories in question were in fact
gone.  I've tried resetting the metadata for these tools under the Galaxy
Admin menu.  Now I need new ideas.  As I mentioned, all of this works
beautifully on a second computer.  It's as if the first Galaxy
installation, on the first computer, got stuck with some outdated
information, and I haven't figured out how to clear it out.


I don't know but you should remove the toolshed= and the revision= from 
your XML files. These fields will be automatically filled by the 
toolshed (inserting the latest tip version of your dependencies and the 
current toolshed url).


Hope this helps a little bit!
Bjoern


Thanks!

Melissa



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Output Excel file

[Björn Grüning]

Hi Patrick,

a nice workaround is the following:

Please create your file.xls file, but not in the path galaxy will 
provide, use a temporary path, for example in the current working dir in 
your wrapper. Than after your program has finished, just move the *.xls 
file to the galaxy output path. Done. Like this.


myprogram.r -i $input -o foo.xls -p 4;
mv foo.xls $output;

Cheers,
Bjoern





Am 04.09.2014 um 23:27 schrieb Patrick Leyshock:

Hello,

I am trying to write a Galaxy tool that will output an Excel file.

Currently the tool wrapper calls an R script, which uses the "xlsx" package
to read and write to and from Excel files.

After being invoked by Galaxy, the script is able to successfully read an
input Excel file:

 suppressPackageStartupMessages(library(xlsx, quietly=TRUE));
 raw.data <- read.xlsx(commandArgs(trailingOnly=TRUE)[1]),
sheetName="input_records");

The script then does its work on the data just read in.  Then, when the
work is done, I'd like to output the results to an Excel file.  Here's
where I run into a problem.  I first tried to output the result like this:

 write.xlsx(processed.data, file=commandArgs(trailingOnly=TRUE)[2]);

but write.xlsx throws an error.  Looks like  ".xls" and ".xlsx" are the
only legal file extensions for the write.xlsx() function.  Inspecting
commandArgs(trailingOnly=TRUE)[2] shows that Galaxy provided a filename
extension of ".dat".

I tried a workaround using this:

 write.xlsx(processed.data,
file=paste(commandArgs(trailingOnly=TRUE)[2]), ".xls", sep="");

The write.xlsx function no longer throws an error (since the file name
supplied as a parameter has an acceptable file extension) but now Galaxy
won't display the result in the History.  If I look in Galaxy's database I
see two files there corresponding to my tool's output.  Supposing that
Galaxy assigned the result the name "dataset_87", then looking in the
Galaxy database I see:

 dataset_87.dat
 dataset_87.dat.xls

"dataset_87.dat" is empty but visible to Galaxy (and so displayed as an
empty dataset in the History window). "dataset_87.dat.xls" has the results
I want but isn't displayed by Galaxy in the History window.

There might be R libraries that can write xls or xlsx files without
requiring a ".xls" or ".xlsx" file extension.  That's a path I'm willing to
explore.  That said, is there a configuration option I can set that'll let
me continue to use write.xlsx()?

I've been working from the suggestions made on this helpful thread:

http://lists.bx.psu.edu/pipermail/galaxy-dev/2011-December/007807.html

so the relevant entries in datatypes_conf.xml are:

 
 

and the relevant additions to binary.py are:

 class Xls(Binary):
 '''Class describing an Excel 2003 (xls) file'''

 file_ext='xls'

 class Xlsx(Binary):
 '''Class describing an Excel 2007 (xlsx) file'''

 file_ext='xlsx'

Any suggestions appreciated.

Thanks, Patrick



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Examples of Galaxy tools in the toolsheds that install and run JAR files properly?

[Björn Grüning]

Hi Melissa,

Am 03.09.2014 um 03:22 schrieb Melissa Cline:

Peter, Pieter and Dave, thank you for the pointers to your tools - they've
been extremely helpful!  Now I can see how the process is supposed to work.
  I'm not really sure where mine is going wrong, but maybe someone here will
have ideas.

So folks, I've had partial success with a repository that includes a
tool_dependencies.xml, which sets two environment variables and moves a JAR
file from REPOSITORY_INSTALL_DIR to INSTALL_DIR.  But the following things
suggest to me that it's only a partial success, and I'm very interested in
any insights to clear them up.


1. I have my repository checked into an internal tool shed.  Somewhere in
the course of development, I specified a buggy dependency, and now I can't
seem to clear it up.  When I go to (re)install my tool from the tool shed,
here's what I see in the way of dependencies:

  Tool dependencies* - these dependencies may not be required by tools in
this repository*
NameVersionTypeOrphanJAR_PAHTset_environmentyes
and as you might imagine, I've checked the obvious things, and there is no
more reference to "JAR_PAHT" (typos and all) anywhere in or around my tool,
or in my galaxy-dist directory.  It seems like there's some old metadata
cruft that hasn't been cleared out.  How can I clear it out?


As Pieter already mentioned you can try to rebuild the metadata 
associated to your repository. You can also try to 'repair' your 
repository. This option should be in your Admin menue of your tool. And 
finally you can remove and reinstall the tool.



For the other problems, is there any change you can upload your repo to 
the test-toolshed? In this way we can have a look at it and reproduce it!


Thanks,
Bjoern



2. When I install my tool, I see the following messages in paster.log:

---

tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager
DEBUG 2014-09-02 17:13:59,279 Building repository dependency
relationships...

172.30.0.22 - - [02/Sep/2014:17:13:59 -0700] "POST
/admin_toolshed/prepare_for_install HTTP/1.1" 200 - "
http://tcga1:1235/admin_toolshed/prepare_for_install?tool_shed_url=http://medbook.ucsc.edu:9009/&repository_ids=decab5ee1e95b10b&changeset_revisions=ff9b02e50bcf";
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/36.0.1985.143 Safari/537.36"

172.30.0.22 - - [02/Sep/2014:17:14:02 -0700] "POST
/admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "
http://tcga1:1235/admin_toolshed/prepare_for_install"; "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/36.0.1985.143 Safari/537.36"

tool_shed.util.shed_util_common DEBUG 2014-09-02 17:14:03,477 Error
attempting to get tool shed status for installed repository start_xena:
HTTP Error 404: Not Found

Attempting older 'check_for_updates' method.

---

Should I be concerned about these last messages?  I don't remember seeing
them with Peter, Pieter and Dave's tools.


3. After my tool has installed, there is no INSTALLATION.log in my
INSTALL_DIR.  Does this mean that the installation process somehow
terminated early?  There is an env.sh file, with the correct values of the
environment variables I'm setting in my tool_dependencies.xml, and my jar
file is copied to INSTALL_DIR.  In my Galaxy window, my tool is indicated
as "Installed", in green.  Here are the last messages I see in paster.log:

tool_shed.galaxy_install.install_manager DEBUG 2014-09-02 17:14:04,630
Changing status for tool dependency installXena from Installing to
Installed.

tool_shed.galaxy_install.install_manager DEBUG 2014-09-02 17:14:04,669 Tool
dependency installXena version 1.0 has been installed in
/inside/home/cline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/3e4683c94d8d.

172.30.0.22 - - [02/Sep/2014:17:13:59 -0700] "POST
/admin_toolshed/manage_repositories HTTP/1.1" 302 - "
http://tcga1:1235/admin_toolshed/prepare_for_install"; "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/36.0.1985.143 Safari/537.36"

172.30.0.22 - - [02/Sep/2014:17:14:04 -0700] "GET
/admin_toolshed/monitor_repository_installation?tool_shed_repository_ids=3f5830403180d620
HTTP/1.1" 200 - "http://tcga1:1235/admin_toolshed/prepare_for_install";
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/36.0.1985.143 Safari/537.36"

172.30.0.22 - - [02/Sep/2014:17:14:05 -0700] "POST
/admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 - "
http://tcga1:1235/admin_toolshed/prepare_for_install"; "Mozilla/5.0
(Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/36.0.1985.143 Safari/537.36"


Thanks for the insights, folks!  It actually seems like my tool is running
post-install, but I just want to make sure that everything is in place
before I go on to the next step.


Melissa





On Sat, Aug 30, 2014 at 5:50 AM, Dave Bouv

Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool

[Björn Grüning]

Am 25.08.2014 um 18:05 schrieb Dooley, Damion:

Ok, I'll be very happy to see what you've accomplished there.  I will read 
through what you've done when I return from vacation in a week!

A key need is to have whatever data comes in show up as linked data in one's 
history to avoid server overhead;
a second objective was to not need to modify existing workflows - as 
long as they could work of data in history that is typed appropriately. 
 So your 'select type' solution sounds intreguing!


And certainly interested in your use of git - I tried using git, using a 1-line 
fasta data format, but git seemed to choke on protein fasta files?
And did it run into performance problems with larger files?  That was my 
experience.  I think I read its authors say that its upper limit was 15gb.


This is probably true for one large file. I'm storing the entire PDB in 
git since a few years. One entry one file and it works fine.


Do you know git annex? https://git-annex.branchable.com/


That was the motivation for writing a simple key-value master file diff system 
that seems to have the same I/O as git on smaller files,
but more reliable for the fasta data case, and no problems with larger files - 
it outputs a new version in the same time it takes to read a master file.
It has drawbacks though - incoming data to compare master with must be sorted 
in 1 line fasta format first.


My intention was to create a universal solution for database tracking. 
So if you can please design your system in such a way that you can store 
arbitrary data, not only fasta files.




Thanks for your input; looking forward to your project writeup...


Wonderful!
Talk to you after your holidays!
Bjoern


Damion

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada
________
From: Björn Grüning [bjoern.gruen...@gmail.com]
Sent: Saturday, August 23, 2014 12:17 AM
To: Dooley, Damion; galaxy-dev@lists.bx.psu.edu
Cc: Hsiao, William
Subject: Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval 
Tool

Hi Damion,

the idea sounds fantastic!
Can we go a step further and use a specific datatype that keeps entire
fasta files versioned and the user can choose which version he wants to
use, in any tool? Please have a look at my talk at GCC2012. Maybe you
are interested in the (old) patches. I would be very interested to
restart this old project.

https://wiki.galaxyproject.org/Events/GCC2012/Abstracts#Keeping_Track_of_Life_Science_Data


Am 23.08.2014 um 03:24 schrieb Dooley, Damion:

We are about to implement a fasta database (file) versioning system as a Galaxy 
tool.  I wanted to get interested people's feedback first before we roll ahead 
with the prototype implementation.  The versioning system aims to:

- Simple makeblastdb or bowtie-build commands, or more specific workflows 
that include dustmasker etc can be implemented.

Does this sound attractive?


I think all of the use cases are covered by the old project mentioned
above. But I did not create a new tool I have created a new 'select
type' everyone can use in all tools. It was using git underneath (yeah,
I have the entire PDB in git and it is working fine :)) but we can
probably change git with a database if you like.

To answer your question: Yes, very attractive!


We're hoping such a vision could handle Fasta databases from 12mb to e.g. 200Gb 
(probably requires makeblastdb in parallel at that scale).

Preliminary work suggests this project is doable via the Galaxy API without 
galaxy customization - does that sound right?!


Yes, as long as the User has an API key.

Cheers,
Bjoern


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Grouping of tools inside of toolbox does not work

[Björn Grüning]

Hi,

I have multiple version of the ncbi-blast+ wrapper installed. But they 
appear twice or a few of them even tripled in the toolbox. Expected 
behavior is only one tool in the toolbox and a select box inside of the 
tool.


I have prepared a docker galaxy-blast container as test case.

https://registry.hub.docker.com/u/bgruening/galaxy-blast/
https://github.com/bgruening/docker-recipes/tree/master/galaxy-blast

Reproducible bugs with Docker!
Cheers,
Bjoern
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool

[Björn Grüning]

Hi Damion,

the idea sounds fantastic!
Can we go a step further and use a specific datatype that keeps entire 
fasta files versioned and the user can choose which version he wants to 
use, in any tool? Please have a look at my talk at GCC2012. Maybe you 
are interested in the (old) patches. I would be very interested to 
restart this old project.


https://wiki.galaxyproject.org/Events/GCC2012/Abstracts#Keeping_Track_of_Life_Science_Data


Am 23.08.2014 um 03:24 schrieb Dooley, Damion:

We are about to implement a fasta database (file) versioning system as a Galaxy 
tool.  I wanted to get interested people's feedback first before we roll ahead 
with the prototype implementation.  The versioning system aims to:

* Enable reproducible research: To recreate a search result at a certain point 
in time we need versioning so that search and mapping tools can look at 
sequence reference databases corresponding to a particular past date.  This 
recall can also explain the difference between what was known in the past vs. 
currently.

* Reduce hard drive space.  Some databases are too big to keep N copies around, 
e.g. 5 years of 16S, updated monthly, is say, 670Mb + 668Mb + 665Mb +   But 
occasionally we want to access past archives fairly quickly.

* Integrate database versioning into Galaxy without adding a lot of complexity.

A bonus would be to enable the efficient sharing of version databases between 
computers/servers.

The solution we think would work centres around a "Versioned Data Retrieval" 
tool (draft image attached) that would work as follows:

1) User selects from a list of databases provided by  "Shared Data > Data Libraries 
> Versioned Data".
   - Each database has a master file that keeps its various versions as a list of 
time-stamped insert/delete transactions of key (fasta id) value (description & 
sequence) pairs.
   - Each master file is managed outside of galaxy via a triggered process on 
regular fasta file imports from data sources like NCBI or other niche sources.
   - We're expecting, due to the nature of fasta archived sequence updates, 
that our master file would only be about 1.1x the latest version in size 
(uncompressed).
2) User enters date / version id to retrieve (validated)
3) If a cached version of that database exists, it is linked into user's 
history.
4) Otherwise a new version of it is created, placed in cache, and linked into 
history.
   - The cached version itself then shows up as linked data under a Data Library 
> Versioned Data subfolder.
5) User can select preconfigured workflow(s) to execute on the selected 
retreived fasta file to regenerate any database products they need.
   - Workflow output data would also be cached in the same way the fasta data 
is - by linking the Galaxy Data Library to it.
   - Workflow execution will be skipped if end data already exists in cache.
   - Simple makeblastdb or bowtie-build commands, or more specific workflows 
that include dustmasker etc can be implemented.

Does this sound attractive?


I think all of the use cases are covered by the old project mentioned 
above. But I did not create a new tool I have created a new 'select 
type' everyone can use in all tools. It was using git underneath (yeah, 
I have the entire PDB in git and it is working fine :)) but we can 
probably change git with a database if you like.


To answer your question: Yes, very attractive!


We're hoping such a vision could handle Fasta databases from 12mb to e.g. 200Gb 
(probably requires makeblastdb in parallel at that scale).

Preliminary work suggests this project is doable via the Galaxy API without 
galaxy customization - does that sound right?!


Yes, as long as the User has an API key.

Cheers,
Bjoern


Feedback really appreciated!

Regards,

Damion Dooley

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Plans for Docker image generation in galaxy

[Björn Grüning]

Hi Marius,

thanks for shifting the discussion over to the mailinglist.

Am 19.08.2014 um 13:13 schrieb Marius van den Beek:

Hi everyone,

it’s my first post to the list, so forgive me if I miss something obvious,
I tried to get all the information I could so if there was already a
discussion I’d be glad if you can point me towards this.



https://trello.com/c/fYB8fMY4/43-docker-in-galaxy-in-docker
https://trello.com/c/tm1fPvyq
https://trello.com/c/8k55T24F
https://github.com/bgruening/galaxy-ipython
https://github.com/bgruening/docker-galaxy-stable

The galaxy-docker-stable image can be used to generate self-defined set 
of Tools with a full working Galaxy instance.



I’m posting to start a discussion about how galaxy is going to handle
interactions with Docker.

I know there are already some tools and resources out there that make use
of Docker. As I understood, we can already specify Docker as a job
destination in the tool xml, along with an image to use (See
https://github.com/apetkau/galaxy-hackathon-2014/tree/master/smalt for a
detailed example). This is cool, and allows for the controlling of
resources, but I think we could take the integration to the next step if we
allowed the generation of Docker images in galaxy.



The Tool Shed can, since the latest release, generate you "ready to go" 
Dockerfiles with all your favorite tools.




I was thinking if there is a trusted baseimage (ie without root access), we
could let users install


Can you please elaborate on this a little bit more?


all the packages they need, commit after the install, and let them use this
new docker image. This could be beneficial for generating new tools with a
sandboxed version of the Galaxy Tool Factory (
http://www.ncbi.nlm.nih.gov/pubmed/23024011).




I am currently working on this (
https://bitbucket.org/mvdbeek/dockertoolfactory), and I would like to have
your opinion on how to manage these user-generated Docker images.

Some questions that I came across:

Do we store the committed images in the user’s history (they could
potentially become very big!)?

How to display available images to the user? Something like a per-user
image registry?

How to identify available images (dataset_id as the tag?)

How to transfer user-generated images to the toolshed?

Is it a good idea at all to store user-generated images in the toolshed?

What do we do if the security of the baseimage is compromised? Obviously

we can blacklist execution of images, but what if somebody installed a
dangerous image from the toolshed, and is not aware of this?


To answer these questions it would be good to have some use-cases for 
your docker tool images.
For example I think most of the time an IPython instance would be 
sufficient. You will only store notebook files in your history (small) 
and execute them on new datasets.


I would be really interested in your use-cases.

Imho, all this should be taken with care, in the end we would like to 
have proper tools and images in the toolshed and reusable for many 
others. I consider IPython Integration and the Toolfactory as bonus and 
tools mostly for developers. Nothing a biologist should deal with. And 
we should take care to not over complicate the tool setup/usage.




Ultimately I think this could be a way to have advanced users run their own
scripts inside galaxy, to generate their own tools and tool-dependencies
inside galaxy, and why not, even have “user-space tools”.

It would bridge the gap between galaxy users and galaxy tool-developers,

so I’m curious what you’re thinking about this.


Absolutely, thanks for bringing this up!
Bjoern




Cheers,

Marius



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] TypeError with 'dict'

[Björn Grüning]

Hi Martin,

please keep galaxy-dev in the CC list.

Am 12.08.2014 um 08:51 schrieb Martin Christiansen:

Hi Björn,

Most certainly. I have posted it below.




-> add here a version number version="0.1"


against hg19

screen_reads.sh $Project.input $Project.samples $Project 
$Samples 


 
 
 galaxy_test1
 Untitled Folder


-> is the white space really needed? If so $Project.input will be two 
words. Use "${Project.input}" to convert it to onw argument



 
 
 
 147406386-700171390
 158256496-700097688
 158337416-700013715
 158337416-700097837
 158357646-700035237
 158458797-700014562
 158479027-700014724
 158479027-700097196
 158499257-700014837
 158499257-700098561
 158742018-700015181
 158802708-700015245
 158802708-700015250
 158802708-700099803
 158802708-700119165
 158822939-700014954
 158883629-700015113
 158883629-700100227
 158883629-700112812
 158924089-700099307
 
 
 
 


-> you don't have any  here.


 
 
 



 
 



-> I'm not sure this will work. workdir is a special dir Galaxy creates 
for you. It will be the working directory where your program get's 
called. So assume your program creates foo.sam files. You can specify it 
like from_work_dir='foo.sam'


The input handling looks also a little bit strange. You do not specify 
any input file in Galaxy. That is not really Galaxy like :)


Cheers,
Bjoern



Reads screened against hg19.



Best,
Martin



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] TypeError with 'dict'

[Björn Grüning]

Hi Martin,

can you show us the code of your tool wrapper?

Thanks,
Bjoern

Am 12.08.2014 um 08:43 schrieb Martin Christiansen:

Hi all,

I'm setting up a local galaxy server and have implemented a tool, but I can't 
even get it to run.
When I try to run it, the terminal says:
TypeError: unhashable type: 'dict'

The error itself arises from a python egg:
File 
"~/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/identity.py",
 line 141 in get
   state = dict.get(self, key, default)

Any suggestions of how to solve this?

Best regards,
Martin Christiansen






___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] testtoolshed : python-2.7 installation error

[Björn Grüning]

Hi,

I updated the python wrappers. Depending on your Galaxy version it 
should work now.


@Greg, we still have the error that  'contact owner' does not seem to 
work :(


Cheers,
Bjoern

Am 07.08.2014 um 12:11 schrieb Geert Vandeweyer:

sorry about that confusion :-)

Yes, I'm talking about the package_python_2.7 repository. The same error
is also present in the test runs of the package_python_3* repo.

Best,

Geert

On 08/07/2014 11:45 AM, Björn Grüning wrote:

Hi Geert,

are you talking about the package_python_* repository?


Am 07.08.2014 um 10:00 schrieb Geert Vandeweyer:

Hi,

I get an installation error on the python 2.7 package in the test
toolshed. I used the 'contact owner' function, but wanted to mention it
here too, as there hasn't been reaction so far. Sorry for double posting
if so.


Oh this is interesting, because we did not get notified. It is from
IUC correct?


Error:

tar (child): 5.2.tar.bz2: Cannot open: No such file or directory

A similar error is in the Test run outputs. I believe it is related to
the following (unnecessary)  line in the tool_dependency.xml:

  ..


This can be. The cwd directory handling changed a lot in recent
versions, due to bugs. And for a few repositories we still have the
workarounds included.

I will upload a new version without this workaround and see if it passes.


located just after the download_file action for the 5.2.tar.bz2 file.

Best,

Geert





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] testtoolshed : python-2.7 installation error

[Björn Grüning]

Hi Geert,

are you talking about the package_python_* repository?


Am 07.08.2014 um 10:00 schrieb Geert Vandeweyer:

Hi,

I get an installation error on the python 2.7 package in the test
toolshed. I used the 'contact owner' function, but wanted to mention it
here too, as there hasn't been reaction so far. Sorry for double posting
if so.


Oh this is interesting, because we did not get notified. It is from IUC 
correct?



Error:

tar (child): 5.2.tar.bz2: Cannot open: No such file or directory

A similar error is in the Test run outputs. I believe it is related to
the following (unnecessary)  line in the tool_dependency.xml:

  ..


This can be. The cwd directory handling changed a lot in recent 
versions, due to bugs. And for a few repositories we still have the 
workarounds included.


I will upload a new version without this workaround and see if it passes.


located just after the download_file action for the 5.2.tar.bz2 file.

Best,

Geert


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Packaging tools with language specific dependencies

[Björn Grüning]

Hi Renato,

Am 05.08.2014 um 17:34 schrieb Renato Alves:

Hi Bjoern,

Thanks for the info and the fix.

So far we had no issues but it's not extensive testing. I'll let you
know if it fails for some reason.

In addition to the shebang change we also started using "perl script"
instead of simply "script". Since "perl" is looked up in PATH, all else
works as expected.


Can you try to explain this a bit more?


It does get a bit tricky if you need to guess where the script is
located. "$(which script)" works but it's hacky. Still, it's the only
way you can "fix" the shebang without changing it before.

PS: There's shutil.which() but only python 3.3 onwards.


Anything I can do to support you? I'm right that the shebang 
modifications work for you?


Do you think that should go into Galaxy Core, so that every perl script 
under /bin/ will get a simple shebang?


Thanks,
Bjoern


Cheers,
Renato

Quoting Björn Grüning on 02-08-2014 15:26:

Hi Renato,

cpanm is not the only problem it seems to be an general perl 'feature'.
How annoying :(

As I figured from:
https://metacpan.org/source/RJBS/perl-5.20.0/Configure#L9162

we can use:-Dstartperl='#!/usr/bin/env perl' to change them via install.
Now every perl script from the main perl has the correct shebang.

Unfortunately, every other package installed with cpanm or Makefile.PL
will have this annoying long shebang.

I'm not sure there is a switch in cpan/cpanm to do that. Under cpan
there are a few repositories, called change_shebang... so I assume we
are not the only ones that have this problem.

So I ended up with:

sed -i
's|#!$INSTALL_DIR/bin/|#!/usr/bin/env |' $INSTALL_DIR/bin/*

I have uploaded it to the testtoolshed and to galaxytools.
If this is working for you, also with depending repositories, I will try
to add this to into the Tool Shed core to make this sed command implicit
for all perl tools.

Cheers,
Bjoern



Am 01.08.2014 um 16:06 schrieb Renato Alves:

Hi BJörn,

We have run into some problems with the perl environment.
This might also affect other tools that rely on a shebang with absolute
paths to the interpreter. Details below.

During the installation of the perl package (we tried package_perl_5_18
from main and testing toolshed), the cpanm script gets installed but
when executed it is invalid/not found.

The issue seems to be the length of the shebang in the script which is
limited to 80 characters. In cpanm it's:

!#/galaxy_dist/tool_dependencies/perl/5.18.1/bgruening/package_perl_5_18/e89824189ec6/bin/perl


That is 92 characters long. When launching the script it gets truncated
at "...perl_5_18/e898241", causing the command to fail with a "bad
interpreter" error.

I was able to workaround this problem by editing the script manually and
replacing the long shebang by: #!/usr/bin/env perl


As it stands it seems relying on the shebang to use the correct
interpreter is a problem. Repositories with long (character wise) names,
versions or owners will be more likely to suffer from this. So will
installations that are not close to the system root.

Renato

Quoting Björn Grüning on 30-07-2014 12:04:

Hi Renato,

Am 30.07.2014 um 12:21 schrieb Renato Alves:

Hi everyone,

Is there any standard or commonly used way to package tools that have
language specific dependencies.

I know that with Python libraries one can use setup_virtualenv and with
Java jars the JAVA_JAR_LIB strategy is used.
Is there anything equivalent for R, Perl and Ruby libraries?


please have a look at:
https://github.com/bgruening/galaxytools/tree/master/test_repositories

If you have any questions I'm happy to help you!
Bjoern


Thanks
Renato





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/






___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Serious start-up delay in next-stable

[Björn Grüning]

Hi all,

does anyone encountered a long start-up delay with the recent 
next-stable and starting a instance from scratch?


For me its hanging at:

tool_shed.galaxy_install.migrate.check DEBUG 2014-08-05 18:39:45,207 
psycopg2 egg successfully loaded for postgres dialect


for 3-5 minutes.

Cheers,
Bjoern
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] cufflinks suite 2.2.1

[Björn Grüning]

Awesome!

Thanks Geert!
Bjoern

Am 04.08.2014 um 22:37 schrieb Geert Vandeweyer:

Hi all,

I've updated the tool wrappers for the cufflinks tools to options
supported by cufflinks 2.2.1. I've also added the cuffquant and cuffnorm
tools, together with the cxb datatype.

As discussed with Jeremy, these updates were welcome :-)

Everything is available on the test toolshed, tested and installed on at
least one instance (http://biomina.be/app/galaxy), although mixing main
and test toolshed installations doesn't work well. The tools are present
twice.

Therefore, feel free to integrate the updated wrappers & tools into the
main cufflinks package (by devteam) after further testing.

Best,

geert

--

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: geert.vandewe...@ua.ac.be
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/in/geertvandeweyer

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] XLS TO CSV

[Björn Grüning]

Hi,

not tat I know. But you can export your xls in Excel as tab separated 
file. Galaxy can handle such tab-delimited files.


Cheers,
Bjoern

Am 04.08.2014 um 14:56 schrieb Mert Mehnur KIRKALI:

Hello,

How can i convert xls file to csv file on galaxy ?

Is that possible ?

Best Regards,Mert.



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] creating datatype repository : no module named binary

[Björn Grüning]

Hi Geert,

please include this at the top:

from galaxy.datatypes.binary import Binary

Cheers,
Bjoern

Am 04.08.2014 um 15:02 schrieb Geert Vandeweyer:

Hi all,

I'm trying to create a new binary datatype in galaxy, but I'm running
into troubles. The current installer is available at
https://testtoolshed.g2.bx.psu.edu/view/geert-vandeweyer/cuffquant_datatype

When I install the repository to a local server, I get the errors below
(no module binary). Since I'm new to defining datatypes, any insights on
what I'm doing wrong is welcome :-)

Best,

Geert


tool_shed.util.repository_dependency_util DEBUG 2014-08-04 14:39:07,313
Building repository dependency relationships...
127.0.0.1 - - [04/Aug/2014:14:39:03 +0200] "POST
/admin_toolshed/prepare_for_install HTTP/1.1" 200 -
"http://127.0.0.1:8080/admin_toolshed/prepare_for_install?tool_shed_url=https://testtoolshed.g2.bx.psu.edu/&repository_ids=80b80f5c54443265&changeset_revisions=e12a09256097";
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Ubuntu Chromium/34.0.1847.116 Chrome/34.0.1847.116 Safari/537.36"
127.0.0.1 - - [04/Aug/2014:14:39:10 +0200] "POST
/admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 -
"http://127.0.0.1:8080/admin_toolshed/prepare_for_install"; "Mozilla/5.0
(X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu
Chromium/34.0.1847.116 Chrome/34.0.1847.116 Safari/537.36"
galaxy.datatypes.registry DEBUG 2014-08-04 14:39:11,362 Loading
datatypes from
/home/gvandeweyer/galaxy-dist/galaxy-dist/database/tmp/tmp-toolshed-acalpdGx1tEP

galaxy.datatypes.registry DEBUG 2014-08-04 14:39:11,363 Exception
importing proprietary code file
/home/gvandeweyer/galaxy-dist/shed_tools/testtoolshed.g2.bx.psu.edu/repos/geert-vandeweyer/cuffquant_datatype/f7801f8191e0/cuffquant_datatype/binary:
No module named binary
galaxy.datatypes.registry ERROR 2014-08-04 14:39:11,363 Error importing
datatype module galaxy.datatypes.binary: 'module' object has no
attribute 'Cuffquant'
Traceback (most recent call last):
   File
"/home/gvandeweyer/galaxy-dist/galaxy-dist/lib/galaxy/datatypes/registry.py",
line 207, in load_datatypes
 datatype_class = getattr( module, datatype_class_name )
AttributeError: 'module' object has no attribute 'Cuffquant'
galaxy.datatypes.registry ERROR 2014-08-04 14:39:11,364 Error calling
method Cuffquant from class :
'module' object has no attribute 'Cuffquant'
Traceback (most recent call last):
   File
"/home/gvandeweyer/galaxy-dist/galaxy-dist/lib/galaxy/datatypes/registry.py",
line 349, in load_datatype_sniffers
 aclass = getattr( module, datatype_class_name )()
AttributeError: 'module' object has no attribute 'Cuffquant'


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Packaging tools with language specific dependencies

[Björn Grüning]

Hi Renato,

cpanm is not the only problem it seems to be an general perl 'feature'. 
How annoying :(


As I figured from:
https://metacpan.org/source/RJBS/perl-5.20.0/Configure#L9162

we can use:-Dstartperl='#!/usr/bin/env perl' to change them via install.
Now every perl script from the main perl has the correct shebang.

Unfortunately, every other package installed with cpanm or Makefile.PL 
will have this annoying long shebang.


I'm not sure there is a switch in cpan/cpanm to do that. Under cpan 
there are a few repositories, called change_shebang... so I assume we 
are not the only ones that have this problem.


So I ended up with:

sed -i 
's|#!$INSTALL_DIR/bin/|#!/usr/bin/env |' $INSTALL_DIR/bin/*


I have uploaded it to the testtoolshed and to galaxytools.
If this is working for you, also with depending repositories, I will try 
to add this to into the Tool Shed core to make this sed command implicit 
for all perl tools.


Cheers,
Bjoern



Am 01.08.2014 um 16:06 schrieb Renato Alves:

Hi BJörn,

We have run into some problems with the perl environment.
This might also affect other tools that rely on a shebang with absolute
paths to the interpreter. Details below.

During the installation of the perl package (we tried package_perl_5_18
from main and testing toolshed), the cpanm script gets installed but
when executed it is invalid/not found.

The issue seems to be the length of the shebang in the script which is
limited to 80 characters. In cpanm it's:

!#/galaxy_dist/tool_dependencies/perl/5.18.1/bgruening/package_perl_5_18/e89824189ec6/bin/perl

That is 92 characters long. When launching the script it gets truncated
at "...perl_5_18/e898241", causing the command to fail with a "bad
interpreter" error.

I was able to workaround this problem by editing the script manually and
replacing the long shebang by: #!/usr/bin/env perl


As it stands it seems relying on the shebang to use the correct
interpreter is a problem. Repositories with long (character wise) names,
versions or owners will be more likely to suffer from this. So will
installations that are not close to the system root.

Renato

Quoting Björn Grüning on 30-07-2014 12:04:

Hi Renato,

Am 30.07.2014 um 12:21 schrieb Renato Alves:

Hi everyone,

Is there any standard or commonly used way to package tools that have
language specific dependencies.

I know that with Python libraries one can use setup_virtualenv and with
Java jars the JAVA_JAR_LIB strategy is used.
Is there anything equivalent for R, Perl and Ruby libraries?


please have a look at:
https://github.com/bgruening/galaxytools/tree/master/test_repositories

If you have any questions I'm happy to help you!
Bjoern


Thanks
Renato





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Announcement: Galaxy IPython Integration - interactive programming in Galaxy

[Björn Grüning]

Hi all,

we proudly present the first release of the Galaxy IPython project.

Galaxy IPython is a visualization plugin which should enable Galaxy 
users with coding skills to easily process their data in the most 
flexible way. With this plugin, it is possible to analyse and 
post-process data without downloading datasets or entire histories. One 
of our aims was to make Galaxy more attractive and accessible to 
bioinformaticians and programmers, and we hope that this project will 
build some bridges.


Screencast: http://www.youtube.com/watch?v=jQDyTuYnn1k

Disclaimer: Even though the Ipython notebooks can be stored and reused, 
this plugin will break the Galaxy philosophy of reproducibility, I feel 
personally bad about that, but I also think it is a great opportunity to 
get more bioinformaticians into Galaxy, and to get Galaxy used more 
often as a teaching resource. By being able to teach not only about 
workflows but also about data analysis tasks often necessary with 
Bioinformatics, Galaxy will be significantly more useful in teaching 
environments.
Keep in mind to write a nice Tool Shed Tool if you catch yourself using 
IPython in Galaxy to often for the same task.


A few features we have up and running:
Use IPython directly in the main window or in the Scratchbook
Completely encapsulated IPython environment with matplotlib, biopython, 
pandas and friends already installed.
IPython runs completely self-contained within a docker container, 
separate from your Galaxy data
Easy access to datasets from your current history via pre-defined 
IPython functions
Manipulate and plot data as you like and export your new files back into 
your Galaxy history
Save IPython Notebooks across analysis sessions in your Galaxy history 
with the click of a button.
View saved IPython Notebooks directly in HTML format, or re-open them to 
continue your analysis.

Self-closing and self-cleaning IPython docker container
Notebooks are secure, only accessible to the intended user


Please follow the installation instruction on our project page under:
https://github.com/bgruening/galaxy-ipython

The underlying IPython Notebook (+Galaxy sugar) is stored under:
https://github.com/bgruening/docker-ipython-notebook
https://registry.hub.docker.com/u/bgruening/docker-ipython-notebook/
You can also install a ipynb datatype:
https://github.com/bgruening/galaxytools/tree/master/datatypes/json
https://testtoolshed.g2.bx.psu.edu/view/iuc/datatyp_ipynb


Comments welcome!
Happy research!
Eric & Björn
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] package_r_2_11_0 and ggplot2

[Björn Grüning]

Hi Evan,

unfortunately I have no access to this repository. I'm only the 
maintainer of the galaxytools in github. But I have CC'ed Dave, he is 
responsible for the devteam packages!


Cheers and thanks for sharing your information!
Bjoern

Am 31.07.2014 um 20:47 schrieb Evan Bollig:

Bjoern,

FYI, I found that package_gatk_1_4 from the main Galaxy toolshed
produces Rscript files for VariantRecalibrator and other commands that
require ggplot_0.8.8 and reshape_0.8.4. This applies to GATK versions
1.4-20 and 1.4-37.

The magic incantation to get a systemwide R v3.1 setup properly is:

r <- getOption("repos");
r["CRAN"] <- "http://cran.rstudio.com/";;
options(repos=r);
install.packages(c('devtools','gsalib'));
require(devtools);
install_url('http://cran.r-project.org/src/contrib/Archive/plyr/plyr_1.7.1.tar.gz');
install_url('http://cran.r-project.org/src/contrib/Archive/reshape/reshape_0.8.4.tar.gz');
install_url('http://cran.r-project.org/src/contrib/Archive/ggplot2/ggplot2_0.8.8.tar.gz');

I'm not sure gsalib is required, but it is better to play it safe IMO.

Can you add this to the package_gatk_1_4 documentation to help others
along? It would be even better if the package_gatk_1_4 could also
install those R libraries itself.

Cheers,

-E
-Evan Bollig
Research Associate | Application Developer | User Support Consultant
Minnesota Supercomputing Institute
599 Walter Library
612 624 1447
e...@msi.umn.edu
boll0...@umn.edu


On Tue, Jul 29, 2014 at 4:22 PM, Björn Grüning
 wrote:



Am 29.07.2014 um 22:45 schrieb Evan Bollig:


Hey Bjoern,

I'm trying to use variant_recalibrator, xy_plot and other tools that
depend on package_r_2_11_0 and require ggplot2 to be installed. Since
CRAN support for R 2.11.0 and 2.15.0 is on the out, perhaps its a
better idea to update the packages the depend on those versions to R
3.0 or 3.1?



Yes, this would be a better idea!



The variant_recalibrator tool generates an Rscript
incompatible with modern versions of ggplot2, so this could be a quite
involved undertaking.



Oh :( ... can we somehow get an old tarball of ggplot2? Have you looked at
older linux disributions, debian stable for example?

I found ggplot on download_store ;)
https://github.com/bgruening/download_store/blob/master/gatk2_R_deps/ggplot2_0.9.3.1.tar.gz

Is that old enough?



Meanwhile I'll look into the download_store option. You download all
of the tarballs and install them individually with R CMD INSTALL?



Not really, we have a nice routine in the toolshed for such tasks:
Have a look at:
https://github.com/bgruening/galaxytools/tree/master/orphan_tool_dependencies/package_deseq2_1_2_10

Note: the order of the packages are important ...

Ciao,
Bjoern



Cheers,

-E
-Evan Bollig
Research Associate | Application Developer | User Support Consultant
Minnesota Supercomputing Institute
599 Walter Library
612 624 1447
e...@msi.umn.edu
boll0...@umn.edu


On Tue, Jul 29, 2014 at 2:46 PM, Björn Grüning
 wrote:


Hi Evan,

do you have somehow access to this tarball? Than we can put it on

https://github.com/bgruening/download_store

I use that to host R packages to save them for reproducibility.



Can I install another package_r and have packages that depend on
package_r_2_11_0 use the newer version?




I don't think so. But that depends very much on your package. Is it not
possible to use a newer R version? In either case you should save all
tarballs somehow to archive them!

Cheers,
Bjoern


-Evan Bollig
Research Associate | Application Developer | User Support Consultant
Minnesota Supercomputing Institute
599 Walter Library
612 624 1447
e...@msi.umn.edu
boll0...@umn.edu
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/






___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Packaging tools with language specific dependencies

[Björn Grüning]

Hi Renato,

Am 30.07.2014 um 12:21 schrieb Renato Alves:

Hi everyone,

Is there any standard or commonly used way to package tools that have
language specific dependencies.

I know that with Python libraries one can use setup_virtualenv and with
Java jars the JAVA_JAR_LIB strategy is used.
Is there anything equivalent for R, Perl and Ruby libraries?


please have a look at:
https://github.com/bgruening/galaxytools/tree/master/test_repositories

If you have any questions I'm happy to help you!
Bjoern


Thanks
Renato





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] package_r_2_11_0 and ggplot2

[Björn Grüning]

Am 29.07.2014 um 22:45 schrieb Evan Bollig:

Hey Bjoern,

I'm trying to use variant_recalibrator, xy_plot and other tools that
depend on package_r_2_11_0 and require ggplot2 to be installed. Since
CRAN support for R 2.11.0 and 2.15.0 is on the out, perhaps its a
better idea to update the packages the depend on those versions to R
3.0 or 3.1?


Yes, this would be a better idea!


The variant_recalibrator tool generates an Rscript
incompatible with modern versions of ggplot2, so this could be a quite
involved undertaking.


Oh :( ... can we somehow get an old tarball of ggplot2? Have you looked 
at older linux disributions, debian stable for example?


I found ggplot on download_store ;)
https://github.com/bgruening/download_store/blob/master/gatk2_R_deps/ggplot2_0.9.3.1.tar.gz

Is that old enough?


Meanwhile I'll look into the download_store option. You download all
of the tarballs and install them individually with R CMD INSTALL?


Not really, we have a nice routine in the toolshed for such tasks:
Have a look at: 
https://github.com/bgruening/galaxytools/tree/master/orphan_tool_dependencies/package_deseq2_1_2_10


Note: the order of the packages are important ...

Ciao,
Bjoern


Cheers,

-E
-Evan Bollig
Research Associate | Application Developer | User Support Consultant
Minnesota Supercomputing Institute
599 Walter Library
612 624 1447
e...@msi.umn.edu
boll0...@umn.edu


On Tue, Jul 29, 2014 at 2:46 PM, Björn Grüning
 wrote:

Hi Evan,

do you have somehow access to this tarball? Than we can put it on

https://github.com/bgruening/download_store

I use that to host R packages to save them for reproducibility.



Can I install another package_r and have packages that depend on
package_r_2_11_0 use the newer version?



I don't think so. But that depends very much on your package. Is it not
possible to use a newer R version? In either case you should save all
tarballs somehow to archive them!

Cheers,
Bjoern


-Evan Bollig
Research Associate | Application Developer | User Support Consultant
Minnesota Supercomputing Institute
599 Walter Library
612 624 1447
e...@msi.umn.edu
boll0...@umn.edu
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] package_r_2_11_0 and ggplot2

[Björn Grüning]

Hi Evan,

do you have somehow access to this tarball? Than we can put it on

https://github.com/bgruening/download_store

I use that to host R packages to save them for reproducibility.


Can I install another package_r and have packages that depend on
package_r_2_11_0 use the newer version?


I don't think so. But that depends very much on your package. Is it not 
possible to use a newer R version? In either case you should save all 
tarballs somehow to archive them!


Cheers,
Bjoern


-Evan Bollig
Research Associate | Application Developer | User Support Consultant
Minnesota Supercomputing Institute
599 Walter Library
612 624 1447
e...@msi.umn.edu
boll0...@umn.edu
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Error while installing DESeq-hts

[Björn Grüning]

Hi Geert and Vipin,

I might also consider the DESeq package and the wrappers from 
galaxytools if you plan to update your wrappers.


https://github.com/bgruening/galaxytools/tree/master/orphan_tool_dependencies/package_deseq2_1_2_10

@Geert: impressive documentation! Thanks a lot for sharing!
Bjoern


Am 29.07.2014 um 15:54 schrieb Geert Vandeweyer:

I know this is a very old entry to the mailing list, but I thought this
might be useful for some people.

Recently, I set up DESeq-hts and its siblings on our galaxy instance. To
succeed, I performed the steps below.

@vipin: would you consider revising the tool_dependencies.xml for these
tools to take these steps into account? If you're interested, I might
compose a preliminary version.  I believe that most, if not all of these
settings can be automated using the tool_dependencies files.

I did not manage to get DEXSeq working, as it throws errors on the
'estimateDispersions' step in R. I'm not sure how to solve this, as it
gives me the same errors using the demonstration data from the pasilla
library, following the manual steps. (R 3.0.2, bioconductor 2.13, DEXSeq
1.8). Any hints on that are welcome.

Best,

Geert

## GALAXY PACKAGES IN TOOLSHED
###
- install the DESeq-hts tools from the main toolshed (cec4b4fb30be from
vipints)
- install R into galaxy (I used package_R_3_0_2 from iuc (50f7e1e71271))
- install package_scipi and package_numpi (from iuc)
- install package_samtools-0.1.18 (I used
devteam/package_samtools_0_1_18/c0f72bdba484)
- install htseq_count package (I used d5edaf8dc974 from lparsons)


## INSTALL DESeq bioconductor package
#
- open R from the commandline (using the R version installed earlier !),
install bioconductor package 'DESeq' :
 - source("http://bioconductor.org/biocLite.R";)
 - biocLite('DESeq')
 - biocLite('DESeq2')
 - biocLite('DEXSeq') (needed extra ubuntu package
libcurl4-gnutls-dev for this)


## complie samtools from source (temporary)
#
- download samtools 0.1.18 source code into a temporary location and unpack
- compile with : "make CXXFLAGS=-fPIC CFLAGS=-fPIC CPPFLAGS=-fPIC"

## install octave
###
- OS-dependent, ubuntu: apt-get install octave

## set up DESeq-hts-1.0
###
- go to
//toolshed.g2.bx.psu.edu/repos/vipints/deseq_hts/cec4b4fb30be/deseq_hts/deseq-hts_1.0


- run : "setup_deseq-hts.sh" and provide paths as requested.
 - NOTE: set samtools path to the source compiled version for now
(shed version fails to find sam.h)!!
 - NOTE: also add NumPy path to the SCIPY_PATH variable
 - rest: example (available in bin/deseq_config.sh):
 export LD_LIBRARY_PATH=
 export ENVIRONMENT=galaxy
 export DESEQ_VERSION=1.12.1
 export
DESEQ_PATH=/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/vipints/deseq_hts/cec4b4fb30be/deseq_hts/deseq-hts_1.0

 export
DESEQ_SRC_PATH=/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/vipints/deseq_hts/cec4b4fb30be/deseq_hts/deseq-hts_1.0/src

 export
DESEQ_BIN_PATH=/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/vipints/deseq_hts/cec4b4fb30be/deseq_hts/deseq-hts_1.0/bin

 export INTERPRETER=octave
 export MATLAB_BIN_PATH=
 export MATLAB_MEX_PATH=
 export MATLAB_INCLUDE_DIR=
 export OCTAVE_BIN_PATH=/opt/software/octave-3.8.0/bin/octave
 export OCTAVE_MKOCT=/opt/software/octave-3.8.0/bin/mkoctfile
 export SAMTOOLS_DIR=/galaxy/packages_sources/samtools-0.1.18_fPIC
 #export
SAMTOOLS_DIR=/galaxy/galaxy_tool_binaries/samtools/0.1.18/devteam/package_samtools_0_1_18/c0f72bdba484/bin/

 export PYTHON_PATH=/galaxy/galaxy_env/bin/python
 export
SCIPY_PATH=/galaxy/galaxy_tool_binaries/scipy/0.12.0/iuc/package_scipy_0_12/984d208b0808/lib/python/:/galaxy/galaxy_tool_binaries/numpy/1.7.1/iuc/package_numpy_1_7/ef12a3a11d5b/lib/python/

 ## USE THE TOOLSHED VERSION, The one you installed bioclite
packages with.
 export
R_PATH=/galaxy/galaxy_tool_binaries/R_3_0_2/3.0.2/iuc/package_r_3_0_2/50f7e1e71271/R


- enter mex directory, run "make octave"
- swap SAMTOOLS_DIR in bin/deseq_config.sh to point to the toolshed
version of samtools 0.1.18

## RUN DESeq-hts-1.0 TEST
##
enter src directory, run test from the toolconf xml file:
"./deseq-hts.sh ../test_data/deseq_c_elegans_WS200-I-regions.gff3
../test_data/deseq_c_elegans_WS200-I-regions_deseq.txt
../test_data/genes.mat
../test_data/deseq_c_elegans_WS200-I-regions-SRX001872.bam
../test_data/deseq_c_elegans_WS200-I-regions-SRX001875.bam"


## set up DESeq-hts-2.0
###
- go to
//toolshed.g2.bx.psu.edu/repos/vipints/deseq_hts/cec4b4fb30be/deseq_hts/deseq-hts_2.0

- similar to 1.0: setup config file,
 - point to src compiled version of samtools
 - add scipy and numpy to scipi_path variable
 - use the toolshed version of R
- in mex, run "make octave"
- swap samtools path

IMPORTANT:

Re: [galaxy-dev] IE 11 issues

[Björn Grüning]

Hi Carl & Dori,

sorry for the misinformation, for some reason I thought the official 
statement was that IE is not supported.


Thanks for clarification!
Bjoern

Am 28.07.2014 um 16:54 schrieb Carl Eberhard:

Hi, Dori

Galaxy supports IE 10+ so this is definitely a bug. I'm checking for bugs
in the history panel now, but in the meantime:
  - can you open the javascript console in your IE and see if any errors are
displayed when doing the two tasks you mention?
http://msdn.microsoft.com/en-us/library/ie/bg182326%28v=vs.85%29.aspx
  - What version of Galaxy does your lab use currently? (You can view the
version by using the 'hg summary' command from the directory from which
Galaxy is run)

As an aside, IE has traditionally veered away from web standards and made
it difficult to program compatible sites. Recent versions of IE have become
much more standards compliant allowing us to support them well enough.

Please don't hesitate to report IE (10+) bugs when using Galaxy.

Thanks for the report and I'll update with my findings soon,
Carl



On Mon, Jul 28, 2014 at 10:42 AM, Björn Grüning 
wrote:


Hi Dori,

the IE is not really supported. There are many issues with IE and it's
hard to deal with that. If possible please you Chrome or FF, both are
supported and tested.

Cheers,
Bjoern

Am 28.07.2014 um 16:40 schrieb Sajdak, Doris:


We are having issues with using Galaxy in Internet Explorer 11 (these
work fine in Chrome & Firefox).


1.   The jobs queued and history doesn't display unless you refresh
the browser window.  Clicking the history reload icon doesn't work.

2.   If I click to edit something in the history, the tabs aren't
clickable.  Attributes are displayed but I can't click on Convert Format,
Datatype, or Permissions.

Anyone else having this problem?  I apologize if this information is
documented somewhere but I was unable to find it.

Thank you for your assistance,
Dori

**
Dori Sajdak
Senior Systems Administrator
State University of NY at Buffalo
Center for Computational Research
701 Ellicott St
Buffalo, NY 14203
Phone: (716) 881-8934
Fax: (716) 849-6656
Web: http://ccr.buffalo.edu
**





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/

  ___

Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] IE 11 issues

[Björn Grüning]

Hi Dori,

the IE is not really supported. There are many issues with IE and it's 
hard to deal with that. If possible please you Chrome or FF, both are 
supported and tested.


Cheers,
Bjoern

Am 28.07.2014 um 16:40 schrieb Sajdak, Doris:

We are having issues with using Galaxy in Internet Explorer 11 (these work fine in 
Chrome & Firefox).


1.   The jobs queued and history doesn't display unless you refresh the 
browser window.  Clicking the history reload icon doesn't work.

2.   If I click to edit something in the history, the tabs aren't 
clickable.  Attributes are displayed but I can't click on Convert Format, 
Datatype, or Permissions.

Anyone else having this problem?  I apologize if this information is documented 
somewhere but I was unable to find it.

Thank you for your assistance,
Dori

**
Dori Sajdak
Senior Systems Administrator
State University of NY at Buffalo
Center for Computational Research
701 Ellicott St
Buffalo, NY 14203
Phone: (716) 881-8934
Fax: (716) 849-6656
Web: http://ccr.buffalo.edu
**





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] [galaxy-iuc] writing datatypes

[Björn Grüning]

Hi Greg,

thanks for the clarification. Please see my comments below.


On Jul 20, 2014, at 3:22 PM, Peter Cock  wrote:


On Sun, Jul 20, 2014 at 6:23 PM, Björn Grüning  wrote:

Hi,

single datatype definitions only work if you haven’t defined any converters.
Let's assume I have a datatype X and want to ship a X -> Y converter (Y -> X
is also possible), we will end up with a dependency loop, or? The X
repository will depend on the Y repository, but Y is depending on X, because
we want to include a Y -> X converter.

Any idea how to solve that?


I don't see a problem here, so I'm hoping I'm correctly understanding the issue.

If we have:

repo_x contains the single datatype X
repo_y contains the single datatype Y
repo_x_to_y_converter contains a tool that converts datatype X to datatype Y 
(this repository also defines 2 dependency relationships, one to repo_x and 
another to repo_y)
repo_y_to_x_cenverter contains a tool that converts datatype Y to datatype X 
(this repository also defines 2 dependency relationships, one to repo_x and 
another to repo_y)

Now if we want to install both the repo_x_to_y_converter and the 
repo_y_to_x_cenverter automatically whenever either one is installed, we have 2 
options:

1) define a 3rd dependency relationshiop for repo_x_to_y_converter to depend on 
repo_y_to_x_cenverter and, similarly a 3rd dependency relationshiop for 
repo_y_to_x_cenverter on repo_x_to_y_converter.  This does indeed
create a circular repository dependency relationship, but the Tool Shed 
installation process will handle it correctly, installing all 4 repositories 
with proper dependency relationships created between them


Does that mean, circular dependencies will be no problem at all?
Do you consider including the converters into the datatypes as 
best-practise? (These converters are implicit-galaxy-converters).

I would have only two repositories with circular dependencies.


2) Instead of creating a circlular dependency relationship between repo_x_to_y_converter 
and repo_y_to_x_cenverter, create an additional suite_definition_x_y repository (of type 
"repository_suite_definition" that defines relationships to 
repo_x_to_y_converter and  repo_y_to_x_cenverter, ultimately installing all 4 
repositories, but without defining any circular dependency relationships.


repo_x_to_y_converter and repo_y_to_x_converter would have dependencies 
on datatype X and Y, so I do not see the need for a suite_definition ... 
or it is some collection like the emboss_datatypes ...


My scenario is more that the converters are not tools, they are implicit 
converters and should _not_ be displayed in the tool panel.
As far as I know they need to be defined inside the datatypes_conf.xml 
file.


I think if circular dependencies are not a problem I will try to 
implement a proof of concept. EMBOSS is now splitted:


https://github.com/bgruening/galaxytools/tree/master/datatypes/emboss_datatypes

Thanks Greg!
Bjoern


Either of the above 2 scenarios will correctly install the 4 repositories.

Let me know if I'm missing something here.

Thanks!

Greg



Excellent example!


How to handle versions of datatypes? Extra repositories for stockholm 1.0
and 1.1? If so ... the associated python file (sniffing, splitting ...)
should be also versioned, or? What happend if I have two stockholm.py files
in my system?


Potentially you might need/want to define those as two different
Galaxy datatypes?


@Peter, can we create a striped-down, python only biopython egg? All parsers
should be included, Bio.SeqIO should be sufficient I think.


Right now, yes in principle (and this is fine from the licence point of view),
but in practise this is a fair chunk of work. However, we are looking at
this - see https://github.com/biopython/biopython/issues/349

Peter

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/




___
galaxy-iuc mailing list
galaxy-...@lists.bx.psu.edu
http://lists.bx.psu.edu/listinfo/galaxy-iuc


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Once-run galaxy archives

[Björn Grüning]

:) great I like it!
Will do it shortly!

Am 22.07.2014 15:36, schrieb Eric Rasche:

Hi Björn,

22.07.2014, 14:26, "Björn Grüning" :

  Hi Eric,

   That sounds like a pretty good idea.  If there was a pre-built image
   available for whatever release I wanted to test against I could just

   cache

   it and (hopefully) get my tests running a bit faster.  I'm not sure

   if

   anyone else is already doing this?

   Also, I remember there being mentioned pre-building docker images for

   each

   release of Galaxy, which would accomplish something similar, but I'm

   not

   really sure how that's being handled.  I think Björn's Docker image

   is kept

   up to date with Galaxy stable each time it's built

   
https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51.

   So, this could be handled by modifying his Dockerfile to build Galaxy

   at

   whatever tagged release you want to test against.

   I will try hard to create with every Galaxy stable release a new Galaxy

   docker image. You can create a github branch with a specific tag, that
   will end up as a new tagged version of the main Galaxy Docker image.

   Try hard to create? No no, what can we do to automatically create these? I'm 
not so familiar with how one might build a galaxy release specific docker 
image, but if you can provide a generalized process, let's stick it in a CI 
server/cron job somewhere and never worry about it again!

  The hardest part is to remind myself ;)
  The procedure is:

  1) create a new branch for the galaxy-docker github account
  2) change the version string in the git-clone command in the Dockerfile
  3) login into the docker-index site and re-associate a new tag to the
  new branch ... click the build button

  I could simplify that a little bit if the galaxy-docker image has its
  own repository. The docker-index has a build-on-push feature.
  But currently every image (all branches) are build again. There is no
  way to only trigger one branch build. Until that is fixed in the
  docker-index I will do it manually.

  So you see any improvements in that setup? Let me know!


Naturally! 1 and 2 could be automated out with a script. 3 could probably be 
fixed with a script, but that requires parsing pages/crafting cURL queries and 
that's less pleasant.

Let's have a new repository just for galaxy-docker images. I'll write up a script to 
check for updates and create+push branches as needed, we can place this in a cron/CI job 
and have it email you whenever it's run to say "hey, associate the branch/trigger a 
build"


  Cheers,
  Bjoern

   One downside to docker is that you need to get it installed on your

   CI

   server, which may or may not be possible (needs a very recent kernel

   for

   example).

   So true. SL7 for the win! :)

   Docker, Docker, Docker!
   Bjoern

   Aaron

   On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche 

   wrote:

   Hi Aaron,

   Yeah, absolutely understandable. I want my tools tested early and

   often.

   I abuse my CI server for everything, especially for building and
   packaging software. In this case I was imagining that I might have

   it

   produce an archive on every tagged release, as well as producing a
   "daily" archive. All of these would be available on some ftp/http

   server

   somewhere with symlinks for latest archives (e.g.
   galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that

   work

   for your use case as well?

   Eric

   On 07/21/2014 03:02 PM, Aaron Petkau wrote:

   Hello Eric,

   Your right about that, downloading the archive, installing all the

   eggs,

   and then updating the database takes a bit of time (especially if

   you're

   like me and like re-running tests on nearly every change you make

   :P).

   I think it would be cool to have a pre-package Galaxy for

   integration

   testing which is quick to setup.  I once thought of downloading

   Björn's

   Docker image from Galaxy Bootstrap and using it that way, but

   thinking

   is about as far as I got with that one.  One problem I could see is

   it

   would have to be re-built on every release of Galaxy you want to

   test

   against (whereas mercurial cloning/pulling makes sure you're always

   up

   to date with the latest code).

   Aaron

   On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche mailto:rasche.e...@yandex.ru>> wrote:

 Hi Aaron,

 Good points, I was considering using galaxy bootstrap. This is
 mostly for the CI folk who want to download an archive, unpack

   it,

 and be ready to install/test their tools. The hg clone and

   egg/db

 steps seem like unnecessary overhead for this specific use

   case.

 Cheers,

 Eric

   ___
   Please keep all replies on the list by using "rep

Re: [galaxy-dev] Once-run galaxy archives

[Björn Grüning]

Hi Eric,


That sounds like a pretty good idea.  If there was a pre-built image
available for whatever release I wanted to test against I could just

cache

it and (hopefully) get my tests running a bit faster.  I'm not sure

if

anyone else is already doing this?

Also, I remember there being mentioned pre-building docker images for

each

release of Galaxy, which would accomplish something similar, but I'm

not

really sure how that's being handled.  I think Björn's Docker image

is kept

up to date with Galaxy stable each time it's built


https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51.

So, this could be handled by modifying his Dockerfile to build Galaxy

at

whatever tagged release you want to test against.


I will try hard to create with every Galaxy stable release a new Galaxy

docker image. You can create a github branch with a specific tag, that
will end up as a new tagged version of the main Galaxy Docker image.


Try hard to create? No no, what can we do to automatically create these? I'm 
not so familiar with how one might build a galaxy release specific docker 
image, but if you can provide a generalized process, let's stick it in a CI 
server/cron job somewhere and never worry about it again!


The hardest part is to remind myself ;)
The procedure is:

1) create a new branch for the galaxy-docker github account
2) change the version string in the git-clone command in the Dockerfile
3) login into the docker-index site and re-associate a new tag to the 
new branch ... click the build button


I could simplify that a little bit if the galaxy-docker image has its 
own repository. The docker-index has a build-on-push feature.
But currently every image (all branches) are build again. There is no 
way to only trigger one branch build. Until that is fixed in the 
docker-index I will do it manually.


So you see any improvements in that setup? Let me know!
Cheers,
Bjoern


One downside to docker is that you need to get it installed on your

CI

server, which may or may not be possible (needs a very recent kernel

for

example).


So true. SL7 for the win! :)

Docker, Docker, Docker!
Bjoern


Aaron


On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche 

wrote:



Hi Aaron,

Yeah, absolutely understandable. I want my tools tested early and

often.


I abuse my CI server for everything, especially for building and
packaging software. In this case I was imagining that I might have

it

produce an archive on every tagged release, as well as producing a
"daily" archive. All of these would be available on some ftp/http

server

somewhere with symlinks for latest archives (e.g.
galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that

work

for your use case as well?

Eric

On 07/21/2014 03:02 PM, Aaron Petkau wrote:

Hello Eric,

Your right about that, downloading the archive, installing all the

eggs,

and then updating the database takes a bit of time (especially if

you're

like me and like re-running tests on nearly every change you make

:P).

I think it would be cool to have a pre-package Galaxy for

integration

testing which is quick to setup.  I once thought of downloading

Björn's

Docker image from Galaxy Bootstrap and using it that way, but

thinking

is about as far as I got with that one.  One problem I could see is

it

would have to be re-built on every release of Galaxy you want to

test

against (whereas mercurial cloning/pulling makes sure you're always

up

to date with the latest code).

Aaron


On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche mailto:rasche.e...@yandex.ru>> wrote:

  Hi Aaron,

  Good points, I was considering using galaxy bootstrap. This is
  mostly for the CI folk who want to download an archive, unpack

it,

  and be ready to install/test their tools. The hg clone and

egg/db

  steps seem like unnecessary overhead for this specific use

case.


  Cheers,

  Eric








___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Once-run galaxy archives

[Björn Grüning]

Hi Aaron and Eric,

Am 21.07.2014 22:58, schrieb Aaron Petkau:

Hello Eric,

That sounds like a pretty good idea.  If there was a pre-built image
available for whatever release I wanted to test against I could just cache
it and (hopefully) get my tests running a bit faster.  I'm not sure if
anyone else is already doing this?

Also, I remember there being mentioned pre-building docker images for each
release of Galaxy, which would accomplish something similar, but I'm not
really sure how that's being handled.  I think Björn's Docker image is kept
up to date with Galaxy stable each time it's built
https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51.
So, this could be handled by modifying his Dockerfile to build Galaxy at
whatever tagged release you want to test against.


I will try hard to create with every Galaxy stable release a new Galaxy 
docker image. You can create a github branch with a specific tag, that 
will end up as a new tagged version of the main Galaxy Docker image.



One downside to docker is that you need to get it installed on your CI
server, which may or may not be possible (needs a very recent kernel for
example).


So true. SL7 for the win! :)

Docker, Docker, Docker!
Bjoern


Aaron


On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche  wrote:


Hi Aaron,

Yeah, absolutely understandable. I want my tools tested early and often.

I abuse my CI server for everything, especially for building and
packaging software. In this case I was imagining that I might have it
produce an archive on every tagged release, as well as producing a
"daily" archive. All of these would be available on some ftp/http server
somewhere with symlinks for latest archives (e.g.
galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work
for your use case as well?

Eric

On 07/21/2014 03:02 PM, Aaron Petkau wrote:

Hello Eric,

Your right about that, downloading the archive, installing all the eggs,
and then updating the database takes a bit of time (especially if you're
like me and like re-running tests on nearly every change you make :P).
I think it would be cool to have a pre-package Galaxy for integration
testing which is quick to setup.  I once thought of downloading Björn's
Docker image from Galaxy Bootstrap and using it that way, but thinking
is about as far as I got with that one.  One problem I could see is it
would have to be re-built on every release of Galaxy you want to test
against (whereas mercurial cloning/pulling makes sure you're always up
to date with the latest code).

Aaron


On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche mailto:rasche.e...@yandex.ru>> wrote:

 Hi Aaron,

 Good points, I was considering using galaxy bootstrap. This is
 mostly for the CI folk who want to download an archive, unpack it,
 and be ready to install/test their tools. The hg clone and egg/db
 steps seem like unnecessary overhead for this specific use case.

 Cheers,

 Eric








___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] How to configure galaxy with a cluster

[Björn Grüning]

Hi Ben,

if the job is in waiting in the queue it's unlikely (not impossible) 
that it is Galaxy fault. Can you recheck your Torque setup and how many 
cores and memory your job has requested?


Ciao,
Bjoern

Am 22.07.2014 10:09, schrieb 王渭巍:

Hi, Bjoern,
 I've tried the latest galaxy version with Torque 4.1.7, and it seems all 
right. But torque version > 4.2 won't work.
 And I tried to submit“fastqc readqc” jobs via torque (runner pbs),  
but the job is always in the queue waiting. I submited “fastqc readqc”local  
(runner local) , and the job finished successfully. So the question is , it 
seems not all the tools can be submitted via torque (or other resource 
manager), right?



王渭巍

From: Björn Grüning
Date: 2014-07-21 01:23
To: 王渭巍; Björn Grüning; galaxy-dev
Subject: Re: [galaxy-dev] How to configure galaxy with a cluster
Hi Ben,

sorry but we do not run a Torque setup.

Do you have any concrete questions or error messages?

Cheers,
Bjoern

Am 17.07.2014 04:10, schrieb 王渭巍:

Hi, Bjoern
  Would you share your  procedure to make some tools to run on a 
cluster.
  I have tried 
https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster using Torque, 
but got errors.
  I think maybe it's job_conf.xml. Would you share yours?  Thanks a lot

Ben


From: Björn Grüning
Date: 2014-07-16 16:34
To: 王渭巍; Thomas Bellembois; galaxy-dev
Subject: Re: [galaxy-dev] How to configure galaxy with a cluster
Hi Ben,

that is not possible at the moment. The idea is to keep the
user-inferface as easy as possible for the user. You, as admin, can
decide which resource a specific tool with a specific input will use.
You will never see any options like that in a tool, but you can write a
tool by yourself if you like, or "enhance" the megablast tool.

Cheers,
Bjoern


Am 16.07.2014 09:43, schrieb 王渭巍:

Thanks a lot, Thomas! It really helps, I added tools section followed your 
suggestion...

here is my job_conf.xml ( I am using Torque,  I have 3 servers. One for galaxy 
server, two for cluster computing.  )










walltime=72:00:00,nodes=1:ppn=8
128







and still no cluster options in "megablast" item.  How can I see cluster 
options in the page, for example, the page will let me choose to use local server or a 
cluster.

Ben



From: Thomas Bellembois
Date: 2014-07-15 17:41
To: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] How to configure galaxy with a cluster
Hello Ben,

you can configure your Galaxy instance to use your cluster in the
job_conf.xml file:

https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster

You can set up your instance to use your cluster by default for all jobs
or only for specific jobs.

Here is a part of my job_conf.xml for example:

   

   

   
   
   

   
   
   
   

   
   
   
 -r yes -b n -cwd -S /bin/bash
-V -pe galaxy 1
   
   
 -r yes -b n -cwd -S /bin/bash
-V -pe galaxy 12
   

   

   
   
   
   
   
   
   
   
   
   
   
   


Moreover you Galaxy user and Galaxy server must be allowed to submit
jobs to your scheduler.

Hope it  helps,

Thomas




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] XML/json code to parse/get/create wrapper files to/from galaxy

[Björn Grüning]

Very cool, I will give it a try :)

Am 21.07.2014 16:27, schrieb Rémy Dernat:

Hi,

I created a little repository on github to parse or create wrapper files. I
have not implemented everything, but it is a good base, i.e. if you want to
create a galaxy wrapper from a standard help output !

Just clone it and have fun :

git clone https://github.com/remyd1/XMLparser-wrapper.git
cd XMLparser-wrapper
python ParserConverterXML.py --help


python ParserConverterXML.py h2gw -c "ls --help" -o ../ls.gw.xml

Any help is welcome on this little project. Exemples : json for workflows...

Best regards,


Remy



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy+Toolshed Docker image?

[Björn Grüning]

Hi Kyle,

not that I know of, but I'm interested in adding such a feature on top 
of the base galaxy image.


You can find the Dockerfile here:
https://github.com/bgruening/docker-recipes

Ciao,
Bjoern

Am 21.07.2014 15:39, schrieb Kyle Ellrott:

I know there is a base docker image for Galaxy stable (
https://registry.hub.docker.com/u/bgruening/galaxy-stable/), but is there a
docker image that will start both a server and a toolshed sever and link
them?
I was hoping that I could use something like that for testing shed based
tool deployment.

Kyle



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] writing datatypes

[Björn Grüning]

Hi,

single datatype definitions only work if you haven’t defined any 
converters. Let's assume I have a datatype X and want to ship a X -> Y 
converter (Y -> X is also possible), we will end up with a dependency 
loop, or? The X repository will depend on the Y repository, but Y is 
depending on X, because we want to include a Y -> X converter.


Any idea how to solve that?

How to handle versions of datatypes? Extra repositories for stockholm 
1.0 and 1.1? If so ... the associated python file (sniffing, splitting 
...) should be also versioned, or? What happend if I have two 
stockholm.py files in my system?


@Peter, can we create a striped-down, python only biopython egg? All 
parsers should be included, Bio.SeqIO should be sufficient I think.


Ciao,
Bjoern

Am 17.07.2014 19:43, schrieb Greg Von Kuster:

Here's a Trello card for this:

https://trello.com/c/s8tfbW4x

On Jul 17, 2014, at 1:38 PM, Peter Cock  wrote:


Good point Greg.

Let's refine this slightly then, a new special ToolShed repository type for
a *single* datatype definition. That avoids this problem :)

(This does not help with suites of very closely related datatypes -
like different
kinds of BLAST database.)

Peter

On Thu, Jul 17, 2014 at 6:35 PM, Greg Von Kuster  wrote:

This would be easy to implement, but could adversely affect reproducibility.
If a repository containing datatypes always had only a single installable
revision (i.e., the chagelog tip), then any datatypes defined in an early
changeset revision that are removed in a later changeset revision would
no longer be available.

Greg

On Jul 17, 2014, at 1:30 PM, Peter Cock  wrote:


On Thu, Jul 17, 2014 at 6:10 PM, Björn Grüning
 wrote:


... but the problem will stay the same ... one [datatype definition] repository
can have multiple versions ...



I like your idea that like tool dependency definitions, this should be a special
repository type on the ToolShed:

Earlier, Björn Grüning  wrote:


Imho datatypes should be handled like "Tool dependency definitions".
There should be only one "installable revsion".



This is something Greg will have to comment on - there may be
ramifications I'm not seeing.

Peter

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] How to configure galaxy with a cluster

[Björn Grüning]

Hi Ben,

sorry but we do not run a Torque setup.

Do you have any concrete questions or error messages?

Cheers,
Bjoern

Am 17.07.2014 04:10, schrieb 王渭巍:

Hi, Bjoern
 Would you share your  procedure to make some tools to run on a cluster.
 I have tried 
https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster using Torque, 
but got errors.
 I think maybe it's job_conf.xml. Would you share yours?  Thanks a lot

Ben


From: Björn Grüning
Date: 2014-07-16 16:34
To: 王渭巍; Thomas Bellembois; galaxy-dev
Subject: Re: [galaxy-dev] How to configure galaxy with a cluster
Hi Ben,

that is not possible at the moment. The idea is to keep the
user-inferface as easy as possible for the user. You, as admin, can
decide which resource a specific tool with a specific input will use.
You will never see any options like that in a tool, but you can write a
tool by yourself if you like, or "enhance" the megablast tool.

Cheers,
Bjoern


Am 16.07.2014 09:43, schrieb 王渭巍:

Thanks a lot, Thomas! It really helps, I added tools section followed your 
suggestion...

here is my job_conf.xml ( I am using Torque,  I have 3 servers. One for galaxy 
server, two for cluster computing.  )










walltime=72:00:00,nodes=1:ppn=8
128







and still no cluster options in "megablast" item.  How can I see cluster 
options in the page, for example, the page will let me choose to use local server or a 
cluster.

Ben



From: Thomas Bellembois
Date: 2014-07-15 17:41
To: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] How to configure galaxy with a cluster
Hello Ben,

you can configure your Galaxy instance to use your cluster in the
job_conf.xml file:

https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster

You can set up your instance to use your cluster by default for all jobs
or only for specific jobs.

Here is a part of my job_conf.xml for example:

  

  

  
  
  

  
  
  
  

  
  
  
-r yes -b n -cwd -S /bin/bash
-V -pe galaxy 1
  
  
-r yes -b n -cwd -S /bin/bash
-V -pe galaxy 12
  

  

  
  
  
  
  
  
  
  
  
  
  
  


Moreover you Galaxy user and Galaxy server must be allowed to submit
jobs to your scheduler.

Hope it  helps,

Thomas




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] writing datatypes

[Björn Grüning]

Am 17.07.2014 18:51, schrieb Peter Cock:

On Thu, Jul 17, 2014 at 5:45 PM, Björn Grüning
 wrote:

Hi,

I think you are right John. Datatypes have many issues in that regard as I
can tell, from a few bug reports. Imho datatypes should be handled like
"Tool dependency definitions". There should be only one "installable
revsion".

But that aside, emboss datatypes are already broken. For example asn1 was
added into Galaxy but it still exists in emboss_datatypes.

Moreover, howto add a proper genbank datatype with sniffer, split and merge
functions? Ideally, every datatype should have its own repository, but that
is an overhead I would like to omit ... any other ideas?

I would love to discuss that issue further, maybe a hangout with Greg and
Peter?

Thanks John for your input,
Bjoern


This could be high level, e.g. "other sequence file formats" repository
covering GenBank, EMBL, SwissProt plain text, UniProt XML, etc;
one for multiple sequence alignments; one for EMBOSS' own output...


That was my initial idea. Starting point is here:
https://github.com/bgruening/galaxytools/tree/master/datatypes


But it wouldn't be that much more work to do one ToolShed repo
per additional file format, would it?


Uploading and creating descriptions in the toolshed will take most of 
the time :)
Lets see if I can use a train trip to do that ... but the problem will 
stay the same ... one repository can have multiple versions ...



One reason I have been meaning to do some of these is familiarity with
many of these formats from looking after/writing parsers in Biopython.

Having this done sooner rather than later ought to head off too many
incompatible datatype names which worries me. Is it too late to adopt
something like the EDAM ontology for the datatypes within Galaxy?

Peter


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] writing datatypes

[Björn Grüning]

Hi,

I think you are right John. Datatypes have many issues in that regard as 
I can tell, from a few bug reports. Imho datatypes should be handled 
like "Tool dependency definitions". There should be only one 
"installable revsion".


But that aside, emboss datatypes are already broken. For example asn1 
was added into Galaxy but it still exists in emboss_datatypes.


Moreover, howto add a proper genbank datatype with sniffer, split and 
merge functions? Ideally, every datatype should have its own repository, 
but that is an overhead I would like to omit ... any other ideas?


I would love to discuss that issue further, maybe a hangout with Greg 
and Peter?


Thanks John for your input,
Bjoern

Am 17.07.2014 18:24, schrieb John Chilton:

Even more out of office than normal so maybe I don't have the throughput to
process this but it sounds like it won't work then. If the new types aren't
going to be loaded than we cannot evolve the datatypes with new
functionality in new repositories.

Perhaps I am missing something,  but in the abstract it seems Galaxy would
have no way of knowing which types are new and which types are old in this
scenario.

Not really my business so feel free to proceed however obviously - just
letting you know that it makes me nervous. I will try to find the time to
understand how the tool shed handles data types so I can speak with less
ignorance in the future.

-John


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] writing datatypes

[Björn Grüning]

Hi Eric,


Forgive me, I'm not 100% clear on the custom plugin system used by galaxy, but if I 
"subclass" from the text data type, will sniffers I implement override text's 
and function? The lack of being able to add an entry to the sniffer section (unlike with 
the tabular example) led me to believe my genbank datatype wouldn't be sniffed.


Thats true, if you want to override functions, you need to subclass it 
on a python level not on the XML level.



Additionally, I'd still like to be able to add completely new datatypes, do you 
know of any working examples of this? As mentioned in my original post, 
duplicating an existing datatype and changing names on it surprisingly doesn't 
work.


https://github.com/bgruening/galaxytools/tree/master/datatypes/msa_datatypes
https://github.com/bgruening/galaxytools/blob/master/chemicaltoolbox/datatypes/datatypes_conf.xml

Is that enough, to get started?


I'd be lovely to have the emboss datatypes split out.


Ok, than lets start :)
I will try to fork emboss into my galaxytools/datatypes repository and 
try to split them. You will get commit access and can improve your 
genbank datatype (and a few more ;)). Finally, we will talk to the 
devteam to rewrite EMBOSS to depend on our separate data type 
repositories. OK?


Ciao,
Bjoenr


Cheers,
Eric

On July 16, 2014 8:34:55 AM CDT, Peter Cock  wrote:

Indeed - ideally (once working) we can upload under the IUC ToolShed as
a
community maintained resource rather than under a personal account
which
becomes a single point of failure (the bus factor).

We (the ICU) have previously discussed doing this so that the EMBOSS
datatypes could become more of a meta-entry depending on other smaller
specific datatype defining ToolShed repositories. But it hasn't reached
the
top of my personal TODO list yet ;)

Peter

On Wed, Jul 16, 2014 at 1:47 PM, Björn Grüning
 wrote:

Hi Eric,

please have a look at:



https://github.com/bgruening/galaxytools/blob/master/datatypes/msa_datatypes/datatypes_conf.xml


You need somthing like:


Lets try to split the EMBOSS datatypes a little bit into small

chunks. E.g.

sequences_datatypes, msa_datatypes ... and so on ...

Cheers,
Bjoern


Am 14.07.2014 20:31, schrieb Eric Rasche:


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

I'm trying to add a new datatype to my galaxy instance for genbank
files, however I'm running into various issues. I've followed the
tutorial
(https://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes)

however that example subclasses tabular, and I'd like to subclass

Text

as they're plain text files, and I'd like to be able to define a

sniffer

for them (not possible if your type=galaxy.datatypes.data:Text)

I figured the call ought to be something like



however, everything I try fails with


Error importing datatype module galaxy.datatypes.data: 'module'

object

has no attribute 'Genbank'



To avoid this particular issue, I tried writing a separate datatype

just

for genbank files (type="galaxy.datatypes.genbank:Genbank"), however
that fails with the same error:


galaxy.datatypes.registry ERROR 2014-07-14 13:23:23,100 Error

importing

datatype module galaxy.datatypes.genbank: 'module' object has no

attribute

'genbank'
Traceback (most recent call last):
File

"/home/hxr/work/galaxy-central/lib/galaxy/datatypes/registry.py",

line 206, in load_datatypes
  module = getattr( module, mod )
AttributeError: 'module' object has no attribute 'genbank'



Here's my lib/galaxy/datatypes/genbank.py looks like:


import pkg_resources
pkg_resources.require( "bx-python" )
import logging
from galaxy.datatypes import data
log = logging.getLogger(__name__)

class Genbank( data.Text ):
  file_ext = "gb"

  def sniff( self, filename ):
  header = open(filename).read(5)
  return header == 'LOCUS'



To debug this, I've tried copying the tabular data type completely,
removed all the classes other than Tabular, and renamed it

"Genbank",

however this fails too with the same error.

Can anyone offer some insight?

Cheers,
Eric
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJTxCHwAAoJEMqDXdrsMcpVmbsQAJ3eFIhZtZmVP9LCz/F9Ywg/
148NJZy4lmxZU0KScJlc8kVDCDSADXIHd0Db/kpJwuUKEX7zei9q2uXfO7sWl3yt
yxrFEdtX/a5SMVsa6F5WZuKwBs0zfvfsnIUoraOgh6nXeJnr53l9mYeWaKB6bi3Z
xAlgJG/kdIR1jRjAimuQf4vMjNgtDQPOmotYBQTytbhsV6/nRzGI8RZAYwQ7GnVs
XYOWFyhzrBgALndVI3BjI21rbRqguhrqr2t7i0Ma7Pp2JmAnNjmUaq70NN3Rueh6
DvnTtxInM1dVOQY+Yam6MCMmAedV1cG+rNGdpP2l82MajQAsMtbXckBXXKcSgyTq
WCFoLVURYO1tHkWyq4ikamfFDHtJp1DogBYhUiPMyRw+CV+3sOvr0U5DcyRdiDsJ
Xcm3ygqYVLGwauNmuN3yGcQcnfypDOOeFs1lppbNe3lw0w3ikZN4Zmu1ec5s1ITK
MEcgBrGYgZrKDRXkx53lnABGpv6mYflYpag7fguDNL8j0lh9beaaNmHr4tmeEcug
VZ1b1EWoLMj/ikJ/vZcluiHPTSTheiAP8Ttvh1WAayq4rKwVtZygaI9IDauqqBQ1
Dgote

Re: [galaxy-dev] writing datatypes

[Björn Grüning]

Hi Eric,

please have a look at:

https://github.com/bgruening/galaxytools/blob/master/datatypes/msa_datatypes/datatypes_conf.xml

You need somthing like:
subclass="True" />


Lets try to split the EMBOSS datatypes a little bit into small chunks. 
E.g. sequences_datatypes, msa_datatypes ... and so on ...


Cheers,
Bjoern


Am 14.07.2014 20:31, schrieb Eric Rasche:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

I'm trying to add a new datatype to my galaxy instance for genbank
files, however I'm running into various issues. I've followed the
tutorial (https://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes)

however that example subclasses tabular, and I'd like to subclass Text
as they're plain text files, and I'd like to be able to define a sniffer
for them (not possible if your type=galaxy.datatypes.data:Text)

I figured the call ought to be something like



however, everything I try fails with


Error importing datatype module galaxy.datatypes.data: 'module' object has no 
attribute 'Genbank'


To avoid this particular issue, I tried writing a separate datatype just
for genbank files (type="galaxy.datatypes.genbank:Genbank"), however
that fails with the same error:


galaxy.datatypes.registry ERROR 2014-07-14 13:23:23,100 Error importing 
datatype module galaxy.datatypes.genbank: 'module' object has no attribute 
'genbank'
Traceback (most recent call last):
   File "/home/hxr/work/galaxy-central/lib/galaxy/datatypes/registry.py", line 
206, in load_datatypes
 module = getattr( module, mod )
AttributeError: 'module' object has no attribute 'genbank'


Here's my lib/galaxy/datatypes/genbank.py looks like:


import pkg_resources
pkg_resources.require( "bx-python" )
import logging
from galaxy.datatypes import data
log = logging.getLogger(__name__)

class Genbank( data.Text ):
 file_ext = "gb"

 def sniff( self, filename ):
 header = open(filename).read(5)
 return header == 'LOCUS'


To debug this, I've tried copying the tabular data type completely,
removed all the classes other than Tabular, and renamed it "Genbank",
however this fails too with the same error.

Can anyone offer some insight?

Cheers,
Eric
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJTxCHwAAoJEMqDXdrsMcpVmbsQAJ3eFIhZtZmVP9LCz/F9Ywg/
148NJZy4lmxZU0KScJlc8kVDCDSADXIHd0Db/kpJwuUKEX7zei9q2uXfO7sWl3yt
yxrFEdtX/a5SMVsa6F5WZuKwBs0zfvfsnIUoraOgh6nXeJnr53l9mYeWaKB6bi3Z
xAlgJG/kdIR1jRjAimuQf4vMjNgtDQPOmotYBQTytbhsV6/nRzGI8RZAYwQ7GnVs
XYOWFyhzrBgALndVI3BjI21rbRqguhrqr2t7i0Ma7Pp2JmAnNjmUaq70NN3Rueh6
DvnTtxInM1dVOQY+Yam6MCMmAedV1cG+rNGdpP2l82MajQAsMtbXckBXXKcSgyTq
WCFoLVURYO1tHkWyq4ikamfFDHtJp1DogBYhUiPMyRw+CV+3sOvr0U5DcyRdiDsJ
Xcm3ygqYVLGwauNmuN3yGcQcnfypDOOeFs1lppbNe3lw0w3ikZN4Zmu1ec5s1ITK
MEcgBrGYgZrKDRXkx53lnABGpv6mYflYpag7fguDNL8j0lh9beaaNmHr4tmeEcug
VZ1b1EWoLMj/ikJ/vZcluiHPTSTheiAP8Ttvh1WAayq4rKwVtZygaI9IDauqqBQ1
Dgotes3vcomlTQXDUEZACyOZDxl7wbAUh0LZVaa2fYNIOoPNPOItUFSjf6YveF88
dLiw3ddVm+BFmczJzRpt
=4m2j
-END PGP SIGNATURE-
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] How to configure galaxy with a cluster

[Björn Grüning]

Hi Ben,

that is not possible at the moment. The idea is to keep the 
user-inferface as easy as possible for the user. You, as admin, can 
decide which resource a specific tool with a specific input will use. 
You will never see any options like that in a tool, but you can write a 
tool by yourself if you like, or "enhance" the megablast tool.


Cheers,
Bjoern


Am 16.07.2014 09:43, schrieb 王渭巍:

Thanks a lot, Thomas! It really helps, I added tools section followed your 
suggestion...

here is my job_conf.xml ( I am using Torque,  I have 3 servers. One for galaxy 
server, two for cluster computing.  )










walltime=72:00:00,nodes=1:ppn=8
128







and still no cluster options in "megablast" item.  How can I see cluster 
options in the page, for example, the page will let me choose to use local server or a 
cluster.

Ben



From: Thomas Bellembois
Date: 2014-07-15 17:41
To: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] How to configure galaxy with a cluster
Hello Ben,

you can configure your Galaxy instance to use your cluster in the
job_conf.xml file:

https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster

You can set up your instance to use your cluster by default for all jobs
or only for specific jobs.

Here is a part of my job_conf.xml for example:

 

 

 
 
 

 
 
 
 

 
 
 
   -r yes -b n -cwd -S /bin/bash
-V -pe galaxy 1
 
 
   -r yes -b n -cwd -S /bin/bash
-V -pe galaxy 12
 

 

 
 
 
 
 
 
 
 
 
 
 
 


Moreover you Galaxy user and Galaxy server must be allowed to submit
jobs to your scheduler.

Hope it  helps,

Thomas




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Clonable Galaxy installation

[Björn Grüning]

Hi Srinivas,

what is the Aim of that project, do you plan to also clone the data, or 
only the Galaxy installation. What should be keeped, all configs, also 
the configs special to your hardware?


I can't really tell you where full paths are stored. grepping and a 
database check will probably tell you.


Cheers,
Bjoern

Am 09.07.2014 00:56, schrieb Srinivas .:

Hi Bjoern,
Thank you for the response. Thanks also for your work on Docker'izing Galaxy. 
Using the Tool Shed to create Dockerfiles also appears to have good potential 
-- thanks for exploring that possibility, Greg.
The crux of my question is on whether it is possible to clone a customized 
Galaxy installation (whether using Docker containers or VM).
As an example, the "shed_tool_data_table_conf.xml" file contains absolute paths to the 
".loc" files containing reference genomes.  Those absolute paths (unless OK to convert them to 
relative paths or a different set of absolute paths) would prevent a clone of the reference installation from 
working properly since each cloned instance would require a unique "galaxy-dist" and supporting set 
of directories (within a common cluster). To illustrate:
1. "galaxy-dist" on the master instance is located within "/export/galaxy/master/". Installing BWA 
using the Tool Shed results in a "shed_tool_data_table_conf.xml" file that contains 
"/export/galaxy/master/galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/bwa_wrappers/b4427dbb6ced/bwa_index.loc"
2. "galaxy-dist" on a clone of the master instance is located within "/export/galaxy/clone-1/".  In order for 
this to work, "shed_tool_data_table_conf.xml" would have to reference the ".loc" files either using relative 
paths or updated absolute paths to reflect the proper location of the ".loc" files.  Can this be done without causing 
unknown / TBD side effect ?
I am looking for input on whether there might be other similar (surmountable / 
unsurmountable) considerations, including path dependencies within the 
database, that would make cloning a reference installation unviable.
Thanks.

Date: Mon, 7 Jul 2014 23:21:56 +0200
From: bjoern.gruen...@gmail.com
To: galaxy-dev@lists.bx.psu.edu; srinivas.gal...@outlook.com
Subject: Re: [galaxy-dev] Clonable Galaxy installation

Hi Srinivas,

I'm not sure I entirely understood your questions, its to late for me
today sorry. But maybe you would like to have a look at:

https://github.com/bgruening/docker-recipes

It's basically a docker base image (galaxy) on top you can create
several specialised images can be created (e.g. galaxy-deeptools). Every
child images is 'just' a set of toolshed tools.

Also have a look at https://trello.com/c/tm1fPvyq and try the new Test
ToolShed feature, which will create Dockerfiles for you!

Let us know how that works for you. It's still all work in progress and
can change or entirely disappear :)
Feedbacks, Pull-Requests welcome!

Cheers,
Bjoern

Am 07.07.2014 23:08, schrieb Srinivas .:

Hello,
I am writing to inquire if there are considerations which would prevent 
creation of multiple instances of Galaxy by cloning a reference Galaxy 
installation containing a pre-configured set of tools and related assets.  Both 
the reference and cloned instances would reside in the same cluster, and are 
each built within dedicated virtual machines . A Docker based approach has yet 
to be attempted but the same question likely applies.
One issue that was observed in attempting this was that the Tool Shed populates 
the “shed_tool_data_table_conf.xml” file with absolute paths to the “.loc” 
files for tools installed via the Tool Shed (even though all path related 
variables are specified using relative paths in the respective configuration 
files).  Might there be other similar dependencies or constraints, including 
those in the database, that would disallow this clonable Galaxy installation 
model from being viable ?
With regard to the Tool Shed updating files with absolute paths: Could the Toolshed be 
configured to use relative paths ? Else, would it be OK to update the "file 
path" variables within the instances created from the reference installation to 
reflect the instance-specific file paths ?
Thanks in advance.
--Srinivas




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 

Re: [galaxy-dev] removing wrapper version info from tools?

[Björn Grüning]

Hi Nik,

sorry for my late response.

Am 09.07.2014 01:23, schrieb Nikhil Joshi:

Hi Bjorn,

So I actually don't use the toolshed at all, we have a galaxy AMI that I
update and maintain. We have many of the canonical tools, but we also have
many custom tool interfaces on there, some of which I wrote. We also use
the latest version of the tool if possible. All of the XML files are local
to the instance and are not from the toolshed. I am referring to the
version that is used in the "" tag. That version number seems
arbitrary and the students that use our Galaxy are constantly confused as
to why the tool page has a different version than the tool itself.However,
it seems like there is no easy way to remove that part, so I think I will
just have to update the version numbers myself.


Please remember you are then forced to maintain the tools all by your 
own and integrate changes from upstream by your own. I'm not sure that 
is worth the effort.



What would be the best way
to have multiple versions of a tool *without* using the toolshed? Any help
is highly appreciated. Thanks!


I don't think that is possible at all. In the end that is one reason the 
Tool Shed project was started. To make it possible to have many 
different tool versions and to spread the maintenance burden over the 
entire community.

Have a look at the galaxytools repository, maintained by several people:
https://github.com/bgruening/galaxytools

We accept PR and you are very welcome to work with us to improve the 
tools, in the end I really think it will save you some time.


An other really easy possibility is ... just remove the tool-version 
from the HTML rendering. If you are not afraid of maintaining in house 
patches, that should be fine and will take time, I guess.


Cheers,
Bjoern


- Nik.


On Tue, Jul 8, 2014 at 1:12 AM, Björn Grüning 
wrote:


Hi Nik,

  Hi Bjorn,


Thanks for your response. It seems to me that the wrapper version is not
really that useful... the thing that is important is the version of the
underlying tool.



Sure, I agree. But on the other side the user should not care about the
version, and is on most cases not interested as long as it works :)
But as pointed out I would also like to shift to a more useful numbering,
that the arbitrary one we are using now.


I can certainly update the versions to be the versions of


the tool, if need be.



That will break many thing, if you combining your tools with the toolshed
and updates would be more complicated. Don't do that, at least not for
tools from the ToolShed. Try to convince the Tool Maintainers.


However, I wasn't aware that you could have multiple


versions of the same tool to choose from. How do I get that to work?



You can simply install multiple versions (revisions) from the same tool
from the Tool Shed. For example multimple Blast version.


  I tried creating two different XML files with the same tool id, but that

didn't work. And how would I make sure the wrapper scripts pointed to the
correct version... it seems like I would have to tailor each script with
an
absolute path to the specific version?



No you need to have the Tool Shed managing that for you. Each version is a
separate tool with the same tool-id. Galaxy keeps track of every dependency
of a certain tool. So the same tool with different versions with different
dependencies will be working over time.

Please read more about all of that in the galaxy wiki, especially about
Tools and the Tool Shed:

https://wiki.galaxyproject.org/ToolShed

Cheers,
Bjoern


  - Nik.



On Mon, Jul 7, 2014 at 11:56 PM, Björn Grüning 


wrote:

  Hi Nik,


you can hack to tool rendering and omit the version string, but I would
not recommend that. The version will be a selectbox as soon as you have
multiple versions from the same tools installed. This is important for
reproducibility.
The best approach would be to fix the version string. Do you have any
suggestions?
For my wrappers I try to go with that version scheme:
{Tool-Version}.{wrapper-version}.

Cheers,
Bjoern

Am 08.07.2014 08:51, schrieb Nikhil Joshi:

  Hi all,


So I am trying to remove the version info (at the top of a tool page)
entirely for all the tools. As I understand, those versions are the
versions of the wrappers and not the tools themselves. Although I could
change that, I would rather just not have the versions on the tool pages
at
all. We currently show all the versions of our tools on our landing
page.
Is there any way to remove the version part of the tool title entirely?
I.e., if it originally says "Map with BWA for Illumina (version
1.0.0)", I
just want it to say "Map with BWA for Illumina".

- Nik.



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists

Re: [galaxy-dev] removing wrapper version info from tools?

[Björn Grüning]

Hi Nik,

Hi Bjorn,

Thanks for your response. It seems to me that the wrapper version is not
really that useful... the thing that is important is the version of the
underlying tool.


Sure, I agree. But on the other side the user should not care about the 
version, and is on most cases not interested as long as it works :)
But as pointed out I would also like to shift to a more useful 
numbering, that the arbitrary one we are using now.


I can certainly update the versions to be the versions of

the tool, if need be.


That will break many thing, if you combining your tools with the 
toolshed and updates would be more complicated. Don't do that, at least 
not for tools from the ToolShed. Try to convince the Tool Maintainers.


However, I wasn't aware that you could have multiple

versions of the same tool to choose from. How do I get that to work?


You can simply install multiple versions (revisions) from the same tool 
from the Tool Shed. For example multimple Blast version.



I tried creating two different XML files with the same tool id, but that
didn't work. And how would I make sure the wrapper scripts pointed to the
correct version... it seems like I would have to tailor each script with an
absolute path to the specific version?


No you need to have the Tool Shed managing that for you. Each version is 
a separate tool with the same tool-id. Galaxy keeps track of every 
dependency of a certain tool. So the same tool with different versions 
with different dependencies will be working over time.


Please read more about all of that in the galaxy wiki, especially about 
Tools and the Tool Shed:


https://wiki.galaxyproject.org/ToolShed

Cheers,
Bjoern


- Nik.


On Mon, Jul 7, 2014 at 11:56 PM, Björn Grüning 
wrote:


Hi Nik,

you can hack to tool rendering and omit the version string, but I would
not recommend that. The version will be a selectbox as soon as you have
multiple versions from the same tools installed. This is important for
reproducibility.
The best approach would be to fix the version string. Do you have any
suggestions?
For my wrappers I try to go with that version scheme:
{Tool-Version}.{wrapper-version}.

Cheers,
Bjoern

Am 08.07.2014 08:51, schrieb Nikhil Joshi:


Hi all,

So I am trying to remove the version info (at the top of a tool page)
entirely for all the tools. As I understand, those versions are the
versions of the wrappers and not the tools themselves. Although I could
change that, I would rather just not have the versions on the tool pages
at
all. We currently show all the versions of our tools on our landing page.
Is there any way to remove the version part of the tool title entirely?
I.e., if it originally says "Map with BWA for Illumina (version 1.0.0)", I
just want it to say "Map with BWA for Illumina".

- Nik.



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/






___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] removing wrapper version info from tools?

[Björn Grüning]

Hi Nik,

you can hack to tool rendering and omit the version string, but I would 
not recommend that. The version will be a selectbox as soon as you have 
multiple versions from the same tools installed. This is important for 
reproducibility.
The best approach would be to fix the version string. Do you have any 
suggestions?
For my wrappers I try to go with that version scheme: 
{Tool-Version}.{wrapper-version}.


Cheers,
Bjoern

Am 08.07.2014 08:51, schrieb Nikhil Joshi:

Hi all,

So I am trying to remove the version info (at the top of a tool page)
entirely for all the tools. As I understand, those versions are the
versions of the wrappers and not the tools themselves. Although I could
change that, I would rather just not have the versions on the tool pages at
all. We currently show all the versions of our tools on our landing page.
Is there any way to remove the version part of the tool title entirely?
I.e., if it originally says "Map with BWA for Illumina (version 1.0.0)", I
just want it to say "Map with BWA for Illumina".

- Nik.



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Clonable Galaxy installation

[Björn Grüning]

Hi Srinivas,

I'm not sure I entirely understood your questions, its to late for me 
today sorry. But maybe you would like to have a look at:


https://github.com/bgruening/docker-recipes

It's basically a docker base image (galaxy) on top you can create 
several specialised images can be created (e.g. galaxy-deeptools). Every 
child images is 'just' a set of toolshed tools.


Also have a look at https://trello.com/c/tm1fPvyq and try the new Test 
ToolShed feature, which will create Dockerfiles for you!


Let us know how that works for you. It's still all work in progress and 
can change or entirely disappear :)

Feedbacks, Pull-Requests welcome!

Cheers,
Bjoern

Am 07.07.2014 23:08, schrieb Srinivas .:

Hello,
I am writing to inquire if there are considerations which would prevent 
creation of multiple instances of Galaxy by cloning a reference Galaxy 
installation containing a pre-configured set of tools and related assets.  Both 
the reference and cloned instances would reside in the same cluster, and are 
each built within dedicated virtual machines . A Docker based approach has yet 
to be attempted but the same question likely applies.
One issue that was observed in attempting this was that the Tool Shed populates 
the “shed_tool_data_table_conf.xml” file with absolute paths to the “.loc” 
files for tools installed via the Tool Shed (even though all path related 
variables are specified using relative paths in the respective configuration 
files).  Might there be other similar dependencies or constraints, including 
those in the database, that would disallow this clonable Galaxy installation 
model from being viable ?
With regard to the Tool Shed updating files with absolute paths: Could the Toolshed be 
configured to use relative paths ? Else, would it be OK to update the "file 
path" variables within the instances created from the reference installation to 
reflect the instance-specific file paths ?
Thanks in advance.
--Srinivas




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Proposed cd-hit tool dependency change in galaxy

[Björn Grüning]

Hi Qi,

thank you very much for you contribution.
I will try to get your patch to the correct person during the upcoming 
Galaxy Conference.


Cheers,
Bjoern


Am 13.06.2014 17:29, schrieb Qi, Chuyang:

Hi all,

When installing and testing the cd-hit tool from jjohnson from the main galaxy 
toolshed into my local instance of galaxy, I encountered a problem where galaxy 
cannot find the cd-hit program.

The error was:
"/bin/sh: 1: cd-hit-est: not found"

As I debugged the problem, I found that there is an bug in the tool dependency. 
It installs the cd-hit programs but never moves the programs into the 
configurable galaxy install directory. This could cause many problems in the 
situation where users don't have cd-hit already installed on their computer and 
must depend on the tool dependency to run the tool.

I propose we add the following code to the tool dependency:


cd-hit
$INSTALL_DIR
  


cd-hit-est
$INSTALL_DIR
  

After I made these changes, I pushed the changes into a new repo on my local 
toolshed, I managed to successfully install cd-hit into a local and dev-test 
instance of galaxy and the cd-hit tool ran successfully.

Thanks for your understanding,
Charles Qi





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] setup_venv doesn't honour set_environment_for_install / installing lxml

[Björn Grüning]

Hi Janis,

that is not working and it is currently not clear if we will change it.
Please see the following Trello Card:

https://trello.com/c/NsLJv9la/61-clean-up-tool-shed-setup-actions

One of my project during the upcoming GCC hackathon is to implement a 
setup_python_environment, like the R, perl or ruby ones. With that in 
place your use case will be easy to implement. If you get your python 
package up and running at that time, this would be great!


Cheers,
Bjoern

Am 24.06.2014 18:57, schrieb Jan Kanis:

Ok, here is the tool_dependencies.xml. This

(also attached, if that survives the list) is what I would like my
tool_dependencies.xml to look like, but it doesn't work because the
installation of lxml fails, as it can't find the libxml2 headers. I am now
using this

as a workaround. It works because lxml is installed from a shell_command
which does include the variables set from the set_environment_for_install
block. This is from the blast2html tool.

Jan


On 24 June 2014 15:30, Dave Bouvier  wrote:


Jan,

In order to help track down this issue, could you provide the
tool_dependencies.xml you're using?

   --Dave B.


On Tue 24 Jun 2014 05:40:30 AM EDT, Jan Kanis wrote:


In a tool_dependency.xml file I want to install python package lxml in
a virtual environment, as a tool I'm building needs it. The python
lxml package requires the libxml2 tool dependency. I have added a
set_environment_for_install action that refers to the libxml2
repository, but when python/pip tries to install lxml it fails,
apparently because it can't find the required headers. This appears to
be because the setup_virtualenv action does not include install
environment variables.

It seems to me that install environment variables should be sourced
for every following action that can do nontrivial things, not just
shell commands.

Alternatively, am I trying to install lxml the wrong way, is there a
better way? (I'm running on python 2.6)

Jan


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/







___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Announcement: Galaxy Docker builds

[Björn Grüning]

Hi,

news from the Galaxy Docker project!

With the recent addition of tags in trusted builds it's now possible to 
create a tagged Docker Images for every stable Galaxy release.


I will try to maintain that and will release every two month, in sync 
with the Galaxy release, a new Docker image.


Feedback welcome!

https://registry.hub.docker.com/u/bgruening/galaxy-stable/tags/manage/

Cheers,
Bjoern

Am 30.04.2014 00:43, schrieb Björn Grüning:

Dear Galaxy Devs and Users,

I'm pleased to announce the Galaxy Docker Image project!

The Galaxy Docker Image is an easy distributable full-fledged Galaxy
installation, that can be used for testing, teaching and presenting new
tools and features.

One of the main goals is to make the access to entire tool suites as
easy as possible. Usually, this includes the setup of a public available
webservice that needs to be maintained, or that the Tool-user needs to
either setup a Galaxy Server by its own or to have Admin access to a
local Galaxy server. With docker, tool developers can create their own
Image with all dependencies and the user only needs to run it within
docker. Ideally suited to make your reviewers happy :)

Currently, I have three test images to play with. The first one,
bgruening/galaxy-stable is the main Image that will do all the heavy
lifting of setting up PostgreSQL, Apache and Galaxy.

bgruening/deepTools and bgruening/chemicaltoolbox are proof of concepts
how easy it is to create an own Image build on top of the main Galaxy
Image.

https://index.docker.io/u/bgruening/galaxy-deeptools/
https://index.docker.io/u/bgruening/galaxy-chemicaltoolbox/

How to use it:
docker run -d -p 8080:80 bgruening/galaxy-deeptools
docker run -d -p 8080:80 -p 8000:8000 bgruening/galaxy-chemicaltoolbox

For a more detailed description please read:
https://github.com/bgruening/docker-recipes/tree/master/galaxy

I would be very much interested in any kind of feedback.
Especially, in two technical questions:

I'm not sure if a "trusted build" from the docker index is the best way
to go. Trusted Builds in docker are super convenient. It is a
post-commit hook from github. Every time something is pushed, the image
will be rebuild. The problem is that I can't tag my images with "trusted
builds" and unless I have missed something, that forces me to create new
repositories every time a new Galaxy release is out.
A better way would be, one repository with different tags:
galaxy-stable, galaxy-feb-2014 and so on. Every Galaxy release will be
just a diff between the old and the new. But this is not possible with
"trusted builds".

One other trick I used is to remove the mercurial history, that saves
180MB. I'm not sure if this is wise, because it probably kills a few use
cases. Currently, the aim of this project is a little bit different from
being a developer environment. So I decided to remove it. Please ping me
if this was a bad decision :)

Thanks for reading!
Bjoern

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Object-Store, setting filetypes crashes Galaxy

[Björn Grüning]

files_path
 return self.object_store.get_filename( self, dir_only=True,
extra_dir=self._extra_files_path or "dataset_%d_files" % self.id )
   File "/usr/local/galaxy/galaxy-dist/lib/galaxy/objectstore/__init__.py",
line 416, in get_filename
 return self.__call_method('get_filename', obj, ObjectNotFound, True,
**kwargs)
   File "/usr/local/galaxy/galaxy-dist/lib/galaxy/objectstore/__init__.py",
line 436, in __call_method
 % ( method, str( obj ), str( kwargs ) ) )
ObjectNotFound: objectstore, __call_method failed: get_filename on
, kwargs: {'dir_only': True,
'extra_dir': 'dataset_118453_files'}

It should be reproducible with the latest Galaxy stable release and the
latest NCBI Blast+ wrappers if you are trying to generate a Blast+ database.

Thanks,
Bjoern



2014-06-12 16:50 GMT+02:00 Björn Grüning :


Hi Nate,

yes that is fixing it for me!

Thanks!
Bjoern

Am 12.06.2014 16:26, schrieb Nate Coraor:


writeable (e.g. what you have new_files_path set to), that'd confirm it.




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

[galaxy-dev] BEDtools wrapper development - help needed

[Björn Grüning]

Hi,

if you are interested in wrapper development you can help to improve the 
current BEDtools in Galaxy. We have wrapped nearly all available 
BEDtools from version 2.19.


A test repository is located here:
 https://testtoolshed.g2.bx.psu.edu/view/bgruening/bedtools_test_bag

and the development github repository is here:
 https://github.com/bgruening/galaxytools/tree/master/bedtools

If you would like to learn how to write wrappers, improve the 
documentation or the unit-test, if you are a BEDtools expert or just 
want to speed up the upload to the Tool Shed ... please drop me a mail 
or clone the repository and go crazy.


Thanks,
Bjoern
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] What is the correct place under Galaxy for a database that's created by a tool?

[Björn Grüning]

Hi,

Am 17.06.2014 21:25, schrieb Melissa Cline:

Good thing to clarify: this is a tool-specific database, created by tools
that are running inside Galaxy but that should persist after the individual
tools are done with their execution.


Should it persist as output data or forever even if I start my workflow 
from scratch?


What will happen to the db if I reload a tool and rerun it. Will it 
extend the database or will it use a new one from scratch?


You have access to the user name if you are running the tool. With that 
you can create a table for every user and store your data user-specific.


Cheers,
Bjoern



On Mon, Jun 16, 2014 at 11:23 PM, Björn Grüning 
wrote:


Hi Melissa,

Am 17.06.2014 01:07, schrieb Melissa Cline:

  Hi folks,


Hopefully this is a quick question.  I'm working on a set of tools that
will fire off a VM from within Galaxy and will then communicate with the
VM.  The VM will create a local database.



Are we talking here about a local Galaxy database or a tool specific
database?

best,
Bjoern


  The vision is that this won't be


a shared database; in a shared Galaxy instance, each user will have his or
her own database.  What is the best place to create this database under
the
Galaxy file system?

Thanks!

Melissa



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/





___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] R bioconductor dependencies when creating toolshed installation

[Björn Grüning]

Hi Stef,

Am 17.06.2014 15:40, schrieb Stef van Lieshout:

Hi Bjoern,

That looks much better indeed ;) The only problem I still have then is
that I need R 3.1.0 for a bioconductor 2.14 package (have send a new
mailing list msg for that). Looking at the xml of other versions it's
not something I will easily do myself.


If you can wait a little bit we (the IUC, or more concrete Dave Bouvier) 
will take care of that and create such a repository.



What will happen if I do not specify the R dependency ("package_r_3_0_3"
in your example code) but do specify the download/install of packages,
guess these get installed in the "default" R instance?


Puh, to be honest, I do not know. I never tested it without a real 
instance. I guess it will pick the default version.



Related to that, how can I call a specific instance of R in de tool.xml
without specifying the full path to the tool. Eg, in the tool.xml I now
do:


   /path/to/lib/R/R-3.1.0/bin/Rscript
   /path/to/galaxy-dist/tools/testdir/tool.R $config


Where normally you can do:


   tool.R $config



You should always use the latter version, without a path to R. Setting 
the correct path or assuming the default should be handled by Galaxy. 
The correct R version will be created with the  tag. You 
can specify 3.1 as soon as we have it :)


You can thank Dave for the new R packages, he spend much time in 
creating a big R binary that can run on almost all architectures.


Cheers,
Bjoern


Thanks again!
Stef


- Original message -
From: Björn Grüning 
To: Stef van Lieshout ,
galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] R bioconductor dependencies when creating
toolshed installation
Date: Tue, 17 Jun 2014 15:17:36 +0200

Hi Stef,

for R packages we have a special installation routine that will
(hapefully) make your life easier.


I'm running into some difficulties on how to setup the installation
procedure for a galaxy tool which executes an R script and has certain
dependencies (mainly bioconductor packages). R can deal with
dependencies, packages can be installed with install.packages (has a
"dependencies" argument) or biocLite() for bioconductor packages.

Yet, now I want my tool to be available at toolsheds. To do this I see
several options:


Great!


1) setting up tool_dependencies.xml with "R CMD INSTALL" for all
packages. BUT: need to download all dependencies before install, and can
older versions still be downloaded? Maybe need to upload them to
toolshed too..


It is all a matter of how reproducible you want to have your tool.
If you want 100% reproducibility, you need to mirror the source packages
somehow, because bioc will not store older versions. At least that is
not guaranteed.

I'm using a special github repository for that purpose:
https://github.com/bgruening/download_store

R CMD INSTALL is not needed, see below.


2) setting up tool_dependencies.xml to call an installation script with
Rscript (where I could use install.packages), BUT: Dependencies are
taken care of. But how do I select specific (older) versions, because if
I dont, installing at different time can give different version.


Older versions is not possible as far as I know.


3) creating a repository for each package and have all of them as
requirement in my galaxy tool. BUT: a lot of work for a lot of
dependencies


Imho, we should have one R repository with a handful of standard
packages included in the toolshed.
Like packages_r_3_0_1. You should depend on that repository and
additionally define one second dependency. Lets say your tool is called
deseq2 than create one additional tool_dependencies.xml file called
package_deseq2_1_2_10. In that definition you will install every
dependency you need in addition to R.

Here is one example:
https://github.com/bgruening/galaxytools/blob/master/deseq2/tool_dependencies.xml
https://github.com/bgruening/galaxytools/blob/master/orphan_tool_dependencies/package_deseq2_1_2_10/tool_dependencies.xml

The really nice part is the "setup_r_environment" function from the
toolshed. It will install source packages for you automatically. All you
need to do is to name the package or, as shown in the example, specify
the location of the source package.

The only downside is that the order of these packages is important. If
you are interested we have a script that will give you the correct
dependency tree of a given package.

Hope that helps,
Bjoern



All have pros and cons, how do people deal with this?

Stef
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/



___
Please keep all re

Re: [galaxy-dev] R bioconductor dependencies when creating toolshed installation

[Björn Grüning]

Hi Stef,

fortunately it is way easier than that. Please have a look at the 
"setup_r_environment" installation routine :)


Cheers,
Bjoern

Am 17.06.2014 11:07, schrieb Stef van Lieshout:

Say option 3 is the way to go, would you say every new version of an R
package should be wrapped in a new galaxy package (and give them names
like "matrixStats_0_10_0") or create one package ("matrixStats") and
update that one if a new version is worth an update. In the first way
there would be an enormous amount of packages ;)

Also if you do need an external R script as you say, how would I
construct my tool_dependencies.xml to execute R code?

And last, if that approach doesn't work out for me, how can copy a file
in the repository to the installation dir? (to execute it with Rscript)

Many thanks,
Stef


- Original message -
From: "Kandalaft, Iyad" 
To: Stef van Lieshout ,
"galaxy-dev@lists.bx.psu.edu" 
Subject: RE: [galaxy-dev] R bioconductor dependencies when creating
toolshed installation
Date: Mon, 16 Jun 2014 18:19:46 +

I would typically recommend Option 3 as it is the best practice.
However, human resources limit this as a viable option even though this
should be the "Gold Standard" that you aim for.  This allows you to
reuse the dependencies later for other tool wrappers AND you don't have
to re-install dependencies every time you make a modification to your
tool wrapper repository.  While briefly looking at Bioconductor, it
seems that they keep old version of packages (ex:
http://www.bioconductor.org/packages/2.13/data/experiment/bin/windows/contrib/3.0/AmpAffyExample_1.2.13.zip),
where using the URLs directly might be advantageous if their BiocLite
doesn't allow you to define which version to install.  You don't
necessarily need to have an external R script for the installation
because many of these commands can be done within the
tool_dependencies.xml.

Regards,


Iyad Kandalaft
Microbial Biodiversity Bioinformatics
Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada
960 Carling Ave.| 960 Ave. Carling
Ottawa, ON| Ottawa (ON) K1A 0C6
E-mail Address / Adresse courriel  iyad.kandal...@agr.gc.ca
Telephone | Téléphone 613-759-1228
Facsimile | Télécopieur 613-759-1701
Teletypewriter | Téléimprimeur 613-773-2600
Government of Canada | Gouvernement du Canada



-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Stef van
Lieshout
Sent: Monday, June 16, 2014 10:04 AM
To: galaxy-dev@lists.bx.psu.edu
Subject: [galaxy-dev] R bioconductor dependencies when creating toolshed
installation

Hi all,

I'm running into some difficulties on how to setup the installation
procedure for a galaxy tool which executes an R script and has certain
dependencies (mainly bioconductor packages). R can deal with
dependencies, packages can be installed with install.packages (has a
"dependencies" argument) or biocLite() for bioconductor packages.

Yet, now I want my tool to be available at toolsheds. To do this I see
several options:

1) setting up tool_dependencies.xml with "R CMD INSTALL" for all
packages. BUT: need to download all dependencies before install, and can
older versions still be downloaded? Maybe need to upload them to
toolshed too..

2) setting up tool_dependencies.xml to call an installation script with
Rscript (where I could use install.packages), BUT: Dependencies are
taken care of. But how do I select specific (older) versions, because if
I dont, installing at different time can give different version.

3) creating a repository for each package and have all of them as
requirement in my galaxy tool. BUT: a lot of work for a lot of
dependencies

All have pros and cons, how do people deal with this?

Stef
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this and other
Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] R bioconductor dependencies when creating toolshed installation

[Björn Grüning]

Hi Stef,

for R packages we have a special installation routine that will 
(hapefully) make your life easier.



I'm running into some difficulties on how to setup the installation
procedure for a galaxy tool which executes an R script and has certain
dependencies (mainly bioconductor packages). R can deal with
dependencies, packages can be installed with install.packages (has a
"dependencies" argument) or biocLite() for bioconductor packages.

Yet, now I want my tool to be available at toolsheds. To do this I see
several options:


Great!


1) setting up tool_dependencies.xml with "R CMD INSTALL" for all
packages. BUT: need to download all dependencies before install, and can
older versions still be downloaded? Maybe need to upload them to
toolshed too..


It is all a matter of how reproducible you want to have your tool.
If you want 100% reproducibility, you need to mirror the source packages 
somehow, because bioc will not store older versions. At least that is 
not guaranteed.


I'm using a special github repository for that purpose:
https://github.com/bgruening/download_store

R CMD INSTALL is not needed, see below.


2) setting up tool_dependencies.xml to call an installation script with
Rscript (where I could use install.packages), BUT: Dependencies are
taken care of. But how do I select specific (older) versions, because if
I dont, installing at different time can give different version.


Older versions is not possible as far as I know.


3) creating a repository for each package and have all of them as
requirement in my galaxy tool. BUT: a lot of work for a lot of
dependencies


Imho, we should have one R repository with a handful of standard 
packages included in the toolshed.
Like packages_r_3_0_1. You should depend on that repository and 
additionally define one second dependency. Lets say your tool is called 
deseq2 than create one additional tool_dependencies.xml file called 
package_deseq2_1_2_10. In that definition you will install every 
dependency you need in addition to R.


Here is one example:
https://github.com/bgruening/galaxytools/blob/master/deseq2/tool_dependencies.xml
https://github.com/bgruening/galaxytools/blob/master/orphan_tool_dependencies/package_deseq2_1_2_10/tool_dependencies.xml

The really nice part is the "setup_r_environment" function from the 
toolshed. It will install source packages for you automatically. All you 
need to do is to name the package or, as shown in the example, specify 
the location of the source package.


The only downside is that the order of these packages is important. If 
you are interested we have a script that will give you the correct 
dependency tree of a given package.


Hope that helps,
Bjoern



All have pros and cons, how do people deal with this?

Stef
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] What is the correct place under Galaxy for a database that's created by a tool?

[Björn Grüning]

Hi Melissa,

Am 17.06.2014 01:07, schrieb Melissa Cline:

Hi folks,

Hopefully this is a quick question.  I'm working on a set of tools that
will fire off a VM from within Galaxy and will then communicate with the
VM.  The VM will create a local database.


Are we talking here about a local Galaxy database or a tool specific 
database?


best,
Bjoern

 The vision is that this won't be

a shared database; in a shared Galaxy instance, each user will have his or
her own database.  What is the best place to create this database under the
Galaxy file system?

Thanks!

Melissa



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Object-Store, setting filetypes crashes Galaxy

[Björn Grüning]

Hi Nate,

yes that is fixing it for me!

Thanks!
Bjoern

Am 12.06.2014 16:26, schrieb Nate Coraor:

writeable (e.g. what you have new_files_path set to), that'd confirm it.

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] "Resume Paused Jobs"

[Björn Grüning]

No problem John, thanks for fixing it so fast!

Am 07.06.2014 19:25, schrieb John Chilton:

I can in fact confirm I broke that. It needs to be conditionally
turned off in some cases now and I had the logic for when it is
display and when not appears to be inverted from what it should be
somehow.

Following patch fixes it for me:

diff --git a/static/scripts/galaxy.tools.js b/static/scripts/galaxy.tools.js
index 91b0597..2fa9ade 100644
--- a/static/scripts/galaxy.tools.js
+++ b/static/scripts/galaxy.tools.js
@@ -110,9 +110,9 @@ define([ "libs/underscore", "mvc/tools" ],
function( _, Tools ) {
  enableSelectBy: function( enableIndex, onValue ) {
  var selectionType = SELECTION_TYPE[onValue];
  if(selectionType["allow_remap"]) {
-$("div#remap-row").css("display", "none");
-} else {
  $("div#remap-row").css("display", "inherit");
+} else {
+$("div#remap-row").css("display", "none");
  }
  this.formRow().find( "i" ).each(function(index, iElement) {
  if(index == enableIndex) {

Will open a stable branch bug fix pull request shortly. Very sorry about that.

-John


On Sat, Jun 7, 2014 at 12:04 PM, Björn Grüning
 wrote:

Hi,

can anyone confirm that "Resume Paused Jobs" on crashed tools is broken in
the latest Galaxy release?

Thanks,
Bjoern
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

[galaxy-dev] "Resume Paused Jobs"

[Björn Grüning]

Hi,

can anyone confirm that "Resume Paused Jobs" on crashed tools is broken 
in the latest Galaxy release?


Thanks,
Bjoern
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] to put or not to put a tool suite into the toolshed

[Björn Grüning]

Hi Wolfgang,


I am maintaining a (still young) suite of command line tools (written in
Python) for identification of mutations in model organism genomes
through whole-genome sequencing (https://sourceforge.net/projects/mimodd/).
MiModD aims at geneticists that do not have much background in
bioinformatics, and it's supposed to make WGS analysis for small model
organism genomes (anything from yeast to fish) possible on regular PCs,
so it's not a cloud/cluster solution.

We do support Galaxy though through a complete set of tool wrappers for
three reasons:
- to provide a graphical user interface
- to offer labs the possibility to install the software on one dedicated
machine, but run analyses from any machine (typically with Windows
installed)
- to keep analysis workflows documented and reproducible.

Currently, we advise users to take advantage of these features and
install a local instance of Galaxy even though it will be running then
only on a single desktop PC (or even just a notebook). After
installation of Galaxy our software simply copies its wrapper xmls over
to the tools folder and modifies the tool_conf file to integrate itself.
In addition, users have to install a bit of other third-party software
(samtools, snap aligner, optionally snpeff) that our code relies on.

[End of lengthy introduction]

So my question is: in what way could the package profit from being
uploaded to a Galaxy toolshed ? I guess it would mean quite some extra
work from my side since I'm not familiar with the whole procedure, so
are there benefits (visibility, ease of installation, etc.) that are
worth the effort ?


Yes, it would mean extra work for you, but it is worth the effort.

One main reason, you already mentioned, visibility. If done right and 
you are supporting your wrapper these will probably be used by many 
different Galaxy Servers out there. Just have a look at a few tools that 
are from the same field as your tools and look at the download counter.


Easy installation is also one reason, as soon as you have a Galaxy 
server it can be trivial to install your tools, suppose you define all 
dependencies. -> But defining dependencies is easy nowadays, and even 
better, many dependencies are already available in the Tool Shed. If you 
depend on numpy, insert a few lines to point to the correct numpy 
dependency and you are done.
We can help you if you encounter any problems in that step, or just drop 
by during the GCC developer workshop. We will give a hands-on how to 
wrap your tools.


Reproducibility: With the Tool Shed you get reproducibility for free. 
You can install many different version of your tools and Galaxy will 
take care to select the correct one to reproduce your results.


You can also share your workflows in the Tool Shed. Convince others from 
your tools, export your workflows and and anyone can test it with a few 
clicks.


Be part of a very nice community, which tries hard to make bioinformatic 
(and more) tools accessible for everyone.


Convinced? :)
Cheers,
Bjoern



Thanks a real lot for any feedback,
Wolfgang

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Manual Tool and dependency installation

[Björn Grüning]

Hi Ravi,

in a nutshell try to install all tools locally on your computer (same 
architecture than your cluster). Copy over all tools and directories. 
Now go to every tool-directory and create a symlink from the folder 
where the env.sh file is located to a folder called default. For example:


/usr/local/tools/samtools/0.0.19/iuc/package_asdf/34213452345/ (here is 
your env.sh file) to /usr/local/tools/samtools/default/


That should work, but your are loosing reproducibility and can't use 
different versions of the same tool.
What about running Galaxy in a docker container 
(https://github.com/bgruening/docker-recipes/tree/master/galaxy) and 
hosting everything there, isolated from your webserver?


Hope that helps a little bit,
Bjoern

Am 14.05.2014 18:43, schrieb Ravi Alla:

Hi All,
Due to the nature of our cluster we are not allowed to have compilers on our 
webserver, which is where our galaxy instance runs. Because of this tool 
dependencies and tools that need to be compiled fail to install through the 
galaxy tool shed webpage. Is there a way for me to install dependencies and 
tools manually? For example I am trying to install the GATK2 wrapper. This 
depends on samtools-0.1.19 from iuc which in turn requires ncurses-5.9. Is 
there instructions somewhere as to where to install these tools such that they 
are tied to the GATK2 wrapper? Pardon me if this is a naive question, but I am 
really out of sorts here.
Thank you
Ravi
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] galaxy is not sourcing tool environment

[Björn Grüning]

Hi Isabelle,

can you show us the "requirements" section from your tool?
Also you need to have a symbolic link from 
tool_dependency_dir/hhsuite/2.0.16/ ponting to 
tool_dependency_dir/hhsuite/default/


In tool_dependency_dir/hhsuite/default/ should be the env.sh file. If 
you have the correct "requirements" section in your tool it should work.


Good luck!
Bjoern


Am 04.06.2014 01:17, schrieb Isabelle Phan:

I am using env.sh to manage my tool dependencies:

# tool_dependency_dir/hhsuite/2.0.16/env.sh
export HHPRED_DIR=/opt/galaxy-dist/tool_dependencies/hhsuite/default
export HHLIB=${HHPRED_DIR}/lib/hh
export PATH=${HHPRED_DIR}/bin:${HHLIB}/scripts:$PATH


Galaxy is not sourcing it. I have to manually source it before I restart
Galaxy.

I am following "Using the env.sh file" and cannot get it to work. What am
I missing?

Any help greatly appreciated,

thanks,

Isabelle




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy customized tool config file

[Björn Grüning]

Hi Vipin,

that is currently not possible but a high priority of the Galaxy 
deavteam. You can follow the status in several trello cards:


https://trello.com/c/VizlCET9
https://trello.com/c/phMelslw

Ciao,
Bjoern

Am 04.06.2014 18:33, schrieb Vipin TS:

Sorry for re-posting! any suggestions to make an interactive tool page.

Thanks you, Vipin


On Tue, May 20, 2014 at 5:25 PM, Vipin TS  wrote:


Hello dev-team,

With the current state, is it possible to write an interactive page for a
tool in Galaxy?

In my case, I m considering to build a tool page with JavaScript embed in
XML file, my expectation is to behave the Galaxy tool page as google
translate page with option instant translation on (display the same text
content in the next box).

I have checked the Admin/Tools/ToolConfigSyntax, not sure whether I missed
anything related this. any comments ?

thanks, Vipin
galaxy.cbio.mskcc.org






___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] devteam packages

[Björn Grüning]

Hi Jan,

they are not available yet, but here is a trello card about that problem:

https://trello.com/c/VZdI9MI8

Best,
Bjoern

Am 04.06.2014 15:59, schrieb Jan Kanis:

Hi all,

Where are the toolshed packages owned by devteam maintained? The galaxy
sources don't seem to include those tool shed packages and I was unable to
find a related repository on github or a similar place.

Jan



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy reports crashing with the latest stable release

[Björn Grüning]

Hi Dannon,

after removing the *pyc files and killing manually the daemon 
(--stop-daemon did not work) fixes the problem for me.


Sorry for the noise!
Bjoern

Am 04.06.2014 13:52, schrieb Dannon Baker:

I'm having trouble reproducing this. The base grid object that's used by
the SpecifiedDateListGrid does have that info_text field -- it's almost as
if you're somehow using an outdated grid module, while the rendering
template is current.

It's a bit brute force, but can you try (from the base of your galaxy
directory) something like:

find . -name '*.pyc' -delete

and restart your reports app?


On Wed, Jun 4, 2014 at 5:10 AM, bjoern.gruen...@googlemail.com <
bjoern.gruen...@gmail.com> wrote:


Hi,

I encountered the following bug during my testing of the latest stable
release:

Error - : 'SpecifiedDateListGrid' object
has no attribute 'info_text'
URL:
http://galaxy.uni-freiburg.de:9001/jobs/specified_date_handler?specified_date=2014-06-04
File
'/usr/local/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/exceptions/errormiddleware.py',
line 144 in __call__
   app_iter = self.application(environ, sr_checker)
File
'/usr/local/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
line 84 in __call__
   return self.application(environ, start_response)
File
'/usr/local/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
line 633 in __call__
   return self.application(environ, start_response)
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py',
line 132 in __call__
   return self.handle_request( environ, start_response )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py',
line 190 in handle_request
   body = method( trans, **kwargs )
File
'/usr/local/galaxy/galaxy-dist/lib/galaxy/webapps/reports/controllers/jobs.py',
line 177 in specified_date_handler
   return self.specified_date_list_grid( trans, **kwd )
File
'/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/helpers/grids.py',
line 296 in __call__
   message = message,
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py',
line 1184 in fill_template
   def fill_template(self, filename, **kwargs):
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/__init__.py',
line 1199 in fill_template_mako
   template_lookup = template_lookup or self.webapp.mako_template_lookup
File
'/usr/local/galaxy/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg/mako/template.py',
line 296 in render
   return runtime._render(self, self.callable_, args, data)
File
'/usr/local/galaxy/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py',
line 660 in _render
   **_kwargs_for_callable(callable_, data))
File
'/usr/local/galaxy/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py',
line 692 in _render_context
   _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File
'/usr/local/galaxy/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py',
line 718 in _exec_template
   callable_(context, *args, **kwargs)
File '/usr/local/galaxy/galaxy-dist/database/compiled_templates/reports/
base.mako.py', line 66 in render_body
   __M_writer(unicode(next.body()))
File '/usr/local/galaxy/galaxy-dist/database/compiled_templates/reports/
grid_base.mako.py', line 91 in render_body
   __M_writer(unicode(self.load()))
File '/usr/local/galaxy/galaxy-dist/database/compiled_templates/reports/
grid_base.mako.py', line 120 in render_load
   __M_writer(unicode( h.to_json_string( self.get_grid_config(
embedded=embedded, insert=insert ) ) ))
File '/usr/local/galaxy/galaxy-dist/database/compiled_templates/reports/
grid_base.mako.py', line 193 in render_get_grid_config
   'info_text' : grid.info_text,
AttributeError: 'SpecifiedDateListGrid' object has no attribute 'info_text'

Trello card is here:
https://trello.com/c/EqQCAcIe


Thanks,
Bjoern


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Cluster Admins - Call for experiences/issues encountered

[Björn Grüning]

Hi,


Yes, I am aware of such. I had intended to update the Cluster docs as
it's the more appropriate page, perhaps linking to it from the community
log page.

I was hoping that people might send in their experiences so I could
aggregate them and write the entries for them. There's always a lot of
apathy towards documentation, and given that this is a feature used by a
select few (even if many organisations use it, only a handful of people
are actually doing the deployment), I figured that they might be more
easily convinced to simply summarise their issues/experiences in a dozen
bulletpoints. Additionally by aggregation we could easily identify
common problems and highlight those for others to see.


That is always good, yes, but hard to motivate people.
My problem is that I do not remember every detail of every problem, it's 
to far away. Galaxy is running for us now. But if I encounter any new 
problem I will let you and the wiki know.



That said...no one has replied to this yet. I fully intend to at least
include mine, however I was holding out for some other replies.


Fingers crossed!
Bjoern


Cheers,
Eric


On 06/02/2014 08:55 AM, Björn Grüning wrote:

Hi Eric,

thanks for caring about that issue.

Do you know about Galaxy Community Hubs?

https://wiki.galaxyproject.org/Community/

The community is encouraged to share experience under that page. Maybe

you can add yours to that page as well?


Cheers,
Bjoern

Am 27.05.2014 18:32, schrieb Eric Rasche:
This is specifically to galaxy administrators and developers who have
worked with galaxy and cluster deployment.

If you have successfully deployed a galaxy instance backed by one (or
more) clusters, I'm looking to hear about your experience;

- the configuration
- any pitfalls encountered
- any extra tools that needed to be installed
- any ugly hacks you had to implement

The current cluster documentation is somewhat lacking, and I'm willing
to compile everyone's experiences deploying to clusters if you're
willing to send them in. I'll update the cluster page on the wiki as I
receive replies and can allocate time.



By way of example, here's the summary of what I would reply to this
email with my experiences deploying to an HTCondor cluster

- HTcondor ran fine, accepted jobs and worked for all users
- Connecting galaxy to it would result in some jobs running/some jobs
failing. It was determined that those jobs which ran, were only on
localhost.
- This was a result of the fact that galaxy's home directory/data was
not exported
- Galaxy was made an LDAP user and a folder created in the correct
location on all computers
- This still failed, as galaxy does not transfer files by default (as
far as I can tell), even though that's available in HTCondor
- Finally galaxy's home directory was NFS exported and things seemed to
work.




Cheers,
Eric


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/



- --
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M University
College Station, TX 77843
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTjI4NAAoJEMqDXdrsMcpVk5YP+waHzbx5MS3g31a3l7hC+84M
g0jakKQM2t6A/516q3YTm3FouGFpitcvxXYaesDDr/AbiRSWY8KGX6nqn8xk4LNH
yxc2Unewfwuzo8hB+PGHUncRnYAIc+BSDr57BDQukY7cuSBV6dymKkipXyim3v3Z
buLFKkby17SAvCmBxHN6cfGKnRs6ZLCh0qAyZaTWwmTmrqg3gY3WUh7jBsIS0qUV
bMEomgQ4vmqsOoEhi2UsA3ph09y8XcmyQCOkKLcR8nszvRcH/3s89YdMScxs9ghU
nGdVaO6B3zXG9QnWc8XgokISCldTcsP5GkKUPcMayz5n9tx2+Mf+BZQdQpGU7qQe
P5hGX0sJstklN1TRoJgvR/e6GBaEfT3yPgHlEi+1L7vlrb57vPYXi2Empph/KYo/
fUKGS0q2joxcg3/z2MNzrHc5+y+H9wR0dYVZrUEn3VBY+bhvaIdwpA4BBjjSjf99
BcvUk7FIKi6eyhCFX8RzU8CF4mXNqtrSXzd26igL8datxW0y7bhYcfiied0RMFQv
zz6Q4fQyV8hz5ufXKyf7Z1fJ+0PqIiGotN1/wD/l09mMFoKqJhvs4510LhCv7WW+
K7/KBJWJ2O8GrcuW4z7vydF+FxRLpkfGERzKI9IeMmxJiAs8OSYmIROsQzxg7alD
MEHRwJf6505DZSTeFDTn
=qHdv
-END PGP SIGNATURE-



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Cluster Admins - Call for experiences/issues encountered

[Björn Grüning]

Hi Eric,

thanks for caring about that issue.

Do you know about Galaxy Community Hubs?

https://wiki.galaxyproject.org/Community/

The community is encouraged to share experience under that page. Maybe 
you can add yours to that page as well?


Cheers,
Bjoern

Am 27.05.2014 18:32, schrieb Eric Rasche:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

This is specifically to galaxy administrators and developers who have
worked with galaxy and cluster deployment.

If you have successfully deployed a galaxy instance backed by one (or
more) clusters, I'm looking to hear about your experience;

- - the configuration
- - any pitfalls encountered
- - any extra tools that needed to be installed
- - any ugly hacks you had to implement

The current cluster documentation is somewhat lacking, and I'm willing
to compile everyone's experiences deploying to clusters if you're
willing to send them in. I'll update the cluster page on the wiki as I
receive replies and can allocate time.



By way of example, here's the summary of what I would reply to this
email with my experiences deploying to an HTCondor cluster

- - HTcondor ran fine, accepted jobs and worked for all users
- - Connecting galaxy to it would result in some jobs running/some jobs
failing. It was determined that those jobs which ran, were only on
localhost.
- - This was a result of the fact that galaxy's home directory/data was
not exported
- - Galaxy was made an LDAP user and a folder created in the correct
location on all computers
- - This still failed, as galaxy does not transfer files by default (as
far as I can tell), even though that's available in HTCondor
- - Finally galaxy's home directory was NFS exported and things seemed to
work.




Cheers,
Eric

- --
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M University
College Station, TX 77843
404-692-2048
e...@tamu.edu
rasche.e...@yandex.ru
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (GNU/Linux)

iQIcBAEBAgAGBQJThL4PAAoJEMqDXdrsMcpVJ0wP/jPFrTxRYnWcavbUxhht8tWE
VDhid2pst7o+qkkLAUylPyNzJz31movsU1Gd2FJLP4+RHb3lhy2GWm8mR9G4JuMM
g7fHAwIRj1qTT5tMXAEN6EoCtrValZH0EkIQPaFUPIiN1wzGTTDP56V+L6f3xQNg
zglk2GTO+hM+AunYGuDugDvdKTPyKgR1K2vaCNdc2kZlvQZe7TlOHZEbXaoDGTY4
vwHcNR0+8fvf6Hs1liCuCuh8nlTBABZLxMl5LgxOaZqbaAVQWsqXNdh1tF/rlERL
+PlNv5HRBWz26iGE04jbAD1pKmxg33Knl3l4MXrUnB5ZJ2dIDJL0YHOhXM3mjI4T
Hf8mv/iVr21xx3zrrAbAlRIt3Npw4DtaGSqBFdKvnxQZygjKEF7ZKTGfA4xLNRHh
4dPElUpv8b3jUe2GqhXyrmHYuBdWqBC5Y/ii0k4c/bHjoXW6yg0mPs7po7i5+5JS
wnO3mfoFhnyEALd45EBprkQY+kx05zjSg4ntpeywl8fIM2uGT0GyANA0/prKhcmB
5PiYUejsaDTY4GigiiZqz0tnC5z7VDyjByfyl7ufNaaCWNRXlTNIzH0ePy0ZjLrU
Sj8XyI1XoOSdtyxxvNNe8OPDxxffnJxi0rres4Q3CC/EbZK4vBF7chsisHd28Q29
M1e+lNdO8FBIoy5+MR3H
=6Jbk
-END PGP SIGNATURE-
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] tool dependencies fail

[Björn Grüning]

Hi,

can you try:


hhsuite


Hope that helps, sorry for the short answer I'm too tired :)
Bjoern

Am 31.05.2014 01:53, schrieb Isabelle Phan:

Hello,

I'm following "Managed Tool Dependencies" to the letter and still can't
get Galaxy to find the executable. Is this page
https://wiki.galaxyproject.org/Admin/Config/ToolDependencies up to date?
If yes, what am I doing wrong?

Any help greatly appreciated!


# universe_wsgi.ini
# I also tried the absolute path, it made no difference
tool_dependency_dir = tool_dependencies

# tool_dependencies is set like instructed:

$ ls tool_dependencies/hhsuite/
2.0.16/  default@
$ ls tool_dependencies/hhsuite/2.0.16/bin

ffindex_build* ffindex_get*   hhalign*   hhblits*   hhconsensus*
hhfilter*  hhmake*hhsearch*


My tool:

hhsuite
Searches single fasta iteratively against hmm db to build a
MSA
hhblits -i $input_file -d $database -oa3m $outfile
etc...


Galaxy throws Error: hhblits not found.


I've tried setting the PATH in the tool_dependencies/hhsuite/2.0.16/env.sh:
#!/bin/bash
# configure PATH to hhsuite binaries
PATH="/opt/galaxy-dist/tool_dependencies/hhsuite/default/bin:$PATH"
export PATH



still getting hhblits not found :-(
What am I doing wrong?

Isabelle


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] genotypeGVCFs in galaxy from GATK 3.0

[Björn Grüning]

Hi Nik,

the current wrappers are hosted here:
https://github.com/bgruening/galaxytools/tree/master/gatk2

and maintained by Nicola Soranzo, Jim Johnson and me.

The current plan is to realease a final version of GATK2.8 and start 
working on GATK3 after that.
I'm in contact the Geraldine Van der Auwera from the GATK devs, we are 
trying to generate some Json files which will describe the commandline 
interface of all GATK programs, with the aim of generating most of the 
wrappers automatically.


If you want to start creating GATK3 wrappers and can't wait longer, I 
would recommend to start from the GATK2 wrappers.


Cheers,
Bjoern

Am 21.05.2014 02:25, schrieb Nikhil Joshi:

Hi all,

Is anyone working on (or has already created) wrappers for GATK 3.0?
Specifically, I'm looking for a wrapper for genotypeGVCFs, but it seems
like I would want all the GATK wrappers to be updated. Any help is highly
appreciated!

- Nik.



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Keeping a tool from running multiple times at once

[Björn Grüning]

Hi Guillaume,

you can set a limit for  "concurrent_jobs" in job_conf.xml file. Please 
have a look at the job_conf.xml.sample_advanced file for more information.


Cheers,
Bjoern

Am 13.05.2014 12:11, schrieb Guillaume Penderia:

Hi,

I am currently working on a workflow with some custom tools, and one of
these tools has to create very big temporary files (like 45Go big).
As this workflow will be used on a lot of file at the same time, I have to
keep it from running more than once or twice at the same time (the other
execution would wait in the queue). If I don't, I'm afraid that some memory
lack or something could cause all the executions to fail and stop.

The problem is : I can't find if it's possible to do that, and if it is,
how do I do it.

Anyone has an idea please ?



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Inform tool interface with data specific to selected dataset

[Björn Grüning]

Hi Eric,

the following should work, but is a real hack and not recommended, I think.

https://galaxy-dist.readthedocs.org/en/latest/_modules/galaxy/tools/parameters/dynamic_options.html




Hope that will work for you.
Bjoern


Am 09.05.2014 16:42, schrieb Eric Rasche:

Howdy All,

I have a set of tools in two parts; the first part builds some databases and
prepares some data structures, the second tool plots a subset of the data in the
data structure. Specifically, the data in the structures are named genomes. The
second tools needs to know which genomes, and I'd like to allow the user to be
able to select from a list rather than typing in manually.

Essentially what I'd like to be able to do is create a set of options for an
 from the results of executing a perl/python script. Is
there any way to do this as of now? (ugly hacks are OK as this is a private 
server)

--
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M University
College Station, TX 77843
404-692-2048 
e...@tamu.edu 



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] identify tool based on qstat name.

[Björn Grüning]

Hi Geert,

it is a converter, defined in datatypes_conf.xml, loacted in 
/lib/galaxy/datatypes/converters/gff_to_fli_converter.xml and with the 
id "CONVERTER_gff_to_fli_0".


Cheers,
Bjoern

Am 09.05.2014 07:59, schrieb Geert Vandeweyer:

Hi,

I would like to know what package/default tools is creating the jobs on
our galaxy server like:

76905_converter_gff_to_fli_0_xx...@email.com

I see these jobs in the pbs queue, and they use a lot of memory, up to
18Gb of Ram. Since I can't find the job in the tool configuration, I
can't set the memory requirements in the job rules :-)

Thanks,

Geert


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Cleanest way to disable job-runs during a long cluster maintenace window?

[Björn Grüning]

Hi Curtis,

we simply remove our galaxy server from the SGE submit host list. So we 
can't submit anything to the Queue. That works quite well.


Cheers,
Bjoern

Am 01.05.2014 21:17, schrieb Curtis Hendrickson (Campus):

Peter,

Thanks for the quick reply. Saved me a lot of google/grep time.

Here's what I did to work-around (added it to the card as a comment). Anyone 
got a more thorough/elegant solution?

Regards,
Curtis


Workaround:

1. Backup migrated_tools_conf.xml, shed_tool_conf.xml and tool_conf.xml
2. edit each to remove all sections
3. edit static/welcome.html to notify users with a big orange announcement

That gives you an empty tool pane, though it doesn't prevent anyone from hitting 
"re-run" on an existing dataset, or doing something through the API.


-Original Message-
From: Peter Cock [mailto:p.j.a.c...@googlemail.com]
Sent: Thursday, May 01, 2014 1:03 PM
To: Curtis Hendrickson (Campus)
Cc: galaxy-dev PSU list-serv (galaxy-dev@lists.bx.psu.edu)
Subject: Re: [galaxy-dev] Cleanest way to disable job-runs during a long 
cluster maintenace window?

Hi Curtis,

This used to be easy - there was a setting on the admin pages to lock new job 
submission, which I would use before a planned shutdown.

See https://trello.com/c/BTDaHy9m/1269-re-institute-job-lock-feature
(and vote on it to help prioritize the issue).

Right now I'm not sure if there is an easy alternative :(

Peter


On Thu, May 1, 2014 at 6:45 PM, Curtis Hendrickson (Campus)  
wrote:

Folks



The cluster under our galaxy server will be down for a WEEK, so it can
be relocated.

The fileserver that hosts the galaxy datasets, the galaxy server and
the galaxy database will stay up.



What’s the most elegant way to disable people’s ability to run new
jobs, without blocking them from browsing existing histories and
downloading/viewing their data?



Our configuration:

 External Auth

 universe_wsgi.ini:use_remote_user =
True

 Job runner: SGE

 universe_wsgi.ini:start_job_runners =
drmaa

 and the new_file_path and job_working_directories are
on a fileserver that will be down.

grep -H /scratch/share/galaxy universe_wsgi.ini

universe_wsgi.ini:file_path = /scratch/share/galaxy/staging  #
available

universe_wsgi.ini:new_file_path = /scratch/share/galaxy/temp  # down

universe_wsgi.ini:job_working_directory =
/scratch/share/galaxy/job_working_directory # down



We’d rather something nicer than just letting the jobs go to the queue
and never get run.



Regards,

Curtis




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this and other
Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Announcement: Galaxy Docker builds

[Björn Grüning]

Dear Galaxy Devs and Users,

I'm pleased to announce the Galaxy Docker Image project!

The Galaxy Docker Image is an easy distributable full-fledged Galaxy 
installation, that can be used for testing, teaching and presenting new 
tools and features.


One of the main goals is to make the access to entire tool suites as 
easy as possible. Usually, this includes the setup of a public available 
webservice that needs to be maintained, or that the Tool-user needs to 
either setup a Galaxy Server by its own or to have Admin access to a 
local Galaxy server. With docker, tool developers can create their own 
Image with all dependencies and the user only needs to run it within 
docker. Ideally suited to make your reviewers happy :)


Currently, I have three test images to play with. The first one, 
bgruening/galaxy-stable is the main Image that will do all the heavy 
lifting of setting up PostgreSQL, Apache and Galaxy.


bgruening/deepTools and bgruening/chemicaltoolbox are proof of concepts 
how easy it is to create an own Image build on top of the main Galaxy Image.


https://index.docker.io/u/bgruening/galaxy-deeptools/
https://index.docker.io/u/bgruening/galaxy-chemicaltoolbox/

How to use it:
docker run -d -p 8080:80 bgruening/galaxy-deeptools
docker run -d -p 8080:80 -p 8000:8000 bgruening/galaxy-chemicaltoolbox

For a more detailed description please read:
https://github.com/bgruening/docker-recipes/tree/master/galaxy

I would be very much interested in any kind of feedback.
Especially, in two technical questions:

I'm not sure if a "trusted build" from the docker index is the best way 
to go. Trusted Builds in docker are super convenient. It is a 
post-commit hook from github. Every time something is pushed, the image 
will be rebuild. The problem is that I can't tag my images with "trusted 
builds" and unless I have missed something, that forces me to create new 
repositories every time a new Galaxy release is out.
A better way would be, one repository with different tags: 
galaxy-stable, galaxy-feb-2014 and so on. Every Galaxy release will be 
just a diff between the old and the new. But this is not possible with 
"trusted builds".


One other trick I used is to remove the mercurial history, that saves 
180MB. I'm not sure if this is wise, because it probably kills a few use 
cases. Currently, the aim of this project is a little bit different from 
being a developer environment. So I decided to remove it. Please ping me 
if this was a bad decision :)


Thanks for reading!
Bjoern
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] [galaxy-iuc] Test ToolShed package_samtools_0_1_19 status error?

[Björn Grüning]

Hi,

I updated the test and the tool shed accordingly to Dave suggestion.
I'm just wondering, the TS tests were passing with the old version, 
anything I should worry about? :)


Thanks and sorry for the inconvenience!
Bjoern


Am 29.04.2014 16:07, schrieb Peter Cock:

On Tue, Apr 29, 2014 at 2:59 PM, Dave Bouvier  wrote:

Peter,

Based on the latest test run, and my own investigations, it would appear
that the samtools compilation process is looking in the wrong include path
for ncurses/ncurses_dll.h. My recommendation would be to replace the
following:

-I$NCURSES_INCLUDE_PATH/ncurses/

with

-I$NCURSES_INCLUDE_PATH/ncurses/ -I$NCURSES_INCLUDE_PATH

in the shell_command for package_samtools_0_1_19's tool_dependencies.xml
that starts with: sed -i -e "s|CFLAGS=\s*-g\s*-Wall\s*-O2\s*|

   --Dave B.


Thanks Dave,

That sounds like it would be more robust to subtle server changes.

I see Bjoern has been working on this today with a slightly different
change - I'll leave this in his capable hands:

https://github.com/bgruening/galaxytools/commits/master/orphan_tool_dependencies/package_samtools_0_1_19/tool_dependencies.xml

Regards,

Peter


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Packaging pycurl on toolshed

[Björn Grüning]

Hi Saket,

is there any script installed with that package? --install-scripts may 
be needed. Also I'm wondering if there is no dependency on curl-lib?


Cheers,
Bjoern

Am 29.04.2014 19:16, schrieb Saket Choudhary:

Any comments on this?
http://testtoolshed.g2.bx.psu.edu/view/saketkc/package_pycurl_7_19_3_1

On 17 April 2014 19:54, Saket Choudhary  wrote:

My attempt to 'package' pycurl on testtoolshed fails with the following error:

src/pycurl.c: In function ‘do_multi_info_read’: src/pycurl.c:3549:15:
warning: call to ‘_curl_easy_getinfo_err_string’ declared with
attribute warning: curl_easy_getinfo expects a pointer to char * for
this info [enabled by default] src/pycurl.c: In function
‘do_curl_getinfo’: src/pycurl.c:2888:19: warning: call to
‘_curl_easy_getinfo_err_curl_slist’ declared with attribute warning:
curl_easy_getinfo expects a pointer to struct curl_slist * for this
info [enabled by default] error: could not create
'/usr/local/share/doc': Permission denied


A local install worked fine. Is this because of libcurl-devel missing
on toolshed instance?


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] A proposed modules extension for toolshed wrappers

[Björn Grüning]

Hi Ida,

now I got it.
I may have an idea, not tested, but you could try the following.

- Add a file under $galaxy_tools/R_env/3.1.0/env.sh
- Create a symlink from $galaxy_tools/R_env/default/ to 
$galaxy_tools/R_env/3.1.0/

- change your requirement tag to
R_env

Now you can put in everything you want have in the env.sh file. That 
file will be sourced as soon as your tool will be executed. For example 
you can load your module.


Hope that works!
Thanks,
Bjoern


Dear  Björn,

maybe I could change the tool, which is not mine, and which I don't want to 
maintain to use a specific R-version that is already available on our cluster 
and
which I can put into my path with "module load R/3.1.0-devel" ( 
http://modules.sourceforge.net/)

there even is a requirements in the wrapper which is not fulfilled (these 
versions are not available at the moment) and still it installed without 
problems.
   
 edgeR
 limma
   

This requirements tags looks also rather inflexible to me. With an additional 
level of user configurable indirection it would be possible to make the tools
fit to different infrastructures without having to use binaries provided by 
somebody else, taking up space for just one tool etc...

Currently for jobs that are not run locally there is a general 
environment_setup_file. There could be one optional environment_setup_file for 
every job (or destination).

In the end I created yet another wrapper.

best,
ido


On Apr 25, 2014, at 2:34 PM, Björn Grüning  wrote:


Hi Ido,

I do not get your question in all detail, but it is possible to define a 
tool_dependencies.xml with a specific R version, + R libraries and use only that 
specific version from your tool with  tags.

For an example please see various R tools from:

https://github.com/bgruening/galaxytools

Cheers,
Bjoern

Am 25.04.2014 14:25, schrieb Ido Tamir:

Hi,
has anything changed in galaxy in this regard?
Any way to modify an environment before a tools is run?

I now have a tool relying on R-devel and bioconductor devel, both of which I 
can load in a module.
The tool comes from the toolshed with xml like:


…


I don't want to hack around in the tool itself, but simply load the necessary 
R-version.

thank you very much,
ido


On Sep 13, 2013, at 3:23 AM, "Guest, Simon"  
wrote:


Just been reading a bit more about the Galaxy packaging system.  Here's a 
slight modification to what I was suggesting that might fit in a bit better.  
Apologies for not being more familiar with the existing system before proposing 
extensions.

Recall that my goal is to support using a system-installed (native) package, at 
a defined version, which I aim to achieve by loading the appropriate 
environment module before running a tool.

We still have tool_dependencies.xml defining a package at a particular version, 
but rather than download and build the source code, there's just a directive 
that says how to pick up the correct program version at runtime, e.g. which 
environment module to load.

So instead of the tool_dependencies.xml fragment:




http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2
make

bwa
$INSTALL_DIR/bin


$INSTALL_DIR/bin






We have something like this (NB: element and attribute names are for 
illustrative purposes only):





bwa/0.6.2





This causes the right thing (module load bwa/0.6.2) to be stuck into the 
dependencies env.sh file when this package is installed from the toolshed.  We 
could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with 
the existing download-and-make one.

We might want a bit of flexibility on what actions are supported (in case we 
want to support Software Collections, for example).

What do you think?

cheers,
Simon

PS: In case it wasn't already clear, solving this problem well is quite 
important to us here at AgResearch.  ;-)


===
Attention: The information contained in this message and/or attachments
from AgResearch Limited is intended only for the persons or entities
to which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipients is prohibited by AgResearch
Limited. If you have received this message in error, please notify the
sender immediately.
===

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subsc

Re: [galaxy-dev] A proposed modules extension for toolshed wrappers

[Björn Grüning]

Hi Ido,

I do not get your question in all detail, but it is possible to define a 
tool_dependencies.xml with a specific R version, + R libraries and use 
only that specific version from your tool with  tags.


For an example please see various R tools from:

https://github.com/bgruening/galaxytools

Cheers,
Bjoern

Am 25.04.2014 14:25, schrieb Ido Tamir:

Hi,
has anything changed in galaxy in this regard?
Any way to modify an environment before a tools is run?

I now have a tool relying on R-devel and bioconductor devel, both of which I 
can load in a module.
The tool comes from the toolshed with xml like:


…


I don't want to hack around in the tool itself, but simply load the necessary 
R-version.

thank you very much,
ido


On Sep 13, 2013, at 3:23 AM, "Guest, Simon"  
wrote:


Just been reading a bit more about the Galaxy packaging system.  Here's a 
slight modification to what I was suggesting that might fit in a bit better.  
Apologies for not being more familiar with the existing system before proposing 
extensions.

Recall that my goal is to support using a system-installed (native) package, at 
a defined version, which I aim to achieve by loading the appropriate 
environment module before running a tool.

We still have tool_dependencies.xml defining a package at a particular version, 
but rather than download and build the source code, there's just a directive 
that says how to pick up the correct program version at runtime, e.g. which 
environment module to load.

So instead of the tool_dependencies.xml fragment:




http://downloads.sourceforge.net/project/bio-bwa/bwa-0.6.2.tar.bz2
make

bwa
$INSTALL_DIR/bin


$INSTALL_DIR/bin






We have something like this (NB: element and attribute names are for 
illustrative purposes only):





bwa/0.6.2





This causes the right thing (module load bwa/0.6.2) to be stuck into the 
dependencies env.sh file when this package is installed from the toolshed.  We 
could call this toolshed tool native_package_bwa_0_6_2, to avoid confusion with 
the existing download-and-make one.

We might want a bit of flexibility on what actions are supported (in case we 
want to support Software Collections, for example).

What do you think?

cheers,
Simon

PS: In case it wasn't already clear, solving this problem well is quite 
important to us here at AgResearch.  ;-)


===
Attention: The information contained in this message and/or attachments
from AgResearch Limited is intended only for the persons or entities
to which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipients is prohibited by AgResearch
Limited. If you have received this message in error, please notify the
sender immediately.
===

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Specifying _JAVA_OPTIONS in job_conf.xml

[Björn Grüning]

Hi John,

I tested it a day long on our server and it seems to work. It actually a 
nice solution, as you can do many things with it I think.


Maybe it's worth to document, that 'foo bar' is necessary. 
The >'<  is important.


Thanks!
Bjoern

Am 22.04.2014 22:01, schrieb Björn Grüning:

:) Speechless ... I will stress test it tomorrow!

Am 22.04.2014 21:24, schrieb John Chilton:

On Tue, Apr 22, 2014 at 1:59 PM, Björn Grüning
 wrote:

Hi John,

Am 22.04.2014 20:45, schrieb John Chilton:


You can create multiple variants of the 10G_memory tool with different
native specifications right? Does this work around result in too much
XML? Dynamic destinations could also be used - if you are interested I
could post a demo rule that you might be able to use.


You are right of course - very sorry. Dynamic destinations and macros
would make sense if there was some way to alter the environment based
on the destination - which I am not sure is possible with
nativeSpecification at least generally. (It sounds like you can with
PBS/Torque).

Here is patch that would add the ability to specify VALUE tags to job destinations in job_conf.xml and
have them auto-populated in jobs.

https://github.com/jmchilton/galaxy-central/compare/job_env?expand=1
https://github.com/jmchilton/galaxy-central/commit/d05e15d712c9a3175552e7c5f7041301507dae1f.patch


Any interest in testing it for me :)?

If not, I can open a PR for it as it is.




The problem with multiple variants (multiple handlers) is that I only
can
specify one ENV variable that is known to the current shell. I don't
think I
can set the variable directly inside the nativeSpecification tag, or?
Moreover, the variable need to have the name _JAVA_OPTIONS.
To solve that I need to set the variable somehow inside the
nativeSpecification tag.

Dynamic destinations may work, but what happens if I change a ENV
inside of
python ... it is not known to the parent process or? Even if it is
known, is
it only known to that specific tool?



The memory card referenced above might help, but a much easier cards
might help this would be...

https://trello.com/c/t1FH1Q5P




Yes, but that is really hackish :). If there is no other way, I will try
something like that ...



Also tool style macros for job_conf.xml might help right - keep size
of XML down?

https://trello.com/c/wt1LflNh



Unfortunately, macros do not work here. In GATK different tools do have
different requirements.
I can probationally work around that with HEAP_SIZE/2 or something like
that, but a job_conf.xml based solution would be better.



Feel free to create another card for this request (allowing
parameterized tools to influence destination parameters) - seems
reasonable.



Ok I will wait a few more hours and than create such a card.
Thanks for your ideas,
Bjoern






-John

On Tue, Apr 22, 2014 at 1:32 PM, Björn Grüning
 wrote:


Hi,

to specify -Xmx and -Xms for java based programs I set my preferred
values
to $_JAVA_OPTIONS and include _JAVA_OPTIONS as nativeSpecification
with
-v
_JAVA_OPTIONS. That worked good so far, but the downside is that I can
not
have different options for different tools. Is there a trick I can
specify
arbitrary $ENVs to a specific tool? Any ideas how to solve that,
without
hacking the -Xmx and -Xms parameters into the wrapper file?

It is related to https://trello.com/c/HGEYPQf6.
I would be interested in workarounds until that card is closed :)

Something like that would be great:

export _JAVA_OPTIONS='-Xmx=6GB'


Thanks!
Bjoern



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Specifying _JAVA_OPTIONS in job_conf.xml

[Björn Grüning]

:) Speechless ... I will stress test it tomorrow!

Am 22.04.2014 21:24, schrieb John Chilton:

On Tue, Apr 22, 2014 at 1:59 PM, Björn Grüning
 wrote:

Hi John,

Am 22.04.2014 20:45, schrieb John Chilton:


You can create multiple variants of the 10G_memory tool with different
native specifications right? Does this work around result in too much
XML? Dynamic destinations could also be used - if you are interested I
could post a demo rule that you might be able to use.


You are right of course - very sorry. Dynamic destinations and macros
would make sense if there was some way to alter the environment based
on the destination - which I am not sure is possible with
nativeSpecification at least generally. (It sounds like you can with
PBS/Torque).

Here is patch that would add the ability to specify VALUE tags to job destinations in job_conf.xml and
have them auto-populated in jobs.

https://github.com/jmchilton/galaxy-central/compare/job_env?expand=1
https://github.com/jmchilton/galaxy-central/commit/d05e15d712c9a3175552e7c5f7041301507dae1f.patch

Any interest in testing it for me :)?

If not, I can open a PR for it as it is.




The problem with multiple variants (multiple handlers) is that I only can
specify one ENV variable that is known to the current shell. I don't think I
can set the variable directly inside the nativeSpecification tag, or?
Moreover, the variable need to have the name _JAVA_OPTIONS.
To solve that I need to set the variable somehow inside the
nativeSpecification tag.

Dynamic destinations may work, but what happens if I change a ENV inside of
python ... it is not known to the parent process or? Even if it is known, is
it only known to that specific tool?



The memory card referenced above might help, but a much easier cards
might help this would be...

https://trello.com/c/t1FH1Q5P




Yes, but that is really hackish :). If there is no other way, I will try
something like that ...



Also tool style macros for job_conf.xml might help right - keep size
of XML down?

https://trello.com/c/wt1LflNh



Unfortunately, macros do not work here. In GATK different tools do have
different requirements.
I can probationally work around that with HEAP_SIZE/2 or something like
that, but a job_conf.xml based solution would be better.



Feel free to create another card for this request (allowing
parameterized tools to influence destination parameters) - seems
reasonable.



Ok I will wait a few more hours and than create such a card.
Thanks for your ideas,
Bjoern






-John

On Tue, Apr 22, 2014 at 1:32 PM, Björn Grüning
 wrote:


Hi,

to specify -Xmx and -Xms for java based programs I set my preferred
values
to $_JAVA_OPTIONS and include _JAVA_OPTIONS as nativeSpecification with
-v
_JAVA_OPTIONS. That worked good so far, but the downside is that I can
not
have different options for different tools. Is there a trick I can
specify
arbitrary $ENVs to a specific tool? Any ideas how to solve that, without
hacking the -Xmx and -Xms parameters into the wrapper file?

It is related to https://trello.com/c/HGEYPQf6.
I would be interested in workarounds until that card is closed :)

Something like that would be great:

export _JAVA_OPTIONS='-Xmx=6GB'


Thanks!
Bjoern



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Specifying _JAVA_OPTIONS in job_conf.xml

[Björn Grüning]

Hi John,

Am 22.04.2014 20:45, schrieb John Chilton:

You can create multiple variants of the 10G_memory tool with different
native specifications right? Does this work around result in too much
XML? Dynamic destinations could also be used - if you are interested I
could post a demo rule that you might be able to use.


The problem with multiple variants (multiple handlers) is that I only 
can specify one ENV variable that is known to the current shell. I don't 
think I can set the variable directly inside the nativeSpecification 
tag, or? Moreover, the variable need to have the name _JAVA_OPTIONS.
To solve that I need to set the variable somehow inside the 
nativeSpecification tag.


Dynamic destinations may work, but what happens if I change a ENV inside 
of python ... it is not known to the parent process or? Even if it is 
known, is it only known to that specific tool?



The memory card referenced above might help, but a much easier cards
might help this would be...

https://trello.com/c/t1FH1Q5P



Yes, but that is really hackish :). If there is no other way, I will try 
something like that ...



Also tool style macros for job_conf.xml might help right - keep size
of XML down?

https://trello.com/c/wt1LflNh


Unfortunately, macros do not work here. In GATK different tools do have 
different requirements.
I can probationally work around that with HEAP_SIZE/2 or something like 
that, but a job_conf.xml based solution would be better.



Feel free to create another card for this request (allowing
parameterized tools to influence destination parameters) - seems
reasonable.


Ok I will wait a few more hours and than create such a card.
Thanks for your ideas,
Bjoern





-John

On Tue, Apr 22, 2014 at 1:32 PM, Björn Grüning
 wrote:

Hi,

to specify -Xmx and -Xms for java based programs I set my preferred values
to $_JAVA_OPTIONS and include _JAVA_OPTIONS as nativeSpecification with -v
_JAVA_OPTIONS. That worked good so far, but the downside is that I can not
have different options for different tools. Is there a trick I can specify
arbitrary $ENVs to a specific tool? Any ideas how to solve that, without
hacking the -Xmx and -Xms parameters into the wrapper file?

It is related to https://trello.com/c/HGEYPQf6.
I would be interested in workarounds until that card is closed :)

Something like that would be great:

export _JAVA_OPTIONS='-Xmx=6GB'


Thanks!
Bjoern



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Specifying _JAVA_OPTIONS in job_conf.xml

[Björn Grüning]

Hi,

to specify -Xmx and -Xms for java based programs I set my preferred 
values to $_JAVA_OPTIONS and include _JAVA_OPTIONS as 
nativeSpecification with -v _JAVA_OPTIONS. That worked good so far, but 
the downside is that I can not have different options for different 
tools. Is there a trick I can specify arbitrary $ENVs to a specific 
tool? Any ideas how to solve that, without hacking the -Xmx and -Xms 
parameters into the wrapper file?


It is related to https://trello.com/c/HGEYPQf6.
I would be interested in workarounds until that card is closed :)

Something like that would be great:
destination="10G_memory">

export _JAVA_OPTIONS='-Xmx=6GB'


Thanks!
Bjoern



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Installing S-Mart on Mac OSX local instance (updated)

[Björn Grüning]

Hi Bernado,

can you try to change your $HOME/shed_tools to a full path?
Also please try my small tutorial here:

https://github.com/bgruening/galaxytools/tree/master/chemicaltoolbox

Hope that will help you!

Cheers - Bjoern

Am 08.04.2014 23:39, schrieb Bernardo Bello:

Hi,

I am having problems to install a tool from tool shed named S_MART on my
local Galaxy. In fact, its the first tool I'm trying to install.
The reason to install S-MART is to use DETR'PROK in Galaxy.

Here is the 'error' Galaxy shows after installing some programs from
toolshed.

I've seen this related post, but I don't know if we are talking about the
same 
problem.


Here are the commands aI've used up to now:

Thanks, Bernardo

P.S. Sorry for the duplicated email,  I mistyped the subject in the
previous one.


hg clone https://bitbucket.org/galaxy/galaxy-dist/
cd galaxy-dist
hg update stable

sh run.sh # gave errors, so see below

# solve problem at startup (
https://www.mail-archive.com/galaxy-dev@lists.bx.psu.edu/msg13354.html)
% export LC_ALL='en_US.UTF-8'
% sh run.sh

done # Go to http://localhost:8080/ and register as popn...@gmail.com
# Setup admin user (https://wiki.galaxyproject.org/Admin/Interface). To
give a user Galaxy admin privileges, add their Galaxy login ( email ) to
the list in the following config setting in the Galaxy configuration file
universe_wsgi.ini.

admin_users = popn...@gmail.com

# Set 'Tool Dependencies'
https://wiki.galaxyproject.org/Admin/Config/ToolDependencies?action=show&redirect=Admin%2FConfig%2FTool+Dependencies
mkdir tool_dependency_dir

done # add tool dependencies folder to 'universe_wsgi.ini' -> Galaxy
provides a method for managing the dependencies of Galaxy tools installed
from the Tool Shed.  In this case, it is simply necessary to set the
tool_dependency_dir option of universe_wsgi.ini to a path writable by the
Galaxy server.

done # restart server

First, I modified the universe_wsgi.ini file by removing the "#" before the
tool_config_file and tool_path lines. #
http://user.list.galaxyproject.org/Installing-new-tools-from-the-tool-shed-td4655241.html

# Then, I created a shed_tools folder in my $HOME directory before
modifying the shed_tool_conf.xml file




done # INSTALL R
done # ADD R PACKAGES
R --slave --no-save --no-restore --quiet -e 'if("RColorBrewer" %in%
rownames(installed.packages()) == FALSE){install.packages("RColorBrewer",
repos = c("http://cran.rstudio.com/";), dependencies = TRUE)}'
R --slave --no-save --no-restore --quiet -e 'if("Hmisc" %in%
rownames(installed.packages()) == FALSE){install.packages("Hmisc", repos =
c("http://cran.rstudio.com/";), dependencies = TRUE)}'




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/