Re: [galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Steve Cassidy]

Ok, probing further trying to understand this. It looks like the job runner is 
trying to call set_meta() from galaxy_ext/metadata/set_metadata.py.  In there 
there’s a line that tries to set up the PATH:

# insert *this* galaxy before all others on sys.path
sys.path.insert( 1, os.path.abspath( os.path.join( os.path.dirname( __file__ ), 
os.pardir, os.pardir ) ) )

In my case the result is:

['/export/galaxy-central/database/job_working_directory/000/3’, 
'/galaxy-central/lib’, 
'/galaxy-central/lib’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python27.zip’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/plat-linux2’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-tk’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-old’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/lib-dynload’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/site-packages’,
 
'/export/galaxy-central/database/job_working_directory/000/3/conda-env/lib/python2.7/site-packages/setuptools-25.1.6-py2.7.egg’
]

Looking for sqlalchemy, I find it installed at:

/galaxy-central/.venv/lib/python2.7/site-packages/sqlalchemy

so clearly that line of code is not doing what it should. I’m guessing that the 
Conda resolver is running the script within a conda env and the BETA magic 
hasn’t quite made it to the right place yet… 

BTW I was trying to understand the flags you mentioned but I can’t find 
GALAXY_CONFIG_ENABLE_BETA_TOOL_COMMAND_ISOLATION anywhere other than in sample 
Dockerfiles - is this some kind of magic incantation that creates a rift in 
space-time…enquiring minds want to know!!!

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 25 Aug 2016, at 9:54 PM, Steve Cassidy  wrote:
> 
> Nah, definitely baby steps…
> 
> so, in the repo you point to there seems to be an error in the Dockerfile, 
> the ENV line should use the var=value syntax to have more than one setting on 
> one line (maybe that’s changed recently in docker).
> 
> with this I built a new docker image, when I run my first tool it takes an 
> age while it’s installing the deps, then it crashes with:
> 
> Traceback (most recent call last):
>  File 
> "/export/galaxy-central/database/job_working_directory/000/1/set_metadata_QUJQLD.py",
>  line 1, in 
>from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
>  File "/galaxy-central/lib/galaxy_ext/metadata/set_metadata.py", line 23, in 
> 
>from sqlalchemy.orm import clear_mappers
> ImportError: No module named sqlalchemy.orm
> 
> and the output:
> 
> discarding /galaxy-central/tool_deps/_conda/bin from PATH
> prepending 
> /export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to 
> PATH
> discarding /galaxy-central/tool_deps/_conda/bin from PATH
> prepending 
> /export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to 
> PATH
> 
> I’m guessing this is some kind of conflict between python versions in and out 
> of conda environments? Surely sqlalchemy would be installed for Galaxy to 
> work?  
> 
> I’ll try to dig around this in the morning but if it rings a bell…
> 
> Steve
> —
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy
> 
>> On 25 Aug 2016, at 8:43 PM, Björn Grüning  wrote:
>> 
>> Hi Steve,
>> 
>> you call this baby-steps? I think this is huge! :)
>> 
>> All what you are missing is to enable conda in Galaxy.
>> Look at Gregs new flavour which is entirely Conda/Galaxy based.
>> 
>> You need to enable these env vars to make Galaxy conda enabled:
>> 
>> https://github.com/gregvonkuster/docker-galaxy-csg/blob/master/Dockerfile#L9
>> 
>> Hope this helps,
>> Bjoern
>> 
>> Am 25.08.2016 um 12:32 schrieb Steve Cassidy:
>>> Hi all,
>>> I’m making baby steps towards having a repeatable installation for my
>>> tools.  But I’m now stuck, so help would be appreciated.
>>> 
>>> I have a tool that works and is in the test toolshed (alveoimport in
>>> Data Sources). 
>>> 
>>> It depends on my python package which is now part of bioconda (pyalveo,
>>> version 0.6). 
>>> 
>>> I can run my tool via planemo, which works I think because I have
>>> pyalveo installed in a local venv.  
>>> 
>>> If I try to run the docker image (derived from  bgruening/galaxy-stable
>>> but with the testtoolshed added) I am able to install my tool, but it
>>> doesn’t pick up the dependency, so it doesn’t work. 
>>> 
>>> I tried running with planemo turning on conda dependency resolution
>>> (following https://pypi.python.org/pypi/planemo/): 
>>> 
>>> planemo serve --galaxy_branch release_16.01 --conda_dependency_resolution .
>>> 
>>> It seems to have a go, but fails:
>>> 
>>> galaxy.tools.

Re: [galaxy-dev] Parameter Storing in Database

[Katherine Beaulieu]

On Thu, Aug 25, 2016 at 2:25 PM, Katherine Beaulieu <
katherine.beaulieu...@gmail.com> wrote:

> Hi Peter,
> No actually I meant in what file is the table job_parameter getting filled
> during execution?
> Katherine
>
> On Fri, Aug 5, 2016 at 3:00 PM, Peter van Heusden  wrote:
>
>> You mean the stuff for the tool-data tables? As I understand it, it is
>> loaded in:
>>
>> lib/galaxy/tools/data/__init__.py
>>
>> There's a ToolDataTableManager and a TabularToolDataTable (with
>> associated TabularToolDataField).
>>
>> Peter
>>
>> On Fri, 5 Aug 2016 at 20:46 Katherine Beaulieu <
>> katherine.beaulieu...@gmail.com> wrote:
>>
>>> Would anyone be able to tell me in what file the storing of tool
>>> parameters into the database occurs?
>>> Thanks!
>>> Katherine
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/mailinglists/
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Steve Cassidy]

Nah, definitely baby steps…

so, in the repo you point to there seems to be an error in the Dockerfile, the 
ENV line should use the var=value syntax to have more than one setting on one 
line (maybe that’s changed recently in docker).

with this I built a new docker image, when I run my first tool it takes an age 
while it’s installing the deps, then it crashes with:

Traceback (most recent call last):
  File 
"/export/galaxy-central/database/job_working_directory/000/1/set_metadata_QUJQLD.py",
 line 1, in 
from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
  File "/galaxy-central/lib/galaxy_ext/metadata/set_metadata.py", line 23, in 

from sqlalchemy.orm import clear_mappers
ImportError: No module named sqlalchemy.orm

and the output:

discarding /galaxy-central/tool_deps/_conda/bin from PATH
prepending 
/export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to 
PATH
discarding /galaxy-central/tool_deps/_conda/bin from PATH
prepending 
/export/galaxy-central/database/job_working_directory/000/1/conda-env/bin to 
PATH

I’m guessing this is some kind of conflict between python versions in and out 
of conda environments? Surely sqlalchemy would be installed for Galaxy to work? 
 

I’ll try to dig around this in the morning but if it rings a bell…

Steve
—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

> On 25 Aug 2016, at 8:43 PM, Björn Grüning  wrote:
> 
> Hi Steve,
> 
> you call this baby-steps? I think this is huge! :)
> 
> All what you are missing is to enable conda in Galaxy.
> Look at Gregs new flavour which is entirely Conda/Galaxy based.
> 
> You need to enable these env vars to make Galaxy conda enabled:
> 
> https://github.com/gregvonkuster/docker-galaxy-csg/blob/master/Dockerfile#L9
> 
> Hope this helps,
> Bjoern
> 
> Am 25.08.2016 um 12:32 schrieb Steve Cassidy:
>> Hi all,
>>  I’m making baby steps towards having a repeatable installation for my
>> tools.  But I’m now stuck, so help would be appreciated.
>> 
>> I have a tool that works and is in the test toolshed (alveoimport in
>> Data Sources). 
>> 
>> It depends on my python package which is now part of bioconda (pyalveo,
>> version 0.6). 
>> 
>> I can run my tool via planemo, which works I think because I have
>> pyalveo installed in a local venv.  
>> 
>> If I try to run the docker image (derived from  bgruening/galaxy-stable
>> but with the testtoolshed added) I am able to install my tool, but it
>> doesn’t pick up the dependency, so it doesn’t work. 
>> 
>> I tried running with planemo turning on conda dependency resolution
>> (following https://pypi.python.org/pypi/planemo/): 
>> 
>> planemo serve --galaxy_branch release_16.01 --conda_dependency_resolution .
>> 
>> It seems to have a go, but fails:
>> 
>> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Building dependency
>> shell command for dependency 'pyalveo'
>> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Find dependency pyalveo
>> version 0.6
>> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned
>> 
>> (isnull? True)
>> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned
>> 
>> (isnull? True)
>> galaxy.tools.deps WARNING 2016-08-25 17:32:22,451 Failed to resolve
>> dependency on 'pyalveo', ignoring
>> 
>> So, what’s the easiest route to a Galaxy deployment with my tool
>> installed. The Docker route would be best I think, but what do I have to
>> add to bgruening/galaxy-stable to have conda find my dependencies.
>> 
>> Thanks in advance,
>> 
>> Steve
>> 
>> —
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy
>> 
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>> 
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
>> 

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Björn Grüning]

Hi Steve,

you call this baby-steps? I think this is huge! :)

All what you are missing is to enable conda in Galaxy.
Look at Gregs new flavour which is entirely Conda/Galaxy based.

You need to enable these env vars to make Galaxy conda enabled:

https://github.com/gregvonkuster/docker-galaxy-csg/blob/master/Dockerfile#L9

Hope this helps,
Bjoern

Am 25.08.2016 um 12:32 schrieb Steve Cassidy:
> Hi all,
>   I’m making baby steps towards having a repeatable installation for my
> tools.  But I’m now stuck, so help would be appreciated.
> 
> I have a tool that works and is in the test toolshed (alveoimport in
> Data Sources). 
> 
> It depends on my python package which is now part of bioconda (pyalveo,
> version 0.6). 
> 
> I can run my tool via planemo, which works I think because I have
> pyalveo installed in a local venv.  
> 
> If I try to run the docker image (derived from  bgruening/galaxy-stable
> but with the testtoolshed added) I am able to install my tool, but it
> doesn’t pick up the dependency, so it doesn’t work. 
> 
> I tried running with planemo turning on conda dependency resolution
> (following https://pypi.python.org/pypi/planemo/): 
> 
> planemo serve --galaxy_branch release_16.01 --conda_dependency_resolution .
> 
> It seems to have a go, but fails:
> 
> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Building dependency
> shell command for dependency 'pyalveo'
> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Find dependency pyalveo
> version 0.6
> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned
> 
> (isnull? True)
> galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned
> 
> (isnull? True)
> galaxy.tools.deps WARNING 2016-08-25 17:32:22,451 Failed to resolve
> dependency on 'pyalveo', ignoring
> 
> So, what’s the easiest route to a Galaxy deployment with my tool
> installed. The Docker route would be best I think, but what do I have to
> add to bgruening/galaxy-stable to have conda find my dependencies.
> 
> Thanks in advance,
> 
> Steve
> 
> —
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy
> 
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Steve Cassidy]

Hi all,
  I’m making baby steps towards having a repeatable installation for my tools.  
But I’m now stuck, so help would be appreciated.

I have a tool that works and is in the test toolshed (alveoimport in Data 
Sources).

It depends on my python package which is now part of bioconda (pyalveo, version 
0.6).

I can run my tool via planemo, which works I think because I have pyalveo 
installed in a local venv.

If I try to run the docker image (derived from  bgruening/galaxy-stable but 
with the testtoolshed added) I am able to install my tool, but it doesn’t pick 
up the dependency, so it doesn’t work.

I tried running with planemo turning on conda dependency resolution (following 
https://pypi.python.org/pypi/planemo/):

planemo serve --galaxy_branch release_16.01 --conda_dependency_resolution .

It seems to have a go, but fails:

galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Building dependency shell 
command for dependency 'pyalveo'
galaxy.tools.deps DEBUG 2016-08-25 17:32:22,449 Find dependency pyalveo version 
0.6
galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned 
 (isnull? 
True)
galaxy.tools.deps DEBUG 2016-08-25 17:32:22,450 Resolver conda returned 
 (isnull? 
True)
galaxy.tools.deps WARNING 2016-08-25 17:32:22,451 Failed to resolve dependency 
on 'pyalveo', ignoring

So, what’s the easiest route to a Galaxy deployment with my tool installed. The 
Docker route would be best I think, but what do I have to add to 
bgruening/galaxy-stable to have conda find my dependencies.

Thanks in advance,

Steve

—
Department of Computing, Macquarie University
http://web.science.mq.edu.au/~cassidy

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/