Re: [galaxy-dev] Galaxy 16.01, eggs vs wheels, and running jobs as the user

2016-02-23 Thread Nate Coraor 
On Tue, Feb 23, 2016 at 4:04 AM, Peter Cock 
wrote:

> On Mon, Feb 22, 2016 at 9:02 PM, Nate Coraor  wrote:
> > On Mon, Feb 22, 2016 at 5:54 AM, Peter Cock 
> > wrote:
> >>
> >> Hi all,
> >>
> >> Last week at the Galaxy Admins video hangout Nate gave an
> >> overview of the way Python dependency handling is changing
> >> from using egg files to using wheel files instead - slide links at:
> >>
> >> ...
> >>
> >> Recently we've been trying to setup a replacement Galaxy instance
> >> where jobs are submitted to our SGE cluster as the linux user
> >> requesting the job. This has been a bumpy road, ...
> >>
> >> ...
> >>
> >> Clearly with Galaxy v16.01 the eggs and $PYTHON_EGG_CACHE
> >> will go away. What happens with wheels, Python virtual environments,
> >> and running jobs as the real user?
> >
> >
> > Hi Peter,
> >
> > Unlike eggs, which can be installed in their package format (as a single
> > .egg file) and used in place, wheels are unpacked and their contents
> reside
> > in the virtualenv's site-packages just as if you had `python setup.py
> > install`ed them, there's no cache directory. The "real user" should be
> able
> > to use Galaxy's virtualenv as long as the appropriate read/execute
> > permissions are set on a common group or other.
> >
> > Galaxy's virtualenv is passed down to the jobs when possible:
> >
> >
> https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh#L17
> >
> > If you need to create a separate virtualenv for jobs to use, you can do
> this
> > with an  tag on the destination:
> >
> >
> >
> https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies.html#galaxy-job-handlers
> >
> > There are instructions in that document on how to create your own Galaxy
> > virtualenv:
> >
> >
> >
> https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies.html#managing-dependencies-manually
> >
> > I have not used the "real user" stuff in a very long time so I haven't
> > tested this, but I can't think of any major roadblocks that should
> prevent
> > it from working.
> >
> > --nate
>
> Thanks Nate, this sounds promising - fingers crossed :)
>
> Maybe we should test the pre-release branch for v16.10 out?
>

Yes, I don't expect much to change before the release, which, fingers
crossed, will be tomorrow or Wednesday.


>
> Peter
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy 16.01, eggs vs wheels, and running jobs as the user

2016-02-22 Thread Nate Coraor 
On Mon, Feb 22, 2016 at 5:54 AM, Peter Cock 
wrote:

> Hi all,
>
> Last week at the Galaxy Admins video hangout Nate gave an
> overview of the way Python dependency handling is changing
> from using egg files to using wheel files instead - slide links at:
>
> https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2016_02_18
>
> I understand the Galaxy v16.01 (2016 January feature freeze)
> release which switches from eggs to wheels is due out shortly -
> hopefully this week?


> Recently we've been trying to setup a replacement Galaxy instance
> where jobs are submitted to our SGE cluster as the linux user
> requesting the job. This has been a bumpy road, e.g.
>
>
> http://dev.list.galaxyproject.org/Python-on-cluster-for-setting-metadata-after-jobs-complete-tt4668736.html
>
> Right now we've run into a problem with the metadata scripts
> trying to access ~/.python-eggs and going back over the archives
> it looks like we may need to set $PYTHON_EGG_CACHE to a
> temp folder on a per-node per-user basis, see e.g.
>
>
> http://dev.list.galaxyproject.org/python-egg-cache-exists-error-tt4656276.html#a4656394
>
> Sadly the wiki does not cover this aspect of running jobs as the
> real user (but the egg settings are about to be obsolete):
>
>
> https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#Submitting_Jobs_as_the_Real_User
>
> Clearly with Galaxy v16.01 the eggs and $PYTHON_EGG_CACHE
> will go away. What happens with wheels, Python virtual environments,
> and running jobs as the real user?
>

Hi Peter,

Unlike eggs, which can be installed in their package format (as a single
.egg file) and used in place, wheels are unpacked and their contents reside
in the virtualenv's site-packages just as if you had `python setup.py
install`ed them, there's no cache directory. The "real user" should be able
to use Galaxy's virtualenv as long as the appropriate read/execute
permissions are set on a common group or other.

Galaxy's virtualenv is passed down to the jobs when possible:


https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh#L17

If you need to create a separate virtualenv for jobs to use, you can do
this with an  tag on the destination:


https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies.html#galaxy-job-handlers

There are instructions in that document on how to create your own Galaxy
virtualenv:


https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies.html#managing-dependencies-manually

I have not used the "real user" stuff in a very long time so I haven't
tested this, but I can't think of any major roadblocks that should prevent
it from working.

--nate


>
> Thanks,
>
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/