[galaxy-dev] Docker and alpine linux

2016-02-22 Thread Tiago Antao
Dear all,

This email is mostly to report a negative result, maybe to help others
_not_ trying something.

I researched the possibility of replacing ubuntu with alpine on the
Docker images (alpine seems to be used more and more on docker images,
with plenty of official containers now based on it and not on debian).

The reason alpine is used, its because it generates very small
containers (a bare bones one is below 10 MB). But, due the the large
dependencies of galaxy, the gain is negligible. Maybe 200 MB or so. 20%
is something, but not a revolution.

This being said, for servers with smaller dependencies (a mail server,
web, ldap, dns...) alpine really reduces the footprint of docker
containers.

Tiago

-- 
"While I may be sending this email outside my normal office hours, I
have no expectation to receive a reply outside yours" - @tomstafford
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy 16.01, eggs vs wheels, and running jobs as the user

2016-02-22 Thread Nate Coraor
On Mon, Feb 22, 2016 at 5:54 AM, Peter Cock 
wrote:

> Hi all,
>
> Last week at the Galaxy Admins video hangout Nate gave an
> overview of the way Python dependency handling is changing
> from using egg files to using wheel files instead - slide links at:
>
> https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2016_02_18
>
> I understand the Galaxy v16.01 (2016 January feature freeze)
> release which switches from eggs to wheels is due out shortly -
> hopefully this week?


> Recently we've been trying to setup a replacement Galaxy instance
> where jobs are submitted to our SGE cluster as the linux user
> requesting the job. This has been a bumpy road, e.g.
>
>
> http://dev.list.galaxyproject.org/Python-on-cluster-for-setting-metadata-after-jobs-complete-tt4668736.html
>
> Right now we've run into a problem with the metadata scripts
> trying to access ~/.python-eggs and going back over the archives
> it looks like we may need to set $PYTHON_EGG_CACHE to a
> temp folder on a per-node per-user basis, see e.g.
>
>
> http://dev.list.galaxyproject.org/python-egg-cache-exists-error-tt4656276.html#a4656394
>
> Sadly the wiki does not cover this aspect of running jobs as the
> real user (but the egg settings are about to be obsolete):
>
>
> https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#Submitting_Jobs_as_the_Real_User
>
> Clearly with Galaxy v16.01 the eggs and $PYTHON_EGG_CACHE
> will go away. What happens with wheels, Python virtual environments,
> and running jobs as the real user?
>

Hi Peter,

Unlike eggs, which can be installed in their package format (as a single
.egg file) and used in place, wheels are unpacked and their contents reside
in the virtualenv's site-packages just as if you had `python setup.py
install`ed them, there's no cache directory. The "real user" should be able
to use Galaxy's virtualenv as long as the appropriate read/execute
permissions are set on a common group or other.

Galaxy's virtualenv is passed down to the jobs when possible:


https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh#L17

If you need to create a separate virtualenv for jobs to use, you can do
this with an  tag on the destination:


https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies.html#galaxy-job-handlers

There are instructions in that document on how to create your own Galaxy
virtualenv:


https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies.html#managing-dependencies-manually

I have not used the "real user" stuff in a very long time so I haven't
tested this, but I can't think of any major roadblocks that should prevent
it from working.

--nate


>
> Thanks,
>
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] JBrowse on dev branch: progress

2016-02-22 Thread Marius van den Beek
I have occasionally seen this.
GNU-coreutils hangs while compiling on ubuntu 14.04 ... there is a
`nanosleep` in the configure step,
which is never returning.
"checking for working nanosleep..."
https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/1079889
Just kill the process (this will happen twice), and it will work.
I will have a look if this is still happening with newer versions of
coreutils.

On 22 February 2016 at 10:43, Peter van Heusden  wrote:

> Ok, second attempt, perl install died with the same error, so I suspect
> you're right about the cause. Eric?
>
> Peter
>
> On 22 February 2016 at 11:41, Peter van Heusden  wrote:
>
>> Thanks Peter
>>
>> I managed, on the second attempt, to get past the gnu coreutils problem.
>> Zipho noticed a weird lockup though: conftest that seemed to be from the
>> gnu coreutils install wanted to listen on port 4001, which is the same port
>> our uwsgi server listens on. The install has now completed but I'll try and
>> find time to go back and verify this.
>>
>> On the second attempt perl seems to be installing, so I'll see if it
>> completes or hits the error you're mentioning.
>>
>> Peter
>>
>> On 22 February 2016 at 11:34, Peter Briggs > > wrote:
>>
>>> Hi Peter
>>>
>>> I think that package_perl_5_18 in the tool shed is broken, I came across
>>> the same error last week on both our production instance and a local 'test'
>>> instance (both v15.10).
>>>
>>> My suspicion is that the following lines are to blame:
>>>
>>> ...
>>> >> sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b"
>>> type="download_file">
>>> http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz
>>> 
>>> tar xfvz
>>> local-lib-1.008009.tar.gz
>>> ...
>>>
>>> Looking at the Galaxy internals I believe that the 'download_file'
>>> action automatically unpacks the archive and cd's into the resulting
>>> directory, hence the failure for the subsequent 'tar xfvz ...' command. But
>>> I haven't had the time yet to verify this or create a patch to see if the
>>> removal of this line addresses the problem.
>>>
>>> I don't think I've seen the problem with 'package_gnu_coreutils_8_22'
>>> that you describe however so it may be that my hypothesis is not correct.
>>>
>>> HTH
>>>
>>> Best wishes
>>>
>>> Peter
>>>
>>>
>>> On 22/02/16 05:10, Peter van Heusden wrote:
>>>
 Hi there Eric and others

 I tried to install the JBrowse tool on a dev branch server running on an
 Ubuntu 14.04 VM.

 It got stuck at the package_gnu_coreutils_8_22 dependency (revision
 ac64dfe4b1fb) - this ended up in "Installing tool dependencies" state.

 If I go look at the tool details it highlights errors
 in package_perl_5_18 which in turn refers to errors in gnu_coreutils
 (just "Error" state) and then this more informative failure for perl:

 "tar (child): local-lib-1.008009.tar.gz: Cannot open: No such file or
 directory
 tar (child): Error is not recoverable: exiting now
 tar: Child returned status 2
 tar: Error is not recoverable: exiting now"

 I wonder if this is a sign of a download error?

 Digging a bit more I find the error "Shutting down process id 21094
 because it generated no output for the defined timeout period of 3600.0
 seconds." for gnu_coreutils.

 I'm trying to do a reinstall of the gnu_coreutils and perl packages to
 see if I can get further.

 Peter


 ___
 Please keep all replies on the list by using "reply all"
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/


>>> --
>>> Peter Briggs peter.bri...@manchester.ac.uk
>>> Bioinformatics Core Facility University of Manchester
>>> B.1083 Michael Smith Bldg Tel: (0161) 2751482
>>>
>>
>>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] JBrowse on dev branch: progress

2016-02-22 Thread Peter van Heusden
Ok, second attempt, perl install died with the same error, so I suspect
you're right about the cause. Eric?

Peter

On 22 February 2016 at 11:41, Peter van Heusden  wrote:

> Thanks Peter
>
> I managed, on the second attempt, to get past the gnu coreutils problem.
> Zipho noticed a weird lockup though: conftest that seemed to be from the
> gnu coreutils install wanted to listen on port 4001, which is the same port
> our uwsgi server listens on. The install has now completed but I'll try and
> find time to go back and verify this.
>
> On the second attempt perl seems to be installing, so I'll see if it
> completes or hits the error you're mentioning.
>
> Peter
>
> On 22 February 2016 at 11:34, Peter Briggs 
> wrote:
>
>> Hi Peter
>>
>> I think that package_perl_5_18 in the tool shed is broken, I came across
>> the same error last week on both our production instance and a local 'test'
>> instance (both v15.10).
>>
>> My suspicion is that the following lines are to blame:
>>
>> ...
>> > sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b"
>> type="download_file">
>> http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz
>> 
>> tar xfvz
>> local-lib-1.008009.tar.gz
>> ...
>>
>> Looking at the Galaxy internals I believe that the 'download_file' action
>> automatically unpacks the archive and cd's into the resulting directory,
>> hence the failure for the subsequent 'tar xfvz ...' command. But I haven't
>> had the time yet to verify this or create a patch to see if the removal of
>> this line addresses the problem.
>>
>> I don't think I've seen the problem with 'package_gnu_coreutils_8_22'
>> that you describe however so it may be that my hypothesis is not correct.
>>
>> HTH
>>
>> Best wishes
>>
>> Peter
>>
>>
>> On 22/02/16 05:10, Peter van Heusden wrote:
>>
>>> Hi there Eric and others
>>>
>>> I tried to install the JBrowse tool on a dev branch server running on an
>>> Ubuntu 14.04 VM.
>>>
>>> It got stuck at the package_gnu_coreutils_8_22 dependency (revision
>>> ac64dfe4b1fb) - this ended up in "Installing tool dependencies" state.
>>>
>>> If I go look at the tool details it highlights errors
>>> in package_perl_5_18 which in turn refers to errors in gnu_coreutils
>>> (just "Error" state) and then this more informative failure for perl:
>>>
>>> "tar (child): local-lib-1.008009.tar.gz: Cannot open: No such file or
>>> directory
>>> tar (child): Error is not recoverable: exiting now
>>> tar: Child returned status 2
>>> tar: Error is not recoverable: exiting now"
>>>
>>> I wonder if this is a sign of a download error?
>>>
>>> Digging a bit more I find the error "Shutting down process id 21094
>>> because it generated no output for the defined timeout period of 3600.0
>>> seconds." for gnu_coreutils.
>>>
>>> I'm trying to do a reinstall of the gnu_coreutils and perl packages to
>>> see if I can get further.
>>>
>>> Peter
>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>http://galaxyproject.org/search/mailinglists/
>>>
>>>
>> --
>> Peter Briggs peter.bri...@manchester.ac.uk
>> Bioinformatics Core Facility University of Manchester
>> B.1083 Michael Smith Bldg Tel: (0161) 2751482
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/