[galaxy-dev] Docker and alpine linux
Dear all, This email is mostly to report a negative result, maybe to help others _not_ trying something. I researched the possibility of replacing ubuntu with alpine on the Docker images (alpine seems to be used more and more on docker images, with plenty of official containers now based on it and not on debian). The reason alpine is used, its because it generates very small containers (a bare bones one is below 10 MB). But, due the the large dependencies of galaxy, the gain is negligible. Maybe 200 MB or so. 20% is something, but not a revolution. This being said, for servers with smaller dependencies (a mail server, web, ldap, dns...) alpine really reduces the footprint of docker containers. Tiago -- "While I may be sending this email outside my normal office hours, I have no expectation to receive a reply outside yours" - @tomstafford ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy 16.01, eggs vs wheels, and running jobs as the user
On Mon, Feb 22, 2016 at 5:54 AM, Peter Cockwrote: > Hi all, > > Last week at the Galaxy Admins video hangout Nate gave an > overview of the way Python dependency handling is changing > from using egg files to using wheel files instead - slide links at: > > https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2016_02_18 > > I understand the Galaxy v16.01 (2016 January feature freeze) > release which switches from eggs to wheels is due out shortly - > hopefully this week? > Recently we've been trying to setup a replacement Galaxy instance > where jobs are submitted to our SGE cluster as the linux user > requesting the job. This has been a bumpy road, e.g. > > > http://dev.list.galaxyproject.org/Python-on-cluster-for-setting-metadata-after-jobs-complete-tt4668736.html > > Right now we've run into a problem with the metadata scripts > trying to access ~/.python-eggs and going back over the archives > it looks like we may need to set $PYTHON_EGG_CACHE to a > temp folder on a per-node per-user basis, see e.g. > > > http://dev.list.galaxyproject.org/python-egg-cache-exists-error-tt4656276.html#a4656394 > > Sadly the wiki does not cover this aspect of running jobs as the > real user (but the egg settings are about to be obsolete): > > > https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#Submitting_Jobs_as_the_Real_User > > Clearly with Galaxy v16.01 the eggs and $PYTHON_EGG_CACHE > will go away. What happens with wheels, Python virtual environments, > and running jobs as the real user? > Hi Peter, Unlike eggs, which can be installed in their package format (as a single .egg file) and used in place, wheels are unpacked and their contents reside in the virtualenv's site-packages just as if you had `python setup.py install`ed them, there's no cache directory. The "real user" should be able to use Galaxy's virtualenv as long as the appropriate read/execute permissions are set on a common group or other. Galaxy's virtualenv is passed down to the jobs when possible: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh#L17 If you need to create a separate virtualenv for jobs to use, you can do this with an tag on the destination: https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies.html#galaxy-job-handlers There are instructions in that document on how to create your own Galaxy virtualenv: https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies.html#managing-dependencies-manually I have not used the "real user" stuff in a very long time so I haven't tested this, but I can't think of any major roadblocks that should prevent it from working. --nate > > Thanks, > > Peter > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] JBrowse on dev branch: progress
I have occasionally seen this. GNU-coreutils hangs while compiling on ubuntu 14.04 ... there is a `nanosleep` in the configure step, which is never returning. "checking for working nanosleep..." https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/1079889 Just kill the process (this will happen twice), and it will work. I will have a look if this is still happening with newer versions of coreutils. On 22 February 2016 at 10:43, Peter van Heusdenwrote: > Ok, second attempt, perl install died with the same error, so I suspect > you're right about the cause. Eric? > > Peter > > On 22 February 2016 at 11:41, Peter van Heusden wrote: > >> Thanks Peter >> >> I managed, on the second attempt, to get past the gnu coreutils problem. >> Zipho noticed a weird lockup though: conftest that seemed to be from the >> gnu coreutils install wanted to listen on port 4001, which is the same port >> our uwsgi server listens on. The install has now completed but I'll try and >> find time to go back and verify this. >> >> On the second attempt perl seems to be installing, so I'll see if it >> completes or hits the error you're mentioning. >> >> Peter >> >> On 22 February 2016 at 11:34, Peter Briggs > > wrote: >> >>> Hi Peter >>> >>> I think that package_perl_5_18 in the tool shed is broken, I came across >>> the same error last week on both our production instance and a local 'test' >>> instance (both v15.10). >>> >>> My suspicion is that the following lines are to blame: >>> >>> ... >>> >> sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b" >>> type="download_file"> >>> http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz >>> >>> tar xfvz >>> local-lib-1.008009.tar.gz >>> ... >>> >>> Looking at the Galaxy internals I believe that the 'download_file' >>> action automatically unpacks the archive and cd's into the resulting >>> directory, hence the failure for the subsequent 'tar xfvz ...' command. But >>> I haven't had the time yet to verify this or create a patch to see if the >>> removal of this line addresses the problem. >>> >>> I don't think I've seen the problem with 'package_gnu_coreutils_8_22' >>> that you describe however so it may be that my hypothesis is not correct. >>> >>> HTH >>> >>> Best wishes >>> >>> Peter >>> >>> >>> On 22/02/16 05:10, Peter van Heusden wrote: >>> Hi there Eric and others I tried to install the JBrowse tool on a dev branch server running on an Ubuntu 14.04 VM. It got stuck at the package_gnu_coreutils_8_22 dependency (revision ac64dfe4b1fb) - this ended up in "Installing tool dependencies" state. If I go look at the tool details it highlights errors in package_perl_5_18 which in turn refers to errors in gnu_coreutils (just "Error" state) and then this more informative failure for perl: "tar (child): local-lib-1.008009.tar.gz: Cannot open: No such file or directory tar (child): Error is not recoverable: exiting now tar: Child returned status 2 tar: Error is not recoverable: exiting now" I wonder if this is a sign of a download error? Digging a bit more I find the error "Shutting down process id 21094 because it generated no output for the defined timeout period of 3600.0 seconds." for gnu_coreutils. I'm trying to do a reinstall of the gnu_coreutils and perl packages to see if I can get further. Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ >>> -- >>> Peter Briggs peter.bri...@manchester.ac.uk >>> Bioinformatics Core Facility University of Manchester >>> B.1083 Michael Smith Bldg Tel: (0161) 2751482 >>> >> >> > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] JBrowse on dev branch: progress
Ok, second attempt, perl install died with the same error, so I suspect you're right about the cause. Eric? Peter On 22 February 2016 at 11:41, Peter van Heusdenwrote: > Thanks Peter > > I managed, on the second attempt, to get past the gnu coreutils problem. > Zipho noticed a weird lockup though: conftest that seemed to be from the > gnu coreutils install wanted to listen on port 4001, which is the same port > our uwsgi server listens on. The install has now completed but I'll try and > find time to go back and verify this. > > On the second attempt perl seems to be installing, so I'll see if it > completes or hits the error you're mentioning. > > Peter > > On 22 February 2016 at 11:34, Peter Briggs > wrote: > >> Hi Peter >> >> I think that package_perl_5_18 in the tool shed is broken, I came across >> the same error last week on both our production instance and a local 'test' >> instance (both v15.10). >> >> My suspicion is that the following lines are to blame: >> >> ... >> > sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b" >> type="download_file"> >> http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz >> >> tar xfvz >> local-lib-1.008009.tar.gz >> ... >> >> Looking at the Galaxy internals I believe that the 'download_file' action >> automatically unpacks the archive and cd's into the resulting directory, >> hence the failure for the subsequent 'tar xfvz ...' command. But I haven't >> had the time yet to verify this or create a patch to see if the removal of >> this line addresses the problem. >> >> I don't think I've seen the problem with 'package_gnu_coreutils_8_22' >> that you describe however so it may be that my hypothesis is not correct. >> >> HTH >> >> Best wishes >> >> Peter >> >> >> On 22/02/16 05:10, Peter van Heusden wrote: >> >>> Hi there Eric and others >>> >>> I tried to install the JBrowse tool on a dev branch server running on an >>> Ubuntu 14.04 VM. >>> >>> It got stuck at the package_gnu_coreutils_8_22 dependency (revision >>> ac64dfe4b1fb) - this ended up in "Installing tool dependencies" state. >>> >>> If I go look at the tool details it highlights errors >>> in package_perl_5_18 which in turn refers to errors in gnu_coreutils >>> (just "Error" state) and then this more informative failure for perl: >>> >>> "tar (child): local-lib-1.008009.tar.gz: Cannot open: No such file or >>> directory >>> tar (child): Error is not recoverable: exiting now >>> tar: Child returned status 2 >>> tar: Error is not recoverable: exiting now" >>> >>> I wonder if this is a sign of a download error? >>> >>> Digging a bit more I find the error "Shutting down process id 21094 >>> because it generated no output for the defined timeout period of 3600.0 >>> seconds." for gnu_coreutils. >>> >>> I'm trying to do a reinstall of the gnu_coreutils and perl packages to >>> see if I can get further. >>> >>> Peter >>> >>> >>> ___ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>>https://lists.galaxyproject.org/ >>> >>> To search Galaxy mailing lists use the unified search at: >>>http://galaxyproject.org/search/mailinglists/ >>> >>> >> -- >> Peter Briggs peter.bri...@manchester.ac.uk >> Bioinformatics Core Facility University of Manchester >> B.1083 Michael Smith Bldg Tel: (0161) 2751482 >> > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/