[galaxy-dev] Docker and alpine linux

[Tiago Antao]

Dear all,

This email is mostly to report a negative result, maybe to help others
_not_ trying something.

I researched the possibility of replacing ubuntu with alpine on the
Docker images (alpine seems to be used more and more on docker images,
with plenty of official containers now based on it and not on debian).

The reason alpine is used, its because it generates very small
containers (a bare bones one is below 10 MB). But, due the the large
dependencies of galaxy, the gain is negligible. Maybe 200 MB or so. 20%
is something, but not a revolution.

This being said, for servers with smaller dependencies (a mail server,
web, ldap, dns...) alpine really reduces the footprint of docker
containers.

Tiago

-- 
"While I may be sending this email outside my normal office hours, I
have no expectation to receive a reply outside yours" - @tomstafford
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy 16.01, eggs vs wheels, and running jobs as the user

[Nate Coraor]

On Mon, Feb 22, 2016 at 5:54 AM, Peter Cock 
wrote:

> Hi all,
>
> Last week at the Galaxy Admins video hangout Nate gave an
> overview of the way Python dependency handling is changing
> from using egg files to using wheel files instead - slide links at:
>
> https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2016_02_18
>
> I understand the Galaxy v16.01 (2016 January feature freeze)
> release which switches from eggs to wheels is due out shortly -
> hopefully this week?


> Recently we've been trying to setup a replacement Galaxy instance
> where jobs are submitted to our SGE cluster as the linux user
> requesting the job. This has been a bumpy road, e.g.
>
>
> http://dev.list.galaxyproject.org/Python-on-cluster-for-setting-metadata-after-jobs-complete-tt4668736.html
>
> Right now we've run into a problem with the metadata scripts
> trying to access ~/.python-eggs and going back over the archives
> it looks like we may need to set $PYTHON_EGG_CACHE to a
> temp folder on a per-node per-user basis, see e.g.
>
>
> http://dev.list.galaxyproject.org/python-egg-cache-exists-error-tt4656276.html#a4656394
>
> Sadly the wiki does not cover this aspect of running jobs as the
> real user (but the egg settings are about to be obsolete):
>
>
> https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#Submitting_Jobs_as_the_Real_User
>
> Clearly with Galaxy v16.01 the eggs and $PYTHON_EGG_CACHE
> will go away. What happens with wheels, Python virtual environments,
> and running jobs as the real user?
>

Hi Peter,

Unlike eggs, which can be installed in their package format (as a single
.egg file) and used in place, wheels are unpacked and their contents reside
in the virtualenv's site-packages just as if you had `python setup.py
install`ed them, there's no cache directory. The "real user" should be able
to use Galaxy's virtualenv as long as the appropriate read/execute
permissions are set on a common group or other.

Galaxy's virtualenv is passed down to the jobs when possible:


https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh#L17

If you need to create a separate virtualenv for jobs to use, you can do
this with an  tag on the destination:


https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies.html#galaxy-job-handlers

There are instructions in that document on how to create your own Galaxy
virtualenv:


https://docs.galaxyproject.org/en/release_16.01/admin/framework_dependencies.html#managing-dependencies-manually

I have not used the "real user" stuff in a very long time so I haven't
tested this, but I can't think of any major roadblocks that should prevent
it from working.

--nate


>
> Thanks,
>
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Plans for workflow & parallelisation work?

[John Chilton]

Peter -

My plans for pre-GCC workflow work are sort of outlined in this issue:
https://github.com/galaxyproject/planemo/issues/408 (I want an
abstract for GCC and BOSC like "Planemo – A Scientific Workflow SDK").

I've been doing most of my work out of this branch
https://github.com/galaxyproject/galaxy/compare/dev...common-workflow-language:cwl.
It has my work in progress on CWL support, collection operations
(rejected once from Galaxy here
https://github.com/galaxyproject/galaxy/pull/1313) but these are so
important I'm going to take another stab at pushing them into Galaxy,
and work on expression tools to produce values that will hopefully tie
back into workflows as connections for non-data parameters - both as
Galaxy native enties and CWL based enties.

There have been some completely valid complaints about the background
workflow scheduling being slow and buggy, these will need to be fixed
by 16.04 since all workflows will be executed this way as of then. I
hope also to take another pass at subworkflows - better tracking of
sources, allowing upgrading subworkflow steps, fixing glaring bugs
like https://github.com/galaxyproject/galaxy/issues/1739.

Peter C. mentioned splitting and joining files into/from collections
in workflows based on the datatype methods (so hooking into
parallelism) - I have some initial WIP on this here
https://github.com/jmchilton/galaxy/commit/c4d93acdb3b0f89b970b7c3d17b965be8ab3ba30
as part of this branch
https://github.com/jmchilton/galaxy/tree/split_merge_collections. I
spent a couple hours on it - I think if I spent a day or two on it I'd
have a usable prototype to hack on - I don't remember thinking there
were any big hurdles I was encountering in doing that. (So the answer
to your last question is a definitive yes.)

Sam started a bunch of work here with completely replacing the
workflow form with an API driven one here
https://github.com/galaxyproject/galaxy/pull/1249. I know he hopes to
have that done in 16.04 - it will allow us to delete a bunch of paths
through the workflow code and should allow future developments to be
made more rapidly. It will ensure everything is coming through the API
also - which means Galaxy's test coverage of workflow stuff will be
much higher (given our depth of workflow API tests).

I'm happy to have a hangout to discuss this more, I consider the
planemo issue something of a roadmap for what I want to work on in the
first half of 2016 - but I might get pulled away or told the project
has other priorities.

As for scheduling workflows instead of jobs - this is intriguing and
really would probably be needed to get streaming working well in
Galaxy. So I would say - I want to work on it someday - but I probably
won't get to it in 2016. If others want to hack on it, that is
fantastic but it is also a difficult feat.  (At least scheduling out
and optimizing pieces of the workflow, Kyle Ellrott, Dannon, and I had
some interesting ideas about scheduling whole workflows on local
Galaxy instances running on a cluster and just collecting the outputs
- that would be significantly more doable given I sort of sculpted the
changes made to backgrounding workflows to preserve things for doing
that - though the work left is probably still a hard task).

Hope this helps.

-John

On Mon, Feb 22, 2016 at 7:57 AM, Peter van Heusden  wrote:
> Hi there
>
> I see from the PR landing in Galaxy and the comments on things like issue
> #1701 (https://github.com/galaxyproject/galaxy/issues/1701) that there's
> lots of work happening on the workflow side of Galaxy. This is an area of
> interest at SANBI too, so we'd like to coordinate development efforts as
> much as possible. To this end:
>
> 1) Are there forks to track so we can see what new code is landing?
> 2) Is there a roadmap for workflow work or perhaps can we have a Hangout to
> talk about this?
> 3) Specifically in terms of workflows and parallelisation: are there any
> plans to work on running workflows as opposed to just generating lots of
> jobs? I know this is a major change to how Galaxy works - it would mean
> something like submitting a workflow specification to a job runner that is
> located on the cluster, and then returning the results of workflow
> execution.
> 4) Currently parallelisation in Galaxy is supported using two mechanisms:
> collections and dataset splitters/tasks. Are there plans on extending and
> harmonising Galaxy's parallelisation capabilities?
>
> Thanks,
> Peter
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client

Re: [galaxy-dev] cleanup

[Hans-Rudolf Hotz]

Hi Philipp



Since the new galaxy version, I am unable to perform a cleanup anymore.

 From the galaxy root directory I want to start the script
cleanup_datasets.py with the following line:

$ python cleanup_datasets.py /export/galaxy-central/config/galaxy.ini -d
-60 -1

I get the error message:

Can’t open file ‘cleanup_datasets.py’: no such file or directory.



Have you tried:

$ python scripts/cleanup_datasets/cleanup_datasets.py 
/export/galaxy-central/config/galaxy.ini -d -60 -1




But when I go into the folder in which the script is located it doesn’t
work as well.


what is the error this time? Is it "ImportError: No module named galaxy"?



Regards, Hans-Rudolf




For your information: we are running the latest docker-galaxy-stable
version.

I hope somebody has an idea.

Cheers,

Philipp

--

Philipp Rathert, PhD

Institute of Biochemistry

University Stuttgart

Pfaffenwaldring 55

D-70569 Stuttgart

Germany

Tel: +49 711 685 64388

Fax: +49 711 685 64392

philipp.rath...@ibc.uni-stuttgart.de


http://www.ibc.uni-stuttgart.de/

http://www.ibc.uni-stuttgart.de/mitarbeiter/Rathert/?__locale=en



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Collapse tool panel and history panel

[Makis Ladoukakis]

The git log ..origin/master command produced no output so I guess I have the 
latest release (15.10.1 maybe?). The git pull command confirmed that. Should I 
use a release from the dev branch?

Kind regards,
Makis

Date: Fri, 19 Feb 2016 08:39:35 -0500
Subject: Re: [galaxy-dev] Collapse tool panel and history panel
From: dannon.ba...@gmail.com
To: makis4e...@hotmail.com
CC: galaxy-...@lists.bx.psu.edu

Sure, I can help.  Which version of Galaxy are you currently running?  I didn't 
check the history very well and the fix I suggested will work for the 
forthcoming 16.01 release (or current dev branch), but not 15.10.

On Fri, Feb 19, 2016 at 8:32 AM, Makis Ladoukakis  
wrote:



Hello Dannon,

Sorry for asking but can you help me out a little bit with that? I searched in 
my galaxy directory in order to add those two lines you mention in the link but 
there is no galaxy/scripts/apps/analysis.js file and neither any file 
containing the string 'analysisPage.right.historyView'. 

Any advice?

Thank you,
Makis
Date: Mon, 28 Dec 2015 14:30:02 -0500
Subject: Re: [galaxy-dev] Collapse tool panel and history panel
From: dannon.ba...@gmail.com
To: makis4e...@hotmail.com
CC: galaxy-...@lists.bx.psu.edu

Hi Makis,
We've restructured much of the client code fairly recently.  Something like 
this should work for you now:
https://gist.github.com/dannon/bd470d9c70019b07cb8b


Sorry for the slow response!
-Dannon

On Sat, Dec 19, 2015 at 2:57 PM, Makis Ladoukakis  
wrote:



Shamelessly bumping my own question... Does anyone have any ideas about 
collapsing the history and tool panel?

Thank you,
Makis Ladoukakis

From: makis4e...@hotmail.com
To: galaxy-...@lists.bx.psu.edu
Date: Wed, 16 Dec 2015 13:41:19 +0200
Subject: [galaxy-dev] Collapse tool panel and history panel




Hello,

Does anyone know how to start my Galaxy instance with the history panel and 
tool panel collapsed by default?

I used to do it by editing the galaxy/templates/base/base_panels.mako script 
and adding

lp.do_toggle();

and 

rp.do_toggle();

inside the if sections of %if self.has_left_panel: and %if 
self.has_right_panel: respectively but with my new installed instance this 
doesn't seem to work anymore.

Any ideas anyone?

Thank you,
Makis Ladoukakis



  

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/ 
  

___

Please keep all replies on the list by using "reply all"

in your mail client.  To manage your subscriptions to this

and other Galaxy lists, please use the interface at:

  https://lists.galaxyproject.org/



To search Galaxy mailing lists use the unified search at:

  http://galaxyproject.org/search/mailinglists/

  

  ___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Galaxy 16.01, eggs vs wheels, and running jobs as the user

[Peter Cock]

Hi all,

Last week at the Galaxy Admins video hangout Nate gave an
overview of the way Python dependency handling is changing
from using egg files to using wheel files instead - slide links at:

https://wiki.galaxyproject.org/Community/GalaxyAdmins/Meetups/2016_02_18

I understand the Galaxy v16.01 (2016 January feature freeze)
release which switches from eggs to wheels is due out shortly -
hopefully this week?

Recently we've been trying to setup a replacement Galaxy instance
where jobs are submitted to our SGE cluster as the linux user
requesting the job. This has been a bumpy road, e.g.

http://dev.list.galaxyproject.org/Python-on-cluster-for-setting-metadata-after-jobs-complete-tt4668736.html

Right now we've run into a problem with the metadata scripts
trying to access ~/.python-eggs and going back over the archives
it looks like we may need to set $PYTHON_EGG_CACHE to a
temp folder on a per-node per-user basis, see e.g.

http://dev.list.galaxyproject.org/python-egg-cache-exists-error-tt4656276.html#a4656394

Sadly the wiki does not cover this aspect of running jobs as the
real user (but the egg settings are about to be obsolete):

https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#Submitting_Jobs_as_the_Real_User

Clearly with Galaxy v16.01 the eggs and $PYTHON_EGG_CACHE
will go away. What happens with wheels, Python virtual environments,
and running jobs as the real user?

Thanks,

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] JBrowse on dev branch: progress

[Marius van den Beek]

I have occasionally seen this.
GNU-coreutils hangs while compiling on ubuntu 14.04 ... there is a
`nanosleep` in the configure step,
which is never returning.
"checking for working nanosleep..."
https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/1079889
Just kill the process (this will happen twice), and it will work.
I will have a look if this is still happening with newer versions of
coreutils.

On 22 February 2016 at 10:43, Peter van Heusden  wrote:

> Ok, second attempt, perl install died with the same error, so I suspect
> you're right about the cause. Eric?
>
> Peter
>
> On 22 February 2016 at 11:41, Peter van Heusden  wrote:
>
>> Thanks Peter
>>
>> I managed, on the second attempt, to get past the gnu coreutils problem.
>> Zipho noticed a weird lockup though: conftest that seemed to be from the
>> gnu coreutils install wanted to listen on port 4001, which is the same port
>> our uwsgi server listens on. The install has now completed but I'll try and
>> find time to go back and verify this.
>>
>> On the second attempt perl seems to be installing, so I'll see if it
>> completes or hits the error you're mentioning.
>>
>> Peter
>>
>> On 22 February 2016 at 11:34, Peter Briggs > > wrote:
>>
>>> Hi Peter
>>>
>>> I think that package_perl_5_18 in the tool shed is broken, I came across
>>> the same error last week on both our production instance and a local 'test'
>>> instance (both v15.10).
>>>
>>> My suspicion is that the following lines are to blame:
>>>
>>> ...
>>> >> sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b"
>>> type="download_file">
>>> http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz
>>> 
>>> tar xfvz
>>> local-lib-1.008009.tar.gz
>>> ...
>>>
>>> Looking at the Galaxy internals I believe that the 'download_file'
>>> action automatically unpacks the archive and cd's into the resulting
>>> directory, hence the failure for the subsequent 'tar xfvz ...' command. But
>>> I haven't had the time yet to verify this or create a patch to see if the
>>> removal of this line addresses the problem.
>>>
>>> I don't think I've seen the problem with 'package_gnu_coreutils_8_22'
>>> that you describe however so it may be that my hypothesis is not correct.
>>>
>>> HTH
>>>
>>> Best wishes
>>>
>>> Peter
>>>
>>>
>>> On 22/02/16 05:10, Peter van Heusden wrote:
>>>
 Hi there Eric and others

 I tried to install the JBrowse tool on a dev branch server running on an
 Ubuntu 14.04 VM.

 It got stuck at the package_gnu_coreutils_8_22 dependency (revision
 ac64dfe4b1fb) - this ended up in "Installing tool dependencies" state.

 If I go look at the tool details it highlights errors
 in package_perl_5_18 which in turn refers to errors in gnu_coreutils
 (just "Error" state) and then this more informative failure for perl:

 "tar (child): local-lib-1.008009.tar.gz: Cannot open: No such file or
 directory
 tar (child): Error is not recoverable: exiting now
 tar: Child returned status 2
 tar: Error is not recoverable: exiting now"

 I wonder if this is a sign of a download error?

 Digging a bit more I find the error "Shutting down process id 21094
 because it generated no output for the defined timeout period of 3600.0
 seconds." for gnu_coreutils.

 I'm trying to do a reinstall of the gnu_coreutils and perl packages to
 see if I can get further.

 Peter


 ___
 Please keep all replies on the list by using "reply all"
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/


>>> --
>>> Peter Briggs peter.bri...@manchester.ac.uk
>>> Bioinformatics Core Facility University of Manchester
>>> B.1083 Michael Smith Bldg Tel: (0161) 2751482
>>>
>>
>>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] JBrowse on dev branch: progress

[Peter van Heusden]

Ok, second attempt, perl install died with the same error, so I suspect
you're right about the cause. Eric?

Peter

On 22 February 2016 at 11:41, Peter van Heusden  wrote:

> Thanks Peter
>
> I managed, on the second attempt, to get past the gnu coreutils problem.
> Zipho noticed a weird lockup though: conftest that seemed to be from the
> gnu coreutils install wanted to listen on port 4001, which is the same port
> our uwsgi server listens on. The install has now completed but I'll try and
> find time to go back and verify this.
>
> On the second attempt perl seems to be installing, so I'll see if it
> completes or hits the error you're mentioning.
>
> Peter
>
> On 22 February 2016 at 11:34, Peter Briggs 
> wrote:
>
>> Hi Peter
>>
>> I think that package_perl_5_18 in the tool shed is broken, I came across
>> the same error last week on both our production instance and a local 'test'
>> instance (both v15.10).
>>
>> My suspicion is that the following lines are to blame:
>>
>> ...
>> > sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b"
>> type="download_file">
>> http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz
>> 
>> tar xfvz
>> local-lib-1.008009.tar.gz
>> ...
>>
>> Looking at the Galaxy internals I believe that the 'download_file' action
>> automatically unpacks the archive and cd's into the resulting directory,
>> hence the failure for the subsequent 'tar xfvz ...' command. But I haven't
>> had the time yet to verify this or create a patch to see if the removal of
>> this line addresses the problem.
>>
>> I don't think I've seen the problem with 'package_gnu_coreutils_8_22'
>> that you describe however so it may be that my hypothesis is not correct.
>>
>> HTH
>>
>> Best wishes
>>
>> Peter
>>
>>
>> On 22/02/16 05:10, Peter van Heusden wrote:
>>
>>> Hi there Eric and others
>>>
>>> I tried to install the JBrowse tool on a dev branch server running on an
>>> Ubuntu 14.04 VM.
>>>
>>> It got stuck at the package_gnu_coreutils_8_22 dependency (revision
>>> ac64dfe4b1fb) - this ended up in "Installing tool dependencies" state.
>>>
>>> If I go look at the tool details it highlights errors
>>> in package_perl_5_18 which in turn refers to errors in gnu_coreutils
>>> (just "Error" state) and then this more informative failure for perl:
>>>
>>> "tar (child): local-lib-1.008009.tar.gz: Cannot open: No such file or
>>> directory
>>> tar (child): Error is not recoverable: exiting now
>>> tar: Child returned status 2
>>> tar: Error is not recoverable: exiting now"
>>>
>>> I wonder if this is a sign of a download error?
>>>
>>> Digging a bit more I find the error "Shutting down process id 21094
>>> because it generated no output for the defined timeout period of 3600.0
>>> seconds." for gnu_coreutils.
>>>
>>> I'm trying to do a reinstall of the gnu_coreutils and perl packages to
>>> see if I can get further.
>>>
>>> Peter
>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>http://galaxyproject.org/search/mailinglists/
>>>
>>>
>> --
>> Peter Briggs peter.bri...@manchester.ac.uk
>> Bioinformatics Core Facility University of Manchester
>> B.1083 Michael Smith Bldg Tel: (0161) 2751482
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] JBrowse on dev branch: progress

[Peter van Heusden]

Thanks Peter

I managed, on the second attempt, to get past the gnu coreutils problem.
Zipho noticed a weird lockup though: conftest that seemed to be from the
gnu coreutils install wanted to listen on port 4001, which is the same port
our uwsgi server listens on. The install has now completed but I'll try and
find time to go back and verify this.

On the second attempt perl seems to be installing, so I'll see if it
completes or hits the error you're mentioning.

Peter

On 22 February 2016 at 11:34, Peter Briggs 
wrote:

> Hi Peter
>
> I think that package_perl_5_18 in the tool shed is broken, I came across
> the same error last week on both our production instance and a local 'test'
> instance (both v15.10).
>
> My suspicion is that the following lines are to blame:
>
> ...
>  sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b"
> type="download_file">
> http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz
> 
> tar xfvz
> local-lib-1.008009.tar.gz
> ...
>
> Looking at the Galaxy internals I believe that the 'download_file' action
> automatically unpacks the archive and cd's into the resulting directory,
> hence the failure for the subsequent 'tar xfvz ...' command. But I haven't
> had the time yet to verify this or create a patch to see if the removal of
> this line addresses the problem.
>
> I don't think I've seen the problem with 'package_gnu_coreutils_8_22' that
> you describe however so it may be that my hypothesis is not correct.
>
> HTH
>
> Best wishes
>
> Peter
>
>
> On 22/02/16 05:10, Peter van Heusden wrote:
>
>> Hi there Eric and others
>>
>> I tried to install the JBrowse tool on a dev branch server running on an
>> Ubuntu 14.04 VM.
>>
>> It got stuck at the package_gnu_coreutils_8_22 dependency (revision
>> ac64dfe4b1fb) - this ended up in "Installing tool dependencies" state.
>>
>> If I go look at the tool details it highlights errors
>> in package_perl_5_18 which in turn refers to errors in gnu_coreutils
>> (just "Error" state) and then this more informative failure for perl:
>>
>> "tar (child): local-lib-1.008009.tar.gz: Cannot open: No such file or
>> directory
>> tar (child): Error is not recoverable: exiting now
>> tar: Child returned status 2
>> tar: Error is not recoverable: exiting now"
>>
>> I wonder if this is a sign of a download error?
>>
>> Digging a bit more I find the error "Shutting down process id 21094
>> because it generated no output for the defined timeout period of 3600.0
>> seconds." for gnu_coreutils.
>>
>> I'm trying to do a reinstall of the gnu_coreutils and perl packages to
>> see if I can get further.
>>
>> Peter
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>http://galaxyproject.org/search/mailinglists/
>>
>>
> --
> Peter Briggs peter.bri...@manchester.ac.uk
> Bioinformatics Core Facility University of Manchester
> B.1083 Michael Smith Bldg Tel: (0161) 2751482
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] JBrowse on dev branch: progress

[Peter Briggs]

Forgot to 'reply-to-list', sorry:

Hi Peter

I think that package_perl_5_18 in the tool shed is broken, I came across 
the same error last week on both our production instance and a local 
'test' instance (both v15.10).


My suspicion is that the following lines are to blame:

...
sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b" 
type="download_file">

http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz

tar xfvz 
local-lib-1.008009.tar.gz

...

Looking at the Galaxy internals I believe that the 'download_file' 
action automatically unpacks the archive and cd's into the resulting 
directory, hence the failure for the subsequent 'tar xfvz ...' command. 
But I haven't had the time yet to verify this or create a patch to see 
if the removal of this line addresses the problem.


I don't think I've seen the problem with 'package_gnu_coreutils_8_22' 
that you describe however so it may be that my hypothesis is not correct.


HTH

Best wishes

Peter

On 22/02/16 05:10, Peter van Heusden wrote:

Hi there Eric and others

I tried to install the JBrowse tool on a dev branch server running on an
Ubuntu 14.04 VM.

It got stuck at the package_gnu_coreutils_8_22 dependency (revision
ac64dfe4b1fb) - this ended up in "Installing tool dependencies" state.

If I go look at the tool details it highlights errors
in package_perl_5_18 which in turn refers to errors in gnu_coreutils
(just "Error" state) and then this more informative failure for perl:

"tar (child): local-lib-1.008009.tar.gz: Cannot open: No such file or
directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now"

I wonder if this is a sign of a download error?

Digging a bit more I find the error "Shutting down process id 21094
because it generated no output for the defined timeout period of 3600.0
seconds." for gnu_coreutils.

I'm trying to do a reinstall of the gnu_coreutils and perl packages to
see if I can get further.

Peter


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/



--
Peter Briggs peter.bri...@manchester.ac.uk
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Plans for workflow & parallelisation work?

[Peter Cock]

On Mon, Feb 22, 2016 at 7:57 AM, Peter van Heusden  wrote:
> Hi there
>
> ...
>
> 4) Currently parallelisation in Galaxy is supported using two mechanisms:
> collections and dataset splitters/tasks. Are there plans on extending and
> harmonising Galaxy's parallelisation capabilities?

I'm not sure there is anything formal, but chatting to John and others
at GCC2015 we recognised that the split/merge capabilities in the
Python datatype classes have a lot of functional overlap between
splitting and merging for datasets into collections.

https://wiki.galaxyproject.org/Events/GCC2015/BoFs/DataSplittingAndParallelism

One idea we mooted was defining (pseudo) tools for dataset splitting
and merging using the existing datatype classes, with similar integration
into the framework as the datatype converter tools.

i.e. You could in principle merge a collection of text files using the
text datatype's merge functionality (which is essentially a cat
command).

There are a lot of details to think about, particularly for splitting
where currently tool wrappers using parallelisation have some
control (e.g. split a large FASTA file into chunks of 1000 sequences),
which might need to be exposed in any UI for creating a collection
from a single file.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/