[galaxy-dev] Re: Ansible Galaxy using root user for role installation

[Peter van Heusden]

It is not actually, in my investigation, coming from an Ansible default.
There's code inside the Galaxy role that tries to figure out which user to
use for remote_user (the one used in SSH) and when it is run with `become:
true` ends up incorrectly choosing root. However people more knowledgeable
than me don't have this problem. Perhaps something changed in some version
somewhere?

Peter

On Mon, 15 Feb 2021 at 17:16, Michael Moore 
wrote:

> Anything attempting to login remotely as root must be rejected.  Peter's
> policy approach is good for that.  Most modern linices will not allow a
> remote root login, which is correct.  There are SSH ways to accomplish the
> same thing, but depend on trust at both ends.  The defaulting by Ansible is
> reprehensible, but usually should be expected to fail.
>
>
> On Mon, Feb 15, 2021 at 5:15 AM Peter van Heusden  wrote:
>
>> Hi there
>>
>> I found I had this problem too, when the `become: true` applied to the
>> whole Galaxy play. My current top level playbook (at
>> https://github.com/pvanheus/masters_galaxy_ansible/blob/master/galaxy.yml#L25)
>> now applies `become: true` as it is needed, and does not apply it to the
>> `galaxyproject.galaxy` role. I found that this avoids Ansible trying to log
>> in as the root user.
>>
>> Peter
>>
>> On Mon, 15 Feb 2021 at 07:21, Sandra Maksimovic <
>> sandra.maksimo...@mcri.edu.au> wrote:
>>
>>> Hi there,
>>>
>>>
>>>
>>> Would anyone happen to know why Ansible tries to connect to a remote
>>> host (CentOS 7) using 'root' during Galaxy role installation?
>>>
>>>
>>>
>>> My playbook has been configured to connect to the remote host using a
>>> different user account via SSH, however, the role installation is not
>>> honoring this for some reason and continues to use root. The remote host
>>> has had SSH root login disabled for security reasons and setting various
>>> galaxy user/privsep/become/remote user variables in the group_vars has not
>>> had any effect.
>>>
>>>
>>>
>>> TASK [galaxyproject.galaxy : Create galaxy_root] *
>>>
>>> fatal: [server_name]: UNREACHABLE! => {"changed": false, "msg":
>>> "Invalid/incorrect password: Permission denied, please try
>>> again.\r\nPermission denied, please try again.\r\nroot@server_name:
>>> Permission denied (publickey,password).", "unreachable": true}
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Sandra
>>>
>>>
>>> This e-mail and any attachments to it (the "Communication") are, unless
>>> otherwise stated, confidential, may contain copyright material and is for
>>> the use only of the intended recipient. If you receive the Communication in
>>> error, please notify the sender immediately by return e-mail, delete the
>>> Communication and the return e-mail, and do not read, copy, retransmit or
>>> otherwise deal with it. Any views expressed in the Communication are those
>>> of the individual sender only, unless expressly stated to be those of
>>> Murdoch Children’s Research Institute (MCRI) ABN 21 006 566 972 or any of
>>> its related entities. MCRI does not accept liability in connection with the
>>> integrity of or errors in the Communication, computer virus, data
>>> corruption, interference or delay arising from or in respect of the
>>> Communication.
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   %(web_page_url)s
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   %(web_page_url)s
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Re: Ansible Galaxy using root user for role installation

[Peter van Heusden]

Hi there

I found I had this problem too, when the `become: true` applied to the
whole Galaxy play. My current top level playbook (at
https://github.com/pvanheus/masters_galaxy_ansible/blob/master/galaxy.yml#L25)
now applies `become: true` as it is needed, and does not apply it to the
`galaxyproject.galaxy` role. I found that this avoids Ansible trying to log
in as the root user.

Peter

On Mon, 15 Feb 2021 at 07:21, Sandra Maksimovic <
sandra.maksimo...@mcri.edu.au> wrote:

> Hi there,
>
>
>
> Would anyone happen to know why Ansible tries to connect to a remote host
> (CentOS 7) using 'root' during Galaxy role installation?
>
>
>
> My playbook has been configured to connect to the remote host using a
> different user account via SSH, however, the role installation is not
> honoring this for some reason and continues to use root. The remote host
> has had SSH root login disabled for security reasons and setting various
> galaxy user/privsep/become/remote user variables in the group_vars has not
> had any effect.
>
>
>
> TASK [galaxyproject.galaxy : Create galaxy_root] *
>
> fatal: [server_name]: UNREACHABLE! => {"changed": false, "msg":
> "Invalid/incorrect password: Permission denied, please try
> again.\r\nPermission denied, please try again.\r\nroot@server_name:
> Permission denied (publickey,password).", "unreachable": true}
>
>
>
> Thanks,
>
> Sandra
>
>
> This e-mail and any attachments to it (the "Communication") are, unless
> otherwise stated, confidential, may contain copyright material and is for
> the use only of the intended recipient. If you receive the Communication in
> error, please notify the sender immediately by return e-mail, delete the
> Communication and the return e-mail, and do not read, copy, retransmit or
> otherwise deal with it. Any views expressed in the Communication are those
> of the individual sender only, unless expressly stated to be those of
> Murdoch Children’s Research Institute (MCRI) ABN 21 006 566 972 or any of
> its related entities. MCRI does not accept liability in connection with the
> integrity of or errors in the Communication, computer virus, data
> corruption, interference or delay arising from or in respect of the
> Communication.
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] _remove_dead_weakref error even after deleting .venv (Galaxy 18.09)

[Peter van Heusden]

I solved this problem by deleting the virtualenv in $GALAXY_ROOT and
recreating it from one of my cluster worker nodes. System python on the
worker nodes is 2.7.15rc1 from the Ubuntu 18.04 repository. System python
on the Galaxy server is 2.7.12 from the Ubuntu 16.04 repository. Since the
virtual environment used by Galaxy contains symbolic links to system python
modules, it seems there is a dependency between the virtual environment and
system python? I tried different methods for creating the virtual env but
nothing worked except for creating it on one of the cluster nodes. I look
forward to testing the virtualenv-in-conda method that Marius PRed in here (
https://github.com/galaxyproject/galaxy/pull/6975) in the future.

One final "gotcha" I discovered - creating the virtualenv with `bash
run.sh` does not install `watchdog`, a module which is necessary if
watching the tool directories for changes is enabled. In certain cases this
can lead to an exception.

Peter

On Tue, 11 Dec 2018 at 17:15, Peter van Heusden  wrote:

> Hi there
>
> I'm trying to run Windowmasker in Galaxy and receiving the following error:
>
> "computing the genome length
> pass 1
> pass 2
> optimizing the data structure
> Traceback (most recent call last):
>   File
> "/tools/software/galaxy/galaxy1/database/jobs/006/6476/set_metadata_bUlLkt.py",
> line 1, in 
> from galaxy_ext.metadata.set_metadata import set_metadata;
> set_metadata()
>   File
> "/tools/software/galaxy/galaxy1/lib/galaxy_ext/metadata/set_metadata.py",
> line 14, in 
> import logging
>   File "/usr/lib/python2.7/logging/__init__.py", line 26, in 
> import sys, os, time, cStringIO, traceback, warnings, weakref,
> collections
>   File "/usr/lib/python2.7/weakref.py", line 14, in 
> from _weakref import (
> ImportError: cannot import name _remove_dead_weakref"
>
> The usual suggested fix for this is to remove the Galaxy virtual
> environment (.venv) and recreated it. I have done that but the error
> persists. Galaxy is running here on Ubuntu 16.04 with Python 2.7.12. Any
> advice?
>
> Thanks!
> Peter
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] _remove_dead_weakref error even after deleting .venv (Galaxy 18.09)

[Peter van Heusden]

Hi there

I'm trying to run Windowmasker in Galaxy and receiving the following error:

"computing the genome length
pass 1
pass 2
optimizing the data structure
Traceback (most recent call last):
  File
"/tools/software/galaxy/galaxy1/database/jobs/006/6476/set_metadata_bUlLkt.py",
line 1, in 
from galaxy_ext.metadata.set_metadata import set_metadata;
set_metadata()
  File
"/tools/software/galaxy/galaxy1/lib/galaxy_ext/metadata/set_metadata.py",
line 14, in 
import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in 
import sys, os, time, cStringIO, traceback, warnings, weakref,
collections
  File "/usr/lib/python2.7/weakref.py", line 14, in 
from _weakref import (
ImportError: cannot import name _remove_dead_weakref"

The usual suggested fix for this is to remove the Galaxy virtual
environment (.venv) and recreated it. I have done that but the error
persists. Galaxy is running here on Ubuntu 16.04 with Python 2.7.12. Any
advice?

Thanks!
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Adding the Ilifu cloud to launch.usegalaxy.org

[Peter van Heusden]

Hi there

While I will continue to work on getting a Cloudlaunch server running
locally, could the Ilifu cloud (http://www.ilifu.ac.za/) be added as an
option on https://launch.usegalaxy.org/catalog ? What details would be
required?

And then - what are the requirements for adding appliances to the appliance
catalogue?

Thanks,
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Running Cloudlaunch against a new OpenStack cloud

[Peter van Heusden]

Hi there

I've got a bit further, but now I've got a problem - how do you access the
admin panel when you have the complete site set up?

1) If Debug = False (as it is for a production setup), Django doesn't load
static files (e.g. CSS).

2) When going via the nginx proxy, I can't figure out what URL will direct
me to the admin page.

Peter
P.S. I'm doing some work on the ansible to get this all set up - I'll do a
PR once everything seems to be working.

On Fri, 19 Oct 2018 at 01:25, Alexandru Mahmoud  wrote:

> Hello!
>
> I am new to the team, so this will be a brief response to only one of your
> questions while waiting for someone else to get to the email (and I am not
> replying all so that people don’t dismiss it as answered):
>
> For 3 specifically, you can access the admin panel at http://127.0.0.1(or
> localhost):8000/admin, where you should be able to login with the user
> created with the ‘createsuperuser’ command. Under “Appliances” or
> “Applications” (depending on the version of cloudlaunch) you’ll be able to
> see and change/add configurations for apps.
> If you want to see in more detail how the appliances are being set up and
> how the launch config is used in code, look for the "Backend component
> name", then find the corresponding backend plugin in the cloudlaunch repo.
> I hope that is a helpful start, and feel free to email me if nobody
> responds in a reasonable amount of time to your other questions, as I can
> get more answers in person and relay them to you.
>
> Best,
> Alex Mahmoud
>
>
> On Oct 18, 2018, at 3:45 PM, Peter van Heusden  wrote:
>
> Hi there
>
> I'd like to set up cloudlaunch to be able to set up on-demand Galaxy
> servers and associated clusters on the new Ilifu OpenStack cloud. From
> looking around the repositories it looks like Cloudlaunch is at least 2
> components:
>
> 1. The cloudlaunch server
> 2. The UI - either cloudlaunch-ui or cloudlaunch-cli
>
> I'm trying to get cloudlaunch server installed on a Ubuntu 18.04 VM. The
> only requirement seems to be python3-pip. For postgresql support I
> installed the python3-psycopg2 module, the postgresql-10 server and create
> a user and db (both named cloud). Then I installed the server with `pip3
> install cloudlaunch-server` and ran commands:
>
> cloudlaunch-server django migrate
> cloudlaunch-server django createsuperuser
> cloudlaunch-server django runserver
>
> The next step (https://github.com/galaxyproject/cloudlaunch) refers to
> redis-server. Installing this package seemed to satisfy that requirement,
> running it manually didn't work so I just ran the celery command as
> illustrated. And finally relaunched the django server.
>
> Then nodejs and npm were installed, the cloudlaunch-ui cloned from github,
> the various npm package specified installed and finally npm start brings up
> the cloudlaunch user interface.
>
> So far so good but:
>
> 1. Is there more extensive documentation anywhere? Specifically
> documentation about running this in production.
>
> 2. Is there a docker container that can be used instead of all this manual
> installation?
>
> 3. How are appliances configured and added to the Appliance Catalogue?
>
> Thanks!
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/
>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] regarding galaxy kickstart

[Peter van Heusden]

I don't have an answer to your 2. but Galaxy Kickstart sets up Galaxy
controlled by supervisord. So you can restart e.g. the Galaxy web front end
with:

sudo supervisorctl restart galaxy:galaxy_web

On Tue, 16 Oct 2018 at 08:16 Rui Wang  wrote:

> Hi Folks,
>
> I spent sometime playing with the kickstart, and sort of got it working.
> Still a couple things not clear:
>
> 1. Is it true that there is no start/stop? My guess is that whenever you
> need to restart, just re-run the playbook? Anything you want to update(for
> example add new tools), you re-run the updated playbook too, correct?
> Before you re-run it, do you have to manually kill everything?
>
> 2. Seems tools could easily added by editing the tool list. How about
> genome data? What if I want to add a specific repo or genome data? Is there
> a role for this purpose?
>
> Please give me a hand if you could. Any input will be greatly appreciated.
> :-)
>
> Thanks so much,
> Rui
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Running Cloudlaunch against a new OpenStack cloud

[Peter van Heusden]

Hi there

I'd like to set up cloudlaunch to be able to set up on-demand Galaxy
servers and associated clusters on the new Ilifu OpenStack cloud. From
looking around the repositories it looks like Cloudlaunch is at least 2
components:

1. The cloudlaunch server
2. The UI - either cloudlaunch-ui or cloudlaunch-cli

I'm trying to get cloudlaunch server installed on a Ubuntu 18.04 VM. The
only requirement seems to be python3-pip. For postgresql support I
installed the python3-psycopg2 module, the postgresql-10 server and create
a user and db (both named cloud). Then I installed the server with `pip3
install cloudlaunch-server` and ran commands:

cloudlaunch-server django migrate
cloudlaunch-server django createsuperuser
cloudlaunch-server django runserver

The next step (https://github.com/galaxyproject/cloudlaunch) refers to
redis-server. Installing this package seemed to satisfy that requirement,
running it manually didn't work so I just ran the celery command as
illustrated. And finally relaunched the django server.

Then nodejs and npm were installed, the cloudlaunch-ui cloned from github,
the various npm package specified installed and finally npm start brings up
the cloudlaunch user interface.

So far so good but:

1. Is there more extensive documentation anywhere? Specifically
documentation about running this in production.

2. Is there a docker container that can be used instead of all this manual
installation?

3. How are appliances configured and added to the Appliance Catalogue?

Thanks!
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Error when trying to update tool vcf_to_alignment in testtoolshed

[Peter van Heusden]

Hi there

I had a failed updated (network problem) of the sanbi-uwc tool
vcf_to_alignment and now when I try and do an update I get:

$ planemo shed_update --shed_target testtoolshed
cd '/home/pvh/Documents/code/SANBI/tools-sanbi-uwc/tools/vcf_to_alignment'
&& git rev-parse HEAD
cd '/home/pvh/Documents/code/SANBI/tools-sanbi-uwc/tools/vcf_to_alignment'
&& git diff --quiet
Could not update vcf_to_alignment
('Connection aborted.', RemoteDisconnected('Remote end closed connection
without response',))
Repository metadata updated successfully for repository 'vcf_to_alignment'
on the test Tool Shed.
Failed to update repository contents for repository 'vcf_to_alignment' on
the test Tool Shed.

Nothing that I try on my side seems to resolve this. Any ideas? Is the
problem on the toolshed side?

Thanks,
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Using Singularity images to resolve dependences in Galaxy 18.05

[Peter van Heusden]

Thanks Björn!

So it is a job_conf thing rather than a dependency manager config thing?

>From the job_conf.xml example I am not clear how it expects to find the
container images? How is the search path configured, what naming scheme is
used for containers? And does it "pull" containers automatically (like the
conda dependency manager does) or should I pre-populate the system with
container images?

Thanks,
Peter

On Wed, 19 Sep 2018 at 10:15 Björn Grüning 
wrote:

> Hi Peter,
>
> yes this is working :)
> Usegalaxy.eu is using it for a few tools.
>
> Documentation is here ;)
>
> https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced#L484
>
> Also the Docker compose image is can also use Singularity if you want to
> reverse engineer it :)
>
> Ciao,
> Bjoern
>
> Am 19.09.2018 um 10:12 schrieb Peter van Heusden:
> > Hi there
> >
> > Is it possible to use Singularity images to resolve dependencies in
> > Galaxy 18.05? If so is there documentation somewhere?
> >
> > I have a Galaxy server where for at least some of the dependencies conda
> > will not be a good option so I would like to be able to use singularity
> > if at all possible.
> >
> > Thanks!
> > Peter
> >
> >
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> >http://galaxyproject.org/search/
> >
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Using Singularity images to resolve dependences in Galaxy 18.05

[Peter van Heusden]

Hi there

Is it possible to use Singularity images to resolve dependencies in Galaxy
18.05? If so is there documentation somewhere?

I have a Galaxy server where for at least some of the dependencies conda
will not be a good option so I would like to be able to use singularity if
at all possible.

Thanks!
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"

[Peter van Heusden]

Yes that was my conclusion, thanks Peter.

As a result I switched to a new table named (shapeit2_ref) and a new tool
name (data_manager_shapeit2_reference) and from planemo side upload seemed
to succeed but the tool is in an error state. I resorted to manually
editing the tool-data table.

Data managers are a bit of an ugly duckling part of Galaxy - not receiving
as much love as other kinds of tools.

Peter


On Tue, 18 Sep 2018 at 14:22 Peter Cock  wrote:

> Hi Peter vH,
>
> I wonder if adding columns is not supported - because some tools
> would define the table without the extra column? You might have
> to define a new data table instead (with largely the same content).
> But to be clear, I am not an expert in this area of Galaxy.
>
> However, it does seem the IUC could lay out some *.loc file best
> practises for tool authors. There is currently a TODO entry here:
>
>
> https://galaxy-iuc-standards.readthedocs.io/en/latest/best_practices/tool_xml.html#data-managers
>
> Peter C.
>
> On Mon, Sep 17, 2018 at 10:56 PM Peter van Heusden 
> wrote:
>
>> A bit more tracing leads me to suspect that what is happening is that a
>> merge is being called here:
>>
>>
>> https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/data/__init__.py#L120
>>
>> from:
>>
>>
>> https://github.com/galaxyproject/galaxy/blob/dev/lib/tool_shed/tools/data_table_manager.py#L88
>>
>> i.e. the toolshed maintains an in-memory copy of tool-data tables and if
>> the list of columns in an incoming tool-data table is not the same as the
>> in-memory version the error I saw happens.
>>
>> Which leaves me with the question: how is one meant to update the column
>> list of a tool data table?
>>
>> Peter
>>
>>
>>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"

[Peter van Heusden]

A bit more tracing leads me to suspect that what is happening is that a
merge is being called here:

https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/data/__init__.py#L120

from:

https://github.com/galaxyproject/galaxy/blob/dev/lib/tool_shed/tools/data_table_manager.py#L88

i.e. the toolshed maintains an in-memory copy of tool-data tables and if
the list of columns in an incoming tool-data table is not the same as the
in-memory version the error I saw happens.

Which leaves me with the question: how is one meant to update the column
list of a tool data table?

Peter

On Mon, 17 Sep 2018 at 13:23 Peter van Heusden  wrote:

> Thanks - which is probably because of the previous failures with
> `shed_update` - which relate back to the topic of my original email. I
> suspect that failure from `planemo shed_update` has led to this "not
> installable" state. And I don't know how to solve the problem from
> `shed_update` which *looks* like it is coming from a clash between previous
> and current definitions of the tool-data table.
>
> Peter
>
> On Mon, 17 Sep 2018 at 12:58 Nicola Soranzo 
> wrote:
>
>> Hi Peter,
>> if you open the page for data_manager_shapeit_reference in the test
>> ToolShed you will see that revision 5 and 6 are not installable, they both
>> have "*This revision can be installed:* False" . If you instead select
>> revision 3 it is installable.
>>
>> Cheers,
>> Nicola
>>
>>
>> On 17/09/18 11:51, Peter van Heusden wrote:
>>
>> I'm following up on my previous email. I somehow managed to get `planemo
>> shed_update` to work but now I seem to have an inconsistent view of the
>> Test toolshed. This is how it looks from the Admin panel of my Galaxy
>> server:
>>
>> [image: https://files.gitter.im/galaxyproject/Lobby/k2qH/image.png]
>> Notice that it shows 3 installable revisions for the data manager, last
>> updated 13th of September. And this is how it looks from the Test toolshed
>> website:
>>
>> [image: https://files.gitter.im/galaxyproject/Lobby/ymY9/image.png]
>>
>>
>> Notice that it has 6 installabale revisions for the data manager. Is my
>> Galaxy server perhaps caching something?
>>
>> Peter
>>
>>
>> On Sat, 15 Sep 2018 at 09:57 Peter van Heusden  wrote:
>>
>>> Hi there
>>>
>>> I'm working on a data manager to keep track of the reference data
>>> collection used by the SHAPEIT2 tool. I had to add a column to the data
>>> manager but when I try and do a `planemo shed_update` I get the error:
>>>
>>> Unexpected HTTP status code: 500: {"err_msg": "Error attempting to parse
>>> file tool_data_table_conf.xml.sample: Merging tabular data tables with non
>>> matching columns is not allowed: shapeit_ref:{'path': 2,
>>> 'reference_prefix': 3, 'name': 1, 'value': 0, 'map_prefix': 4} !=
>>> shapeit_ref:{'name': 1, 'value': 0, 'sample_prefix': 5, 'path': 2,
>>> 'reference_prefix': 3, 'map_prefix': 4}"}
>>>
>>> This is for data_manager_shapeit_reference in testtoolshed, which is on
>>> github at
>>>
>>> https://github.com/pvanheus/data_manager_shapeit_reference
>>>
>>> As far as I can tell the columns do accord. Is there any way to see
>>> which files are being merged to help try and debug this error?
>>>
>>> Thanks!
>>> Peter
>>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/
>>
>>
>>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"

[Peter van Heusden]

Thanks - which is probably because of the previous failures with
`shed_update` - which relate back to the topic of my original email. I
suspect that failure from `planemo shed_update` has led to this "not
installable" state. And I don't know how to solve the problem from
`shed_update` which *looks* like it is coming from a clash between previous
and current definitions of the tool-data table.

Peter

On Mon, 17 Sep 2018 at 12:58 Nicola Soranzo 
wrote:

> Hi Peter,
> if you open the page for data_manager_shapeit_reference in the test
> ToolShed you will see that revision 5 and 6 are not installable, they both
> have "*This revision can be installed:* False" . If you instead select
> revision 3 it is installable.
>
> Cheers,
> Nicola
>
>
> On 17/09/18 11:51, Peter van Heusden wrote:
>
> I'm following up on my previous email. I somehow managed to get `planemo
> shed_update` to work but now I seem to have an inconsistent view of the
> Test toolshed. This is how it looks from the Admin panel of my Galaxy
> server:
>
> [image: https://files.gitter.im/galaxyproject/Lobby/k2qH/image.png]
> Notice that it shows 3 installable revisions for the data manager, last
> updated 13th of September. And this is how it looks from the Test toolshed
> website:
>
> [image: https://files.gitter.im/galaxyproject/Lobby/ymY9/image.png]
>
>
> Notice that it has 6 installabale revisions for the data manager. Is my
> Galaxy server perhaps caching something?
>
> Peter
>
>
> On Sat, 15 Sep 2018 at 09:57 Peter van Heusden  wrote:
>
>> Hi there
>>
>> I'm working on a data manager to keep track of the reference data
>> collection used by the SHAPEIT2 tool. I had to add a column to the data
>> manager but when I try and do a `planemo shed_update` I get the error:
>>
>> Unexpected HTTP status code: 500: {"err_msg": "Error attempting to parse
>> file tool_data_table_conf.xml.sample: Merging tabular data tables with non
>> matching columns is not allowed: shapeit_ref:{'path': 2,
>> 'reference_prefix': 3, 'name': 1, 'value': 0, 'map_prefix': 4} !=
>> shapeit_ref:{'name': 1, 'value': 0, 'sample_prefix': 5, 'path': 2,
>> 'reference_prefix': 3, 'map_prefix': 4}"}
>>
>> This is for data_manager_shapeit_reference in testtoolshed, which is on
>> github at
>>
>> https://github.com/pvanheus/data_manager_shapeit_reference
>>
>> As far as I can tell the columns do accord. Is there any way to see which
>> files are being merged to help try and debug this error?
>>
>> Thanks!
>> Peter
>>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"

[Peter van Heusden]

I'm following up on my previous email. I somehow managed to get `planemo
shed_update` to work but now I seem to have an inconsistent view of the
Test toolshed. This is how it looks from the Admin panel of my Galaxy
server:

[image: https://files.gitter.im/galaxyproject/Lobby/k2qH/image.png]
Notice that it shows 3 installable revisions for the data manager, last
updated 13th of September. And this is how it looks from the Test toolshed
website:

[image: https://files.gitter.im/galaxyproject/Lobby/ymY9/image.png]


Notice that it has 6 installabale revisions for the data manager. Is my
Galaxy server perhaps caching something?

Peter


On Sat, 15 Sep 2018 at 09:57 Peter van Heusden  wrote:

> Hi there
>
> I'm working on a data manager to keep track of the reference data
> collection used by the SHAPEIT2 tool. I had to add a column to the data
> manager but when I try and do a `planemo shed_update` I get the error:
>
> Unexpected HTTP status code: 500: {"err_msg": "Error attempting to parse
> file tool_data_table_conf.xml.sample: Merging tabular data tables with non
> matching columns is not allowed: shapeit_ref:{'path': 2,
> 'reference_prefix': 3, 'name': 1, 'value': 0, 'map_prefix': 4} !=
> shapeit_ref:{'name': 1, 'value': 0, 'sample_prefix': 5, 'path': 2,
> 'reference_prefix': 3, 'map_prefix': 4}"}
>
> This is for data_manager_shapeit_reference in testtoolshed, which is on
> github at
>
> https://github.com/pvanheus/data_manager_shapeit_reference
>
> As far as I can tell the columns do accord. Is there any way to see which
> files are being merged to help try and debug this error?
>
> Thanks!
> Peter
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Problems installing Galaxy 18.01

[Peter van Heusden]

Thanks to Marius and others, I now have a working Galaxy server, including
CVMFS support, as documented in the complete blog post:
http://pvh.wp.sanbi.ac.za/2018/03/18/galaxy-18-01-install/

The problem I was encountering with installations was related to incorrect
environment configuration, which led to the `hg` binary not being found in
the PATH and thus the install failing. Another fault led to this failure
being a silent failure.

I've linked to the working configuration in the blog, but here is a direct
link:

https://gist.github.com/pvanheus/589f84507a26ffa49e3bdf3bd938f25c

It is rather too unwieldy to paste here.

Note that while the says [program:web], there is only one section in the
supervisord configuration. The server is running in a uWSGI + Mules
configuration.

Thanks!
Peter



On Mon, 19 Mar 2018 at 11:17 Peter van Heusden <p...@sanbi.ac.za> wrote:

> First to answer your questions:
>
> I'm on release_18.01 branch,
> commit c58ca749c26bbb3c4ca8750f2c37fd1bb6bd7999.
>
> Thanks for the note about nginx-extras, I've installed that and it fixes
> the upload
> directives problem.
>
> Then on to the main problem. I used "Get Data" as the default location
> because I'm really just testing the install. The install goes from "New"
> status to "Error" status pretty much immediately:
>
> [image: image.png]
>
> This is not a bowtie2-specific problem, I get the same behaviour when
> trying to install fastqc.
>
> Thanks for your reply,
> Peter
> P.S. running already-installed tools works
>
> On Mon, 19 Mar 2018 at 09:44 Marius van den Beek <m.vandenb...@gmail.com>
> wrote:
>
>> Hi Peter,
>>
>> does the tool show up in the tool panel (Looks like you've installed it
>> into "Get Data")?
>> What commit are you on ? There were some last minute bugfixes to the
>> release_18.01
>> prior to the release, so if the checkout is a few days old an upgrade
>> (and an uninstall/install cycle) might help.
>>
>> Also you mentioned the lack of the upload_store in nginx in nginx, you
>> need to install nginx-extras, not nginx.
>>
>> Hope that helps,
>> Marius
>>
>> On 18 March 2018 at 21:17, Peter van Heusden <p...@sanbi.ac.za> wrote:
>>
>>> Hi there
>>>
>>> My newly installed Galaxy 18.01 server seems to be unable to install
>>> bowtie (the devteam version). I'm documenting my installation progress at:
>>>
>>> http://pvh.wp.sanbi.ac.za/2018/03/18/galaxy-18-01-install/
>>>
>>> As noted in the last paragraph:
>>>
>>> "My next test was to try and install the bowtie2 tool from the toolshed.
>>> This failed, with the tool status going from New to Error with not much
>>> interesting left in the logs."
>>>
>>> The log during the time when I'm trying to install bowtie2 is:
>>>
>>>
>>>1. [pid: 8856|app: 0|req: 35/48] 10.8.0.2 () {48 vars in 1108 bytes}
>>>[Sun Mar 18 19:14:55 2018] GET
>>>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install 
>>> =>
>>>generated 650 bytes in 26 msecs (HTTP/1.1 302) 3headers in 269 bytes
>>>(1 switches on core 1)
>>>2. 10.8.0.2 - - [18/Mar/2018:19:14:57 +] "GET
>>>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install
>>>HTTP/1.1" 302 - "
>>>
>>> https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b;
>>>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like
>>>Gecko) Chrome/65.0.3325.146 Safari/537.36"
>>>3. [pid: 8856|app: 0|req: 37/49] 10.8.0.2 () {48 vars in 1108 bytes}
>>>[Sun Mar 18 19:14:57 2018] GET
>>>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install 
>>> =>
>>>generated 650 bytes in 24 msecs (HTTP/1.1 302) 3headers in 269 bytes
>>>(1 switches on core 0)
>>>4. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,375 [p:8856,w:2,m:0] [
>>>uWSGIWorker2Core2] Dependency bowtie2 not found.
>>>5. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,376 [p:8856,w:2,m:0] [
>>>uWSGIWorker2Core2] Dependency samtools not found.
>>>6. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [
>>>uWSGIWorker2Core3] Dependency bowtie2 not found.
>>>7. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [
>>>uWSGIWorker2Core3] Dependency samtools not found.
>>>8. 10.8.0.2 - - [18/Mar/2018:19:14:55 +] "GET
>>>
>>> /admin_toolshed/prepare_for_i

Re: [galaxy-dev] Problems installing Galaxy 18.01

[Peter van Heusden]

First to answer your questions:

I'm on release_18.01 branch,
commit c58ca749c26bbb3c4ca8750f2c37fd1bb6bd7999.

Thanks for the note about nginx-extras, I've installed that and it fixes
the upload
directives problem.

Then on to the main problem. I used "Get Data" as the default location
because I'm really just testing the install. The install goes from "New"
status to "Error" status pretty much immediately:

[image: image.png]

This is not a bowtie2-specific problem, I get the same behaviour when
trying to install fastqc.

Thanks for your reply,
Peter
P.S. running already-installed tools works

On Mon, 19 Mar 2018 at 09:44 Marius van den Beek <m.vandenb...@gmail.com>
wrote:

> Hi Peter,
>
> does the tool show up in the tool panel (Looks like you've installed it
> into "Get Data")?
> What commit are you on ? There were some last minute bugfixes to the
> release_18.01
> prior to the release, so if the checkout is a few days old an upgrade (and
> an uninstall/install cycle) might help.
>
> Also you mentioned the lack of the upload_store in nginx in nginx, you
> need to install nginx-extras, not nginx.
>
> Hope that helps,
> Marius
>
> On 18 March 2018 at 21:17, Peter van Heusden <p...@sanbi.ac.za> wrote:
>
>> Hi there
>>
>> My newly installed Galaxy 18.01 server seems to be unable to install
>> bowtie (the devteam version). I'm documenting my installation progress at:
>>
>> http://pvh.wp.sanbi.ac.za/2018/03/18/galaxy-18-01-install/
>>
>> As noted in the last paragraph:
>>
>> "My next test was to try and install the bowtie2 tool from the toolshed.
>> This failed, with the tool status going from New to Error with not much
>> interesting left in the logs."
>>
>> The log during the time when I'm trying to install bowtie2 is:
>>
>>
>>1. [pid: 8856|app: 0|req: 35/48] 10.8.0.2 () {48 vars in 1108 bytes} [Sun
>>Mar 18 19:14:55 2018] GET
>>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install =>
>>generated 650 bytes in 26 msecs (HTTP/1.1 302) 3headers in 269 bytes (
>>1 switches on core 1)
>>2. 10.8.0.2 - - [18/Mar/2018:19:14:57 +] "GET
>>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install
>>HTTP/1.1" 302 - "
>>
>> https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b;
>>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko
>>) Chrome/65.0.3325.146 Safari/537.36"
>>3. [pid: 8856|app: 0|req: 37/49] 10.8.0.2 () {48 vars in 1108 bytes} [Sun
>>Mar 18 19:14:57 2018] GET
>>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install =>
>>generated 650 bytes in 24 msecs (HTTP/1.1 302) 3headers in 269 bytes (
>>1 switches on core 0)
>>4. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,375 [p:8856,w:2,m:0] [
>>uWSGIWorker2Core2] Dependency bowtie2 not found.
>>5. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,376 [p:8856,w:2,m:0] [
>>uWSGIWorker2Core2] Dependency samtools not found.
>>6. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [
>>uWSGIWorker2Core3] Dependency bowtie2 not found.
>>7. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [
>>uWSGIWorker2Core3] Dependency samtools not found.
>>8. 10.8.0.2 - - [18/Mar/2018:19:14:55 +] "GET
>>
>> /admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F%
>>2Ftoolshed.g2.bx.psu.edu%2F HTTP/1.1" 200 - "
>>
>> https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b;
>>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko
>>) Chrome/65.0.3325.146 Safari/537.36"
>>9. [pid: 8856|app: 0|req: 38/50] 10.8.0.2 () {48 vars in 1274 bytes} [Sun
>>Mar 18 19:14:55 2018] GET
>>
>> /admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F%
>>2Ftoolshed.g2.bx.psu.edu%2F => generated 19062 bytes in 7001 msecs (
>>HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 2)
>>10. 10.8.0.2 - - [18/Mar/2018:19:14:57 +] "GET
>>
>> /admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F%
>>2Ftoolshed.g2.bx.psu.edu%2F HTTP/1.1" 200 - "
>>
>> https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b;
>>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko
&

[galaxy-dev] Problems installing Galaxy 18.01

[Peter van Heusden]

Hi there

My newly installed Galaxy 18.01 server seems to be unable to install bowtie
(the devteam version). I'm documenting my installation progress at:

http://pvh.wp.sanbi.ac.za/2018/03/18/galaxy-18-01-install/

As noted in the last paragraph:

"My next test was to try and install the bowtie2 tool from the toolshed.
This failed, with the tool status going from New to Error with not much
interesting left in the logs."

The log during the time when I'm trying to install bowtie2 is:


   1. [pid: 8856|app: 0|req: 35/48] 10.8.0.2 () {48 vars in 1108 bytes} [Sun
   Mar 18 19:14:55 2018] GET
   /admin_toolshed/manage_repository?id=f597429621d6eb2b=install =>
   generated 650 bytes in 26 msecs (HTTP/1.1 302) 3headers in 269
bytes (1 switches
   on core 1)
   2. 10.8.0.2 - - [18/Mar/2018:19:14:57 +] "GET
   /admin_toolshed/manage_repository?id=f597429621d6eb2b=install
   HTTP/1.1" 302 - "
   
https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b;
   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/65.0.3325.146 Safari/537.36"
   3. [pid: 8856|app: 0|req: 37/49] 10.8.0.2 () {48 vars in 1108 bytes} [Sun
   Mar 18 19:14:57 2018] GET
   /admin_toolshed/manage_repository?id=f597429621d6eb2b=install =>
   generated 650 bytes in 24 msecs (HTTP/1.1 302) 3headers in 269
bytes (1 switches
   on core 0)
   4. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,375 [p:8856,w:2,m:0] [
   uWSGIWorker2Core2] Dependency bowtie2 not found.
   5. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,376 [p:8856,w:2,m:0] [
   uWSGIWorker2Core2] Dependency samtools not found.
   6. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [
   uWSGIWorker2Core3] Dependency bowtie2 not found.
   7. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [
   uWSGIWorker2Core3] Dependency samtools not found.
   8. 10.8.0.2 - - [18/Mar/2018:19:14:55 +] "GET
   
/admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F%
   2Ftoolshed.g2.bx.psu.edu%2F HTTP/1.1" 200 - "
   
https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b;
   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/65.0.3325.146 Safari/537.36"
   9. [pid: 8856|app: 0|req: 38/50] 10.8.0.2 () {48 vars in 1274 bytes} [Sun
   Mar 18 19:14:55 2018] GET
   
/admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F%
   2Ftoolshed.g2.bx.psu.edu%2F => generated 19062 bytes in 7001 msecs (HTTP/
   1.1 200) 2 headers in 73 bytes (1 switches on core 2)
   10. 10.8.0.2 - - [18/Mar/2018:19:14:57 +] "GET
   
/admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F%
   2Ftoolshed.g2.bx.psu.edu%2F HTTP/1.1" 200 - "
   
https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b;
   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/65.0.3325.146 Safari/537.36"
   11. [pid: 8856|app: 0|req: 38/51] 10.8.0.2 () {48 vars in 1274 bytes} [Sun
   Mar 18 19:14:57 2018] GET
   
/admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F%
   2Ftoolshed.g2.bx.psu.edu%2F => generated 19062 bytes in 5987 msecs (HTTP/
   1.1 200) 2 headers in 73 bytes (1 switches on core 3)
   12. galaxy.tools.toolbox.base DEBUG 2018-03-18 19:15:14,128 [p:8846,w:1
   ,m:0] [uWSGIWorker1Core1] Appending to tool panel section: Get Data
   13. galaxy.tools.toolbox.base DEBUG 2018-03-18 19:15:14,143 [p:8846,w:1
   ,m:0] [uWSGIWorker1Core1] Appending to tool panel section: Get Data
   14. 10.8.0.2 - - [18/Mar/2018:19:15:11 +] "POST
   /admin_toolshed/prepare_for_install HTTP/1.1" 200 - "
   
https://galaxy.sanbi.ac.za/admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F;
   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/65.0.3325.146Safari/537.36"
   15. [pid: 8846|app: 0|req: 14/52] 10.8.0.2 () {56 vars in 1329 bytes} [Sun
   Mar 18 19:15:11 2018] POST /admin_toolshed/prepare_for_install =>
   generated 11133 bytes in 2509 msecs (HTTP/1.1 200) 2 headers in 73 bytes
   (1 switches on core 1)
   16. galaxy.tools.toolbox.base DEBUG 2018-03-18 19:15:15,296 [p:8856,w:2
   ,m:0] [uWSGIWorker2Core1] Appending to tool panel section: Get Data
   17. 10.8.0.2 - - [18/Mar/2018:19:15:15 +] "POST
   /admin_toolshed/install_repositories HTTP/1.1" 200 - "
   https://galaxy.sanbi.ac.za/admin_toolshed/prepare_for_install; "Mozilla/
   5.0 (X11; Linux x86_64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/65.0
   .3325.146 Safari/537.36"
   18. [pid: 8856|app: 0|req: 39/53] 10.8.0.2 () {54 vars in 1152 bytes} [Sun
   Mar 18 19:15:15 2018] POST /admin_toolshed/install_repositories =>
   generated 20 bytes in 206 msecs (HTTP/1.1 

Re: [galaxy-dev] Improved DRMAAJobRunner (Peter van Heusden)

[Peter van Heusden]

Hi Matthias

My job_conf.xml still has the old config of running upload locally, so it
has a section





after the  section. Are you using a proxy such as nginx with
your Galaxy server?

Thanks,
Peter

On Mon, 10 Jul 2017 at 11:50 Matthias Bernt <m.be...@ufz.de> wrote:

> Hi Peter,
>
> FYI: just found on the galaxy documentation a paragraph that even
> recommends running upload on the cluster:
>
> https://galaxyproject.org/admin/config/performance/cluster/
>
> """
> If your cluster nodes have Internet access (NAT is okay) and you want to
> run the data source tools (upload, ucsc, etc.) on the cluster (doing so
> is highly recommended), set new_file_path in galaxy.ini to a directory
> somewhere in your shared filesystem:
> """
>
> Cheers,
> Matthias
>
>
> On 10.07.2017 11:11, Matthias Bernt wrote:
> > Dear Peter,
> >
> > my job_conf.xml is attached.
> >
> >
> > Best,
> > Matthuas
> >
> > On 08.07.2017 18:00, galaxy-dev-requ...@lists.galaxyproject.org wrote:
> >> Send galaxy-dev mailing list submissions to
> >> galaxy-dev@lists.galaxyproject.org
> >>
> >> To subscribe or unsubscribe via the World Wide Web, visit
> >> https://lists.galaxyproject.org/listinfo/galaxy-dev
> >> or, via email, send a message with subject or body 'help' to
> >> galaxy-dev-requ...@lists.galaxyproject.org
> >>
> >> You can reach the person managing the list at
> >> galaxy-dev-ow...@lists.galaxyproject.org
> >>
> >> When replying, please edit your Subject line so it is more specific
> >> than "Re: Contents of galaxy-dev digest..."
> >>
> >>
> >> HEY!  This is important!  If you reply to a thread in a digest, please
> >> 1. Change the subject of your response from "Galaxy-dev Digest Vol
> >> ..." to the original subject for the thread.
> >> 2. Strip out everything else in the digest that is not part of the
> >> thread you are responding to.
> >>
> >> Why?
> >> 1. This will keep the subject meaningful.  People will have some idea
> >> from the subject line if they should read it or not.
> >> 2. Not doing this greatly increases the number of emails that match
> >> search queries, but that aren't actually informative.
> >>
> >> Today's Topics:
> >>
> >> 1. Re: any introduce for how Mako and JavaScript work ingalaxy?
> >>(Daniel Blankenberg)
> >> 2. Re: ustacks - bad interpreter (Björn Grüning)
> >> 3. Re: Improved DRMAAJobRunner (Peter van Heusden) (Matthias Bernt)
> >> 4. Re: Improved DRMAAJobRunner (Peter van Heusden)
> >>(Peter van Heusden)
> >>
> >>
> >> --
> >>
> >> Message: 1
> >> Date: Thu, 6 Jul 2017 22:36:27 -0400
> >> From: Daniel Blankenberg <dan.blankenb...@gmail.com>
> >> To: Steven Shen <ishenwei...@gmail.com>
> >> Cc: galaxy-dev@lists.galaxyproject.org
> >> Subject: Re: [galaxy-dev] any introduce for how Mako and JavaScript
> >> work ingalaxy?
> >> Message-ID:
> >> <CAE9nRFg+Edj2=3+K=6+FZCxj5=hb4togw+lm++e+sd1+j2+...@mail.gmail.com
> >
> >> Content-Type: text/plain; charset="utf-8"
> >>
> >> Hi Steven,
> >>
> >> For a quick first pass, have you taken a look at https://github.com/
> >> galaxyproject/galaxy/blob/dev/client/README.md?
> >>
> >>
> >> Thanks for using Galaxy,
> >>
> >> Dan
> >>
> >> On Thu, Jul 6, 2017 at 10:11 PM, Steven Shen <ishenwei...@gmail.com>
> >> wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> I want to make some change for my local galaxy web, actually I can edit
> >>> galaxy_path/static/scripts/bundled/*.js directly.
> >>>
> >>> It seems mako templates are used for galaxy, as well as many JavaScript
> >>> files, but I don't know how they work. So is there any introduce for
> how
> >>> mako and JavaScript files working in galaxy?
> >>>
> >>> Thank you very much.
> >>>
> >>> Steven
> >>>
> >>> ___
> >>> Please keep all replies on the list by using "reply all"
> >>> in your mail client.  To manage your subscriptions to this
> >>> and other Galax

Re: [galaxy-dev] Improved DRMAAJobRunner (Peter van Heusden)

[Peter van Heusden]

On Sat, 8 Jul 2017 at 11:56 Matthias Bernt  wrote:

> Hi Peter,
>
> > Code looks interesting. This is for Univa Grid Engine?
>
> In its current state its tailored for the Univa Grid Engine. But the drmaa
> library code that is based on job_info() and wait() should run on all grid
> engines.
> For the command line based code changes might be necessary for the
> different grid engines. (Currently there is only a small bug, because
> wait() is currently
> called twice for finished jobs: in the repeated polling and then in the
> final check. The second call won't work which causes a fallback to qacct. I
> will fix this
> soon.)
> I guess that the output of qstat and qacct will be different. But I think
> this can be configured one just needs a way to get the info which grid
> engine is running.
>
> > In terms of the upload jobs, are those not designed to be run as 'local'
> jobs and not with the 'real user' setting?
>
> Sounds reasonable, but this is not what is happening on my installation of
> galaxy. Any idea where I could start to find the problem.
>
> I'm not an expert on this, but what does your job_conf.xml look like?

Peter

> Best,
> Matthias ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Improved DRMAAJobRunner

[Peter van Heusden]

Code looks interesting. This is for Univa Grid Engine?

In terms of the upload jobs, are those not designed to be run as 'local'
jobs and not with the 'real user' setting?

On Fri, 7 Jul 2017 at 17:32 Matthias Bernt  wrote:

> Dear list,
>
> I have implemented a new DRMAA job runner allows to detect runtime and
> memory violations. This is done by using the drmaa library functions
> job_info + wait or as fallback the commandline tools qstat + qacct. This
> has the additional advantage that the runner works also in setups using
> the external runner scripts to start jobs as real user (the original
> DRMAAJobRunner can not query these jobs at all).
>
> You may have a look here:
>
> https://github.com/galaxyproject/galaxy/pull/4275
>
> I have successfully tested the but settings: galaxy user/real user
> submits the jobs. Also the resubmission in case of memory and time
> violations seems to work.
>
> Would be great to get some comments.
>
> One problem that I encountered is that upload jobs do not work in the
> real user setting (for the original and the new runner): The permissions
> of the uploaded file in
> /gpfs1/data/galaxy_server/galaxy-dev/database/tmp/ are not changed. Any
> idea what needs to be changed to get this running?
>
> Best,
> Matthias
>
> --
>
> ---
> Matthias Bernt
> Bioinformatics Service
> Molekulare Systembiologie (MOLSYB)
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> Phone +49 341 235 482296 <+49%20341%20235482296>,
> m.be...@ufz.de, www.ufz.de
>
> Sitz der Gesellschaft/Registered Office: Leipzig
> Registergericht/Registration Office: Amtsgericht Leipzig
> Handelsregister Nr./Trade Register Nr.: B 4703
> Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
> MinDirig Wilfried Kraus
> Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
> Prof. Dr. Dr. h.c. Georg Teutsch
> Administrative Geschäftsführerin/ Administrative Managing Director:
> Prof. Dr. Heike Graßmann
> ---
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] "Best practice" for maintaining genome list in Galaxy

[Peter van Heusden]

Hi there

Is there current "best practice" for maintaining a collection of reference
genome data in a Galaxy server? I.e. filling in the dbkeys.loc and
all_fasta.loc table and associated data. I know of the following
discussions on this topic:

1. The data integration page:
https://galaxyproject.org/admin/data-integration/
2. The data managers page:
https://galaxyproject.org/admin/tools/data-managers/
3. This biostars answer: https://biostar.usegalaxy.org/p/7176/
4. This page on custom genomes that seems to be user / history oriented:
https://galaxyproject.org/learn/custom-genomes/
5. This page on rsyncing from the Galaxy reference data collection:
https://galaxyproject.org/admin/use-galaxy-rsync/ - quick test gives me
rsync error though
6. This guide to using the human reference data:
https://biostar.usegalaxy.org/p/14777/
7. I hear the usegalaxy.org reference collection is available via cvmfs - I
think this was discussed in a Galaxy Admins meetup at some point.

None of this is comprehensive. There are 2 sets of questions for me (in my
role as Galaxy admin):

1. If I want to make reference genomes and common indices (like for HISAT
and BWA) available for e.g. human and mouse, what is the best way to do
this?

2. If I want to add a genome for a non-model organism (e.g. M. tuberculosis
or L. calcarifer) available, what is the best way to do this? Which data
manager should or could I use?

Thanks,
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] composite datatypes: uploading and optional files

[Peter van Heusden]

Hi there

We've got a composite datatype (neostore) for Neo4j databases. Currently
the upload is disabled because to upload you need to select each individual
file of the database directory (which is tedious). Is there a more
efficient way to upload composite datatypes? E.g. as a tar.gz or zip file?
If not, has anyone looked into supporting this? (I see @jmchilton was noted
as working on a composite uploading for 17.05)

Secondly, we're moving to support Neo4j 3.2.0. In this version there is a
minor (1 file) change in the files used to store the database. Can
composite datatypes have "optional files"?

Thanks,
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Perl module problem with vcftools in conda + new tool

[Peter van Heusden]

Hi there

I'm trying to add a wrapper for the vcftools vcf-consensus tool. There is
an existing tool collection for vcftools in tools-devteam, so I'm add it
there. Wrapper is at:

https://github.com/pvanheus/tools-devteam/tree/master/tool_collections/vcftools/vcftools_consensus

I tested this using:

planemo conda_install .
planemo serve --conda_prefix /home/pvh/miniconda2  --con
da_dependency_resolution

When the tool executes I get the error:

  Fatal error: Exit code 2 () Can't locate Vcf.pm in @INC

Which implies that the PERL5LIB is not being set correctly when the tool
runs.

Using planemo test gives me:

No tests were executed - see Galaxy output for details.

Strangely enough on the pull request the Travis test passes (
https://github.com/galaxyproject/tools-devteam/pull/402).

Any idea how to fix this?

Thanks!
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Loading a GUI on galaxy

[Peter van Heusden]

Hi

Galaxy supports visualisation plugins and then it also supports interactive
environments. I don't know how WEKA VisualizaPanel works - is it a web
application? Does it require a server?

On Fri, 26 Aug 2016 at 20:52 Lijo John  wrote:

> Hello All,
> I want to know, whether it is possible to embed graphical user interface
> like that of WEKA VisualizePanel tool in galaxy ?
> I am able to load  the GUI on a local instance of galaxy. But when i am
> trying to access the galaxy server through a remote connection, this GUI is
> not able to load on remote desktop.
>
> Any comments or suggestions would be of great interest !
> --
> Regards,
> Lijo John
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Travis CI for Galaxy repositories?

[Peter van Heusden]

Hi everyone

We maintain a tools collection on Github for tools that are not yet (or
will never be) added to tools-iuc:
https://github.com/SANBI-SA/tools-sanbi-uwc

I tried copying the .travis.yml from tools-iuc to our repository, and it
largely works, except for the planemo upload part, which fails with this
error:

Repository created
cd '/home/pvh/Documents/code/SANBI/pilon/tools-sanbi-uwc/tools/pilon' &&
git rev-parse HEAD
cd '/home/pvh/Documents/code/SANBI/pilon/tools-sanbi-uwc/tools/pilon' &&
git diff --quiet
Could not update pilon
Unexpected response from galaxy: 500: {"content_alert": "", "err_msg":
"Metadata may have been defined for some items in revision 'f2477672a950'.
Correct the following problems if necessary and reset
metadata.pilon.xml<\/b> - This file requires an entry in the
tool_data_table_conf.xml file. Upload a file named
tool_data_table_conf.xml.sample to the repository that includes the
required entry to correct this error."}

This is related to the pilon tool in that repo (tools/pilon). The tool in
question optionally uses a dataset from all_fasta, I'm not sure if that is
relevant for the error. BTW this error happened on push, not on PR.

Any ideas?

Thanks,
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Parameter Storing in Database

[Peter van Heusden]

You mean the stuff for the tool-data tables? As I understand it, it is
loaded in:

lib/galaxy/tools/data/__init__.py

There's a ToolDataTableManager and a TabularToolDataTable (with associated
TabularToolDataField).

Peter

On Fri, 5 Aug 2016 at 20:46 Katherine Beaulieu <
katherine.beaulieu...@gmail.com> wrote:

> Would anyone be able to tell me in what file the storing of tool
> parameters into the database occurs?
> Thanks!
> Katherine
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] run-as-user and Galaxy 16.04

[Peter van Heusden]

Hi there

Does anyone have the run-as-user support working with Galaxy 16.04? If so,
could you describe your config?

Thanks,
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Building tool dependencies

[Peter van Heusden]

Actually, for recent Galaxy, I'd recommend making a conda recipe for
UMI-tools and contributing it to bioconda:

https://bioconda.github.io/

Then your tool can just depend on this and the conda dependency resolver
(default in Galaxy since 16.01) can find it. Others can comment on the
recent work on conda/toolshed integration.

Peter

On 5 July 2016 at 08:56, Matthias De Smet  wrote:

>
> >
> > Hi
> >
> > I’m trying to put together a tool + dependency package to use umi-tools (
> https://github.com/CGATOxford/UMI-tools) from Galaxy.
> > I’m at a loss on how I should organise this.
> >
> > The package is available from either github or pip. Which one is
> preferable to use? How do I  make sure the package dependencies are
> installed? I’ve found the  and installing it with  type="shell_command">python setup.py install. This, however, tries
> to install the package to /usr/bin/ which isn’t desirable (obviously).
> >
> > Thanks a lot!
> > M
> >
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Mapping requirements to conda recipe names (granularity?)

[Peter van Heusden]

Hi there

I'm trying to satisfy requirements on our RNA-seq oriented Galaxy server at
SANBI using conda/bioconda. There's a disconnect between the names used in
tool dependency tags and the names of conda recipes though, for example:

1) the BAM to BigWig converter (tool ID CONVERTER_bam_to_bigwig_0) has a
requirement for ucsc_tools. The tool it needs, bedGraphToBigWig, is
provided by the ucsc-bedgraphtobigwig package, so in this case ucsc_tools
is a broader requirement than what bioconda provides.

2) the featurecounts tool requires featurecounts, which is part of the
subreads recipe, so in this case featurecounts is a narrow requirement than
what is provided in bioconda.

In the second case, one could say subreads provides the featurecounts
requirement. In the first case one could say the tool requires a narrower
requirement than what it specifies, although the bioconda recipes split a
single upstream package (ucsc_tools) into multiple recipes that invariable
will be updated together, a somewhat strange choice.

Right now I'm resolving these problems by manually creating environments
(e.g. __package__featurecounts@__unversioned__) that will get picked up by
the conda dependency resolver, but I'd like to hear from the list what the
best way to support this is? Probably the easiest is keeping the tool
requirements definition in line with what the conda recipe is called, but
in some cases a mapping table between requirements and conda recipe names
could be useful.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Auto detection of output format Galaxy custom tool

[Peter van Heusden]

Hi Katherine

I presume this is related to the tool described here (
https://biostar.usegalaxy.org/p/18319/)? I think there is general interest
in the creation of a tool like this - see the recent email from Miu ki Yip.
If you could share the code and XML (e.g. via GitHub) perhaps we could
assist with what you are trying to write.

Peter

On 30 June 2016 at 19:01, Peter Cock  wrote:

> Maybe sharing your tool XML file would be best - is it on GitHub?
>
> Peter
>
> On Wed, Jun 29, 2016 at 7:43 PM, Katherine Beaulieu
>  wrote:
> > Ok, so I don't give the user the option to select the correct data type
> to
> > reduce the amount of stuff the user has to know when using the tool.
> >
> > I do leave the format as auto and Galaxy just leaves it as the generic
> data
> > type 'data'.
> >
> > As for how the format is defined in Galaxy, I'm not quite sure what the
> > answer to this question is because I am just testing the tool with really
> > simple text files, with extension .txt. and it can't seem to pick it up,
> it
> > just leaves it as 'data'. Also not quite sure how to test the Python
> code in
> > the sniff function outside of Galaxy... (only starting working with
> Galaxy 3
> > weeks ago).
> >
> > Let me know if I should provide you with the tool config file if that
> would
> > be a little more useful. It's quite straightforward
> > Thanks for helping with this.
> >
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Implementing iRODS with Galaxy

[Peter van Heusden]

Hi Miuki

There's two aspects to iRODS and Galaxy. Firstly, there is the Galaxy
object store which can store objects in iRODS. This is entirely hidden from
the user and they have no idea that their datasets are in iRODS. Some work
on that happened earlier in this year, see:

https://wiki.galaxyproject.org/Community/Log/2015/GalaxyIrodsConnection

Then there is the question of iRODS import/export. See this thread from
earlier this year: http://dev.list.galaxyproject.org/iRODS-td4669076.html -
Iyad Kandalaft (http://www.goc411.ca/en/8192/Iyad-Kandalaft) mentioned some
future work on import/export. As part of the B3Africa project some
colleagues of ours are also investigating iRODS import/export. I think it
might be useful to make a GitHub issue describing the problem and have
people comment on there.

I had a brief look at the iRODS python client library and it seems to be in
a relatively early state of development (
https://github.com/irods/python-irodsclient) but it is definitely usable.

On 29 June 2016 at 21:40, Yip, Miu ki  wrote:

> Hello Galaxy Team,
>
> We're from the Bioinformatics Shared Resources (BSR) at Cold Spring Harbor
> Labs and we're looking to implement irods on a newly installed version of
> Galaxy.
>
> We want to incorporate iRODS (http://irods.org/). Is there a recommended
> way to do this, or no?
>
> Thanks in advance for the advice.
>
> Best,
> Miuki Yip
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Infinite recursion on conda dependency manager

[Peter van Heusden]

Hi there

I've got a package in our lab conda channel (its still in testing phase)
called goget. This is specified as a requirement for the build_ctb_gene
tool like this:


   goget


When the tool runs, I see infinite recursion on the conda side:

https://gist.github.com/pvanheus/c16a07a2c954741a311881cc7606df8d

Any ideas?

Thanks,
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] The state of the MOTHUR merge?

[Peter van Heusden]

Thanks Björn, no rush!

Is anyone here doing 16S on Galaxy? I've been looking into options, and
FastQC, Trimmomatic, PEAR and the MOTHUR tools seem to give a viable
workflow. Other options are QIIME or integrating the UPARSE pipeline as I
understand. Any comments appreciated!

Thanks,
Peter

On 3 June 2016 at 15:53, Björn Grüning <bjoern.gruen...@gmail.com> wrote:

> Me finishing my re review and pressing the merge button.
> I tend to merge as it is at the moment and do some small adjustments
> afterwards ... This weekend Peter, this weekend! :)
>
> Am 03.06.2016 um 13:37 schrieb Peter van Heusden:
> > PR #429 on tools-iuc contains the updated MOTHUR toolsuite for Galaxy.
> >
> > https://github.com/galaxyproject/tools-iuc/pull/429
> >
> > It *looks* like it is ready to merge - what is still needed before it
> > merges into tools-iuc?
> >
> > Thanks!
> > Peter
> >
> >
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >   https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> >   http://galaxyproject.org/search/mailinglists/
> >
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] The state of the MOTHUR merge?

[Peter van Heusden]

PR #429 on tools-iuc contains the updated MOTHUR toolsuite for Galaxy.

https://github.com/galaxyproject/tools-iuc/pull/429

It *looks* like it is ready to merge - what is still needed before it
merges into tools-iuc?

Thanks!
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Scaling tools: data available? (Peter van Heusden)

[Peter van Heusden]

Thanks Damion

We've got the collectl based metric collection stuff working and that dumps
data in the Galaxy DB (which you can pull out using the API).

Thanks,
Peter

On 17 March 2016 at 10:45, Dooley, Damion <damion.doo...@bccdc.ca> wrote:

> If you are looking for a simple python program to log cpu% and memory of
> another executable, then analyze.py might do.  Its in
> https://github.com/Public-Health-Bioinformatics/kipper/tree/master/RDP-test
> -case along with a few scripts that show examples of its use.
>
> Damion
> >>
> >> On 02/21/2016 11:13 PM, Peter van Heusden wrote:
> >>
> >> Hi there
> >>
> >> We're researching the resource (CPU time and memory) requirements for
> >>RNA
> >> STAR at the moment. Specifically, we'd like to build up a database of
> >>input
> >> size to resource usage so that we can use this to feed a dynamic
> >> destination mapper (currently our STAR configuration uses a thumbsuck
> >>for
> >> RAM and CPU requirements that is not optimal but works in many cases).
> >>
> >> Has anyone collected these metrics before? And more generally, is anyone
> >> collecting metrics for a range of bioinformatics tools? A database like
> >> this could be very useful!
> >>
> >> Thanks!
> >> Peter
> >>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] planemo update failing on testtoolshed

[Peter van Heusden]

The repo is empty, so there is nothing to delete at this stage. And I can't
figure out how to delete the thing entirely.

I looked at the splitFasta tool, and also the splitfasta tool from rnateam
(in the test toolshed) and these split into 1 file for each sequence as
opposed to a certain number of chunks.

Peter

On 18 March 2016 at 21:11, Björn Grüning <bjoern.gruen...@gmail.com> wrote:

> Hi Peter,
>
> can you try to delete files with the web interface?
>
> I saw the tool name, do you know this small tool?
>
> https://github.com/bgruening/galaxytools/blob/master/tools/splitfasta/splitFasta.xml
>
> Cheers,
> Bjoern
>
> Am 18.03.2016 um 20:03 schrieb Peter van Heusden:
> > Hi there
> >
> > I'm trying to update a newly created tool on the test tool shed and I
> > get this error:
> >
> > $ planemo shed_update -t testtoolshed
> > cd '/home/pvh/PycharmProjects/split_fasta_chunks' && git rev-parse HEAD
> > cd '/home/pvh/PycharmProjects/split_fasta_chunks' && git diff --quiet
> > Could not update split_fasta_chunks
> > Unexpected response from galaxy: 500: 
> >   Internal Server Error
> >   
> > Internal Server Error
> > The server has either erred or is incapable of performing
> > the requested operation.
> >
> > 
> > 
> > 
> > WSGI Server
> >   
> > 
> >
> > Repository metadata updated.
> > Failed to update a repository.
> >
> > I might have accidentally created a "bad" repository that is leading to
> > this error. This is with version 0.23.0. Alternately, is there a way to
> > delete the tool so that I can try and recreate it?
> >
> > Thanks!
> > Peter
> >
> >
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >   https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> >   http://galaxyproject.org/search/mailinglists/
> >
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] planemo update failing on testtoolshed

[Peter van Heusden]

Hi there

I'm trying to update a newly created tool on the test tool shed and I get
this error:

$ planemo shed_update -t testtoolshed
cd '/home/pvh/PycharmProjects/split_fasta_chunks' && git rev-parse HEAD
cd '/home/pvh/PycharmProjects/split_fasta_chunks' && git diff --quiet
Could not update split_fasta_chunks
Unexpected response from galaxy: 500: 
  Internal Server Error
  
Internal Server Error
The server has either erred or is incapable of performing
the requested operation.




WSGI Server
  


Repository metadata updated.
Failed to update a repository.

I might have accidentally created a "bad" repository that is leading to
this error. This is with version 0.23.0. Alternately, is there a way to
delete the tool so that I can try and recreate it?

Thanks!
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Scaling tools: data available?

[Peter van Heusden]

Hi Eric

I'm having a look at the GRT. We're collecting metrics on our local Galaxy
server, I'm just trying to understand how you envision these metrics
getting into the GRT?

Peter

On 22 February 2016 at 07:31, Eric Rasche <e...@tamu.edu> wrote:

> Hi peter,
>
> I've been working on this in the past but got caught up in other projects.
> There's work for a server-side component here (maybe 50% done), the client
> side stuff could all be done through collectl but I was having issues
> actually collecting collectl data that had put the project on hold
> temporarily.
>
> https://github.com/erasche/galactic-radio-telescope
>
> I'd be very, very interested in finishing this project up.
>
> Cheers,
> Eric
>
>
> On 02/21/2016 11:13 PM, Peter van Heusden wrote:
>
> Hi there
>
> We're researching the resource (CPU time and memory) requirements for RNA
> STAR at the moment. Specifically, we'd like to build up a database of input
> size to resource usage so that we can use this to feed a dynamic
> destination mapper (currently our STAR configuration uses a thumbsuck for
> RAM and CPU requirements that is not optimal but works in many cases).
>
> Has anyone collected these metrics before? And more generally, is anyone
> collecting metrics for a range of bioinformatics tools? A database like
> this could be very useful!
>
> Thanks!
> Peter
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
>
> --
> --
> Eric Rasche
> Programmer II
>
> Center for Phage Technology
> Rm 312A, BioBio
> Texas A University
> College Station, TX 77843
> 404-692-2048
> e...@tamu.edu
> Not responding quickly enough?
> <https://calendar.google.com/calendar/embed?src=esr%40tamu.edu=America/Chicago>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] novoalign in Galaxy

[Peter van Heusden]

Hi there

We're busy integrating novoalign in our local Galaxy - there doesn't seem
to be a tool around for this, but I recall some people saying they were
using (single threaded) novoalign with Galaxy. Does anyone know if tool XML
has already been written? If not, we'll just have to write it.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Broken package_perl_5_18, was Re: JBrowse on dev branch: progress

[Peter van Heusden]

I knew there was an email that I forgot to send! Nicola Soranzo fixed this
with galaxy PR https://github.com/galaxyproject/galaxy/pull/1775 - sometime
in November last year Eric added whitespace to the perl package (and a
bunch of other things) that broke the download_file action, so Nicola's
patch strips whitespace. (The conversation about this happened on IRC and
the PR)

I tested the PR and it makes the perl install work (which makes the JBrowse
installl work, yay!).

Peter
On 25 Feb 2016 6:10 PM, "Peter Briggs" <peter.bri...@manchester.ac.uk>
wrote:

> Hello Peter
>
> Re the failing install of 'package_perl_5_18', I've investigated further
> now and have submitted a patch which seems to fix the problem:
>
> https://github.com/galaxyproject/tools-iuc/pull/616
>
> Not sure if this is of any immediate help,
>
> Best wishes
>
> Peter
>
> On 22/02/16 09:43, Peter van Heusden wrote:
>
>> Ok, second attempt, perl install died with the same error, so I suspect
>> you're right about the cause. Eric?
>>
>> Peter
>>
>> On 22 February 2016 at 11:41, Peter van Heusden <p...@sanbi.ac.za
>> <mailto:p...@sanbi.ac.za>> wrote:
>>
>> Thanks Peter
>>
>> I managed, on the second attempt, to get past the gnu coreutils
>> problem. Zipho noticed a weird lockup though: conftest that seemed
>> to be from the gnu coreutils install wanted to listen on port 4001,
>> which is the same port our uwsgi server listens on. The install has
>> now completed but I'll try and find time to go back and verify this.
>>
>> On the second attempt perl seems to be installing, so I'll see if it
>> completes or hits the error you're mentioning.
>>
>> Peter
>>
>> On 22 February 2016 at 11:34, Peter Briggs
>> <peter.bri...@manchester.ac.uk
>> <mailto:peter.bri...@manchester.ac.uk>> wrote:
>>
>> Hi Peter
>>
>> I think that package_perl_5_18 in the tool shed is broken, I
>> came across the same error last week on both our production
>> instance and a local 'test' instance (both v15.10).
>>
>> My suspicion is that the following lines are to blame:
>>
>> ...
>>  >
>> sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b"
>> type="download_file">
>>
>> http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz
>>  
>>  tar xfvz
>> local-lib-1.008009.tar.gz
>> ...
>>
>> Looking at the Galaxy internals I believe that the
>> 'download_file' action automatically unpacks the archive and
>> cd's into the resulting directory, hence the failure for the
>> subsequent 'tar xfvz ...' command. But I haven't had the time
>>     yet to verify this or create a patch to see if the removal of
>> this line addresses the problem.
>>
>> I don't think I've seen the problem with
>> 'package_gnu_coreutils_8_22' that you describe however so it may
>> be that my hypothesis is not correct.
>>
>> HTH
>>
>> Best wishes
>>
>> Peter
>>
>>
>> On 22/02/16 05:10, Peter van Heusden wrote:
>>
>> Hi there Eric and others
>>
>> I tried to install the JBrowse tool on a dev branch server
>> running on an
>> Ubuntu 14.04 VM.
>>
>> It got stuck at the package_gnu_coreutils_8_22 dependency
>> (revision
>> ac64dfe4b1fb) - this ended up in "Installing tool
>> dependencies" state.
>>
>> If I go look at the tool details it highlights errors
>> in package_perl_5_18 which in turn refers to errors in
>> gnu_coreutils
>> (just "Error" state) and then this more informative failure
>> for perl:
>>
>> "tar (child): local-lib-1.008009.tar.gz: Cannot open: No
>> such file or
>> directory
>> tar (child): Error is not recoverable: exiting now
>> tar: Child returned status 2
>> tar: Error is not recoverable: exiting now"
>>
>> I wonder if this is a sign of a download error?
>>
>> Digging a bit more I find the error "Shutting down process
>> id 

Re: [galaxy-dev] JBrowse on dev branch: progress

[Peter van Heusden]

Ok, second attempt, perl install died with the same error, so I suspect
you're right about the cause. Eric?

Peter

On 22 February 2016 at 11:41, Peter van Heusden <p...@sanbi.ac.za> wrote:

> Thanks Peter
>
> I managed, on the second attempt, to get past the gnu coreutils problem.
> Zipho noticed a weird lockup though: conftest that seemed to be from the
> gnu coreutils install wanted to listen on port 4001, which is the same port
> our uwsgi server listens on. The install has now completed but I'll try and
> find time to go back and verify this.
>
> On the second attempt perl seems to be installing, so I'll see if it
> completes or hits the error you're mentioning.
>
> Peter
>
> On 22 February 2016 at 11:34, Peter Briggs <peter.bri...@manchester.ac.uk>
> wrote:
>
>> Hi Peter
>>
>> I think that package_perl_5_18 in the tool shed is broken, I came across
>> the same error last week on both our production instance and a local 'test'
>> instance (both v15.10).
>>
>> My suspicion is that the following lines are to blame:
>>
>> ...
>> > sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b"
>> type="download_file">
>> http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz
>> 
>> tar xfvz
>> local-lib-1.008009.tar.gz
>> ...
>>
>> Looking at the Galaxy internals I believe that the 'download_file' action
>> automatically unpacks the archive and cd's into the resulting directory,
>> hence the failure for the subsequent 'tar xfvz ...' command. But I haven't
>> had the time yet to verify this or create a patch to see if the removal of
>> this line addresses the problem.
>>
>> I don't think I've seen the problem with 'package_gnu_coreutils_8_22'
>> that you describe however so it may be that my hypothesis is not correct.
>>
>> HTH
>>
>> Best wishes
>>
>> Peter
>>
>>
>> On 22/02/16 05:10, Peter van Heusden wrote:
>>
>>> Hi there Eric and others
>>>
>>> I tried to install the JBrowse tool on a dev branch server running on an
>>> Ubuntu 14.04 VM.
>>>
>>> It got stuck at the package_gnu_coreutils_8_22 dependency (revision
>>> ac64dfe4b1fb) - this ended up in "Installing tool dependencies" state.
>>>
>>> If I go look at the tool details it highlights errors
>>> in package_perl_5_18 which in turn refers to errors in gnu_coreutils
>>> (just "Error" state) and then this more informative failure for perl:
>>>
>>> "tar (child): local-lib-1.008009.tar.gz: Cannot open: No such file or
>>> directory
>>> tar (child): Error is not recoverable: exiting now
>>> tar: Child returned status 2
>>> tar: Error is not recoverable: exiting now"
>>>
>>> I wonder if this is a sign of a download error?
>>>
>>> Digging a bit more I find the error "Shutting down process id 21094
>>> because it generated no output for the defined timeout period of 3600.0
>>> seconds." for gnu_coreutils.
>>>
>>> I'm trying to do a reinstall of the gnu_coreutils and perl packages to
>>> see if I can get further.
>>>
>>> Peter
>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>http://galaxyproject.org/search/mailinglists/
>>>
>>>
>> --
>> Peter Briggs peter.bri...@manchester.ac.uk
>> Bioinformatics Core Facility University of Manchester
>> B.1083 Michael Smith Bldg Tel: (0161) 2751482
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Plans for workflow & parallelisation work?

[Peter van Heusden]

Hi there

I see from the PR landing in Galaxy and the comments on things like
issue #1701 (https://github.com/galaxyproject/galaxy/issues/1701) that
there's lots of work happening on the workflow side of Galaxy. This is an
area of interest at SANBI too, so we'd like to coordinate development
efforts as much as possible. To this end:

1) Are there forks to track so we can see what new code is landing?
2) Is there a roadmap for workflow work or perhaps can we have a Hangout to
talk about this?
3) Specifically in terms of workflows and parallelisation: are there any
plans to work on running workflows as opposed to just generating lots of
jobs? I know this is a major change to how Galaxy works - it would mean
something like submitting a workflow specification to a job runner that is
located on the cluster, and then returning the results of workflow
execution.
4) Currently parallelisation in Galaxy is supported using two mechanisms:
collections and dataset splitters/tasks. Are there plans on extending and
harmonising Galaxy's parallelisation capabilities?

Thanks,
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Scaling tools: data available?

[Peter van Heusden]

Hi there

We're researching the resource (CPU time and memory) requirements for RNA
STAR at the moment. Specifically, we'd like to build up a database of input
size to resource usage so that we can use this to feed a dynamic
destination mapper (currently our STAR configuration uses a thumbsuck for
RAM and CPU requirements that is not optimal but works in many cases).

Has anyone collected these metrics before? And more generally, is anyone
collecting metrics for a range of bioinformatics tools? A database like
this could be very useful!

Thanks!
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy broken after update to 16.01

[Peter van Heusden]

Mattias, what does your supervisord config look like? Specifically, does it
contain an environment variable to direct the Galaxy handler to the correct
virtualenv?

Peter

On 9 February 2016 at 16:03, Matthias De Smet 
wrote:

> Hi all,
>
>
> We were tracking the galaxy master branch on our local server and are
> looking to migrate to v16.01.
> The galaxy processes are managed bij supervisor as shown here (
> https://wiki.galaxyproject.org/Admin/Config/Performance/Scaling).
> After updating all dependencies in our venv we tried to restart the server
> but got the following issue:
>
> Loading paste environment:
> config:/home/galaxy/galaxy-dist/config/galaxy.ini
> Traceback (most recent call last):
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py",
> line 247, in loadapp
> return loadobj(APP, uri, name=name, **kw)
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py",
> line 271, in loadobj
> global_conf=global_conf)
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py",
> line 296, in loadcontext
> global_conf=global_conf)
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py",
> line 320, in _loadconfig
> return loader.get_context(object_type, name, global_conf)
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py",
> line 458, in get_context
> section)
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py",
> line 517, in _context_from_explicit
> value = import_string(found_expr)
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py",
> line 22, in import_string
> return pkg_resources.EntryPoint.parse("x=" + s).load(False)
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 2380, in load
> return self.resolve()
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/pkg_resources/__init__.py",
> line 2386, in resolve
> module = __import__(self.module_name, fromlist=['__name__'], level=0)
>   File "lib/galaxy/web/buildapp.py", line 5, in 
> from galaxy.webapps.galaxy.buildapp import app_factory
>   File "lib/galaxy/webapps/galaxy/buildapp.py", line 15, in 
> import galaxy.app
>   File "lib/galaxy/app.py", line 9, in 
> import galaxy.queues
>   File "lib/galaxy/queues.py", line 9, in 
> from kombu import Exchange, Queue, Connection
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/kombu/__init__.py",
> line 67, in __getattr__
> module = __import__(object_origins[name], None, None, [name])
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/kombu/entity.py",
> line 10, in 
> from .abstract import MaybeChannelBound, Object
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/kombu/abstract.py",
> line 12, in 
> from .connection import maybe_channel
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/kombu/connection.py",
> line 19, in 
> from kombu import exceptions
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/kombu/exceptions.py",
> line 12, in 
> from amqp import ChannelError, ConnectionError, ResourceError
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/amqp/__init__.py",
> line 34, in 
> from .connection import Connection  # noqa
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/amqp/connection.py",
> line 40, in 
> from .transport import create_transport
>   File
> "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/amqp/transport.py",
> line 21, in 
> import ssl
>   File "/Shared/Software/Cellar/python/2.7.11/lib/python2.7/ssl.py", line
> 97, in 
> import _ssl # if we can't import it, let the error
> propagate
> ImportError:
> /home/galaxy/galaxy-dist/.venv/lib/python2.7/lib-dynload/_ssl.so: undefined
> symbol: SSL_CTX_set_alpn_protos
>
> So far we have tried:
> - rebuilding the venv
> - reinstalling python
>
> Funny thing is, when we start the configuration with the “run.sh” script
> instead of through supervisor everything seems to work just fine.
>
> We’re at a loss, please help!
> Thanks!
> Matthias
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to 

Re: [galaxy-dev] Trimmomatic doesn't recognise pair collection

[Peter van Heusden]

Hi Peter

We found that it was a problem in the Galaxy code - it was fixed in the
some of the patches that went towards the 16.01 release.

Thanks,
Peter


On 4 February 2016 at 13:47, Peter Briggs <peter.bri...@manchester.ac.uk>
wrote:

> Hello Peter
>
> I'm the author of the Trimmomatic tool on the toolshed (i.e.
> https://toolshed.g2.bx.psu.edu/view/pjbriggs/trimmomatic) - apologies for
> not responding sooner.
>
> The tool works for me on our production server (v15.10 from github) when
> using a pair of fastqs as input, by which I mean that it allows me to
> select a fastq-pair dataset as input and produces 2 fastq-pair datasets.
> (I'm haven't tried on a list of pairs.)
>
> What do you see when you say that the tool 'is not working'? I'm wondering
> if it is a problem with the tool itself, or something to do with the
> underlying Galaxy infrastructure when using the head of the dev branch).
>
> Best wishes
>
> Peter
>
>
> On 22/01/16 13:53, Peter van Heusden wrote:
>
>> Hi there
>>
>> I've got Trimmomatic installed (from the toolshed) into our local Galaxy
>> server (server is latest commit on the dev branch from GitHub). It
>> refuses to see a dataset pair as a valid input, however.
>>
>> I've tested this with two types: firstly a pair made out of 2 individual
>> datasets (both fastqsanger) and then also a list of pairs. I've tested
>> this with 'planemo serve' and the tool works, but for some reason it is
>> not working on our server. Any ideas?
>>
>> Thanks,
>> Peter
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>http://galaxyproject.org/search/mailinglists/
>>
>>
> --
> Peter Briggs peter.bri...@manchester.ac.uk
> Bioinformatics Core Facility University of Manchester
> B.1083 Michael Smith Bldg Tel: (0161) 2751482
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] planemo reports test failure when test succeeds

[Peter van Heusden]

Zipho looked at this over here and had some ideas about how some warning
output might trigger this spurious failure - I haven't had time to follow
up on that theory.

On 4 February 2016 at 14:26, Peter Briggs <peter.bri...@manchester.ac.uk>
wrote:

> Hello again Peter
>
> I also see this when using planemo to run the tool tests both on
> trimmomatic and also on another of our tools i.e. one or more of the tests
> will be reported as 'failed' on the command line when the HTML report
> indicates that all tests executed successfully.
>
> Now I think about it, planemo itself must exit with status 0 in these
> cases as my tests on Travis-CI also report success, for example see:
>
> https://travis-ci.org/fls-bioinformatics-core/galaxy-tools/jobs/104643315
>
> Scroll to the bottom of the log and the last test is reported as 'failed',
> however the Travis test overall passes. I hadn't really thought about it
> before since the planemo exit status seems to be correct (i.e. in cases
> where there are genuine tool test failures, it will return a non-zero
> status at the end which causes Travis to correctly report a failure).
>
> (Btw this referenced above used planemo 0.22.2 but I've seen it in older
> versions too.)
>
> I don't think I've ever seen planemo report a 'passed' test as 'failed',
> but generally I check the HTML to verify whether the failures are real or
> not.
>
> Best wishes
>
> Peter
>
>
> On 22/01/16 11:40, Peter van Heusden wrote:
>
>> Hi there
>>
>> When testing the latest trimmomatic tool (from
>>
>> https://github.com/fls-bioinformatics-core/galaxy-tools/tree/master/trimmomatic
>> ,
>> planemo test reports an error on the last test, but according to the
>> test report the test has succeeded (see attached HTML and JSON). For the
>> test setup you need to set some environment variables:
>>
>> TRIMMOMATIC_DIR - the location of the trimmomatic jar
>> TRIMMOMATIC_JAR - the name of the jar file
>> TRIMMOMATIC_ADAPTERS_DIR - location of adapters
>>
>> This is with planemo 0.22.2 and a fresh $HOME/.planemo.
>>
>> Any ideas?
>>
>> Peter
>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>http://galaxyproject.org/search/mailinglists/
>>
>>
> --
> Peter Briggs peter.bri...@manchester.ac.uk
> Bioinformatics Core Facility University of Manchester
> B.1083 Michael Smith Bldg Tel: (0161) 2751482
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Working on the admin interface: JS vs Mako

[Peter van Heusden]

Yeah I noticed that about the DependencyResolversView.

In terms of writing JS - where's a good example to look at? I tried digging
around the client/ directories but couldn't make sense of how it works.

Thanks,
Peter

On 1 February 2016 at 16:48, John Chilton <jmchil...@gmail.com> wrote:

> Peter,
>
>   We would like to replace all the mako with JS, if I was going to put
> a bunch of effort into admin pages I'd start by reworking what was
> there to use JavaScript and the API. That is me however, I have lots
> of time to put into Galaxy fundamentals and refactoring. This is more
> work and I am happy to field PRs that modify mako for the time
> being.(https://github.com/galaxyproject/galaxy/pull/1632/files looks
> great for instance).
>
>   A middle ground between rewriting everything and still depending on
> mako is to dump dictionaries to JavaScript in the mako and render
> things in JS. This will make it easier to adapt your changes once the
> pages are ultimately rewritten to JavaScript SPAs. In the case 1632 -
> that DependencyResolversView class produces dictionaries that can be
> dumped as JSON easily into the page and JavaScript could be used to
> build up the tables in
> templates/webapps/tool_shed/repository/common.mako.
>
> -John
>
>
> On Mon, Feb 1, 2016 at 11:51 AM, Peter van Heusden <p...@sanbi.ac.za>
> wrote:
> > Hi there
> >
> > Zipho and I'd would like to make some changes to the admin pages,
> including
> > the tool info page and the local data page. These are currently rendered
> > from mako templates - should we change these or is the interface going
> to be
> > rewritten in client side / javascript in the near(ish) future?
> >
> > Thanks,
> > Peter
> >
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >   https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> >   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] galaxy job manager with celery and rabbitmq

[Peter van Heusden]

Galaxy implements the job manager using rabbitmq and celery? Surely you
mean Galaxy *does not* implement celery (at this point). As far as I know
it is planned for the future.

Peter

On 25 January 2016 at 15:37, 杜云飞  wrote:

> Hi,
>   We are reading the source code for galaxy job manager. We found that the
> job manager works like amqp. We want to know why galaxy implements the  job
> manager using rabbitmq and celery. What is the difference between  them?
> Thanks!
> Best regards,
>forest
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Trimmomatic doesn't recognise pair collection

[Peter van Heusden]

Hi there

I've got Trimmomatic installed (from the toolshed) into our local Galaxy
server (server is latest commit on the dev branch from GitHub). It refuses
to see a dataset pair as a valid input, however.

I've tested this with two types: firstly a pair made out of 2 individual
datasets (both fastqsanger) and then also a list of pairs. I've tested this
with 'planemo serve' and the tool works, but for some reason it is not
working on our server. Any ideas?

Thanks,
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Ansible roles: galaxy_server_dir vs galaxy_home_dir and galaxy_root

[Peter van Heusden]

Hi there

We've just finished provisioning a new Galaxy server here using the
ansible-galaxy and ansible-galaxy-extras roles, with some extra roles from
Nate and our own creation. In the ansible-galaxy role Galaxy's install
location is set using the galaxy_server_dir variable, but in
ansible-galaxy-extras the variables galaxy_home_dir and galaxy_root
(presumably for distinct locations for the galaxy user's home directory and
the root of the Galaxy installation) are used.

Is this difference deliberate? Attached is our Ansible playbook - we just
configured the variables to all point to the same location (and set the
galaxy_venv_dir to be .venv not venv). I thought the two roles were meant
to work together, so this difference is somewhat confusing.

Thanks,
Peter


galaxydev.yml
Description: application/yaml
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Job metrics configuration

[Peter van Heusden]

Hi there

I'm trying to understand how the job metrics configuration works. Reading
through various examples, it looks like it works together with the dynamic
destination support.

First on the dynamic destination support, this works, as I understand it,
by importing modules (files ending in .py) from the directory set as the
rules_module, where rules_module is a parameter for the dynamic job plugin.
As in this snipped from the usegalaxy configuration:



usegalaxy.jobs.rules



So import usegalaxy.jobs.rules, find where this resolves to as a directory,
find all .py files in that directory. If rules_module is not specified, the
default is galaxy.jobs.rules (under galaxy/lib).

So these .py files are imported and within them are functions that are used
in dynamic destinations (those with runner="dynamic") and in turn tools are
directed to use these destinations.

Now the job metrics configuration supplies both a user interface and then a
set of keys that are made accessible to dynamic runners as key value pairs
on the job.parameters attribute (passed in as the third argument, i.e.
job). Specifically the job metrics configuration exists under a key
'__job_resource'.

Is this all correct? Is there a more straightforward way of moving from job
resource specification to job configuration than this?

Thanks,
Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Job metrics configuration

[Peter van Heusden]

Hi there John

Thanks for the clarification. My initial aim was, exactly, to expose those
parameters that, as you say, some people don't think should be exposed.

My use case is to allow jobs with different native parameters to be
scheduled on our clouster from our Galaxy server. Perhaps a better approach
is to use dynamic destinations that try and estimate appropriate values for
e.g. hvmem, but at present our users are used to setting such parameters
themselves.

Peter

On 11 January 2016 at 17:32, John Chilton <jmchil...@gmail.com> wrote:

> So job metrics are completely different than what you described here.
> Job metrics are enabled by default and can be configured fairly
> directly by modifying config/job_metrics_conf.xml. New plugins are
> relatively easy to write and can be added as python files to
> lib/galaxy/jobs/metrics/instrumenters/.
>
> What you described is what is used on usegalaxy.org to allow users to
> have some input on how jobs are scheduled. I call these job resources
> parameters - in job_conf.xml particular tools can be mapped to these
> parameters and - yes you are correct dynamic job destination functions
> can consume them.
>
> The dynamic job destinations definitely shouldn't be accessing these
> through job.parameters keyed on __job_resource. The dynamic
> destination function should just take in a parameter named
> resource_params and then the function will be passed in.
>
> Dynamic Destinations are documented here:
>
> https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping
>
> You are correct job resource parameters have not been documented*,
> there is an open issue here:
> https://github.com/galaxyproject/galaxy/issues/1266
>
> * It should be noted that while this a frequently requested feature
> and used on usegalaxy.org - the feature is somewhat controversial. The
> thinking among some is that is exposes a level of detail that Galaxy
> is supposed to hide. I don't share this particular concern, but the
> implementation does worry me - what it does to persisted job state and
> parameter handling is fairly hacky and I'm somewhat surprised we
> haven't encountered more problems. This is part of the reason not why
> this hasn't been documented, but why I feel less bad that it hasn't
> been documented than other stuff I have done that also hasn't been
> documented :|.
>
> -John
>
> On Mon, Jan 11, 2016 at 1:56 PM, Peter van Heusden <p...@sanbi.ac.za>
> wrote:
> > Hi there
> >
> > I'm trying to understand how the job metrics configuration works. Reading
> > through various examples, it looks like it works together with the
> dynamic
> > destination support.
> >
> > First on the dynamic destination support, this works, as I understand
> it, by
> > importing modules (files ending in .py) from the directory set as the
> > rules_module, where rules_module is a parameter for the dynamic job
> plugin.
> > As in this snipped from the usegalaxy configuration:
> >
> > 
> > 
> > usegalaxy.jobs.rules
> > 
> >
> >
> > So import usegalaxy.jobs.rules, find where this resolves to as a
> directory,
> > find all .py files in that directory. If rules_module is not specified,
> the
> > default is galaxy.jobs.rules (under galaxy/lib).
> >
> > So these .py files are imported and within them are functions that are
> used
> > in dynamic destinations (those with runner="dynamic") and in turn tools
> are
> > directed to use these destinations.
> >
> > Now the job metrics configuration supplies both a user interface and
> then a
> > set of keys that are made accessible to dynamic runners as key value
> pairs
> > on the job.parameters attribute (passed in as the third argument, i.e.
> job).
> > Specifically the job metrics configuration exists under a key
> > '__job_resource'.
> >
> > Is this all correct? Is there a more straightforward way of moving from
> job
> > resource specification to job configuration than this?
> >
> > Thanks,
> > Peter
> >
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >   https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> >   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Multiple web servers and handlers Galaxy failed

[Peter van Heusden]

Hi Makis, I recently implemented something like this using uWSGI with a
nginx frontend. Would you consider using uWSGI instead of your current
approach?
On 4 Dec 2015 09:40, "Makis Ladoukakis"  wrote:

> Sorry for bumping my own question but I didn't get any replies. Has anyone
> succesfully installed a production Galaxy instance?
>
> I could really use some advice.
>
> Thank you,
> Makis
>
> --
> From: makis4e...@hotmail.com
> To: galaxy-...@lists.bx.psu.edu; galaxy-dev@lists.galaxyproject.org
> Date: Tue, 1 Dec 2015 18:03:44 +0200
> Subject: [galaxy-dev] Multiple web servers and handlers Galaxy failed
>
> Hello everyone,
>
> I am trying to setup a Galaxy instance for a multi-user production
> environment and I tried to follow the instructions here:
>
> https://wiki.galaxyproject.org/Admin/Config/Performance/Scaling
> https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer
> https://wiki.galaxyproject.org/Admin/Config/ApacheProxy
>
> but when I tried to start my galaxy instance (GALAXY_RUN_ALL=1 sh run.sh
> --daemon) I got the following error in my browser:Bad Gateway The proxy
> server received an invalid response from an upstream server.
>
>
> I don't get any errors from the paster.log files so I am guessing the
> error is somewhere in my apache configuration although I can't see where it
> could be.
>
>
> Has anyone encountered this problem before?
>
> Kind regards,
> Makis
>
>
> P.S. My apache configuration is as follows:
>
> I added that to my apache httpd.conf file:
>
> #for galaxy
> #RewriteRule ^/galaxy(.*) http://localhost:8091$1 [P]
> RewriteRule ^/galaxy$ /galaxy/ [R]
> RewriteRule ^/galaxy/static/style/(.*)
> /home/galaxy/galaxy/static/june_2007_style/blue/$1 [L]
> RewriteRule ^/galaxy/static/scripts/(.*)
> /home/galaxy/galaxy/static/scripts/packed/$1 [L]
> RewriteRule ^/galaxy/static/(.*) /home/galaxy/galaxy/static/$1 [L]
> RewriteRule ^/galaxy/favicon.ico
> /home/galaxy/galaxy/static/favicon.ico [L]
> RewriteRule ^/galaxy/robots.txt /home/galaxy/galaxy/static/robots.txt
> [L]
>
> LoadModule uwsgi_module /etc/httpd/modules/mod_uwsgi.so
>
> 
> Sethandler uwsgi-handler
> uWSGISocket 127.0.0.1:4001
> uWSGImaxVars 512
> 
>
> I am also attaching my job_conf.xml and galaxy.ini file.
>
>
> ___ Please keep
> all replies on the list by using "reply all" in your mail client. To manage
> your subscriptions to this and other Galaxy lists, please use the interface
> at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use
> the unified search at: http://galaxyproject.org/search/mailinglists/
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Multiple web servers and handlers Galaxy failed

[Peter van Heusden]

I've written up my experiences in a blog:

http://pvh.wp.sanbi.ac.za/2015/12/04/faster-galaxy-with-uwsgi/

The only Galaxy side changes were the ones mentioned on the Scaling page.
The Bad Gateway message means that Apache can't talk to the uWSGI so what
is the state of uWSGI?

Are you using supervisord? Does supervisorctl show the uWSGI process
running? Does ps ax |grep uwsgi show it running? What does sudo lsof -i
:4001 show is listening on port 4001? That would be the starting point of
my debugging.

Peter
On 4 Dec 2015 10:57, "Makis Ladoukakis"  wrote:

> It is uwsgi that I am trying but with apache. Could you let me know what
> changes you made to the configuration of Galaxy?
>
> Thank you,
> Makis
>
> --
> Date: Fri, 4 Dec 2015 10:01:56 +0200
> Subject: Re: [galaxy-dev] Multiple web servers and handlers Galaxy failed
> From: p...@sanbi.ac.za
> To: makis4e...@hotmail.com
> CC: galaxy-...@lists.bx.psu.edu; galaxy-dev@lists.galaxyproject.org
>
> Hi Makis, I recently implemented something like this using uWSGI with a
> nginx frontend. Would you consider using uWSGI instead of your current
> approach?
> On 4 Dec 2015 09:40, "Makis Ladoukakis"  wrote:
>
> Sorry for bumping my own question but I didn't get any replies. Has anyone
> succesfully installed a production Galaxy instance?
>
> I could really use some advice.
>
> Thank you,
> Makis
>
> --
> From: makis4e...@hotmail.com
> To: galaxy-...@lists.bx.psu.edu; galaxy-dev@lists.galaxyproject.org
> Date: Tue, 1 Dec 2015 18:03:44 +0200
> Subject: [galaxy-dev] Multiple web servers and handlers Galaxy failed
>
> Hello everyone,
>
> I am trying to setup a Galaxy instance for a multi-user production
> environment and I tried to follow the instructions here:
>
> https://wiki.galaxyproject.org/Admin/Config/Performance/Scaling
> https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer
> https://wiki.galaxyproject.org/Admin/Config/ApacheProxy
>
> but when I tried to start my galaxy instance (GALAXY_RUN_ALL=1 sh run.sh
> --daemon) I got the following error in my browser:Bad Gateway The proxy
> server received an invalid response from an upstream server.
>
>
> I don't get any errors from the paster.log files so I am guessing the
> error is somewhere in my apache configuration although I can't see where it
> could be.
>
>
> Has anyone encountered this problem before?
>
> Kind regards,
> Makis
>
>
> P.S. My apache configuration is as follows:
>
> I added that to my apache httpd.conf file:
>
> #for galaxy
> #RewriteRule ^/galaxy(.*) http://localhost:8091$1 [P]
> RewriteRule ^/galaxy$ /galaxy/ [R]
> RewriteRule ^/galaxy/static/style/(.*)
> /home/galaxy/galaxy/static/june_2007_style/blue/$1 [L]
> RewriteRule ^/galaxy/static/scripts/(.*)
> /home/galaxy/galaxy/static/scripts/packed/$1 [L]
> RewriteRule ^/galaxy/static/(.*) /home/galaxy/galaxy/static/$1 [L]
> RewriteRule ^/galaxy/favicon.ico
> /home/galaxy/galaxy/static/favicon.ico [L]
> RewriteRule ^/galaxy/robots.txt /home/galaxy/galaxy/static/robots.txt
> [L]
>
> LoadModule uwsgi_module /etc/httpd/modules/mod_uwsgi.so
>
> 
> Sethandler uwsgi-handler
> uWSGISocket 127.0.0.1:4001
> uWSGImaxVars 512
> 
>
> I am also attaching my job_conf.xml and galaxy.ini file.
>
>
> ___ Please keep
> all replies on the list by using "reply all" in your mail client. To manage
> your subscriptions to this and other Galaxy lists, please use the interface
> at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use
> the unified search at: http://galaxyproject.org/search/mailinglists/
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Peter van Heusden]

These names have meaning in CollectedDatasetMatch:

designation means designation if that exists else name
name means name
dbkey means dbkey
ext means ext
visible means visible


On 21 October 2015 at 08:44, Steve Cassidy <steve.cass...@mq.edu.au> wrote:

> Thanks, yes that works, though I'm not really sure what the difference is
> between __name__ and __designation__. They both seem to have the same
> effect in my example.
>
> Thanks for your help.
>
> Steve
>
>
> On 21 October 2015 at 16:49, Peter van Heusden <p...@sanbi.ac.za> wrote:
>
>> Since you're using a directory, you can use one of the built in patterns:
>>
>> DEFAULT_EXTRA_FILENAME_PATTERN = 
>> r"primary_DATASET_ID_(?P[^_]+)_(?P[^_]+)_(?P[^_]+)(_(?P[^_]+))?"
>>
>> NAMED_PATTERNS = {
>> "__default__": DEFAULT_EXTRA_FILENAME_PATTERN,
>> "__name__": r"(?P.*)",
>> "__designation__": r"(?P.*)",
>> "__name_and_ext__": r"(?P.*)\.(?P[^\.]+)?",
>> "__designation_and_ext__": r"(?P.*)\.(?P[^\._]+)?",
>> }
>>
>> In terms of docs, I don't know what the future is - the Galaxy wiki or 
>> http://galaxy.readthedocs.org/en/master/
>>
>>
>> On 21 October 2015 at 04:26, Steve Cassidy <steve.cass...@mq.edu.au>
>> wrote:
>>
>>> Ah, thankyou, yes, I can now get results by using patterns to match the
>>> output.  I used your example but prepend 'simple' to the filename and then
>>> searches for that with:
>>>
>>> 
>>> this solves the problem for the sample script but not generally since in
>>> general I can't predict the filenames that will be generated - this is a
>>> tool for downloading data from a repository which could be text, audio or
>>> video data.
>>>
>>> If I don't use the 'simple' prefix and omit the file extension I still
>>> get my data but I also get three other files which are temporary scripts
>>> generated by galaxy and placed in the working directory.  So, back to
>>> trying to put things in a subdirectory. It turns out that the issue I was
>>> having was as you pointed out earlier, the directory attribute to
>>> discover_datasets doesn't allow variables, so I need to write to a fixed
>>> directory name:
>>>
>>> 
>>> This now works!
>>>
>>> I had thought that I'd need to use a unique directory name but since
>>> galaxy runs each job in a separate directory, this isn't required.  My real
>>> tool now works too after following the same pattern.
>>>
>>> Thanks for your help.  I'll see if I can write this up in a blog post.
>>>
>>> Steve
>>>
>>>
>>>
>>> On 21 October 2015 at 00:06, Peter van Heusden <p...@sanbi.ac.za> wrote:
>>>
>>>> I poked around at your tool XML and the code a bit and the problem is
>>>> directory="$job_name". Galaxy expects to collect files from the job's
>>>> working directory - basically the current working directory the job runs
>>>> in. The directory= argument doesn't have variables expanded as far as I can
>>>> tell. In any event it is used in walk_over_extra_files() that is in
>>>> lib/galaxy/tools/parameters/output_collect.py - if you look there you see
>>>> that it is simply appended to the job's working directory.
>>>>
>>>> So if you use:
>>>>
>>>> 
>>>>
>>>> (note the  and  - this is effectively the regexp
>>>> r"(?P.*)\.txt" with the < and > escaped out)
>>>>
>>>> And alter the code so that it just writes files to the current
>>>> directory, then you'll pick up the files one.txt, two,txt and three.txt.
>>>>
>>>> Peter
>>>>
>>>> On 20 October 2015 at 12:28, Steve Cassidy <steve.cass...@mq.edu.au>
>>>> wrote:
>>>>
>>>>> Sorry, it was just an example of a tool that works - the file that it
>>>>> writes out is put into that directory, so I assume that's where my files
>>>>> should end up too.
>>>>>
>>>>> Steve
>>>>>
>>>>> On 20 October 2015 at 21:12, Peter van Heusden <p...@sanbi.ac.za>
>>>>> wrote:
>>>>>
>>>>>> Sorry, I don't understand - what does the Upload File tool have to do
>>>>>> with this?
>>>>>>
>>>>&g

Re: [galaxy-dev] Tools that make datasets

[Peter van Heusden]

Just a quick check - did you refresh your history to confirm that the
dataset *is* empty? We had the same thing at SANBI but it turns out that
Galaxy creates an empty output collection and then only populates it
sometime after job completion (this is a know UI bug).

See:
http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/

On 20 October 2015 at 08:48, Steve Cassidy  wrote:

> Hi all,
>   I'm trying to understand how to write a tool that generates a dataset
> rather than a single output file.  I've tried following all of the examples
> but I'm stuck, so I thought I would distil down the simplest example I
> could write and ask for help here.
>
> So here's my example:
>
> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55
>
> it's a simple python script that writes three files to a directory named
> for the single input parameter.
>
> I think one of the problems I'm having is knowing where to write the
> output to. I've run this under planemo serve and the job runs, creating the
> output directory within the 'job_working_directory/000/1/SampleDataset'
> directory, however my dataset doesn't contain anything so clearly my
> outputs directive isn't working:
>
> 
> 
>  directory="$job_name" />
> 
> 
>
> ($job_name is the name of the directory that is being written to,
> SampleDataset in this case)
>
> Any help in getting this example working would be appreciated.
>
> Thanks,
>
> Steve
>
>
>
>
>
> --
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy/
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Peter van Heusden]

I suspect that the problem might be in the  then. I'm
not an export on this, but "__name_and_ext__" turns into the
regexp r"(?P.*)\.(?P[^\.]+)?" in
lib/galaxy/tools/parameters/output_collect.py, and is used by the
DatasetCollector (line 358). This looks like it should match the filenames
you're creating, but I'm not 100% sure how that code works. One thing I
notice is the "directory" argument. If you write jobs to the current
directory instead of "output_path" can you get it to work?

Peter

On 20 October 2015 at 09:52, Steve Cassidy <steve.cass...@mq.edu.au> wrote:

> Thanks Peter,
>   I did see that proviso somewhere but no, refreshing doesn't help.
>
> That page was one of those that I referred to getting to this point.
>
> Steve
>
> On 20 October 2015 at 18:33, Peter van Heusden <p...@sanbi.ac.za> wrote:
>
>> Just a quick check - did you refresh your history to confirm that the
>> dataset *is* empty? We had the same thing at SANBI but it turns out that
>> Galaxy creates an empty output collection and then only populates it
>> sometime after job completion (this is a know UI bug).
>>
>> See:
>> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/
>>
>> On 20 October 2015 at 08:48, Steve Cassidy <steve.cass...@mq.edu.au>
>> wrote:
>>
>>> Hi all,
>>>   I'm trying to understand how to write a tool that generates a dataset
>>> rather than a single output file.  I've tried following all of the examples
>>> but I'm stuck, so I thought I would distil down the simplest example I
>>> could write and ask for help here.
>>>
>>> So here's my example:
>>>
>>> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55
>>>
>>> it's a simple python script that writes three files to a directory named
>>> for the single input parameter.
>>>
>>> I think one of the problems I'm having is knowing where to write the
>>> output to. I've run this under planemo serve and the job runs, creating the
>>> output directory within the 'job_working_directory/000/1/SampleDataset'
>>> directory, however my dataset doesn't contain anything so clearly my
>>> outputs directive isn't working:
>>>
>>> 
>>> 
>>> >> directory="$job_name" />
>>> 
>>> 
>>>
>>> ($job_name is the name of the directory that is being written to,
>>> SampleDataset in this case)
>>>
>>> Any help in getting this example working would be appreciated.
>>>
>>> Thanks,
>>>
>>> Steve
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Department of Computing, Macquarie University
>>> http://web.science.mq.edu.au/~cassidy/
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/mailinglists/
>>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>
>
>
> --
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tools that make datasets

[Peter van Heusden]

I poked around at your tool XML and the code a bit and the problem is
directory="$job_name". Galaxy expects to collect files from the job's
working directory - basically the current working directory the job runs
in. The directory= argument doesn't have variables expanded as far as I can
tell. In any event it is used in walk_over_extra_files() that is in
lib/galaxy/tools/parameters/output_collect.py - if you look there you see
that it is simply appended to the job's working directory.

So if you use:



(note the  and  - this is effectively the regexp
r"(?P.*)\.txt" with the < and > escaped out)

And alter the code so that it just writes files to the current directory,
then you'll pick up the files one.txt, two,txt and three.txt.

Peter

On 20 October 2015 at 12:28, Steve Cassidy <steve.cass...@mq.edu.au> wrote:

> Sorry, it was just an example of a tool that works - the file that it
> writes out is put into that directory, so I assume that's where my files
> should end up too.
>
> Steve
>
> On 20 October 2015 at 21:12, Peter van Heusden <p...@sanbi.ac.za> wrote:
>
>> Sorry, I don't understand - what does the Upload File tool have to do
>> with this?
>>
>> On 20 October 2015 at 11:49, Steve Cassidy <steve.cass...@mq.edu.au>
>> wrote:
>>
>>> Yes, I'm sure that's where the problem lies. Writing out to the current
>>> directory doesn't work.  The files get written to
>>> 'job_working_directory/000/1/' but if I run the Upload File tool the result
>>> is placed in 'files/000/'.  I think I need to work out where to write the
>>> files, I found some references to $__new_file_path__ but that doesn't seem
>>> to help.
>>>
>>> Steve
>>>
>>>
>>>
>>> On 20 October 2015 at 19:57, Peter van Heusden <p...@sanbi.ac.za> wrote:
>>>
>>>> I suspect that the problem might be in the  then.
>>>> I'm not an export on this, but "__name_and_ext__" turns into the
>>>> regexp r"(?P.*)\.(?P[^\.]+)?" in
>>>> lib/galaxy/tools/parameters/output_collect.py, and is used by the
>>>> DatasetCollector (line 358). This looks like it should match the filenames
>>>> you're creating, but I'm not 100% sure how that code works. One thing I
>>>> notice is the "directory" argument. If you write jobs to the current
>>>> directory instead of "output_path" can you get it to work?
>>>>
>>>> Peter
>>>>
>>>> On 20 October 2015 at 09:52, Steve Cassidy <steve.cass...@mq.edu.au>
>>>> wrote:
>>>>
>>>>> Thanks Peter,
>>>>>   I did see that proviso somewhere but no, refreshing doesn't help.
>>>>>
>>>>> That page was one of those that I referred to getting to this point.
>>>>>
>>>>> Steve
>>>>>
>>>>> On 20 October 2015 at 18:33, Peter van Heusden <p...@sanbi.ac.za>
>>>>> wrote:
>>>>>
>>>>>> Just a quick check - did you refresh your history to confirm that the
>>>>>> dataset *is* empty? We had the same thing at SANBI but it turns out that
>>>>>> Galaxy creates an empty output collection and then only populates it
>>>>>> sometime after job completion (this is a know UI bug).
>>>>>>
>>>>>> See:
>>>>>> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/
>>>>>>
>>>>>> On 20 October 2015 at 08:48, Steve Cassidy <steve.cass...@mq.edu.au>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>   I'm trying to understand how to write a tool that generates a
>>>>>>> dataset rather than a single output file.  I've tried following all of 
>>>>>>> the
>>>>>>> examples but I'm stuck, so I thought I would distil down the simplest
>>>>>>> example I could write and ask for help here.
>>>>>>>
>>>>>>> So here's my example:
>>>>>>>
>>>>>>> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55
>>>>>>>
>>>>>>> it's a simple python script that writes three files to a directory
>>>>>>> named for the single input parameter.
>>>>>>>
>>>>>>> I think one of the problems I'm having is knowing where to write the
>>>>>>> output to. I've run thi

Re: [galaxy-dev] Failure when preparing job on blastp

[Peter van Heusden]

Previous revision of ncbi_blast_plus works like a charm, thanks.

On 21 July 2015 at 19:57, Peter Cock p.j.a.c...@googlemail.com wrote:

 Thanks Peter,

 Yes, you are using the latest release published to
 the Tool Shed, aka v0.1.03 (internal changes):

 https://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/c16c30e9ad5b

 I've filed this as an issue on the GitHub repository,
 https://github.com/peterjc/galaxy_blast/issues/69

 If you are also on GitHub we can discuss this there, where
 I've CC'd John and Bjoern for their thoughts.

 Something simple you could try now is going back to an older
 ncbi_blast_plus via the Tool Shed? The previous release
 2fe07f50a41e aka v0.1.01 used the alternative form
 $db_opts.histdb.extra_files_path rather than the current
 $db_opts.histdb.files_path which appears to be failing.

 Thanks,

 Peter

 On Tue, Jul 21, 2015 at 6:17 PM, Peter van Heusden p...@sanbi.ac.za
 wrote:
  This is release_15.07 (last commit listed is
  39f6730e5f50b54f372b8c1e17d86d8936b5899f) and ncbi_blast_plus
 c16c30e9ad5b.
  So no, its not an old Galaxy.
 
  On 21 July 2015 at 17:37, Peter Cock p.j.a.c...@googlemail.com wrote:
 
  Hi Peter v H,
 
  Which version of Galaxy are you running?
 
  Which version of ncbi_blast_tools are you running? (ideally down to
  the ToolShed revision)
 
  My guess is that since this used to be .extra_files_path perhaps your
  Galaxy is too old for .files_path to work?
 
 
 https://github.com/peterjc/galaxy_blast/commit/2bbbf7f361ada20bf1b5bd777fae6e0cb52c4a38
 
  Thanks,
 
  Peter C.
 
  On Tue, Jul 21, 2015 at 3:08 PM, Peter van Heusden p...@sanbi.ac.za
  wrote:
   Hi there
  
   I'm trying to use the blastp tools from the ncbi_blast_tools, but
 every
   time
   I try and run a job I get the following error:
  
   Traceback (most recent call last):
 File
  
   /net/
 datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/lib/galaxy/jobs/runners/__init__.py
 ,
   line 163, in prepare_job
   job_wrapper.prepare()
 File
  
   /net/
 datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/lib/galaxy/jobs/__init__.py
 ,
   line 859, in prepare
   self.command_line, self.extra_filenames,
 self.environment_variables
   =
   tool_evaluator.build()
 File
  
   /net/
 datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/lib/galaxy/tools/evaluation.py
 ,
   line 422, in build
   raise e
   NotFound: cannot find 'files_path' while searching for
   'db_opts.histdb.files_path'
  
   This is happening when expanding the tool template. The BLAST db in in
   the
   history (thus the reference to histdb) but why can it not find this
   attribute?
  
   Peter
  
   ___
   Please keep all replies on the list by using reply all
   in your mail client.  To manage your subscriptions to this
   and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/
  
   To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/
 
 
 
  ___
  Please keep all replies on the list by using reply all
  in your mail client.  To manage your subscriptions to this
  and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
 
  To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Failure when preparing job on blastp

[Peter van Heusden]

Hi there

I'm trying to use the blastp tools from the ncbi_blast_tools, but every
time I try and run a job I get the following error:

Traceback (most recent call last):
  File /net/
datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/lib/galaxy/jobs/runners/__init__.py,
line 163, in prepare_job
job_wrapper.prepare()
  File /net/
datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/lib/galaxy/jobs/__init__.py,
line 859, in prepare
self.command_line, self.extra_filenames, self.environment_variables =
tool_evaluator.build()
  File /net/
datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/lib/galaxy/tools/evaluation.py,
line 422, in build
raise e
NotFound: cannot find 'files_path' while searching for
'db_opts.histdb.files_path'

This is happening when expanding the tool template. The BLAST db in in the
history (thus the reference to histdb) but why can it not find this
attribute?

Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Troubleshooting

[Peter van Heusden]

Hi Lionel

Sounds like Galaxy does not know what format your FASTQ file is. When you
click on it, what format does it show? Is it simple fastq? And how did you
get it into Galaxy?

You might need to manually specify the format by clicking on the Edit
Attributes button (the pencil icon) and select the Datatype tab. Then set
it to fastqsanger. This should allow the analysis to continue.

On 20 July 2015 at 01:01, Lionel Mavoungou lionel@gmail.com wrote:

 Hello,
 I have a problem when I try to align en FATSQ file for a ChIP-seq.
 I have done a FASTQ grooming because it has been performed on Illumina. No
 my file is a .FASTSANGER.
 However my file is not recognized. I see this : No fastqsanger,
 fastqillumina or fastqsolexa dataset collection available.

 How could I fix it. It seems that my fastq file as well as my fastqsanger
 are not recognized.

 Thanks a lot,

 Lionel

 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Adding packages for bedtools and ucsc-utils for Trackster?

[Peter van Heusden]

Thanks Bjoern

My initial install didn't pick up the Perl dependencies correctly, but I
re-installed and now the install seems to have the required modules but is
missing something in the environment variables. Specifically I get an error
import JSON.pm, see below. Note that the
jbrowse/1.11.6/iuc/package_jbrowse_1_11_6/d5ab749c9e97/lib/perl5 directory
isn't included in @INC and in fact the paths that *are* included seem to
imply that the wrong perl interpreter is being picked up.


Fatal error: Exit code 1 ()
Can't locate JSON.pm in @INC (you may need to install the JSON module)
(@INC contains:
/net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5
/etc/perl /usr/local/lib/perl/5.18.2 /usr/local/share/perl/5.18.2
/usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.18
/usr/share/perl/5.18 /usr/local/lib/site_perl .) at
/net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5/Bio/JBrowse/JSON.pm
line 10.
BEGIN failed--compilation aborted at
/net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5/Bio/JBrowse/JSON.pm
line 10.
Compilation failed in require at
/net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5/Bio/JBrowse/Cmd/FormatSequences.pm
line 21.
BEGIN failed--compilation aborted at
/net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5/Bio/JBrowse/Cmd/FormatSequences.pm
line 21.
Compilation failed in require at bin/prepare-refseqs.pl line 7.
BEGIN failed--compilation aborted at bin/prepare-refseqs.pl line 7.
Traceback (most recent call last):
  File 
/net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/2c9e5136b416/jbrowse/jbrowse.py,
line 44, in module
process_genome(jbrowse_dir, os.path.realpath(args.genome.name))
  File 
/net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/2c9e5136b416/jbrowse/jbrowse.py,
line 17, in process_genome
subprocess.check_output(['perl', 'bin/prepare-refseqs.pl',
'--fasta', genome], cwd=jbrowse_dir)
  File /usr/lib/python2.7/subprocess.py, line 573, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['perl',
'bin/prepare-refseqs.pl', '--fasta',
'/net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/files/000/dataset_3.dat']'
returned non-zero exit status 2


On 18 June 2015 at 10:01, Bjoern Gruening bjoern.gruen...@gmail.com wrote:

  Peter,

 I'm in a hurry ... have a look at
 https://github.com/galaxyproject/tools-iuc/tree/master/tools/jbrowse
 Developed by IUC member Eric Rasche.

 Cheers,
 Bjoern


 On 18.06.2015 10:00, Peter van Heusden wrote:

 I've tried that - I've added the Environment Modules for the relevant
 tools to the local_env.sh file, so they're there in the environment when
 Galaxy runs. These tools are scheduled via our cluster, however, so I don't
 think the environment passes across to them. They depend on these Galaxy
 packages, however, and those (bedtools and ucsc-utils) are now installed,
 so I'm not sure why it can't pick them up from the shed_tools.

 On 18 June 2015 at 09:10, Hans-Rudolf Hotz h...@fmi.ch wrote:



 On 06/18/2015 09:06 AM, Peter van Heusden wrote:

 Hi there

 I'm trying to visualise some of our genome data using Trackster. Initial
 input is 790,000 lines of GFF3. When I try and create the visualisation,
 I get errors from two tools:

 Convert GFF to Interval Index on data 3
 -- an out of memory error  - I'll try and fix this by upping the memory
 limit for our Galaxy jobs

 Convert BED, GFF, or VCF to BigWig on data 3
 -- bedtools and bedGraphToBigWig not found errors. I tried to resolve
 these by installing the ucsc-tools and bedtools packages - and they did
 install correctly - but I still get these file not found errors.


  Hi Peter

 make sure 'galaxy' (ie the user galaxy is running as) has the newly
 installed tools in its path. You might need to add them to run.sh and
 restart the server.


 Hans-Rudolf


  Any idea how to fix this second problem?

 Thanks!
 Peter
 P.S. as an alternative to Trackster, does anyone have Galaxy interfaced
 to JBrowse?


  ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists

Re: [galaxy-dev] Nginx Config

[Peter van Heusden]

Wow that's a lot of upstreams... what is your Galaxy configuration?

On 26 November 2014 at 20:26, Nicholas Kline nx...@case.edu wrote:

 Hi,

 I am trying to setup a local production Galaxy running on Ubuntu,
 Nginx, PostgreSQL, and ProFTD.

 When visiting http://localhost, 502 Bad Gateway is returned.
 Visiting https://localhost returns unable to connect. I was
 expecting to see the Galaxy website instead.

 My Nginx configuration file is included below. Thank you for your help.



 ###
 # /etc/nginx/nginx.conf
 ###

 user www-data;
 worker_processes 4;
 pid /run/nginx.pid;

 events {
 worker_connections 768;
 # multi_accept on;
 }

 http {

 ##
 # Basic Settings
 ##

 sendfile on;
 tcp_nopush on;
 tcp_nodelay on;
 keepalive_timeout 65;
 types_hash_max_size 2048;
 # server_tokens off;

 # server_names_hash_bucket_size 64;
 # server_name_in_redirect off;

 include /etc/nginx/mime.types;
 default_type application/octet-stream;

 ##
 # Logging Settings
 ##

 access_log /var/log/nginx/access.log;
 error_log /var/log/nginx/error.log;

 ##
 # Gzip Settings
 ##

 gzip on;
 gzip_http_version 1.1;
 gzip_vary on;
 gzip_comp_level 4;
 gzip_proxied any;
 gzip_types text/plain text/css application/x-javascript text/xml
 application/xml text/javascript application/json;
 gzip_buffers 16 8k;
 gzip_disable MSIE [1-6].(?!.*SV1);

 ##
 # nginx-naxsi config
 ##
 # Uncomment it if you installed nginx-naxsi
 ##

 #include /etc/nginx/naxsi_core.rules;

 ##
 # nginx-passenger config
 ##
 # Uncomment it if you installed nginx-passenger
 ##

 #passenger_root /usr;
 #passenger_ruby /usr/bin/ruby;

 ##
 # Virtual Host Configs
 ##

 include /etc/nginx/conf.d/*.conf;
 #include /etc/nginx/sites-enabled/*;

 ##
 # Galaxy
 ##

 upstream galaxy_app {
 # As defined in galaxy config file universe_wsgi.ini
 server localhost:8080;
 server localhost:8081;
 server localhost:8082;
 server localhost:8083;
 server localhost:8084;
 server localhost:8085;
 server localhost:8086;
 server localhost:8087;
 server localhost:8088;
 server localhost:8089;
 }

 server {
 # Maximum upload size that can be handled by POST requests
 through nginx.
 # Disabled because using FTP instead of web interface.
 #client_max_body_size 10G;

 location / {
 proxy_pass   http://galaxy_app;
 proxy_set_header   X-Forwarded-Host $host;
 proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
 # If using SSL, makes Galaxy aware of what type of URL it
 should generate for external sites like Biomart.
 proxy_set_header   X-URL-SCHEME https;
 }

 # nginx is more efficient at serving static content.
 # expires 24h is for caching purposes.
 location /static {
 alias /home/galaxy/galaxy-dist/static;
 expires 24h;
 }
 location /static/style {
 alias /home/galaxy/galaxy-dist/static/june_2007_style/blue;
 expires 24h;
 }
 location /static/scripts {
 alias /home/galaxy/galaxy-dist/static/scripts/packed;
 }
 location /favicon.ico {
 alias /home/galaxy/galaxy-dist/static/favicon.ico;
 }
 location /robots.txt {
 alias /home/galaxy/galaxy-dist/static/robots.txt;
 }

 # Use nginx to send files (downloads) instead of Galaxy.
 location /_x_accel_redirect/ {
 internal;
 alias /;
 }
 }
 }



 ###
 # /var/log/nginx/error.log
 ###

 2014/11/25 15:43:21 [error] 56433#0: *23 connect() failed (111:
 Connection refused) while connecting to upstream, client: 127.0.0.1,
 server: , request: GET / HTTP/1.1, upstream:
 http://127.0.0.1:8089/;, host: localhost
 2014/11/25 15:43:21 [error] 56433#0: *23 connect() failed (111:
 Connection refused) while connecting to upstream, client: 127.0.0.1,
 server: , request: GET / HTTP/1.1, upstream:
 http://127.0.0.1:8080/;, host: localhost
 2014/11/25 15:43:21 [error] 56433#0: *23 connect() failed (111:
 Connection refused) while connecting to upstream, client: 127.0.0.1,
 server: , request: GET / HTTP/1.1, upstream:
 http://127.0.0.1:8081/;, host: localhost
 2014/11/25 15:43:21 [error] 56433#0: *23 connect() failed (111:
 Connection refused) while connecting to upstream, client: 127.0.0.1,
 server: , request: GET / HTTP/1.1, upstream:
 http://127.0.0.1:8082/;, host: localhost
 2014/11/25 15:43:21 [error] 56433#0: *23 connect() failed (111:
 Connection refused) while connecting to upstream, client: 127.0.0.1,
 server: , request: GET /