[galaxy-dev] Re: Ansible Galaxy using root user for role installation
It is not actually, in my investigation, coming from an Ansible default. There's code inside the Galaxy role that tries to figure out which user to use for remote_user (the one used in SSH) and when it is run with `become: true` ends up incorrectly choosing root. However people more knowledgeable than me don't have this problem. Perhaps something changed in some version somewhere? Peter On Mon, 15 Feb 2021 at 17:16, Michael Moore wrote: > Anything attempting to login remotely as root must be rejected. Peter's > policy approach is good for that. Most modern linices will not allow a > remote root login, which is correct. There are SSH ways to accomplish the > same thing, but depend on trust at both ends. The defaulting by Ansible is > reprehensible, but usually should be expected to fail. > > > On Mon, Feb 15, 2021 at 5:15 AM Peter van Heusden wrote: > >> Hi there >> >> I found I had this problem too, when the `become: true` applied to the >> whole Galaxy play. My current top level playbook (at >> https://github.com/pvanheus/masters_galaxy_ansible/blob/master/galaxy.yml#L25) >> now applies `become: true` as it is needed, and does not apply it to the >> `galaxyproject.galaxy` role. I found that this avoids Ansible trying to log >> in as the root user. >> >> Peter >> >> On Mon, 15 Feb 2021 at 07:21, Sandra Maksimovic < >> sandra.maksimo...@mcri.edu.au> wrote: >> >>> Hi there, >>> >>> >>> >>> Would anyone happen to know why Ansible tries to connect to a remote >>> host (CentOS 7) using 'root' during Galaxy role installation? >>> >>> >>> >>> My playbook has been configured to connect to the remote host using a >>> different user account via SSH, however, the role installation is not >>> honoring this for some reason and continues to use root. The remote host >>> has had SSH root login disabled for security reasons and setting various >>> galaxy user/privsep/become/remote user variables in the group_vars has not >>> had any effect. >>> >>> >>> >>> TASK [galaxyproject.galaxy : Create galaxy_root] * >>> >>> fatal: [server_name]: UNREACHABLE! => {"changed": false, "msg": >>> "Invalid/incorrect password: Permission denied, please try >>> again.\r\nPermission denied, please try again.\r\nroot@server_name: >>> Permission denied (publickey,password).", "unreachable": true} >>> >>> >>> >>> Thanks, >>> >>> Sandra >>> >>> >>> This e-mail and any attachments to it (the "Communication") are, unless >>> otherwise stated, confidential, may contain copyright material and is for >>> the use only of the intended recipient. If you receive the Communication in >>> error, please notify the sender immediately by return e-mail, delete the >>> Communication and the return e-mail, and do not read, copy, retransmit or >>> otherwise deal with it. Any views expressed in the Communication are those >>> of the individual sender only, unless expressly stated to be those of >>> Murdoch Children’s Research Institute (MCRI) ABN 21 006 566 972 or any of >>> its related entities. MCRI does not accept liability in connection with the >>> integrity of or errors in the Communication, computer virus, data >>> corruption, interference or delay arising from or in respect of the >>> Communication. >>> ___ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> %(web_page_url)s >>> >>> To search Galaxy mailing lists use the unified search at: >>> http://galaxyproject.org/search/ >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> %(web_page_url)s >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/ > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] Re: Ansible Galaxy using root user for role installation
Hi there I found I had this problem too, when the `become: true` applied to the whole Galaxy play. My current top level playbook (at https://github.com/pvanheus/masters_galaxy_ansible/blob/master/galaxy.yml#L25) now applies `become: true` as it is needed, and does not apply it to the `galaxyproject.galaxy` role. I found that this avoids Ansible trying to log in as the root user. Peter On Mon, 15 Feb 2021 at 07:21, Sandra Maksimovic < sandra.maksimo...@mcri.edu.au> wrote: > Hi there, > > > > Would anyone happen to know why Ansible tries to connect to a remote host > (CentOS 7) using 'root' during Galaxy role installation? > > > > My playbook has been configured to connect to the remote host using a > different user account via SSH, however, the role installation is not > honoring this for some reason and continues to use root. The remote host > has had SSH root login disabled for security reasons and setting various > galaxy user/privsep/become/remote user variables in the group_vars has not > had any effect. > > > > TASK [galaxyproject.galaxy : Create galaxy_root] * > > fatal: [server_name]: UNREACHABLE! => {"changed": false, "msg": > "Invalid/incorrect password: Permission denied, please try > again.\r\nPermission denied, please try again.\r\nroot@server_name: > Permission denied (publickey,password).", "unreachable": true} > > > > Thanks, > > Sandra > > > This e-mail and any attachments to it (the "Communication") are, unless > otherwise stated, confidential, may contain copyright material and is for > the use only of the intended recipient. If you receive the Communication in > error, please notify the sender immediately by return e-mail, delete the > Communication and the return e-mail, and do not read, copy, retransmit or > otherwise deal with it. Any views expressed in the Communication are those > of the individual sender only, unless expressly stated to be those of > Murdoch Children’s Research Institute (MCRI) ABN 21 006 566 972 or any of > its related entities. MCRI does not accept liability in connection with the > integrity of or errors in the Communication, computer virus, data > corruption, interference or delay arising from or in respect of the > Communication. > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > %(web_page_url)s > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: %(web_page_url)s To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] _remove_dead_weakref error even after deleting .venv (Galaxy 18.09)
I solved this problem by deleting the virtualenv in $GALAXY_ROOT and recreating it from one of my cluster worker nodes. System python on the worker nodes is 2.7.15rc1 from the Ubuntu 18.04 repository. System python on the Galaxy server is 2.7.12 from the Ubuntu 16.04 repository. Since the virtual environment used by Galaxy contains symbolic links to system python modules, it seems there is a dependency between the virtual environment and system python? I tried different methods for creating the virtual env but nothing worked except for creating it on one of the cluster nodes. I look forward to testing the virtualenv-in-conda method that Marius PRed in here ( https://github.com/galaxyproject/galaxy/pull/6975) in the future. One final "gotcha" I discovered - creating the virtualenv with `bash run.sh` does not install `watchdog`, a module which is necessary if watching the tool directories for changes is enabled. In certain cases this can lead to an exception. Peter On Tue, 11 Dec 2018 at 17:15, Peter van Heusden wrote: > Hi there > > I'm trying to run Windowmasker in Galaxy and receiving the following error: > > "computing the genome length > pass 1 > pass 2 > optimizing the data structure > Traceback (most recent call last): > File > "/tools/software/galaxy/galaxy1/database/jobs/006/6476/set_metadata_bUlLkt.py", > line 1, in > from galaxy_ext.metadata.set_metadata import set_metadata; > set_metadata() > File > "/tools/software/galaxy/galaxy1/lib/galaxy_ext/metadata/set_metadata.py", > line 14, in > import logging > File "/usr/lib/python2.7/logging/__init__.py", line 26, in > import sys, os, time, cStringIO, traceback, warnings, weakref, > collections > File "/usr/lib/python2.7/weakref.py", line 14, in > from _weakref import ( > ImportError: cannot import name _remove_dead_weakref" > > The usual suggested fix for this is to remove the Galaxy virtual > environment (.venv) and recreated it. I have done that but the error > persists. Galaxy is running here on Ubuntu 16.04 with Python 2.7.12. Any > advice? > > Thanks! > Peter > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] _remove_dead_weakref error even after deleting .venv (Galaxy 18.09)
Hi there I'm trying to run Windowmasker in Galaxy and receiving the following error: "computing the genome length pass 1 pass 2 optimizing the data structure Traceback (most recent call last): File "/tools/software/galaxy/galaxy1/database/jobs/006/6476/set_metadata_bUlLkt.py", line 1, in from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata() File "/tools/software/galaxy/galaxy1/lib/galaxy_ext/metadata/set_metadata.py", line 14, in import logging File "/usr/lib/python2.7/logging/__init__.py", line 26, in import sys, os, time, cStringIO, traceback, warnings, weakref, collections File "/usr/lib/python2.7/weakref.py", line 14, in from _weakref import ( ImportError: cannot import name _remove_dead_weakref" The usual suggested fix for this is to remove the Galaxy virtual environment (.venv) and recreated it. I have done that but the error persists. Galaxy is running here on Ubuntu 16.04 with Python 2.7.12. Any advice? Thanks! Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] Adding the Ilifu cloud to launch.usegalaxy.org
Hi there While I will continue to work on getting a Cloudlaunch server running locally, could the Ilifu cloud (http://www.ilifu.ac.za/) be added as an option on https://launch.usegalaxy.org/catalog ? What details would be required? And then - what are the requirements for adding appliances to the appliance catalogue? Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] Running Cloudlaunch against a new OpenStack cloud
Hi there I've got a bit further, but now I've got a problem - how do you access the admin panel when you have the complete site set up? 1) If Debug = False (as it is for a production setup), Django doesn't load static files (e.g. CSS). 2) When going via the nginx proxy, I can't figure out what URL will direct me to the admin page. Peter P.S. I'm doing some work on the ansible to get this all set up - I'll do a PR once everything seems to be working. On Fri, 19 Oct 2018 at 01:25, Alexandru Mahmoud wrote: > Hello! > > I am new to the team, so this will be a brief response to only one of your > questions while waiting for someone else to get to the email (and I am not > replying all so that people don’t dismiss it as answered): > > For 3 specifically, you can access the admin panel at http://127.0.0.1(or > localhost):8000/admin, where you should be able to login with the user > created with the ‘createsuperuser’ command. Under “Appliances” or > “Applications” (depending on the version of cloudlaunch) you’ll be able to > see and change/add configurations for apps. > If you want to see in more detail how the appliances are being set up and > how the launch config is used in code, look for the "Backend component > name", then find the corresponding backend plugin in the cloudlaunch repo. > I hope that is a helpful start, and feel free to email me if nobody > responds in a reasonable amount of time to your other questions, as I can > get more answers in person and relay them to you. > > Best, > Alex Mahmoud > > > On Oct 18, 2018, at 3:45 PM, Peter van Heusden wrote: > > Hi there > > I'd like to set up cloudlaunch to be able to set up on-demand Galaxy > servers and associated clusters on the new Ilifu OpenStack cloud. From > looking around the repositories it looks like Cloudlaunch is at least 2 > components: > > 1. The cloudlaunch server > 2. The UI - either cloudlaunch-ui or cloudlaunch-cli > > I'm trying to get cloudlaunch server installed on a Ubuntu 18.04 VM. The > only requirement seems to be python3-pip. For postgresql support I > installed the python3-psycopg2 module, the postgresql-10 server and create > a user and db (both named cloud). Then I installed the server with `pip3 > install cloudlaunch-server` and ran commands: > > cloudlaunch-server django migrate > cloudlaunch-server django createsuperuser > cloudlaunch-server django runserver > > The next step (https://github.com/galaxyproject/cloudlaunch) refers to > redis-server. Installing this package seemed to satisfy that requirement, > running it manually didn't work so I just ran the celery command as > illustrated. And finally relaunched the django server. > > Then nodejs and npm were installed, the cloudlaunch-ui cloned from github, > the various npm package specified installed and finally npm start brings up > the cloudlaunch user interface. > > So far so good but: > > 1. Is there more extensive documentation anywhere? Specifically > documentation about running this in production. > > 2. Is there a docker container that can be used instead of all this manual > installation? > > 3. How are appliances configured and added to the Appliance Catalogue? > > Thanks! > Peter > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/ > > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] regarding galaxy kickstart
I don't have an answer to your 2. but Galaxy Kickstart sets up Galaxy controlled by supervisord. So you can restart e.g. the Galaxy web front end with: sudo supervisorctl restart galaxy:galaxy_web On Tue, 16 Oct 2018 at 08:16 Rui Wang wrote: > Hi Folks, > > I spent sometime playing with the kickstart, and sort of got it working. > Still a couple things not clear: > > 1. Is it true that there is no start/stop? My guess is that whenever you > need to restart, just re-run the playbook? Anything you want to update(for > example add new tools), you re-run the updated playbook too, correct? > Before you re-run it, do you have to manually kill everything? > > 2. Seems tools could easily added by editing the tool list. How about > genome data? What if I want to add a specific repo or genome data? Is there > a role for this purpose? > > Please give me a hand if you could. Any input will be greatly appreciated. > :-) > > Thanks so much, > Rui > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] Running Cloudlaunch against a new OpenStack cloud
Hi there I'd like to set up cloudlaunch to be able to set up on-demand Galaxy servers and associated clusters on the new Ilifu OpenStack cloud. From looking around the repositories it looks like Cloudlaunch is at least 2 components: 1. The cloudlaunch server 2. The UI - either cloudlaunch-ui or cloudlaunch-cli I'm trying to get cloudlaunch server installed on a Ubuntu 18.04 VM. The only requirement seems to be python3-pip. For postgresql support I installed the python3-psycopg2 module, the postgresql-10 server and create a user and db (both named cloud). Then I installed the server with `pip3 install cloudlaunch-server` and ran commands: cloudlaunch-server django migrate cloudlaunch-server django createsuperuser cloudlaunch-server django runserver The next step (https://github.com/galaxyproject/cloudlaunch) refers to redis-server. Installing this package seemed to satisfy that requirement, running it manually didn't work so I just ran the celery command as illustrated. And finally relaunched the django server. Then nodejs and npm were installed, the cloudlaunch-ui cloned from github, the various npm package specified installed and finally npm start brings up the cloudlaunch user interface. So far so good but: 1. Is there more extensive documentation anywhere? Specifically documentation about running this in production. 2. Is there a docker container that can be used instead of all this manual installation? 3. How are appliances configured and added to the Appliance Catalogue? Thanks! Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] Error when trying to update tool vcf_to_alignment in testtoolshed
Hi there I had a failed updated (network problem) of the sanbi-uwc tool vcf_to_alignment and now when I try and do an update I get: $ planemo shed_update --shed_target testtoolshed cd '/home/pvh/Documents/code/SANBI/tools-sanbi-uwc/tools/vcf_to_alignment' && git rev-parse HEAD cd '/home/pvh/Documents/code/SANBI/tools-sanbi-uwc/tools/vcf_to_alignment' && git diff --quiet Could not update vcf_to_alignment ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)) Repository metadata updated successfully for repository 'vcf_to_alignment' on the test Tool Shed. Failed to update repository contents for repository 'vcf_to_alignment' on the test Tool Shed. Nothing that I try on my side seems to resolve this. Any ideas? Is the problem on the toolshed side? Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] Using Singularity images to resolve dependences in Galaxy 18.05
Thanks Björn! So it is a job_conf thing rather than a dependency manager config thing? >From the job_conf.xml example I am not clear how it expects to find the container images? How is the search path configured, what naming scheme is used for containers? And does it "pull" containers automatically (like the conda dependency manager does) or should I pre-populate the system with container images? Thanks, Peter On Wed, 19 Sep 2018 at 10:15 Björn Grüning wrote: > Hi Peter, > > yes this is working :) > Usegalaxy.eu is using it for a few tools. > > Documentation is here ;) > > https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced#L484 > > Also the Docker compose image is can also use Singularity if you want to > reverse engineer it :) > > Ciao, > Bjoern > > Am 19.09.2018 um 10:12 schrieb Peter van Heusden: > > Hi there > > > > Is it possible to use Singularity images to resolve dependencies in > > Galaxy 18.05? If so is there documentation somewhere? > > > > I have a Galaxy server where for at least some of the dependencies conda > > will not be a good option so I would like to be able to use singularity > > if at all possible. > > > > Thanks! > > Peter > > > > > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > >https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > >http://galaxyproject.org/search/ > > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] Using Singularity images to resolve dependences in Galaxy 18.05
Hi there Is it possible to use Singularity images to resolve dependencies in Galaxy 18.05? If so is there documentation somewhere? I have a Galaxy server where for at least some of the dependencies conda will not be a good option so I would like to be able to use singularity if at all possible. Thanks! Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"
Yes that was my conclusion, thanks Peter. As a result I switched to a new table named (shapeit2_ref) and a new tool name (data_manager_shapeit2_reference) and from planemo side upload seemed to succeed but the tool is in an error state. I resorted to manually editing the tool-data table. Data managers are a bit of an ugly duckling part of Galaxy - not receiving as much love as other kinds of tools. Peter On Tue, 18 Sep 2018 at 14:22 Peter Cock wrote: > Hi Peter vH, > > I wonder if adding columns is not supported - because some tools > would define the table without the extra column? You might have > to define a new data table instead (with largely the same content). > But to be clear, I am not an expert in this area of Galaxy. > > However, it does seem the IUC could lay out some *.loc file best > practises for tool authors. There is currently a TODO entry here: > > > https://galaxy-iuc-standards.readthedocs.io/en/latest/best_practices/tool_xml.html#data-managers > > Peter C. > > On Mon, Sep 17, 2018 at 10:56 PM Peter van Heusden > wrote: > >> A bit more tracing leads me to suspect that what is happening is that a >> merge is being called here: >> >> >> https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/data/__init__.py#L120 >> >> from: >> >> >> https://github.com/galaxyproject/galaxy/blob/dev/lib/tool_shed/tools/data_table_manager.py#L88 >> >> i.e. the toolshed maintains an in-memory copy of tool-data tables and if >> the list of columns in an incoming tool-data table is not the same as the >> in-memory version the error I saw happens. >> >> Which leaves me with the question: how is one meant to update the column >> list of a tool data table? >> >> Peter >> >> >> ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"
A bit more tracing leads me to suspect that what is happening is that a merge is being called here: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/data/__init__.py#L120 from: https://github.com/galaxyproject/galaxy/blob/dev/lib/tool_shed/tools/data_table_manager.py#L88 i.e. the toolshed maintains an in-memory copy of tool-data tables and if the list of columns in an incoming tool-data table is not the same as the in-memory version the error I saw happens. Which leaves me with the question: how is one meant to update the column list of a tool data table? Peter On Mon, 17 Sep 2018 at 13:23 Peter van Heusden wrote: > Thanks - which is probably because of the previous failures with > `shed_update` - which relate back to the topic of my original email. I > suspect that failure from `planemo shed_update` has led to this "not > installable" state. And I don't know how to solve the problem from > `shed_update` which *looks* like it is coming from a clash between previous > and current definitions of the tool-data table. > > Peter > > On Mon, 17 Sep 2018 at 12:58 Nicola Soranzo > wrote: > >> Hi Peter, >> if you open the page for data_manager_shapeit_reference in the test >> ToolShed you will see that revision 5 and 6 are not installable, they both >> have "*This revision can be installed:* False" . If you instead select >> revision 3 it is installable. >> >> Cheers, >> Nicola >> >> >> On 17/09/18 11:51, Peter van Heusden wrote: >> >> I'm following up on my previous email. I somehow managed to get `planemo >> shed_update` to work but now I seem to have an inconsistent view of the >> Test toolshed. This is how it looks from the Admin panel of my Galaxy >> server: >> >> [image: https://files.gitter.im/galaxyproject/Lobby/k2qH/image.png] >> Notice that it shows 3 installable revisions for the data manager, last >> updated 13th of September. And this is how it looks from the Test toolshed >> website: >> >> [image: https://files.gitter.im/galaxyproject/Lobby/ymY9/image.png] >> >> >> Notice that it has 6 installabale revisions for the data manager. Is my >> Galaxy server perhaps caching something? >> >> Peter >> >> >> On Sat, 15 Sep 2018 at 09:57 Peter van Heusden wrote: >> >>> Hi there >>> >>> I'm working on a data manager to keep track of the reference data >>> collection used by the SHAPEIT2 tool. I had to add a column to the data >>> manager but when I try and do a `planemo shed_update` I get the error: >>> >>> Unexpected HTTP status code: 500: {"err_msg": "Error attempting to parse >>> file tool_data_table_conf.xml.sample: Merging tabular data tables with non >>> matching columns is not allowed: shapeit_ref:{'path': 2, >>> 'reference_prefix': 3, 'name': 1, 'value': 0, 'map_prefix': 4} != >>> shapeit_ref:{'name': 1, 'value': 0, 'sample_prefix': 5, 'path': 2, >>> 'reference_prefix': 3, 'map_prefix': 4}"} >>> >>> This is for data_manager_shapeit_reference in testtoolshed, which is on >>> github at >>> >>> https://github.com/pvanheus/data_manager_shapeit_reference >>> >>> As far as I can tell the columns do accord. Is there any way to see >>> which files are being merged to help try and debug this error? >>> >>> Thanks! >>> Peter >>> >> >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/ >> >> >> ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"
Thanks - which is probably because of the previous failures with `shed_update` - which relate back to the topic of my original email. I suspect that failure from `planemo shed_update` has led to this "not installable" state. And I don't know how to solve the problem from `shed_update` which *looks* like it is coming from a clash between previous and current definitions of the tool-data table. Peter On Mon, 17 Sep 2018 at 12:58 Nicola Soranzo wrote: > Hi Peter, > if you open the page for data_manager_shapeit_reference in the test > ToolShed you will see that revision 5 and 6 are not installable, they both > have "*This revision can be installed:* False" . If you instead select > revision 3 it is installable. > > Cheers, > Nicola > > > On 17/09/18 11:51, Peter van Heusden wrote: > > I'm following up on my previous email. I somehow managed to get `planemo > shed_update` to work but now I seem to have an inconsistent view of the > Test toolshed. This is how it looks from the Admin panel of my Galaxy > server: > > [image: https://files.gitter.im/galaxyproject/Lobby/k2qH/image.png] > Notice that it shows 3 installable revisions for the data manager, last > updated 13th of September. And this is how it looks from the Test toolshed > website: > > [image: https://files.gitter.im/galaxyproject/Lobby/ymY9/image.png] > > > Notice that it has 6 installabale revisions for the data manager. Is my > Galaxy server perhaps caching something? > > Peter > > > On Sat, 15 Sep 2018 at 09:57 Peter van Heusden wrote: > >> Hi there >> >> I'm working on a data manager to keep track of the reference data >> collection used by the SHAPEIT2 tool. I had to add a column to the data >> manager but when I try and do a `planemo shed_update` I get the error: >> >> Unexpected HTTP status code: 500: {"err_msg": "Error attempting to parse >> file tool_data_table_conf.xml.sample: Merging tabular data tables with non >> matching columns is not allowed: shapeit_ref:{'path': 2, >> 'reference_prefix': 3, 'name': 1, 'value': 0, 'map_prefix': 4} != >> shapeit_ref:{'name': 1, 'value': 0, 'sample_prefix': 5, 'path': 2, >> 'reference_prefix': 3, 'map_prefix': 4}"} >> >> This is for data_manager_shapeit_reference in testtoolshed, which is on >> github at >> >> https://github.com/pvanheus/data_manager_shapeit_reference >> >> As far as I can tell the columns do accord. Is there any way to see which >> files are being merged to help try and debug this error? >> >> Thanks! >> Peter >> > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/ > > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"
I'm following up on my previous email. I somehow managed to get `planemo shed_update` to work but now I seem to have an inconsistent view of the Test toolshed. This is how it looks from the Admin panel of my Galaxy server: [image: https://files.gitter.im/galaxyproject/Lobby/k2qH/image.png] Notice that it shows 3 installable revisions for the data manager, last updated 13th of September. And this is how it looks from the Test toolshed website: [image: https://files.gitter.im/galaxyproject/Lobby/ymY9/image.png] Notice that it has 6 installabale revisions for the data manager. Is my Galaxy server perhaps caching something? Peter On Sat, 15 Sep 2018 at 09:57 Peter van Heusden wrote: > Hi there > > I'm working on a data manager to keep track of the reference data > collection used by the SHAPEIT2 tool. I had to add a column to the data > manager but when I try and do a `planemo shed_update` I get the error: > > Unexpected HTTP status code: 500: {"err_msg": "Error attempting to parse > file tool_data_table_conf.xml.sample: Merging tabular data tables with non > matching columns is not allowed: shapeit_ref:{'path': 2, > 'reference_prefix': 3, 'name': 1, 'value': 0, 'map_prefix': 4} != > shapeit_ref:{'name': 1, 'value': 0, 'sample_prefix': 5, 'path': 2, > 'reference_prefix': 3, 'map_prefix': 4}"} > > This is for data_manager_shapeit_reference in testtoolshed, which is on > github at > > https://github.com/pvanheus/data_manager_shapeit_reference > > As far as I can tell the columns do accord. Is there any way to see which > files are being merged to help try and debug this error? > > Thanks! > Peter > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] Problems installing Galaxy 18.01
Thanks to Marius and others, I now have a working Galaxy server, including CVMFS support, as documented in the complete blog post: http://pvh.wp.sanbi.ac.za/2018/03/18/galaxy-18-01-install/ The problem I was encountering with installations was related to incorrect environment configuration, which led to the `hg` binary not being found in the PATH and thus the install failing. Another fault led to this failure being a silent failure. I've linked to the working configuration in the blog, but here is a direct link: https://gist.github.com/pvanheus/589f84507a26ffa49e3bdf3bd938f25c It is rather too unwieldy to paste here. Note that while the says [program:web], there is only one section in the supervisord configuration. The server is running in a uWSGI + Mules configuration. Thanks! Peter On Mon, 19 Mar 2018 at 11:17 Peter van Heusden <p...@sanbi.ac.za> wrote: > First to answer your questions: > > I'm on release_18.01 branch, > commit c58ca749c26bbb3c4ca8750f2c37fd1bb6bd7999. > > Thanks for the note about nginx-extras, I've installed that and it fixes > the upload > directives problem. > > Then on to the main problem. I used "Get Data" as the default location > because I'm really just testing the install. The install goes from "New" > status to "Error" status pretty much immediately: > > [image: image.png] > > This is not a bowtie2-specific problem, I get the same behaviour when > trying to install fastqc. > > Thanks for your reply, > Peter > P.S. running already-installed tools works > > On Mon, 19 Mar 2018 at 09:44 Marius van den Beek <m.vandenb...@gmail.com> > wrote: > >> Hi Peter, >> >> does the tool show up in the tool panel (Looks like you've installed it >> into "Get Data")? >> What commit are you on ? There were some last minute bugfixes to the >> release_18.01 >> prior to the release, so if the checkout is a few days old an upgrade >> (and an uninstall/install cycle) might help. >> >> Also you mentioned the lack of the upload_store in nginx in nginx, you >> need to install nginx-extras, not nginx. >> >> Hope that helps, >> Marius >> >> On 18 March 2018 at 21:17, Peter van Heusden <p...@sanbi.ac.za> wrote: >> >>> Hi there >>> >>> My newly installed Galaxy 18.01 server seems to be unable to install >>> bowtie (the devteam version). I'm documenting my installation progress at: >>> >>> http://pvh.wp.sanbi.ac.za/2018/03/18/galaxy-18-01-install/ >>> >>> As noted in the last paragraph: >>> >>> "My next test was to try and install the bowtie2 tool from the toolshed. >>> This failed, with the tool status going from New to Error with not much >>> interesting left in the logs." >>> >>> The log during the time when I'm trying to install bowtie2 is: >>> >>> >>>1. [pid: 8856|app: 0|req: 35/48] 10.8.0.2 () {48 vars in 1108 bytes} >>>[Sun Mar 18 19:14:55 2018] GET >>>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install >>> => >>>generated 650 bytes in 26 msecs (HTTP/1.1 302) 3headers in 269 bytes >>>(1 switches on core 1) >>>2. 10.8.0.2 - - [18/Mar/2018:19:14:57 +] "GET >>>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install >>>HTTP/1.1" 302 - " >>> >>> https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b; >>>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like >>>Gecko) Chrome/65.0.3325.146 Safari/537.36" >>>3. [pid: 8856|app: 0|req: 37/49] 10.8.0.2 () {48 vars in 1108 bytes} >>>[Sun Mar 18 19:14:57 2018] GET >>>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install >>> => >>>generated 650 bytes in 24 msecs (HTTP/1.1 302) 3headers in 269 bytes >>>(1 switches on core 0) >>>4. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,375 [p:8856,w:2,m:0] [ >>>uWSGIWorker2Core2] Dependency bowtie2 not found. >>>5. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,376 [p:8856,w:2,m:0] [ >>>uWSGIWorker2Core2] Dependency samtools not found. >>>6. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [ >>>uWSGIWorker2Core3] Dependency bowtie2 not found. >>>7. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [ >>>uWSGIWorker2Core3] Dependency samtools not found. >>>8. 10.8.0.2 - - [18/Mar/2018:19:14:55 +] "GET >>> >>> /admin_toolshed/prepare_for_i
Re: [galaxy-dev] Problems installing Galaxy 18.01
First to answer your questions: I'm on release_18.01 branch, commit c58ca749c26bbb3c4ca8750f2c37fd1bb6bd7999. Thanks for the note about nginx-extras, I've installed that and it fixes the upload directives problem. Then on to the main problem. I used "Get Data" as the default location because I'm really just testing the install. The install goes from "New" status to "Error" status pretty much immediately: [image: image.png] This is not a bowtie2-specific problem, I get the same behaviour when trying to install fastqc. Thanks for your reply, Peter P.S. running already-installed tools works On Mon, 19 Mar 2018 at 09:44 Marius van den Beek <m.vandenb...@gmail.com> wrote: > Hi Peter, > > does the tool show up in the tool panel (Looks like you've installed it > into "Get Data")? > What commit are you on ? There were some last minute bugfixes to the > release_18.01 > prior to the release, so if the checkout is a few days old an upgrade (and > an uninstall/install cycle) might help. > > Also you mentioned the lack of the upload_store in nginx in nginx, you > need to install nginx-extras, not nginx. > > Hope that helps, > Marius > > On 18 March 2018 at 21:17, Peter van Heusden <p...@sanbi.ac.za> wrote: > >> Hi there >> >> My newly installed Galaxy 18.01 server seems to be unable to install >> bowtie (the devteam version). I'm documenting my installation progress at: >> >> http://pvh.wp.sanbi.ac.za/2018/03/18/galaxy-18-01-install/ >> >> As noted in the last paragraph: >> >> "My next test was to try and install the bowtie2 tool from the toolshed. >> This failed, with the tool status going from New to Error with not much >> interesting left in the logs." >> >> The log during the time when I'm trying to install bowtie2 is: >> >> >>1. [pid: 8856|app: 0|req: 35/48] 10.8.0.2 () {48 vars in 1108 bytes} [Sun >>Mar 18 19:14:55 2018] GET >>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install => >>generated 650 bytes in 26 msecs (HTTP/1.1 302) 3headers in 269 bytes ( >>1 switches on core 1) >>2. 10.8.0.2 - - [18/Mar/2018:19:14:57 +] "GET >>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install >>HTTP/1.1" 302 - " >> >> https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b; >>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko >>) Chrome/65.0.3325.146 Safari/537.36" >>3. [pid: 8856|app: 0|req: 37/49] 10.8.0.2 () {48 vars in 1108 bytes} [Sun >>Mar 18 19:14:57 2018] GET >>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install => >>generated 650 bytes in 24 msecs (HTTP/1.1 302) 3headers in 269 bytes ( >>1 switches on core 0) >>4. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,375 [p:8856,w:2,m:0] [ >>uWSGIWorker2Core2] Dependency bowtie2 not found. >>5. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,376 [p:8856,w:2,m:0] [ >>uWSGIWorker2Core2] Dependency samtools not found. >>6. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [ >>uWSGIWorker2Core3] Dependency bowtie2 not found. >>7. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [ >>uWSGIWorker2Core3] Dependency samtools not found. >>8. 10.8.0.2 - - [18/Mar/2018:19:14:55 +] "GET >> >> /admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F% >>2Ftoolshed.g2.bx.psu.edu%2F HTTP/1.1" 200 - " >> >> https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b; >>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko >>) Chrome/65.0.3325.146 Safari/537.36" >>9. [pid: 8856|app: 0|req: 38/50] 10.8.0.2 () {48 vars in 1274 bytes} [Sun >>Mar 18 19:14:55 2018] GET >> >> /admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F% >>2Ftoolshed.g2.bx.psu.edu%2F => generated 19062 bytes in 7001 msecs ( >>HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 2) >>10. 10.8.0.2 - - [18/Mar/2018:19:14:57 +] "GET >> >> /admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F% >>2Ftoolshed.g2.bx.psu.edu%2F HTTP/1.1" 200 - " >> >> https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b; >>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko &
[galaxy-dev] Problems installing Galaxy 18.01
Hi there My newly installed Galaxy 18.01 server seems to be unable to install bowtie (the devteam version). I'm documenting my installation progress at: http://pvh.wp.sanbi.ac.za/2018/03/18/galaxy-18-01-install/ As noted in the last paragraph: "My next test was to try and install the bowtie2 tool from the toolshed. This failed, with the tool status going from New to Error with not much interesting left in the logs." The log during the time when I'm trying to install bowtie2 is: 1. [pid: 8856|app: 0|req: 35/48] 10.8.0.2 () {48 vars in 1108 bytes} [Sun Mar 18 19:14:55 2018] GET /admin_toolshed/manage_repository?id=f597429621d6eb2b=install => generated 650 bytes in 26 msecs (HTTP/1.1 302) 3headers in 269 bytes (1 switches on core 1) 2. 10.8.0.2 - - [18/Mar/2018:19:14:57 +] "GET /admin_toolshed/manage_repository?id=f597429621d6eb2b=install HTTP/1.1" 302 - " https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36" 3. [pid: 8856|app: 0|req: 37/49] 10.8.0.2 () {48 vars in 1108 bytes} [Sun Mar 18 19:14:57 2018] GET /admin_toolshed/manage_repository?id=f597429621d6eb2b=install => generated 650 bytes in 24 msecs (HTTP/1.1 302) 3headers in 269 bytes (1 switches on core 0) 4. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,375 [p:8856,w:2,m:0] [ uWSGIWorker2Core2] Dependency bowtie2 not found. 5. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,376 [p:8856,w:2,m:0] [ uWSGIWorker2Core2] Dependency samtools not found. 6. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [ uWSGIWorker2Core3] Dependency bowtie2 not found. 7. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [ uWSGIWorker2Core3] Dependency samtools not found. 8. 10.8.0.2 - - [18/Mar/2018:19:14:55 +] "GET /admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F% 2Ftoolshed.g2.bx.psu.edu%2F HTTP/1.1" 200 - " https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36" 9. [pid: 8856|app: 0|req: 38/50] 10.8.0.2 () {48 vars in 1274 bytes} [Sun Mar 18 19:14:55 2018] GET /admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F% 2Ftoolshed.g2.bx.psu.edu%2F => generated 19062 bytes in 7001 msecs (HTTP/ 1.1 200) 2 headers in 73 bytes (1 switches on core 2) 10. 10.8.0.2 - - [18/Mar/2018:19:14:57 +] "GET /admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F% 2Ftoolshed.g2.bx.psu.edu%2F HTTP/1.1" 200 - " https://galaxy.sanbi.ac.za/admin_toolshed/manage_repository?id=f597429621d6eb2b; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36" 11. [pid: 8856|app: 0|req: 38/51] 10.8.0.2 () {48 vars in 1274 bytes} [Sun Mar 18 19:14:57 2018] GET /admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F% 2Ftoolshed.g2.bx.psu.edu%2F => generated 19062 bytes in 5987 msecs (HTTP/ 1.1 200) 2 headers in 73 bytes (1 switches on core 3) 12. galaxy.tools.toolbox.base DEBUG 2018-03-18 19:15:14,128 [p:8846,w:1 ,m:0] [uWSGIWorker1Core1] Appending to tool panel section: Get Data 13. galaxy.tools.toolbox.base DEBUG 2018-03-18 19:15:14,143 [p:8846,w:1 ,m:0] [uWSGIWorker1Core1] Appending to tool panel section: Get Data 14. 10.8.0.2 - - [18/Mar/2018:19:15:11 +] "POST /admin_toolshed/prepare_for_install HTTP/1.1" 200 - " https://galaxy.sanbi.ac.za/admin_toolshed/prepare_for_install?changeset_revisions=dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146Safari/537.36" 15. [pid: 8846|app: 0|req: 14/52] 10.8.0.2 () {56 vars in 1329 bytes} [Sun Mar 18 19:15:11 2018] POST /admin_toolshed/prepare_for_install => generated 11133 bytes in 2509 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 1) 16. galaxy.tools.toolbox.base DEBUG 2018-03-18 19:15:15,296 [p:8856,w:2 ,m:0] [uWSGIWorker2Core1] Appending to tool panel section: Get Data 17. 10.8.0.2 - - [18/Mar/2018:19:15:15 +] "POST /admin_toolshed/install_repositories HTTP/1.1" 200 - " https://galaxy.sanbi.ac.za/admin_toolshed/prepare_for_install; "Mozilla/ 5.0 (X11; Linux x86_64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/65.0 .3325.146 Safari/537.36" 18. [pid: 8856|app: 0|req: 39/53] 10.8.0.2 () {54 vars in 1152 bytes} [Sun Mar 18 19:15:15 2018] POST /admin_toolshed/install_repositories => generated 20 bytes in 206 msecs (HTTP/1.1
Re: [galaxy-dev] Improved DRMAAJobRunner (Peter van Heusden)
Hi Matthias My job_conf.xml still has the old config of running upload locally, so it has a section after the section. Are you using a proxy such as nginx with your Galaxy server? Thanks, Peter On Mon, 10 Jul 2017 at 11:50 Matthias Bernt <m.be...@ufz.de> wrote: > Hi Peter, > > FYI: just found on the galaxy documentation a paragraph that even > recommends running upload on the cluster: > > https://galaxyproject.org/admin/config/performance/cluster/ > > """ > If your cluster nodes have Internet access (NAT is okay) and you want to > run the data source tools (upload, ucsc, etc.) on the cluster (doing so > is highly recommended), set new_file_path in galaxy.ini to a directory > somewhere in your shared filesystem: > """ > > Cheers, > Matthias > > > On 10.07.2017 11:11, Matthias Bernt wrote: > > Dear Peter, > > > > my job_conf.xml is attached. > > > > > > Best, > > Matthuas > > > > On 08.07.2017 18:00, galaxy-dev-requ...@lists.galaxyproject.org wrote: > >> Send galaxy-dev mailing list submissions to > >> galaxy-dev@lists.galaxyproject.org > >> > >> To subscribe or unsubscribe via the World Wide Web, visit > >> https://lists.galaxyproject.org/listinfo/galaxy-dev > >> or, via email, send a message with subject or body 'help' to > >> galaxy-dev-requ...@lists.galaxyproject.org > >> > >> You can reach the person managing the list at > >> galaxy-dev-ow...@lists.galaxyproject.org > >> > >> When replying, please edit your Subject line so it is more specific > >> than "Re: Contents of galaxy-dev digest..." > >> > >> > >> HEY! This is important! If you reply to a thread in a digest, please > >> 1. Change the subject of your response from "Galaxy-dev Digest Vol > >> ..." to the original subject for the thread. > >> 2. Strip out everything else in the digest that is not part of the > >> thread you are responding to. > >> > >> Why? > >> 1. This will keep the subject meaningful. People will have some idea > >> from the subject line if they should read it or not. > >> 2. Not doing this greatly increases the number of emails that match > >> search queries, but that aren't actually informative. > >> > >> Today's Topics: > >> > >> 1. Re: any introduce for how Mako and JavaScript work ingalaxy? > >>(Daniel Blankenberg) > >> 2. Re: ustacks - bad interpreter (Björn Grüning) > >> 3. Re: Improved DRMAAJobRunner (Peter van Heusden) (Matthias Bernt) > >> 4. Re: Improved DRMAAJobRunner (Peter van Heusden) > >>(Peter van Heusden) > >> > >> > >> -- > >> > >> Message: 1 > >> Date: Thu, 6 Jul 2017 22:36:27 -0400 > >> From: Daniel Blankenberg <dan.blankenb...@gmail.com> > >> To: Steven Shen <ishenwei...@gmail.com> > >> Cc: galaxy-dev@lists.galaxyproject.org > >> Subject: Re: [galaxy-dev] any introduce for how Mako and JavaScript > >> work ingalaxy? > >> Message-ID: > >> <CAE9nRFg+Edj2=3+K=6+FZCxj5=hb4togw+lm++e+sd1+j2+...@mail.gmail.com > > > >> Content-Type: text/plain; charset="utf-8" > >> > >> Hi Steven, > >> > >> For a quick first pass, have you taken a look at https://github.com/ > >> galaxyproject/galaxy/blob/dev/client/README.md? > >> > >> > >> Thanks for using Galaxy, > >> > >> Dan > >> > >> On Thu, Jul 6, 2017 at 10:11 PM, Steven Shen <ishenwei...@gmail.com> > >> wrote: > >> > >>> Hi everyone, > >>> > >>> I want to make some change for my local galaxy web, actually I can edit > >>> galaxy_path/static/scripts/bundled/*.js directly. > >>> > >>> It seems mako templates are used for galaxy, as well as many JavaScript > >>> files, but I don't know how they work. So is there any introduce for > how > >>> mako and JavaScript files working in galaxy? > >>> > >>> Thank you very much. > >>> > >>> Steven > >>> > >>> ___ > >>> Please keep all replies on the list by using "reply all" > >>> in your mail client. To manage your subscriptions to this > >>> and other Galax
Re: [galaxy-dev] Improved DRMAAJobRunner (Peter van Heusden)
On Sat, 8 Jul 2017 at 11:56 Matthias Berntwrote: > Hi Peter, > > > Code looks interesting. This is for Univa Grid Engine? > > In its current state its tailored for the Univa Grid Engine. But the drmaa > library code that is based on job_info() and wait() should run on all grid > engines. > For the command line based code changes might be necessary for the > different grid engines. (Currently there is only a small bug, because > wait() is currently > called twice for finished jobs: in the repeated polling and then in the > final check. The second call won't work which causes a fallback to qacct. I > will fix this > soon.) > I guess that the output of qstat and qacct will be different. But I think > this can be configured one just needs a way to get the info which grid > engine is running. > > > In terms of the upload jobs, are those not designed to be run as 'local' > jobs and not with the 'real user' setting? > > Sounds reasonable, but this is not what is happening on my installation of > galaxy. Any idea where I could start to find the problem. > > I'm not an expert on this, but what does your job_conf.xml look like? Peter > Best, > Matthias ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
Re: [galaxy-dev] Improved DRMAAJobRunner
Code looks interesting. This is for Univa Grid Engine? In terms of the upload jobs, are those not designed to be run as 'local' jobs and not with the 'real user' setting? On Fri, 7 Jul 2017 at 17:32 Matthias Berntwrote: > Dear list, > > I have implemented a new DRMAA job runner allows to detect runtime and > memory violations. This is done by using the drmaa library functions > job_info + wait or as fallback the commandline tools qstat + qacct. This > has the additional advantage that the runner works also in setups using > the external runner scripts to start jobs as real user (the original > DRMAAJobRunner can not query these jobs at all). > > You may have a look here: > > https://github.com/galaxyproject/galaxy/pull/4275 > > I have successfully tested the but settings: galaxy user/real user > submits the jobs. Also the resubmission in case of memory and time > violations seems to work. > > Would be great to get some comments. > > One problem that I encountered is that upload jobs do not work in the > real user setting (for the original and the new runner): The permissions > of the uploaded file in > /gpfs1/data/galaxy_server/galaxy-dev/database/tmp/ are not changed. Any > idea what needs to be changed to get this running? > > Best, > Matthias > > -- > > --- > Matthias Bernt > Bioinformatics Service > Molekulare Systembiologie (MOLSYB) > Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/ > Helmholtz Centre for Environmental Research GmbH - UFZ > Permoserstraße 15, 04318 Leipzig, Germany > Phone +49 341 235 482296 <+49%20341%20235482296>, > m.be...@ufz.de, www.ufz.de > > Sitz der Gesellschaft/Registered Office: Leipzig > Registergericht/Registration Office: Amtsgericht Leipzig > Handelsregister Nr./Trade Register Nr.: B 4703 > Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: > MinDirig Wilfried Kraus > Wissenschaftlicher Geschäftsführer/Scientific Managing Director: > Prof. Dr. Dr. h.c. Georg Teutsch > Administrative Geschäftsführerin/ Administrative Managing Director: > Prof. Dr. Heike Graßmann > --- > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] "Best practice" for maintaining genome list in Galaxy
Hi there Is there current "best practice" for maintaining a collection of reference genome data in a Galaxy server? I.e. filling in the dbkeys.loc and all_fasta.loc table and associated data. I know of the following discussions on this topic: 1. The data integration page: https://galaxyproject.org/admin/data-integration/ 2. The data managers page: https://galaxyproject.org/admin/tools/data-managers/ 3. This biostars answer: https://biostar.usegalaxy.org/p/7176/ 4. This page on custom genomes that seems to be user / history oriented: https://galaxyproject.org/learn/custom-genomes/ 5. This page on rsyncing from the Galaxy reference data collection: https://galaxyproject.org/admin/use-galaxy-rsync/ - quick test gives me rsync error though 6. This guide to using the human reference data: https://biostar.usegalaxy.org/p/14777/ 7. I hear the usegalaxy.org reference collection is available via cvmfs - I think this was discussed in a Galaxy Admins meetup at some point. None of this is comprehensive. There are 2 sets of questions for me (in my role as Galaxy admin): 1. If I want to make reference genomes and common indices (like for HISAT and BWA) available for e.g. human and mouse, what is the best way to do this? 2. If I want to add a genome for a non-model organism (e.g. M. tuberculosis or L. calcarifer) available, what is the best way to do this? Which data manager should or could I use? Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] composite datatypes: uploading and optional files
Hi there We've got a composite datatype (neostore) for Neo4j databases. Currently the upload is disabled because to upload you need to select each individual file of the database directory (which is tedious). Is there a more efficient way to upload composite datatypes? E.g. as a tar.gz or zip file? If not, has anyone looked into supporting this? (I see @jmchilton was noted as working on a composite uploading for 17.05) Secondly, we're moving to support Neo4j 3.2.0. In this version there is a minor (1 file) change in the files used to store the database. Can composite datatypes have "optional files"? Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/
[galaxy-dev] Perl module problem with vcftools in conda + new tool
Hi there I'm trying to add a wrapper for the vcftools vcf-consensus tool. There is an existing tool collection for vcftools in tools-devteam, so I'm add it there. Wrapper is at: https://github.com/pvanheus/tools-devteam/tree/master/tool_collections/vcftools/vcftools_consensus I tested this using: planemo conda_install . planemo serve --conda_prefix /home/pvh/miniconda2 --con da_dependency_resolution When the tool executes I get the error: Fatal error: Exit code 2 () Can't locate Vcf.pm in @INC Which implies that the PERL5LIB is not being set correctly when the tool runs. Using planemo test gives me: No tests were executed - see Galaxy output for details. Strangely enough on the pull request the Travis test passes ( https://github.com/galaxyproject/tools-devteam/pull/402). Any idea how to fix this? Thanks! Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Loading a GUI on galaxy
Hi Galaxy supports visualisation plugins and then it also supports interactive environments. I don't know how WEKA VisualizaPanel works - is it a web application? Does it require a server? On Fri, 26 Aug 2016 at 20:52 Lijo Johnwrote: > Hello All, > I want to know, whether it is possible to embed graphical user interface > like that of WEKA VisualizePanel tool in galaxy ? > I am able to load the GUI on a local instance of galaxy. But when i am > trying to access the galaxy server through a remote connection, this GUI is > not able to load on remote desktop. > > Any comments or suggestions would be of great interest ! > -- > Regards, > Lijo John > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Travis CI for Galaxy repositories?
Hi everyone We maintain a tools collection on Github for tools that are not yet (or will never be) added to tools-iuc: https://github.com/SANBI-SA/tools-sanbi-uwc I tried copying the .travis.yml from tools-iuc to our repository, and it largely works, except for the planemo upload part, which fails with this error: Repository created cd '/home/pvh/Documents/code/SANBI/pilon/tools-sanbi-uwc/tools/pilon' && git rev-parse HEAD cd '/home/pvh/Documents/code/SANBI/pilon/tools-sanbi-uwc/tools/pilon' && git diff --quiet Could not update pilon Unexpected response from galaxy: 500: {"content_alert": "", "err_msg": "Metadata may have been defined for some items in revision 'f2477672a950'. Correct the following problems if necessary and reset metadata.pilon.xml<\/b> - This file requires an entry in the tool_data_table_conf.xml file. Upload a file named tool_data_table_conf.xml.sample to the repository that includes the required entry to correct this error."} This is related to the pilon tool in that repo (tools/pilon). The tool in question optionally uses a dataset from all_fasta, I'm not sure if that is relevant for the error. BTW this error happened on push, not on PR. Any ideas? Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Parameter Storing in Database
You mean the stuff for the tool-data tables? As I understand it, it is loaded in: lib/galaxy/tools/data/__init__.py There's a ToolDataTableManager and a TabularToolDataTable (with associated TabularToolDataField). Peter On Fri, 5 Aug 2016 at 20:46 Katherine Beaulieu < katherine.beaulieu...@gmail.com> wrote: > Would anyone be able to tell me in what file the storing of tool > parameters into the database occurs? > Thanks! > Katherine > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] run-as-user and Galaxy 16.04
Hi there Does anyone have the run-as-user support working with Galaxy 16.04? If so, could you describe your config? Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Building tool dependencies
Actually, for recent Galaxy, I'd recommend making a conda recipe for UMI-tools and contributing it to bioconda: https://bioconda.github.io/ Then your tool can just depend on this and the conda dependency resolver (default in Galaxy since 16.01) can find it. Others can comment on the recent work on conda/toolshed integration. Peter On 5 July 2016 at 08:56, Matthias De Smetwrote: > > > > > Hi > > > > I’m trying to put together a tool + dependency package to use umi-tools ( > https://github.com/CGATOxford/UMI-tools) from Galaxy. > > I’m at a loss on how I should organise this. > > > > The package is available from either github or pip. Which one is > preferable to use? How do I make sure the package dependencies are > installed? I’ve found the and installing it with type="shell_command">python setup.py install. This, however, tries > to install the package to /usr/bin/ which isn’t desirable (obviously). > > > > Thanks a lot! > > M > > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Mapping requirements to conda recipe names (granularity?)
Hi there I'm trying to satisfy requirements on our RNA-seq oriented Galaxy server at SANBI using conda/bioconda. There's a disconnect between the names used in tool dependency tags and the names of conda recipes though, for example: 1) the BAM to BigWig converter (tool ID CONVERTER_bam_to_bigwig_0) has a requirement for ucsc_tools. The tool it needs, bedGraphToBigWig, is provided by the ucsc-bedgraphtobigwig package, so in this case ucsc_tools is a broader requirement than what bioconda provides. 2) the featurecounts tool requires featurecounts, which is part of the subreads recipe, so in this case featurecounts is a narrow requirement than what is provided in bioconda. In the second case, one could say subreads provides the featurecounts requirement. In the first case one could say the tool requires a narrower requirement than what it specifies, although the bioconda recipes split a single upstream package (ucsc_tools) into multiple recipes that invariable will be updated together, a somewhat strange choice. Right now I'm resolving these problems by manually creating environments (e.g. __package__featurecounts@__unversioned__) that will get picked up by the conda dependency resolver, but I'd like to hear from the list what the best way to support this is? Probably the easiest is keeping the tool requirements definition in line with what the conda recipe is called, but in some cases a mapping table between requirements and conda recipe names could be useful. Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Auto detection of output format Galaxy custom tool
Hi Katherine I presume this is related to the tool described here ( https://biostar.usegalaxy.org/p/18319/)? I think there is general interest in the creation of a tool like this - see the recent email from Miu ki Yip. If you could share the code and XML (e.g. via GitHub) perhaps we could assist with what you are trying to write. Peter On 30 June 2016 at 19:01, Peter Cockwrote: > Maybe sharing your tool XML file would be best - is it on GitHub? > > Peter > > On Wed, Jun 29, 2016 at 7:43 PM, Katherine Beaulieu > wrote: > > Ok, so I don't give the user the option to select the correct data type > to > > reduce the amount of stuff the user has to know when using the tool. > > > > I do leave the format as auto and Galaxy just leaves it as the generic > data > > type 'data'. > > > > As for how the format is defined in Galaxy, I'm not quite sure what the > > answer to this question is because I am just testing the tool with really > > simple text files, with extension .txt. and it can't seem to pick it up, > it > > just leaves it as 'data'. Also not quite sure how to test the Python > code in > > the sniff function outside of Galaxy... (only starting working with > Galaxy 3 > > weeks ago). > > > > Let me know if I should provide you with the tool config file if that > would > > be a little more useful. It's quite straightforward > > Thanks for helping with this. > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Implementing iRODS with Galaxy
Hi Miuki There's two aspects to iRODS and Galaxy. Firstly, there is the Galaxy object store which can store objects in iRODS. This is entirely hidden from the user and they have no idea that their datasets are in iRODS. Some work on that happened earlier in this year, see: https://wiki.galaxyproject.org/Community/Log/2015/GalaxyIrodsConnection Then there is the question of iRODS import/export. See this thread from earlier this year: http://dev.list.galaxyproject.org/iRODS-td4669076.html - Iyad Kandalaft (http://www.goc411.ca/en/8192/Iyad-Kandalaft) mentioned some future work on import/export. As part of the B3Africa project some colleagues of ours are also investigating iRODS import/export. I think it might be useful to make a GitHub issue describing the problem and have people comment on there. I had a brief look at the iRODS python client library and it seems to be in a relatively early state of development ( https://github.com/irods/python-irodsclient) but it is definitely usable. On 29 June 2016 at 21:40, Yip, Miu kiwrote: > Hello Galaxy Team, > > We're from the Bioinformatics Shared Resources (BSR) at Cold Spring Harbor > Labs and we're looking to implement irods on a newly installed version of > Galaxy. > > We want to incorporate iRODS (http://irods.org/). Is there a recommended > way to do this, or no? > > Thanks in advance for the advice. > > Best, > Miuki Yip > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Infinite recursion on conda dependency manager
Hi there I've got a package in our lab conda channel (its still in testing phase) called goget. This is specified as a requirement for the build_ctb_gene tool like this: goget When the tool runs, I see infinite recursion on the conda side: https://gist.github.com/pvanheus/c16a07a2c954741a311881cc7606df8d Any ideas? Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] The state of the MOTHUR merge?
Thanks Björn, no rush! Is anyone here doing 16S on Galaxy? I've been looking into options, and FastQC, Trimmomatic, PEAR and the MOTHUR tools seem to give a viable workflow. Other options are QIIME or integrating the UPARSE pipeline as I understand. Any comments appreciated! Thanks, Peter On 3 June 2016 at 15:53, Björn Grüning <bjoern.gruen...@gmail.com> wrote: > Me finishing my re review and pressing the merge button. > I tend to merge as it is at the moment and do some small adjustments > afterwards ... This weekend Peter, this weekend! :) > > Am 03.06.2016 um 13:37 schrieb Peter van Heusden: > > PR #429 on tools-iuc contains the updated MOTHUR toolsuite for Galaxy. > > > > https://github.com/galaxyproject/tools-iuc/pull/429 > > > > It *looks* like it is ready to merge - what is still needed before it > > merges into tools-iuc? > > > > Thanks! > > Peter > > > > > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ > > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] The state of the MOTHUR merge?
PR #429 on tools-iuc contains the updated MOTHUR toolsuite for Galaxy. https://github.com/galaxyproject/tools-iuc/pull/429 It *looks* like it is ready to merge - what is still needed before it merges into tools-iuc? Thanks! Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Scaling tools: data available? (Peter van Heusden)
Thanks Damion We've got the collectl based metric collection stuff working and that dumps data in the Galaxy DB (which you can pull out using the API). Thanks, Peter On 17 March 2016 at 10:45, Dooley, Damion <damion.doo...@bccdc.ca> wrote: > If you are looking for a simple python program to log cpu% and memory of > another executable, then analyze.py might do. Its in > https://github.com/Public-Health-Bioinformatics/kipper/tree/master/RDP-test > -case along with a few scripts that show examples of its use. > > Damion > >> > >> On 02/21/2016 11:13 PM, Peter van Heusden wrote: > >> > >> Hi there > >> > >> We're researching the resource (CPU time and memory) requirements for > >>RNA > >> STAR at the moment. Specifically, we'd like to build up a database of > >>input > >> size to resource usage so that we can use this to feed a dynamic > >> destination mapper (currently our STAR configuration uses a thumbsuck > >>for > >> RAM and CPU requirements that is not optimal but works in many cases). > >> > >> Has anyone collected these metrics before? And more generally, is anyone > >> collecting metrics for a range of bioinformatics tools? A database like > >> this could be very useful! > >> > >> Thanks! > >> Peter > >> > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] planemo update failing on testtoolshed
The repo is empty, so there is nothing to delete at this stage. And I can't figure out how to delete the thing entirely. I looked at the splitFasta tool, and also the splitfasta tool from rnateam (in the test toolshed) and these split into 1 file for each sequence as opposed to a certain number of chunks. Peter On 18 March 2016 at 21:11, Björn Grüning <bjoern.gruen...@gmail.com> wrote: > Hi Peter, > > can you try to delete files with the web interface? > > I saw the tool name, do you know this small tool? > > https://github.com/bgruening/galaxytools/blob/master/tools/splitfasta/splitFasta.xml > > Cheers, > Bjoern > > Am 18.03.2016 um 20:03 schrieb Peter van Heusden: > > Hi there > > > > I'm trying to update a newly created tool on the test tool shed and I > > get this error: > > > > $ planemo shed_update -t testtoolshed > > cd '/home/pvh/PycharmProjects/split_fasta_chunks' && git rev-parse HEAD > > cd '/home/pvh/PycharmProjects/split_fasta_chunks' && git diff --quiet > > Could not update split_fasta_chunks > > Unexpected response from galaxy: 500: > > Internal Server Error > > > > Internal Server Error > > The server has either erred or is incapable of performing > > the requested operation. > > > > > > > > > > WSGI Server > > > > > > > > Repository metadata updated. > > Failed to update a repository. > > > > I might have accidentally created a "bad" repository that is leading to > > this error. This is with version 0.23.0. Alternately, is there a way to > > delete the tool so that I can try and recreate it? > > > > Thanks! > > Peter > > > > > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ > > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] planemo update failing on testtoolshed
Hi there I'm trying to update a newly created tool on the test tool shed and I get this error: $ planemo shed_update -t testtoolshed cd '/home/pvh/PycharmProjects/split_fasta_chunks' && git rev-parse HEAD cd '/home/pvh/PycharmProjects/split_fasta_chunks' && git diff --quiet Could not update split_fasta_chunks Unexpected response from galaxy: 500: Internal Server Error Internal Server Error The server has either erred or is incapable of performing the requested operation. WSGI Server Repository metadata updated. Failed to update a repository. I might have accidentally created a "bad" repository that is leading to this error. This is with version 0.23.0. Alternately, is there a way to delete the tool so that I can try and recreate it? Thanks! Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Scaling tools: data available?
Hi Eric I'm having a look at the GRT. We're collecting metrics on our local Galaxy server, I'm just trying to understand how you envision these metrics getting into the GRT? Peter On 22 February 2016 at 07:31, Eric Rasche <e...@tamu.edu> wrote: > Hi peter, > > I've been working on this in the past but got caught up in other projects. > There's work for a server-side component here (maybe 50% done), the client > side stuff could all be done through collectl but I was having issues > actually collecting collectl data that had put the project on hold > temporarily. > > https://github.com/erasche/galactic-radio-telescope > > I'd be very, very interested in finishing this project up. > > Cheers, > Eric > > > On 02/21/2016 11:13 PM, Peter van Heusden wrote: > > Hi there > > We're researching the resource (CPU time and memory) requirements for RNA > STAR at the moment. Specifically, we'd like to build up a database of input > size to resource usage so that we can use this to feed a dynamic > destination mapper (currently our STAR configuration uses a thumbsuck for > RAM and CPU requirements that is not optimal but works in many cases). > > Has anyone collected these metrics before? And more generally, is anyone > collecting metrics for a range of bioinformatics tools? A database like > this could be very useful! > > Thanks! > Peter > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > > > -- > -- > Eric Rasche > Programmer II > > Center for Phage Technology > Rm 312A, BioBio > Texas A University > College Station, TX 77843 > 404-692-2048 > e...@tamu.edu > Not responding quickly enough? > <https://calendar.google.com/calendar/embed?src=esr%40tamu.edu=America/Chicago> > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] novoalign in Galaxy
Hi there We're busy integrating novoalign in our local Galaxy - there doesn't seem to be a tool around for this, but I recall some people saying they were using (single threaded) novoalign with Galaxy. Does anyone know if tool XML has already been written? If not, we'll just have to write it. Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Broken package_perl_5_18, was Re: JBrowse on dev branch: progress
I knew there was an email that I forgot to send! Nicola Soranzo fixed this with galaxy PR https://github.com/galaxyproject/galaxy/pull/1775 - sometime in November last year Eric added whitespace to the perl package (and a bunch of other things) that broke the download_file action, so Nicola's patch strips whitespace. (The conversation about this happened on IRC and the PR) I tested the PR and it makes the perl install work (which makes the JBrowse installl work, yay!). Peter On 25 Feb 2016 6:10 PM, "Peter Briggs" <peter.bri...@manchester.ac.uk> wrote: > Hello Peter > > Re the failing install of 'package_perl_5_18', I've investigated further > now and have submitted a patch which seems to fix the problem: > > https://github.com/galaxyproject/tools-iuc/pull/616 > > Not sure if this is of any immediate help, > > Best wishes > > Peter > > On 22/02/16 09:43, Peter van Heusden wrote: > >> Ok, second attempt, perl install died with the same error, so I suspect >> you're right about the cause. Eric? >> >> Peter >> >> On 22 February 2016 at 11:41, Peter van Heusden <p...@sanbi.ac.za >> <mailto:p...@sanbi.ac.za>> wrote: >> >> Thanks Peter >> >> I managed, on the second attempt, to get past the gnu coreutils >> problem. Zipho noticed a weird lockup though: conftest that seemed >> to be from the gnu coreutils install wanted to listen on port 4001, >> which is the same port our uwsgi server listens on. The install has >> now completed but I'll try and find time to go back and verify this. >> >> On the second attempt perl seems to be installing, so I'll see if it >> completes or hits the error you're mentioning. >> >> Peter >> >> On 22 February 2016 at 11:34, Peter Briggs >> <peter.bri...@manchester.ac.uk >> <mailto:peter.bri...@manchester.ac.uk>> wrote: >> >> Hi Peter >> >> I think that package_perl_5_18 in the tool shed is broken, I >> came across the same error last week on both our production >> instance and a local 'test' instance (both v15.10). >> >> My suspicion is that the following lines are to blame: >> >> ... >> > >> sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b" >> type="download_file"> >> >> http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz >> >> tar xfvz >> local-lib-1.008009.tar.gz >> ... >> >> Looking at the Galaxy internals I believe that the >> 'download_file' action automatically unpacks the archive and >> cd's into the resulting directory, hence the failure for the >> subsequent 'tar xfvz ...' command. But I haven't had the time >> yet to verify this or create a patch to see if the removal of >> this line addresses the problem. >> >> I don't think I've seen the problem with >> 'package_gnu_coreutils_8_22' that you describe however so it may >> be that my hypothesis is not correct. >> >> HTH >> >> Best wishes >> >> Peter >> >> >> On 22/02/16 05:10, Peter van Heusden wrote: >> >> Hi there Eric and others >> >> I tried to install the JBrowse tool on a dev branch server >> running on an >> Ubuntu 14.04 VM. >> >> It got stuck at the package_gnu_coreutils_8_22 dependency >> (revision >> ac64dfe4b1fb) - this ended up in "Installing tool >> dependencies" state. >> >> If I go look at the tool details it highlights errors >> in package_perl_5_18 which in turn refers to errors in >> gnu_coreutils >> (just "Error" state) and then this more informative failure >> for perl: >> >> "tar (child): local-lib-1.008009.tar.gz: Cannot open: No >> such file or >> directory >> tar (child): Error is not recoverable: exiting now >> tar: Child returned status 2 >> tar: Error is not recoverable: exiting now" >> >> I wonder if this is a sign of a download error? >> >> Digging a bit more I find the error "Shutting down process >> id
Re: [galaxy-dev] JBrowse on dev branch: progress
Ok, second attempt, perl install died with the same error, so I suspect you're right about the cause. Eric? Peter On 22 February 2016 at 11:41, Peter van Heusden <p...@sanbi.ac.za> wrote: > Thanks Peter > > I managed, on the second attempt, to get past the gnu coreutils problem. > Zipho noticed a weird lockup though: conftest that seemed to be from the > gnu coreutils install wanted to listen on port 4001, which is the same port > our uwsgi server listens on. The install has now completed but I'll try and > find time to go back and verify this. > > On the second attempt perl seems to be installing, so I'll see if it > completes or hits the error you're mentioning. > > Peter > > On 22 February 2016 at 11:34, Peter Briggs <peter.bri...@manchester.ac.uk> > wrote: > >> Hi Peter >> >> I think that package_perl_5_18 in the tool shed is broken, I came across >> the same error last week on both our production instance and a local 'test' >> instance (both v15.10). >> >> My suspicion is that the following lines are to blame: >> >> ... >> > sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b" >> type="download_file"> >> http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz >> >> tar xfvz >> local-lib-1.008009.tar.gz >> ... >> >> Looking at the Galaxy internals I believe that the 'download_file' action >> automatically unpacks the archive and cd's into the resulting directory, >> hence the failure for the subsequent 'tar xfvz ...' command. But I haven't >> had the time yet to verify this or create a patch to see if the removal of >> this line addresses the problem. >> >> I don't think I've seen the problem with 'package_gnu_coreutils_8_22' >> that you describe however so it may be that my hypothesis is not correct. >> >> HTH >> >> Best wishes >> >> Peter >> >> >> On 22/02/16 05:10, Peter van Heusden wrote: >> >>> Hi there Eric and others >>> >>> I tried to install the JBrowse tool on a dev branch server running on an >>> Ubuntu 14.04 VM. >>> >>> It got stuck at the package_gnu_coreutils_8_22 dependency (revision >>> ac64dfe4b1fb) - this ended up in "Installing tool dependencies" state. >>> >>> If I go look at the tool details it highlights errors >>> in package_perl_5_18 which in turn refers to errors in gnu_coreutils >>> (just "Error" state) and then this more informative failure for perl: >>> >>> "tar (child): local-lib-1.008009.tar.gz: Cannot open: No such file or >>> directory >>> tar (child): Error is not recoverable: exiting now >>> tar: Child returned status 2 >>> tar: Error is not recoverable: exiting now" >>> >>> I wonder if this is a sign of a download error? >>> >>> Digging a bit more I find the error "Shutting down process id 21094 >>> because it generated no output for the defined timeout period of 3600.0 >>> seconds." for gnu_coreutils. >>> >>> I'm trying to do a reinstall of the gnu_coreutils and perl packages to >>> see if I can get further. >>> >>> Peter >>> >>> >>> ___ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>>https://lists.galaxyproject.org/ >>> >>> To search Galaxy mailing lists use the unified search at: >>>http://galaxyproject.org/search/mailinglists/ >>> >>> >> -- >> Peter Briggs peter.bri...@manchester.ac.uk >> Bioinformatics Core Facility University of Manchester >> B.1083 Michael Smith Bldg Tel: (0161) 2751482 >> > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Plans for workflow & parallelisation work?
Hi there I see from the PR landing in Galaxy and the comments on things like issue #1701 (https://github.com/galaxyproject/galaxy/issues/1701) that there's lots of work happening on the workflow side of Galaxy. This is an area of interest at SANBI too, so we'd like to coordinate development efforts as much as possible. To this end: 1) Are there forks to track so we can see what new code is landing? 2) Is there a roadmap for workflow work or perhaps can we have a Hangout to talk about this? 3) Specifically in terms of workflows and parallelisation: are there any plans to work on running workflows as opposed to just generating lots of jobs? I know this is a major change to how Galaxy works - it would mean something like submitting a workflow specification to a job runner that is located on the cluster, and then returning the results of workflow execution. 4) Currently parallelisation in Galaxy is supported using two mechanisms: collections and dataset splitters/tasks. Are there plans on extending and harmonising Galaxy's parallelisation capabilities? Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Scaling tools: data available?
Hi there We're researching the resource (CPU time and memory) requirements for RNA STAR at the moment. Specifically, we'd like to build up a database of input size to resource usage so that we can use this to feed a dynamic destination mapper (currently our STAR configuration uses a thumbsuck for RAM and CPU requirements that is not optimal but works in many cases). Has anyone collected these metrics before? And more generally, is anyone collecting metrics for a range of bioinformatics tools? A database like this could be very useful! Thanks! Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy broken after update to 16.01
Mattias, what does your supervisord config look like? Specifically, does it contain an environment variable to direct the Galaxy handler to the correct virtualenv? Peter On 9 February 2016 at 16:03, Matthias De Smetwrote: > Hi all, > > > We were tracking the galaxy master branch on our local server and are > looking to migrate to v16.01. > The galaxy processes are managed bij supervisor as shown here ( > https://wiki.galaxyproject.org/Admin/Config/Performance/Scaling). > After updating all dependencies in our venv we tried to restart the server > but got the following issue: > > Loading paste environment: > config:/home/galaxy/galaxy-dist/config/galaxy.ini > Traceback (most recent call last): > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", > line 247, in loadapp > return loadobj(APP, uri, name=name, **kw) > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", > line 271, in loadobj > global_conf=global_conf) > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", > line 296, in loadcontext > global_conf=global_conf) > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", > line 320, in _loadconfig > return loader.get_context(object_type, name, global_conf) > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", > line 458, in get_context > section) > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", > line 517, in _context_from_explicit > value = import_string(found_expr) > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", > line 22, in import_string > return pkg_resources.EntryPoint.parse("x=" + s).load(False) > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/pkg_resources/__init__.py", > line 2380, in load > return self.resolve() > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/pkg_resources/__init__.py", > line 2386, in resolve > module = __import__(self.module_name, fromlist=['__name__'], level=0) > File "lib/galaxy/web/buildapp.py", line 5, in > from galaxy.webapps.galaxy.buildapp import app_factory > File "lib/galaxy/webapps/galaxy/buildapp.py", line 15, in > import galaxy.app > File "lib/galaxy/app.py", line 9, in > import galaxy.queues > File "lib/galaxy/queues.py", line 9, in > from kombu import Exchange, Queue, Connection > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/kombu/__init__.py", > line 67, in __getattr__ > module = __import__(object_origins[name], None, None, [name]) > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/kombu/entity.py", > line 10, in > from .abstract import MaybeChannelBound, Object > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/kombu/abstract.py", > line 12, in > from .connection import maybe_channel > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/kombu/connection.py", > line 19, in > from kombu import exceptions > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/kombu/exceptions.py", > line 12, in > from amqp import ChannelError, ConnectionError, ResourceError > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/amqp/__init__.py", > line 34, in > from .connection import Connection # noqa > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/amqp/connection.py", > line 40, in > from .transport import create_transport > File > "/home/galaxy/galaxy-dist/.venv/lib/python2.7/site-packages/amqp/transport.py", > line 21, in > import ssl > File "/Shared/Software/Cellar/python/2.7.11/lib/python2.7/ssl.py", line > 97, in > import _ssl # if we can't import it, let the error > propagate > ImportError: > /home/galaxy/galaxy-dist/.venv/lib/python2.7/lib-dynload/_ssl.so: undefined > symbol: SSL_CTX_set_alpn_protos > > So far we have tried: > - rebuilding the venv > - reinstalling python > > Funny thing is, when we start the configuration with the “run.sh” script > instead of through supervisor everything seems to work just fine. > > We’re at a loss, please help! > Thanks! > Matthias > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to
Re: [galaxy-dev] Trimmomatic doesn't recognise pair collection
Hi Peter We found that it was a problem in the Galaxy code - it was fixed in the some of the patches that went towards the 16.01 release. Thanks, Peter On 4 February 2016 at 13:47, Peter Briggs <peter.bri...@manchester.ac.uk> wrote: > Hello Peter > > I'm the author of the Trimmomatic tool on the toolshed (i.e. > https://toolshed.g2.bx.psu.edu/view/pjbriggs/trimmomatic) - apologies for > not responding sooner. > > The tool works for me on our production server (v15.10 from github) when > using a pair of fastqs as input, by which I mean that it allows me to > select a fastq-pair dataset as input and produces 2 fastq-pair datasets. > (I'm haven't tried on a list of pairs.) > > What do you see when you say that the tool 'is not working'? I'm wondering > if it is a problem with the tool itself, or something to do with the > underlying Galaxy infrastructure when using the head of the dev branch). > > Best wishes > > Peter > > > On 22/01/16 13:53, Peter van Heusden wrote: > >> Hi there >> >> I've got Trimmomatic installed (from the toolshed) into our local Galaxy >> server (server is latest commit on the dev branch from GitHub). It >> refuses to see a dataset pair as a valid input, however. >> >> I've tested this with two types: firstly a pair made out of 2 individual >> datasets (both fastqsanger) and then also a list of pairs. I've tested >> this with 'planemo serve' and the tool works, but for some reason it is >> not working on our server. Any ideas? >> >> Thanks, >> Peter >> >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >>https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >>http://galaxyproject.org/search/mailinglists/ >> >> > -- > Peter Briggs peter.bri...@manchester.ac.uk > Bioinformatics Core Facility University of Manchester > B.1083 Michael Smith Bldg Tel: (0161) 2751482 > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] planemo reports test failure when test succeeds
Zipho looked at this over here and had some ideas about how some warning output might trigger this spurious failure - I haven't had time to follow up on that theory. On 4 February 2016 at 14:26, Peter Briggs <peter.bri...@manchester.ac.uk> wrote: > Hello again Peter > > I also see this when using planemo to run the tool tests both on > trimmomatic and also on another of our tools i.e. one or more of the tests > will be reported as 'failed' on the command line when the HTML report > indicates that all tests executed successfully. > > Now I think about it, planemo itself must exit with status 0 in these > cases as my tests on Travis-CI also report success, for example see: > > https://travis-ci.org/fls-bioinformatics-core/galaxy-tools/jobs/104643315 > > Scroll to the bottom of the log and the last test is reported as 'failed', > however the Travis test overall passes. I hadn't really thought about it > before since the planemo exit status seems to be correct (i.e. in cases > where there are genuine tool test failures, it will return a non-zero > status at the end which causes Travis to correctly report a failure). > > (Btw this referenced above used planemo 0.22.2 but I've seen it in older > versions too.) > > I don't think I've ever seen planemo report a 'passed' test as 'failed', > but generally I check the HTML to verify whether the failures are real or > not. > > Best wishes > > Peter > > > On 22/01/16 11:40, Peter van Heusden wrote: > >> Hi there >> >> When testing the latest trimmomatic tool (from >> >> https://github.com/fls-bioinformatics-core/galaxy-tools/tree/master/trimmomatic >> , >> planemo test reports an error on the last test, but according to the >> test report the test has succeeded (see attached HTML and JSON). For the >> test setup you need to set some environment variables: >> >> TRIMMOMATIC_DIR - the location of the trimmomatic jar >> TRIMMOMATIC_JAR - the name of the jar file >> TRIMMOMATIC_ADAPTERS_DIR - location of adapters >> >> This is with planemo 0.22.2 and a fresh $HOME/.planemo. >> >> Any ideas? >> >> Peter >> >> >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >>https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >>http://galaxyproject.org/search/mailinglists/ >> >> > -- > Peter Briggs peter.bri...@manchester.ac.uk > Bioinformatics Core Facility University of Manchester > B.1083 Michael Smith Bldg Tel: (0161) 2751482 > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Working on the admin interface: JS vs Mako
Yeah I noticed that about the DependencyResolversView. In terms of writing JS - where's a good example to look at? I tried digging around the client/ directories but couldn't make sense of how it works. Thanks, Peter On 1 February 2016 at 16:48, John Chilton <jmchil...@gmail.com> wrote: > Peter, > > We would like to replace all the mako with JS, if I was going to put > a bunch of effort into admin pages I'd start by reworking what was > there to use JavaScript and the API. That is me however, I have lots > of time to put into Galaxy fundamentals and refactoring. This is more > work and I am happy to field PRs that modify mako for the time > being.(https://github.com/galaxyproject/galaxy/pull/1632/files looks > great for instance). > > A middle ground between rewriting everything and still depending on > mako is to dump dictionaries to JavaScript in the mako and render > things in JS. This will make it easier to adapt your changes once the > pages are ultimately rewritten to JavaScript SPAs. In the case 1632 - > that DependencyResolversView class produces dictionaries that can be > dumped as JSON easily into the page and JavaScript could be used to > build up the tables in > templates/webapps/tool_shed/repository/common.mako. > > -John > > > On Mon, Feb 1, 2016 at 11:51 AM, Peter van Heusden <p...@sanbi.ac.za> > wrote: > > Hi there > > > > Zipho and I'd would like to make some changes to the admin pages, > including > > the tool info page and the local data page. These are currently rendered > > from mako templates - should we change these or is the interface going > to be > > rewritten in client side / javascript in the near(ish) future? > > > > Thanks, > > Peter > > > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] galaxy job manager with celery and rabbitmq
Galaxy implements the job manager using rabbitmq and celery? Surely you mean Galaxy *does not* implement celery (at this point). As far as I know it is planned for the future. Peter On 25 January 2016 at 15:37, 杜云飞wrote: > Hi, > We are reading the source code for galaxy job manager. We found that the > job manager works like amqp. We want to know why galaxy implements the job > manager using rabbitmq and celery. What is the difference between them? > Thanks! > Best regards, >forest > > > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Trimmomatic doesn't recognise pair collection
Hi there I've got Trimmomatic installed (from the toolshed) into our local Galaxy server (server is latest commit on the dev branch from GitHub). It refuses to see a dataset pair as a valid input, however. I've tested this with two types: firstly a pair made out of 2 individual datasets (both fastqsanger) and then also a list of pairs. I've tested this with 'planemo serve' and the tool works, but for some reason it is not working on our server. Any ideas? Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Ansible roles: galaxy_server_dir vs galaxy_home_dir and galaxy_root
Hi there We've just finished provisioning a new Galaxy server here using the ansible-galaxy and ansible-galaxy-extras roles, with some extra roles from Nate and our own creation. In the ansible-galaxy role Galaxy's install location is set using the galaxy_server_dir variable, but in ansible-galaxy-extras the variables galaxy_home_dir and galaxy_root (presumably for distinct locations for the galaxy user's home directory and the root of the Galaxy installation) are used. Is this difference deliberate? Attached is our Ansible playbook - we just configured the variables to all point to the same location (and set the galaxy_venv_dir to be .venv not venv). I thought the two roles were meant to work together, so this difference is somewhat confusing. Thanks, Peter galaxydev.yml Description: application/yaml ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Job metrics configuration
Hi there I'm trying to understand how the job metrics configuration works. Reading through various examples, it looks like it works together with the dynamic destination support. First on the dynamic destination support, this works, as I understand it, by importing modules (files ending in .py) from the directory set as the rules_module, where rules_module is a parameter for the dynamic job plugin. As in this snipped from the usegalaxy configuration: usegalaxy.jobs.rules So import usegalaxy.jobs.rules, find where this resolves to as a directory, find all .py files in that directory. If rules_module is not specified, the default is galaxy.jobs.rules (under galaxy/lib). So these .py files are imported and within them are functions that are used in dynamic destinations (those with runner="dynamic") and in turn tools are directed to use these destinations. Now the job metrics configuration supplies both a user interface and then a set of keys that are made accessible to dynamic runners as key value pairs on the job.parameters attribute (passed in as the third argument, i.e. job). Specifically the job metrics configuration exists under a key '__job_resource'. Is this all correct? Is there a more straightforward way of moving from job resource specification to job configuration than this? Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Job metrics configuration
Hi there John Thanks for the clarification. My initial aim was, exactly, to expose those parameters that, as you say, some people don't think should be exposed. My use case is to allow jobs with different native parameters to be scheduled on our clouster from our Galaxy server. Perhaps a better approach is to use dynamic destinations that try and estimate appropriate values for e.g. hvmem, but at present our users are used to setting such parameters themselves. Peter On 11 January 2016 at 17:32, John Chilton <jmchil...@gmail.com> wrote: > So job metrics are completely different than what you described here. > Job metrics are enabled by default and can be configured fairly > directly by modifying config/job_metrics_conf.xml. New plugins are > relatively easy to write and can be added as python files to > lib/galaxy/jobs/metrics/instrumenters/. > > What you described is what is used on usegalaxy.org to allow users to > have some input on how jobs are scheduled. I call these job resources > parameters - in job_conf.xml particular tools can be mapped to these > parameters and - yes you are correct dynamic job destination functions > can consume them. > > The dynamic job destinations definitely shouldn't be accessing these > through job.parameters keyed on __job_resource. The dynamic > destination function should just take in a parameter named > resource_params and then the function will be passed in. > > Dynamic Destinations are documented here: > > https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping > > You are correct job resource parameters have not been documented*, > there is an open issue here: > https://github.com/galaxyproject/galaxy/issues/1266 > > * It should be noted that while this a frequently requested feature > and used on usegalaxy.org - the feature is somewhat controversial. The > thinking among some is that is exposes a level of detail that Galaxy > is supposed to hide. I don't share this particular concern, but the > implementation does worry me - what it does to persisted job state and > parameter handling is fairly hacky and I'm somewhat surprised we > haven't encountered more problems. This is part of the reason not why > this hasn't been documented, but why I feel less bad that it hasn't > been documented than other stuff I have done that also hasn't been > documented :|. > > -John > > On Mon, Jan 11, 2016 at 1:56 PM, Peter van Heusden <p...@sanbi.ac.za> > wrote: > > Hi there > > > > I'm trying to understand how the job metrics configuration works. Reading > > through various examples, it looks like it works together with the > dynamic > > destination support. > > > > First on the dynamic destination support, this works, as I understand > it, by > > importing modules (files ending in .py) from the directory set as the > > rules_module, where rules_module is a parameter for the dynamic job > plugin. > > As in this snipped from the usegalaxy configuration: > > > > > > > > usegalaxy.jobs.rules > > > > > > > > So import usegalaxy.jobs.rules, find where this resolves to as a > directory, > > find all .py files in that directory. If rules_module is not specified, > the > > default is galaxy.jobs.rules (under galaxy/lib). > > > > So these .py files are imported and within them are functions that are > used > > in dynamic destinations (those with runner="dynamic") and in turn tools > are > > directed to use these destinations. > > > > Now the job metrics configuration supplies both a user interface and > then a > > set of keys that are made accessible to dynamic runners as key value > pairs > > on the job.parameters attribute (passed in as the third argument, i.e. > job). > > Specifically the job metrics configuration exists under a key > > '__job_resource'. > > > > Is this all correct? Is there a more straightforward way of moving from > job > > resource specification to job configuration than this? > > > > Thanks, > > Peter > > > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > https://lists.galaxyproject.org/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Multiple web servers and handlers Galaxy failed
Hi Makis, I recently implemented something like this using uWSGI with a nginx frontend. Would you consider using uWSGI instead of your current approach? On 4 Dec 2015 09:40, "Makis Ladoukakis"wrote: > Sorry for bumping my own question but I didn't get any replies. Has anyone > succesfully installed a production Galaxy instance? > > I could really use some advice. > > Thank you, > Makis > > -- > From: makis4e...@hotmail.com > To: galaxy-...@lists.bx.psu.edu; galaxy-dev@lists.galaxyproject.org > Date: Tue, 1 Dec 2015 18:03:44 +0200 > Subject: [galaxy-dev] Multiple web servers and handlers Galaxy failed > > Hello everyone, > > I am trying to setup a Galaxy instance for a multi-user production > environment and I tried to follow the instructions here: > > https://wiki.galaxyproject.org/Admin/Config/Performance/Scaling > https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer > https://wiki.galaxyproject.org/Admin/Config/ApacheProxy > > but when I tried to start my galaxy instance (GALAXY_RUN_ALL=1 sh run.sh > --daemon) I got the following error in my browser:Bad Gateway The proxy > server received an invalid response from an upstream server. > > > I don't get any errors from the paster.log files so I am guessing the > error is somewhere in my apache configuration although I can't see where it > could be. > > > Has anyone encountered this problem before? > > Kind regards, > Makis > > > P.S. My apache configuration is as follows: > > I added that to my apache httpd.conf file: > > #for galaxy > #RewriteRule ^/galaxy(.*) http://localhost:8091$1 [P] > RewriteRule ^/galaxy$ /galaxy/ [R] > RewriteRule ^/galaxy/static/style/(.*) > /home/galaxy/galaxy/static/june_2007_style/blue/$1 [L] > RewriteRule ^/galaxy/static/scripts/(.*) > /home/galaxy/galaxy/static/scripts/packed/$1 [L] > RewriteRule ^/galaxy/static/(.*) /home/galaxy/galaxy/static/$1 [L] > RewriteRule ^/galaxy/favicon.ico > /home/galaxy/galaxy/static/favicon.ico [L] > RewriteRule ^/galaxy/robots.txt /home/galaxy/galaxy/static/robots.txt > [L] > > LoadModule uwsgi_module /etc/httpd/modules/mod_uwsgi.so > > > Sethandler uwsgi-handler > uWSGISocket 127.0.0.1:4001 > uWSGImaxVars 512 > > > I am also attaching my job_conf.xml and galaxy.ini file. > > > ___ Please keep > all replies on the list by using "reply all" in your mail client. To manage > your subscriptions to this and other Galaxy lists, please use the interface > at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use > the unified search at: http://galaxyproject.org/search/mailinglists/ > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Multiple web servers and handlers Galaxy failed
I've written up my experiences in a blog: http://pvh.wp.sanbi.ac.za/2015/12/04/faster-galaxy-with-uwsgi/ The only Galaxy side changes were the ones mentioned on the Scaling page. The Bad Gateway message means that Apache can't talk to the uWSGI so what is the state of uWSGI? Are you using supervisord? Does supervisorctl show the uWSGI process running? Does ps ax |grep uwsgi show it running? What does sudo lsof -i :4001 show is listening on port 4001? That would be the starting point of my debugging. Peter On 4 Dec 2015 10:57, "Makis Ladoukakis"wrote: > It is uwsgi that I am trying but with apache. Could you let me know what > changes you made to the configuration of Galaxy? > > Thank you, > Makis > > -- > Date: Fri, 4 Dec 2015 10:01:56 +0200 > Subject: Re: [galaxy-dev] Multiple web servers and handlers Galaxy failed > From: p...@sanbi.ac.za > To: makis4e...@hotmail.com > CC: galaxy-...@lists.bx.psu.edu; galaxy-dev@lists.galaxyproject.org > > Hi Makis, I recently implemented something like this using uWSGI with a > nginx frontend. Would you consider using uWSGI instead of your current > approach? > On 4 Dec 2015 09:40, "Makis Ladoukakis" wrote: > > Sorry for bumping my own question but I didn't get any replies. Has anyone > succesfully installed a production Galaxy instance? > > I could really use some advice. > > Thank you, > Makis > > -- > From: makis4e...@hotmail.com > To: galaxy-...@lists.bx.psu.edu; galaxy-dev@lists.galaxyproject.org > Date: Tue, 1 Dec 2015 18:03:44 +0200 > Subject: [galaxy-dev] Multiple web servers and handlers Galaxy failed > > Hello everyone, > > I am trying to setup a Galaxy instance for a multi-user production > environment and I tried to follow the instructions here: > > https://wiki.galaxyproject.org/Admin/Config/Performance/Scaling > https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer > https://wiki.galaxyproject.org/Admin/Config/ApacheProxy > > but when I tried to start my galaxy instance (GALAXY_RUN_ALL=1 sh run.sh > --daemon) I got the following error in my browser:Bad Gateway The proxy > server received an invalid response from an upstream server. > > > I don't get any errors from the paster.log files so I am guessing the > error is somewhere in my apache configuration although I can't see where it > could be. > > > Has anyone encountered this problem before? > > Kind regards, > Makis > > > P.S. My apache configuration is as follows: > > I added that to my apache httpd.conf file: > > #for galaxy > #RewriteRule ^/galaxy(.*) http://localhost:8091$1 [P] > RewriteRule ^/galaxy$ /galaxy/ [R] > RewriteRule ^/galaxy/static/style/(.*) > /home/galaxy/galaxy/static/june_2007_style/blue/$1 [L] > RewriteRule ^/galaxy/static/scripts/(.*) > /home/galaxy/galaxy/static/scripts/packed/$1 [L] > RewriteRule ^/galaxy/static/(.*) /home/galaxy/galaxy/static/$1 [L] > RewriteRule ^/galaxy/favicon.ico > /home/galaxy/galaxy/static/favicon.ico [L] > RewriteRule ^/galaxy/robots.txt /home/galaxy/galaxy/static/robots.txt > [L] > > LoadModule uwsgi_module /etc/httpd/modules/mod_uwsgi.so > > > Sethandler uwsgi-handler > uWSGISocket 127.0.0.1:4001 > uWSGImaxVars 512 > > > I am also attaching my job_conf.xml and galaxy.ini file. > > > ___ Please keep > all replies on the list by using "reply all" in your mail client. To manage > your subscriptions to this and other Galaxy lists, please use the interface > at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use > the unified search at: http://galaxyproject.org/search/mailinglists/ > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tools that make datasets
These names have meaning in CollectedDatasetMatch: designation means designation if that exists else name name means name dbkey means dbkey ext means ext visible means visible On 21 October 2015 at 08:44, Steve Cassidy <steve.cass...@mq.edu.au> wrote: > Thanks, yes that works, though I'm not really sure what the difference is > between __name__ and __designation__. They both seem to have the same > effect in my example. > > Thanks for your help. > > Steve > > > On 21 October 2015 at 16:49, Peter van Heusden <p...@sanbi.ac.za> wrote: > >> Since you're using a directory, you can use one of the built in patterns: >> >> DEFAULT_EXTRA_FILENAME_PATTERN = >> r"primary_DATASET_ID_(?P[^_]+)_(?P[^_]+)_(?P[^_]+)(_(?P[^_]+))?" >> >> NAMED_PATTERNS = { >> "__default__": DEFAULT_EXTRA_FILENAME_PATTERN, >> "__name__": r"(?P.*)", >> "__designation__": r"(?P.*)", >> "__name_and_ext__": r"(?P.*)\.(?P[^\.]+)?", >> "__designation_and_ext__": r"(?P.*)\.(?P[^\._]+)?", >> } >> >> In terms of docs, I don't know what the future is - the Galaxy wiki or >> http://galaxy.readthedocs.org/en/master/ >> >> >> On 21 October 2015 at 04:26, Steve Cassidy <steve.cass...@mq.edu.au> >> wrote: >> >>> Ah, thankyou, yes, I can now get results by using patterns to match the >>> output. I used your example but prepend 'simple' to the filename and then >>> searches for that with: >>> >>> >>> this solves the problem for the sample script but not generally since in >>> general I can't predict the filenames that will be generated - this is a >>> tool for downloading data from a repository which could be text, audio or >>> video data. >>> >>> If I don't use the 'simple' prefix and omit the file extension I still >>> get my data but I also get three other files which are temporary scripts >>> generated by galaxy and placed in the working directory. So, back to >>> trying to put things in a subdirectory. It turns out that the issue I was >>> having was as you pointed out earlier, the directory attribute to >>> discover_datasets doesn't allow variables, so I need to write to a fixed >>> directory name: >>> >>> >>> This now works! >>> >>> I had thought that I'd need to use a unique directory name but since >>> galaxy runs each job in a separate directory, this isn't required. My real >>> tool now works too after following the same pattern. >>> >>> Thanks for your help. I'll see if I can write this up in a blog post. >>> >>> Steve >>> >>> >>> >>> On 21 October 2015 at 00:06, Peter van Heusden <p...@sanbi.ac.za> wrote: >>> >>>> I poked around at your tool XML and the code a bit and the problem is >>>> directory="$job_name". Galaxy expects to collect files from the job's >>>> working directory - basically the current working directory the job runs >>>> in. The directory= argument doesn't have variables expanded as far as I can >>>> tell. In any event it is used in walk_over_extra_files() that is in >>>> lib/galaxy/tools/parameters/output_collect.py - if you look there you see >>>> that it is simply appended to the job's working directory. >>>> >>>> So if you use: >>>> >>>> >>>> >>>> (note the and - this is effectively the regexp >>>> r"(?P.*)\.txt" with the < and > escaped out) >>>> >>>> And alter the code so that it just writes files to the current >>>> directory, then you'll pick up the files one.txt, two,txt and three.txt. >>>> >>>> Peter >>>> >>>> On 20 October 2015 at 12:28, Steve Cassidy <steve.cass...@mq.edu.au> >>>> wrote: >>>> >>>>> Sorry, it was just an example of a tool that works - the file that it >>>>> writes out is put into that directory, so I assume that's where my files >>>>> should end up too. >>>>> >>>>> Steve >>>>> >>>>> On 20 October 2015 at 21:12, Peter van Heusden <p...@sanbi.ac.za> >>>>> wrote: >>>>> >>>>>> Sorry, I don't understand - what does the Upload File tool have to do >>>>>> with this? >>>>>> >>>>&g
Re: [galaxy-dev] Tools that make datasets
Just a quick check - did you refresh your history to confirm that the dataset *is* empty? We had the same thing at SANBI but it turns out that Galaxy creates an empty output collection and then only populates it sometime after job completion (this is a know UI bug). See: http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/ On 20 October 2015 at 08:48, Steve Cassidywrote: > Hi all, > I'm trying to understand how to write a tool that generates a dataset > rather than a single output file. I've tried following all of the examples > but I'm stuck, so I thought I would distil down the simplest example I > could write and ask for help here. > > So here's my example: > > https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55 > > it's a simple python script that writes three files to a directory named > for the single input parameter. > > I think one of the problems I'm having is knowing where to write the > output to. I've run this under planemo serve and the job runs, creating the > output directory within the 'job_working_directory/000/1/SampleDataset' > directory, however my dataset doesn't contain anything so clearly my > outputs directive isn't working: > > > > directory="$job_name" /> > > > > ($job_name is the name of the directory that is being written to, > SampleDataset in this case) > > Any help in getting this example working would be appreciated. > > Thanks, > > Steve > > > > > > -- > Department of Computing, Macquarie University > http://web.science.mq.edu.au/~cassidy/ > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > https://lists.galaxyproject.org/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tools that make datasets
I suspect that the problem might be in the then. I'm not an export on this, but "__name_and_ext__" turns into the regexp r"(?P.*)\.(?P[^\.]+)?" in lib/galaxy/tools/parameters/output_collect.py, and is used by the DatasetCollector (line 358). This looks like it should match the filenames you're creating, but I'm not 100% sure how that code works. One thing I notice is the "directory" argument. If you write jobs to the current directory instead of "output_path" can you get it to work? Peter On 20 October 2015 at 09:52, Steve Cassidy <steve.cass...@mq.edu.au> wrote: > Thanks Peter, > I did see that proviso somewhere but no, refreshing doesn't help. > > That page was one of those that I referred to getting to this point. > > Steve > > On 20 October 2015 at 18:33, Peter van Heusden <p...@sanbi.ac.za> wrote: > >> Just a quick check - did you refresh your history to confirm that the >> dataset *is* empty? We had the same thing at SANBI but it turns out that >> Galaxy creates an empty output collection and then only populates it >> sometime after job completion (this is a know UI bug). >> >> See: >> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/ >> >> On 20 October 2015 at 08:48, Steve Cassidy <steve.cass...@mq.edu.au> >> wrote: >> >>> Hi all, >>> I'm trying to understand how to write a tool that generates a dataset >>> rather than a single output file. I've tried following all of the examples >>> but I'm stuck, so I thought I would distil down the simplest example I >>> could write and ask for help here. >>> >>> So here's my example: >>> >>> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55 >>> >>> it's a simple python script that writes three files to a directory named >>> for the single input parameter. >>> >>> I think one of the problems I'm having is knowing where to write the >>> output to. I've run this under planemo serve and the job runs, creating the >>> output directory within the 'job_working_directory/000/1/SampleDataset' >>> directory, however my dataset doesn't contain anything so clearly my >>> outputs directive isn't working: >>> >>> >>> >>> >> directory="$job_name" /> >>> >>> >>> >>> ($job_name is the name of the directory that is being written to, >>> SampleDataset in this case) >>> >>> Any help in getting this example working would be appreciated. >>> >>> Thanks, >>> >>> Steve >>> >>> >>> >>> >>> >>> -- >>> Department of Computing, Macquarie University >>> http://web.science.mq.edu.au/~cassidy/ >>> >>> ___ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> https://lists.galaxyproject.org/ >>> >>> To search Galaxy mailing lists use the unified search at: >>> http://galaxyproject.org/search/mailinglists/ >>> >> >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> https://lists.galaxyproject.org/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ >> > > > > -- > Department of Computing, Macquarie University > http://web.science.mq.edu.au/~cassidy/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tools that make datasets
I poked around at your tool XML and the code a bit and the problem is directory="$job_name". Galaxy expects to collect files from the job's working directory - basically the current working directory the job runs in. The directory= argument doesn't have variables expanded as far as I can tell. In any event it is used in walk_over_extra_files() that is in lib/galaxy/tools/parameters/output_collect.py - if you look there you see that it is simply appended to the job's working directory. So if you use: (note the and - this is effectively the regexp r"(?P.*)\.txt" with the < and > escaped out) And alter the code so that it just writes files to the current directory, then you'll pick up the files one.txt, two,txt and three.txt. Peter On 20 October 2015 at 12:28, Steve Cassidy <steve.cass...@mq.edu.au> wrote: > Sorry, it was just an example of a tool that works - the file that it > writes out is put into that directory, so I assume that's where my files > should end up too. > > Steve > > On 20 October 2015 at 21:12, Peter van Heusden <p...@sanbi.ac.za> wrote: > >> Sorry, I don't understand - what does the Upload File tool have to do >> with this? >> >> On 20 October 2015 at 11:49, Steve Cassidy <steve.cass...@mq.edu.au> >> wrote: >> >>> Yes, I'm sure that's where the problem lies. Writing out to the current >>> directory doesn't work. The files get written to >>> 'job_working_directory/000/1/' but if I run the Upload File tool the result >>> is placed in 'files/000/'. I think I need to work out where to write the >>> files, I found some references to $__new_file_path__ but that doesn't seem >>> to help. >>> >>> Steve >>> >>> >>> >>> On 20 October 2015 at 19:57, Peter van Heusden <p...@sanbi.ac.za> wrote: >>> >>>> I suspect that the problem might be in the then. >>>> I'm not an export on this, but "__name_and_ext__" turns into the >>>> regexp r"(?P.*)\.(?P[^\.]+)?" in >>>> lib/galaxy/tools/parameters/output_collect.py, and is used by the >>>> DatasetCollector (line 358). This looks like it should match the filenames >>>> you're creating, but I'm not 100% sure how that code works. One thing I >>>> notice is the "directory" argument. If you write jobs to the current >>>> directory instead of "output_path" can you get it to work? >>>> >>>> Peter >>>> >>>> On 20 October 2015 at 09:52, Steve Cassidy <steve.cass...@mq.edu.au> >>>> wrote: >>>> >>>>> Thanks Peter, >>>>> I did see that proviso somewhere but no, refreshing doesn't help. >>>>> >>>>> That page was one of those that I referred to getting to this point. >>>>> >>>>> Steve >>>>> >>>>> On 20 October 2015 at 18:33, Peter van Heusden <p...@sanbi.ac.za> >>>>> wrote: >>>>> >>>>>> Just a quick check - did you refresh your history to confirm that the >>>>>> dataset *is* empty? We had the same thing at SANBI but it turns out that >>>>>> Galaxy creates an empty output collection and then only populates it >>>>>> sometime after job completion (this is a know UI bug). >>>>>> >>>>>> See: >>>>>> http://pvh.wp.sanbi.ac.za/2015/09/18/adventures-in-galaxy-output-collections/ >>>>>> >>>>>> On 20 October 2015 at 08:48, Steve Cassidy <steve.cass...@mq.edu.au> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> I'm trying to understand how to write a tool that generates a >>>>>>> dataset rather than a single output file. I've tried following all of >>>>>>> the >>>>>>> examples but I'm stuck, so I thought I would distil down the simplest >>>>>>> example I could write and ask for help here. >>>>>>> >>>>>>> So here's my example: >>>>>>> >>>>>>> https://gist.github.com/stevecassidy/0fa45ad5853faacb5f55 >>>>>>> >>>>>>> it's a simple python script that writes three files to a directory >>>>>>> named for the single input parameter. >>>>>>> >>>>>>> I think one of the problems I'm having is knowing where to write the >>>>>>> output to. I've run thi
Re: [galaxy-dev] Failure when preparing job on blastp
Previous revision of ncbi_blast_plus works like a charm, thanks. On 21 July 2015 at 19:57, Peter Cock p.j.a.c...@googlemail.com wrote: Thanks Peter, Yes, you are using the latest release published to the Tool Shed, aka v0.1.03 (internal changes): https://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/c16c30e9ad5b I've filed this as an issue on the GitHub repository, https://github.com/peterjc/galaxy_blast/issues/69 If you are also on GitHub we can discuss this there, where I've CC'd John and Bjoern for their thoughts. Something simple you could try now is going back to an older ncbi_blast_plus via the Tool Shed? The previous release 2fe07f50a41e aka v0.1.01 used the alternative form $db_opts.histdb.extra_files_path rather than the current $db_opts.histdb.files_path which appears to be failing. Thanks, Peter On Tue, Jul 21, 2015 at 6:17 PM, Peter van Heusden p...@sanbi.ac.za wrote: This is release_15.07 (last commit listed is 39f6730e5f50b54f372b8c1e17d86d8936b5899f) and ncbi_blast_plus c16c30e9ad5b. So no, its not an old Galaxy. On 21 July 2015 at 17:37, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Peter v H, Which version of Galaxy are you running? Which version of ncbi_blast_tools are you running? (ideally down to the ToolShed revision) My guess is that since this used to be .extra_files_path perhaps your Galaxy is too old for .files_path to work? https://github.com/peterjc/galaxy_blast/commit/2bbbf7f361ada20bf1b5bd777fae6e0cb52c4a38 Thanks, Peter C. On Tue, Jul 21, 2015 at 3:08 PM, Peter van Heusden p...@sanbi.ac.za wrote: Hi there I'm trying to use the blastp tools from the ncbi_blast_tools, but every time I try and run a job I get the following error: Traceback (most recent call last): File /net/ datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/lib/galaxy/jobs/runners/__init__.py , line 163, in prepare_job job_wrapper.prepare() File /net/ datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/lib/galaxy/jobs/__init__.py , line 859, in prepare self.command_line, self.extra_filenames, self.environment_variables = tool_evaluator.build() File /net/ datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/lib/galaxy/tools/evaluation.py , line 422, in build raise e NotFound: cannot find 'files_path' while searching for 'db_opts.histdb.files_path' This is happening when expanding the tool template. The BLAST db in in the history (thus the reference to histdb) but why can it not find this attribute? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Failure when preparing job on blastp
Hi there I'm trying to use the blastp tools from the ncbi_blast_tools, but every time I try and run a job I get the following error: Traceback (most recent call last): File /net/ datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/lib/galaxy/jobs/runners/__init__.py, line 163, in prepare_job job_wrapper.prepare() File /net/ datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/lib/galaxy/jobs/__init__.py, line 859, in prepare self.command_line, self.extra_filenames, self.environment_variables = tool_evaluator.build() File /net/ datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/lib/galaxy/tools/evaluation.py, line 422, in build raise e NotFound: cannot find 'files_path' while searching for 'db_opts.histdb.files_path' This is happening when expanding the tool template. The BLAST db in in the history (thus the reference to histdb) but why can it not find this attribute? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Troubleshooting
Hi Lionel Sounds like Galaxy does not know what format your FASTQ file is. When you click on it, what format does it show? Is it simple fastq? And how did you get it into Galaxy? You might need to manually specify the format by clicking on the Edit Attributes button (the pencil icon) and select the Datatype tab. Then set it to fastqsanger. This should allow the analysis to continue. On 20 July 2015 at 01:01, Lionel Mavoungou lionel@gmail.com wrote: Hello, I have a problem when I try to align en FATSQ file for a ChIP-seq. I have done a FASTQ grooming because it has been performed on Illumina. No my file is a .FASTSANGER. However my file is not recognized. I see this : No fastqsanger, fastqillumina or fastqsolexa dataset collection available. How could I fix it. It seems that my fastq file as well as my fastqsanger are not recognized. Thanks a lot, Lionel ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Adding packages for bedtools and ucsc-utils for Trackster?
Thanks Bjoern My initial install didn't pick up the Perl dependencies correctly, but I re-installed and now the install seems to have the required modules but is missing something in the environment variables. Specifically I get an error import JSON.pm, see below. Note that the jbrowse/1.11.6/iuc/package_jbrowse_1_11_6/d5ab749c9e97/lib/perl5 directory isn't included in @INC and in fact the paths that *are* included seem to imply that the wrong perl interpreter is being picked up. Fatal error: Exit code 1 () Can't locate JSON.pm in @INC (you may need to install the JSON module) (@INC contains: /net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5 /etc/perl /usr/local/lib/perl/5.18.2 /usr/local/share/perl/5.18.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.18 /usr/share/perl/5.18 /usr/local/lib/site_perl .) at /net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5/Bio/JBrowse/JSON.pm line 10. BEGIN failed--compilation aborted at /net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5/Bio/JBrowse/JSON.pm line 10. Compilation failed in require at /net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5/Bio/JBrowse/Cmd/FormatSequences.pm line 21. BEGIN failed--compilation aborted at /net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/job_working_directory/000/29/dataset_29_files/JBrowse-1.11.6/bin/../src/perl5/Bio/JBrowse/Cmd/FormatSequences.pm line 21. Compilation failed in require at bin/prepare-refseqs.pl line 7. BEGIN failed--compilation aborted at bin/prepare-refseqs.pl line 7. Traceback (most recent call last): File /net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/2c9e5136b416/jbrowse/jbrowse.py, line 44, in module process_genome(jbrowse_dir, os.path.realpath(args.genome.name)) File /net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/jbrowse/2c9e5136b416/jbrowse/jbrowse.py, line 17, in process_genome subprocess.check_output(['perl', 'bin/prepare-refseqs.pl', '--fasta', genome], cwd=jbrowse_dir) File /usr/lib/python2.7/subprocess.py, line 573, in check_output raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command '['perl', 'bin/prepare-refseqs.pl', '--fasta', '/net/datasrv3hs.sanbi.ac.za/cip0/software/galaxy/galaxysrv/galaxy/database/files/000/dataset_3.dat']' returned non-zero exit status 2 On 18 June 2015 at 10:01, Bjoern Gruening bjoern.gruen...@gmail.com wrote: Peter, I'm in a hurry ... have a look at https://github.com/galaxyproject/tools-iuc/tree/master/tools/jbrowse Developed by IUC member Eric Rasche. Cheers, Bjoern On 18.06.2015 10:00, Peter van Heusden wrote: I've tried that - I've added the Environment Modules for the relevant tools to the local_env.sh file, so they're there in the environment when Galaxy runs. These tools are scheduled via our cluster, however, so I don't think the environment passes across to them. They depend on these Galaxy packages, however, and those (bedtools and ucsc-utils) are now installed, so I'm not sure why it can't pick them up from the shed_tools. On 18 June 2015 at 09:10, Hans-Rudolf Hotz h...@fmi.ch wrote: On 06/18/2015 09:06 AM, Peter van Heusden wrote: Hi there I'm trying to visualise some of our genome data using Trackster. Initial input is 790,000 lines of GFF3. When I try and create the visualisation, I get errors from two tools: Convert GFF to Interval Index on data 3 -- an out of memory error - I'll try and fix this by upping the memory limit for our Galaxy jobs Convert BED, GFF, or VCF to BigWig on data 3 -- bedtools and bedGraphToBigWig not found errors. I tried to resolve these by installing the ucsc-tools and bedtools packages - and they did install correctly - but I still get these file not found errors. Hi Peter make sure 'galaxy' (ie the user galaxy is running as) has the newly installed tools in its path. You might need to add them to run.sh and restart the server. Hans-Rudolf Any idea how to fix this second problem? Thanks! Peter P.S. as an alternative to Trackster, does anyone have Galaxy interfaced to JBrowse? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists
Re: [galaxy-dev] Nginx Config
Wow that's a lot of upstreams... what is your Galaxy configuration? On 26 November 2014 at 20:26, Nicholas Kline nx...@case.edu wrote: Hi, I am trying to setup a local production Galaxy running on Ubuntu, Nginx, PostgreSQL, and ProFTD. When visiting http://localhost, 502 Bad Gateway is returned. Visiting https://localhost returns unable to connect. I was expecting to see the Galaxy website instead. My Nginx configuration file is included below. Thank you for your help. ### # /etc/nginx/nginx.conf ### user www-data; worker_processes 4; pid /run/nginx.pid; events { worker_connections 768; # multi_accept on; } http { ## # Basic Settings ## sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; # server_tokens off; # server_names_hash_bucket_size 64; # server_name_in_redirect off; include /etc/nginx/mime.types; default_type application/octet-stream; ## # Logging Settings ## access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; ## # Gzip Settings ## gzip on; gzip_http_version 1.1; gzip_vary on; gzip_comp_level 4; gzip_proxied any; gzip_types text/plain text/css application/x-javascript text/xml application/xml text/javascript application/json; gzip_buffers 16 8k; gzip_disable MSIE [1-6].(?!.*SV1); ## # nginx-naxsi config ## # Uncomment it if you installed nginx-naxsi ## #include /etc/nginx/naxsi_core.rules; ## # nginx-passenger config ## # Uncomment it if you installed nginx-passenger ## #passenger_root /usr; #passenger_ruby /usr/bin/ruby; ## # Virtual Host Configs ## include /etc/nginx/conf.d/*.conf; #include /etc/nginx/sites-enabled/*; ## # Galaxy ## upstream galaxy_app { # As defined in galaxy config file universe_wsgi.ini server localhost:8080; server localhost:8081; server localhost:8082; server localhost:8083; server localhost:8084; server localhost:8085; server localhost:8086; server localhost:8087; server localhost:8088; server localhost:8089; } server { # Maximum upload size that can be handled by POST requests through nginx. # Disabled because using FTP instead of web interface. #client_max_body_size 10G; location / { proxy_pass http://galaxy_app; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # If using SSL, makes Galaxy aware of what type of URL it should generate for external sites like Biomart. proxy_set_header X-URL-SCHEME https; } # nginx is more efficient at serving static content. # expires 24h is for caching purposes. location /static { alias /home/galaxy/galaxy-dist/static; expires 24h; } location /static/style { alias /home/galaxy/galaxy-dist/static/june_2007_style/blue; expires 24h; } location /static/scripts { alias /home/galaxy/galaxy-dist/static/scripts/packed; } location /favicon.ico { alias /home/galaxy/galaxy-dist/static/favicon.ico; } location /robots.txt { alias /home/galaxy/galaxy-dist/static/robots.txt; } # Use nginx to send files (downloads) instead of Galaxy. location /_x_accel_redirect/ { internal; alias /; } } } ### # /var/log/nginx/error.log ### 2014/11/25 15:43:21 [error] 56433#0: *23 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: GET / HTTP/1.1, upstream: http://127.0.0.1:8089/;, host: localhost 2014/11/25 15:43:21 [error] 56433#0: *23 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: GET / HTTP/1.1, upstream: http://127.0.0.1:8080/;, host: localhost 2014/11/25 15:43:21 [error] 56433#0: *23 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: GET / HTTP/1.1, upstream: http://127.0.0.1:8081/;, host: localhost 2014/11/25 15:43:21 [error] 56433#0: *23 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: GET / HTTP/1.1, upstream: http://127.0.0.1:8082/;, host: localhost 2014/11/25 15:43:21 [error] 56433#0: *23 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: GET /