Re: [galaxy-dev] Configuring Galaxy, Pulsar and Conda
Hi John, thank you for your help. Now it works! Il 07/02/2017 21:20, John Chilton ha scritto: Hello, Thanks for working on this - Pulsar still hasn't quite caught up with Galaxy in terms of support for Conda but we are getting there. So I noticed two things that should help today when I was trying to recreate your problem - the first is that Conda support in Pulsar requires this PR (https://github.com/galaxyproject/pulsar/pull/126) which isn't part of 0.7.3 - the pulsar you are targeting. I did an update of Pulsar though so that should help - release 0.7.4 should be available as of today. Additionally, I noticed that the default for the Pulsar client option "dependency_resolution" in Galaxy was broken. So to get this to work at all - you will need to update your job_conf.xml and specify that Pulsar should resolve dependencies and not Galaxy. This means adding remote to your Pulsar destination. I opened a Pull Request here for the Galaxy change - so hopefully this will be the default by the time 17.01 is released. The logs should make it clear if there are additional problems after those changes I think. One think I would recommend is adding a versionless Conda resolver also - so you may want to make that dependency resolvers file look like this instead: This may not be strictly needed based on the tools you are attempting to get this to work for though. Thanks for your patience and let me know how it goes. -John On Fri, Feb 3, 2017 at 6:18 AM, ric wrote: Hi, I'm tryng to configure a Pulsar server to resolve dependencies using Conda but unfortunately it does not work. When I try to execute, for example, bowtie2, the standard error in the Galaxy machine says: "/home/galaxy/pulsar/files/staging/67/command.sh: line 60: bowtie2-build: command not found" because Conda in the Pulsar machine does not download this dependency. Here is my app.yml: dependency_resolvers_config_file: dependency_resolvers_conf.xml My dependency_resolvers_conf.xml The log of the Galaxy server: galaxy.tools DEBUG 2017-01-19 14:50:18,910 Validated and populated state for tool request (97.265 ms) galaxy.tools.actions INFO 2017-01-19 14:50:19,069 Handled output named output for tool toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0 (81.926 ms) galaxy.tools.actions INFO 2017-01-19 14:50:19,113 Added output datasets to history (43.950 ms) galaxy.tools.actions INFO 2017-01-19 14:50:19,179 Verified access to datasets for Job[unflushed,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0] (33.690 ms) galaxy.tools.actions INFO 2017-01-19 14:50:19,180 Setup for job Job[unflushed,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0] complete, ready to flush (66.261 ms) galaxy.tools.actions INFO 2017-01-19 14:50:19,240 Flushed transaction for job Job[id=67,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0] (59.840 ms) galaxy.tools.execute DEBUG 2017-01-19 14:50:19,240 Tool [toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0] created job [67] (296.972 ms) galaxy.tools.execute DEBUG 2017-01-19 14:50:19,266 Executed 1 job(s) for tool toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0 request: (355.402 ms) 156.148.160.117 - - [19/Jan/2017:14:50:18 +0200] "POST /api/tools HTTP/1.1" 200 - "http://156.148.120.96:8080/"; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" 156.148.160.117 - - [19/Jan/2017:14:50:19 +0200] "GET /api/webhooks/tool HTTP/1.1" 200 - "http://156.148.120.96:8080/"; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" 156.148.160.117 - - [19/Jan/2017:14:50:19 +0200] "GET /api/histories/1cd8e2f6b131e891/contents?details=b847e822bdc195d0%2C40876639881ca029%2C915ae9a80309f157%2Cfd767e8b7355e21a%2C9ce617df390851fc%2C3cc0effd29705aa3%2Cb701da857886499b&order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2017-01-19T12%3A11%3A40.000Z&qv=False&qv=False HTTP/1.1" 200 - "http://156.148.120.96:8080/"; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" galaxy.jobs DEBUG 2017-01-19 14:50:19,912 (67) Working directory for job is: /home/galaxy/galaxy_test_pulsar/database/jobs_directory/000/67 galaxy.jobs.handler DEBUG 2017-01-19 14:50:19,931 (67) Dispatching to pulsar runner galaxy.jobs DEBUG 2017-01-19 14:50:20,013 (67) Persisting job destination (destination id: remote_cluster) galaxy.jobs.runners DEBUG 2017-01-19 14:50:20,044 Job [67] queued (112.446 ms) galaxy.jobs.handler INFO 2017-01-19 14:50:20,090 (67) Job dispatched galaxy.jobs.runners.pulsar INFO 2017-01-19 14:50:20,205 pulsar_version is 0.7.3 galaxy.util.object_wrapper WARNING 2017-01-19 14:50:20,644 Unable to create dynamic subclass for , None: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases galaxy.tools.deps DEBUG 2017-01-19 14:50:20,8
Re: [galaxy-dev] Configuring Galaxy, Pulsar and Conda
Hello, Thanks for working on this - Pulsar still hasn't quite caught up with Galaxy in terms of support for Conda but we are getting there. So I noticed two things that should help today when I was trying to recreate your problem - the first is that Conda support in Pulsar requires this PR (https://github.com/galaxyproject/pulsar/pull/126) which isn't part of 0.7.3 - the pulsar you are targeting. I did an update of Pulsar though so that should help - release 0.7.4 should be available as of today. Additionally, I noticed that the default for the Pulsar client option "dependency_resolution" in Galaxy was broken. So to get this to work at all - you will need to update your job_conf.xml and specify that Pulsar should resolve dependencies and not Galaxy. This means adding remote to your Pulsar destination. I opened a Pull Request here for the Galaxy change - so hopefully this will be the default by the time 17.01 is released. The logs should make it clear if there are additional problems after those changes I think. One think I would recommend is adding a versionless Conda resolver also - so you may want to make that dependency resolvers file look like this instead: This may not be strictly needed based on the tools you are attempting to get this to work for though. Thanks for your patience and let me know how it goes. -John On Fri, Feb 3, 2017 at 6:18 AM, ric wrote: > Hi, > I'm tryng to configure a Pulsar server to resolve dependencies using Conda > but unfortunately it does not work. > When I try to execute, for example, bowtie2, the standard error in the > Galaxy machine says: "/home/galaxy/pulsar/files/staging/67/command.sh: line > 60: bowtie2-build: command not found" because Conda in the Pulsar machine > does not download this dependency. > > Here is my app.yml: > > dependency_resolvers_config_file: dependency_resolvers_conf.xml > > > My dependency_resolvers_conf.xml > > > > > > > The log of the Galaxy server: > > galaxy.tools DEBUG 2017-01-19 14:50:18,910 Validated and populated state for > tool request (97.265 ms) > galaxy.tools.actions INFO 2017-01-19 14:50:19,069 Handled output named > output for tool toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0 > (81.926 ms) > galaxy.tools.actions INFO 2017-01-19 14:50:19,113 Added output datasets to > history (43.950 ms) > galaxy.tools.actions INFO 2017-01-19 14:50:19,179 Verified access to > datasets for > Job[unflushed,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0] > (33.690 ms) > galaxy.tools.actions INFO 2017-01-19 14:50:19,180 Setup for job > Job[unflushed,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0] > complete, ready to flush (66.261 ms) > galaxy.tools.actions INFO 2017-01-19 14:50:19,240 Flushed transaction for > job > Job[id=67,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0] > (59.840 ms) > galaxy.tools.execute DEBUG 2017-01-19 14:50:19,240 Tool > [toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0] created job > [67] (296.972 ms) > galaxy.tools.execute DEBUG 2017-01-19 14:50:19,266 Executed 1 job(s) for > tool toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0 request: > (355.402 ms) > 156.148.160.117 - - [19/Jan/2017:14:50:18 +0200] "POST /api/tools HTTP/1.1" > 200 - "http://156.148.120.96:8080/"; "Mozilla/5.0 (X11; Linux x86_64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" > 156.148.160.117 - - [19/Jan/2017:14:50:19 +0200] "GET /api/webhooks/tool > HTTP/1.1" 200 - "http://156.148.120.96:8080/"; "Mozilla/5.0 (X11; Linux > x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 > Safari/537.36" > 156.148.160.117 - - [19/Jan/2017:14:50:19 +0200] "GET > /api/histories/1cd8e2f6b131e891/contents?details=b847e822bdc195d0%2C40876639881ca029%2C915ae9a80309f157%2Cfd767e8b7355e21a%2C9ce617df390851fc%2C3cc0effd29705aa3%2Cb701da857886499b&order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2017-01-19T12%3A11%3A40.000Z&qv=False&qv=False > HTTP/1.1" 200 - "http://156.148.120.96:8080/"; "Mozilla/5.0 (X11; Linux > x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 > Safari/537.36" > galaxy.jobs DEBUG 2017-01-19 14:50:19,912 (67) Working directory for job is: > /home/galaxy/galaxy_test_pulsar/database/jobs_directory/000/67 > galaxy.jobs.handler DEBUG 2017-01-19 14:50:19,931 (67) Dispatching to pulsar > runner > galaxy.jobs DEBUG 2017-01-19 14:50:20,013 (67) Persisting job destination > (destination id: remote_cluster) > galaxy.jobs.runners DEBUG 2017-01-19 14:50:20,044 Job [67] queued (112.446 > ms) > galaxy.jobs.handler INFO 2017-01-19 14:50:20,090 (67) Job dispatched > galaxy.jobs.runners.pulsar INFO 2017-01-19 14:50:20,205 pulsar_version is > 0.7.3 > galaxy.util.object_wrapper WARNING 2017-01-19 14:50:20,644 Unable to create > dynamic subclass for , None: metaclass conflict: the > metaclass of a derived class must be a (non-strict) subclass of the > metaclasses of all its bases > galaxy.t
[galaxy-dev] Configuring Galaxy, Pulsar and Conda
Hi, I'm tryng to configure a Pulsar server to resolve dependencies using Conda but unfortunately it does not work. When I try to execute, for example, bowtie2, the standard error in the Galaxy machine says: "/home/galaxy/pulsar/files/staging/67/command.sh: line 60: bowtie2-build: command not found" because Conda in the Pulsar machine does not download this dependency. Here is my app.yml: dependency_resolvers_config_file: dependency_resolvers_conf.xml My dependency_resolvers_conf.xml The log of the Galaxy server: galaxy.tools DEBUG 2017-01-19 14:50:18,910 Validated and populated state for tool request (97.265 ms) galaxy.tools.actions INFO 2017-01-19 14:50:19,069 Handled output named output for tool toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0 (81.926 ms) galaxy.tools.actions INFO 2017-01-19 14:50:19,113 Added output datasets to history (43.950 ms) galaxy.tools.actions INFO 2017-01-19 14:50:19,179 Verified access to datasets for Job[unflushed,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0] (33.690 ms) galaxy.tools.actions INFO 2017-01-19 14:50:19,180 Setup for job Job[unflushed,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0] complete, ready to flush (66.261 ms) galaxy.tools.actions INFO 2017-01-19 14:50:19,240 Flushed transaction for job Job[id=67,tool_id=toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0] (59.840 ms) galaxy.tools.execute DEBUG 2017-01-19 14:50:19,240 Tool [toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0] created job [67] (296.972 ms) galaxy.tools.execute DEBUG 2017-01-19 14:50:19,266 Executed 1 job(s) for tool toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.0 request: (355.402 ms) 156.148.160.117 - - [19/Jan/2017:14:50:18 +0200] "POST /api/tools HTTP/1.1" 200 - "http://156.148.120.96:8080/"; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" 156.148.160.117 - - [19/Jan/2017:14:50:19 +0200] "GET /api/webhooks/tool HTTP/1.1" 200 - "http://156.148.120.96:8080/"; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" 156.148.160.117 - - [19/Jan/2017:14:50:19 +0200] "GET /api/histories/1cd8e2f6b131e891/contents?details=b847e822bdc195d0%2C40876639881ca029%2C915ae9a80309f157%2Cfd767e8b7355e21a%2C9ce617df390851fc%2C3cc0effd29705aa3%2Cb701da857886499b&order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2017-01-19T12%3A11%3A40.000Z&qv=False&qv=False HTTP/1.1" 200 - "http://156.148.120.96:8080/"; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" galaxy.jobs DEBUG 2017-01-19 14:50:19,912 (67) Working directory for job is: /home/galaxy/galaxy_test_pulsar/database/jobs_directory/000/67 galaxy.jobs.handler DEBUG 2017-01-19 14:50:19,931 (67) Dispatching to pulsar runner galaxy.jobs DEBUG 2017-01-19 14:50:20,013 (67) Persisting job destination (destination id: remote_cluster) galaxy.jobs.runners DEBUG 2017-01-19 14:50:20,044 Job [67] queued (112.446 ms) galaxy.jobs.handler INFO 2017-01-19 14:50:20,090 (67) Job dispatched galaxy.jobs.runners.pulsar INFO 2017-01-19 14:50:20,205 pulsar_version is 0.7.3 galaxy.util.object_wrapper WARNING 2017-01-19 14:50:20,644 Unable to create dynamic subclass for , None: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases galaxy.tools.deps DEBUG 2017-01-19 14:50:20,820 Find dependency bowtie2 version 2.3.0 galaxy.tools.deps DEBUG 2017-01-19 14:50:20,821 Using dependency bowtie2 version 2.3.0 of type conda galaxy.tools.deps DEBUG 2017-01-19 14:50:20,821 Find dependency samtools version 1.3.1 galaxy.tools.deps DEBUG 2017-01-19 14:50:20,821 Using dependency samtools version 1.3.1 of type conda galaxy.tools.deps.conda_util DEBUG 2017-01-19 14:50:20,822 Executing command: /home/galaxy/galaxy_test_pulsar/database/dependencies/_conda/bin/conda list --name __bowtie2@2.3.0 --export > ./database/tmp/jobdepsUaJpU60c3b56cd60a211bacfadd5c45280e5adba95113fa75f9e47b09cad555f49ae2b/__bowtie2@2.3.0 galaxy.tools.deps.conda_util DEBUG 2017-01-19 14:50:21,220 Executing command: /home/galaxy/galaxy_test_pulsar/database/dependencies/_conda/bin/conda create -y --unknown --offline --prefix /home/galaxy/galaxy_test_pulsar/database/jobs_directory/000/67/conda-env --copy --file ./database/tmp/jobdepsUaJpU60c3b56cd60a211bacfadd5c45280e5adba95113fa75f9e47b09cad555f49ae2b/__bowtie2@2.3.0 > /dev/null galaxy.tools.deps.conda_util DEBUG 2017-01-19 14:50:23,885 Executing command: /home/galaxy/galaxy_test_pulsar/database/dependencies/_conda/bin/conda clean --tarballs -y 156.148.160.117 - - [19/Jan/2017:14:50:24 +0200] "GET /api/histories/1cd8e2f6b131e891/contents?details=b847e822bdc195d0%2C40876639881ca029%2C915ae9a80309f157%2Cfd767e8b7355e21a%2C9ce617df390851fc%2C3cc0effd29705aa3%2Cb701da857886499b&order=hid&v=dev&q=update