Re: [galaxy-dev] Download URL not accepted in tool_dependencies.xml
Thanks, then I will run toolshed from a separate galaxy-central clone if that's advisable - different from the Galaxy service. I prefer to keep galaxy-dist on the stable branch. In this process, I will move the tools from the old toolshed to my new toolshed running from galaxy-central. When I have time, I will upload everything to the test-toolshed. Cheers, Joachim Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 09/30/2013 05:11 PM, Bjoern Gruening wrote: Hi Joachim, I had such problems if my XML file is invalid. Than I get immediately the 'installed:' text and can click install again, without any change. Also I would really recommend to reset your toolshed. These white ghosts can have unpredictable effects. If you can upload your tools to the test tool shed I can try to install it debug it. Please make sure that you run the latest Galaxy central version. Greg and Dave invested a lot of time to get rid of white ghosts. But these changes are only in central. Cheers, Bjoern Still having a hard time and some strange situations. It is almost too much to report via email. Basically, the tool_dependencies.xml get recognised sometimes. Please, watch this video to see what I mean (1'30''), and pause as you like: https://dl.dropboxusercontent.com/u/18352887/toolshedissues.ogv I am afraid I cannot invest this amount of time to make my repo's public-proof. There are weird things happening which make no sense to me. And debugging of tool_dependencies.xml is very hard, since I find almost no logs of this process. Joachim. Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 09/30/2013 02:53 PM, Dave Bouvier wrote: Joachim, For sourceforge URLs, you can safely omit the question mark and everything after it, since that's part of the browser download load balancing process. When downloading with wget, curl, or python's urllib, downloads.sourceforge.net does its own behind-the-scenes load balancing. As for hyphens, I don't believe they need to be escaped, since they're not XML entities like the ampersand is. The following download_by_url action should work: action type=download_by_urlhttp://downloads.sourceforge.net/project/transpose/transpose/transpose-2.0/2.0/transpose-2.0.zip/action --Dave B. On 09/30/2013 08:45 AM, Joachim Jacob | VIB | wrote: Hi all, Not fixed yet. Somehow the download does not work: I have escaped the ampersands and the hyphens. The download seems to have to proceed, since I receive no error from the download step. The next step in the dependencies file gives an error: mv: cannot stat transpose.zip': No such file or directory Cheers, J Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 09/30/2013 02:06 PM, Peter Cock wrote: You must escape any ampersand in the XML as amp; and then it should work. Peter On Mon, Sep 30, 2013 at 12:51 PM, Joachim Jacob | VIB | joachim.ja...@vib.be wrote: Hi all, The download URL seems not to be accepted in tool_dependencies.xml. It is an URL from sourceforge, the direct link to a package. The error upon uploading the tool to my toolshed: ** Metadata may have been defined for some items in revision '5665a799775d'. Correct the following problems if necessary and reset metadata. tool_dependencies.xml - Exception attempting to parse /mnt/toolsheddb/database/000/repo_26/tool_dependencies.xml: not well-formed (invalid token): line 6, column 149 ** The tool_dependencies.xml file: ** 1 ?xml version=1.0? 2 tool_dependency 3 package name=transpose version=2.0.0 4install version=1.0 5 actions 6 action type=download_by_urlhttp://downloads.sourceforge.net/project/transpose/transpose/transpose-2.0/2.0/transpose-2.0.zip?r=ts=1380535239use_mirror=surfnet/action 7action type=shell_commandmkdir bin/action 8 action type=shell_commandunzip transpose-2.0.zip/action action type=shell_commandcd transpose-2.0/src/action action type=shell_commandgcc transpose.c -o transpose/action action type=move_file source$INSTALL_DIR/transpose-2.0/src/transpose/source destination$INSTALL_DIR/bin/destination /action action type=shell_commandchmod +x $INSTALL_DIR/bin/transpose/action action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install readme Compiling transpose and putting in the path. /readme /package /tool_dependency ** Cheers, Joachim -- Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team ___ Please keep all replies on the list by using reply all in your mail client. To manage your
Re: [galaxy-dev] Download URL not accepted in tool_dependencies.xml
Hi Joachim, I don't think that is possible at the moment (its not a general problem, but in this cycle it seems to be). Currently I can not install from the test toolshed in my stable test branch. Because the tool shed is not forward compatible (yet). I can be wrong, but if you get an error, that is probably the reason. There is a trello card about forward compatibility, somewhere. Ciao, Bjoern Thanks, then I will run toolshed from a separate galaxy-central clone if that's advisable - different from the Galaxy service. I prefer to keep galaxy-dist on the stable branch. In this process, I will move the tools from the old toolshed to my new toolshed running from galaxy-central. When I have time, I will upload everything to the test-toolshed. Cheers, Joachim Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 09/30/2013 05:11 PM, Bjoern Gruening wrote: Hi Joachim, I had such problems if my XML file is invalid. Than I get immediately the 'installed:' text and can click install again, without any change. Also I would really recommend to reset your toolshed. These white ghosts can have unpredictable effects. If you can upload your tools to the test tool shed I can try to install it debug it. Please make sure that you run the latest Galaxy central version. Greg and Dave invested a lot of time to get rid of white ghosts. But these changes are only in central. Cheers, Bjoern Still having a hard time and some strange situations. It is almost too much to report via email. Basically, the tool_dependencies.xml get recognised sometimes. Please, watch this video to see what I mean (1'30''), and pause as you like: https://dl.dropboxusercontent.com/u/18352887/toolshedissues.ogv I am afraid I cannot invest this amount of time to make my repo's public-proof. There are weird things happening which make no sense to me. And debugging of tool_dependencies.xml is very hard, since I find almost no logs of this process. Joachim. Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 09/30/2013 02:53 PM, Dave Bouvier wrote: Joachim, For sourceforge URLs, you can safely omit the question mark and everything after it, since that's part of the browser download load balancing process. When downloading with wget, curl, or python's urllib, downloads.sourceforge.net does its own behind-the-scenes load balancing. As for hyphens, I don't believe they need to be escaped, since they're not XML entities like the ampersand is. The following download_by_url action should work: action type=download_by_urlhttp://downloads.sourceforge.net/project/transpose/transpose/transpose-2.0/2.0/transpose-2.0.zip/action --Dave B. On 09/30/2013 08:45 AM, Joachim Jacob | VIB | wrote: Hi all, Not fixed yet. Somehow the download does not work: I have escaped the ampersands and the hyphens. The download seems to have to proceed, since I receive no error from the download step. The next step in the dependencies file gives an error: mv: cannot stat transpose.zip': No such file or directory Cheers, J Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 09/30/2013 02:06 PM, Peter Cock wrote: You must escape any ampersand in the XML as amp; and then it should work. Peter On Mon, Sep 30, 2013 at 12:51 PM, Joachim Jacob | VIB | joachim.ja...@vib.be wrote: Hi all, The download URL seems not to be accepted in tool_dependencies.xml. It is an URL from sourceforge, the direct link to a package. The error upon uploading the tool to my toolshed: ** Metadata may have been defined for some items in revision '5665a799775d'. Correct the following problems if necessary and reset metadata. tool_dependencies.xml - Exception attempting to parse /mnt/toolsheddb/database/000/repo_26/tool_dependencies.xml: not well-formed (invalid token): line 6, column 149 ** The tool_dependencies.xml file: ** 1 ?xml version=1.0? 2 tool_dependency 3 package name=transpose version=2.0.0 4install version=1.0 5 actions 6 action type=download_by_urlhttp://downloads.sourceforge.net/project/transpose/transpose/transpose-2.0/2.0/transpose-2.0.zip?r=ts=1380535239use_mirror=surfnet/action 7action type=shell_commandmkdir bin/action 8 action type=shell_commandunzip transpose-2.0.zip/action action type=shell_commandcd transpose-2.0/src/action action type=shell_commandgcc transpose.c -o transpose/action action type=move_file source$INSTALL_DIR/transpose-2.0/src/transpose/source destination$INSTALL_DIR/bin/destination /action action type=shell_commandchmod +x
[galaxy-dev] multiple input files
Dear all We frequently find ourselves in situations where a tool needs to be run with a lot of input files. For example, run the GATK UnifiedGenotyper with easily dozens of bam files. Using the repeat in this case requires quite a bit of clicking. Is there a more conventient way fo doing this? Maybe similar to the multi-file-select that is possible for workflow inputs? I saw some older discussions on this or similar issues, but I am a bit lost what the current official stable proposed solution for this is. Thanks for your help Ulf ** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] multiple input files
On Tue, Oct 1, 2013 at 1:27 PM, Ulf Schaefer ulf.schae...@phe.gov.uk wrote: Dear all We frequently find ourselves in situations where a tool needs to be run with a lot of input files. For example, run the GATK UnifiedGenotyper with easily dozens of bam files. Using the repeat in this case requires quite a bit of clicking. Is there a more conventient way fo doing this? Maybe similar to the multi-file-select that is possible for workflow inputs? I saw some older discussions on this or similar issues, but I am a bit lost what the current official stable proposed solution for this is. Thanks for your help Ulf Use param type=data multiple=true ... instead of using repeat ...param type=data .../repeat. I asked John Chilton about this recently via Twitter, and it is now on the wiki, http://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax Simple example here: https://bitbucket.org/galaxyp/galaxyp-toolshed-iquant/src/tip/iquant.xml?at=default Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] multiple input files
Thanks Peter,
Yeah, looking over unified_genotyper.xml, you would probably just
want to replace:
#for $i, $input_bam in enumerate( $reference_source.input_bams ):
-d -I ${input_bam.input_bam} ${input_bam.input_bam.ext}
gatk_input_${i}
#if str( $input_bam.input_bam.metadata.bam_index ) != None:
-d ${input_bam.input_bam.metadata.bam_index}
bam_index gatk_input_${i} ##hardcode galaxy ext type as bam_index
#end if
#end for
with
#for $i, $input_bam in enumerate( $reference_source.input_bams ):
-d -I ${input_bam} ${input_bam.ext} gatk_input_${i}
#if str( $input_bam.metadata.bam_index ) != None:
-d ${input_bam.metadata.bam_index} bam_index
gatk_input_${i} ##hardcode galaxy ext type as bam_index
#end if
#end for
And:
repeat name=input_bams title=BAM file min=1
help=-I,--input_file amp;lt;input_fileamp;gt;
param name=input_bam type=data format=bam label=BAM file
validator type=unspecified_build /
validator type=dataset_metadata_in_data_table
table_name=gatk_picard_indexes metadata_name=dbkey
metadata_column=dbkey message=Sequences are not currently available
for the specified build. / !-- fixme!!! this needs to be a select
--
/param
/repeat
with
param name=input_bams type=data multiple=true
format=bam label=BAM file
validator type=unspecified_build /
validator type=dataset_metadata_in_data_table
table_name=gatk_picard_indexes metadata_name=dbkey
metadata_column=dbkey message=Sequences are not currently available
for the specified build. / !-- fixme!!! this needs to be a select
--
/param
And:
repeat name=input_bams title=BAM file min=1
help=-I,--input_file amp;lt;input_fileamp;gt;
param name=input_bam type=data format=bam label=BAM file
/param
/repeat
with
param name=input_bams multiple=true type=data
format=bam label=BAM file
/param
I have not tested these specific changes, so your millage may vary. I
have no clue if those validators in that second block are going to
work with multiple=true. If you test it out and there is some
problem, please let me know I can try to fix it.
I don't know if the Galaxy team wants to start replacing these blocks
in its tools - this change would break existing workflows built on the
unified genotyper and I am not sure the Galaxy team has any interest
in using these kind of data blocks going forward. They are working out
great for us on Galaxy-P though.
-John
On Tue, Oct 1, 2013 at 7:57 AM, Peter Cock p.j.a.c...@googlemail.com wrote:
On Tue, Oct 1, 2013 at 1:27 PM, Ulf Schaefer ulf.schae...@phe.gov.uk wrote:
Dear all
We frequently find ourselves in situations where a tool needs to be run
with a lot of input files. For example, run the GATK UnifiedGenotyper
with easily dozens of bam files.
Using the repeat in this case requires quite a bit of clicking. Is
there a more conventient way fo doing this? Maybe similar to the
multi-file-select that is possible for workflow inputs?
I saw some older discussions on this or similar issues, but I am a bit
lost what the current official stable proposed solution for this is.
Thanks for your help
Ulf
Use param type=data multiple=true ... instead of using
repeat ...param type=data .../repeat.
I asked John Chilton about this recently via Twitter, and it is
now on the wiki,
http://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax
Simple example here:
https://bitbucket.org/galaxyp/galaxyp-toolshed-iquant/src/tip/iquant.xml?at=default
Peter
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] multiple input files
Hi Ulf, What I do: 1. make a history, doing the steps you want to do on one input file 2. create a workflow of that history 3. assemble all input files in one history 4. run the workflow and select the multiple input files to run the workflow on. 5. Optionally: send the results to a new history, for every input file you will get a new history, properly named. Hope this helps, Joachim Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 10/01/2013 02:27 PM, Ulf Schaefer wrote: Dear all We frequently find ourselves in situations where a tool needs to be run with a lot of input files. For example, run the GATK UnifiedGenotyper with easily dozens of bam files. Using the repeat in this case requires quite a bit of clicking. Is there a more conventient way fo doing this? Maybe similar to the multi-file-select that is possible for workflow inputs? I saw some older discussions on this or similar issues, but I am a bit lost what the current official stable proposed solution for this is. Thanks for your help Ulf ** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] multiple input files
Sorry, my answer doesn't fit your question. :-) J Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 10/01/2013 03:08 PM, Joachim Jacob | VIB | wrote: Hi Ulf, What I do: 1. make a history, doing the steps you want to do on one input file 2. create a workflow of that history 3. assemble all input files in one history 4. run the workflow and select the multiple input files to run the workflow on. 5. Optionally: send the results to a new history, for every input file you will get a new history, properly named. Hope this helps, Joachim Joachim Jacob Contact details: http://www.bits.vib.be/index.php/about/80-team On 10/01/2013 02:27 PM, Ulf Schaefer wrote: Dear all We frequently find ourselves in situations where a tool needs to be run with a lot of input files. For example, run the GATK UnifiedGenotyper with easily dozens of bam files. Using the repeat in this case requires quite a bit of clicking. Is there a more conventient way fo doing this? Maybe similar to the multi-file-select that is possible for workflow inputs? I saw some older discussions on this or similar issues, but I am a bit lost what the current official stable proposed solution for this is. Thanks for your help Ulf ** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] multiple input files
Dear Peter, dear John, dear all
Thanks very much. This is exactly what I need and the code changes
proposed work perfectly as they are. At a short glance even the
validators work as intended.
Now for the optional special bonus:
Is there a way to define the size of the boxes? The ones I see have 4
lines. Can I make them resizable somehow or define a larger size?
(size=10 or resize=true does not seem to do the trick.) Not a very
pressing issue though.
@Joachim: I am using the same approach elsewhere, but the difference is
that it runs a tool multiple times, instead of running it once on
multiple inputs. But thanks anyway.
Cheers
Ulf
On 01/10/13 14:08, John Chilton wrote:
Thanks Peter,
Yeah, looking over unified_genotyper.xml, you would probably just
want to replace:
#for $i, $input_bam in enumerate( $reference_source.input_bams ):
-d -I ${input_bam.input_bam} ${input_bam.input_bam.ext}
gatk_input_${i}
#if str( $input_bam.input_bam.metadata.bam_index ) != None:
-d ${input_bam.input_bam.metadata.bam_index}
bam_index gatk_input_${i} ##hardcode galaxy ext type as bam_index
#end if
#end for
with
#for $i, $input_bam in enumerate( $reference_source.input_bams ):
-d -I ${input_bam} ${input_bam.ext} gatk_input_${i}
#if str( $input_bam.metadata.bam_index ) != None:
-d ${input_bam.metadata.bam_index} bam_index
gatk_input_${i} ##hardcode galaxy ext type as bam_index
#end if
#end for
And:
repeat name=input_bams title=BAM file min=1
help=-I,--input_file amp;lt;input_fileamp;gt;
param name=input_bam type=data format=bam label=BAM
file
validator type=unspecified_build /
validator type=dataset_metadata_in_data_table
table_name=gatk_picard_indexes metadata_name=dbkey
metadata_column=dbkey message=Sequences are not currently available
for the specified build. / !-- fixme!!! this needs to be a select
--
/param
/repeat
with
param name=input_bams type=data multiple=true
format=bam label=BAM file
validator type=unspecified_build /
validator type=dataset_metadata_in_data_table
table_name=gatk_picard_indexes metadata_name=dbkey
metadata_column=dbkey message=Sequences are not currently available
for the specified build. / !-- fixme!!! this needs to be a select
--
/param
And:
repeat name=input_bams title=BAM file min=1
help=-I,--input_file amp;lt;input_fileamp;gt;
param name=input_bam type=data format=bam label=BAM
file
/param
/repeat
with
param name=input_bams multiple=true type=data
format=bam label=BAM file
/param
I have not tested these specific changes, so your millage may vary. I
have no clue if those validators in that second block are going to
work with multiple=true. If you test it out and there is some
problem, please let me know I can try to fix it.
I don't know if the Galaxy team wants to start replacing these blocks
in its tools - this change would break existing workflows built on the
unified genotyper and I am not sure the Galaxy team has any interest
in using these kind of data blocks going forward. They are working out
great for us on Galaxy-P though.
-John
On Tue, Oct 1, 2013 at 7:57 AM, Peter Cock p.j.a.c...@googlemail.com wrote:
On Tue, Oct 1, 2013 at 1:27 PM, Ulf Schaefer ulf.schae...@phe.gov.uk wrote:
Dear all
We frequently find ourselves in situations where a tool needs to be run
with a lot of input files. For example, run the GATK UnifiedGenotyper
with easily dozens of bam files.
Using the repeat in this case requires quite a bit of clicking. Is
there a more conventient way fo doing this? Maybe similar to the
multi-file-select that is possible for workflow inputs?
I saw some older discussions on this or similar issues, but I am a bit
lost what the current official stable proposed solution for this is.
Thanks for your help
Ulf
Use param type=data multiple=true ... instead of using
repeat ...param type=data .../repeat.
I asked John Chilton about this recently via Twitter, and it is
now on the wiki,
http://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax
Simple example here:
https://bitbucket.org/galaxyp/galaxyp-toolshed-iquant/src/tip/iquant.xml?at=default
Peter
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
**
The information contained in the EMail and any attachments is confidential and
intended solely
Re: [galaxy-dev] Dynamic data library
On 10/01/2013 03:53 PM, Cole, Nathan (NIH/NCI) [C] wrote: Thank you both for your responses. I will be looking into both of these. With regard to the from_file option to add the sample selection into the tool: I assume this means that the metadata and everything is loaded into galaxy at the time the tool is run. This depends on how you write your tool. Do you just wanna read the ie fastq file or do you also wanna read the meta data. Also, how is the meta data accessible? eg. is it stored in a txt file at the same location as the fastq file? Does this create a copy of the loaded file or simply read it in place? Also are there any efficiency issues created using this method, outside of the tool run time increase due to the load of the data taking place in-tool? It should just read it in place Hans-Rudolf Thanks, Nathan -Original Message- From: Hans-Rudolf Hotz [mailto:h...@fmi.ch] Sent: Tuesday, October 01, 2013 4:07 AM To: Cole, Nathan (NIH/NCI) [C] Cc: Martin Čech; galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Dynamic data library Hi Nathan Do you have many tools working with those samples or just a few? If you only have a limited, predefined set of tools you might wanna consider adding the sample selection into the tool. You can use the from_file, or from_data_table options to dynamically create sample selection list. You can even drill down a hierarchical list. Have a look at ~/tools/annotation_profiler/annotation_profiler.xml which uses the file ~/tool-data/annotation_profiler_options.xml All you need to do is keeping the file in sync with the directory structure of your samples directory Regards, Hans-Rudolf On 09/30/2013 09:48 PM, Martin Čech wrote: Hi Nathan, Dannon answered similar question few days ago: There's an import mechanism in libraries that'll allow you to simply link to the file on disk without copy/upload. I believe the example_watch_folder.py sample script (in the distribution) does just this via the API, if you want an example. This might be what you are looking for. Martin On Mon, Sep 30, 2013 at 2:43 PM, Cole, Nathan (NIH/NCI) [C] nathan.c...@nih.gov mailto:nathan.c...@nih.gov wrote: Hello, we’ve set up a local Galaxy instance in our genotyping and next-gen sequencing lab with local Apache LDAP (AD) integration, NFS mounts to a large NAS, and cluster integration coming. Do to the high volume of samples and staff that will be using the system, I want to set up data libraries (without copying to Galaxy). This is obviously no problem the first time, however I was wondering if there was a way to make a library, added from a system path, be dynamic so that it would stay synchronized with the underlying file structure? __ __ If a try dynamic library is not possible, is there a method for adding files to an existing library via that same system path that would not duplicate all of the original files in the data library? __ __ I did some scouring of the list and found some old unanswered questions and some things tangentially related topics, but I was unable to find a true answer or solution to my problem. Any information on how to do the tasks above or other solutions to provide the same functionality would be greatly appreciated. __ __ Thanks, Nathan __ __ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] cleanup_datasets.py with postgresql 8.1
Any feedback on this? Our local instance of Galaxy is ready to go live and this is one of the remaining issues keeping us from more wide-spread usage. Can anyone confirm that I should be using scripts/cleanup_datasets/cleanup_datasets.py when I'm using postgresql? Are there any limits on the versions of postgresql? Thanks in advance, Malcolm On Thursday 26 September 2013 17:16:16 Jennifer Jackson wrote: Posting to galaxy-...@bx.psu.edumailto:galaxy-...@bx.psu.edu to give the question better exposure to the development community. Please remove the galaxy-u...@bx.psu.edumailto:galaxy-u...@bx.psu.edu mailing list from all replies - we want to avoid double posts. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev On 9/26/13 7:29 AM, Malcolm Tobias wrote: All, My understanding is that the cleanup_datasets.py under scripts/cleanup_datasets should be compatible with 'older' versions of postgresql. I'm running 8.1 under CentOS 5. When I attempt to run the scripts, it fails to clean any data sets. From the logs, I'm noticing messages like: database_connection contains an unknown SQLAlchemy database dialect: postgresql In my universe_wsgi.ini I've defined the database_connection as: database_connection = postgresql://galaxy:@localhost:5432/galaxydb and am confident it's working as Galaxy works fine, other than this problem with deleting older datasets. The pgcleanup.py script isn't an option, as it requires postgresql =9.1. I noticed yet another version of cleanup_datasets.py under cron which looks like it might be compatible with postgres, but am leary of trying it as I'm not sure what it means by 1: database directory to clean. I'm not sure what directory to point this to, nor am I confident that just deleting data from the directory is the safest thing. Any suggestions on how to proceed? Thanks! Malcolm -- Malcolm Tobias 314.362.1594 ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson http://galaxyproject.org -- Malcolm Tobias 314.362.1594 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] 2 RNAseq files in queue to map w/Tophat in Main since last Thursday, please advise
Greetings, I have had two RNAseq files in queue to map with Top hat in Galaxy Main since last Thursday (History began Sept 19). Yesterday I got word that NGS was stalled, my jobs were mistakenly believed to be moving afterwards when they weren't- but are now under review again. This morning the 2 mapping jobs are still waiting to run. I have a data review meeting this thursday and hope you can advise me on contingency plans to get these 2 mapping jobs running. Can you tell me whether I should delete the jobs and re-run them as new? My concern is that they are still believed to be moving as was indicated by one of your galaxy stewards, but are actually still stalled in space. I have a data meeting Thursday (10/4) and hoped to be ready by starting this project Sept 19th. Can you tell me if you are aware of another public Galaxy server (other than Main) with 'Top Hat' for mapping RNAseq data? I joined Cistrome (Harvard) but they only have Cufflinks (for previously mapped RNAseq analysis) below is my history thank you very much for your help https://main.g2.bx.psu.edu/u/trstueve/h/rnaseqjegcsc3wkrun130924-1 https://main.g2.bx.psu.edu/history/sharing# Theresa Ryan Stueve 510-225-8894 (text better than voicemail for quick response) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Local Galaxy Install - Debug Report FROM address as Admin and not User?
Howdy, On our local Galaxy install, everytime a user submits a bug report, it does email all the admins -- as defined in universe_wsgi.ini -- but the FROM address is not of the user, but rather the first email address from the admin list. Is this normal? May we have it changed so the FROM address is that of the user? Thanks, -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local Galaxy Install - Debug Report FROM address as Admin and not User?
Hi Adam, you are right about the behaviour, this is what Galaxy does. However it sends the email to the user too, so Reply all will reach the user's email address. This is how the feature was meant to be used. Martin On Tue, Oct 1, 2013 at 12:59 PM, Adam Brenner aebre...@uci.edu wrote: Howdy, On our local Galaxy install, everytime a user submits a bug report, it does email all the admins -- as defined in universe_wsgi.ini -- but the FROM address is not of the user, but rather the first email address from the admin list. Is this normal? May we have it changed so the FROM address is that of the user? Thanks, -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local Galaxy Install - Debug Report FROM address as Admin and not User?
Martin, Thanks for the response. Yes, I do notice the reply-to address does include the user, but wouldn't it be better if the FROM address was from the actual user reporting the error? When checking my email, for example, I do not want to see emails that are FROM me, to MYSELF when its really from our users. Do you know which of the python files contain the code that relates to this? Or point me in the right direction so I can change this? Thanks! -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu On Tue, Oct 1, 2013 at 10:52 AM, Martin Čech mar...@bx.psu.edu wrote: Hi Adam, you are right about the behaviour, this is what Galaxy does. However it sends the email to the user too, so Reply all will reach the user's email address. This is how the feature was meant to be used. Martin On Tue, Oct 1, 2013 at 12:59 PM, Adam Brenner aebre...@uci.edu wrote: Howdy, On our local Galaxy install, everytime a user submits a bug report, it does email all the admins -- as defined in universe_wsgi.ini -- but the FROM address is not of the user, but rather the first email address from the admin list. Is this normal? May we have it changed so the FROM address is that of the user? Thanks, -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local Galaxy Install - Debug Report FROM address as Admin and not User?
Adam, it is function *report_error* in /lib/galaxy/webapps/galaxy/controllers/dataset.py you need to set this line * frm = to_address * to * frm = email* and you should be all set. Martin On Tue, Oct 1, 2013 at 2:12 PM, Adam Brenner aebre...@uci.edu wrote: Martin, Thanks for the response. Yes, I do notice the reply-to address does include the user, but wouldn't it be better if the FROM address was from the actual user reporting the error? When checking my email, for example, I do not want to see emails that are FROM me, to MYSELF when its really from our users. Do you know which of the python files contain the code that relates to this? Or point me in the right direction so I can change this? Thanks! -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu On Tue, Oct 1, 2013 at 10:52 AM, Martin Čech mar...@bx.psu.edu wrote: Hi Adam, you are right about the behaviour, this is what Galaxy does. However it sends the email to the user too, so Reply all will reach the user's email address. This is how the feature was meant to be used. Martin On Tue, Oct 1, 2013 at 12:59 PM, Adam Brenner aebre...@uci.edu wrote: Howdy, On our local Galaxy install, everytime a user submits a bug report, it does email all the admins -- as defined in universe_wsgi.ini -- but the FROM address is not of the user, but rather the first email address from the admin list. Is this normal? May we have it changed so the FROM address is that of the user? Thanks, -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] configuring auth_ldap for galaxy
I think I am nearly there with the auth_ldap, but when I attempt to
access the site I get the following message:
Access to Galaxy is denied
Galaxy is configured to authenticate users via an external method (such as
HTTP authentication in Apache), but a username was not provided by the
upstream (proxy) server. This is generally due to a misconfiguration in
the upstream server.
Please contact your local Galaxy administrator.
I am using a service account in the configuration and wonder if this
could be a permissions issue.
Thanks,
Iry
On 9/18/13 10:10 AM, Maddhi, Srinivas srinivas-mad...@uiowa.edu wrote:
Another galaxy + nginx + LDAP user here.
Iry,
I have it working as well with help from Galaxy Wiki docs and mailing list
archives. The configure options that were used to compile nginx:
--add-module=../nginx-auth-ldap.20130823
--add-module=../nginx-upload-module-2.2 --with-pcre=../pcre-8.33
--with-http_ssl_module --conf-path=/usr/local/etc/nginx/nginx.conf
--pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --user=galaxy-qa
--group=galaxy-qa --with-debug
I also supplied the proxy_set_header X-URL-SCHEME https; directive in
the / and /api location blocks.
Good luck!
John,
Thanks for the detailed response to Iry's question, and docs for this
setup.
Related question: Are you aware of a mechanism that would transmit LDAP
attributes, say mail for example, to downstream apps ? I have that
question in to the author of that module and haven't heard back.
The need for that: in my use case, the email address associated with
remote_user is not remote_u...@example.com but rather
first_name-last_n...@example.com. As such, I am stuck, less than
ideally, with requiring users to use first_name-last_n...@example.com as
their username instead of just short_name. Our previous solution to this
was to hack galaxy-dist/lib/galaxy/web/framework/__init__.py which I would
like to avoid for ease of maintainability purposes.
Thank you.
-Original Message-
From: John Chilton chil...@msi.umn.edu
Date: Wednesday, September 18, 2013 7:49 AM
To: Iry Witham iry.wit...@jax.org
Cc: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] configuring auth_ldap for galaxy
You have to compile nginx with the auth-ldap module. Have you done
this? I am doing using CloudBioLinux and here is how it is being done
in that context, but it would probably be easier just doing it by
hand:
https://github.com/chapmanb/cloudbiolinux/blob/master/cloudbio/galaxy/__i
n
it__.py
I have posted my information on configuring CloudMan's nginx for LDAP
auth, but you can ignore the CloudMan stuff:
https://production-galaxy-instances-with-cloudman-and-cloudbiolinux.readt
h
edocs.org/en/latest/#configuring-nginx-conf
http {
auth_ldap_url
ldap://ldap.example.com/dc=example,dc=com?uid?sub?(objectClass=person);
#auth_ldap_binddn cn=nginx,ou=service,dc=example,dc=com;
#auth_ldap_binddn_passwd mYsUperPas55W0Rd
#auth_ldap_group_attribute uniquemember; # default 'member'
#auth_ldap_group_attribute_is_dn on; # default on
...
}
location / {
auth_ldap_require valid_user;
auth_ldap LDAP Auth Source Description;
proxy_set_header REMOTE_USER $remote_user;
proxy_pass http://galaxy_app;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-URL-SCHEME https;
...
}
# For API access, set REMOTE_USER if available so Galaxy
# session based requests are let through, if REMOTE_USER is not
# available pass the request through and let Galaxy determine
# if a key is present and valid.
location /api {
proxy_set_header REMOTE_USER $remote_user;
proxy_pass http://galaxy_app;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
-John
On Wed, Sep 18, 2013 at 7:37 AM, Iry Witham iry.wit...@jax.org wrote:
Hi Team,
Has anyone had any experience with setting up auth_ldap for nginx? I
am
attempting to make this work, but nginx fails to start with the
following
message:
Starting nginx nginx: [emerg] unknown directive auth_ldap_url in
/usr/local/nginx/conf/nginx.conf:48
startproc: exit status of parent of /usr/local/nginx/sbin/nginx: 1
failed
Any help would be appreciated.
Thanks,
Iry Witham
The information in this email, including attachments, may be
confidential
and is intended solely for the addressee(s). If you believe you
received
this email by mistake, please notify the sender by return email as soon
as
possible.
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
[galaxy-dev] Test post, please ignore
(Just testing that my Exchange Server admin guy has managed to disable the stupid legal disclaimers on my postings to this list.) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency NumPy
I've done some investigating, and it looks like there is an irregularity with the installation of package_numpy_1_7, and somehow package_biopython_1_62 is not finding the (successfully) installed numpy dependency. I duplicated the dependency chain for seq_select_by_id on my local tool shed, and got the following error for biopython's tool dependency record when installing seq_select_by_id into my local Galaxy instance: export PYTHONPATH=$PYTHONPATH:/var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python export PATH=$PATH:$PATH_NUMPY export PYTHONPATH=$PYTHONPATH:$PYTHONPATH_NUMPY python setup.py install --install-lib /var/galaxy/tool_dependencies/biopython/1.62/devteam/package_biopython_1_62/c3c8665cb0af/lib/python STDOUT running install Numerical Python (NumPy) is not installed. This package is required for many Biopython features. Please install it before you install Biopython. You can install Biopython anyway, but anything dependent on NumPy will not work. If you do this, and later install NumPy, you should then re-install Biopython. You can find NumPy at http://numpy.scipy.org Note that when I installed seq_select_by_id, numpy installed correctly, and PYTHONPATH was populated with the right values. --Dave B. On 09/30/2013 09:10 AM, Bjoern Gruening wrote: Am Montag, den 30.09.2013, 14:02 +0100 schrieb Peter Cock: On Mon, Sep 30, 2013 at 1:58 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, It looks like there may be a version conflict between package_numpy and either a system numpy or the Galaxy numpy egg. I'll investigate and keep you updated on my progress. --Dave B. Thanks Dave, If Tool authors can assume NumPy is already present via the Galaxy numpy egg and/or the system Python's numpy, that might be simpler... The question is how to import the eggs into the tools? Is that possible at all? For example pysam is shipped with Galaxy right? But is there a way to use it inside of a tool? But I think even if that is possible we need separate pysam/numpy repositories for reproducibility. I agree with Björn, it's probably not a good idea to use Galaxy eggs in tools and their dependencies, since the egg versions can change at any time, which might impact reproducibilty. Cheers, Bjoern Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] how to generate .len file for custom genome? (trackster error)
Trackster complains to me when I try to load up my custom reference genome to compare some sample data to. could not load chroms for this dbkey I think it's because I'm using the newest g1k human ref genome v37, and I don't know if the hg19.len file works with it... I generated my own twobit file for it, but now I think I need the .len file... To be honest I'm kind of lost. Where does trackster even look for my genome, and how do I generate a .len file? (I read the visualizations page and the custom genome docs page) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
