[galaxy-dev] Question on setting metadata on upload via API

I'm looking at extending the metadata fields for one of the supported file types. The files can get VERY large, and since I'm creating those files, I'd like to save as metadata some of the information I have on the contents. Specifically, I'd like to tag the files with information about the samp

Re: [galaxy-dev] Question on setting metadata on upload via API

riginal Message- From: Nate Coraor [mailto:n...@bx.psu.edu] Sent: Friday, May 06, 2011 8:12 AM To: Duddy, John Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Question on setting metadata on upload via API Duddy, John wrote: > I'm looking at extending the metadata fields for on

[galaxy-dev] Unable to set metadata in API call

I need to be able to set some metadata in some custom data types. For now, I'm just trying to set the value of the 'misc_info' field. The client script is this: put( sys.argv[1], sys.argv[2], { 'update_type' : 'metadata', 'misc_info' : 'meta data msg' } ) and my API method is as follows. It exec

Re: [galaxy-dev] Unable to set metadata in API call

..@illumina.com -Original Message- From: Nate Coraor [mailto:n...@bx.psu.edu] Sent: Monday, May 09, 2011 8:27 AM To: Duddy, John Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Unable to set metadata in API call Duddy, John wrote: > I need to be able to set some metadata in some cust

Re: [galaxy-dev] A tool with no inputs

Doesn't this violate one of the basic tenets of Galaxy - reproducibility? Without the ability to provide full traceability to the inputs, one can make no guarantees about the outputs. John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3

[galaxy-dev] Getting binary programs into Galaxy distribution?

There is a C program for merging Gzip files (gzjoin) that I'd love to rely on for a core Galaxy capability. Is there a standard way to get things like this included in Galaxy? Recoding it in Python would be a bit of a pain, and might be a lot slower due to the IO layer not allowing the reuse of

Re: [galaxy-dev] Getting binary programs into Galaxy distribution?

e is stuff that happens on first start, such as copying *example files. Would it be far-fetched to compile the program at that stage? From: James Taylor [ja...@jamestaylor.org] Sent: Tuesday, May 24, 2011 11:51 PM To: Duddy, John Cc: galaxy-...@bx.p

Re: [galaxy-dev] Getting binary programs into Galaxy distribution?

92121 Tel: 858-736-3584 E-mail: jdu...@illumina.com -Original Message- From: Nate Coraor [mailto:n...@bx.psu.edu] Sent: Wednesday, May 25, 2011 9:24 AM To: Duddy, John Cc: James Taylor; galaxy-...@bx.psu.edu Dev Subject: Re: [galaxy-dev] Getting binary programs into Galaxy distribution? D

[galaxy-dev] Is dynamic associated information per dataset possible?

We'd like to be able to associate fixed things (project, Sample, sequencer used) with user's FASTQ files, and we'd also like to allow users to associate dynamic, site-specific stuff with the sequencing run. Currently, users track their runs using a CSV sample sheet, and often they add columns to

Re: [galaxy-dev] Is dynamic associated information per dataset possible?

26, 2011 1:38 AM To: Duddy, John Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Is dynamic associated information per dataset possible? Duddy, John wrote: > We'd like to be able to associate fixed things (project, Sample, sequencer > used) with user's FASTQ files, a

Re: [galaxy-dev] Is dynamic associated information per dataset possible?

ginal Message- From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Thursday, May 26, 2011 2:18 AM To: Duddy, John Cc: galaxy-dev Subject: Re: [galaxy-dev] Is dynamic associated information per dataset possible? In addition to Data Library templates, which are useful after the sequencer dat

[galaxy-dev] Accessing Data Library Template fields in tools?

I have my data in a data library and have a form template defined so I can enter the sample information. So, I import a data file into a history and want to run a tool on it. Can I pass the values of those form templates to my tool? Sort of line ${input.form_field_id} ? Thanks! John Duddy Sr.

[galaxy-dev] Sharing authentication between Galaxy and other WSGI apps on the same web server (with custom UI)?

I'd like to have Galaxy and another application installed on the same Apache server and have the user authenticate only once. I think I understand how to do that by deferring authentication to Apache (instead of using Galaxy's built-in database). So far, so good, I think. What I'm wondering is

Re: [galaxy-dev] Sharing authentication between Galaxy and other WSGI apps on the same web server (with custom UI)?

Thanks! That's perfect. John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jdu...@illumina.com -Original Message- From: Assaf Gordon [mailto:gor...@cshl.edu] Sent: Monday, June 20, 2011 12:09 PM To: Duddy,

[galaxy-dev] Getting a list of workflows a user can run via the API

I am doing an integration with Galaxy, and part of what I need to do is trigger workflows. To do that, I need to list them. I can do this if the user owns the workflows, but the API does not return workflows that have been shared with the user. Is there a way via the API to discover the sharing

Re: [galaxy-dev] Cluster setup - shared temporary directory

I can give you a very good example - if you are doing alignment and for some reason need to convert the input file before operating on them, such that you need a complete copy, /tmp may not have enough room. I have had this happen to me running lots of instances of an aligner, temporarily using

Re: [galaxy-dev] Problems with Galaxy on a mapped drive

We had similar problems on NFS mounts to Isilon. We traced it to the default timeout for attribute caching on NFS mounts, which does not force a re-read of directory contents (hence file existence or size) for up to 30 seconds. We worked around it by adding no-ac to the mount, but this can drast

Re: [galaxy-dev] using Galaxy for map/reduce

I did something similar, but implemented as an evolution of the original "basic" parallelism (see BWA), that: - Moved the splitting of input files into the datatype classes - Allowed any number of inputs to be split, as long as they were the same datatype (so they were mutually consistent - think

[galaxy-dev] Question on timing with API, running workflows, and setting metadata

I'd like to have an external program that registers a file by absolute path (link, not upload) in a data library, then immediately starts a workflow on it. My question is to whether or not that will work in the general case where: - The system is configured to set metadata externally -

[galaxy-dev] Customizing/reusing the workflows/run.mako template

I'd like to integrate with Galaxy and launch workflows, and I am hoping to reuse Galaxy's support for prompting the user for items that are tagged as being set at runtime. It looks to me that I could load the workflows/run page in a frame on my app (on the same server) and reuse it. The proble

Re: [galaxy-dev] Customizing/reusing the workflows/run.mako template

lt;mailto:jdu...@illumina.com> From: James Taylor [mailto:ja...@jamestaylor.org] Sent: Tuesday, August 09, 2011 5:41 PM To: Duddy, John Cc: galaxy-dev Subject: Re: [galaxy-dev] Customizing/reusing the workflows/run.mako template John, the prefixes like "22|" are added to the inputs a

[galaxy-dev] Modifying how runtime inputs are resolved when running a workflow

This is probably something only the Galaxy devs can answer, but I thought I'd give it a shot in the wider community. Some of you are doing some very complicated stuff. If you have a workflow with several input blocks, you might have multiple fastq files you need to provide. A good example of th

[galaxy-dev] "multiple" inputs in tools used in workflows?

While snooping around the Galaxy code, I noticed that some tool features are not supported in workflows, only in histories. Is there a list somewhere that lists the restrictions? Specifically, are "multiple" inputs supported? John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Cent

[galaxy-dev] Storing a dict as metadata

I'd like to have a datatype with a dict as metadata. This dict() would store file offsets to enable seeking around to process different sections of the file. How do I add a dictionary data metadata element? John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego

Re: [galaxy-dev] Storing a dict as metadata

Sent: Friday, August 26, 2011 5:37 AM To: Duddy, John Cc: galaxy-dev Subject: Re: [galaxy-dev] Storing a dict as metadata Hey John, are you sure you don't want to use a "converted dataset" rather than a metadata element for this. This is how we handle most types of secondary index

Re: [galaxy-dev] using Galaxy for map/reduce

se seek to move the file pointer so each task can grab it's part. On Tue, Aug 2, 2011 at 10:54 AM, Duddy, John wrote: > I did something similar, but implemented as an evolution of the original > "basic" parallelism (see BWA), that: > - Moved the splitting of input files int

Re: [galaxy-dev] Storing a dict as metadata

or [mailto:ja...@jamestaylor.org] Sent: Friday, August 26, 2011 1:52 PM To: Duddy, John Cc: galaxy-dev Subject: Re: [galaxy-dev] Storing a dict as metadata Not currently, but since a converted dataset is just a dataset, you could reuse all of the existing upload mechanism, and just add the conv

Re: [galaxy-dev] outputting different numbers of files based on variables?

The BWA tool in NGS mapping does just what you want, just for inputs. The general idea is to use a element and define your extra output in a block. John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jdu...@illumina.com

Re: [galaxy-dev] [galaxy-user] Add library to dataset performance metric: developer vs production instances

We routinely put large compressed fastq files into data libraries by that method (linking, no copy) and it is very fast, since the patch that stopped it decompressing the files. You should probably make sure you specify the file format (fastqsanger) so Galaxy does not attempt to sniff the file

[galaxy-dev] Running the doctet unit tests in Galaxy

There are several files in Datatypes with doctest tests in them. Is there a convenient wrapper script to run them all? John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jdu...@illumina.com _

[galaxy-dev] Tool shed and datatypes

Can we introduce new file types via tools in the tool shed? It seems Galaxy can load them if they are in the datatypes configuration file. Does tool installation automate the editing of that file? John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121

Re: [galaxy-dev] Tool shed and datatypes

e'd have to add a datatype. John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jdu...@illumina.com<mailto:jdu...@illumina.com> From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Wednesday, October 05, 2011

Re: [galaxy-dev] Tool shed and datatypes

n Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jdu...@illumina.com -Original Message- From: Peter Cock [mailto:p.j.a.c...@googlemail.com] Sent: Thursday, October 06, 2011 1:28 AM To: Duddy, John Cc: Greg Von Kuster;

Re: [galaxy-dev] Tool shed and datatypes

ed IO benefits of smaller writes. John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jdu...@illumina.com -Original Message- From: Peter Cock [mailto:p.j.a.c...@googlemail.com] Sent: Thursday, October 06, 2011 8:16 AM To:

Re: [galaxy-dev] Tool shed and datatypes

dy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jdu...@illumina.com -Original Message- From: Peter Cock [mailto:p.j.a.c...@googlemail.com] Sent: Thursday, October 06, 2011 9:19 AM To: Duddy, John Cc: Greg Von Kuster; galax

Re: [galaxy-dev] Tool shed and datatypes

f new support is added to the > Galaxy code base, an entry for that new datatype should be manually added to > the table at the same time. There may be benefits to enabling automatic > changes to datatype entries that already exist in the file (e.g., adding a > new converter for an ex

Re: [galaxy-dev] What's causing this error?

You mention that you moved it to an NFS volume - but it seems you also moved to a grid configuration using PBS? If that's the case, what you are seeing might be an issue with NFS attribute caching or write caching, which causes files created from one machine to not appear until some time later

Re: [galaxy-dev] Looks like actual user breaks splitting

com> From: Chorny, Ilya Sent: Wednesday, November 02, 2011 11:50 AM To: Duddy, John Cc: Nate Coraor (n...@bx.psu.edu); galaxy-dev@lists.bx.psu.edu Subject: Looks like actual user breaks splitting Hey John, Any thoughts? Ilya Traceback (most recent call last): File "/home/galaxy/ich

Re: [galaxy-dev] Looks like actual user breaks splitting

esday, November 02, 2011 12:24 PM To: Duddy, John Cc: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu Subject: Re: Looks like actual user breaks splitting John, Ilya, I get further with sequence type inputs but it looks like JobWrapper.get_output_datasets_and_fnames() is not returning the right thing

Re: [galaxy-dev] Looks like actual user breaks splitting

2:22 PM To: Duddy, John Cc: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu Subject: Re: Looks like actual user breaks splitting Hi John, It looks like the first issue is related to the change from get_output_fnames() -> compute_outputs(). When outputs_to_working_directory = False (default) this

Re: [galaxy-dev] Looks like actual user breaks splitting

-Original Message- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Thursday, November 03, 2011 3:53 PM To: Duddy, John Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Looks like actual user breaks splitting Nate Coraor (n...@bx.psu.edu) wrote: > Duddy, John wr

Re: [galaxy-dev] Tool shed and datatypes

to:p.j.a.c...@googlemail.com] Sent: Tuesday, November 08, 2011 3:29 PM To: Duddy, John Cc: Greg Von Kuster; galaxy-dev@lists.bx.psu.edu; Nate Coraor Subject: Re: [galaxy-dev] Tool shed and datatypes On Thu, Oct 6, 2011 at 5:45 PM, Duddy, John wrote: > GZIP files are definitely our plan. I just finished tes

Re: [galaxy-dev] Tool shed and datatypes

- From: Peter Cock [mailto:p.j.a.c...@googlemail.com] Sent: Tuesday, November 08, 2011 3:29 PM To: Duddy, John Cc: Greg Von Kuster; galaxy-dev@lists.bx.psu.edu; Nate Coraor Subject: Re: [galaxy-dev] Tool shed and datatypes On Thu, Oct 6, 2011 at 5:45 PM, Duddy, John wrote: > GZIP files are definit

Re: [galaxy-dev] Tool shed and datatypes

Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jdu...@illumina.com -Original Message- From: Peter Cock [mailto:p.j.a.c...@googlemail.com] Sent: Tuesday, November 08, 2011 4:04 PM To: Duddy, John Cc: Greg Von Kuster; galax

Re: [galaxy-dev] Galaxy in a frame?

You can put it in an iframe, and if you are serving it from the same host as your main app, you can also inject javascript/CSS to control certain aspects, such as styling and event propagation related to navigation. However, this is not a trivial exercise. John Duddy Sr. Staff Software Enginee

[galaxy-dev] Bootstrapping the Galaxy installation process: populating the database

We want to automate certain aspects of setup. We already create the target database and put it in universe_wsgi.ini, but we want to plug some values into the tables during installation. Currently, the schema is created the first time Galaxy is run. I did not see a way to cause that to happen vi