Hi Piotr,
In our proteomics lab, a protein sample is fractionated (by e.g. pH)
before analysis in a nr of sample fractions. The fractions are then run
through the mass spectrometer one at a time. Each fraction yields a data
file.
The mass spec data is then matched to peptides by searching a FASTA
file, termed target, with protein sequences. Afterwards the matches are
statistically scored by machine learning. To do this, the data is also
matched with a scrambled FASTA file, termed decoy. Each fraction is
matched to a target and decoy file, which yields two match-files per
fraction.
The machine learning tool thus picks a target and a decoy matchfile and
puts statistical significances on the matches. In order for this to be
correct, it needs to pick matchfiles that correspond, ie that are
derived from the same fraction.
In our lab, we have not yet looked at John Chilton's (I think) work with
the m: data sets, and our parallel processing is done inside galaxy,
using its split and merge functions to divide a job into tasks. Each
task is sent as a separate job to sge, I think, but others may know more
about this than I.
I really have to get back to my holiday now, cheers,
jorrit
On 08/01/2013 04:17 AM, piotr.s...@csiro.au wrote:
Hi Jorrit,
Thank you for your explanation. Would you be able to give us an
example of what do you mean by fractions and when the task_%d are
being used to pick files. Just want to make sure we have good
understanding of the problem that you solved.
Also, I vaguely remember seeing 'data parallelism" mentioned somewhere
with relation to the m: data sets. Do you currently support in any
way automatic distribution of processing of such datasets to parallel
environments (e.g. array jobs in sge or such?)
Cheers,
-Piotr
*From:*Jorrit Boekel [mailto:jorrit.boe...@scilifelab.se]
*Sent:* Wednesday, July 31, 2013 8:18 PM
*To:* Khassapov, Alex (CSIRO IM&T, Clayton)
*Cc:* p.j.a.c...@googlemail.com; jmchil...@gmail.com;
galaxy-dev@lists.bx.psu.edu; Szul, Piotr (ICT Centre, Marsfield);
Burdett, Neil (ICT Centre, Herston - RBWH)
*Subject:* Re: Appending _task_%d suffix to multi files
Hi Alex,
In our lab, files are often fractions of an experiments, but they are
named by their creators in whatever way they like. I put that code in
to standardize fraction naming, in case a tool needs input from two
files that originate from the same fraction (but have been treated in
different ways). In those cases, in my fork, Galaxy always picks the
files with the same task_%d numbers.
I can't help you very much right now, as I'm currently away from work
until October, but I hope this explains why its in there.
cheers,
jorrit
On 07/31/2013 04:15 AM, alex.khassa...@csiro.au
<mailto:alex.khassa...@csiro.au> wrote:
Hi guys,
We've been using Galaxy for a year now, we created our own Galaxy
fork where we were making changes to adapt Galaxy to our
requirements. As we need "multiple file dataset" - we were using
Johns' fork for that initially.
Now we are trying to use "The most updated version of the multiple
file dataset stuff" https://bitbucket.org/msiappdev/galaxy-extras/
directly as we don't want to maintain our own version.
One of the problems we have - when we upload multiple files -
their file names are changed (_task_%d suffix is added to their
names).
On our branch we simply removed the code which does it, but now we
wonder if it is possible to avoid this renaming somehow? I.e. make
it configurable?
Is it really necessary to change the file names?
-Alex
-----Original Message-----
From: galaxy-dev-boun...@lists.bx.psu.edu
<mailto:galaxy-dev-boun...@lists.bx.psu.edu>
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Jorrit
Boekel
Sent: Thursday, 25 October 2012 8:35 PM
To: Peter Cock
Cc: galaxy-dev@lists.bx.psu.edu <mailto:galaxy-dev@lists.bx.psu.edu>
Subject: Re: [galaxy-dev] the multi job splitter
I keep the files matched by keeping a _task_%d suffix to their
names. So each task is matched with its correct counterpart with
the same number.
cheers,
jorrit
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
