On 12-Mar-08, at 3:44 PM, Roger Hull wrote:
Hi Syed,
Hi Roger,
I'll let Syed reply to you in detail to your questions but let me put
this into the context of our future developments.
Thanks for the advice. I think we see quite big risks in doing
changes which depend on our understanding too much of how your code
works. This could present problems in maintaining the code we have
written, or changed, in the future, when BioMart is upgraded or we
need to add new features.
I would like to ask some questions about BioMart upgrades:
(A) Will you maintain backward compatibility between a BioMart 0.6
installation which gets its data from a remote BioMart MartService,
when the remote BioMart is upgraded to 0.7,..., 1.0, etc ?
We do currently maintain the compatibility between our 0.6 service and
all other marts which run earlier versions. The central sever is a
'translation point' which maintains backward compatibility.
Not all of the BioMart servers accessible from our central server are
0.6 but you can query them in uniform fashion. It is our intention to
maintain this compatibility in a similar way in the future.
(B) Do you have a date for BioMart 0.7, and have you published a
there a list of enhancements and bug fixes for that release?
the preliminary ETA is april
(C) There seem to be a number of BioMart installations where people
have modified your code. As I said, I'm reluctant to follow this
path, but I wonder if you could consider adding functions, hooks, or
similar mechanisms in your code so that the BioMart behaviour can be
changed in various ways without modifying your code? (The files
header.tt and footer.tt are useful, but it is limited what can be
done by adding code to these files.)
yes, the web code will undergo a major re-organization for 0.8 which
will coincide with the release of the new configuration system. One of
the main goals of this 're-organization' is to provide a flexible
framework by
which people can extend the code both in terms of web GUI but also
things like visualization etc. This is still at a very early stage so
any suggestions are very welcome.
A couple of suggestions, based on what I have been wanting to do:
(C1) I would like to call your AJAX functions for my own
purposes. But your function doAjaxMagic(toDo) only supports the two
values of toDo = 'countByAjax' and 'resultsByAjax'. Could you
provide a general purpose function which anyone could use? - with an
argument to specify the URL which will handle the request and
another to supply a function to handle the results (preferably
supporting POST as well as GET). Then use this function internally
to implement doAjaxMagic.
If I modify your current function, or make a modified copy, then I
have to maintain this code in the future if you change the function
internally (e.g. to support new browser versions).
(C2) Could you implement, and document, a callback function in
perl, which has as input arguments the results of a query in a
parsable format (maybe XML), so that by default this function
returns the input results unchanged. Then if the user wants to
filter or otherwise modify the result data, this can be done by
adding code to this function, and modifying the data before
returning it to the caller. [OK, it's not quite as simple as this,
because you batch the results data and return a certain number of
result rows at a time, and if some are filtered out, some more have
to be processed to make up the number, but I'm sure a solution can
be found.]
When I looked at your code to see how and where I might do this type
of result modification, so that various formats (HTML, CSV, etc) and
various output methods (display in MartView, download to a file),
are all supported, it needed quite a lot of study of your code, and
might well present difficulties to maintain. You have a perl API
based upon BioMart::QueryRunner, which you use internally in your
code, but the only way I can see to get at the results using this
API is $query_runner->printResults(), which gives already formatted
results. I want to get at the results before formatting, so I don't
have to parse data formatted in various ways, and reformat the data
after modification.
Apart from filtering the results, another common requirement would
be to add links to fields when the results are to be formatted as
HTML, or to add another column of fields fetched from a local non-
BioMart database, using one of the BioMart attributes in the results
as an index to retrieve data from this database.
(D) As I mentioned earlier, the Count returned by BioMart is "how
many rows in the main table of the dataset match your filters so
far". In the case of PRIDE, this is the number of experiments. But
often I need to know the number of rows returned from the query, not
the number of rows from the main table - for example I want to know
the number of distinct proteins or peptides which match my filters.
Will you support a more usual "number of rows returned" count in the
future?
There are two problems here: one the counts for multiple mains, the
second the count of the actual number of rows returned. As far as the
mains are concerned (which I think would solve your protein problem
for PRIDE) we do intend to provide a proper count for each main table
ei, your query has selected this many experiments, proteins etc ...
how many mains you happen to have there as oppose to what we support
now which is a top main. As far as the number of rows is concerned the
implementation is really trivial for single datasets and the only
concern there was the unpredictable performance. For large datasets
this frequently ends up taking longer than the preview of the resutls
for the actual query so it is not always works nice in an interactive
environment. A bigger challenge is to provide a row count for
federated queries. We are currently thinking of all possible scenarios
for all of them
a.
Regards,
Roger.
Syed Haider wrote:
Hi Roger,
On Tue, 2008-02-26 at 14:15 +0000, Roger Hull wrote:
Hi Syed,
Ah, I misunderstood you. Yes, maintenance is important to me. But
can
you give me some advice on how to use this file?
(1) Where in biomart.tt do I embed resultspage.tt? Which code do I
copy across into resultspage.tt (copy rather than move?)
biomart.tt contains HTML and [%%][**] code. The latter are the
notations
used to carry out simplistic programming in terms of perl CPAN
package
'template toolkit' also referred to as TT2.
HTML code is quite self-explanatory [% results_string %] is pasted
initially by Web.pm but carries no significance. Later on, when user
interacts with the MartView, there is a
<td id="resultsTableId" ....
which gets updated by javascript in martview.js using AJAX, which you
search in the said file using keyword AJAX.
I would recommend first see how the biomart.tt works. When you change
something in this file, no need to restart server or anything, just
refresh browser and you will see the changes. However, as you know by
now thats not the case if you change javascript or Web.pm.
Once you are confident what you want to do/change, then better
tranfer
things over to resultspage.tt which is just to keep things clean,
otherwise you can leave your changes in biomart.tt as well.
hope this sets you in the right direction :)
cheers
syed
(2) How do I get the data which would have been displayed, in
order to
filter it, etc
All I really want is to write a function which will be called by the
BioMart code, the input arguments to the function give me the data
which would have been displayed or exported, and the output is my
modified data (maybe with less rows, because I have filtered some
out;
maybe with extra columns which I have added; ...), for BioMart to
display or export as usual.
Since I don't want to modify the presentation of the results, but
modify the the results data, I thought that this would most likely
be
somewhere in the perl code which gets the data from the database,
rather than in the HTML.
Regards,
Roger.
Syed Haider wrote:
Yes, you can use this file and embedd it into biomart.tt just
like all
other templates are calling/embedding sub-units (templates).
Of course you are free to modify biomart.tt and leave this out,
however
just as you mentioned 'maintenance', I thought would be better if
you
can simply add a line to biomart.tt and copy across the relevant
code
over to resultspage.tt and maintain your own resultsPanel.
cheers
syed
On Tue, 2008-02-26 at 13:46 +0000, Roger Hull wrote:
Hi Syed,
Are you sure about resultspage.tt? - this file doesn't seem to
be used.
Regards,
Roger
Syed Haider wrote:
Hi Roger,
to manipulate the results for say sorting, filtering etc etc,
have a
loot at
biomart-perl/conf/templates/default/resultspage.tt
you can modify this panel according to your needs.
cheers
syed
On Mon, 2008-02-25 at 17:41 +0000, Roger Hull wrote:
Hi,
The URL API provides a good solution for canned queries which
can be
implemented as standard BioMart queries. But I also need to
implement
another type of canned query, which takes data from a BioMart
and
post-processes it in various ways. I can do this using the
perl API,
for example, and display the results in a new web page.
However, what I would like to do is to make use of the existing
MartView features for displaying and downloading the data, so
that the
user has a more uniform experience. Also, for some types of
canned
query, I would like the user to be able to change the
filtering and/or
attribute selections and press "Results" to get a new set of
post-processed results, displayed in the MartView page.
Is there some function that I could hook into which delivered
the
results internally within MartView? Of course I want to
minimise the
code changes, for future maintainability, but if there is one
place
(or a small number of places) where I could insert a function
call
which would post-process the data, this should be manageable.
I also need to do the same for the Count function. Currently
the Count
returned is "how many rows in the main table of the dataset
match your
filters so far". In the case of PRIDE, this is the number of
experiments. But often I need to know the number of rows
returned from
the query, not the number of rows from the main table - for
example I
want to know the number of distinct proteins or peptides which
match
my filters.
Regards,
Roger.
