Re: [mart-dev] Canned queries 2 - How to modify results set before display

[Roger Hull]

Hi Syed, Thanks for the advice. I think we see quite big risks in doing changes which depend on our understanding too much of how your code works. This could present problems in maintaining the code we have written, or changed, in the future, when BioMart is upgraded or we need to add new features. I would like to ask some questions about BioMart upgrades: (A) Will you maintain backward compatibility between a BioMart 0.6 installation which gets its data from a remote BioMart MartService, when the remote BioMart is upgraded to 0.7,..., 1.0, etc ? (B) Do you have a date for BioMart 0.7, and have you published a there a list of enhancements and bug fixes for that release? (C) There seem to be a number of BioMart installations where people have modified your code. As I said, I'm reluctant to follow this path, but I wonder if you could consider adding functions, hooks, or similar mechanisms in your code so that the BioMart behaviour can be changed in various ways without modifying your code? (The files header.tt and footer.tt are useful, but it is limited what can be done by adding code to these files.)  A couple of suggestions, based on what I have been wanting to do:     (C1) I would like to call your AJAX functions for my own purposes. But your function doAjaxMagic(toDo) only supports the two values of toDo = 'countByAjax' and 'resultsByAjax'. Could you provide a general purpose function which anyone could use? - with an argument to specify the URL which will handle the request and another to supply a function to handle the results (preferably supporting POST as well as GET). Then use this function internally to implement doAjaxMagic. If I modify your current function, or make a modified copy, then I have to maintain this code in the future if you change the function internally (e.g. to support new browser versions).     (C2) Could you implement, and document, a callback function in perl, which has as input arguments the results of a query in a parsable format (maybe XML), so that by default this function returns the input results unchanged. Then if the user wants to filter or otherwise modify the result data, this can be done by adding code to this function, and modifying the data before returning it to the caller. [OK, it's not quite as simple as this, because you batch the results data and return a certain number of result rows at a time, and if some are filtered out, some more have to be processed to make up the number, but I'm sure a solution can be found.] When I looked at your code to see how and where I might do this type of result modification, so that various formats (HTML, CSV, etc) and various output methods (display in MartView, download to a file), are all supported, it needed quite a lot of study of your code, and might well present difficulties to maintain. You have a perl API based upon BioMart::QueryRunner, which you use internally in your code, but the only way I can see to get at the results using this API is $query_runner->printResults(), which gives already formatted results. I want to get at the results before formatting, so I don't have to parse data formatted in various ways, and reformat the data after modification. Apart from filtering the results, another common requirement would be to add links to fields when the results are to be formatted as HTML, or to add another column of fields fetched from a local non-BioMart database, using one of the BioMart attributes in the results as an index to retrieve data from this database. (D) As I mentioned earlier, the Count returned by BioMart is "how many rows in the main table of the dataset match your filters so far". In the case of PRIDE, this is the number of experiments. But often I need to know the number of rows returned from the query, not the number of rows from the main table - for example I want to know the number of distinct proteins or peptides which match my filters. Will you support a more usual "number of rows returned" count in the future? Regards, Roger. Syed Haider wrote: Hi Roger, On Tue, 2008-02-26 at 14:15 +0000, Roger Hull wrote: Hi Syed, Ah, I misunderstood you. Yes, maintenance is important to me. But can you give me some advice on how to use this file? (1) Where in biomart.tt do I embed resultspage.tt? Which code do I copy across into resultspage.tt (copy rather than move?) biomart.tt contains HTML and [%%][**] code. The latter are the notations used to carry out simplistic programming in terms of perl CPAN package 'template toolkit' also referred to as TT2. HTML code is quite self-explanatory [% results_string %] is pasted initially by Web.pm but carries no significance. Later on, when user interacts with the MartView, there is a <td id="resultsTableId" .... which gets updated by _javascript_ in martview.js using AJAX, which you search in the said file using keyword AJAX. I would recommend first see how the biomart.tt works. When you change something in this file, no need to restart server or anything, just refresh browser and you will see the changes. However, as you know by now thats not the case if you change _javascript_ or Web.pm. Once you are confident what you want to do/change, then better tranfer things over to resultspage.tt which is just to keep things clean, otherwise you can leave your changes in biomart.tt as well. hope this sets you in the right direction :) cheers syed (2) How do I get the data which would have been displayed, in order to filter it, etc All I really want is to write a function which will be called by the BioMart code, the input arguments to the function give me the data which would have been displayed or exported, and the output is my modified data (maybe with less rows, because I have filtered some out; maybe with extra columns which I have added; ...), for BioMart to display or export as usual. Since I don't want to modify the presentation of the results, but modify the the results data, I thought that this would most likely be somewhere in the perl code which gets the data from the database, rather than in the HTML. Regards, Roger. Syed Haider wrote: Yes, you can use this file and embedd it into biomart.tt just like all other templates are calling/embedding sub-units (templates). Of course you are free to modify biomart.tt and leave this out, however just as you mentioned 'maintenance', I thought would be better if you can simply add a line to biomart.tt and copy across the relevant code over to resultspage.tt and maintain your own resultsPanel. cheers syed On Tue, 2008-02-26 at 13:46 +0000, Roger Hull wrote: Hi Syed, Are you sure about resultspage.tt? - this file doesn't seem to be used. Regards, Roger Syed Haider wrote: Hi Roger, to manipulate the results for say sorting, filtering etc etc, have a loot at biomart-perl/conf/templates/default/resultspage.tt you can modify this panel according to your needs. cheers syed On Mon, 2008-02-25 at 17:41 +0000, Roger Hull wrote: Hi, The URL API provides a good solution for canned queries which can be implemented as standard BioMart queries. But I also need to implement another type of canned query, which takes data from a BioMart and post-processes it in various ways. I can do this using the perl API, for example, and display the results in a new web page. However, what I would like to do is to make use of the existing MartView features for displaying and downloading the data, so that the user has a more uniform experience. Also, for some types of canned query, I would like the user to be able to change the filtering and/or attribute selections and press "Results" to get a new set of post-processed results, displayed in the MartView page. Is there some function that I could hook into which delivered the results internally within MartView? Of course I want to minimise the code changes, for future maintainability, but if there is one place (or a small number of places) where I could insert a function call which would post-process the data, this should be manageable. I also need to do the same for the Count function. Currently the Count returned is "how many rows in the main table of the dataset match your filters so far". In the case of PRIDE, this is the number of experiments. But often I need to know the number of rows returned from the query, not the number of rows from the main table - for example I want to know the number of distinct proteins or peptides which match my filters. Regards, Roger.