[base] JEP extra value calculator again...
Hi all. Sorry to bother you again with this point, but I still does not understand everything... I tried to define a new type, called float colored: it is a float, colored with min value as 0, max as 0, mid as 5 I want to add 2 values to my data: -2 * (ch(1) / ch(2)) -3 * (ch(1) / ch(2)) (Probably stupid for the analysis, but I just want to check the functionality) So, I tried the JEP extra value calculator, specifying float coloured as a data type, and 2 * (ch(1) / ch(2)) as the expression. Then, I give a look to the data, and it works: the numbers are preceded with a little coloured square, the colour depends on the value. Great, indeed. So, I tried a second run, this time with 3 * (ch(1) / ch(2)) as expression. It got the following error message: The bioassay set already has extra values of the type: Extra value type[id=2824; name=float colored] This sounds very strange to me: If float colored is an Extra Value Type, (this sounds to me as some data type, it's not right ?) why should it be applied ONLY one time ? Something else, which is probably a bug. When I go to Administrate/Types/Extra value types, I click on any extra type declared (for instance for editing the type), I get the following: Base version2.1 (build #2847; schema #20) Web server Apache Tomcat/5.5.20 Database Dialectorg.hibernate.dialect.PostgreSQLDialect JDBC Driver org.postgresql.Driver (version 8.1) Java runtimeJava(TM) 2 Runtime Environment, Standard Edition (1.5.0_06-b05), Sun Microsystems Inc. http://java.sun.com/ Operating systemLinux i386 2.6.16-2-em64t-p4-smp Memory Total: 155.3 MB Free: 30.9 MB Max: 986.1 MB Browser Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.8.1) Gecko/20061010 Firefox/2.0 Error message /usr/local/apache-tomcat-5.5.20/work/Catalina/localhost/base2/org/apache/jsp/admin/extravaluetypes/view_005fextravaluetype_jsp.java (Permission denied) Stack trace ...at java.io.FileOutputStream.open(Native Method) ...at java.io.FileOutputStream.(Unknown Source) ...at java.io.FileOutputStream.(Unknown Source) ...at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:140) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:295) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:276) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:264) ...at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:563) ...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:305) ...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) ...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) ...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) ...at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672) ...at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463) ...at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398) ...at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301) ...at org.apache.jasper.runtime.PageContextImpl.doForward(PageContextImpl.java:688) ...at org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:658) ...at org.apache.jsp.admin.extravaluetypes.index_jsp._jspService(index_jsp.java:278) ...at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334) ...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) ...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) ...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) ...at net.sf.basedb.clients.web.servlet.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:71) ...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) ...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) ...at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) ...at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) ...at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) ...at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) ...at
Re: [base] JEP extra value calculator again...
Emmanuel Courcelle wrote: Hi all. Sorry to bother you again with this point, but I still does not understand everything... I tried to define a new type, called float colored: it is a float, colored with min value as 0, max as 0, mid as 5 I want to add 2 values to my data: -2 * (ch(1) / ch(2)) -3 * (ch(1) / ch(2)) (Probably stupid for the analysis, but I just want to check the functionality) So, I tried the JEP extra value calculator, specifying float coloured as a data type, and 2 * (ch(1) / ch(2)) as the expression. Then, I give a look to the data, and it works: the numbers are preceded with a little coloured square, the colour depends on the value. Great, indeed. So, I tried a second run, this time with 3 * (ch(1) / ch(2)) as expression. It got the following error message: The bioassay set already has extra values of the type: Extra value type[id=2824; name=float colored] You can only calculate one extra value of each type per bioassayset. This sounds very strange to me: If float colored is an Extra Value Type, (this sounds to me as some data type, it's not right ?) why should it be applied ONLY one time ? Because the extra value type itself can be used in further calculations, filters, by plugins, etc. You then refer to it by the External ID of the extra value type. If there are more than one value for each type, it is not possible to know which one to choose. Something else, which is probably a bug. When I go to Administrate/Types/Extra value types, I click on any extra type declared (for instance for editing the type), I get the following: It looks like some internal Tomcat error. Try removing everything from the /usr/local/apache-tomcat-5.5.20/work/Catalina/localhost/base2/ directory. /Nicklas Base version 2.1 (build #2847; schema #20) Web serverApache Tomcat/5.5.20 Database Dialect org.hibernate.dialect.PostgreSQLDialect JDBC Driver org.postgresql.Driver (version 8.1) Java runtime Java(TM) 2 Runtime Environment, Standard Edition (1.5.0_06-b05), Sun Microsystems Inc. http://java.sun.com/ Operating system Linux i386 2.6.16-2-em64t-p4-smp MemoryTotal: 155.3 MB Free: 30.9 MB Max: 986.1 MB Browser Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.8.1) Gecko/20061010 Firefox/2.0 Error message /usr/local/apache-tomcat-5.5.20/work/Catalina/localhost/base2/org/apache/jsp/admin/extravaluetypes/view_005fextravaluetype_jsp.java (Permission denied) Stack trace ...at java.io.FileOutputStream.open(Native Method) ...at java.io.FileOutputStream.(Unknown Source) ...at java.io.FileOutputStream.(Unknown Source) ...at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:140) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:295) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:276) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:264) ...at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:563) ...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:305) ...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) ...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) ...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) ...at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672) ...at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463) ...at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398) ...at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301) ...at org.apache.jasper.runtime.PageContextImpl.doForward(PageContextImpl.java:688) ...at org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:658) ...at org.apache.jsp.admin.extravaluetypes.index_jsp._jspService(index_jsp.java:278) ...at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334) ...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) ...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) ...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) ...at
Re: [base] Offline files
That sounds like a hell of a lot more work than what I have done so far :-) My first shot at this has been quite basic and it only creates new RawBioAssays and a single Experiment that these belong to. It could of course be extended to do more stuff, but my time allowance for this is limited and I don't think I'll be able to spend a lot more time on this than I already have. I think the upshot of all this discussion has been: 1. people need a batch importer (desperately, in fact), and 2. a plugin is much better and will (hopefully) be possible With that in mind, I'll go away and look into turning my stuff into a plugin now, and when I have got the basic version of that working (i.e. file upload, RawBioAssay and Experiment creation only, as above) I'll make the code available to the community somehow and someone else can pick it up from there if they want to make it fancier. I hope that's okay -- even with no extra effort from someone else it'll still mean that a hell of a lot of effort can be saved. Perhaps the dev team can point me in the right direction initially. My colleague Iain here is just working on a file importer plugin that handles Excel files and his code starts off like this: public class ExcelFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.RAWBIOASSAY, GuiContext.Type.ITEM)); So am I right in thinking that if I change this to the following... public class ZipFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.EXPERIMENT, GuiContext.Type.ITEM)); then I'll get an import tab created in the Experiment list view which will let me import the zip file using my plugin? We're planning to make the Excel plugin available for everyone too BTW, once it's finished. Cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = -Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Bob MacCallum Sent: 29 November 2006 18:36 To: BASE ML Subject: Re: [base] Offline files It looks like I let the cat out of the bag on this one... Some discussion among the community would be good to establish what we want from a bulk loader. For example, I would like all objects created and linked, from samples (with annotations) right up to raw bioassays (grouped into an experiment as Micha does, ideally). Obviously to do this you need to specify the relationships between all the samples, annotations, dyes, hybs, data files, but that could easily end up being as much work as manually loading the data ;-) Therefore you have to make compromises; the main one being that you treat every sample/extract/etc in exactly the same way (same protocols, same amounts, same labels, same array design). This would probably work for most people - but let's discuss it anyway... One issue with the bulk loader I wrote for BASE 1 is that if your experiment involves a dye swap for one biological replicate, the bulk loader will label each extract with both dyes for *all* replicates (leaving you with unused labelled extracts after linking to the hybs). One quite low cost way to define your experiment for a bulk loader is Tab2MAGE (soon to be MAGE-TAB, see http://www.biomedcentral.com/1471-2105/7/489/abstract ) - although it feels strange to say this because I have been *exporting* Tab2MAGE from BASE 1.2 rather than importing it. The input files for our bulk loader are specified here: http://wiki.vectorbase.org/vectorbase/index.php/MicroarrayExperiments (look for the example file links) but note that the BASE-specific parameters (protocols, raw file parsers, array designs, etc) are configured by the user in an interactive process before the upload. Our bulk loader is also 2-channel only... I have a feeling that the Tab2MAGE parser could be adapted into a BASE 2 bulk loader somehow, although I think all Tab2MAGE code is Perl. I think MAGE-TAB is something we could get the average bench scientist to provide (with a little help MGEDifying the annotations). So far every experiment I have loaded has come with some kind of spreadsheet (each one in a different format of course), so the leap to MAGE-TAB is not so great. Ok, enough from me... Micha Bayer writes: -Original Message- From: [EMAIL PROTECTED] [mailto:basedb- users- [EMAIL PROTECTED] On Behalf Of Nicklas Nordborg Sent: 29 November
Re: [base] JEP extra value calculator again...
Hi Emmanuel, The Catalina/work directory caches copies of your compiled JSP pages. It looks from the stacktrace like the JSP compiler can't write to the work directory because you don't have permissions to it -- check what they are set to. Cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = -Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Emmanuel Courcelle Sent: 30 November 2006 09:23 To: BASE ML Subject: [base] JEP extra value calculator again... Hi all. Sorry to bother you again with this point, but I still does not understand everything... I tried to define a new type, called float colored: it is a float, colored with min value as 0, max as 0, mid as 5 I want to add 2 values to my data: -2 * (ch(1) / ch(2)) -3 * (ch(1) / ch(2)) (Probably stupid for the analysis, but I just want to check the functionality) So, I tried the JEP extra value calculator, specifying float coloured as a data type, and 2 * (ch(1) / ch(2)) as the expression. Then, I give a look to the data, and it works: the numbers are preceded with a little coloured square, the colour depends on the value. Great, indeed. So, I tried a second run, this time with 3 * (ch(1) / ch(2)) as expression. It got the following error message: The bioassay set already has extra values of the type: Extra value type[id=2824; name=float colored] This sounds very strange to me: If float colored is an Extra Value Type, (this sounds to me as some data type, it's not right ?) why should it be applied ONLY one time ? Something else, which is probably a bug. When I go to Administrate/Types/Extra value types, I click on any extra type declared (for instance for editing the type), I get the following: Base version 2.1 (build #2847; schema #20) Web serverApache Tomcat/5.5.20 Database Dialect org.hibernate.dialect.PostgreSQLDialect JDBC Driver org.postgresql.Driver (version 8.1) Java runtime Java(TM) 2 Runtime Environment, Standard Edition (1.5.0_06-b05), Sun Microsystems Inc. http://java.sun.com/ Operating system Linux i386 2.6.16-2-em64t-p4-smp MemoryTotal: 155.3 MB Free: 30.9 MB Max: 986.1 MB Browser Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.8.1) Gecko/20061010 Firefox/2.0 Error message /usr/local/apache-tomcat- 5.5.20/work/Catalina/localhost/base2/org/apache/jsp/admin/extravaluetype s/ view_005fextravaluetype_jsp.java (Permission denied) Stack trace ...at java.io.FileOutputStream.open(Native Method) ...at java.io.FileOutputStream.(Unknown Source) ...at java.io.FileOutputStream.(Unknown Source) ...at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:140) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:295) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:276) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:264) ...at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.ja va :563) ...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.ja va :305) ...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) ...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica ti onFilterChain.java:252) ...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt er Chain.java:173) ...at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc he r.java:672) ...at org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicatio nD ispatcher.java:463) ...at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDisp at cher.java:398) ...at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispat ch er.java:301) ...at org.apache.jasper.runtime.PageContextImpl.doForward(PageContextImpl.java :6 88) ...at org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:6 58 ) ...at org.apache.jsp.admin.extravaluetypes.index_jsp._jspService(index_jsp.jav a: 278) ...at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.ja va :334) ...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) ...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at
Re: [base] Offline files
Hi, public class ZipFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.EXPERIMENT, GuiContext.Type.ITEM)); then I'll get an import tab created in the Experiment list view which will let me import the zip file using my plugin? That code is correct but to see an import tab you will have to specify that the plugin is an import plugin .For instance: public MainType getMainType() { return Plugin.MainType.IMPORT; } Please, see the plugin examples in base2 source code for more information on this. We're planning to make the Excel plugin available for everyone too BTW, once it's finished. Cool, this will be good for the community. Dominic -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Micha Bayer Sent: 30 November 2006 09:52 To: BASE ML Subject: Re: [base] Offline files That sounds like a hell of a lot more work than what I have done so far :-) My first shot at this has been quite basic and it only creates new RawBioAssays and a single Experiment that these belong to. It could of course be extended to do more stuff, but my time allowance for this is limited and I don't think I'll be able to spend a lot more time on this than I already have. I think the upshot of all this discussion has been: 1. people need a batch importer (desperately, in fact), and 2. a plugin is much better and will (hopefully) be possible With that in mind, I'll go away and look into turning my stuff into a plugin now, and when I have got the basic version of that working (i.e. file upload, RawBioAssay and Experiment creation only, as above) I'll make the code available to the community somehow and someone else can pick it up from there if they want to make it fancier. I hope that's okay -- even with no extra effort from someone else it'll still mean that a hell of a lot of effort can be saved. Perhaps the dev team can point me in the right direction initially. My colleague Iain here is just working on a file importer plugin that handles Excel files and his code starts off like this: public class ExcelFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.RAWBIOASSAY, GuiContext.Type.ITEM)); So am I right in thinking that if I change this to the following... public class ZipFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.EXPERIMENT, GuiContext.Type.ITEM)); then I'll get an import tab created in the Experiment list view which will let me import the zip file using my plugin? We're planning to make the Excel plugin available for everyone too BTW, once it's finished. Cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = -Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Bob MacCallum Sent: 29 November 2006 18:36 To: BASE ML Subject: Re: [base] Offline files It looks like I let the cat out of the bag on this one... Some discussion among the community would be good to establish what we want from a bulk loader. For example, I would like all objects created and linked, from samples (with annotations) right up to raw bioassays (grouped into an experiment as Micha does, ideally). Obviously to do this you need to specify the relationships between all the samples, annotations, dyes, hybs, data files, but that could easily end up being as much work as manually loading the data ;-) Therefore you have to make compromises; the main one being that you treat every sample/extract/etc in exactly the same way (same protocols, same amounts, same labels, same array design). This would probably work for most people - but let's discuss it anyway... One issue with the bulk loader I wrote for BASE 1 is that if your experiment involves a dye swap for one biological replicate, the bulk loader will label each extract with both dyes for *all* replicates (leaving you with unused labelled extracts after linking to the hybs). One quite low cost way to define your experiment for a bulk loader is Tab2MAGE (soon to be MAGE-TAB, see http://www.biomedcentral.com/1471-2105/7/489/abstract ) - although it feels strange to say this because I have been *exporting* Tab2MAGE from BASE 1.2 rather than importing it. The input files for our bulk
Re: [base] Offline files
Micha Bayer wrote: So am I right in thinking that if I change this to the following... public class ZipFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.EXPERIMENT, GuiContext.Type.ITEM)); then I'll get an import tab created in the Experiment list view which will let me import the zip file using my plugin? If you want the plugin to appear on the list page you should use, GuiContext.Type.List. The GuiContext.Type.ITEM makes it appear in the view page for a single experiment. /Nicklas - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
Hi Micha and others, Of course we have to share the workload. I'd rather help develop something that is already taking shape (because I'm still not very familiar with the API). Let's see what people think about MAGE-TAB etc, but hopefully I can carry on where you leave off. That's not a 100% commitment, but I hope I'll have the time/ability to do this. Just one thought, isn't another option to cannibalise the migration code to make a command-line bulk importer? I guess that's no good for sites who want the users to submit through the web interface... Another thought: my BASE 1 loaders were bolted on top of BASE and did all the sample creation etc through HTTP requests to the BASE 1 GUI with a Perl LWP robot. These could be modified and bolted on to BASE 2, although the approach is very sensitive to GUI changes, and just doesn't feel like a long-term solution to me... (the motivation to do it this way with BASE 1 was to take the path of least resistance and avoid learning the PHP API). cheers, Bob. Micha Bayer writes: That sounds like a hell of a lot more work than what I have done so far :-) My first shot at this has been quite basic and it only creates new RawBioAssays and a single Experiment that these belong to. It could of course be extended to do more stuff, but my time allowance for this is limited and I don't think I'll be able to spend a lot more time on this than I already have. I think the upshot of all this discussion has been: 1. people need a batch importer (desperately, in fact), and 2. a plugin is much better and will (hopefully) be possible With that in mind, I'll go away and look into turning my stuff into a plugin now, and when I have got the basic version of that working (i.e. file upload, RawBioAssay and Experiment creation only, as above) I'll make the code available to the community somehow and someone else can pick it up from there if they want to make it fancier. I hope that's okay -- even with no extra effort from someone else it'll still mean that a hell of a lot of effort can be saved. Perhaps the dev team can point me in the right direction initially. My colleague Iain here is just working on a file importer plugin that handles Excel files and his code starts off like this: public class ExcelFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.RAWBIOASSAY, GuiContext.Type.ITEM)); So am I right in thinking that if I change this to the following... public class ZipFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.EXPERIMENT, GuiContext.Type.ITEM)); then I'll get an import tab created in the Experiment list view which will let me import the zip file using my plugin? We're planning to make the Excel plugin available for everyone too BTW, once it's finished. Cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = -Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Bob MacCallum Sent: 29 November 2006 18:36 To: BASE ML Subject: Re: [base] Offline files It looks like I let the cat out of the bag on this one... Some discussion among the community would be good to establish what we want from a bulk loader. For example, I would like all objects created and linked, from samples (with annotations) right up to raw bioassays (grouped into an experiment as Micha does, ideally). Obviously to do this you need to specify the relationships between all the samples, annotations, dyes, hybs, data files, but that could easily end up being as much work as manually loading the data ;-) Therefore you have to make compromises; the main one being that you treat every sample/extract/etc in exactly the same way (same protocols, same amounts, same labels, same array design). This would probably work for most people - but let's discuss it anyway... One issue with the bulk loader I wrote for BASE 1 is that if your experiment involves a dye swap for one biological replicate, the bulk loader will label each extract with both dyes for *all* replicates (leaving you with unused labelled extracts after linking to the hybs). One quite low cost way to define your experiment for a bulk loader is Tab2MAGE
Re: [base] JEP extra value calculator again...
Micha Bayer a écrit : Hi Emmanuel, The Catalina/work directory caches copies of your compiled JSP pages. It looks from the stacktrace like the JSP compiler can't write to the work directory because you don't have permissions to it -- check what they are set to. Right, Micha, some files and directories did in fact belong to root The pb is fixed now Thanks for your help ! -- Emmanuel COURCELLE[EMAIL PROTECTED] L.I.P.M. (UMR CNRS-INRA 2594/441) tel (33) 5-61-28-54-50 B.P.52627 - 31326 CASTANET TOLOSAN Cedex - FRANCE -- - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
There is a possibility to get access to the subversion repository and trac environment just for this kind of stuff. Check out http://baseplugins.thep.lu.se Trac environment allows for discussions about the importer through the ticketing system. Jari Micha Bayer wrote: That sounds like a hell of a lot more work than what I have done so far :-) My first shot at this has been quite basic and it only creates new RawBioAssays and a single Experiment that these belong to. It could of course be extended to do more stuff, but my time allowance for this is limited and I don't think I'll be able to spend a lot more time on this than I already have. I think the upshot of all this discussion has been: 1. people need a batch importer (desperately, in fact), and 2. a plugin is much better and will (hopefully) be possible With that in mind, I'll go away and look into turning my stuff into a plugin now, and when I have got the basic version of that working (i.e. file upload, RawBioAssay and Experiment creation only, as above) I'll make the code available to the community somehow and someone else can pick it up from there if they want to make it fancier. I hope that's okay -- even with no extra effort from someone else it'll still mean that a hell of a lot of effort can be saved. Perhaps the dev team can point me in the right direction initially. My colleague Iain here is just working on a file importer plugin that handles Excel files and his code starts off like this: public class ExcelFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.RAWBIOASSAY, GuiContext.Type.ITEM)); So am I right in thinking that if I change this to the following... public class ZipFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.EXPERIMENT, GuiContext.Type.ITEM)); then I'll get an import tab created in the Experiment list view which will let me import the zip file using my plugin? We're planning to make the Excel plugin available for everyone too BTW, once it's finished. Cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = -Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Bob MacCallum Sent: 29 November 2006 18:36 To: BASE ML Subject: Re: [base] Offline files It looks like I let the cat out of the bag on this one... Some discussion among the community would be good to establish what we want from a bulk loader. For example, I would like all objects created and linked, from samples (with annotations) right up to raw bioassays (grouped into an experiment as Micha does, ideally). Obviously to do this you need to specify the relationships between all the samples, annotations, dyes, hybs, data files, but that could easily end up being as much work as manually loading the data ;-) Therefore you have to make compromises; the main one being that you treat every sample/extract/etc in exactly the same way (same protocols, same amounts, same labels, same array design). This would probably work for most people - but let's discuss it anyway... One issue with the bulk loader I wrote for BASE 1 is that if your experiment involves a dye swap for one biological replicate, the bulk loader will label each extract with both dyes for *all* replicates (leaving you with unused labelled extracts after linking to the hybs). One quite low cost way to define your experiment for a bulk loader is Tab2MAGE (soon to be MAGE-TAB, see http://www.biomedcentral.com/1471-2105/7/489/abstract ) - although it feels strange to say this because I have been *exporting* Tab2MAGE from BASE 1.2 rather than importing it. The input files for our bulk loader are specified here: http://wiki.vectorbase.org/vectorbase/index.php/MicroarrayExperiments (look for the example file links) but note that the BASE-specific parameters (protocols, raw file parsers, array designs, etc) are configured by the user in an interactive process before the upload. Our bulk loader is also 2-channel only... I have a feeling that the Tab2MAGE parser could be adapted into a BASE 2 bulk loader somehow, although I think all Tab2MAGE code is Perl. I think MAGE-TAB is something we could get the average bench scientist to provide (with a little help MGEDifying the annotations). So far every experiment I have loaded has come with
Re: [base] Offline files
A command line bulk importer ... well, we are just starting to look at adding remote services to BASE. Web services or something similar might be useful in this context. Jari Bob MacCallum wrote: Hi Micha and others, Of course we have to share the workload. I'd rather help develop something that is already taking shape (because I'm still not very familiar with the API). Let's see what people think about MAGE-TAB etc, but hopefully I can carry on where you leave off. That's not a 100% commitment, but I hope I'll have the time/ability to do this. Just one thought, isn't another option to cannibalise the migration code to make a command-line bulk importer? I guess that's no good for sites who want the users to submit through the web interface... Another thought: my BASE 1 loaders were bolted on top of BASE and did all the sample creation etc through HTTP requests to the BASE 1 GUI with a Perl LWP robot. These could be modified and bolted on to BASE 2, although the approach is very sensitive to GUI changes, and just doesn't feel like a long-term solution to me... (the motivation to do it this way with BASE 1 was to take the path of least resistance and avoid learning the PHP API). cheers, Bob. Micha Bayer writes: That sounds like a hell of a lot more work than what I have done so far :-) My first shot at this has been quite basic and it only creates new RawBioAssays and a single Experiment that these belong to. It could of course be extended to do more stuff, but my time allowance for this is limited and I don't think I'll be able to spend a lot more time on this than I already have. I think the upshot of all this discussion has been: 1. people need a batch importer (desperately, in fact), and 2. a plugin is much better and will (hopefully) be possible With that in mind, I'll go away and look into turning my stuff into a plugin now, and when I have got the basic version of that working (i.e. file upload, RawBioAssay and Experiment creation only, as above) I'll make the code available to the community somehow and someone else can pick it up from there if they want to make it fancier. I hope that's okay -- even with no extra effort from someone else it'll still mean that a hell of a lot of effort can be saved. Perhaps the dev team can point me in the right direction initially. My colleague Iain here is just working on a file importer plugin that handles Excel files and his code starts off like this: public class ExcelFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.RAWBIOASSAY, GuiContext.Type.ITEM)); So am I right in thinking that if I change this to the following... public class ZipFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.EXPERIMENT, GuiContext.Type.ITEM)); then I'll get an import tab created in the Experiment list view which will let me import the zip file using my plugin? We're planning to make the Excel plugin available for everyone too BTW, once it's finished. Cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = -Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Bob MacCallum Sent: 29 November 2006 18:36 To: BASE ML Subject: Re: [base] Offline files It looks like I let the cat out of the bag on this one... Some discussion among the community would be good to establish what we want from a bulk loader. For example, I would like all objects created and linked, from samples (with annotations) right up to raw bioassays (grouped into an experiment as Micha does, ideally). Obviously to do this you need to specify the relationships between all the samples, annotations, dyes, hybs, data files, but that could easily end up being as much work as manually loading the data ;-) Therefore you have to make compromises; the main one being that you treat every sample/extract/etc in exactly the same way (same protocols, same amounts, same labels, same array design). This would probably work for most people - but let's discuss it anyway... One issue with the bulk loader I wrote for BASE 1 is that if your experiment involves a
Re: [base] Offline files
Philippe Rocca-Serra writes: Hi Everyone, As Dominic pointed out, we are working anyway on an import into BASE2 using tab2mage format in the short time (see ticket 338), and once MAGE-TAB specifications are stable move on to support this format too. excellent! There is a MAGE-TAB jamboree happening in Philadelphia next week, so I will count on Helen to update us on the outcome of this meeting so we can possibly expedite the process. will there be Java code to parse MAGE-TAB? I guess that would speed up the development of the BASE importer. The tab2mage import will prove no doubt extremely useful to exchange data and to get data from repositories as pointed out So everything is in the pipeline and we jsut need a little bit a patience, days have only 24 hours cheers Philippe -- Philippe Rocca-Serra (Ph.D) Nutri/Toxicogenomics Internal Coordinator EBI- The European Bioinformatics Institute EMBL Outstation Hinxton Wellcome Trust Genome Campus CB10 1SD Cambridge, UK email: [EMAIL PROTECTED] Tel: +44 (0) 1223 492 553 Fax: +44 (0) 1223 494 468 http://www.ebi.ac.uk/microarray/Projects/tox-nutri/index.html Helen Parkinson wrote: Hi, this is an interesting problem and one we have been dealing with at ArrayExpress as well. For really large experiments the only thing that that has worked for submissions is a spreadsheet - currently we use tab2mage for this purpose. *tab2mage*.sourceforge.net/ You might think about an importer that takes a spreadsheet, or partial spreadsheet plus files in an archive as input, e.g. MAGETAB, this would allow people to pull in other people's data downloaded from public repositories, and allow biologists to format up their own data easily. And as spreadsheet export from BASE is planned the import would also be nice, round tripping is also a good test of format and application cheers Helen Jari Häkkinen wrote: There is a possibility to get access to the subversion repository and trac environment just for this kind of stuff. Check out http://baseplugins.thep.lu.se Trac environment allows for discussions about the importer through the ticketing system. Jari Micha Bayer wrote: That sounds like a hell of a lot more work than what I have done so far :-) My first shot at this has been quite basic and it only creates new RawBioAssays and a single Experiment that these belong to. It could of course be extended to do more stuff, but my time allowance for this is limited and I don't think I'll be able to spend a lot more time on this than I already have. I think the upshot of all this discussion has been: 1. people need a batch importer (desperately, in fact), and 2. a plugin is much better and will (hopefully) be possible With that in mind, I'll go away and look into turning my stuff into a plugin now, and when I have got the basic version of that working (i.e. file upload, RawBioAssay and Experiment creation only, as above) I'll make the code available to the community somehow and someone else can pick it up from there if they want to make it fancier. I hope that's okay -- even with no extra effort from someone else it'll still mean that a hell of a lot of effort can be saved. Perhaps the dev team can point me in the right direction initially. My colleague Iain here is just working on a file importer plugin that handles Excel files and his code starts off like this: public class ExcelFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.RAWBIOASSAY, GuiContext.Type.ITEM)); So am I right in thinking that if I change this to the following... public class ZipFileImporter extends AbstractPlugin implements InteractivePlugin, AutoDetectingImporter { private static final SetGuiContext guiContexts = Collections.singleton(new GuiContext(Item.EXPERIMENT, GuiContext.Type.ITEM)); then I'll get an import tab created in the Experiment list view which will let me import the zip file using my plugin? We're planning to make the Excel plugin available for everyone too BTW, once it's finished. Cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm