Feasibility of using OODT file manager in Airavata
Hi all, I have been working on integrating a file staging server for Apache Airavata as a part of my GSoC project. Main purpose of doing this is to manage and store input/ output files for Airavata experiments. Main use case is as follows. 1. Clients (users) can push input files and metadata to file server (OODT file manager) and should receive the url/ identifier of uploaded file to provide to Airavata 2. Users submit experiments to Airavata with the url of uploaded input file 3. Airavata should be able to fetch the input file over SCP or some other mechanism. If the file server exists in same machine where Airavata also has been hosted, Airavata can directly fetch the file from the repository folder. 4. Once the experiment (job) is done, Airavata should be able to push output files to file server and notify the client. 5. Client should be able to download the output file. In addition to that, enforcing security in uploading and downloading files is required. I tried OODT file manager by following this [1] tutorial and managed to push and download files using command line tools. Before going in to integration, I need to know some details about the feasibility of using OODT file manager for this scenario. 1. Can OODT file manager simulate the role of file server that I have mentioned earlier without using other modules of OODT project? 2. Is there any licensing issue if I use OODT file manager in Airavata? 3. What is the preferred way of using file manager client to talk to the file manager programatically? Simply, is there a Java API for file manager client rather than cli commands? [1] https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide Thank You Dimuthu -- Regards W.Dimuthu Upeksha Undergraduate Department of Computer Science And Engineering University of Moratuwa, Sri Lanka
Re: Problems Using Serializable Metadata after Loading Validation Layer
All, I found the issue. Using System.setProperties() and filling it from properties read from filemanager.properties clears out other properties setup by the system which was needed in the XML calls for SerializableMetadata. I did the above call to setup properties needed by the XMLValidationLayer To fix set only the properties you need individually. This adds to the System properties, not erasing them. System.setProperty(org.apache.oodt.cas.filemgr.repositorymgr.dirs, ...); System.setProperty(org.apache.oodt.cas.filemgr.validation.dirs, ...); -Michael On Thu, Jul 30, 2015 at 4:20 PM, Michael Starch starc...@umich.edu wrote: Here is the stack trace, but this only happens after a completely unrelated peice of the process load the XML Validation Layer. -Michael java.lang.NullPointerException at com.sun.org.apache.xml.internal.serializer.ToStream.init(ToStream.java:143) at com.sun.org.apache.xml.internal.serializer.ToXMLStream.init(ToXMLStream.java:67) at com.sun.org.apache.xml.internal.serializer.ToUnknownStream.init(ToUnknownStream.java:143) at com.sun.org.apache.xalan.internal.xsltc.runtime.output.TransletOutputHandlerFactory.getSerializationHandler(TransletOutputHandlerFactory.java:160) at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getOutputHandler(TransformerImpl.java:461) at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:344) at org.apache.oodt.cas.metadata.SerializableMetadata.writeMetadataToXmlStream(SerializableMetadata.java:157) On Thu, Jul 30, 2015 at 4:16 PM, Chris Mattmann chris.mattm...@gmail.com wrote: Mike can you give some specific line numbers? I can help look — Chris Mattmann chris.mattm...@gmail.com -Original Message- From: mdsta...@gmail.com on behalf of Michael Starch starc...@umich.edu Reply-To: dev@oodt.apache.org Date: Thursday, July 30, 2015 at 4:03 PM To: dev@oodt.apache.org Subject: Problems Using Serializable Metadata after Loading Validation Layer All, I am getting a NullPointerException deep in the XML library if I try to use the SerializableMetadata's write to xml function after I load in the XML Validation Layer from the filemanager. However, if I remove the call to load in the XML Validation Layer, everything works fine. Any ideas as to what might cause this issue? Thanks, Michael
Re: Osgi stuff
Friday is for sharing(I think) or its certainly for drinking. Before I hit beer o'clock I thought I'd share this little clip, as I'm hoping we can merge some of this stuff back upstream to offer an alternative to OPSUI. https://slack-files.com/T02531WE8-F08EFGLH1-85c44e77d8 (Hopefully you can see that video) Its very bare bones at the moment, but I have a central karaf server that can accept remote services via DOSGI(currently SOAP CXF, but could also be Hazelcast or another transport mechanism), anyway the window on the right is a websocket connection that mimics what would eventually be a webpage offering live stats about your OODT cluster. The cool thing is the window in the middle is a Karaf wrapped filemanager and it publishes its service via the DOSGI interface and could be located anywhere in the world, but you still get access to its full API (I know there is some rest stuff in the filemanager, but this can be replicated for all the other components without me having to change their code initially). You can see how it registers a new filemanager when I start it up and pushes it to my websocket. So, very rough, but provides some cool insight into some of the stuff that we can do with OSGI/Karaf containers. Happy friday. Tom On Fri, Jul 24, 2015 at 8:02 AM, Tom Barber tom.bar...@meteorite.bi wrote: Thanks Paul I certainly think the natural modularity of oodt components lends itself nicely to osgi integration and usage. Tom On 23 Jul 2015 21:01, Ramirez, Paul M (398M) paul.m.rami...@jpl.nasa.gov wrote: +1 sounds great to me. --Paul Paul Ramirez, M.S. Technical Group Supervisor Computer Science for Data Intensive Applications (398M) Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 158-264, Mailstop: 158-242 Email: paul.m.rami...@jpl.nasa.govmailto:paul.m.rami...@jpl.nasa.gov Office: 818-354-1015 Cell: 818-395-8194 On Jul 23, 2015, at 11:23 AM, Chris Mattmann mattm...@apache.orgmailto: mattm...@apache.org wrote: Totally agree! JIRA issue time and if no one gets to it, maybe GSoC or student project? — Chris Mattmann chris.mattm...@gmail.commailto:chris.mattm...@gmail.com -Original Message- From: Tom Barber tom.bar...@meteorite.bi Reply-To: dev@oodt.apache.org Date: Thursday, July 23, 2015 at 11:21 AM To: dev@oodt.apache.org Subject: Osgi stuff Taking that off of there and onto here about your mega bundle stuff. One thing I would like to do with this little project over time is create a karaf feature so people could run feature:install oodt.xyz and karaf spin up working OODT components radix style. I think that would be very useful. On 23 Jul 2015 19:08, chrismattmann g...@git.apache.org wrote: Github user chrismattmann commented on the pull request: https://github.com/apache/oodt/pull/23#issuecomment-124189993 gotcha. That's fine, well then I wouldn't change the default packaging of the components from jar to bundle then. As you are doing in a profile anyways with the plugin and dependencies, make the packaging also something that is dependent on the profile. I am +1 as long as it doesn't change the default behavior of jar packaging on these. @buggtb --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
File Manager - Error in ingeting a file to server
Hi all, I followed the tutorial [1] about OODT file server and managed to configure and start on the port 9000. Then I sent following request to ingest a file ./filemgr-client --url http://localhost:9000 --operation --ingestProduct --productName blah.txt --productStructure Flat --productTypeName GenericFile --metadataFile file:///tmp/blah.txt.met --refs file:///tmp/blah.txt As response I get following error. What could be the reason for this error? Jul 30, 2015 11:36:36 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient init INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties] Jul 30, 2015 11:36:37 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors INFO: Running Met Extractor: [org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor] for product type: [GenericFile] Jul 30, 2015 11:36:37 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors INFO: Running Met Extractor: [org.apache.oodt.cas.filemgr.metadata.extractors.examples.MimeTypeExtractor] for product type: [GenericFile] Jul 30, 2015 11:36:37 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors INFO: Running Met Extractor: [org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocationExtractor] for product type: [GenericFile] java.lang.IllegalArgumentException: URI is not absolute at java.io.File.init(File.java:416) at org.apache.oodt.cas.filemgr.versioning.VersioningUtils.createBasicDataStoreRefsFlat(VersioningUtils.java:219) at org.apache.oodt.cas.filemgr.versioning.BasicVersioner.createDataStoreReferences(BasicVersioner.java:123) at org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocationExtractor.doExtract(FinalFileLocationExtractor.java:80) at org.apache.oodt.cas.filemgr.metadata.extractors.AbstractFilemgrMetExtractor.extractMetadata(AbstractFilemgrMetExtractor.java:57) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.runExtractors(XmlRpcFileManager.java:) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.addMetadata(XmlRpcFileManager.java:1065) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:722) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) at java.lang.Thread.run(Thread.java:745) org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@77b774dd] : URI is not absolute at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104) at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71) at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198) at org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:113) at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) Jul 30, 2015 11:36:37 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient ingestProduct SEVERE: Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@a2431d0] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@77b774dd] : URI is not absolute -- rolling back ingest Jul 30, 2015 11:36:37 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog removeProductDocument FINE: LuceneCatalog: remove document from index for product: [null] java.lang.Exception: Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@a2431d0] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product
