Sorry about the delay, here’s my replies Dimuthu: > >Hi all, > >I have been working on integrating a file staging server for Apache >Airavata as a part of my GSoC project. Main purpose of doing this is to >manage and store input/ output files for Airavata experiments. > >Main use case is as follows. > >1. Clients (users) can push input files and metadata to file server (OODT >file manager) and should receive the url/ identifier of uploaded file to >provide to Airavata
The best way to do this is to (in addition to the file manager), deploy the FM prod web service, here [1]. It’s a simple WAR file. You can configure the WAR by making sure you have environment variables set for each of the named variables inside of ‘[‘ and ‘]’ in this file: http://svn.apache.org/repos/asf/oodt/trunk/webapp/fmprod/src/main/webapp/ME TA-INF/context.xml For example, FILEMGR_URL by default is http://localhost:9000. Make sure the env vars are set before starting Tomcat and/or Jetty for your web app server. The FM prod provides REST-ful perm links to products by their IDs, etc. The API is here: https://cwiki.apache.org/confluence/display/OODT/File+Manager+REST+API >2. Users submit experiments to Airavata with the url of uploaded input >file You can use the File Manager Ingest API [2] and/or Crawler to take care of this. >3. Airavata should be able to fetch the input file over SCP or some other >mechanism. If the file server exists in same machine where Airavata also >has been hosted, Airavata can directly fetch the file from the repository >folder. For this you’ll want to use PushPull [3]. Another OODT component. Docs are here: https://cwiki.apache.org/confluence/x/KraoAQ > 4. Once the experiment (job) is done, Airavata should be able to push >output files to file server and notify the client. >5. Client should be able to download the output file. Got it. > >In addition to that, enforcing security in uploading and downloading >files is required. > >I tried OODT file manager by following this [1] tutorial and managed to >push and download files using command line tools. Before going in to >integration, I need to know some details about the feasibility of using >OODT file manager for this scenario. > >1. Can OODT file manager simulate the role of file server that I have >mentioned earlier without using other modules of OODT project? Yep. >2. Is there any licensing issue if I use OODT file manager in Airavata? Nope. >3. What is the preferred way of using file manager client to talk to the >file manager programatically? Simply, is there a Java API for file >manager client rather than cli commands? > From your use case scenario above, I would just install the entire OODT stack. Since you are going to be running jobs, and pushing data and so forth, I would seriously have a look at OODT’s CAS-PGE job. It does all that. It stages a file/data/set of files automatically from the file manager and other places to a job working directory. Allows you to integrate and execute any algorithm, cmd line, API or otherwise, then provides an automatic mechanism to extract metadata (using Tika) and to Crawl and capture the results using the OODT crawler and file manager. RADIX will install the entire OODT stack and is a good start point for testing this stuff: https://cwiki.apache.org/confluence/x/zoOOAQ Please try RADIX out and keep the questions coming. Thanks! Cheers, Chris [1] http://search.maven.org/remotecontent?filepath=org/apache/oodt/cas-product/ 0.9/cas-product-0.9.war [2] http://s.apache.org/nYf [3] http://svn.apache.org/repos/asf/oodt/trunk/pushpull/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: DImuthu Upeksha <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Thursday, August 6, 2015 at 8:28 PM To: "[email protected]" <[email protected]> Cc: "[email protected]" <[email protected]> Subject: Re: Feasibility of using OODT file manager in Airavata >Hi Chris, > > >Any update on this? > > >On Tue, Aug 4, 2015 at 12:05 AM, Chris Mattmann ><[email protected]> wrote: > >I’ll provide a detailed answer on this today. Sorry been on baby >duties and enjoying every minute of it! :) > >— >Chris Mattmann >[email protected] > > > > > > >-----Original Message----- >From: "Pierce, Marlon" <[email protected]> >Reply-To: <[email protected]> >Date: Monday, August 3, 2015 at 8:14 AM >To: "[email protected]" <[email protected]>, >"[email protected]" <[email protected]> >Subject: Re: Feasibility of using OODT file manager in Airavata > >>Hi Dimuthu- >> >>In regards to question #2, there should not be any licensing issues. >> >>Marlon >> >> >>From: DImuthu Upeksha >><[email protected]<mailto:[email protected]>> >>Reply-To: "[email protected]<mailto:[email protected]>" >><[email protected]<mailto:[email protected]>> >>Date: Saturday, August 1, 2015 at 12:40 AM >>To: "[email protected]<mailto:[email protected]>" >><[email protected]<mailto:[email protected]>>, >>"[email protected]<mailto:[email protected]>" >><[email protected]<mailto:[email protected]>> >>Subject: Feasibility of using OODT file manager in Airavata >> >>Hi all, >> >>I have been working on integrating a file staging server for Apache >>Airavata as a part of my GSoC project. Main purpose of doing this is to >>manage and store input/ output files for Airavata experiments. >> >>Main use case is as follows. >> >>1. Clients (users) can push input files and metadata to file server (OODT >>file manager) and should receive the url/ identifier of uploaded file to >>provide to Airavata >>2. Users submit experiments to Airavata with the url of uploaded input >>file >>3. Airavata should be able to fetch the input file over SCP or some other >>mechanism. If the file server exists in same machine where Airavata also >>has been hosted, Airavata can directly fetch the file from the repository >>folder. >> 4. Once the experiment (job) is done, Airavata should be able to push >>output files to file server and notify the client. >>5. Client should be able to download the output file. >> >>In addition to that, enforcing security in uploading and downloading >>files is required. >> >>I tried OODT file manager by following this [1] tutorial and managed to >>push and download files using command line tools. Before going in to >>integration, I need to know some details about the feasibility of using >>OODT file manager for this scenario. >> >>1. Can OODT file manager simulate the role of file server that I have >>mentioned earlier without using other modules of OODT project? >>2. Is there any licensing issue if I use OODT file manager in Airavata? >>3. What is the preferred way of using file manager client to talk to the >>file manager programatically? Simply, is there a Java API for file >>manager client rather than cli commands? >> >>[1] >>https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide >> >>Thank You >>Dimuthu >>-- >>Regards >> >>W.Dimuthu Upeksha >>Undergraduate >> >>Department of Computer Science And Engineering >> >>University of Moratuwa, Sri Lanka > > > > > > > > > > >-- >Regards >W.Dimuthu Upeksha >Undergraduate >Department of Computer Science And Engineering >University of Moratuwa, Sri Lanka > >
