Re: filemgr ingestion
Thanks for clearing that out Chris, I think that the capability of the filemgr should be considered as a test on its own and does not actually fall into the category which we are currently benchmarking for the baseline WAN/LAN performance of the tools. There should be a separate test section for archiving... Thanks for the help! Etienne On Sep 10, 2014 6:15 PM, Chris Mattmann mattm...@apache.org wrote: Hi Etienne, Thanks for your query. The default protocol used by the RemoteDataTransfer is XML-RPC. It's a call-back mechanism that I baked up myself with a configurable chunk size parameter. My suggestion is rather than use that DataTransfer, I would leverage: (1) a distributed filesystem like HadoopFS, Tachyon (from BDAS), GlusterFS, or even NFS that logically mounts a set of local shared nothing disks as a virtual global mount (2) use the LocalDataTransfer in the File Manager, with #1 in place The main reason for this is that it allows the people who work on distributed and reliable filesystems that solve those problems in a great way, it allows us in the OODT community to take advantage of that work without having to write all the complex functionality ourselves. That way we just have light-weight insulated pieces/components in OODT that rely on the workhorses that do things right. HTH! Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Etienne Koen koe...@gmail.com Date: Wednesday, September 10, 2014 1:57 AM To: Shakeh Khudikyan shakeh.e.khudik...@jpl.nasa.gov Cc: Chris Mattmann chris.mattm...@gmail.com Subject: filemgr ingestion Hi Shakeh, What is the default protocol that filemgr uses for RemoteDataTransfer ingestion? Also, is the a way to specify the protocol? Cheers Etienne
Re: filemgr ingestion
No problem Etienne, and yes that's a fair statement! :) Cheers man. Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Etienne Koen koe...@gmail.com Date: Wednesday, September 10, 2014 9:37 AM To: Chris Mattmann mattm...@apache.org Cc: Shakeh Khudikyan shakeh.e.khudik...@jpl.nasa.gov, dev@oodt.apache.org dev@oodt.apache.org Subject: Re: filemgr ingestion Thanks for clearing that out Chris, I think that the capability of the filemgr should be considered as a test on its own and does not actually fall into the category which we are currently benchmarking for the baseline WAN/LAN performance of the tools. There should be a separate test section for archiving... Thanks for the help! Etienne On Sep 10, 2014 6:15 PM, Chris Mattmann mattm...@apache.org wrote: Hi Etienne, Thanks for your query. The default protocol used by the RemoteDataTransfer is XML-RPC. It's a call-back mechanism that I baked up myself with a configurable chunk size parameter. My suggestion is rather than use that DataTransfer, I would leverage: (1) a distributed filesystem like HadoopFS, Tachyon (from BDAS), GlusterFS, or even NFS that logically mounts a set of local shared nothing disks as a virtual global mount (2) use the LocalDataTransfer in the File Manager, with #1 in place The main reason for this is that it allows the people who work on distributed and reliable filesystems that solve those problems in a great way, it allows us in the OODT community to take advantage of that work without having to write all the complex functionality ourselves. That way we just have light-weight insulated pieces/components in OODT that rely on the workhorses that do things right. HTH! Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Etienne Koen koe...@gmail.com Date: Wednesday, September 10, 2014 1:57 AM To: Shakeh Khudikyan shakeh.e.khudik...@jpl.nasa.gov Cc: Chris Mattmann chris.mattm...@gmail.com Subject: filemgr ingestion Hi Shakeh, What is the default protocol that filemgr uses for RemoteDataTransfer ingestion? Also, is the a way to specify the protocol? Cheers Etienne
Re: Review Request 22791: Streaming OODT Changes
On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote: http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh, line 1 https://reviews.apache.org/r/22791/diff/2/?file=659631#file659631line1 Mike, instead of directly including cluster-tools in oodt, can you simply make: # A Maven AntRun script (build.xml) or something that downloads the cluster-tools as part of the Resource Manager build? We shouldn't have to maintain these scripts in OODT. Michael Starch wrote: Download from where? Do we have a place to upload unmaintained OPS scripts? Chris Mattmann wrote: THanks Mike. Ask @paulramirez about how to use Maven AntRun or assembly.xml - you could theoretically reference Mesos if it has a distribution for its scripts in the Central repo - you could ref the dist dependency as a dependency and then unpack them dynamically into a directory. If it doesn't have a Maven dist assembly for the Mesos scripts, then my recommendation would simply be to: 1. Create a build.xml that downloads (e.g., from Mesos trunk or a tag in Apache git) those scripts into whatever OODT build directory you want (inside of Resource Manager probably makes the most sense) 2. Use the Maven Ant-Run plugin to call that build.xml in resource/pom.xml 3. Consider using a Maven profile (e.g., mvn -Pstreaming) to insulate your streaming profile cluster-tools download To be clear...these are scripts I created to setup and run our cluster for use under OODT. They currently just handle the supporting Mesos components, but eventually will startup oodt in a streaming-enabled mode as well. The mesos project knows nothing about them. If we do not want them maintained by OODT then I can remove them from the checkin, however; in my opinion the inculsion of these scripts goes towards making a more ops-friendly OODT package. This is one of the current goals OODT is working toward. Am I missing something here? - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22791/#review52324 --- On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22791/ --- (Updated Sept. 4, 2014, 5:23 p.m.) Review request for oodt, Lewis McGibbney and Chris Mattmann. Repository: oodt Description --- This patch contains all the changes needed to add in streaming oodt into the oodt svn repository. There are four main portions: -Mesos Framework for Resource Manager (Prototype working) -Spark Runner for Workflow Manager (Prototype working) -Filemanager streaming type (In development) -Deployment and cluster management scripts (In development) Where can this stuff be put so that it is available to use, even while it is in development? Diffs - http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800
Re: OODT 0.7 status
Hey Tom, Once I fix the Resource Manager, I would like to volunteer to be the RM for 0.7 and push! That sound OK? So maybe RC tomorrow or latest Friday and 0.7 release next week if VOTE'ing goes well? Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Tom Barber tom.bar...@meteorite.bi Date: Thursday, September 4, 2014 8:35 AM To: dev@oodt.apache.org dev@oodt.apache.org Subject: OODT 0.7 status Hello folks, I dunno if I've just missed an email or what but whats the 0.7 status? Code freeze? still open? packaged and ready to roll? Thanks Tom -- Tom Barber | Technical Director meteorite bi T: +44 20 8133 3730 W: www.meteorite.bi http://www.meteorite.bi | Skype: meteorite.consulting A: Surrey Technology Centre, Surrey Research Park, Guildford, GU2 7YG, UK
