----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22791/ -----------------------------------------------------------
(Updated Oct. 24, 2014, 10:35 p.m.) Review request for oodt, Lewis McGibbney and Chris Mattmann. Changes ------- Breaking out sub patches, so work isn't held up waiting for all issues to be resolved. Corrected work may now be committed, while other work in progress. It will be done in an order that does not leave OODT broken wiaitng for dependant changes to be entered. Filemanager changes moved to: https://reviews.apache.org/r/27172 Repository: oodt Description (updated) ------- This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository. There are four main portions: -Mesos Framework for Resource Manager (Prototype working) -Spark Runner for Workflow Manager (Prototype working) -Filemanager "streaming" type (In development) -Deployment and cluster management scripts (In development) Where can this stuff be put so that it is available to use, even while it is in development? Note: Filemanager work (and corrections here-in) have been moved to sub-patch review: https://reviews.apache.org/r/27172 Diffs ----- http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties PRE-CREATION Diff: https://reviews.apache.org/r/22791/diff/ Testing ------- Basic functionality tests done for both the resource-manger and workflow manager pieces. Filemanager have been tested to properly ingest a "GenericStream" type with the lucene catalog only. Thanks, Michael Starch
