> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote: > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh, > > line 1 > > <https://reviews.apache.org/r/22791/diff/2/?file=659631#file659631line1> > > > > Mike, instead of directly including cluster-tools in oodt, can you > > simply make: > > > > # A Maven AntRun script (build.xml) or something that downloads the > > cluster-tools as part of the Resource Manager build? We shouldn't have to > > maintain these scripts in OODT. > > Michael Starch wrote: > Download from where? Do we have a place to upload unmaintained OPS > scripts? > > Chris Mattmann wrote: > THanks Mike. Ask @paulramirez about how to use Maven AntRun or > assembly.xml - you could theoretically reference Mesos if it has a > distribution for its scripts in the Central repo - you could ref the dist > dependency as a dependency and then unpack them dynamically into a directory. > If it doesn't have a Maven dist assembly for the Mesos scripts, then my > recommendation would simply be to: > > 1. Create a build.xml that downloads (e.g., from Mesos trunk or a tag in > Apache git) those scripts into whatever OODT build directory you want (inside > of Resource Manager probably makes the most sense) > 2. Use the Maven Ant-Run plugin to call that build.xml in resource/pom.xml > 3. Consider using a Maven profile (e.g., mvn -Pstreaming) to insulate > your streaming profile cluster-tools download > > Michael Starch wrote: > To be clear...these are scripts I created to setup and run our cluster > for use under OODT. They currently just handle the supporting Mesos > components, but eventually will startup oodt in a streaming-enabled mode as > well. The mesos project knows nothing about them. If we do not want them > maintained by OODT then I can remove them from the checkin, however; in my > opinion the inculsion of these scripts goes towards making a more > ops-friendly OODT package. This is one of the current goals OODT is working > toward. > > Am I missing something here? > > Chris Mattmann wrote: > Hey Mike, sorry about that: those scripts looked like this: > > https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=tree;f=bin;h=8497ec15f623cb08cd7a690e79e387be51225c34;hb=HEAD > > I thought some of those were Mesos scripts, but they are not, my bad! Can > you please add the ALv2 license headers to them at the > top? Then we're good to go on this part of my review, sorry again!
Moved to: https://reviews.apache.org/r/27179/ Apache headers have been added. - Michael ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22791/#review52324 ----------------------------------------------------------- On Oct. 24, 2014, 10:35 p.m., Michael Starch wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22791/ > ----------------------------------------------------------- > > (Updated Oct. 24, 2014, 10:35 p.m.) > > > Review request for oodt, Lewis McGibbney and Chris Mattmann. > > > Repository: oodt > > > Description > ------- > > This patch contains all the changes needed to add in "streaming oodt" into > the oodt svn repository. > > There are four main portions: > -Mesos Framework for Resource Manager (Prototype working) > -Spark Runner for Workflow Manager (Prototype working) > -Filemanager "streaming" type (In development) > -Deployment and cluster management scripts (In development) > > Where can this stuff be put so that it is available to use, even while it is > in development? > > > Note: Filemanager work (and corrections here-in) have been moved to sub-patch > review: https://reviews.apache.org/r/27172 > > > Diffs > ----- > > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java > 1617800 > http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/Scheduler.java > 1617800 > > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/proto/resc.proto > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/assembly/assembly.xml > PRE-CREATION > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/bin/streamer > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/KafkaPublisher.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/publisher/Publisher.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/InputStreamReader.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/Reader.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/reader/StreamEmptyException.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStreamArcheaic.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/system/MultiSourceStreamer.java > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-actions.xml > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/cmd-line-options.xml > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/logging.properties > PRE-CREATION > > http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties > PRE-CREATION > > Diff: https://reviews.apache.org/r/22791/diff/ > > > Testing > ------- > > Basic functionality tests done for both the resource-manger and workflow > manager pieces. Filemanager have been tested to properly ingest a > "GenericStream" type with the lucene catalog only. > > > Thanks, > > Michael Starch > >
