subject:"Review Request 22791\: Streaming OODT Changes"

Re: Review Request 22791: Streaming OODT Changes

2014-12-10 Thread Chris Mattmann


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review64661
---

Ship it!


Ship It!

- Chris Mattmann


On Dec. 10, 2014, 9:41 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Dec. 10, 2014, 9:41 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Note: Filemanager work (and corrections here-in) have been moved to sub-patch 
> review: https://reviews.apache.org/r/27172
> 
> Resource manager changes sent to: https://reviews.apache.org/r/27773/ and 
> https://reviews.apache.org/r/28917/
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java
>  1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFramework

Re: Review Request 22791: Streaming OODT Changes

2014-12-10 Thread Michael Starch


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/
---

(Updated Dec. 10, 2014, 9:41 p.m.)


Review request for oodt, Lewis McGibbney and Chris Mattmann.


Repository: oodt


Description (updated)
---

This patch contains all the changes needed to add in "streaming oodt" into the 
oodt svn repository.

There are four main portions:
   -Mesos Framework for Resource Manager (Prototype working)
   -Spark Runner for Workflow Manager (Prototype working)
   -Filemanager "streaming" type (In development)
   -Deployment and cluster management scripts (In development)

Where can this stuff be put so that it is available to use, even while it is in 
development?


Note: Filemanager work (and corrections here-in) have been moved to sub-patch 
review: https://reviews.apache.org/r/27172

Resource manager changes sent to: https://reviews.apache.org/r/27773/ and 
https://reviews.apache.org/r/28917/


Diffs
-

  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
 PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
 PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java
 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/

Re: Review Request 22791: Streaming OODT Changes

2014-11-01 Thread Michael Starch



> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml, line 274
> > 
> >
> > Custom Maven repos are difficult since the Central repository is 
> > phasing them out:
> > 
> > 
> > http://blog.sonatype.com/2010/03/why-external-repos-are-being-phased-out-of-central/
> > 
> > If we absolutely need to ref this repo, can you make it a Maven 
> > profile, not enabled by default, so that users won't have the issue for 
> > this when downloading and building OODT?

These jars exist in the standard repo, so finding them there.


> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java,
> >  line 1
> > 
> >
> > All of these files need ALv2 license headers.

Added ALv2 headers.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
---


On Oct. 24, 2014, 10:35 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Oct. 24, 2014, 10:35 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Note: Filemanager work (and corrections here-in) have been moved to sub-patch 
> review: https://reviews.apache.org/r/27172
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/

Re: Review Request 22791: Streaming OODT Changes

2014-10-24 Thread Michael Starch



> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java,
> >  line 82
> > 
> >
> > how about a similarly derived STREAM loc here, perhaps the same as the 
> > FILENAME suggestion using UUID?

See: https://reviews.apache.org/r/27172


> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java,
> >  line 30
> > 
> >
> > not sure what this change is?

Added guard clause for ProductStructure, so users cannot end up with an unknown 
structure.  All tests that are in violation of this have been fixed. See: 
https://reviews.apache.org/r/27172


> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java,
> >  line 76
> > 
> >
> > rather than silently do something, maybe take this out until there is 
> > something to do?

Done see: https://reviews.apache.org/r/27172


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
---


On Oct. 24, 2014, 10:35 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Oct. 24, 2014, 10:35 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Note: Filemanager work (and corrections here-in) have been moved to sub-patch 
> review: https://reviews.apache.org/r/27172
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
>  16

Re: Review Request 22791: Streaming OODT Changes

2014-10-24 Thread Michael Starch



> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java,
> >  line 79
> > 
> >
> > how about instead of NA, for the filename, we call it a "stream-UUID" 
> > where we generate a unique stream UUID as the FILENAME field.

Shifted this work to sub-patch: https://reviews.apache.org/r/27172


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
---


On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Sept. 4, 2014, 5:23 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java
>  1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/tr

Re: Review Request 22791: Streaming OODT Changes

2014-10-24 Thread Michael Starch


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/
---

(Updated Oct. 24, 2014, 10:35 p.m.)


Review request for oodt, Lewis McGibbney and Chris Mattmann.


Changes
---

Breaking out sub patches, so work isn't held up waiting for all issues to be 
resolved.  Corrected work may now be committed, while other work in progress.  
It will be done in an order that does not leave OODT broken wiaitng for 
dependant changes to be entered.

Filemanager changes moved to: https://reviews.apache.org/r/27172


Repository: oodt


Description (updated)
---

This patch contains all the changes needed to add in "streaming oodt" into the 
oodt svn repository.

There are four main portions:
   -Mesos Framework for Resource Manager (Prototype working)
   -Spark Runner for Workflow Manager (Prototype working)
   -Filemanager "streaming" type (In development)
   -Deployment and cluster management scripts (In development)

Where can this stuff be put so that it is available to use, even while it is in 
development?


Note: Filemanager work (and corrections here-in) have been moved to sub-patch 
review: https://reviews.apache.org/r/27172


Diffs
-

  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
 PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
 PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java
 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException

Re: Review Request 22791: Streaming OODT Changes

2014-10-24 Thread Michael Starch



> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh,
> >  line 1
> > 
> >
> > Mike, instead of directly including cluster-tools in oodt, can you 
> > simply make:
> > 
> > # A Maven AntRun script (build.xml) or something that downloads the 
> > cluster-tools as part of the Resource Manager build? We shouldn't have to 
> > maintain these scripts in OODT.
> 
> Michael Starch wrote:
> Download from where? Do we have a place to upload unmaintained OPS 
> scripts?
> 
> Chris Mattmann wrote:
> THanks Mike. Ask @paulramirez about how to use Maven AntRun or 
> assembly.xml - you could theoretically reference Mesos if it has a 
> distribution for its scripts in the Central repo - you could ref the dist 
> dependency as a dependency and then unpack them dynamically into a directory. 
> If it doesn't have a Maven dist assembly for the Mesos scripts, then my 
> recommendation would simply be to:
> 
> 1. Create a build.xml that downloads (e.g., from Mesos trunk or a tag in 
> Apache git) those scripts into whatever OODT build directory you want (inside 
> of Resource Manager probably makes the most sense)
> 2. Use the Maven Ant-Run plugin to call that build.xml in resource/pom.xml
> 3. Consider using a Maven profile (e.g., mvn -Pstreaming) to insulate 
> your streaming profile cluster-tools download
> 
> Michael Starch wrote:
> To be clear...these are scripts I created to setup and run our cluster 
> for use under OODT.  They currently just handle the supporting Mesos 
> components, but eventually will startup oodt in a streaming-enabled mode as 
> well. The mesos project knows nothing about them.  If we do not want them 
> maintained by OODT then I can remove them from the checkin, however; in my 
> opinion the inculsion of these scripts goes towards making a more 
> ops-friendly OODT package.  This is one of the current goals OODT is working 
> toward.
> 
> Am I missing something here?
> 
> Chris Mattmann wrote:
> Hey Mike, sorry about that: those scripts looked like this:
> 
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=tree;f=bin;h=8497ec15f623cb08cd7a690e79e387be51225c34;hb=HEAD
> 
> I thought some of those were Mesos scripts, but they are not, my bad! Can 
> you please add the ALv2 license headers to them at the
> top? Then we're good to go on this part of my review, sorry again!

Moved to: https://reviews.apache.org/r/27179/ Apache headers have been added.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
---


On Oct. 24, 2014, 10:35 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Oct. 24, 2014, 10:35 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Note: Filemanager work (and corrections here-in) have been moved to sub-patch 
> review: https://reviews.apache.org/r/27172
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION

Re: Review Request 22791: Streaming OODT Changes

2014-09-10 Thread Michael Starch



> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh,
> >  line 1
> > 
> >
> > Mike, instead of directly including cluster-tools in oodt, can you 
> > simply make:
> > 
> > # A Maven AntRun script (build.xml) or something that downloads the 
> > cluster-tools as part of the Resource Manager build? We shouldn't have to 
> > maintain these scripts in OODT.
> 
> Michael Starch wrote:
> Download from where? Do we have a place to upload unmaintained OPS 
> scripts?
> 
> Chris Mattmann wrote:
> THanks Mike. Ask @paulramirez about how to use Maven AntRun or 
> assembly.xml - you could theoretically reference Mesos if it has a 
> distribution for its scripts in the Central repo - you could ref the dist 
> dependency as a dependency and then unpack them dynamically into a directory. 
> If it doesn't have a Maven dist assembly for the Mesos scripts, then my 
> recommendation would simply be to:
> 
> 1. Create a build.xml that downloads (e.g., from Mesos trunk or a tag in 
> Apache git) those scripts into whatever OODT build directory you want (inside 
> of Resource Manager probably makes the most sense)
> 2. Use the Maven Ant-Run plugin to call that build.xml in resource/pom.xml
> 3. Consider using a Maven profile (e.g., mvn -Pstreaming) to insulate 
> your streaming profile cluster-tools download

To be clear...these are scripts I created to setup and run our cluster for use 
under OODT.  They currently just handle the supporting Mesos components, but 
eventually will startup oodt in a streaming-enabled mode as well. The mesos 
project knows nothing about them.  If we do not want them maintained by OODT 
then I can remove them from the checkin, however; in my opinion the inculsion 
of these scripts goes towards making a more ops-friendly OODT package.  This is 
one of the current goals OODT is working toward.

Am I missing something here?


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
---


On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Sept. 4, 2014, 5:23 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.a

Re: Review Request 22791: Streaming OODT Changes

2014-09-04 Thread Chris Mattmann



> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh,
> >  line 1
> > 
> >
> > Mike, instead of directly including cluster-tools in oodt, can you 
> > simply make:
> > 
> > # A Maven AntRun script (build.xml) or something that downloads the 
> > cluster-tools as part of the Resource Manager build? We shouldn't have to 
> > maintain these scripts in OODT.
> 
> Michael Starch wrote:
> Download from where? Do we have a place to upload unmaintained OPS 
> scripts?

THanks Mike. Ask @paulramirez about how to use Maven AntRun or assembly.xml - 
you could theoretically reference Mesos if it has a distribution for its 
scripts in the Central repo - you could ref the dist dependency as a dependency 
and then unpack them dynamically into a directory. If it doesn't have a Maven 
dist assembly for the Mesos scripts, then my recommendation would simply be to:

1. Create a build.xml that downloads (e.g., from Mesos trunk or a tag in Apache 
git) those scripts into whatever OODT build directory you want (inside of 
Resource Manager probably makes the most sense)
2. Use the Maven Ant-Run plugin to call that build.xml in resource/pom.xml
3. Consider using a Maven profile (e.g., mvn -Pstreaming) to insulate your 
streaming profile cluster-tools download


- Chris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
---


On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Sept. 4, 2014, 5:23 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/fi

Re: Review Request 22791: Streaming OODT Changes

2014-09-04 Thread Michael Starch



> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh,
> >  line 1
> > 
> >
> > Mike, instead of directly including cluster-tools in oodt, can you 
> > simply make:
> > 
> > # A Maven AntRun script (build.xml) or something that downloads the 
> > cluster-tools as part of the Resource Manager build? We shouldn't have to 
> > maintain these scripts in OODT.

Download from where? Do we have a place to upload unmaintained OPS scripts?


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
---


On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Sept. 4, 2014, 5:23 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java
>  1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/

Re: Review Request 22791: Streaming OODT Changes

2014-09-04 Thread Chris Mattmann


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
---



http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh


Mike, instead of directly including cluster-tools in oodt, can you simply 
make:

# A Maven AntRun script (build.xml) or something that downloads the 
cluster-tools as part of the Resource Manager build? We shouldn't have to 
maintain these scripts in OODT.



http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml


Custom Maven repos are difficult since the Central repository is phasing 
them out:


http://blog.sonatype.com/2010/03/why-external-repos-are-being-phased-out-of-central/

If we absolutely need to ref this repo, can you make it a Maven profile, 
not enabled by default, so that users won't have the issue for this when 
downloading and building OODT?



http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java


how about instead of NA, for the filename, we call it a "stream-UUID" where 
we generate a unique stream UUID as the FILENAME field.



http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java


how about a similarly derived STREAM loc here, perhaps the same as the 
FILENAME suggestion using UUID?



http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java


rather than silently do something, maybe take this out until there is 
something to do?



http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java


not sure what this change is?



http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java


All of these files need ALv2 license headers.



http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java


We shoudln't be checking in generated files - can this simply be generated 
during the Resmgr build?



http://svn.apache.org/repos/asf/oodt/trunk/streamer/pom.xml


if we don't need this, remove it.



http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/java/org/apache/oodt/cas/streamer/streams/MultiFileSequentialInputStream.java.bak


ALv2 headers here and everywhere



http://svn.apache.org/repos/asf/oodt/trunk/streamer/src/main/resources/streamer.properties


remove JPL specific URLs.


- Chris Mattmann


On Sept. 4, 2014, 5:23 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Sept. 4, 2014, 5:23 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster

Re: Review Request 22791: Streaming OODT Changes

2014-09-04 Thread Michael Starch


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/
---

(Updated Sept. 4, 2014, 5:23 p.m.)


Review request for oodt, Lewis McGibbney and Chris Mattmann.


Changes
---

Chris, Lewis any other comments or can I get a first pass checked in?


Repository: oodt


Description (updated)
---

This patch contains all the changes needed to add in "streaming oodt" into the 
oodt svn repository.

There are four main portions:
   -Mesos Framework for Resource Manager (Prototype working)
   -Spark Runner for Workflow Manager (Prototype working)
   -Filemanager "streaming" type (In development)
   -Deployment and cluster management scripts (In development)

Where can this stuff be put so that it is available to use, even while it is in 
development?


Diffs
-

  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
 PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
 PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java
 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/o

Re: Review Request 22791: Streaming OODT Changes

2014-08-14 Thread Sean Kelly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review50634
---


Do the supporting scripts have to be BASH-specific? We'd show better 
compatibility with a wider range of Unix systems if we could stick with pure 
/bin/sh.

- Sean Kelly


On Aug. 13, 2014, 10:56 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Aug. 13, 2014, 10:56 p.m.)
> 
> 
> Review request for oodt.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java
>  1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMeso

Re: Review Request 22791: Streaming OODT Changes

2014-08-14 Thread Michael Starch



> On Aug. 13, 2014, 11:44 p.m., Lewis McGibbney wrote:
> > Hey Michael,
> > Can you please talk a bit about how streaming works for the FileMgr?
> > I am really interested about that.

The filemanager has two primary functions: catalog metadata, and hold reference 
to a file.  To extrapolate this to streams I needed to capture: stream 
metadata, and a stream handle (product name). 

Therefore, I created a new product structure that contains no references, and 
does not transfer data. Therefore the metadata can be cataloged, and a stream 
handle (name) can be stored as the product name, thus achieving both goals.  
Now the filemgr has three structures (FLAT -- file, HIERARCHICAL -- directory 
of files, STREAM -- no files, just metadata).

The actual data of the stream is captured in Kafka where the stream name is 
called a "topic".  Data can be captured by Kafka using standard Kafka 
data-flows or using a new daemon that will stream into Kafka.  If a standard 
data-flow is needed  then the metadata is stored in the filemanager using a 
single interaction and if the streaming daemon is used, the metadata store 
still uses a single interaction.  Therefore, by separating the streaming of the 
actual data into a new daemon, the burden on the filemanager is reduced from 
continuous interactions streaming in chunks of data to a single interaction per 
stream.

This is the mechanisms used to allow catalog, query and management of streams, 
while separating out the handling of the stream data to maintain filemanager 
efficiency.

Any questions/comments/suggestions?


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review50528
---


On Aug. 13, 2014, 10:56 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Aug. 13, 2014, 10:56 p.m.)
> 
> 
> Review request for oodt.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/

Re: Review Request 22791: Streaming OODT Changes

2014-08-14 Thread Lewis McGibbney



> On Aug. 13, 2014, 11:44 p.m., Lewis McGibbney wrote:
> > Hey Michael,
> > Can you please talk a bit about how streaming works for the FileMgr?
> > I am really interested about that.
> 
> Michael Starch wrote:
> The filemanager has two primary functions: catalog metadata, and hold 
> reference to a file.  To extrapolate this to streams I needed to capture: 
> stream metadata, and a stream handle (product name). 
> 
> Therefore, I created a new product structure that contains no references, 
> and does not transfer data. Therefore the metadata can be cataloged, and a 
> stream handle (name) can be stored as the product name, thus achieving both 
> goals.  Now the filemgr has three structures (FLAT -- file, HIERARCHICAL -- 
> directory of files, STREAM -- no files, just metadata).
> 
> The actual data of the stream is captured in Kafka where the stream name 
> is called a "topic".  Data can be captured by Kafka using standard Kafka 
> data-flows or using a new daemon that will stream into Kafka.  If a standard 
> data-flow is needed  then the metadata is stored in the filemanager using a 
> single interaction and if the streaming daemon is used, the metadata store 
> still uses a single interaction.  Therefore, by separating the streaming of 
> the actual data into a new daemon, the burden on the filemanager is reduced 
> from continuous interactions streaming in chunks of data to a single 
> interaction per stream.
> 
> This is the mechanisms used to allow catalog, query and management of 
> streams, while separating out the handling of the stream data to maintain 
> filemanager efficiency.
> 
> Any questions/comments/suggestions?

Thanks for detailed description. It is going to take me a bit of time to digest 
this... hopefully I will be able to regurgitate some useful comments later 
today/tonight.


- Lewis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review50528
---


On Aug. 13, 2014, 10:56 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Aug. 13, 2014, 10:56 p.m.)
> 
> 
> Review request for oodt.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/struc

Re: Review Request 22791: Streaming OODT Changes

2014-08-14 Thread Michael Starch



> On Aug. 14, 2014, 8:43 p.m., Sean Kelly wrote:
> > Do the supporting scripts have to be BASH-specific? We'd show better 
> > compatibility with a wider range of Unix systems if we could stick with 
> > pure /bin/sh.

Currently the scripts use BASH specific features.  These scripts are not 
required to run the new components, they only simplify the process of starting 
up things and getting a basic environment running.  They fall into the category 
of operations scripts that help the user get an out-of-the-box version of the 
code running.  There are new desires to create "operations"/basic usage scripts 
for OODT to help OODT run out-of-the-box in a preconfigured way.  Do we have a 
standard for these type of scripts?

If compatibility of most systemes is desired then there are a number of other 
questionable things in these scripts...like the usage of "screen".


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review50634
---


On Aug. 13, 2014, 10:56 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Aug. 13, 2014, 10:56 p.m.)
> 
> 
> Review request for oodt.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java
>  1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java
>  PRE-CREATION 
>   
>

Re: Review Request 22791: Streaming OODT Changes

2014-08-14 Thread Sean Kelly


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review50658
---

Ship it!


Hah! I didn't even notice the call to "screen".

OK, I'm fine with BASH the. Thanks.

- Sean Kelly


On Aug. 13, 2014, 10:56 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Aug. 13, 2014, 10:56 p.m.)
> 
> 
> Review request for oodt.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java
>  1617800 
>   http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java
>  PRE-CREATION 
>   
> http://svn.apache.org/

Re: Review Request 22791: Streaming OODT Changes

2014-08-13 Thread Michael Starch


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/
---

(Updated Aug. 13, 2014, 10:56 p.m.)


Review request for oodt.


Changes
---

Updating to the latest patch


Repository: oodt


Description
---

This patch contains all the changes needed to add in "streaming oodt" into the 
oodt svn repository.

There are four main portions:
   -Mesos Framework for Resource Manager (Prototype working)
   -Spark Runner for Workflow Manager (Prototype working)
   -Filemanager "streaming" type (In development)
   -Deployment and cluster management scripts (In development)

Where can this stuff be put so that it is available to use, even while it is in 
development?


Diffs (updated)
-

  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
 PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
 PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java
 1617800 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java
 PRE-CREATION 
  
http://s

Review Request 22791: Streaming OODT Changes

2014-06-19 Thread Michael Starch


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/
---

Review request for oodt.


Repository: oodt


Description
---

This patch contains all the changes needed to add in "streaming oodt" into the 
oodt svn repository.

There are four main portions:
   -Mesos Framework for Resource Manager (Prototype working)
   -Spark Runner for Workflow Manager (Prototype working)
   -Filemanager "streaming" type (In development)
   -Deployment and cluster management scripts (In development)

Where can this stuff be put so that it is available to use, even while it is in 
development?


Diffs
-

  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
 PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
 PRE-CREATION 
  http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java
 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerClient.java
 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java
 1604032 
  http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1604032 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonito

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Re: Review Request 22791: Streaming OODT Changes

Review Request 22791: Streaming OODT Changes

19 matches

Site Navigation

Mail list logo

Footer information