Re: Review Request 22791: Streaming OODT Changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22791/ --- (Updated Aug. 13, 2014, 10:56 p.m.) Review request for oodt. Changes --- Updating to the latest patch Repository: oodt Description --- This patch contains all the changes needed to add in "streaming oodt" into the oodt svn repository. There are four main portions: -Mesos Framework for Resource Manager (Prototype working) -Spark Runner for Workflow Manager (Prototype working) -Filemanager "streaming" type (In development) -Deployment and cluster management scripts (In development) Where can this stuff be put so that it is available to use, even while it is in development? Diffs (updated) - http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/VersioningUtils.java 1617800 http://svn.apache.org/repos/asf/oodt/trunk/resource/pom.xml 1617800 http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManager.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/batchmgr/MesosBatchManagerFactory.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/MesosUtilities.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/OODTExecutor.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosScheduler.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/exception/MesosFrameworkException.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/proto/ResourceProto.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitor.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/monitor/MesosMonitorFactory.java PRE-CREATION http://s
Re: Remote data transfer
Thanks guys. Etienne, I hope you don't mind but I've copied dev@oodt.apache.org on this email. That way you can tap into the entire Apache OODT community for help. The URI has authority component is usually an error indicating that you have referenced some environment variable in your config (e.g., filemgr.properties in the etc directory) but that variable isn't defined. E.g., maybe you have a *.policy.dirs property set to file://[SOME_UNDEFINED_VARIABLE]/path/dir/ and SOME_UNDEFINED_VARIABLE is undefined. Can you check that to see if that's the root cause of this issue? Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Etienne Koen Date: Wednesday, August 13, 2014 1:42 AM To: Thomas Bennett Cc: "cschol...@ska.ac.za" , Chris Mattmann Subject: RE: Remote data transfer >Hi Tom, > >I get the following error when using the argument: > >ERROR: Failed to ingest product 'blah.txt' : URI has an authority >component > >Here both the server and client were using port 9000 > >I get this when both the server and client are running on the same port > >When communicating on different ports I get: > ><-- some I/O / HTTP exceptions --> >... >... > >ERROR: Failed to ingest product 'blah.txt' : Connection refused > >Server:9000 and Client:431 > >Do you know what any of this mean? > >Cheers >Etienne > > >From: Thomas Bennett [tho...@ska.ac.za] >Sent: Wednesday, August 13, 2014 10:02 AM >To: Etienne Koen >Cc: cschol...@ska.ac.za; chris.mattm...@gmail.com >Subject: Re: Remote data transfer > >Hey Etienne, > >I've been out of the office the last week but I'm back now. > >./filemgr-client --url http://localhost:9000 --operation --ingestProduct >--productName blah.txt --productStructure Flat --productTypeName >GenericFile --metadataFile file:///tmp/blah.txt.met --refs >file:///tmp/blah.txt > >How would this line be modified to achieve what I want to do? I see there >is also an argument --clientTransfer --dataTransfer but I am not sure >what java class to use for this? > >You will need to specify the filemgr remotely ie: --url >http://192.168.0.1 - are you doing this? > >I've done remote file transfer before I'll see if I can remember how to >do it. > >Can I log into the CHPC with the usual credentials? > >Cheers, >Tom >-- >Thomas Bennett > >SKA South Africa >Science Processing Team > >Office: +27 21 5067341 >Mobile: +27 79 5237105 > > >Disclaimer: This E-mail message, including any attachments, is intended >only for the person or entity to which it is addressed, and may contain >confidential information. Each page attached hereto must also be read in >conjunction with this disclaimer. >If you are not the intended recipient you are hereby notified that any >disclosure, copying, distribution or reliance upon the contents of this >e-mail is strictly prohibited. E.&O.E. > >Disclaimer: This E-mail message, including any attachments, is intended >only for the person or entity to which it is addressed, and may contain >confidential information. Each page attached hereto must also be read in >conjunction with this disclaimer. >If you are not the intended recipient you are hereby notified that any >disclosure, copying, distribution or reliance upon the contents of this >e-mail is strictly prohibited.E.&O.E.
