Hi Etienne, For a good description of the push pull architecture checkout the following web link:
http://oodt.apache.org/components/maven/pushpull/development/developer.html I think that this represents a good method to use due to its extensibility. I will at some point start implementing this component to transfer data from site to our archive in Cape Town. Currently I use NFS which is not ideal. As for a typical user scenario - this might be a natural extension point into Virtual Observatory interface. Cheers, Tom On Mon, Aug 18, 2014 at 11:36 AM, Etienne Koen <etien...@scs-space.com> wrote: > Hi Tomas and all, > > I came across the push/pull tutorial on > https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide > . > > Would this guide be more appropriate to download files that have been > archived by the file manager and represent a typical user scenario? > > Regards > Etienne > ________________________________________ > From: Thomas Bennett [tho...@ska.ac.za] > Sent: Friday, August 15, 2014 9:54 AM > To: Etienne Koen > Cc: Mattmann, Chris A (3980); cschol...@ska.ac.za; dev@oodt.apache.org > Subject: Re: Remote data transfer > > Hi Etienne, > > There are various methods you can use to download the data. > > See this page: > > https://cwiki.apache.org/confluence/display/OODT/Getting+products+from+a+remote+FileManager > > Recently there is some great work that has been done on using a REST API - > this exists on svn trunk. I don't think it has been released yet. > > https://cwiki.apache.org/confluence/display/OODT/File+Manager+REST+API > > To use these components you will need to deploy tomcat or jetty. > > Shout if you need some help. > > Cheers, > Tom > > > > > On Thu, Aug 14, 2014 at 4:31 PM, Etienne Koen <etien...@scs-space.com > <mailto:etien...@scs-space.com>> wrote: > Hi Chris and Tom, > > As I have mentioned before in my previous email, I have managed to ingest > a file to a remote location using the filemgr-client. I am also able to > query the information remotely using for example the query_tool in this way: > > $ ./query_tool --url http://192.168.0.10:9000 --lucene -query > 'CAS.ProductName:blah.txt' > > 978ca28e-23b0-11e4-87fb-4f1c29029486 > > What component would I use for searching and downloading the actual > product from the remote file manager? Is the filemgr-client or query_tool > capable of doing this? > > Are there any tutorials you would recommend? > > Thanks > Etienne > > ________________________________________ > From: Mattmann, Chris A (3980) [chris.a.mattm...@jpl.nasa.gov<mailto: > chris.a.mattm...@jpl.nasa.gov>] > Sent: Wednesday, August 13, 2014 6:04 PM > To: Etienne Koen; Thomas Bennett > Cc: cschol...@ska.ac.za<mailto:cschol...@ska.ac.za>; dev@oodt.apache.org > <mailto:dev@oodt.apache.org>; Mattmann, Chris A (3980) > Subject: Re: Remote data transfer > > Thanks guys. > > Etienne, I hope you don't mind but I've copied dev@oodt.apache.org<mailto: > dev@oodt.apache.org> > > on this email. That way you can tap into the entire Apache OODT > community for help. > > The URI has authority component is usually an error indicating > that you have referenced some environment variable in your config > (e.g., filemgr.properties in the etc directory) but that variable > isn't defined. E.g., maybe you have a *.policy.dirs property set > to file://[SOME_UNDEFINED_VARIABLE]/path/dir/ and SOME_UNDEFINED_VARIABLE > is undefined. > > Can you check that to see if that's the root cause of this issue? > > Cheers, > Chris > > ------------------------ > Chris Mattmann > chris.mattm...@gmail.com<mailto:chris.mattm...@gmail.com> > > > > > -----Original Message----- > From: Etienne Koen <etien...@scs-space.com<mailto:etien...@scs-space.com>> > Date: Wednesday, August 13, 2014 1:42 AM > To: Thomas Bennett <tho...@ska.ac.za<mailto:tho...@ska.ac.za>> > Cc: "cschol...@ska.ac.za<mailto:cschol...@ska.ac.za>" <cschol...@ska.ac.za > <mailto:cschol...@ska.ac.za>>, Chris Mattmann > <chris.mattm...@gmail.com<mailto:chris.mattm...@gmail.com>> > Subject: RE: Remote data transfer > > >Hi Tom, > > > >I get the following error when using the argument: > > > >ERROR: Failed to ingest product 'blah.txt' : URI has an authority > >component > > > >Here both the server and client were using port 9000 > > > >I get this when both the server and client are running on the same port > > > >When communicating on different ports I get: > > > ><-- some I/O / HTTP exceptions --> > >... > >... > > > >ERROR: Failed to ingest product 'blah.txt' : Connection refused > > > >Server:9000 and Client:431 > > > >Do you know what any of this mean? > > > >Cheers > >Etienne > > > >________________________________________ > >From: Thomas Bennett [tho...@ska.ac.za<mailto:tho...@ska.ac.za>] > >Sent: Wednesday, August 13, 2014 10:02 AM > >To: Etienne Koen > >Cc: cschol...@ska.ac.za<mailto:cschol...@ska.ac.za>; > chris.mattm...@gmail.com<mailto:chris.mattm...@gmail.com> > >Subject: Re: Remote data transfer > > > >Hey Etienne, > > > >I've been out of the office the last week but I'm back now. > > > >./filemgr-client --url http://localhost:9000 --operation --ingestProduct > >--productName blah.txt --productStructure Flat --productTypeName > >GenericFile --metadataFile file:///tmp/blah.txt.met --refs > >file:///tmp/blah.txt > > > >How would this line be modified to achieve what I want to do? I see there > >is also an argument --clientTransfer --dataTransfer but I am not sure > >what java class to use for this? > > > >You will need to specify the filemgr remotely ie: --url > >http://192.168.0.1 - are you doing this? > > > >I've done remote file transfer before I'll see if I can remember how to > >do it. > > > >Can I log into the CHPC with the usual credentials? > > > >Cheers, > >Tom > >-- > >Thomas Bennett > > > >SKA South Africa > >Science Processing Team > > > >Office: +27 21 5067341<tel:%2B27%2021%205067341> > >Mobile: +27 79 5237105<tel:%2B27%2079%205237105> > > > >________________________________ > >Disclaimer: This E-mail message, including any attachments, is intended > >only for the person or entity to which it is addressed, and may contain > >confidential information. Each page attached hereto must also be read in > >conjunction with this disclaimer. > >If you are not the intended recipient you are hereby notified that any > >disclosure, copying, distribution or reliance upon the contents of this > >e-mail is strictly prohibited. E.&O.E. > > > >Disclaimer: This E-mail message, including any attachments, is intended > >only for the person or entity to which it is addressed, and may contain > >confidential information. Each page attached hereto must also be read in > >conjunction with this disclaimer. > >If you are not the intended recipient you are hereby notified that any > >disclosure, copying, distribution or reliance upon the contents of this > >e-mail is strictly prohibited. E.&O.E. > > > Disclaimer: This E-mail message, including any attachments, is intended > only for the person or entity to which it is addressed, and may contain > confidential information. Each page attached hereto must also be read in > conjunction with this disclaimer. > If you are not the intended recipient you are hereby notified that any > disclosure, copying, distribution or reliance upon the contents of this > e-mail is strictly prohibited. E.&O.E. > > Disclaimer: This E-mail message, including any attachments, is intended > only for the person or entity to which it is addressed, and may contain > confidential information. Each page attached hereto must also be read in > conjunction with this disclaimer. > If you are not the intended recipient you are hereby notified that any > disclosure, copying, distribution or reliance upon the contents of this > e-mail is strictly prohibited. E.&O.E. > > > > -- > Thomas Bennett > > SKA South Africa > Science Processing Team > > Office: +27 21 5067341 > Mobile: +27 79 5237105 > > ________________________________ > Disclaimer: This E-mail message, including any attachments, is intended > only for the person or entity to which it is addressed, and may contain > confidential information. Each page attached hereto must also be read in > conjunction with this disclaimer. > If you are not the intended recipient you are hereby notified that any > disclosure, copying, distribution or reliance upon the contents of this > e-mail is strictly prohibited. E.&O.E. > > Disclaimer: This E-mail message, including any attachments, is intended > only for the person or entity to which it is addressed, and may contain > confidential information. Each page attached hereto must also be read in > conjunction with this disclaimer. > If you are not the intended recipient you are hereby notified that any > disclosure, copying, distribution or reliance upon the contents of this > e-mail is strictly prohibited. E.&O.E. > -- Thomas Bennett SKA South Africa Science Processing Team Office: +27 21 5067341 Mobile: +27 79 5237105
