Hi Lewis, using the latest docker buggtb/oodt image, which i assume is .8 here’s the command i’m running to test the upload
filemgr-client --url http://localhost:9000 --operation --ingestProduct --productName test --productStructure Flat --productTypeName GenericFile --metadataFile file:///root/test.txt.met --refs file:///root/test.txt i verified that i can upload to the path using the s3 tools on the box / with same credentials i put in the properties file here’s the full exception returned: rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt at org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752) ... 12 more Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable to calculate a request signature: Algorithm HmacSHA1 not available at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71) at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57) at com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) at org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76) ... 13 more Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Algorithm HmacSHA1 not available at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90) at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68) ... 20 more Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not available at javax.crypto.Mac.getInstance(Mac.java:176) at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86) ... 21 more org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1] : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104) at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71) at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198) at org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) java.lang.Exception: Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1] : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303) at org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) ERROR: Failed to ingest product 'test' : Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1] : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt > On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney <[email protected]> > wrote: > > Hi John, > What beau on of OODT are you using? > The file types should be encapsulated by the factory implementation I would > guess. It should be. Case of mike this the default filemgr store then off > you go. > Can you provide a paste of how your arriving at your exception please? > Lewia > > On Wednesday, March 11, 2015, John Reynolds <[email protected]> wrote: > >> Should S3DataTransferer work with the generic file type from the get go or >> is there anything i need to change >> i was using the radix docker but getting 'Caused by: >> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not available’ >> i tried downloading the jce but no dice >> i might try just building on my local machine and trying again >> >> Thanks >> >>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney < >> [email protected] <javascript:;>> wrote: >>> >>> Hey John, >>> You're right, you've nearly answered everything here ;) >>> OK, so your properties can be located here >>> >> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124 >>> You'll see the usual key, value pairs in there which you can ad before >>> build the project and packaging filemgr.properties along with your >> compiled >>> code. >>> Regarding the following, >>> >>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <[email protected] >> <javascript:;>> wrote: >>> >>>> >>>> in the near future, i will want to pass in a security token as well to >> the >>>> S3 client. to do so i would extend or modify the s3datatransfer class >>>> (which expires after an hour), what’s the best practice to read in this >>>> transient data from the command line (or wherever else i would initiate >> a >>>> job from) since this wouldnt go in a properties file >>> >>> >>> Do you mean that your security token expires hourly? As oppose to your s3 >>> client? >>> Lets try and defined exactly what you are after here. >>> Thanks >>> Lewis >> >> > > -- > *Lewis*
