John, Can you open filemgr-client sh script? It may set the JAVA_EXT_JARS there. If so, it is clobbering the default path for "extension" jars, and your java encryption jars are not being picked up. If it does set JAVA_EXT_JARS you have two options:
1. Move all your encryption jars into FILEMGR_HOME/lib/ 2. update filemgr-client script to us classpath to specify the jars in the FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS -Michael On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <[email protected]> wrote: > Hi Michael > yeah it’s openjdk 1.7 (“1.7.0_75") > i did download the the unlimited encryption jar from oracle and replaced > the local_policy / us_export_policy jars in javahome/jre/lib/security > more i read, maybe limited by jce.jar > > i dont have anything special set for extension jars > > > > > On Mar 12, 2015, at 10:35 AM, Michael Starch <[email protected]> wrote: > > > > John, > > > > What version of the JDK are you running, and what is your extension jars > > environment variable set to. Do you have the java cryptology jar > included > > (Oracle JDK usually has this, I don't know if Open JDK does). > > > > "Algorithm HmacSHA1 not available" is usually thrown when Java cannot > find > > the java crypto jar used to calculate the given hash. > > > > -Michael > > > > On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <[email protected]> > wrote: > > > >> Hi Lewis, > >> using the latest docker buggtb/oodt image, which i assume is .8 > >> here’s the command i’m running to test the upload > >> > >> filemgr-client --url http://localhost:9000 --operation --ingestProduct > >> --productName test --productStructure Flat --productTypeName GenericFile > >> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt > >> > >> i verified that i can upload to the path using the s3 tools on the box / > >> with same credentials i put in the properties file > >> > >> here’s the full exception returned: > >> > >> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >> Failed to upload product reference /root/test.txt to S3 at > >> usr/src/oodt/data/archive/test/test.txt > >> at > >> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> at > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:606) > >> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) > >> at > >> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) > >> at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) > >> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) > >> at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) > >> at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) > >> at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) > >> at java.lang.Thread.run(Thread.java:745) > >> Caused by: > >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >> Failed to upload product reference /root/test.txt to S3 at > >> usr/src/oodt/data/archive/test/test.txt > >> at > >> > org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78) > >> at > >> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752) > >> ... 12 more > >> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a > >> request signature: Unable to calculate a request signature: Algorithm > >> HmacSHA1 not available > >> at > >> > com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71) > >> at > >> > com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57) > >> at > >> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128) > >> at > >> > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330) > >> at > >> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > >> at > >> > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > >> at > >> > com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) > >> at > >> > org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76) > >> ... 13 more > >> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a > >> request signature: Algorithm HmacSHA1 not available > >> at > >> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90) > >> at > >> > com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68) > >> ... 20 more > >> Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 > not > >> available > >> at javax.crypto.Mac.getInstance(Mac.java:176) > >> at > >> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86) > >> ... 21 more > >> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: > >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error > >> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 > ] > >> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >> Failed to upload product reference /root/test.txt to S3 at > >> usr/src/oodt/data/archive/test/test.txt > >> at > >> > org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104) > >> at > >> > org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71) > >> at > >> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73) > >> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) > >> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185) > >> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178) > >> at > >> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198) > >> at > >> > org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) > >> at > >> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) > >> at > >> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) > >> at > >> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) > >> java.lang.Exception: Failed to ingest product > >> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : > >> java.lang.Exception: > >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error > >> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 > ] > >> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >> Failed to upload product reference /root/test.txt to S3 at > >> usr/src/oodt/data/archive/test/test.txt > >> at > >> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303) > >> at > >> > org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) > >> at > >> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) > >> at > >> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) > >> at > >> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) > >> ERROR: Failed to ingest product 'test' : Failed to ingest product > >> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : > >> java.lang.Exception: > >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error > >> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 > ] > >> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >> Failed to upload product reference /root/test.txt to S3 at > >> usr/src/oodt/data/archive/test/test.txt > >> > >> > >>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney < > >> [email protected]> wrote: > >>> > >>> Hi John, > >>> What beau on of OODT are you using? > >>> The file types should be encapsulated by the factory implementation I > >> would > >>> guess. It should be. Case of mike this the default filemgr store then > off > >>> you go. > >>> Can you provide a paste of how your arriving at your exception please? > >>> Lewia > >>> > >>> On Wednesday, March 11, 2015, John Reynolds <[email protected]> > wrote: > >>> > >>>> Should S3DataTransferer work with the generic file type from the get > go > >> or > >>>> is there anything i need to change > >>>> i was using the radix docker but getting 'Caused by: > >>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not > >> available’ > >>>> i tried downloading the jce but no dice > >>>> i might try just building on my local machine and trying again > >>>> > >>>> Thanks > >>>> > >>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney < > >>>> [email protected] <javascript:;>> wrote: > >>>>> > >>>>> Hey John, > >>>>> You're right, you've nearly answered everything here ;) > >>>>> OK, so your properties can be located here > >>>>> > >>>> > >> > https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124 > >>>>> You'll see the usual key, value pairs in there which you can ad > before > >>>>> build the project and packaging filemgr.properties along with your > >>>> compiled > >>>>> code. > >>>>> Regarding the following, > >>>>> > >>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds <[email protected] > >>>> <javascript:;>> wrote: > >>>>> > >>>>>> > >>>>>> in the near future, i will want to pass in a security token as well > to > >>>> the > >>>>>> S3 client. to do so i would extend or modify the s3datatransfer > class > >>>>>> (which expires after an hour), what’s the best practice to read in > >> this > >>>>>> transient data from the command line (or wherever else i would > >> initiate > >>>> a > >>>>>> job from) since this wouldnt go in a properties file > >>>>> > >>>>> > >>>>> Do you mean that your security token expires hourly? As oppose to > your > >> s3 > >>>>> client? > >>>>> Lets try and defined exactly what you are after here. > >>>>> Thanks > >>>>> Lewis > >>>> > >>>> > >>> > >>> -- > >>> *Lewis* > >> > >> > >
