Thanks Michael, this works for me (filemanager starts), however i’m still getting the same datatransferer error
> On Mar 12, 2015, at 1:52 PM, Michael Starch <[email protected]> wrote: > > John, > > I should be more verbose. The java classpath traditionally did not pick up > multiple jars, so it was super labor intensive to setup. These days you > can use * inside the classpath to pick up multiple jars....but it must be > in " " because otherwise the shell will glob the * if it is outside of > quotes. > > If this doesn't work, try the other recommendations in: > > > http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath > > -Michael > > > On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch <[email protected]> wrote: > >> John, >> >> Change: >> -classpath "$FILEMGR_HOME"/lib \ >> To: >> -classpath "$FILEMGR_HOME/lib/*" \ >> >> -Michael >> >> >> On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds <[email protected]> >> wrote: >> >>> Thanks Michael, if i modify the filemgr-client to look like this (at the >>> end) >>> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \ >>> -classpath "$FILEMGR_HOME"/lib \ >>> >>> -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties >>> \ >>> -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \ >>> >>> -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml >>> \ >>> >>> -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml >>> \ >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“ >>> >>> (replacing ext jars with -classpath) then i get >>> Error: Could not find or load main class >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient >>> i assume i’m doing something wrong with classpath but not sure what >>> >>>> On Mar 12, 2015, at 11:31 AM, Michael Starch <[email protected]> >>> wrote: >>>> >>>> John, >>>> >>>> Can you open filemgr-client sh script? It may set the JAVA_EXT_JARS >>>> there. If so, it is clobbering the default path for "extension" jars, >>> and >>>> your java encryption jars are not being picked up. If it does set >>>> JAVA_EXT_JARS you have two options: >>>> >>>> 1. Move all your encryption jars into FILEMGR_HOME/lib/ >>>> 2. update filemgr-client script to us classpath to specify the jars in >>> the >>>> FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS >>>> >>>> >>>> -Michael >>>> >>>> >>>> On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <[email protected]> >>> wrote: >>>> >>>>> Hi Michael >>>>> yeah it’s openjdk 1.7 (“1.7.0_75") >>>>> i did download the the unlimited encryption jar from oracle and >>> replaced >>>>> the local_policy / us_export_policy jars in javahome/jre/lib/security >>>>> more i read, maybe limited by jce.jar >>>>> >>>>> i dont have anything special set for extension jars >>>>> >>>>> >>>>> >>>>>> On Mar 12, 2015, at 10:35 AM, Michael Starch <[email protected]> >>> wrote: >>>>>> >>>>>> John, >>>>>> >>>>>> What version of the JDK are you running, and what is your extension >>> jars >>>>>> environment variable set to. Do you have the java cryptology jar >>>>> included >>>>>> (Oracle JDK usually has this, I don't know if Open JDK does). >>>>>> >>>>>> "Algorithm HmacSHA1 not available" is usually thrown when Java cannot >>>>> find >>>>>> the java crypto jar used to calculate the given hash. >>>>>> >>>>>> -Michael >>>>>> >>>>>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <[email protected]> >>>>> wrote: >>>>>> >>>>>>> Hi Lewis, >>>>>>> using the latest docker buggtb/oodt image, which i assume is .8 >>>>>>> here’s the command i’m running to test the upload >>>>>>> >>>>>>> filemgr-client --url http://localhost:9000 --operation >>> --ingestProduct >>>>>>> --productName test --productStructure Flat --productTypeName >>> GenericFile >>>>>>> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt >>>>>>> >>>>>>> i verified that i can upload to the path using the s3 tools on the >>> box / >>>>>>> with same credentials i put in the properties file >>>>>>> >>>>>>> here’s the full exception returned: >>>>>>> >>>>>>> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>>>>>> Failed to upload product reference /root/test.txt to S3 at >>>>>>> usr/src/oodt/data/archive/test/test.txt >>>>>>> at >>>>>>> >>>>> >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>> at >>>>>>> >>>>> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>> at >>>>>>> >>>>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) >>>>>>> at >>>>>>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) >>>>>>> at >>> org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) >>>>>>> at >>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) >>>>>>> at >>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) >>>>>>> at >>> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) >>>>>>> at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) >>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>> Caused by: >>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>>>>>> Failed to upload product reference /root/test.txt to S3 at >>>>>>> usr/src/oodt/data/archive/test/test.txt >>>>>>> at >>>>>>> >>>>> >>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78) >>>>>>> at >>>>>>> >>>>> >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752) >>>>>>> ... 12 more >>>>>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a >>>>>>> request signature: Unable to calculate a request signature: Algorithm >>>>>>> HmacSHA1 not available >>>>>>> at >>>>>>> >>>>> >>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71) >>>>>>> at >>>>>>> >>>>> >>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57) >>>>>>> at >>>>>>> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128) >>>>>>> at >>>>>>> >>>>> >>> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330) >>>>>>> at >>>>>>> >>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) >>>>>>> at >>>>>>> >>>>> >>> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) >>>>>>> at >>>>>>> >>>>> >>> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) >>>>>>> at >>>>>>> >>>>> >>> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76) >>>>>>> ... 13 more >>>>>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a >>>>>>> request signature: Algorithm HmacSHA1 not available >>>>>>> at >>>>>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90) >>>>>>> at >>>>>>> >>>>> >>> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68) >>>>>>> ... 20 more >>>>>>> Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 >>>>> not >>>>>>> available >>>>>>> at javax.crypto.Mac.getInstance(Mac.java:176) >>>>>>> at >>>>>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86) >>>>>>> ... 21 more >>>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: >>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: >>> Error >>>>>>> ingesting product >>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 >>>>> ] >>>>>>> : >>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>>>>>> Failed to upload product reference /root/test.txt to S3 at >>>>>>> usr/src/oodt/data/archive/test/test.txt >>>>>>> at >>>>>>> >>>>> >>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104) >>>>>>> at >>>>>>> >>>>> >>> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71) >>>>>>> at >>>>>>> >>> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73) >>>>>>> at >>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) >>>>>>> at >>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185) >>>>>>> at >>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178) >>>>>>> at >>>>>>> >>>>> >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198) >>>>>>> at >>>>>>> >>>>> >>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) >>>>>>> at >>>>>>> >>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) >>>>>>> at >>>>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) >>>>>>> at >>>>>>> >>>>> >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) >>>>>>> java.lang.Exception: Failed to ingest product >>>>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : >>>>>>> java.lang.Exception: >>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: >>> Error >>>>>>> ingesting product >>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 >>>>> ] >>>>>>> : >>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>>>>>> Failed to upload product reference /root/test.txt to S3 at >>>>>>> usr/src/oodt/data/archive/test/test.txt >>>>>>> at >>>>>>> >>>>> >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303) >>>>>>> at >>>>>>> >>>>> >>> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) >>>>>>> at >>>>>>> >>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) >>>>>>> at >>>>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) >>>>>>> at >>>>>>> >>>>> >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) >>>>>>> ERROR: Failed to ingest product 'test' : Failed to ingest product >>>>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : >>>>>>> java.lang.Exception: >>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: >>> Error >>>>>>> ingesting product >>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 >>>>> ] >>>>>>> : >>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>>>>>> Failed to upload product reference /root/test.txt to S3 at >>>>>>> usr/src/oodt/data/archive/test/test.txt >>>>>>> >>>>>>> >>>>>>>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney < >>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>> Hi John, >>>>>>>> What beau on of OODT are you using? >>>>>>>> The file types should be encapsulated by the factory implementation >>> I >>>>>>> would >>>>>>>> guess. It should be. Case of mike this the default filemgr store >>> then >>>>> off >>>>>>>> you go. >>>>>>>> Can you provide a paste of how your arriving at your exception >>> please? >>>>>>>> Lewia >>>>>>>> >>>>>>>> On Wednesday, March 11, 2015, John Reynolds <[email protected]> >>>>> wrote: >>>>>>>> >>>>>>>>> Should S3DataTransferer work with the generic file type from the >>> get >>>>> go >>>>>>> or >>>>>>>>> is there anything i need to change >>>>>>>>> i was using the radix docker but getting 'Caused by: >>>>>>>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not >>>>>>> available’ >>>>>>>>> i tried downloading the jce but no dice >>>>>>>>> i might try just building on my local machine and trying again >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney < >>>>>>>>> [email protected] <javascript:;>> wrote: >>>>>>>>>> >>>>>>>>>> Hey John, >>>>>>>>>> You're right, you've nearly answered everything here ;) >>>>>>>>>> OK, so your properties can be located here >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124 >>>>>>>>>> You'll see the usual key, value pairs in there which you can ad >>>>> before >>>>>>>>>> build the project and packaging filemgr.properties along with your >>>>>>>>> compiled >>>>>>>>>> code. >>>>>>>>>> Regarding the following, >>>>>>>>>> >>>>>>>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds < >>> [email protected] >>>>>>>>> <javascript:;>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> in the near future, i will want to pass in a security token as >>> well >>>>> to >>>>>>>>> the >>>>>>>>>>> S3 client. to do so i would extend or modify the s3datatransfer >>>>> class >>>>>>>>>>> (which expires after an hour), what’s the best practice to read >>> in >>>>>>> this >>>>>>>>>>> transient data from the command line (or wherever else i would >>>>>>> initiate >>>>>>>>> a >>>>>>>>>>> job from) since this wouldnt go in a properties file >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Do you mean that your security token expires hourly? As oppose to >>>>> your >>>>>>> s3 >>>>>>>>>> client? >>>>>>>>>> Lets try and defined exactly what you are after here. >>>>>>>>>> Thanks >>>>>>>>>> Lewis >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Lewis* >>>>>>> >>>>>>> >>>>> >>>>> >>> >>> >>
