John, I should be more verbose. The java classpath traditionally did not pick up multiple jars, so it was super labor intensive to setup. These days you can use * inside the classpath to pick up multiple jars....but it must be in " " because otherwise the shell will glob the * if it is outside of quotes.
If this doesn't work, try the other recommendations in: http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath -Michael On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch <[email protected]> wrote: > John, > > Change: > -classpath "$FILEMGR_HOME"/lib \ > To: > -classpath "$FILEMGR_HOME/lib/*" \ > > -Michael > > > On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds <[email protected]> > wrote: > >> Thanks Michael, if i modify the filemgr-client to look like this (at the >> end) >> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \ >> -classpath "$FILEMGR_HOME"/lib \ >> >> -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties >> \ >> -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \ >> >> -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml >> \ >> >> -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml >> \ >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“ >> >> (replacing ext jars with -classpath) then i get >> Error: Could not find or load main class >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient >> i assume i’m doing something wrong with classpath but not sure what >> >> > On Mar 12, 2015, at 11:31 AM, Michael Starch <[email protected]> >> wrote: >> > >> > John, >> > >> > Can you open filemgr-client sh script? It may set the JAVA_EXT_JARS >> > there. If so, it is clobbering the default path for "extension" jars, >> and >> > your java encryption jars are not being picked up. If it does set >> > JAVA_EXT_JARS you have two options: >> > >> > 1. Move all your encryption jars into FILEMGR_HOME/lib/ >> > 2. update filemgr-client script to us classpath to specify the jars in >> the >> > FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS >> > >> > >> > -Michael >> > >> > >> > On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <[email protected]> >> wrote: >> > >> >> Hi Michael >> >> yeah it’s openjdk 1.7 (“1.7.0_75") >> >> i did download the the unlimited encryption jar from oracle and >> replaced >> >> the local_policy / us_export_policy jars in javahome/jre/lib/security >> >> more i read, maybe limited by jce.jar >> >> >> >> i dont have anything special set for extension jars >> >> >> >> >> >> >> >>> On Mar 12, 2015, at 10:35 AM, Michael Starch <[email protected]> >> wrote: >> >>> >> >>> John, >> >>> >> >>> What version of the JDK are you running, and what is your extension >> jars >> >>> environment variable set to. Do you have the java cryptology jar >> >> included >> >>> (Oracle JDK usually has this, I don't know if Open JDK does). >> >>> >> >>> "Algorithm HmacSHA1 not available" is usually thrown when Java cannot >> >> find >> >>> the java crypto jar used to calculate the given hash. >> >>> >> >>> -Michael >> >>> >> >>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <[email protected]> >> >> wrote: >> >>> >> >>>> Hi Lewis, >> >>>> using the latest docker buggtb/oodt image, which i assume is .8 >> >>>> here’s the command i’m running to test the upload >> >>>> >> >>>> filemgr-client --url http://localhost:9000 --operation >> --ingestProduct >> >>>> --productName test --productStructure Flat --productTypeName >> GenericFile >> >>>> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt >> >>>> >> >>>> i verified that i can upload to the path using the s3 tools on the >> box / >> >>>> with same credentials i put in the properties file >> >>>> >> >>>> here’s the full exception returned: >> >>>> >> >>>> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >> >>>> Failed to upload product reference /root/test.txt to S3 at >> >>>> usr/src/oodt/data/archive/test/test.txt >> >>>> at >> >>>> >> >> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) >> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >>>> at >> >>>> >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >>>> at >> >>>> >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>>> at java.lang.reflect.Method.invoke(Method.java:606) >> >>>> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) >> >>>> at >> >>>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) >> >>>> at >> org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) >> >>>> at >> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) >> >>>> at >> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) >> >>>> at >> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) >> >>>> at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) >> >>>> at java.lang.Thread.run(Thread.java:745) >> >>>> Caused by: >> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >> >>>> Failed to upload product reference /root/test.txt to S3 at >> >>>> usr/src/oodt/data/archive/test/test.txt >> >>>> at >> >>>> >> >> >> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78) >> >>>> at >> >>>> >> >> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752) >> >>>> ... 12 more >> >>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a >> >>>> request signature: Unable to calculate a request signature: Algorithm >> >>>> HmacSHA1 not available >> >>>> at >> >>>> >> >> >> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71) >> >>>> at >> >>>> >> >> >> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57) >> >>>> at >> >>>> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128) >> >>>> at >> >>>> >> >> >> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330) >> >>>> at >> >>>> >> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) >> >>>> at >> >>>> >> >> >> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) >> >>>> at >> >>>> >> >> >> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) >> >>>> at >> >>>> >> >> >> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76) >> >>>> ... 13 more >> >>>> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a >> >>>> request signature: Algorithm HmacSHA1 not available >> >>>> at >> >>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90) >> >>>> at >> >>>> >> >> >> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68) >> >>>> ... 20 more >> >>>> Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 >> >> not >> >>>> available >> >>>> at javax.crypto.Mac.getInstance(Mac.java:176) >> >>>> at >> >>>> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86) >> >>>> ... 21 more >> >>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: >> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: >> Error >> >>>> ingesting product >> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 >> >> ] >> >>>> : >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >> >>>> Failed to upload product reference /root/test.txt to S3 at >> >>>> usr/src/oodt/data/archive/test/test.txt >> >>>> at >> >>>> >> >> >> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104) >> >>>> at >> >>>> >> >> >> org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71) >> >>>> at >> >>>> >> org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73) >> >>>> at >> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) >> >>>> at >> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185) >> >>>> at >> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178) >> >>>> at >> >>>> >> >> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198) >> >>>> at >> >>>> >> >> >> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) >> >>>> at >> >>>> >> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) >> >>>> at >> >>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) >> >>>> at >> >>>> >> >> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) >> >>>> java.lang.Exception: Failed to ingest product >> >>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : >> >>>> java.lang.Exception: >> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: >> Error >> >>>> ingesting product >> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 >> >> ] >> >>>> : >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >> >>>> Failed to upload product reference /root/test.txt to S3 at >> >>>> usr/src/oodt/data/archive/test/test.txt >> >>>> at >> >>>> >> >> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303) >> >>>> at >> >>>> >> >> >> org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) >> >>>> at >> >>>> >> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) >> >>>> at >> >>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) >> >>>> at >> >>>> >> >> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) >> >>>> ERROR: Failed to ingest product 'test' : Failed to ingest product >> >>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : >> >>>> java.lang.Exception: >> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: >> Error >> >>>> ingesting product >> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 >> >> ] >> >>>> : >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >> >>>> Failed to upload product reference /root/test.txt to S3 at >> >>>> usr/src/oodt/data/archive/test/test.txt >> >>>> >> >>>> >> >>>>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney < >> >>>> [email protected]> wrote: >> >>>>> >> >>>>> Hi John, >> >>>>> What beau on of OODT are you using? >> >>>>> The file types should be encapsulated by the factory implementation >> I >> >>>> would >> >>>>> guess. It should be. Case of mike this the default filemgr store >> then >> >> off >> >>>>> you go. >> >>>>> Can you provide a paste of how your arriving at your exception >> please? >> >>>>> Lewia >> >>>>> >> >>>>> On Wednesday, March 11, 2015, John Reynolds <[email protected]> >> >> wrote: >> >>>>> >> >>>>>> Should S3DataTransferer work with the generic file type from the >> get >> >> go >> >>>> or >> >>>>>> is there anything i need to change >> >>>>>> i was using the radix docker but getting 'Caused by: >> >>>>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not >> >>>> available’ >> >>>>>> i tried downloading the jce but no dice >> >>>>>> i might try just building on my local machine and trying again >> >>>>>> >> >>>>>> Thanks >> >>>>>> >> >>>>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney < >> >>>>>> [email protected] <javascript:;>> wrote: >> >>>>>>> >> >>>>>>> Hey John, >> >>>>>>> You're right, you've nearly answered everything here ;) >> >>>>>>> OK, so your properties can be located here >> >>>>>>> >> >>>>>> >> >>>> >> >> >> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124 >> >>>>>>> You'll see the usual key, value pairs in there which you can ad >> >> before >> >>>>>>> build the project and packaging filemgr.properties along with your >> >>>>>> compiled >> >>>>>>> code. >> >>>>>>> Regarding the following, >> >>>>>>> >> >>>>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds < >> [email protected] >> >>>>>> <javascript:;>> wrote: >> >>>>>>> >> >>>>>>>> >> >>>>>>>> in the near future, i will want to pass in a security token as >> well >> >> to >> >>>>>> the >> >>>>>>>> S3 client. to do so i would extend or modify the s3datatransfer >> >> class >> >>>>>>>> (which expires after an hour), what’s the best practice to read >> in >> >>>> this >> >>>>>>>> transient data from the command line (or wherever else i would >> >>>> initiate >> >>>>>> a >> >>>>>>>> job from) since this wouldnt go in a properties file >> >>>>>>> >> >>>>>>> >> >>>>>>> Do you mean that your security token expires hourly? As oppose to >> >> your >> >>>> s3 >> >>>>>>> client? >> >>>>>>> Lets try and defined exactly what you are after here. >> >>>>>>> Thanks >> >>>>>>> Lewis >> >>>>>> >> >>>>>> >> >>>>> >> >>>>> -- >> >>>>> *Lewis* >> >>>> >> >>>> >> >> >> >> >> >> >
