John, Did you remove the code at the top of the script that sets JAVA_EXT_JARS? Are you running the oracle java still?
If both are true, can you send me the latest error? -Michael On Thu, Mar 12, 2015 at 5:14 PM, John Reynolds <jreyno...@vpicu.net> wrote: > Thanks Michael, this works for me (filemanager starts), however i’m still > getting the same datatransferer error > > > On Mar 12, 2015, at 1:52 PM, Michael Starch <starc...@umich.edu> wrote: > > > > John, > > > > I should be more verbose. The java classpath traditionally did not pick > up > > multiple jars, so it was super labor intensive to setup. These days you > > can use * inside the classpath to pick up multiple jars....but it must be > > in " " because otherwise the shell will glob the * if it is outside of > > quotes. > > > > If this doesn't work, try the other recommendations in: > > > > > > > http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath > > > > -Michael > > > > > > On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch <starc...@umich.edu> > wrote: > > > >> John, > >> > >> Change: > >> -classpath "$FILEMGR_HOME"/lib \ > >> To: > >> -classpath "$FILEMGR_HOME/lib/*" \ > >> > >> -Michael > >> > >> > >> On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds <jreyno...@vpicu.net> > >> wrote: > >> > >>> Thanks Michael, if i modify the filemgr-client to look like this (at > the > >>> end) > >>> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \ > >>> -classpath "$FILEMGR_HOME"/lib \ > >>> > >>> > -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties > >>> \ > >>> > -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \ > >>> > >>> > -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml > >>> \ > >>> > >>> > -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml > >>> \ > >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“ > >>> > >>> (replacing ext jars with -classpath) then i get > >>> Error: Could not find or load main class > >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient > >>> i assume i’m doing something wrong with classpath but not sure what > >>> > >>>> On Mar 12, 2015, at 11:31 AM, Michael Starch <starc...@umich.edu> > >>> wrote: > >>>> > >>>> John, > >>>> > >>>> Can you open filemgr-client sh script? It may set the JAVA_EXT_JARS > >>>> there. If so, it is clobbering the default path for "extension" jars, > >>> and > >>>> your java encryption jars are not being picked up. If it does set > >>>> JAVA_EXT_JARS you have two options: > >>>> > >>>> 1. Move all your encryption jars into FILEMGR_HOME/lib/ > >>>> 2. update filemgr-client script to us classpath to specify the jars in > >>> the > >>>> FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS > >>>> > >>>> > >>>> -Michael > >>>> > >>>> > >>>> On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds <jreyno...@vpicu.net> > >>> wrote: > >>>> > >>>>> Hi Michael > >>>>> yeah it’s openjdk 1.7 (“1.7.0_75") > >>>>> i did download the the unlimited encryption jar from oracle and > >>> replaced > >>>>> the local_policy / us_export_policy jars in javahome/jre/lib/security > >>>>> more i read, maybe limited by jce.jar > >>>>> > >>>>> i dont have anything special set for extension jars > >>>>> > >>>>> > >>>>> > >>>>>> On Mar 12, 2015, at 10:35 AM, Michael Starch <starc...@umich.edu> > >>> wrote: > >>>>>> > >>>>>> John, > >>>>>> > >>>>>> What version of the JDK are you running, and what is your extension > >>> jars > >>>>>> environment variable set to. Do you have the java cryptology jar > >>>>> included > >>>>>> (Oracle JDK usually has this, I don't know if Open JDK does). > >>>>>> > >>>>>> "Algorithm HmacSHA1 not available" is usually thrown when Java > cannot > >>>>> find > >>>>>> the java crypto jar used to calculate the given hash. > >>>>>> > >>>>>> -Michael > >>>>>> > >>>>>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds <jreyno...@vpicu.net > > > >>>>> wrote: > >>>>>> > >>>>>>> Hi Lewis, > >>>>>>> using the latest docker buggtb/oodt image, which i assume is .8 > >>>>>>> here’s the command i’m running to test the upload > >>>>>>> > >>>>>>> filemgr-client --url http://localhost:9000 --operation > >>> --ingestProduct > >>>>>>> --productName test --productStructure Flat --productTypeName > >>> GenericFile > >>>>>>> --metadataFile file:///root/test.txt.met --refs > file:///root/test.txt > >>>>>>> > >>>>>>> i verified that i can upload to the path using the s3 tools on the > >>> box / > >>>>>>> with same credentials i put in the properties file > >>>>>>> > >>>>>>> here’s the full exception returned: > >>>>>>> > >>>>>>> > rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >>>>>>> > org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >>>>>>> Failed to upload product reference /root/test.txt to S3 at > >>>>>>> usr/src/oodt/data/archive/test/test.txt > >>>>>>> at > >>>>>>> > >>>>> > >>> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) > >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>>>>>> at > >>>>>>> > >>>>> > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >>>>>>> at > >>>>>>> > >>>>> > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) > >>>>>>> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) > >>>>>>> at > >>>>>>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) > >>>>>>> at > >>> org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) > >>>>>>> at > >>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) > >>>>>>> at > >>> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) > >>>>>>> at > >>> org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) > >>>>>>> at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) > >>>>>>> at java.lang.Thread.run(Thread.java:745) > >>>>>>> Caused by: > >>>>>>> > org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >>>>>>> Failed to upload product reference /root/test.txt to S3 at > >>>>>>> usr/src/oodt/data/archive/test/test.txt > >>>>>>> at > >>>>>>> > >>>>> > >>> > org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78) > >>>>>>> at > >>>>>>> > >>>>> > >>> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752) > >>>>>>> ... 12 more > >>>>>>> Caused by: com.amazonaws.AmazonClientException: Unable to > calculate a > >>>>>>> request signature: Unable to calculate a request signature: > Algorithm > >>>>>>> HmacSHA1 not available > >>>>>>> at > >>>>>>> > >>>>> > >>> > com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71) > >>>>>>> at > >>>>>>> > >>>>> > >>> > com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57) > >>>>>>> at > >>>>>>> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128) > >>>>>>> at > >>>>>>> > >>>>> > >>> > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330) > >>>>>>> at > >>>>>>> > >>> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > >>>>>>> at > >>>>>>> > >>>>> > >>> > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > >>>>>>> at > >>>>>>> > >>>>> > >>> > com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) > >>>>>>> at > >>>>>>> > >>>>> > >>> > org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76) > >>>>>>> ... 13 more > >>>>>>> Caused by: com.amazonaws.AmazonClientException: Unable to > calculate a > >>>>>>> request signature: Algorithm HmacSHA1 not available > >>>>>>> at > >>>>>>> > com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90) > >>>>>>> at > >>>>>>> > >>>>> > >>> > com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68) > >>>>>>> ... 20 more > >>>>>>> Caused by: java.security.NoSuchAlgorithmException: Algorithm > HmacSHA1 > >>>>> not > >>>>>>> available > >>>>>>> at javax.crypto.Mac.getInstance(Mac.java:176) > >>>>>>> at > >>>>>>> > com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86) > >>>>>>> ... 21 more > >>>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: > >>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: > >>> Error > >>>>>>> ingesting product > >>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 > >>>>> ] > >>>>>>> : > >>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >>>>>>> Failed to upload product reference /root/test.txt to S3 at > >>>>>>> usr/src/oodt/data/archive/test/test.txt > >>>>>>> at > >>>>>>> > >>>>> > >>> > org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104) > >>>>>>> at > >>>>>>> > >>>>> > >>> > org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71) > >>>>>>> at > >>>>>>> > >>> > org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73) > >>>>>>> at > >>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) > >>>>>>> at > >>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185) > >>>>>>> at > >>> org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178) > >>>>>>> at > >>>>>>> > >>>>> > >>> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198) > >>>>>>> at > >>>>>>> > >>>>> > >>> > org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) > >>>>>>> at > >>>>>>> > >>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) > >>>>>>> at > >>>>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) > >>>>>>> at > >>>>>>> > >>>>> > >>> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) > >>>>>>> java.lang.Exception: Failed to ingest product > >>>>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : > >>>>>>> java.lang.Exception: > >>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: > >>> Error > >>>>>>> ingesting product > >>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 > >>>>> ] > >>>>>>> : > >>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >>>>>>> Failed to upload product reference /root/test.txt to S3 at > >>>>>>> usr/src/oodt/data/archive/test/test.txt > >>>>>>> at > >>>>>>> > >>>>> > >>> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303) > >>>>>>> at > >>>>>>> > >>>>> > >>> > org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) > >>>>>>> at > >>>>>>> > >>> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) > >>>>>>> at > >>>>>>> org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) > >>>>>>> at > >>>>>>> > >>>>> > >>> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1350) > >>>>>>> ERROR: Failed to ingest product 'test' : Failed to ingest product > >>>>>>> [org.apache.oodt.cas.filemgr.structs.Product@7f25ab41] : > >>>>>>> java.lang.Exception: > >>>>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: > >>> Error > >>>>>>> ingesting product > >>> [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1 > >>>>> ] > >>>>>>> : > >>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >>>>>>> Failed to upload product reference /root/test.txt to S3 at > >>>>>>> usr/src/oodt/data/archive/test/test.txt > >>>>>>> > >>>>>>> > >>>>>>>> On Mar 11, 2015, at 6:21 PM, Lewis John Mcgibbney < > >>>>>>> lewis.mcgibb...@gmail.com> wrote: > >>>>>>>> > >>>>>>>> Hi John, > >>>>>>>> What beau on of OODT are you using? > >>>>>>>> The file types should be encapsulated by the factory > implementation > >>> I > >>>>>>> would > >>>>>>>> guess. It should be. Case of mike this the default filemgr store > >>> then > >>>>> off > >>>>>>>> you go. > >>>>>>>> Can you provide a paste of how your arriving at your exception > >>> please? > >>>>>>>> Lewia > >>>>>>>> > >>>>>>>> On Wednesday, March 11, 2015, John Reynolds <jreyno...@vpicu.net> > >>>>> wrote: > >>>>>>>> > >>>>>>>>> Should S3DataTransferer work with the generic file type from the > >>> get > >>>>> go > >>>>>>> or > >>>>>>>>> is there anything i need to change > >>>>>>>>> i was using the radix docker but getting 'Caused by: > >>>>>>>>> java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not > >>>>>>> available’ > >>>>>>>>> i tried downloading the jce but no dice > >>>>>>>>> i might try just building on my local machine and trying again > >>>>>>>>> > >>>>>>>>> Thanks > >>>>>>>>> > >>>>>>>>>> On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney < > >>>>>>>>> lewis.mcgibb...@gmail.com <javascript:;>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hey John, > >>>>>>>>>> You're right, you've nearly answered everything here ;) > >>>>>>>>>> OK, so your properties can be located here > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>> > >>> > https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124 > >>>>>>>>>> You'll see the usual key, value pairs in there which you can ad > >>>>> before > >>>>>>>>>> build the project and packaging filemgr.properties along with > your > >>>>>>>>> compiled > >>>>>>>>>> code. > >>>>>>>>>> Regarding the following, > >>>>>>>>>> > >>>>>>>>>> On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds < > >>> jreyno...@vpicu.net > >>>>>>>>> <javascript:;>> wrote: > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> in the near future, i will want to pass in a security token as > >>> well > >>>>> to > >>>>>>>>> the > >>>>>>>>>>> S3 client. to do so i would extend or modify the s3datatransfer > >>>>> class > >>>>>>>>>>> (which expires after an hour), what’s the best practice to read > >>> in > >>>>>>> this > >>>>>>>>>>> transient data from the command line (or wherever else i would > >>>>>>> initiate > >>>>>>>>> a > >>>>>>>>>>> job from) since this wouldnt go in a properties file > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Do you mean that your security token expires hourly? As oppose > to > >>>>> your > >>>>>>> s3 > >>>>>>>>>> client? > >>>>>>>>>> Lets try and defined exactly what you are after here. > >>>>>>>>>> Thanks > >>>>>>>>>> Lewis > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> *Lewis* > >>>>>>> > >>>>>>> > >>>>> > >>>>> > >>> > >>> > >> > >
