Re: Amazon S3 data transfer for filemanager
Thanks Michael! i didnt swap out openjdk for oracle version, i’ll try that next. > On Mar 13, 2015, at 9:10 AM, Michael Starch wrote: > > John, > > Did you remove the code at the top of the script that sets JAVA_EXT_JARS? > Are you running the oracle java still? > > If both are true, can you send me the latest error? > > -Michael > > > > On Thu, Mar 12, 2015 at 5:14 PM, John Reynolds wrote: > >> Thanks Michael, this works for me (filemanager starts), however i’m still >> getting the same datatransferer error >> >>> On Mar 12, 2015, at 1:52 PM, Michael Starch wrote: >>> >>> John, >>> >>> I should be more verbose. The java classpath traditionally did not pick >> up >>> multiple jars, so it was super labor intensive to setup. These days you >>> can use * inside the classpath to pick up multiple jarsbut it must be >>> in " " because otherwise the shell will glob the * if it is outside of >>> quotes. >>> >>> If this doesn't work, try the other recommendations in: >>> >>> >>> >> http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath >>> >>> -Michael >>> >>> >>> On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch >> wrote: >>> John, Change: -classpath "$FILEMGR_HOME"/lib \ To: -classpath "$FILEMGR_HOME/lib/*" \ -Michael On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds wrote: > Thanks Michael, if i modify the filemgr-client to look like this (at >> the > end) > "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \ > -classpath "$FILEMGR_HOME"/lib \ > > >> -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties > \ > >> -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \ > > >> -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml > \ > > >> -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml > \ > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“ > > (replacing ext jars with -classpath) then i get > Error: Could not find or load main class > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient > i assume i’m doing something wrong with classpath but not sure what > >> On Mar 12, 2015, at 11:31 AM, Michael Starch > wrote: >> >> John, >> >> Can you open filemgr-client sh script? It may set the JAVA_EXT_JARS >> there. If so, it is clobbering the default path for "extension" jars, > and >> your java encryption jars are not being picked up. If it does set >> JAVA_EXT_JARS you have two options: >> >> 1. Move all your encryption jars into FILEMGR_HOME/lib/ >> 2. update filemgr-client script to us classpath to specify the jars in > the >> FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS >> >> >> -Michael >> >> >> On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds > wrote: >> >>> Hi Michael >>> yeah it’s openjdk 1.7 (“1.7.0_75") >>> i did download the the unlimited encryption jar from oracle and > replaced >>> the local_policy / us_export_policy jars in javahome/jre/lib/security >>> more i read, maybe limited by jce.jar >>> >>> i dont have anything special set for extension jars >>> >>> >>> On Mar 12, 2015, at 10:35 AM, Michael Starch > wrote: John, What version of the JDK are you running, and what is your extension > jars environment variable set to. Do you have the java cryptology jar >>> included (Oracle JDK usually has this, I don't know if Open JDK does). "Algorithm HmacSHA1 not available" is usually thrown when Java >> cannot >>> find the java crypto jar used to calculate the given hash. -Michael On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds >> >>> wrote: > Hi Lewis, > using the latest docker buggtb/oodt image, which i assume is .8 > here’s the command i’m running to test the upload > > filemgr-client --url http://localhost:9000 --operation > --ingestProduct > --productName test --productStructure Flat --productTypeName > GenericFile > --metadataFile file:///root/test.txt.met --refs >> file:///root/test.txt > > i verified that i can upload to the path using the s3 tools on the > box / > with same credentials i put in the properties file > > here’s the full exception returned: > > >> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > Failed to upload product reference /root/test.txt to S3 at > usr/src/oodt/da
Re: Amazon S3 data transfer for filemanager
John, Did you remove the code at the top of the script that sets JAVA_EXT_JARS? Are you running the oracle java still? If both are true, can you send me the latest error? -Michael On Thu, Mar 12, 2015 at 5:14 PM, John Reynolds wrote: > Thanks Michael, this works for me (filemanager starts), however i’m still > getting the same datatransferer error > > > On Mar 12, 2015, at 1:52 PM, Michael Starch wrote: > > > > John, > > > > I should be more verbose. The java classpath traditionally did not pick > up > > multiple jars, so it was super labor intensive to setup. These days you > > can use * inside the classpath to pick up multiple jarsbut it must be > > in " " because otherwise the shell will glob the * if it is outside of > > quotes. > > > > If this doesn't work, try the other recommendations in: > > > > > > > http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath > > > > -Michael > > > > > > On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch > wrote: > > > >> John, > >> > >> Change: > >>-classpath "$FILEMGR_HOME"/lib \ > >> To: > >>-classpath "$FILEMGR_HOME/lib/*" \ > >> > >> -Michael > >> > >> > >> On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds > >> wrote: > >> > >>> Thanks Michael, if i modify the filemgr-client to look like this (at > the > >>> end) > >>> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \ > >>> -classpath "$FILEMGR_HOME"/lib \ > >>> > >>> > -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties > >>> \ > >>> > -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \ > >>> > >>> > -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml > >>> \ > >>> > >>> > -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml > >>> \ > >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“ > >>> > >>> (replacing ext jars with -classpath) then i get > >>> Error: Could not find or load main class > >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient > >>> i assume i’m doing something wrong with classpath but not sure what > >>> > On Mar 12, 2015, at 11:31 AM, Michael Starch > >>> wrote: > > John, > > Can you open filemgr-client sh script? It may set the JAVA_EXT_JARS > there. If so, it is clobbering the default path for "extension" jars, > >>> and > your java encryption jars are not being picked up. If it does set > JAVA_EXT_JARS you have two options: > > 1. Move all your encryption jars into FILEMGR_HOME/lib/ > 2. update filemgr-client script to us classpath to specify the jars in > >>> the > FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS > > > -Michael > > > On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds > >>> wrote: > > > Hi Michael > > yeah it’s openjdk 1.7 (“1.7.0_75") > > i did download the the unlimited encryption jar from oracle and > >>> replaced > > the local_policy / us_export_policy jars in javahome/jre/lib/security > > more i read, maybe limited by jce.jar > > > > i dont have anything special set for extension jars > > > > > > > >> On Mar 12, 2015, at 10:35 AM, Michael Starch > >>> wrote: > >> > >> John, > >> > >> What version of the JDK are you running, and what is your extension > >>> jars > >> environment variable set to. Do you have the java cryptology jar > > included > >> (Oracle JDK usually has this, I don't know if Open JDK does). > >> > >> "Algorithm HmacSHA1 not available" is usually thrown when Java > cannot > > find > >> the java crypto jar used to calculate the given hash. > >> > >> -Michael > >> > >> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds > > > wrote: > >> > >>> Hi Lewis, > >>> using the latest docker buggtb/oodt image, which i assume is .8 > >>> here’s the command i’m running to test the upload > >>> > >>> filemgr-client --url http://localhost:9000 --operation > >>> --ingestProduct > >>> --productName test --productStructure Flat --productTypeName > >>> GenericFile > >>> --metadataFile file:///root/test.txt.met --refs > file:///root/test.txt > >>> > >>> i verified that i can upload to the path using the s3 tools on the > >>> box / > >>> with same credentials i put in the properties file > >>> > >>> here’s the full exception returned: > >>> > >>> > rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >>> > org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >>> Failed to upload product reference /root/test.txt to S3 at > >>> usr/src/oodt/data/archive/test/test.txt > >>> at > >>> > > > >>> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) > >>> at sun.reflect.NativeMethodAccessorImpl.invoke0
Re: Amazon S3 data transfer for filemanager
Thanks Michael, this works for me (filemanager starts), however i’m still getting the same datatransferer error > On Mar 12, 2015, at 1:52 PM, Michael Starch wrote: > > John, > > I should be more verbose. The java classpath traditionally did not pick up > multiple jars, so it was super labor intensive to setup. These days you > can use * inside the classpath to pick up multiple jarsbut it must be > in " " because otherwise the shell will glob the * if it is outside of > quotes. > > If this doesn't work, try the other recommendations in: > > > http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath > > -Michael > > > On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch wrote: > >> John, >> >> Change: >>-classpath "$FILEMGR_HOME"/lib \ >> To: >>-classpath "$FILEMGR_HOME/lib/*" \ >> >> -Michael >> >> >> On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds >> wrote: >> >>> Thanks Michael, if i modify the filemgr-client to look like this (at the >>> end) >>> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \ >>> -classpath "$FILEMGR_HOME"/lib \ >>> >>> -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties >>> \ >>> -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \ >>> >>> -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml >>> \ >>> >>> -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml >>> \ >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“ >>> >>> (replacing ext jars with -classpath) then i get >>> Error: Could not find or load main class >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient >>> i assume i’m doing something wrong with classpath but not sure what >>> On Mar 12, 2015, at 11:31 AM, Michael Starch >>> wrote: John, Can you open filemgr-client sh script? It may set the JAVA_EXT_JARS there. If so, it is clobbering the default path for "extension" jars, >>> and your java encryption jars are not being picked up. If it does set JAVA_EXT_JARS you have two options: 1. Move all your encryption jars into FILEMGR_HOME/lib/ 2. update filemgr-client script to us classpath to specify the jars in >>> the FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS -Michael On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds >>> wrote: > Hi Michael > yeah it’s openjdk 1.7 (“1.7.0_75") > i did download the the unlimited encryption jar from oracle and >>> replaced > the local_policy / us_export_policy jars in javahome/jre/lib/security > more i read, maybe limited by jce.jar > > i dont have anything special set for extension jars > > > >> On Mar 12, 2015, at 10:35 AM, Michael Starch >>> wrote: >> >> John, >> >> What version of the JDK are you running, and what is your extension >>> jars >> environment variable set to. Do you have the java cryptology jar > included >> (Oracle JDK usually has this, I don't know if Open JDK does). >> >> "Algorithm HmacSHA1 not available" is usually thrown when Java cannot > find >> the java crypto jar used to calculate the given hash. >> >> -Michael >> >> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds > wrote: >> >>> Hi Lewis, >>> using the latest docker buggtb/oodt image, which i assume is .8 >>> here’s the command i’m running to test the upload >>> >>> filemgr-client --url http://localhost:9000 --operation >>> --ingestProduct >>> --productName test --productStructure Flat --productTypeName >>> GenericFile >>> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt >>> >>> i verified that i can upload to the path using the s3 tools on the >>> box / >>> with same credentials i put in the properties file >>> >>> here’s the full exception returned: >>> >>> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >>> Failed to upload product reference /root/test.txt to S3 at >>> usr/src/oodt/data/archive/test/test.txt >>> at >>> > >>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> > >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> > >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) >>> at >>> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) >>> at >>>
Re: Amazon S3 data transfer for filemanager
John, I should be more verbose. The java classpath traditionally did not pick up multiple jars, so it was super labor intensive to setup. These days you can use * inside the classpath to pick up multiple jarsbut it must be in " " because otherwise the shell will glob the * if it is outside of quotes. If this doesn't work, try the other recommendations in: http://stackoverflow.com/questions/219585/setting-multiple-jars-in-java-classpath -Michael On Thu, Mar 12, 2015 at 1:48 PM, Michael Starch wrote: > John, > > Change: > -classpath "$FILEMGR_HOME"/lib \ > To: > -classpath "$FILEMGR_HOME/lib/*" \ > > -Michael > > > On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds > wrote: > >> Thanks Michael, if i modify the filemgr-client to look like this (at the >> end) >> "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \ >> -classpath "$FILEMGR_HOME"/lib \ >> >> -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties >> \ >> -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \ >> >> -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml >> \ >> >> -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml >> \ >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“ >> >> (replacing ext jars with -classpath) then i get >> Error: Could not find or load main class >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient >> i assume i’m doing something wrong with classpath but not sure what >> >> > On Mar 12, 2015, at 11:31 AM, Michael Starch >> wrote: >> > >> > John, >> > >> > Can you open filemgr-client sh script? It may set the JAVA_EXT_JARS >> > there. If so, it is clobbering the default path for "extension" jars, >> and >> > your java encryption jars are not being picked up. If it does set >> > JAVA_EXT_JARS you have two options: >> > >> > 1. Move all your encryption jars into FILEMGR_HOME/lib/ >> > 2. update filemgr-client script to us classpath to specify the jars in >> the >> > FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS >> > >> > >> > -Michael >> > >> > >> > On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds >> wrote: >> > >> >> Hi Michael >> >> yeah it’s openjdk 1.7 (“1.7.0_75") >> >> i did download the the unlimited encryption jar from oracle and >> replaced >> >> the local_policy / us_export_policy jars in javahome/jre/lib/security >> >> more i read, maybe limited by jce.jar >> >> >> >> i dont have anything special set for extension jars >> >> >> >> >> >> >> >>> On Mar 12, 2015, at 10:35 AM, Michael Starch >> wrote: >> >>> >> >>> John, >> >>> >> >>> What version of the JDK are you running, and what is your extension >> jars >> >>> environment variable set to. Do you have the java cryptology jar >> >> included >> >>> (Oracle JDK usually has this, I don't know if Open JDK does). >> >>> >> >>> "Algorithm HmacSHA1 not available" is usually thrown when Java cannot >> >> find >> >>> the java crypto jar used to calculate the given hash. >> >>> >> >>> -Michael >> >>> >> >>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds >> >> wrote: >> >>> >> Hi Lewis, >> using the latest docker buggtb/oodt image, which i assume is .8 >> here’s the command i’m running to test the upload >> >> filemgr-client --url http://localhost:9000 --operation >> --ingestProduct >> --productName test --productStructure Flat --productTypeName >> GenericFile >> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt >> >> i verified that i can upload to the path using the s3 tools on the >> box / >> with same credentials i put in the properties file >> >> here’s the full exception returned: >> >> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >> Failed to upload product reference /root/test.txt to S3 at >> usr/src/oodt/data/archive/test/test.txt >> at >> >> >> >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) >> at >> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) >> at >> org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) >> at >> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) >> at >> org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) >> at >> org.apache.xmlrpc.WebServer$Connec
Re: Amazon S3 data transfer for filemanager
John, Change: -classpath "$FILEMGR_HOME"/lib \ To: -classpath "$FILEMGR_HOME/lib/*" \ -Michael On Thu, Mar 12, 2015 at 1:37 PM, John Reynolds wrote: > Thanks Michael, if i modify the filemgr-client to look like this (at the > end) > "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \ > -classpath "$FILEMGR_HOME"/lib \ > > -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties > \ > -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \ > > -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml > \ > > -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml > \ > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“ > > (replacing ext jars with -classpath) then i get > Error: Could not find or load main class > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient > i assume i’m doing something wrong with classpath but not sure what > > > On Mar 12, 2015, at 11:31 AM, Michael Starch wrote: > > > > John, > > > > Can you open filemgr-client sh script? It may set the JAVA_EXT_JARS > > there. If so, it is clobbering the default path for "extension" jars, > and > > your java encryption jars are not being picked up. If it does set > > JAVA_EXT_JARS you have two options: > > > > 1. Move all your encryption jars into FILEMGR_HOME/lib/ > > 2. update filemgr-client script to us classpath to specify the jars in > the > > FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS > > > > > > -Michael > > > > > > On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds > wrote: > > > >> Hi Michael > >> yeah it’s openjdk 1.7 (“1.7.0_75") > >> i did download the the unlimited encryption jar from oracle and replaced > >> the local_policy / us_export_policy jars in javahome/jre/lib/security > >> more i read, maybe limited by jce.jar > >> > >> i dont have anything special set for extension jars > >> > >> > >> > >>> On Mar 12, 2015, at 10:35 AM, Michael Starch > wrote: > >>> > >>> John, > >>> > >>> What version of the JDK are you running, and what is your extension > jars > >>> environment variable set to. Do you have the java cryptology jar > >> included > >>> (Oracle JDK usually has this, I don't know if Open JDK does). > >>> > >>> "Algorithm HmacSHA1 not available" is usually thrown when Java cannot > >> find > >>> the java crypto jar used to calculate the given hash. > >>> > >>> -Michael > >>> > >>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds > >> wrote: > >>> > Hi Lewis, > using the latest docker buggtb/oodt image, which i assume is .8 > here’s the command i’m running to test the upload > > filemgr-client --url http://localhost:9000 --operation > --ingestProduct > --productName test --productStructure Flat --productTypeName > GenericFile > --metadataFile file:///root/test.txt.met --refs file:///root/test.txt > > i verified that i can upload to the path using the s3 tools on the > box / > with same credentials i put in the properties file > > here’s the full exception returned: > > rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > Failed to upload product reference /root/test.txt to S3 at > usr/src/oodt/data/archive/test/test.txt > at > > >> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) > at > org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) > at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) > at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) > at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) > at > org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) > at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) > at java.lang.Thread.run(Thread.java:745) > Caused by: > org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > Failed to upload product reference /root/test.txt to S3 at > usr/src/oodt/data/archive/test/test.txt > at > > >> > org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78) > at > > >> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752) > ... 12 more > Caused by: com.ama
Re: Amazon S3 data transfer for filemanager
Thanks Michael, if i modify the filemgr-client to look like this (at the end) "$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \ -classpath "$FILEMGR_HOME"/lib \ -Dorg.apache.oodt.cas.filemgr.properties="$FILEMGR_HOME"/etc/filemgr.properties \ -Djava.util.logging.config.file="$FILEMGR_HOME"/etc/logging.properties \ -Dorg.apache.oodt.cas.cli.action.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-actions.xml \ -Dorg.apache.oodt.cas.cli.option.spring.config=file:"$FILEMGR_HOME"/policy/cmd-line-options.xml \ org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient "$@“ (replacing ext jars with -classpath) then i get Error: Could not find or load main class org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient i assume i’m doing something wrong with classpath but not sure what > On Mar 12, 2015, at 11:31 AM, Michael Starch wrote: > > John, > > Can you open filemgr-client sh script? It may set the JAVA_EXT_JARS > there. If so, it is clobbering the default path for "extension" jars, and > your java encryption jars are not being picked up. If it does set > JAVA_EXT_JARS you have two options: > > 1. Move all your encryption jars into FILEMGR_HOME/lib/ > 2. update filemgr-client script to us classpath to specify the jars in the > FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS > > > -Michael > > > On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds wrote: > >> Hi Michael >> yeah it’s openjdk 1.7 (“1.7.0_75") >> i did download the the unlimited encryption jar from oracle and replaced >> the local_policy / us_export_policy jars in javahome/jre/lib/security >> more i read, maybe limited by jce.jar >> >> i dont have anything special set for extension jars >> >> >> >>> On Mar 12, 2015, at 10:35 AM, Michael Starch wrote: >>> >>> John, >>> >>> What version of the JDK are you running, and what is your extension jars >>> environment variable set to. Do you have the java cryptology jar >> included >>> (Oracle JDK usually has this, I don't know if Open JDK does). >>> >>> "Algorithm HmacSHA1 not available" is usually thrown when Java cannot >> find >>> the java crypto jar used to calculate the given hash. >>> >>> -Michael >>> >>> On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds >> wrote: >>> Hi Lewis, using the latest docker buggtb/oodt image, which i assume is .8 here’s the command i’m running to test the upload filemgr-client --url http://localhost:9000 --operation --ingestProduct --productName test --productStructure Flat --productTypeName GenericFile --metadataFile file:///root/test.txt.met --refs file:///root/test.txt i verified that i can upload to the path using the s3 tools on the box / with same credentials i put in the properties file here’s the full exception returned: rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt at >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt at >> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78) at >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752) ... 12 more Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable to calculate a request signature: Algorithm HmacSHA1 not available at >> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71) at >> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57) at
Re: Amazon S3 data transfer for filemanager
John, Can you open filemgr-client sh script? It may set the JAVA_EXT_JARS there. If so, it is clobbering the default path for "extension" jars, and your java encryption jars are not being picked up. If it does set JAVA_EXT_JARS you have two options: 1. Move all your encryption jars into FILEMGR_HOME/lib/ 2. update filemgr-client script to us classpath to specify the jars in the FILEMGRHOME/lib directory and remove the use of JAVA_EXT_JARS -Michael On Thu, Mar 12, 2015 at 11:12 AM, John Reynolds wrote: > Hi Michael > yeah it’s openjdk 1.7 (“1.7.0_75") > i did download the the unlimited encryption jar from oracle and replaced > the local_policy / us_export_policy jars in javahome/jre/lib/security > more i read, maybe limited by jce.jar > > i dont have anything special set for extension jars > > > > > On Mar 12, 2015, at 10:35 AM, Michael Starch wrote: > > > > John, > > > > What version of the JDK are you running, and what is your extension jars > > environment variable set to. Do you have the java cryptology jar > included > > (Oracle JDK usually has this, I don't know if Open JDK does). > > > > "Algorithm HmacSHA1 not available" is usually thrown when Java cannot > find > > the java crypto jar used to calculate the given hash. > > > > -Michael > > > > On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds > wrote: > > > >> Hi Lewis, > >> using the latest docker buggtb/oodt image, which i assume is .8 > >> here’s the command i’m running to test the upload > >> > >> filemgr-client --url http://localhost:9000 --operation --ingestProduct > >> --productName test --productStructure Flat --productTypeName GenericFile > >> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt > >> > >> i verified that i can upload to the path using the s3 tools on the box / > >> with same credentials i put in the properties file > >> > >> here’s the full exception returned: > >> > >> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >> Failed to upload product reference /root/test.txt to S3 at > >> usr/src/oodt/data/archive/test/test.txt > >>at > >> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) > >>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>at > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >>at > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>at java.lang.reflect.Method.invoke(Method.java:606) > >>at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) > >>at > >> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) > >>at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) > >>at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) > >>at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) > >>at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) > >>at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) > >>at java.lang.Thread.run(Thread.java:745) > >> Caused by: > >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > >> Failed to upload product reference /root/test.txt to S3 at > >> usr/src/oodt/data/archive/test/test.txt > >>at > >> > org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78) > >>at > >> > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752) > >>... 12 more > >> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a > >> request signature: Unable to calculate a request signature: Algorithm > >> HmacSHA1 not available > >>at > >> > com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71) > >>at > >> > com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57) > >>at > >> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128) > >>at > >> > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330) > >>at > >> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > >>at > >> > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > >>at > >> > com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) > >>at > >> > org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76) > >>... 13 more > >> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a > >> request signature: Algorithm HmacSHA1 not available > >>at > >> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90) > >>at > >> > com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(Abstract
Re: Amazon S3 data transfer for filemanager
Hi Michael yeah it’s openjdk 1.7 (“1.7.0_75") i did download the the unlimited encryption jar from oracle and replaced the local_policy / us_export_policy jars in javahome/jre/lib/security more i read, maybe limited by jce.jar i dont have anything special set for extension jars > On Mar 12, 2015, at 10:35 AM, Michael Starch wrote: > > John, > > What version of the JDK are you running, and what is your extension jars > environment variable set to. Do you have the java cryptology jar included > (Oracle JDK usually has this, I don't know if Open JDK does). > > "Algorithm HmacSHA1 not available" is usually thrown when Java cannot find > the java crypto jar used to calculate the given hash. > > -Michael > > On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds wrote: > >> Hi Lewis, >> using the latest docker buggtb/oodt image, which i assume is .8 >> here’s the command i’m running to test the upload >> >> filemgr-client --url http://localhost:9000 --operation --ingestProduct >> --productName test --productStructure Flat --productTypeName GenericFile >> --metadataFile file:///root/test.txt.met --refs file:///root/test.txt >> >> i verified that i can upload to the path using the s3 tools on the box / >> with same credentials i put in the properties file >> >> here’s the full exception returned: >> >> rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >> Failed to upload product reference /root/test.txt to S3 at >> usr/src/oodt/data/archive/test/test.txt >>at >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) >>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>at java.lang.reflect.Method.invoke(Method.java:606) >>at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) >>at >> org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) >>at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) >>at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) >>at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) >>at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) >>at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) >>at java.lang.Thread.run(Thread.java:745) >> Caused by: >> org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >> Failed to upload product reference /root/test.txt to S3 at >> usr/src/oodt/data/archive/test/test.txt >>at >> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78) >>at >> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752) >>... 12 more >> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a >> request signature: Unable to calculate a request signature: Algorithm >> HmacSHA1 not available >>at >> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71) >>at >> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57) >>at >> com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128) >>at >> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330) >>at >> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) >>at >> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) >>at >> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) >>at >> org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76) >>... 13 more >> Caused by: com.amazonaws.AmazonClientException: Unable to calculate a >> request signature: Algorithm HmacSHA1 not available >>at >> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90) >>at >> com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68) >>... 20 more >> Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not >> available >>at javax.crypto.Mac.getInstance(Mac.java:176) >>at >> com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86) >>... 21 more >> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: >> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error >> ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1] >> : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: >> Failed to upload product reference /root/test.txt to S3 at >> usr/src/oodt/data/archive/test/test.txt >>at >> org.apache.xmlrpc.Xm
Re: Amazon S3 data transfer for filemanager
John, What version of the JDK are you running, and what is your extension jars environment variable set to. Do you have the java cryptology jar included (Oracle JDK usually has this, I don't know if Open JDK does). "Algorithm HmacSHA1 not available" is usually thrown when Java cannot find the java crypto jar used to calculate the given hash. -Michael On Thu, Mar 12, 2015 at 9:06 AM, John Reynolds wrote: > Hi Lewis, > using the latest docker buggtb/oodt image, which i assume is .8 > here’s the command i’m running to test the upload > > filemgr-client --url http://localhost:9000 --operation --ingestProduct > --productName test --productStructure Flat --productTypeName GenericFile > --metadataFile file:///root/test.txt.met --refs file:///root/test.txt > > i verified that i can upload to the path using the s3 tools on the box / > with same credentials i put in the properties file > > here’s the full exception returned: > > rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > Failed to upload product reference /root/test.txt to S3 at > usr/src/oodt/data/archive/test/test.txt > at > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) > at > org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) > at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) > at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) > at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) > at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) > at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) > at java.lang.Thread.run(Thread.java:745) > Caused by: > org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > Failed to upload product reference /root/test.txt to S3 at > usr/src/oodt/data/archive/test/test.txt > at > org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78) > at > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752) > ... 12 more > Caused by: com.amazonaws.AmazonClientException: Unable to calculate a > request signature: Unable to calculate a request signature: Algorithm > HmacSHA1 not available > at > com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71) > at > com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57) > at > com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128) > at > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330) > at > com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > at > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > at > com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) > at > org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76) > ... 13 more > Caused by: com.amazonaws.AmazonClientException: Unable to calculate a > request signature: Algorithm HmacSHA1 not available > at > com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90) > at > com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68) > ... 20 more > Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not > available > at javax.crypto.Mac.getInstance(Mac.java:176) > at > com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86) > ... 21 more > org.apache.xmlrpc.XmlRpcException: java.lang.Exception: > org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error > ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1] > : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: > Failed to upload product reference /root/test.txt to S3 at > usr/src/oodt/data/archive/test/test.txt > at > org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104) > at > org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71) > at > org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73) > at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) > at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcCli
Re: Amazon S3 data transfer for filemanager
Hi Lewis, using the latest docker buggtb/oodt image, which i assume is .8 here’s the command i’m running to test the upload filemgr-client --url http://localhost:9000 --operation --ingestProduct --productName test --productStructure Flat --productTypeName GenericFile --metadataFile file:///root/test.txt.met --refs file:///root/test.txt i verified that i can upload to the path using the s3 tools on the box / with same credentials i put in the properties file here’s the full exception returned: rg.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:768) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt at org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:78) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:752) ... 12 more Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable to calculate a request signature: Algorithm HmacSHA1 not available at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71) at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:57) at com.amazonaws.services.s3.internal.S3Signer.sign(S3Signer.java:128) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:330) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) at org.apache.oodt.cas.filemgr.datatransfer.S3DataTransferer.transferProduct(S3DataTransferer.java:76) ... 13 more Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Algorithm HmacSHA1 not available at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90) at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68) ... 20 more Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not available at javax.crypto.Mac.getInstance(Mac.java:176) at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:86) ... 21 more org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@6454bbe1] : org.apache.oodt.cas.filemgr.structs.exceptions.DataTransferException: Failed to upload product reference /root/test.txt to S3 at usr/src/oodt/data/archive/test/test.txt at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104) at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71) at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198) at org.apache.oodt.cas.filemgr.cli.action.IngestProductCliAction.execute(IngestProductCliAction.java:112) at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.
Re: Amazon S3 data transfer for filemanager
Hi John, What beau on of OODT are you using? The file types should be encapsulated by the factory implementation I would guess. It should be. Case of mike this the default filemgr store then off you go. Can you provide a paste of how your arriving at your exception please? Lewia On Wednesday, March 11, 2015, John Reynolds wrote: > Should S3DataTransferer work with the generic file type from the get go or > is there anything i need to change > i was using the radix docker but getting 'Caused by: > java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not available’ > i tried downloading the jce but no dice > i might try just building on my local machine and trying again > > Thanks > > > On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney < > lewis.mcgibb...@gmail.com > wrote: > > > > Hey John, > > You're right, you've nearly answered everything here ;) > > OK, so your properties can be located here > > > https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124 > > You'll see the usual key, value pairs in there which you can ad before > > build the project and packaging filemgr.properties along with your > compiled > > code. > > Regarding the following, > > > > On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds > wrote: > > > >> > >> in the near future, i will want to pass in a security token as well to > the > >> S3 client. to do so i would extend or modify the s3datatransfer class > >> (which expires after an hour), what’s the best practice to read in this > >> transient data from the command line (or wherever else i would initiate > a > >> job from) since this wouldnt go in a properties file > > > > > > Do you mean that your security token expires hourly? As oppose to your s3 > > client? > > Lets try and defined exactly what you are after here. > > Thanks > > Lewis > > -- *Lewis*
Re: Amazon S3 data transfer for filemanager
Should S3DataTransferer work with the generic file type from the get go or is there anything i need to change i was using the radix docker but getting 'Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacSHA1 not available’ i tried downloading the jce but no dice i might try just building on my local machine and trying again Thanks > On Mar 10, 2015, at 6:19 PM, Lewis John Mcgibbney > wrote: > > Hey John, > You're right, you've nearly answered everything here ;) > OK, so your properties can be located here > https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124 > You'll see the usual key, value pairs in there which you can ad before > build the project and packaging filemgr.properties along with your compiled > code. > Regarding the following, > > On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds wrote: > >> >> in the near future, i will want to pass in a security token as well to the >> S3 client. to do so i would extend or modify the s3datatransfer class >> (which expires after an hour), what’s the best practice to read in this >> transient data from the command line (or wherever else i would initiate a >> job from) since this wouldnt go in a properties file > > > Do you mean that your security token expires hourly? As oppose to your s3 > client? > Lets try and defined exactly what you are after here. > Thanks > Lewis
Re: Amazon S3 data transfer for filemanager
Hey John, You're right, you've nearly answered everything here ;) OK, so your properties can be located here https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/filemgr.properties#L114-L124 You'll see the usual key, value pairs in there which you can ad before build the project and packaging filemgr.properties along with your compiled code. Regarding the following, On Tue, Mar 10, 2015 at 4:31 PM, John Reynolds wrote: > > in the near future, i will want to pass in a security token as well to the > S3 client. to do so i would extend or modify the s3datatransfer class > (which expires after an hour), what’s the best practice to read in this > transient data from the command line (or wherever else i would initiate a > job from) since this wouldnt go in a properties file Do you mean that your security token expires hourly? As oppose to your s3 client? Lets try and defined exactly what you are after here. Thanks Lewis
Re: Amazon S3 data transfer for filemanager
Ok, i think i’m exploring / answering my own questions :) I found the implementation of S3DataTransferer / Factory and it looks like what i need to start in the near future, i will want to pass in a security token as well to the S3 client. to do so i would extend or modify the s3datatransfer class (which expires after an hour), what’s the best practice to read in this transient data from the command line (or wherever else i would initiate a job from) since this wouldnt go in a properties file > On Mar 10, 2015, at 3:48 PM, John Reynolds wrote: > > Hi, > i’m new to the terminology of OODT (and OODT) so please bear with me. > i’m looking to do a couple things with OODT and integration with AWS, and > wanted to see if anyone has done this already. > (I searched through the mail archive, and the wiki, and havent found quite > what i’m looking for, other than some references to an S3 Datatransferer, > that may or may not be complete) > > So, what i’m looking to do: > 1. ingest files into oodt file manager with S3 as the source > 2. store files in S3 that have been ingested by file manager (from an > arbitrary source) > > what would be the general strategy for doing each of those things (what > classes need to be implemented / extended) > or if someone has done this already, let me know > > Thanks! > John Reynolds > >
Amazon S3 data transfer for filemanager
Hi, i’m new to the terminology of OODT (and OODT) so please bear with me. i’m looking to do a couple things with OODT and integration with AWS, and wanted to see if anyone has done this already. (I searched through the mail archive, and the wiki, and havent found quite what i’m looking for, other than some references to an S3 Datatransferer, that may or may not be complete) So, what i’m looking to do: 1. ingest files into oodt file manager with S3 as the source 2. store files in S3 that have been ingested by file manager (from an arbitrary source) what would be the general strategy for doing each of those things (what classes need to be implemented / extended) or if someone has done this already, let me know Thanks! John Reynolds
