can we use -libjars with the Sqoop tool?
Yes. -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. ________________________________ From: Jane Wayne <[email protected]> To: [email protected] Sent: Thursday, June 6, 2013 5:56 AM Subject: Re: can we use -libjars with the Sqoop tool? i am running the Sqoop tool as follows. export LIBJARS=sqoop-1.4.3.jar,mysql-connector-java-5.1.25-bin.jar,ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar org.apache.sqoop.Sqoop import -libjars ${LIBJARS} --connect jdbc:mysql://mysql/mydb --username myuser --password mypass --table data --verbose --append --warehouse-dir /sqoop i am getting this stack trace. java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.sqoop.mapreduce.RawKeyTextOutputFormat at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996) at org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248) at org.apache.hadoop.mapred.Task.initialize(Task.java:501) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: java.lang.ClassNotFoundException: org.apache.sqoop.mapreduce.RawKeyTextOutputFormat at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) when i looked at the sqoop-1.4.3.jar, clearly, org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.class exists. any ideas what's going on? On Thu, Jun 6, 2013 at 1:54 AM, Venkat Ranganathan <[email protected]> wrote: sqoop tools implement the ToolRunner interface. So, you can pass -libjars to sqoop (but make sure it is the first option among the list of the options > > >For example, sqoop import -libjars $LIBJARS other options > >Venkat > > > >On Wed, Jun 5, 2013 at 5:23 PM, Jane Wayne <[email protected]> wrote: > >hi, >> >> >>i am wondering if i can run the org.apache.sqoop.Sqoop tool like other tools >>from the command line using the -libjars flag/option/parameter? >> >> >>i have tried the following. >> >> >>export >>LIBJARS=ant-contrib-1.0b3.jar,ant-eclipse-1.0-jvm1.2.jar,avro-1.5.3.jar,avro-ipc-1.5.3.jar,avro-mapred-1.5.3.jar,commons-io-1.4.jar,hsqldb-1.8.0.10.jar,ivy-2.1.0.jar,jackson-core-asl-1.7.3.jar,jackson-mapper-asl-1.7.3.jar,jopt-simple-3.2.jar,paranamer-2.3.jar,snappy-java-1.0.3.2.jar >> >> >>hadoop --config path/to/sqoop-cluster-config-dir jar sqoop-1.4.3.jar >>org.apache.sqoop.Sqoop -libjars ${LIBJARS} import --connect >>jdbc:mysql://mysql/mydb --username myuser --password mypass --table data >>--verbose --append --warehouse-dir /sqoop >> >> >>what i get is the message below. >> >> >>No such sqoop tool: -libjars. See 'sqoop help'. >> >> >> >>obviously, this problem has something to do with sqoop's parsing of the >>command line options. shouldn't sqoop allow hadoop's command line options and >>its own at the same time? >> >> >>in the meanwhile, i will try to tinker a bit with that Sqoop class to see if >>i can get it to work with -libjars. >> >> >>references that i found regarding -libjars. >>1. http://grepalex.com/2013/02/25/hadoop-libjars/ >>2. >>http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/ >> >> >>thanks, >
