Hey man, It looks like certain Hive jars are missing from the job for some reason. Seems like we need to add more jars to https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteFromInitializer.java#L71 and https://github.com/apache/sqoop/blob/sqoop2/connector/connector-kite/src/main/java/org/apache/sqoop/connector/kite/KiteToInitializer.java#L78 .
I've created https://issues.apache.org/jira/browse/SQOOP-2489 to track this bug. I've also created https://issues.apache.org/jira/browse/SQOOP-2490 to provide a facility to work around these kinds of issues in the future. Sqoop2 is a work in progress and still needs some battle testing. With that in mind, can you use the Avro integration instead? Otherwise, you might need to rebuild Sqoop2 with the fix to get this working at the moment. Again, https://issues.apache.org/jira/browse/SQOOP-2490 will change all of that. -Abe On Fri, Aug 14, 2015 at 1:10 AM, Lee S <[email protected]> wrote: > Hi all: > I try to import from rdms to hive with kite connector in sqoop-shell. > And I submit the job successfully , but when I try to track the job staus > on yarn web-ui, I found there are errors in container log as below. > And the job continues running ,never stops. > The log shows *java.lang.NoClassDefFoundError: > org/apache/hadoop/hive/ql/io/HiveOutputFormat.* > > *Anybody can help?* > > *Showing 4096 bytes. Click here > <http://pdm-03:8042/node/containerlogs/container_1439241066552_0044_01_000002/root/stderr/?start=0> > for > full log* > > *utFormatLoadExecutor - SqoopOutputFormatLoadExecutor consumer thread is > starting > 2015-08-14 15:39:10,765 [OutputFormatLoader-consumer] INFO > org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Running loader class > org.apache.sqoop.connector.kite.KiteLoader > 2015-08-14 15:39:10,771 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - > Starting progress service > 2015-08-14 15:39:10,772 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - > Running extractor class org.apache.sqoop.connector.jdbc.GenericJdbcExtractor > 2015-08-14 15:39:10,981 [OutputFormatLoader-consumer] INFO > org.apache.sqoop.connector.kite.KiteLoader - Constructed temporary dataset > URI: > dataset:hive:wangjun/temp_5bfec97713e04374b2f2efde2dc5e4de?auth:host=pdm-03&auth:port=9083 > 2015-08-14 15:39:11,093 [main] INFO > org.apache.sqoop.connector.jdbc.GenericJdbcExtractor - Using query: SELECT > id FROM bcpdm.history WHERE 1368 <= id AND id <= 1546 > 2015-08-14 15:39:11,537 [OutputFormatLoader-consumer] ERROR > org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - Error while loading > data out of MR job. > java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/io/HiveOutputFormat > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:190) > at > org.kitesdk.data.spi.hive.HiveUtils.getHiveParquetOutputFormat(HiveUtils.java:446) > at org.kitesdk.data.spi.hive.HiveUtils.<clinit>(HiveUtils.java:91) > at > org.kitesdk.data.spi.hive.HiveManagedMetadataProvider.create(HiveManagedMetadataProvider.java:83) > at > org.kitesdk.data.spi.hive.HiveManagedDatasetRepository.create(HiveManagedDatasetRepository.java:77) > at org.kitesdk.data.Datasets.create(Datasets.java:239) > at org.kitesdk.data.Datasets.create(Datasets.java:307) > at org.kitesdk.data.Datasets.create(Datasets.java:335) > at > org.apache.sqoop.connector.kite.KiteDatasetExecutor.createDataset(KiteDatasetExecutor.java:70) > at > org.apache.sqoop.connector.kite.KiteLoader.getExecutor(KiteLoader.java:52) > at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:62) > at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:36) > at > org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:250) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.ql.io.HiveOutputFormat > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > ... 31 more > 2015-08-14 15:39:11,540 [main] INFO org.apache.sqoop.job.mr.SqoopMapper - > Stopping progress service > 2015-08-14 15:39:11,540 [main] INFO > org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor - > SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed* > > > p.s. sqoop version: 1.99.6 > hadoop version 2.6.0 >
