Hi Jarcec, I'm currently running Hadoop 2.0.2.1-alpha. So it should have that fix in theory.
-Kyle On Fri, May 3, 2013 at 2:41 PM, Jarek Jarcec Cecho <[email protected]>wrote: > Hi Kyle, > what Hadoop version are you using? > > Based on the exception it seems that you're using YARN which is suggesting > Hadoop 2. I hit similar NullPointerException during release candidate > testing on Hadoop 2.0.0-alpha. There is a known HDFS bug HADOOP-8726 [1] > that is causing this. It was fixed in 2.0.2-alpha, so I'm wondering if > you're using 2.0.0 or 2.0.1 by any chance? > > Jarcec > > Links: > 1: https://issues.apache.org/jira/browse/HADOOP-8726 > > On Fri, May 03, 2013 at 12:11:05PM -0700, Kyle B wrote: > > Hello, > > > > I am migrating from Sqoop 1.4.3 to Sqoop 2, and am kind of stumbling my > way > > through the intro stuff. I'd like to import data from a MySQL database > into > > HDFS, and a simple import seems to be failing in 2, which works fine in > 1. > > > > - From (1.4.3) - > > sqoop import --connect jdbc:mysql://127.0.0.1:3306/db --username kyle -P > > --table kyle_table --target-dir /user/kyle/table > > > > - To (1.99.2) - > > sqoop:000> create job --xid 1 --type import > > Creating job for connection with id 1 > > Please fill following values to create new job object > > Name: test > > > > Database configuration > > > > Schema name: > > Table name: kyle_table > > Table SQL statement: > > Table column names: * > > Partition column name: > > Boundary query: > > > > Output configuration > > > > Storage type: > > 0 : HDFS > > Choose: 0 > > Output format: > > 0 : TEXT_FILE > > 1 : SEQUENCE_FILE > > Choose: 0 > > Output directory: /user/kyle/table2 > > > > Throttling resources > > > > Extractors: > > Loaders: > > New job was successfully created with validation status FINE and > > persistent id 12 > > sqoop:000> submission start --jid 12 > > Submission details > > Job id: 12 > > Status: BOOTING > > Creation date: 2013-05-03 11:38:01 MST > > Last update date: 2013-05-03 11:38:01 MST > > External Id: job_1367275490217_0075 > > http://server:8088/proxy/application_1367275490217_0075/ > > Progress: Progress is not available > > sqoop:000> submission status --jid 12 > > Exception has occurred during processing command > > Server has returned exception: Exception: java.lang.NullPointerException > > Message: > > > > > > -- > > On both cases, I see the jobs made their way to the job history, and on > the > > first, the output was saved to HDFS. On the second, I just have a blank > > /user/kyle/table2 folder, and the task failed in Hadoop. > > > > > > - Hadoop Logs - > > 2013-05-03 11:38:10,172 WARN [main] org.apache.hadoop.mapred.YarnChild: > > Exception running child : java.lang.NullPointerException > > at java.lang.String.<init>(Unknown Source) > > at > > > org.apache.sqoop.job.mr.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:77) > > at > > > org.apache.sqoop.job.mr.ConfigurationUtils.getConnectorConnection(ConfigurationUtils.java:38) > > at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:69) > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333) > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Unknown Source) > > at > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367) > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) > > > > 2013-05-03 11:38:10,172 ERROR [OutputFormatLoader-consumer] > > org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: Error while > loading > > data out of MR job. > > java.lang.NullPointerException > > at java.lang.String.<init>(Unknown Source) > > at > > > org.apache.sqoop.job.mr.ConfigurationUtils.loadConfiguration(ConfigurationUtils.java:77) > > at > > > org.apache.sqoop.job.mr.ConfigurationUtils.getFrameworkConnection(ConfigurationUtils.java:50) > > at > > > org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:204) > > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > > at java.util.concurrent.FutureTask.run(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > > > > > - sqoop.log - > > 2013-05-03 11:38:01,439 DEBUG repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] > > Tx count-begin: 1, rollback: false > > 2013-05-03 11:38:01,452 DEBUG repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] > > Tx count-commit: 1, rollback: false > > 2013-05-03 11:38:01,452 DEBUG repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] > > Tx count-close: 0, rollback: false > > 2013-05-03 11:38:01,452 INFO repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] > > Attempting transaction commit > > 2013-05-03 11:38:01,453 DEBUG repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] > > Tx count-begin: 1, rollback: false > > 2013-05-03 11:38:01,464 DEBUG repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] > > Tx count-commit: 1, rollback: false > > 2013-05-03 11:38:01,464 DEBUG repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] > > Tx count-close: 0, rollback: false > > 2013-05-03 11:38:01,464 INFO repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] > > Attempting transaction commit > > 2013-05-03 11:38:01,465 DEBUG framework.FrameworkManager > > > [org.apache.sqoop.framework.FrameworkManager.submit(FrameworkManager.java:417)] > > Using callbacks: > > > Importer{initializer=org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer, > > destroyer=org.apache.sqoop.connector.jdbc.GenericJdbcImportDestroyer, > > partitioner=org.apache.sqoop.connector.jdbc.GenericJdbcImportPartitioner, > > extractor=org.apache.sqoop.connector.jdbc.GenericJdbcImportExtractor} > > 2013-05-03 11:38:01,482 DEBUG jdbc.GenericJdbcImportInitializer > > > [org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configurePartitionProperties(GenericJdbcImportInitializer.java:154)] > > Using minMaxQuery: SELECT MIN(id), MAX(id) FROM kyle_table > > 2013-05-03 11:38:01,483 INFO jdbc.GenericJdbcImportInitializer > > > [org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configurePartitionProperties(GenericJdbcImportInitializer.java:169)] > > Boundaries: min=1, max=4, columnType=4 > > 2013-05-03 11:38:01,483 INFO jdbc.GenericJdbcImportInitializer > > > [org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configureTableProperties(GenericJdbcImportInitializer.java:263)] > > Using dataSql: SELECT * FROM kyle_table WHERE ${CONDITIONS} > > 2013-05-03 11:38:01,483 INFO jdbc.GenericJdbcImportInitializer > > > [org.apache.sqoop.connector.jdbc.GenericJdbcImportInitializer.configureTableProperties(GenericJdbcImportInitializer.java:264)] > > Field names: * > > 2013-05-03 11:38:01,483 DEBUG repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] > > Tx count-begin: 1, rollback: false > > 2013-05-03 11:38:01,485 DEBUG repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] > > Tx count-commit: 1, rollback: false > > 2013-05-03 11:38:01,485 DEBUG repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] > > Tx count-close: 0, rollback: false > > 2013-05-03 11:38:01,485 INFO repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] > > Attempting transaction commit > > 2013-05-03 11:38:01,485 DEBUG mapreduce.MapreduceSubmissionEngine > > > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)] > > Adding jar to the job: > > > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-common-1.99.2.jar > > 2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine > > > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)] > > Adding jar to the job: > > > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-core-1.99.2.jar > > 2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine > > > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)] > > Adding jar to the job: > > > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-spi-1.99.2.jar > > 2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine > > > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)] > > Adding jar to the job: > > > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-execution-mapreduce-1.99.2-hadoop200.jar > > 2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine > > > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)] > > Adding jar to the job: > > > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-connector-generic-jdbc-1.99.2.jar > > 2013-05-03 11:38:01,486 DEBUG mapreduce.MapreduceSubmissionEngine > > > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)] > > Adding jar to the job: > > > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/json-simple-1.1.jar > > 2013-05-03 11:38:01,487 DEBUG mapreduce.MapreduceSubmissionEngine > > > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)] > > Adding jar to the job: > > > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/mysql-connector-java-5.1.18-bin.jar > > 2013-05-03 11:38:01,487 DEBUG mapreduce.MapreduceSubmissionEngine > > > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:211)] > > Adding jar to the job: > > > file:/home/hdfs/sqoop-1.99.2-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/guava-11.0.2.jar > > 2013-05-03 11:38:01,519 WARN mapreduce.JobSubmitter > > > [org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:138)] > > Hadoop command-line option parsing not performed. Implement the Tool > > interface and execute your application with ToolRunner to remedy this. > > 2013-05-03 11:38:01,913 WARN mapreduce.JobSubmitter > > > [org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:247)] > > No job jar file set. User classes may not be found. See Job or > > Job#setJar(String). > > 2013-05-03 11:38:01,944 DEBUG mr.SqoopInputFormat > > > [org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:74)] > > Partition: 1 <= id AND id < 2 > > 2013-05-03 11:38:01,944 DEBUG mr.SqoopInputFormat > > > [org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:74)] > > Partition: 2 <= id AND id < 3 > > 2013-05-03 11:38:01,945 DEBUG mr.SqoopInputFormat > > > [org.apache.sqoop.job.mr.SqoopInputFormat.getSplits(SqoopInputFormat.java:74)] > > Partition: 3 <= id AND id <= 4 > > 2013-05-03 11:38:02,010 WARN conf.Configuration > > > [org.apache.hadoop.conf.Configuration.warnOnceIfDeprecated(Configuration.java:816)] > > mapred.map.tasks.speculative.execution is deprecated. Instead, use > > mapreduce.map.speculative > > 2013-05-03 11:38:02,010 WARN conf.Configuration > > > [org.apache.hadoop.conf.Configuration.warnOnceIfDeprecated(Configuration.java:816)] > > mapred.reduce.tasks.speculative.execution is deprecated. Instead, use > > mapreduce.reduce.speculative > > 2013-05-03 11:38:02,126 DEBUG mapreduce.MapreduceSubmissionEngine > > > [org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:272)] > > Executed new map-reduce job with id job_1367275490217_0075 > > 2013-05-03 11:38:02,126 DEBUG repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] > > Tx count-begin: 1, rollback: false > > 2013-05-03 11:38:02,129 DEBUG repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] > > Tx count-commit: 1, rollback: false > > 2013-05-03 11:38:02,129 DEBUG repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] > > Tx count-close: 0, rollback: false > > 2013-05-03 11:38:02,129 INFO repository.JdbcRepositoryTransaction > > > [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] > > Attempting transaction commit > > > > > > > > -- > > Can anyone point out anything i'm missing? I'm following the quick start > > for creating a connection and first job. I would have thought the above > > would have dumped kyle_table to hdfs:/user/kyle/table2 . Am I using the > > right method to import a table to HDFS using Sqoop 2? > > > > Thanks, > > > > -Kyle >
