Thanks Abe, This is the output: <snip> Warning: /usr/lib/hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/hadoop-2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-flume-1.4.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 14/06/16 08:54:19 DEBUG tool.BaseSqoopTool: Enabled debug logging. 14/06/16 08:54:19 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 14/06/16 08:54:19 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory 14/06/16 08:54:19 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory 14/06/16 08:54:19 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:mysql: 14/06/16 08:54:19 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 14/06/16 08:54:19 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.MySQLManager@1537e1d 14/06/16 08:54:19 INFO tool.CodeGenTool: Beginning code generation 14/06/16 08:54:19 DEBUG manager.SqlManager: Execute getColumnTypesRawQuery : SELECT t.* FROM `test1` AS t LIMIT 1 14/06/16 08:54:19 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection. 14/06/16 08:54:24 DEBUG manager.SqlManager: Using fetchSize for next query: -2147483648 14/06/16 08:54:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test1` AS t LIMIT 1 14/06/16 08:54:24 DEBUG orm.ClassWriter: selected columns: 14/06/16 08:54:24 DEBUG orm.ClassWriter: index 14/06/16 08:54:24 DEBUG orm.ClassWriter: bel 14/06/16 08:54:24 DEBUG orm.ClassWriter: gesp 14/06/16 08:54:24 DEBUG orm.ClassWriter: bd 14/06/16 08:54:24 DEBUG orm.ClassWriter: gd1 14/06/16 08:54:24 DEBUG orm.ClassWriter: gd2 14/06/16 08:54:24 DEBUG manager.SqlManager: Using fetchSize for next query: -2147483648 14/06/16 08:54:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test1` AS t LIMIT 1 14/06/16 08:54:24 DEBUG orm.ClassWriter: Writing source file: /tmp/sqoop-hduser/compile/e278e1e92490b537a25cab0ea827c875/test1.java 14/06/16 08:54:24 DEBUG orm.ClassWriter: Table name: test1 14/06/16 08:54:24 DEBUG orm.ClassWriter: Columns: index:12, bel:4, gesp:4, bd:4, gd1:4, gd2:4, 14/06/16 08:54:24 DEBUG orm.ClassWriter: sourceFilename is test1.java 14/06/16 08:54:24 DEBUG orm.CompilationManager: Found existing /tmp/sqoop-hduser/compile/e278e1e92490b537a25cab0ea827c875/ 14/06/16 08:54:24 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.4.0/share/hadoop/mapreduce Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf at org.apache.sqoop.orm.CompilationManager.findHadoopCoreJar(CompilationManager.java:109) at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:137) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100) at org.apache.sqoop.Sqoop.run(Sqoop.java:145) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) at org.apache.sqoop.Sqoop.main(Sqoop.java:238) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 11 more </snip>
In the past such issues turned out to be caused by a wrong classpath. However, I assume Sqoop is building the classpath on its own depending on the input for HADOOP_*_HOME?! There are no MR logs. Cheers Seb. ________________________________ Von: Abraham Elmahrek [[email protected]] Gesendet: Sonntag, 15. Juni 2014 23:54 An: [email protected] Betreff: Re: using Sqoop 1.4.4 with Hadoop 2.4.0 You can add --verbose to the beginning of your command (right after the word "export") and you'll get some more info. Could you copy/paste the output of your sqoop command with this flag? There should also be logs in the MapReduce job/task logs. In this case, I think the application and container logs may have more information. They are usually located under '/var/log/hadoop-yarn/'. Have you considered installing these components using a vendor package or BigTop? On Sun, Jun 15, 2014 at 12:42 AM, Sebastian Gäde <[email protected]<mailto:[email protected]>> wrote: Hi, Thanks Abe. Yes, Sqoop and Hadoop were downloaded from a tarball. I set HADOOP_COMMON_HOME to /opt/hadoop-2.4.0 and HADOOP_MAPRED_HOME to /opt/hadoop-2.4.0/share/hadoop/yarn using the conf/sqoop-env.sh. I also tried /opt/hadoop-2.4.0/share/hadoop/mapreduce for HADOOP_MAPRED_HOME as I did not know which JAR I need. However, the error was the same. Is there a way to enable DEBUG messages? This sometimes helped me in the passed with other Hadoop issues. Cheers Seb. Am 14.06.2014 06:37, schrieb Abraham Elmahrek: Hey there, The exception is indicative of a missing jar from the class path. It seems like the hadoop-mapreduce-client-core.jar file is missing. Have you set HADOOP_COMMON_HOME and HADOOP_MAPRED_HOME? How did you install Sqoop and Hadoop? If you've downloaded a tarball, you may need to set the environment variables specified above. For more info see http://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_controlling_the_hadoop_installation. -Abe On Fri, Jun 13, 2014 at 4:15 AM, Sebastian Gäde <[email protected]<mailto:[email protected]> <mailto:[email protected]<mailto:[email protected]>>> wrote: Hi, I’d like to export MR result (text) files from HDFS to MySQL. My cluster is running Hadoop 2.4.0. When I’m running ./sqoop export –connect jdbc:mysql://mysqlserver/sqoop –username user –password pass –table test1 –export-dir /output I get the below exception. I assume, sqoop is using the old MR API? Can I change this somehow to use the new API, then I'd be able to point $HADOOP_MAPRED_HOME to the right position?! 14/06/13 13:06:44 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 14/06/13 13:06:44 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 14/06/13 13:06:44 INFO tool.CodeGenTool: Beginning code generation 14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test1` AS t LIMIT 1 14/06/13 13:06:50 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `test1` AS t LIMIT 1 14/06/13 13:06:50 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop-2.4.0/share/hadoop/yarn Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf at org.apache.sqoop.orm.CompilationManager.findHadoopCoreJar(CompilationManager .java:109) at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:137) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100) at org.apache.sqoop.Sqoop.run(Sqoop.java:145) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) at org.apache.sqoop.Sqoop.main(Sqoop.java:238) Thanks very much! Sebastian
