[
https://issues.apache.org/jira/browse/HIVE-8588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-8588:
---------------------------------
Description:
This is originally discovered by [~deepesh]
When running a Sqoop integration test from WebHCat
{noformat}
curl --show-error -d command="export -libjars
hdfs:///tmp/mysql-connector-java.jar --connect
jdbc:mysql://deepesh-c6-1.cs1cloud.internal/sqooptest --username sqoop
--password passwd --export-dir /tmp/templeton_test_data/sqoop --table person"
-d statusdir=sqoop.output -X POST
"http://deepesh-c6-1.cs1cloud.internal:50111/templeton/v1/sqoop?user.name=hrt_qa";
{noformat}
the job is failing with the following error:
{noformat}
$ hadoop fs -cat /user/hrt_qa/sqoop.output/stderr
14/10/15 23:52:53 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5.2.2.0.0-897
14/10/15 23:52:53 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
14/10/15 23:52:54 INFO manager.MySQLManager: Preparing to use a MySQL streaming
resultset.
14/10/15 23:52:54 INFO tool.CodeGenTool: Beginning code generation
14/10/15 23:52:54 ERROR sqoop.Sqoop: Got exception running Sqoop:
java.lang.RuntimeException: Could not load db driver class:
com.mysql.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class:
com.mysql.jdbc.Driver
at
org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:848)
at
org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at
org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at
org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at
org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at
org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at
org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
{noformat}
Note that the Sqoop tar bundle does not contain the JDBC connector jar. I think
the problem here maybe that the mysql connector jar added to libjars isn't
available to the Sqoop tool which first connects to the database through JDBC
driver to collect some table information before running the MR job. libjars
will only add the connector jar for the MR job and not the local one.
NO PRECOMMIT TESTS
was:
This is originally discovered by [~deepesh]
When running a Sqoop integration test from WebHCat
{noformat}
curl --show-error -d command="export -libjars
hdfs:///tmp/mysql-connector-java.jar --connect
jdbc:mysql://deepesh-c6-1.cs1cloud.internal/sqooptest --username sqoop
--password passwd --export-dir /tmp/templeton_test_data/sqoop --table person"
-d statusdir=sqoop.output -X POST
"http://deepesh-c6-1.cs1cloud.internal:50111/templeton/v1/sqoop?user.name=hrt_qa";
{noformat}
the job is failing with the following error:
{noformat}
$ hadoop fs -cat /user/hrt_qa/sqoop.output/stderr
14/10/15 23:52:53 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5.2.2.0.0-897
14/10/15 23:52:53 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
14/10/15 23:52:54 INFO manager.MySQLManager: Preparing to use a MySQL streaming
resultset.
14/10/15 23:52:54 INFO tool.CodeGenTool: Beginning code generation
14/10/15 23:52:54 ERROR sqoop.Sqoop: Got exception running Sqoop:
java.lang.RuntimeException: Could not load db driver class:
com.mysql.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class:
com.mysql.jdbc.Driver
at
org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:848)
at
org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
at
org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
at
org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at
org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at
org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at
org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
{noformat}
Note that the Sqoop tar bundle does not contain the JDBC connector jar. I think
the problem here maybe that the mysql connector jar added to libjars isn't
available to the Sqoop tool which first connects to the database through JDBC
driver to collect some table information before running the MR job. libjars
will only add the connector jar for the MR job and not the local one.
> sqoop REST endpoint fails to send appropriate JDBC driver to the cluster
> ------------------------------------------------------------------------
>
> Key: HIVE-8588
> URL: https://issues.apache.org/jira/browse/HIVE-8588
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
> Affects Versions: 0.14.0
> Reporter: Eugene Koifman
> Assignee: Eugene Koifman
>
> This is originally discovered by [~deepesh]
> When running a Sqoop integration test from WebHCat
> {noformat}
> curl --show-error -d command="export -libjars
> hdfs:///tmp/mysql-connector-java.jar --connect
> jdbc:mysql://deepesh-c6-1.cs1cloud.internal/sqooptest --username sqoop
> --password passwd --export-dir /tmp/templeton_test_data/sqoop --table person"
> -d statusdir=sqoop.output -X POST
> "http://deepesh-c6-1.cs1cloud.internal:50111/templeton/v1/sqoop?user.name=hrt_qa";
> {noformat}
> the job is failing with the following error:
> {noformat}
> $ hadoop fs -cat /user/hrt_qa/sqoop.output/stderr
> 14/10/15 23:52:53 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5.2.2.0.0-897
> 14/10/15 23:52:53 WARN tool.BaseSqoopTool: Setting your password on the
> command-line is insecure. Consider using -P instead.
> 14/10/15 23:52:54 INFO manager.MySQLManager: Preparing to use a MySQL
> streaming resultset.
> 14/10/15 23:52:54 INFO tool.CodeGenTool: Beginning code generation
> 14/10/15 23:52:54 ERROR sqoop.Sqoop: Got exception running Sqoop:
> java.lang.RuntimeException: Could not load db driver class:
> com.mysql.jdbc.Driver
> java.lang.RuntimeException: Could not load db driver class:
> com.mysql.jdbc.Driver
> at
> org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:848)
> at
> org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
> at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736)
> at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759)
> at
> org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269)
> at
> org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
> at
> org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
> at
> org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
> at
> org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
> at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64)
> at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> {noformat}
> Note that the Sqoop tar bundle does not contain the JDBC connector jar. I
> think the problem here maybe that the mysql connector jar added to libjars
> isn't available to the Sqoop tool which first connects to the database
> through JDBC driver to collect some table information before running the MR
> job. libjars will only add the connector jar for the MR job and not the local
> one.
> NO PRECOMMIT TESTS
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
