Hey Attila, Sorry for the confusion, I left out the mysql-connector from that list. That's just what's in there by default and then I install the mysql connector before running anything. I just ran a test without the mariadb connector in there and somehow it's still picking up that MariaDbServerPreparedStatement which makes no sense to me.
On Fri, Aug 5, 2016 at 11:58 AM, Attila Szabo <[email protected]> wrote: > Hi Geren, > > My gut feeling is the presence of the mariadb-connector-java.jar in your > lib directory and the absence of the mysql-connector-java-5.1.21.jar. > AFAIK as mariadb is a fork of MySQL it's also listening to the jdbc:mysql: > protocol, and most probably that's the reason why it tries to use mariadb > related preparedstatement instead of the mysql one. > > Could you please try it by removing the mariadb connector and copy there > the mysql connector with your exact version? > > Thanks > M. > > On Fri, Aug 5, 2016 at 8:30 AM, Geren White <[email protected]> wrote: > >> Attached is the verbose log. >> >> I'm using 5.1.21 of the mysql connector. I've also tried with the latest >> (5.1.39) and that didn't work. >> >> For jdbc drivers in the lib directory there are: >> >> mariadb-connector-java.jar >> >> postgresql-jdbc.jar >> >> hsqldb-1.8.0.10.jar >> >> On Thu, Aug 4, 2016 at 8:36 PM, Attila Szabo <[email protected]> wrote: >> >>> Hi Geren, >>> >>> For further investigation: >>> Could you please send us the whole verbose logfile for further >>> investigation? >>> On the top could you please check which version of the MySQL JDBC driver >>> is present in the lib directory of Sqoop? >>> We would also appreciate if you could tell which other JDBC drivers >>> located there! >>> >>> Many thanks, >>> M. >>> >>> On Thu, Aug 4, 2016 at 1:44 PM, Geren White <[email protected]> wrote: >>> >>>> Hello all, >>>> >>>> I'm having trouble importing data from a mysql database (version 5.26) >>>> with sqoop (version 1.4.6). >>>> >>>> I have a number of imports that are happening in AWS EMR and all of >>>> them are working that go against tables. I have one import that goes >>>> against a view that is not working. It works on our dev database but for >>>> some reason not on our prod server. >>>> >>>> Here's the command I'm running: >>>> sqoop import --connect jdbc:mysql://$databaseHost/inventory >>>> --connection-manager org.apache.sqoop.manager.MySQLManager --username >>>> $user_name >>>> --password-file=s3://$s3Bucket/dibs/settings/sqoop_db_credentials.properties >>>> --as-avrodatafile --target-dir /user/hadoop/inventory/category_view >>>> --delete-target-dir --table category_view --split-by id --verbose >>>> >>>> I've tried with and without specifying the jdbc driver and without >>>> specifying the connection-manager. I've also tried overriding the default >>>> SQL it is using but nothing seems to help. I've also seen a number of other >>>> users with this issue that all point to the table_definition_cache but >>>> playing with those values has not fixed the issue. >>>> >>>> From the below stacktrace you can see that it is using a >>>> MariaDbServerPreparedStatement. This seems weird to me and would >>>> expect it to be using the Mysql specific PreparedStatement class. Is this >>>> the normal behavior? >>>> >>>> Here's the stacktrace: >>>> >>>> 2016-08-04 19:49:40,589 INFO org.apache.sqoop.manager.SqlManager (main): >>>> Executing SQL statement: SELECT t.* FROM `category_view` AS t LIMIT 1 >>>> 2016-08-04 19:49:40,601 ERROR org.apache.sqoop.manager.SqlManager (main): >>>> Error executing statement: java.sql.BatchUpdateException: Prepared >>>> statement needs to be re-prepared >>>> java.sql.BatchUpdateException: Prepared statement needs to be re-prepared >>>> at >>>> org.mariadb.jdbc.MariaDbServerPreparedStatement.execute(MariaDbServerPreparedStatement.java:374) >>>> at >>>> org.mariadb.jdbc.MariaDbServerPreparedStatement.executeQuery(MariaDbServerPreparedStatement.java:341) >>>> at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:758) >>>> at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:767) >>>> at >>>> org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270) >>>> at >>>> org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241) >>>> at >>>> org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227) >>>> at >>>> org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295) >>>> at >>>> org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833) >>>> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645) >>>> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107) >>>> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478) >>>> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) >>>> at org.apache.sqoop.Sqoop.run(Sqoop.java:143) >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) >>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) >>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) >>>> at org.apache.sqoop.Sqoop.main(Sqoop.java:236) >>>> Caused by: java.sql.SQLException: Prepared statement needs to be >>>> re-prepared >>>> at >>>> org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:136) >>>> at >>>> org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69) >>>> at >>>> org.mariadb.jdbc.MariaDbServerPreparedStatement.executeQueryEpilog(MariaDbServerPreparedStatement.java:336) >>>> at >>>> org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:291) >>>> at >>>> org.mariadb.jdbc.MariaDbServerPreparedStatement.execute(MariaDbServerPreparedStatement.java:369) >>>> ... 18 more >>>> Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Prepared >>>> statement needs to be re-prepared >>>> at >>>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:479) >>>> at >>>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executePreparedQuery(AbstractQueryProtocol.java:592) >>>> at >>>> org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:279) >>>> ... 19 more >>>> 2016-08-04 19:49:40,606 ERROR org.apache.sqoop.tool.ImportTool (main): >>>> Encountered IOException running import job: java.io.IOException: No >>>> columns to generate for ClassWriter >>>> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651) >>>> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107) >>>> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478) >>>> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) >>>> at org.apache.sqoop.Sqoop.run(Sqoop.java:143) >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) >>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) >>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) >>>> at org.apache.sqoop.Sqoop.main(Sqoop.java:236) >>>> >>>> >>>> Any thoughts/recommendations would be greatly appreciated. >>>> >>>> Thanks, >>>> -- >>>> *Geren White | Senior Java Developer* >>>> *(e)* [email protected] >>>> >>> >>> >>> >>> -- >>> Best regards, >>> >>> Attila Szabo >>> Sotware Engineer >>> >>> <http://www.cloudera.com> >>> >> >> >> >> -- >> *Geren White | Senior Java Developer* >> *(e)* [email protected] >> > > > > -- > Best regards, > > Attila Szabo > Sotware Engineer > > <http://www.cloudera.com> > -- *Geren White | Senior Java Developer* *(e)* [email protected]
