Hey Geren, My advise would be for you trying to create a clean environment. If you can do a full clean install that would be the best. On the other hand (if it's not an option) please try to find all of the occurences of the mariadb connector and delete them.
Meanwhile I've checked the content of the mysql connector jar (latest one) and there must be 0 org.mariadb.* related class. So most probably some of the mariadb connector jars are hiding on your cluster. Cheers M. On Fri, Aug 5, 2016 at 9:03 AM, Geren White <[email protected]> wrote: > Hey Attila, > Sorry for the confusion, I left out the mysql-connector from that list. > That's just what's in there by default and then I install the mysql > connector before running anything. I just ran a test without the mariadb > connector in there and somehow it's still picking up that > MariaDbServerPreparedStatement which makes no sense to me. > > > On Fri, Aug 5, 2016 at 11:58 AM, Attila Szabo <[email protected]> wrote: > >> Hi Geren, >> >> My gut feeling is the presence of the mariadb-connector-java.jar in your >> lib directory and the absence of the mysql-connector-java-5.1.21.jar. >> AFAIK as mariadb is a fork of MySQL it's also listening to the >> jdbc:mysql: protocol, and most probably that's the reason why it tries to >> use mariadb related preparedstatement instead of the mysql one. >> >> Could you please try it by removing the mariadb connector and copy there >> the mysql connector with your exact version? >> >> Thanks >> M. >> >> On Fri, Aug 5, 2016 at 8:30 AM, Geren White <[email protected]> wrote: >> >>> Attached is the verbose log. >>> >>> I'm using 5.1.21 of the mysql connector. I've also tried with the latest >>> (5.1.39) and that didn't work. >>> >>> For jdbc drivers in the lib directory there are: >>> >>> mariadb-connector-java.jar >>> >>> postgresql-jdbc.jar >>> >>> hsqldb-1.8.0.10.jar >>> >>> On Thu, Aug 4, 2016 at 8:36 PM, Attila Szabo <[email protected]> >>> wrote: >>> >>>> Hi Geren, >>>> >>>> For further investigation: >>>> Could you please send us the whole verbose logfile for further >>>> investigation? >>>> On the top could you please check which version of the MySQL JDBC >>>> driver is present in the lib directory of Sqoop? >>>> We would also appreciate if you could tell which other JDBC drivers >>>> located there! >>>> >>>> Many thanks, >>>> M. >>>> >>>> On Thu, Aug 4, 2016 at 1:44 PM, Geren White <[email protected]> wrote: >>>> >>>>> Hello all, >>>>> >>>>> I'm having trouble importing data from a mysql database (version 5.26) >>>>> with sqoop (version 1.4.6). >>>>> >>>>> I have a number of imports that are happening in AWS EMR and all of >>>>> them are working that go against tables. I have one import that goes >>>>> against a view that is not working. It works on our dev database but for >>>>> some reason not on our prod server. >>>>> >>>>> Here's the command I'm running: >>>>> sqoop import --connect jdbc:mysql://$databaseHost/inventory >>>>> --connection-manager org.apache.sqoop.manager.MySQLManager --username >>>>> $user_name --password-file=s3://$s3Bucket >>>>> /dibs/settings/sqoop_db_credentials.properties --as-avrodatafile >>>>> --target-dir /user/hadoop/inventory/category_view --delete-target-dir >>>>> --table category_view --split-by id --verbose >>>>> >>>>> I've tried with and without specifying the jdbc driver and without >>>>> specifying the connection-manager. I've also tried overriding the default >>>>> SQL it is using but nothing seems to help. I've also seen a number of >>>>> other >>>>> users with this issue that all point to the table_definition_cache but >>>>> playing with those values has not fixed the issue. >>>>> >>>>> From the below stacktrace you can see that it is using a >>>>> MariaDbServerPreparedStatement. This seems weird to me and would >>>>> expect it to be using the Mysql specific PreparedStatement class. Is this >>>>> the normal behavior? >>>>> >>>>> Here's the stacktrace: >>>>> >>>>> 2016-08-04 19:49:40,589 INFO org.apache.sqoop.manager.SqlManager (main): >>>>> Executing SQL statement: SELECT t.* FROM `category_view` AS t LIMIT 1 >>>>> 2016-08-04 19:49:40,601 ERROR org.apache.sqoop.manager.SqlManager (main): >>>>> Error executing statement: java.sql.BatchUpdateException: Prepared >>>>> statement needs to be re-prepared >>>>> java.sql.BatchUpdateException: Prepared statement needs to be re-prepared >>>>> at >>>>> org.mariadb.jdbc.MariaDbServerPreparedStatement.execute(MariaDbServerPreparedStatement.java:374) >>>>> at >>>>> org.mariadb.jdbc.MariaDbServerPreparedStatement.executeQuery(MariaDbServerPreparedStatement.java:341) >>>>> at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:758) >>>>> at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:767) >>>>> at >>>>> org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270) >>>>> at >>>>> org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241) >>>>> at >>>>> org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227) >>>>> at >>>>> org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295) >>>>> at >>>>> org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833) >>>>> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645) >>>>> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107) >>>>> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478) >>>>> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) >>>>> at org.apache.sqoop.Sqoop.run(Sqoop.java:143) >>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) >>>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) >>>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) >>>>> at org.apache.sqoop.Sqoop.main(Sqoop.java:236) >>>>> Caused by: java.sql.SQLException: Prepared statement needs to be >>>>> re-prepared >>>>> at >>>>> org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:136) >>>>> at >>>>> org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69) >>>>> at >>>>> org.mariadb.jdbc.MariaDbServerPreparedStatement.executeQueryEpilog(MariaDbServerPreparedStatement.java:336) >>>>> at >>>>> org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:291) >>>>> at >>>>> org.mariadb.jdbc.MariaDbServerPreparedStatement.execute(MariaDbServerPreparedStatement.java:369) >>>>> ... 18 more >>>>> Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Prepared >>>>> statement needs to be re-prepared >>>>> at >>>>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:479) >>>>> at >>>>> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executePreparedQuery(AbstractQueryProtocol.java:592) >>>>> at >>>>> org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:279) >>>>> ... 19 more >>>>> 2016-08-04 19:49:40,606 ERROR org.apache.sqoop.tool.ImportTool (main): >>>>> Encountered IOException running import job: java.io.IOException: No >>>>> columns to generate for ClassWriter >>>>> at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651) >>>>> at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107) >>>>> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478) >>>>> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) >>>>> at org.apache.sqoop.Sqoop.run(Sqoop.java:143) >>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) >>>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) >>>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) >>>>> at org.apache.sqoop.Sqoop.main(Sqoop.java:236) >>>>> >>>>> >>>>> Any thoughts/recommendations would be greatly appreciated. >>>>> >>>>> Thanks, >>>>> -- >>>>> *Geren White | Senior Java Developer* >>>>> *(e)* [email protected] >>>>> >>>> >>>> >>>> >>>> -- >>>> Best regards, >>>> >>>> Attila Szabo >>>> Sotware Engineer >>>> >>>> <http://www.cloudera.com> >>>> >>> >>> >>> >>> -- >>> *Geren White | Senior Java Developer* >>> *(e)* [email protected] >>> >> >> >> >> -- >> Best regards, >> >> Attila Szabo >> Sotware Engineer >> >> <http://www.cloudera.com> >> > > > > -- > *Geren White | Senior Java Developer* > *(e)* [email protected] > -- Best regards, Attila Szabo Sotware Engineer <http://www.cloudera.com>
