Hey man, See inline.
-Abe On Thu, Jul 23, 2015 at 9:39 AM, Chagarlamudi, Prasanth < [email protected]> wrote: > Thanks Abe, > > I tried using sql property to do an update. > > > > MFromConfig mFromConfig = job.getFromJobConfig(); > > mFromConfig.getStringInput("fromJobConfig.inputDirectory" > ).setValue("/data/hdfs_location/prasanth"); > > Here is the query “*INSERT INTO SCHEMA_NAME.TABLE_NAME(col1, col2, col3) > values('?')*” > > MToConfig toJobConfig = job.getToJobConfig(); > > toJobConfig.getStringInput("toJobConfig.sql").setValue("INSERT INTO > SCHEMA_NAME.TABLE_NAME(COL1, COL2, COL3) values('?')"); > > > > I get the following exception > > org.apache.sqoop.common.SqoopException: *GENERIC_JDBC_CONNECTOR_0019:Table > name extraction not supported. - Table name extraction not supported yet.* > > at > org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.getSchema(GenericJdbcToInitializer.java:69) > > at > org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.getSchema(GenericJdbcToInitializer.java:39) > > at > org.apache.sqoop.driver.JobManager.getSchemaForConnector(JobManager.java:461) > > at > org.apache.sqoop.driver.JobManager.createJobRequest(JobManager.java:375) > > at > org.apache.sqoop.driver.JobManager.start(JobManager.java:277) > > at > org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:367) > > at > org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:112) > > at > org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96) > > at > org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79) > > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:646) > > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:723) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) > > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) > > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) > > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606) > > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > > at java.lang.Thread.run(Thread.java:745) > > > > *Is this the right way of using the sql option? If I don’t use the “?” in > the sql I get the following exception* > Jarcec recently created this Jira: https://issues.apache.org/jira/browse/SQOOP-2441 to drop custom insert SQL support. Thoughts? > > org.apache.sqoop.common.SqoopException: *GENERIC_JDBC_CONNECTOR_0013:No > parameter marker in the specified sql* > > at > org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.configureTableProperties(GenericJdbcToInitializer.java:201) > > at > org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.initialize(GenericJdbcToInitializer.java:49) > > at > org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.initialize(GenericJdbcToInitializer.java:39) > > at > org.apache.sqoop.driver.JobManager.initializeConnector(JobManager.java:451) > > at > org.apache.sqoop.driver.JobManager.createJobRequest(JobManager.java:372) > > at > org.apache.sqoop.driver.JobManager.start(JobManager.java:277) > > at > org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:367) > > at > org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:112) > > at > org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96) > > at > org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79) > > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:646) > > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:723) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) > > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) > > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) > > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606) > > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > > at java.lang.Thread.run(Thread.java:745) > > > > *How about extending an existing connector to build update functionality? > How complex would that be?* > Update being upsert? You definitely could copy the Generic JDBC Connector and add it yourself. I even started a similar process for a "fast path" with MySQL a little while back: https://issues.apache.org/jira/browse/SQOOP-1859. If you'd like, you could pick up where I left off there and add functionality as needed? > > > > > Thanks Abe > > Prasanth Chagarlamudi > > > > *From:* Abraham Elmahrek [mailto:[email protected]] > *Sent:* Tuesday, July 21, 2015 3:51 PM > *To:* [email protected] > *Subject:* Re: Update mode on Sqoop2 Export > > > > Prasanth, > > > > I haven't looked at this section of the code in a while, but I know Sqoop2 > doesn't have different connectors for the different relational databases at > the moment. Upsert support varies depending on the database, so I would > imagine it's not supported. > > > > -Abe > > > > On Tue, Jul 21, 2015 at 10:54 AM, Chagarlamudi, Prasanth < > [email protected]> wrote: > > Hello All, > > I am able to run an EXPORT job successfully. Inserts are going fine, but I > am getting an exception when I try to do an update. > > > > Error: org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0018:Error > occurs during loader run at > org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:255) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) Caused by: > org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable > to execute the SQL statement at > org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeBatch(GenericJdbcExecutor.java:189) > at > org.apache.sqoop.connector.jdbc.GenericJdbcLoader.load(GenericJdbcLoader.java:66) > at > org.apache.sqoop.connector.jdbc.GenericJdbcLoader.load(GenericJdbcLoader.java:25) > at > org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:247) > ... 5 more Caused by: java.sql.BatchUpdateException: *ORA-00001: unique > constraint (SCHEMA.TABLENAME_PK)* violated at > oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:11190) > at > oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:244) > at > org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeBatch(GenericJdbcExecutor.java:183) > ... 8 more > > > > Sqoop2 only supports inserts? If not, is there a way to turn the *update* > mode on for Sqoop2? > > > > Thanks > > Prasanth Chagarlamudi > > > ------------------------------ > > > This e-mail and files transmitted with it are confidential, and are > intended solely for the use of the individual or entity to whom this e-mail > is addressed. If you are not the intended recipient, or the employee or > agent responsible to deliver it to the intended recipient, you are hereby > notified that any dissemination, distribution or copying of this > communication is strictly prohibited. If you are not one of the named > recipient(s) or otherwise have reason to believe that you received this > message in error, please immediately notify sender by e-mail, and destroy > the original message. Thank You. > > > > ------------------------------ > > This e-mail and files transmitted with it are confidential, and are > intended solely for the use of the individual or entity to whom this e-mail > is addressed. If you are not the intended recipient, or the employee or > agent responsible to deliver it to the intended recipient, you are hereby > notified that any dissemination, distribution or copying of this > communication is strictly prohibited. If you are not one of the named > recipient(s) or otherwise have reason to believe that you received this > message in error, please immediately notify sender by e-mail, and destroy > the original message. Thank You. >
