Thanks Abe,
I tried using sql property to do an update.
MFromConfig mFromConfig = job.getFromJobConfig();
mFromConfig.getStringInput("fromJobConfig.inputDirectory").setValue("/data/hdfs_location/prasanth");
Here is the query “INSERT INTO SCHEMA_NAME.TABLE_NAME(col1, col2, col3)
values('?')”
MToConfig toJobConfig = job.getToJobConfig();
toJobConfig.getStringInput("toJobConfig.sql").setValue("INSERT INTO
SCHEMA_NAME.TABLE_NAME(COL1, COL2, COL3) values('?')");
I get the following exception
org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0019:Table name
extraction not supported. - Table name extraction not supported yet.
at
org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.getSchema(GenericJdbcToInitializer.java:69)
at
org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.getSchema(GenericJdbcToInitializer.java:39)
at
org.apache.sqoop.driver.JobManager.getSchemaForConnector(JobManager.java:461)
at
org.apache.sqoop.driver.JobManager.createJobRequest(JobManager.java:375)
at org.apache.sqoop.driver.JobManager.start(JobManager.java:277)
at
org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:367)
at
org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:112)
at
org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96)
at
org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:745)
Is this the right way of using the sql option? If I don’t use the “?” in the
sql I get the following exception
org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0013:No
parameter marker in the specified sql
at
org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.configureTableProperties(GenericJdbcToInitializer.java:201)
at
org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.initialize(GenericJdbcToInitializer.java:49)
at
org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.initialize(GenericJdbcToInitializer.java:39)
at
org.apache.sqoop.driver.JobManager.initializeConnector(JobManager.java:451)
at
org.apache.sqoop.driver.JobManager.createJobRequest(JobManager.java:372)
at org.apache.sqoop.driver.JobManager.start(JobManager.java:277)
at
org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:367)
at
org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:112)
at
org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96)
at
org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:745)
How about extending an existing connector to build update functionality? How
complex would that be?
Thanks Abe
Prasanth Chagarlamudi
From: Abraham Elmahrek [mailto:[email protected]]
Sent: Tuesday, July 21, 2015 3:51 PM
To: [email protected]
Subject: Re: Update mode on Sqoop2 Export
Prasanth,
I haven't looked at this section of the code in a while, but I know Sqoop2
doesn't have different connectors for the different relational databases at the
moment. Upsert support varies depending on the database, so I would imagine
it's not supported.
-Abe
On Tue, Jul 21, 2015 at 10:54 AM, Chagarlamudi, Prasanth
<[email protected]<mailto:[email protected]>>
wrote:
Hello All,
I am able to run an EXPORT job successfully. Inserts are going fine, but I am
getting an exception when I try to do an update.
Error: org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0018:Error occurs
during loader run at
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:255)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at
java.util.concurrent.FutureTask.run(FutureTask.java:262) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0002:Unable to
execute the SQL statement at
org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeBatch(GenericJdbcExecutor.java:189)
at
org.apache.sqoop.connector.jdbc.GenericJdbcLoader.load(GenericJdbcLoader.java:66)
at
org.apache.sqoop.connector.jdbc.GenericJdbcLoader.load(GenericJdbcLoader.java:25)
at
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:247)
... 5 more Caused by: java.sql.BatchUpdateException: ORA-00001: unique
constraint (SCHEMA.TABLENAME_PK) violated at
oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:11190)
at
oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:244)
at
org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.executeBatch(GenericJdbcExecutor.java:183)
... 8 more
Sqoop2 only supports inserts? If not, is there a way to turn the update mode on
for Sqoop2?
Thanks
Prasanth Chagarlamudi
________________________________
This e-mail and files transmitted with it are confidential, and are intended
solely for the use of the individual or entity to whom this e-mail is
addressed. If you are not the intended recipient, or the employee or agent
responsible to deliver it to the intended recipient, you are hereby notified
that any dissemination, distribution or copying of this communication is
strictly prohibited. If you are not one of the named recipient(s) or otherwise
have reason to believe that you received this message in error, please
immediately notify sender by e-mail, and destroy the original message. Thank
You.
________________________________
This e-mail and files transmitted with it are confidential, and are intended
solely for the use of the individual or entity to whom this e-mail is
addressed. If you are not the intended recipient, or the employee or agent
responsible to deliver it to the intended recipient, you are hereby notified
that any dissemination, distribution or copying of this communication is
strictly prohibited. If you are not one of the named recipient(s) or otherwise
have reason to believe that you received this message in error, please
immediately notify sender by e-mail, and destroy the original message. Thank
You.
