Thanks Venkat. Thats a good read. Regards,Aravind --------------------------------------------------------------------------------------------Aravind Srinivasan
http://goodcynicsports.blogspot.com I have decided to stick with love. Hate is too great a burden to bear. - MLK, Jr.-------------------------------------------------------------------------------------------- From: Venkat Ranganathan <[email protected]> To: "[email protected]" <[email protected]>; Aravindakshan Srinivasan <[email protected]>; Abraham Elmahrek <[email protected]> Sent: Saturday, May 9, 2015 12:19 AM Subject: Re: Oozie + Sqoop + MySql UPSERT Good solution Abe. Aravind. This blog by Jarcec should answer the question - https://blogs.apache.org/sqoop/entry/connectors_and_drivers_in_the The gist is that if you use —driver, then you use a generic manager which does not provide additional features that may be provided by db specific connection managers Venkat From: Aravindakshan Srinivasan Reply-To: "[email protected]", Aravindakshan Srinivasan Date: Friday, May 8, 2015 at 11:39 AM To: Abraham Elmahrek, "[email protected]" Subject: Re: Oozie + Sqoop + MySql UPSERT Wow, it did work. Why though? Thanks a lot Abraham. Regards,Aravind --------------------------------------------------------------------------------------------Aravind Srinivasan http://goodcynicsports.blogspot.com I have decided to stick with love. Hate is too great a burden to bear. - MLK, Jr.-------------------------------------------------------------------------------------------- From: Abraham Elmahrek <[email protected]> To: "[email protected]" <[email protected]>; Aravindakshan Srinivasan <[email protected]> Sent: Friday, May 8, 2015 11:52 PM Subject: Re: Oozie + Sqoop + MySql UPSERT I'd get rid of --driver and see if that works. On Fri, May 8, 2015 at 11:14 AM, Aravindakshan Srinivasan<[email protected]> wrote: Hi Abraham, Here's the Sqoop command that works. sqoop export --connect jdbc:mysql://myhost.rds.amazonaws.com/DB1 --username XXX -password YYY --table test_table --columns "name,ts,age,salary" --update-key "name" --export-dir /user/aravind/test/year=2015/month=5/day=7/hour=19 --fields-terminated-by "\t" --update-mode allowinsert -m 1 The following workflow does not work. <workflow-app xmlns="uri:oozie:workflow:0.2" name="ara_sqoop-wf"> <start to="ara_move_to_mysql"/> <action name="ara_move_to_mysql"> <sqoop xmlns="uri:oozie:sqoop-action:0.3"><job-tracker>${jobTracker}</job-tracker><name-node>${nameNode}</name-node><arg>export</arg><arg>-libjars</arg><arg>${wf_folder}lib/mysql-connector-java.jar</arg><arg>--connect</arg><arg>jdbc:mysql://myhost.rds.amazonaws.com/${mysql_db}</arg><arg>--driver</arg><arg>com.mysql.jdbc.Driver</arg><arg>--username</arg><arg>${mysql_usern}</arg><arg>-password</arg><arg>${mysql_pswd}</arg><arg>--table</arg><arg>${table_name}</arg><arg>--columns</arg><arg>"name,ts,age,salary"</arg><arg>--update-key</arg><arg>"name"</arg><arg>--export-dir</arg><arg>/user/aravind/test/year=2015/month=5/day=7/hour=19</arg><arg>--fields-terminated-by</arg><arg>"\t"</arg><arg>--update-mode</arg><arg>allowinsert</arg><arg>-m</arg><arg>1</arg> </sqoop> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/></workflow-app> Thanks,Aravind From: Abraham Elmahrek <[email protected]> To: "[email protected]" <[email protected]>; Aravindakshan Srinivasan <[email protected]> Sent: Friday, May 8, 2015 10:36 PM Subject: Re: Oozie + Sqoop + MySql UPSERT Can you send over your oozie job or sqoop command? On Fri, May 8, 2015 at 8:30 AM, Aravindakshan Srinivasan <[email protected]> wrote: Friends, I have an odd problem. I am trying to run an MySql upsert via Sqoop. It works fine from the edge node that has Sqoop 1.4.5 installed. When I try to run via Oozie, I get this error. org.apache.sqoop.tool.ExportTool - Error during export: Mixed update/insert is not supported against the target database This feature is available in the newer versions of Sqoop starting with Sqoop 1.4.3, which is what my Oozie sharelib contains. Still, my first reaction was that the Sqoop jars in Oozie's sharelib are different and hence throwing this error. As you know, the Sqoop via Oozie runs on the Hadoop compute nodes. I tried copying the Sqoop jars from my edge node to the workflow/lib dir. It still didn't work. Since then, I have tried everything I can think of, but can't get past this error. I have even rebuilt the Oozie sharelib with Sqoop 1.4.5 and tried both oozie.libpath and oozie.action.sharelib.for.sqoop pointing to my rebuilt lib. Still no luck. What am I missing? Thanks,Aravind --------------------------------------------------------------------------------------------Aravind Srinivasan http://goodcynicsports.blogspot.com I have decided to stick with love. Hate is too great a burden to bear. - MLK, Jr.--------------------------------------------------------------------------------------------
