Thanks a lot Abe. Regards,Aravind --------------------------------------------------------------------------------------------Aravind Srinivasan
http://goodcynicsports.blogspot.com I have decided to stick with love. Hate is too great a burden to bear. - MLK, Jr.-------------------------------------------------------------------------------------------- From: Abraham Elmahrek <[email protected]> To: Aravindakshan Srinivasan <[email protected]> Cc: "[email protected]" <[email protected]> Sent: Saturday, May 9, 2015 12:14 AM Subject: Re: Oozie + Sqoop + MySql UPSERT --driver tells sqoop to use the generic jdbc manager, which is not what you want. You want upsert mode in the mysql manager. -Abe On Fri, May 8, 2015 at 11:39 AM, Aravindakshan Srinivasan <[email protected]> wrote: Wow, it did work. Why though? Thanks a lot Abraham. Regards,Aravind --------------------------------------------------------------------------------------------Aravind Srinivasan http://goodcynicsports.blogspot.com I have decided to stick with love. Hate is too great a burden to bear. - MLK, Jr.-------------------------------------------------------------------------------------------- From: Abraham Elmahrek <[email protected]> To: "[email protected]" <[email protected]>; Aravindakshan Srinivasan <[email protected]> Sent: Friday, May 8, 2015 11:52 PM Subject: Re: Oozie + Sqoop + MySql UPSERT I'd get rid of --driver and see if that works. On Fri, May 8, 2015 at 11:14 AM, Aravindakshan Srinivasan <[email protected]> wrote: Hi Abraham, Here's the Sqoop command that works. sqoop export --connect jdbc:mysql://myhost.rds.amazonaws.com/DB1 --username XXX -password YYY --table test_table --columns "name,ts,age,salary" --update-key "name" --export-dir /user/aravind/test/year=2015/month=5/day=7/hour=19 --fields-terminated-by "\t" --update-mode allowinsert -m 1 The following workflow does not work. <workflow-app xmlns="uri:oozie:workflow:0.2" name="ara_sqoop-wf"> <start to="ara_move_to_mysql"/> <action name="ara_move_to_mysql"> <sqoop xmlns="uri:oozie:sqoop-action:0.3"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <arg>export</arg> <arg>-libjars</arg> <arg>${wf_folder}lib/mysql-connector-java.jar</arg> <arg>--connect</arg> <arg>jdbc:mysql://myhost.rds.amazonaws.com/${mysql_db}</arg> <arg>--driver</arg> <arg>com.mysql.jdbc.Driver</arg> <arg>--username</arg> <arg>${mysql_usern}</arg> <arg>-password</arg> <arg>${mysql_pswd}</arg> <arg>--table</arg> <arg>${table_name}</arg> <arg>--columns</arg> <arg>"name,ts,age,salary"</arg> <arg>--update-key</arg> <arg>"name"</arg> <arg>--export-dir</arg> <arg>/user/aravind/test/year=2015/month=5/day=7/hour=19</arg> <arg>--fields-terminated-by</arg> <arg>"\t"</arg> <arg>--update-mode</arg> <arg>allowinsert</arg> <arg>-m</arg> <arg>1</arg> </sqoop> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/></workflow-app> Thanks,Aravind From: Abraham Elmahrek <[email protected]> To: "[email protected]" <[email protected]>; Aravindakshan Srinivasan <[email protected]> Sent: Friday, May 8, 2015 10:36 PM Subject: Re: Oozie + Sqoop + MySql UPSERT Can you send over your oozie job or sqoop command? On Fri, May 8, 2015 at 8:30 AM, Aravindakshan Srinivasan <[email protected]> wrote: Friends, I have an odd problem. I am trying to run an MySql upsert via Sqoop. It works fine from the edge node that has Sqoop 1.4.5 installed. When I try to run via Oozie, I get this error. org.apache.sqoop.tool.ExportTool - Error during export: Mixed update/insert is not supported against the target database This feature is available in the newer versions of Sqoop starting with Sqoop 1.4.3, which is what my Oozie sharelib contains. Still, my first reaction was that the Sqoop jars in Oozie's sharelib are different and hence throwing this error. As you know, the Sqoop via Oozie runs on the Hadoop compute nodes. I tried copying the Sqoop jars from my edge node to the workflow/lib dir. It still didn't work. Since then, I have tried everything I can think of, but can't get past this error. I have even rebuilt the Oozie sharelib with Sqoop 1.4.5 and tried both oozie.libpath and oozie.action.sharelib.for.sqoop pointing to my rebuilt lib. Still no luck. What am I missing? Thanks,Aravind --------------------------------------------------------------------------------------------Aravind Srinivasan http://goodcynicsports.blogspot.com I have decided to stick with love. Hate is too great a burden to bear. - MLK, Jr.--------------------------------------------------------------------------------------------
