Hello Markus, I tried with --input-lines-terminated-by instead of lines-terminated-by still, the --column argument is working as it supposed to work. The --column option worked perfectly when I did only the insert option( the initial load) when I am doing the upsert option, it is not considering the order as I mentioned in the --column sqoop argument. Please see the attached sqoop argument and the CSV file screenshot. I could not understand why --column option is not working when I am doing upsert. Andy kind of help is really appreciated.
Column order in ods: id,START_DT,EXP_DT,VAL,QUALITY,PROCES_DT value order in csv file; id,START_DT,EXP_DT,QUALITY,VAL,PROCES_DT --Sqoop command. sqoop export --connect connection_string \ --username xxx \ --password xxxx \ --table xxx \ --export-dir /location/file.csv \ --input-fields-terminated-by ',' \ --input-lines-terminated-by '\n' \ --update-key column_name \ --update-mode allowinsert \ --columns "id,START_DT,EXP_DT,QUALITY,VAL,PROCES_DT" \ -m 1 Best, Ajana Sathian Big Data Intern Cars.com http://ajanacs.com/ On Sun, Dec 3, 2017 at 3:41 PM, Markus Kemper <[email protected]> wrote: > Hey Alana, > > —lines-* and —fields-* are for (sqoop import) and —input-lines-* and > —input-fields-* are for (sqoop export). > > If you do not specify it will use the default which may or may not be > compatible with your data files. > > As a best practice I always recommend explicitly setting all options to > avoid any confusion or assumptions. > > Thanks, Markus > > On Dec 3, 2017, at 12:46, Ajana Chandiruthil Sathian <[email protected]> > wrote: > > Hello Markus, > > Can I ask you one question? Does that make any change? Both are for the > same purpose, right? Please correct me if I am wrong. > > On Sun, Dec 3, 2017 at 10:39 AM, Markus Kemper <[email protected]> > wrote: > >> Hello Ajana, >> >> Have you tried using (--input-lines-terminated-by '\n') instead of >> (--lines-terminated-by >> '\n')? >> >> >> Markus Kemper >> Customer Operations Engineer >> [image: www.cloudera.com] <http://www.cloudera.com> >> >> >> On Sun, Dec 3, 2017 at 8:45 AM, Ajana Chandiruthil Sathian < >> [email protected]> wrote: >> >>> >>> ---------- Forwarded message ---------- >>> From: Ajana Chandiruthil Sathian <[email protected]> >>> Date: Wed, Nov 15, 2017 at 11:09 AM >>> Subject: Upsert option in Sqoop export command. >>> To: [email protected] >>> >>> >>> To whom ever it may concern, >>> >>> I have a csv file in Hadoop and I did Sqoop export to Oracle. The column >>> data type order in ODS is number,date, date,float,varchar and the column >>> data type in the csv file is number,date, float,date,varchar and I used the >>> --columns sqoop argument to get control in column ordering and it worked. >>> But I could not control the column ordering when I was doing the upsert >>> operation in sqoop export. It is giving me misalignment in ODS after >>> Sqoop( please see the attached image). The below given is the sqoop command: >>> >>> sqoop export --connect ConnectionString \ >>> --username xxx \ >>> --password xxxx \ >>> --table tableName \ >>> --export-dir /dir/TestUpdate.txt \ >>> --input-fields-terminated-by ',' \ >>> --lines-terminated-by '\n' \ >>> --update-key column_name \ >>> --update-mode allowinsert \ >>> --columns "id,START_DT,VAL,end_DT,QUALITY" \ >>> -m 4 >>> >>> >> >
