Incremental sqoop without duplication would require merge of avro files, which is not supported. https://issues.apache.org/jira/browse/SQOOP-1094 -------------------------------------------- On Tue, 3/1/16, Ajay Chander <[email protected]> wrote:
Subject: Re: Question_about_Sqoop_Avro To: "[email protected]" <[email protected]>, "Deepak Vohra" <[email protected]> Date: Tuesday, March 1, 2016, 5:41 PM Can someone shed some lights on this? I am trying to perform incremental operation through sqoop to write the data in Avro format. Thanks On Tuesday, March 1, 2016, Ajay Chander <[email protected]> wrote: But when I do the same with '--incremental append' it is working fine except that I see duplicates which is as expected. On Tuesday, March 1, 2016, Deepak Vohra <[email protected]> wrote: '--incremental lastmodified cannot be used in conjunction with --as-avrodatafile' Being binary data avro data may not be incrementable. -------------------------------------------- On Tue, 3/1/16, Ajay Chander <[email protected]> wrote: Subject: Question_about_Sqoop_Avro To: "[email protected]" <[email protected]> Date: Tuesday, March 1, 2016, 12:01 PM Hi Everyone, I just have a question about using sqoop incremental operation with --as-avrodatafile. Let's say initially I did a sqoop import to pull the table 'xyz' from mysql db to HDFS with option '--as-avrodatafile'. Now let us say that I have some new data added to source system(xyz table in MySQL) now I want to run a sqoop incremental job to get those changes updated in HDFS with option '--as-avrodatafile'. When I try to do that it says '--incremental lastmodified cannot be used in conjunction with --as-avrodatafile' Any pointers are appreciated. Thanks.
