Thanks for the info! Would it work well against parquet tables? On Tuesday, March 1, 2016, Jarek Jarcec Cecho <[email protected]> wrote:
> SQOOP-1094 got resolved and will be available in our next release (1.4.7). > > Jarcec > > > On Mar 1, 2016, at 6:32 PM, Deepak Vohra <[email protected] > <javascript:;>> wrote: > > > > Incremental sqoop without duplication would require merge of avro files, > which is not supported. > > https://issues.apache.org/jira/browse/SQOOP-1094 > > -------------------------------------------- > > On Tue, 3/1/16, Ajay Chander <[email protected] <javascript:;>> > wrote: > > > > Subject: Re: Question_about_Sqoop_Avro > > To: "[email protected] <javascript:;>" <[email protected] > <javascript:;>>, "Deepak Vohra" <[email protected] <javascript:;>> > > Date: Tuesday, March 1, 2016, 5:41 PM > > > > Can someone shed some > > lights on this? I am trying to perform incremental operation > > through sqoop to write the data in Avro format. > > Thanks > > > > On > > Tuesday, March 1, 2016, Ajay Chander <[email protected] > <javascript:;>> > > wrote: > > But when > > I do the same with '--incremental append' it is > > working fine except that I see duplicates which is as > > expected. > > On Tuesday, > > March 1, 2016, Deepak Vohra <[email protected] <javascript:;>> wrote: > > '--incremental lastmodified > > cannot be used in conjunction with --as-avrodatafile' > > > > Being binary data avro data may not be incrementable. > > > > -------------------------------------------- > > > > On Tue, 3/1/16, Ajay Chander <[email protected] <javascript:;>> > wrote: > > > > > > > > Subject: Question_about_Sqoop_Avro > > > > To: "[email protected] <javascript:;>" <[email protected] > <javascript:;>> > > > > Date: Tuesday, March 1, 2016, 12:01 PM > > > > > > > > Hi Everyone, > > > > I just have a question about using sqoop > > > > incremental operation with --as-avrodatafile. > > > > Let's say initially I did a sqoop import to > > > > pull the table 'xyz' from mysql db to HDFS with > > > > option '--as-avrodatafile'. Now let us say that I > > have some new data > > > > added to source system(xyz table in MySQL) now I want > > to > > > > run a sqoop incremental job to get those changes updated > > in > > > > HDFS with option '--as-avrodatafile'. When I > > try to do that it > > > > says '--incremental lastmodified cannot be used in > > > > conjunction with > > > > --as-avrodatafile' > > > > Any > > > > pointers are appreciated. Thanks. > > > > > >
