Anyone here have done sqoop incremental operations against Avro or Parquet formats. I would like to know if this can be done. If not, any workarounds ? Thank you!
On Tuesday, March 1, 2016, Ajay Chander <[email protected]> wrote: > Thanks for the info! Would it work well against parquet tables? > > On Tuesday, March 1, 2016, Jarek Jarcec Cecho <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: > >> SQOOP-1094 got resolved and will be available in our next release (1.4.7). >> >> Jarcec >> >> > On Mar 1, 2016, at 6:32 PM, Deepak Vohra <[email protected]> wrote: >> > >> > Incremental sqoop without duplication would require merge of avro >> files, which is not supported. >> > https://issues.apache.org/jira/browse/SQOOP-1094 >> > -------------------------------------------- >> > On Tue, 3/1/16, Ajay Chander <[email protected]> wrote: >> > >> > Subject: Re: Question_about_Sqoop_Avro >> > To: "[email protected]" <[email protected]>, "Deepak Vohra" < >> [email protected]> >> > Date: Tuesday, March 1, 2016, 5:41 PM >> > >> > Can someone shed some >> > lights on this? I am trying to perform incremental operation >> > through sqoop to write the data in Avro format. >> > Thanks >> > >> > On >> > Tuesday, March 1, 2016, Ajay Chander <[email protected]> >> > wrote: >> > But when >> > I do the same with '--incremental append' it is >> > working fine except that I see duplicates which is as >> > expected. >> > On Tuesday, >> > March 1, 2016, Deepak Vohra <[email protected]> wrote: >> > '--incremental lastmodified >> > cannot be used in conjunction with --as-avrodatafile' >> > >> > Being binary data avro data may not be incrementable. >> > >> > -------------------------------------------- >> > >> > On Tue, 3/1/16, Ajay Chander <[email protected]> wrote: >> > >> > >> > >> > Subject: Question_about_Sqoop_Avro >> > >> > To: "[email protected]" <[email protected]> >> > >> > Date: Tuesday, March 1, 2016, 12:01 PM >> > >> > >> > >> > Hi Everyone, >> > >> > I just have a question about using sqoop >> > >> > incremental operation with --as-avrodatafile. >> > >> > Let's say initially I did a sqoop import to >> > >> > pull the table 'xyz' from mysql db to HDFS with >> > >> > option '--as-avrodatafile'. Now let us say that I >> > have some new data >> > >> > added to source system(xyz table in MySQL) now I want >> > to >> > >> > run a sqoop incremental job to get those changes updated >> > in >> > >> > HDFS with option '--as-avrodatafile'. When I >> > try to do that it >> > >> > says '--incremental lastmodified cannot be used in >> > >> > conjunction with >> > >> > --as-avrodatafile' >> > >> > Any >> > >> > pointers are appreciated. Thanks. >> > >> > >> >>
