I faced the same issue, I ended up by running a shell-action (through oozie) after each execution of sqoop.
Inside my option parameter file I had something like this: --whereID >= A and ID <= B After each execution of sqoop, I run a shell script which modifies the "where" clause parameter by the chunk size based on my need. So for each sqoop run, my option parameter becomes: run:1--whereID >= 1 and ID <= 20 run:2--whereID >= 21 and ID <= 40 run:3--whereID >= 41 and ID <= 60 where chunk size = 20. It worked for me! - Tanzir Date: Wed, 15 May 2013 00:02:51 +0530 Subject: Re: Fwd: Incremental Import for Sqoop2 From: [email protected] To: [email protected] Hi, Try using Boundary query or sql statement while creating import job for filtering records. For example: Importing id greater than 100 Boundary query: select min(id),max(id) from table where id>100 Sorry if I am wrong. On Tue, May 14, 2013 at 11:06 PM, Joanne Chan <[email protected]> wrote: Hi Jarek, Thanks for the speedy response! Any idea what the timeline is for this feature? I am trying to utilize sqoop2 to pull new/updated data from mysql to hdfs. Do you know of any workarounds for sqoop2 to get this job done without having to use incremental import? Appreciate your input! On Tue, May 14, 2013 at 1:23 PM, Jarek Jarcec Cecho <[email protected]> wrote: Hi sir, Sqoop 2 currently can't do incremental imports, however implementing this feature is definitely in our plan! Jarcec On Tue, May 14, 2013 at 01:14:32PM -0400, Joanne Chan wrote: > Is incremental import available for sqoop2? If not, is that part of the > plan? > > I can't seem to find any good example of how to create import job with > incremental parameter. > > I am on: > client version: > Sqoop 2.0.0-SNAPSHOT revision c0a43d436fdd9293bf10e4b874d1002441c056b1 > server version: > Sqoop 2.0.0-SNAPSHOT revision c0a43d436fdd9293bf10e4b874d1002441c056b1 -- -- JChan -- Regards Vasanth kumar RJ
