> Oh, great! So, just get the last value from the old job before it's wiped, > set it up as incremental append and also the last-value arg, and it will > sort it out? That's good to know.
Yup, exactly. Jarcec On Tue, Feb 18, 2014 at 02:24:22PM -0500, Devin Suiter RDX wrote: > Oh, great! So, just get the last value from the old job before it's wiped, > set it up as incremental append and also the last-value arg, and it will > sort it out? That's good to know. I thought maybe sqoop -merge to flatten > the sets if you had to start over would be one way to have the same import > directory with everything from both imports once everything was finished. > This way is cleaner and easier. I know Sqoop 2 is better for this sort of > thing, not sure why we ended up using Sqoop 1. > > Thanks for the reply! > > *Devin Suiter* > Jr. Data Solutions Software Engineer > 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212 > Google Voice: 412-256-8556 | www.rdx.com > > > On Tue, Feb 18, 2014 at 2:15 PM, Jarek Jarcec Cecho <[email protected]>wrote: > > > Hi Devin, > > Indeed you are correct. The original Sqoop 1 metastore do not supports > > updates of the stored jobs and the only way how to achieve update is to > > delete the old variant and re-create it. I would like to mention that we've > > significantly improved this concept in Sqoop 2, where the metastore (now > > called repository) is a first class citizen that offers (not only) the > > update capability. > > > > You do not have to be concerned about re-creating incremental jobs in > > Sqoop 1 though. You can use the --last-value argument to instruct Sqoop not > > starting from scratch when re-creating the saved jobs. Sqoop is smart > > enough to realize that this is just the original state that will change on > > every incremental import. > > > > Jarcec > > > > On Thu, Jan 30, 2014 at 09:52:06PM -0500, Devin Suiter RDX wrote: > > > Hello, > > > > > > I am exploring Sqoop, and wondering about a hypothetical problem. What > > > happens if someone needs to change a saved incremental append job in > > Sqoop > > > 1? Say for example the connection string needed to be changed because of > > a > > > network change or something. Shouldn't do that to your cluster, I know, > > > but, things happen. > > > > > > From what I have seen, there isn't any way to edit an in-place job...only > > > to delete it and create a new one. If you have to do that, I'm guessing > > the > > > metastore will drop the history of the incremental append last value. If > > > you build a new incremental append job, that will bring everything over > > > from the beginning, which may not be good to do. If you run an > > incremental > > > import specifying the "from last value" parameter, that won't be a good > > for > > > a saved job, will it? Wouldn't it just begin from that value every time? > > > > > > Is there a way to edit saved job parameters in the metastore directly so > > > you can update the connection string or inject the "last value" value in > > > the new job history after you create it but before running it the first > > > time? Or is something saved by job name? > > > > > > Just wondering if there is a solution here that isn't really apparent in > > > the documentation... > > > > > > Thanks, > > > *Devin Suiter* > > > Jr. Data Solutions Software Engineer > > > 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212 > > > Google Voice: 412-256-8556 | www.rdx.com > >
signature.asc
Description: Digital signature
