[
https://issues.apache.org/jira/browse/SQOOP-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14380306#comment-14380306
]
Veena Basavaraj commented on SQOOP-1803:
----------------------------------------
Took a close look at the patch today. Here are few things that probably differ
from the approach I am using.
1. The Initializer context that extends MutableContext has been modified in my
case, So the destroyer API has no changes.
2. I use convention to indicate the key, for instance the key used for
lastvalue to store its value in the MutableContext, will be
"incrementalConfig.lastValue", so this allows me to use the updateFromJobConfig
API rather than using the entire job update, the updateJob is not necessary in
this case, a granular updateJobConfig input can be done.
3. The code to actually do the final repo call is pretty much in the JobManager
as it is in you patch. The alternative I tried was to put this code in the
commitJob method of the outputcommiter class, using the same semantic as used
to invoke the destroyer code. Here is the destroyer has access to the
MutableContext from the Configuration object.
So If I were to highlight the differences it does come down to
1. Which api do we want to change?
2. Do we want to update the entire job object? or just the config inputs that
the connector wants to store
3. Where do we invoke the "updateJob / update", in Jobmanager UpdateThread? or
inside the OutputCommtter
I should post the RB soon once I have SQOOP-2113 and its sub ticket questions
resolved. Thank you for the patience.
> JobManager and Execution Engine changes: Support for a injecting and pulling
> out configs and job output in connectors
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: SQOOP-1803
> URL: https://issues.apache.org/jira/browse/SQOOP-1803
> Project: Sqoop
> Issue Type: Sub-task
> Reporter: Veena Basavaraj
> Assignee: Veena Basavaraj
> Fix For: 1.99.6
>
> Attachments: SQOOP-1803-POC.patch
>
>
> The details are in the design wiki, as the implementation happens more
> discussions can happen here.
> https://cwiki.apache.org/confluence/display/SQOOP/Delta+Fetch+And+Merge+Design#DeltaFetchAndMergeDesign-Howtogetoutputfromconnectortosqoop?
> The goal is to dynamically inject a IncrementalConfig instance into the
> FromJobConfiguration. The current MFromConfig and MToConfig can already hold
> a list of configs, and a strong sentiment was expressed to keep it as a list,
> why not for the first time actually make use of it and group the incremental
> related configs in one config object
> This task will prepare the FromJobConfiguration from the job config data,
> ExtractorContext with the relevant values from the prev job run
> This task will prepare the ToJobConfiguration from the job config data,
> LoaderContext with the relevant values from the prev job run if any
> We will use DistributedCache to get State information from the Extractor and
> Loader out and finally persist it into the sqoop repository depending on
> SQOOP-1804 once the outputcommitter commit is called
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
