[
https://issues.apache.org/jira/browse/SQOOP-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14250137#comment-14250137
]
Veena Basavaraj commented on SQOOP-1168:
----------------------------------------
So there are a couple of high level question I would like to answer before
going into every detail.
One thing so far everyone seems to have agreed on is that each connector will
say if it supports delta fetch and delta merge in FROM/ TO directions
correspondingly.
As far as Terminology, merge is more relevant since it conveys that it is upto
the connector how to interpret the merge. If one understands what the configs
for delta merge/ fetch are then it is probably more clear
1. The design wiki alludes at Predicate, but they are more than what we really
need. We already have a term "Config" that is used by connectors to say what
are the parameters/ inputs they are expecting to finally create a job. So if a
connector wants to support different forms of delta fetches and delta merges,
it can expose as many config classes as it wants. This is what I meant by
supported delta configs. It is nothing more than a bunch of config objects a
connector will expose.
So what is an example of config? For JDBC is might be like this
{code}
*/
@ConfigClass(validators = {
@Validator(DeltaFetchConfig2.DeltaFetchConfig2Validator.class)})
public class DeltaFetchConfig2 {
@Input(size = 255) public String column;
@Input(size = 255) public Operator operator;
@Input public Sting value;
}
{code}
or
{code}
@ConfigClass(validators = {
@Validator(DeltaFetchConfig.DeltaFetchConfigValidator.class)})
public class DeltaFetchConfig2 {
@Input(size = 255) public String deltaFetchQuery;
{code}
Its upto the connector to define its semantics, No hard rules at all.
Note, it can define more than one config and each of its parameters will have
their own validators and and it can say what is required/optional/ default or
having any of the conditions we want to supports in general for configs. So all
of those can be reused here, there is no extra efforts.
Implementation wise, we already know that the FromJobConfiguration and
ToJobConfiguration is a grouping/list of configs, so we easily support as many
as we want for both DeltaFetch and DeltaMerge. The most flexible thing is they
can call this whatever they want, so if some one writing Foo Connector, wants
to call it "IncrementalAppendConfig" they can.
remaining things I will answer per person who has questions
> Sqoop2: Delta Fetch/ Merge ( formerly called Incremental Import )
> -----------------------------------------------------------------
>
> Key: SQOOP-1168
> URL: https://issues.apache.org/jira/browse/SQOOP-1168
> Project: Sqoop
> Issue Type: Bug
> Reporter: Hari Shreedharan
> Assignee: Veena Basavaraj
> Fix For: 1.99.5
>
>
> The formal design wiki is here
> https://cwiki.apache.org/confluence/display/SQOOP/Delta+Fetch+and+Merge+Design
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
