[
https://issues.apache.org/jira/browse/SQOOP-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated SQOOP-2632:
---------------------------
Description:
Each {{SqoopMapper}} will process one {{SqoopSplit}}. {{SqoopSplit}} is
instantiated by the MR framework. Let's suppose that MR framework loads
{{SqoopSplit}} with classloader A. As {{SqoopSplit}} has one field
{{partition}}, so when {{SqoopSplit}} is loaded by classloader A, class
{{Partition}} will also be loaded by classloader A and field {{partition}} will
be of type {{A.Partition}}.
In {{SqoopMapper}}, we will call {{extractor.extract(extractorContext,
fromConfig, fromJob, partition)}}. To provide classpath isolation, we will load
class {{Extractor}} with a custom classloader {{B}}, so class {{Partition}} in
{{Extractor}} will be loaded by classloader {{B}}. This will cause type
miss-match between {{A.Partition}} and {{B.Partition}} when we call
{{extractor.extract(extractorContext, fromConfig, fromJob, partition}}.
To solve this issue, we need to make sure class {{Partition}} not be loaded by
the custom classloader {{B}}. To achieve this, we need firstly provide a
mechanism to let {{SqoopMapper}} know the class name of {{Partition}}.
was:
Each {{SqoopMapper}} will process one {{SqoopSplit}}. {{SqoopSplit}} is
instantiated by the MR framework. Let's suppose that MR framework loads
{{SqoopSplit}} using classloader A. As {{SqoopSplit}} has one field
{{partition}}, so when {{SqoopSplit}} is loaded by classloader A, class
{{Partition}} will also be loaded by classloader A and field {{partition}} will
be of type {{A.Partition}}.
In {{SqoopMapper}}, we will call {{extractor.extract(extractorContext,
fromConfig, fromJob, partition)}}. To provide classpath isolation, we will load
class {{Extractor}} with a custom classloader {{B}}, so the {{Partition}} class
in {{Extractor}} will be loaded by classloader {{B}} by default. This will
cause type miss-match between {{A.Partition}} and {{B.Partition}}.
To resolve this issue, we need to make sure class {{Partition}} not be loaded
by the custom classloader {{B}}. To do this, we need provide a method to let
{{SqoopMapper}} know the class name of {{Partition}}.
> Sqoop2: add partition class to FROM
> -----------------------------------
>
> Key: SQOOP-2632
> URL: https://issues.apache.org/jira/browse/SQOOP-2632
> Project: Sqoop
> Issue Type: Sub-task
> Reporter: Dian Fu
> Assignee: Dian Fu
> Fix For: 1.99.7
>
> Attachments: SQOOP-2632.001.patch
>
>
> Each {{SqoopMapper}} will process one {{SqoopSplit}}. {{SqoopSplit}} is
> instantiated by the MR framework. Let's suppose that MR framework loads
> {{SqoopSplit}} with classloader A. As {{SqoopSplit}} has one field
> {{partition}}, so when {{SqoopSplit}} is loaded by classloader A, class
> {{Partition}} will also be loaded by classloader A and field {{partition}}
> will be of type {{A.Partition}}.
> In {{SqoopMapper}}, we will call {{extractor.extract(extractorContext,
> fromConfig, fromJob, partition)}}. To provide classpath isolation, we will
> load class {{Extractor}} with a custom classloader {{B}}, so class
> {{Partition}} in {{Extractor}} will be loaded by classloader {{B}}. This will
> cause type miss-match between {{A.Partition}} and {{B.Partition}} when we
> call {{extractor.extract(extractorContext, fromConfig, fromJob, partition}}.
> To solve this issue, we need to make sure class {{Partition}} not be loaded
> by the custom classloader {{B}}. To achieve this, we need firstly provide a
> mechanism to let {{SqoopMapper}} know the class name of {{Partition}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
