[
https://issues.apache.org/jira/browse/PIG-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332881#comment-16332881
]
Rohini Palaniswamy commented on PIG-5253:
-----------------------------------------
Copying [~nkollar]'s comments from review board here
This is an initial patch that adds Hadoop 3 support to Pig in addition to
Hadoop 2.
Major modifications:
* No breaking API change was introduced in Hadoop 3, the current code compiles
with Hadoop 3
* hadoopversion property tells which mode the tests should run, the default is
hadoop 2
* Hadoop 3 introduced a security fix, only whitelisted environment variables
are passed to MiniCluster
* In Hadoop 3 hadoop-site.xml is deprecated, and is replaced by core-site.xml,
hdfs-site.xml and mapred-site.xml. I decided to write the config into all of
these files in MiniCluster.java (into hadoop-site.xml too to stay compatible
with Hadoop 2) for the shake of simplicity, we might want to have different
files for Hadoop 2 and separate the properties for Hadoop 3.
* TestErrorHandling.java: small format change in error message, modified the
assert so it works on both on Hadoop 2 and Hadoop 3
* HadoopShims: code is identical with Hadoop 2, not sure if we need shims any
more. I think we should move it to the src instead.
* Split properties into 3 files: common properties, Hadoop 2 and Hadoop 3
specific properties
* ivy.xml: new config for Hadoop 3
* build.xml: new target to package both hadoop2 and hadoop3 - not sure that
this is needed, if we move shims, the I think we don't need this target
* HBase unit test fails on Hadoop 3 (as per https://hbase.apache.org/book.html
HBase 1.x is not tested against Hadoop 2)
> Pig Hadoop 3 support
> --------------------
>
> Key: PIG-5253
> URL: https://issues.apache.org/jira/browse/PIG-5253
> Project: Pig
> Issue Type: Improvement
> Reporter: Nandor Kollar
> Assignee: Nandor Kollar
> Priority: Major
> Fix For: 0.18.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
