----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65239/ -----------------------------------------------------------
(Updated Sept. 5, 2018, 4:37 p.m.) Review request for pig, Daniel Dai, Koji Noguchi, Rohini Palaniswamy, and Adam Szita. Repository: pig-git Description ------- This is an initial patch that adds Hadoop 3 support to Pig in addition to Hadoop 2. Major modifications: * No breaking API change was introduced in Hadoop 3, the current code compiles with Hadoop 3 * hadoopversion property tells which mode the tests should run, the default is hadoop 2 * Hadoop 3 introduced a security fix, only whitelisted environment variables are passed to MiniCluster * In Hadoop 3 hadoop-site.xml is deprecated, and is replaced by core-site.xml, hdfs-site.xml and mapred-site.xml. I decided to write the config into all of these files in MiniCluster.java (into hadoop-site.xml too to stay compatible with Hadoop 2) for the shake of simplicity, we might want to have different files for Hadoop 2 and separate the properties for Hadoop 3. * TestErrorHandling.java: small format change in error message, modified the assert so it works on both on Hadoop 2 and Hadoop 3 * HadoopShims: code is identical with Hadoop 2, not sure if we need shims any more. I think we should move it to the src instead. * Split properties into 3 files: common properties, Hadoop 2 and Hadoop 3 specific properties * ivy.xml: new config for Hadoop 3 * build.xml: new target to package both hadoop2 and hadoop3 - not sure that this is needed, if we move shims, the I think we don't need this target * HBase unit test fails on Hadoop 3 (as per https://hbase.apache.org/book.html HBase 1.x is not tested against Hadoop 2) Diffs (updated) ----- bin/pig 3fcf165106cccbe75fc1c61ea74732456ae50fc7 bin/pig.py b6c396579c54359f430c6e74d055ec7f27ae2197 build.xml 34697ee5fc7a9fff9a0a6359e10ccb9fa3accb47 ivy.xml 0902b18040de579362abd2e785bdb42ac189a8b0 ivy/libraries-h2.properties PRE-CREATION ivy/libraries-h3.properties PRE-CREATION ivy/libraries.properties ec71472be7e885b8e03b03430c4d274ba84541ea test/org/apache/pig/parser/TestErrorHandling.java 15e09031c360cea5f81609129ac3a6d38d68d3ea test/org/apache/pig/parser/TestQueryParserUtils.java 1c217e3cab9c4b5dc51289a883aa696dcd2feeea test/org/apache/pig/test/MapReduceMiniCluster.java PRE-CREATION test/org/apache/pig/test/MiniCluster.java a7532ad750f06ffae5a03024b1658ff77152c902 test/org/apache/pig/test/MiniGenericCluster.java 674860f880407595d68c4eea2b67e2d6465417fe test/org/apache/pig/test/Util.java 788a72fe3ceca08ec61ae425a393b5b0936454f4 test/org/apache/pig/test/YarnMiniCluster.java 69d808124a4e9be661f1fda25755075dcb6607b1 Diff: https://reviews.apache.org/r/65239/diff/3/ Changes: https://reviews.apache.org/r/65239/diff/2-3/ Testing ------- Thanks, Nandor Kollar
