----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34636/ -----------------------------------------------------------
Review request for DataFu. Repository: datafu Description ------- This updates DataFu to use Hadoop 2.7.0 and Pig 0.14.0. I've fixed all issues that I could find. All the unit tests pass. Some of the issues were already addressd by Daniel Dai's earlier patch, but I found some more problems. Summary of issues: * Hourglass tests failed because we were pulling in avro-tools, which contains Hadoop classes and therefore conflicts. I removed the dependency. I'm not sure why we had this. I also added a handy script to help with this analysis, which uses jarfish. * Hourglass used .toString() for many Path instances. This caused a problem because the resulting string is prefixed with "file:" and therefore isn't a valid path you can use in File. The general fix is to use getName() instead, which gets the simple name and is usually suitable. * Some Pig unit tests had assumptions on tuple ordering in bags. I applied a sort to the actual and expected bags to ensure a consistent comparison. Diffs ----- README.md 8e1b67d build-plugin/src/main/java/org/adrianwalker/multilinestring/MultilineProcessor.java 9abdba5 datafu-hourglass/.gitignore 942515e datafu-hourglass/build.gradle 75a2876 datafu-hourglass/find_dupes.rb PRE-CREATION datafu-hourglass/src/main/java/datafu/hourglass/fs/PathUtils.java c270c7b datafu-hourglass/src/main/java/datafu/hourglass/mapreduce/DistributedCacheHelper.java 62975d1 datafu-hourglass/src/test/java/datafu/hourglass/demo/Examples.java 039822c datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionCollapsingExecutionPlannerTests.java d68ea83 datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionCollapsingJoinTest.java 02aa342 datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionCollapsingTests.java fff1cfd datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionPreservingCollapsingIntegrationTests.java a8f020b datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionPreservingJoinTests.java c41fd39 datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionPreservingTests.java acae96c datafu-hourglass/src/test/java/datafu/hourglass/test/TestAvroJob.java b428003 datafu-hourglass/src/test/java/datafu/hourglass/test/TestBase.java bc52977 datafu-pig/build.gradle ea385d2 datafu-pig/src/test/java/datafu/test/pig/bags/BagTests.java 9bcc384 gradle/dependency-versions.gradle 3b0835f Diff: https://reviews.apache.org/r/34636/diff/ Testing ------- ./gradlew test Thanks, Matthew Hayes
