----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22234/ -----------------------------------------------------------
Review request for giraph, Avery Ching, Sergey Edunov, Igor Kabiljo, and Maja Kabiljo. Repository: giraph-git Description ------- There are many changes here: A new input format - MappingInputFormat - and related dependencies has been defined New hive input format classes to read Mapping table + some examples which can be used for running hellopagerank with modifications in test plan have been defined Changes to main giraph classes to read the mapping & use it for getting partition info - in worker, master & partition sections New mapping section defined to define & declare MappingStore format & some sample implementations The code can take 2 paths based on what the user wants Embed info into vertexId by implementing proper contracts (TranslateEdge.interface + 2 methods in MappingStore) then use an EmbeddedGraphPartitioner to directly read worker info off of vertex ids [translate once & user freely for the rest of app] Always read from the MappingStore (this has more overhead probably because of cache misses in processor) [never translate but pay cost of map lookup each time partition / worker info is needed] Diffs ----- giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ec0ddbb giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java bda967d giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 3337621 giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 6b36418 giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java 95e029d giraph-core/src/main/java/org/apache/giraph/io/MappingInputFormat.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/io/MappingReader.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedMappingInputFormat.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedMappingReader.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/io/iterables/MappingReaderWrapper.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/mapping/AbstractLongByteOps.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/mapping/DefaultEmbeddedLongByteOps.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/mapping/DefaultLongByteOps.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/mapping/LongByteMappingStore.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/mapping/MappingEntry.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/mapping/MappingStore.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/mapping/MappingStoreOps.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/mapping/package-info.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/mapping/translate/LongByteTranslateEdge.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/mapping/translate/TranslateEdge.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/mapping/translate/package-info.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 90dc9f3 giraph-core/src/main/java/org/apache/giraph/master/MasterThread.java 15dbe07 giraph-core/src/main/java/org/apache/giraph/partition/DefaultLongBytePartitionerFactory.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/partition/GraphPartitionerFactory.java 4200d79 giraph-core/src/main/java/org/apache/giraph/partition/HashPartitionerFactory.java 7cc5651 giraph-core/src/main/java/org/apache/giraph/partition/HashRangePartitionerFactory.java 1eeece7 giraph-core/src/main/java/org/apache/giraph/partition/SimpleIntRangePartitionerFactory.java 8ab692f giraph-core/src/main/java/org/apache/giraph/partition/SimpleLongRangePartitionerFactory.java 2989598 giraph-core/src/main/java/org/apache/giraph/partition/SimplePartitionerFactory.java 15b0756 giraph-core/src/main/java/org/apache/giraph/partition/SimpleWorkerPartitioner.java 600d7a3 giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java aff7084 giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java 828eac4 giraph-core/src/main/java/org/apache/giraph/worker/FullInputSplitCallable.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/worker/LocalData.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/worker/MappingInputSplitsCallable.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/worker/MappingInputSplitsCallableFactory.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java e3e04d6 giraph-core/src/test/java/org/apache/giraph/partition/SimpleRangePartitionFactoryTest.java 4e19cd2 giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java 603910b giraph-hive/src/main/java/org/apache/giraph/hive/common/GiraphHiveConstants.java c7ad63b giraph-hive/src/main/java/org/apache/giraph/hive/common/HiveUtils.java 2388673 giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/AbstractHiveToMapping.java PRE-CREATION giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveMappingInputFormat.java PRE-CREATION giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveMappingReader.java PRE-CREATION giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveToMapping.java PRE-CREATION giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/SimpleHiveToMapping.java PRE-CREATION giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/LongByteHiveToMapping.java PRE-CREATION giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/LongInt2ByteHiveToMapping.java PRE-CREATION giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/package-info.java PRE-CREATION giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/package-info.java PRE-CREATION Diff: https://reviews.apache.org/r/22234/diff/ Testing ------- ran pagerank jobs multiple times mvn clean verify Thanks, Pavan Kumar Athivarapu
