----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/22234/#review44791 -----------------------------------------------------------
giraph-core/src/main/java/org/apache/giraph/partition/DefaultLongBytePartitionerFactory.java <https://reviews.apache.org/r/22234/#comment79329> Lets rename it to something meaningful - MappingStoreLongPartitionerFactory There is no reason to have ByteWritable anywhere in it, just put '?'. - Igor Kabiljo On June 4, 2014, 3:11 p.m., Pavan Kumar Athivarapu wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/22234/ > ----------------------------------------------------------- > > (Updated June 4, 2014, 3:11 p.m.) > > > Review request for giraph, Avery Ching, Sergey Edunov, Igor Kabiljo, and Maja > Kabiljo. > > > Repository: giraph-git > > > Description > ------- > > There are many changes here: > > A new input format - MappingInputFormat - and related dependencies has been > defined > New hive input format classes to read Mapping table + some examples which can > be used for running hellopagerank with modifications in test plan have been > defined > Changes to main giraph classes to read the mapping & use it for getting > partition info - in worker, master & partition sections > New mapping section defined to define & declare MappingStore format & some > sample implementations > The code can take 2 paths based on what the user wants > > Embed info into vertexId by implementing proper contracts > (TranslateEdge.interface + 2 methods in MappingStore) > then use an EmbeddedGraphPartitioner to directly read worker info off of > vertex ids [translate once & user freely for the rest of app] > > Always read from the MappingStore (this has more overhead probably because of > cache misses in processor) [never translate but pay cost of map lookup each > time partition / worker info is needed] > > > Diffs > ----- > > giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ec0ddbb > > giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java > bda967d > giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 3337621 > giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java > 6b36418 > > giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java > 95e029d > giraph-core/src/main/java/org/apache/giraph/io/MappingInputFormat.java > PRE-CREATION > giraph-core/src/main/java/org/apache/giraph/io/MappingReader.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedMappingInputFormat.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedMappingReader.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/io/iterables/MappingReaderWrapper.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/mapping/AbstractLongByteOps.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/mapping/DefaultEmbeddedLongByteOps.java > PRE-CREATION > giraph-core/src/main/java/org/apache/giraph/mapping/DefaultLongByteOps.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/mapping/LongByteMappingStore.java > PRE-CREATION > giraph-core/src/main/java/org/apache/giraph/mapping/MappingEntry.java > PRE-CREATION > giraph-core/src/main/java/org/apache/giraph/mapping/MappingStore.java > PRE-CREATION > giraph-core/src/main/java/org/apache/giraph/mapping/MappingStoreOps.java > PRE-CREATION > giraph-core/src/main/java/org/apache/giraph/mapping/package-info.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/mapping/translate/LongByteTranslateEdge.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/mapping/translate/TranslateEdge.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/mapping/translate/package-info.java > PRE-CREATION > giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java > 90dc9f3 > giraph-core/src/main/java/org/apache/giraph/master/MasterThread.java > 15dbe07 > > giraph-core/src/main/java/org/apache/giraph/partition/DefaultLongBytePartitionerFactory.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/partition/GraphPartitionerFactory.java > 4200d79 > > giraph-core/src/main/java/org/apache/giraph/partition/HashPartitionerFactory.java > 7cc5651 > > giraph-core/src/main/java/org/apache/giraph/partition/HashRangePartitionerFactory.java > 1eeece7 > > giraph-core/src/main/java/org/apache/giraph/partition/SimpleIntRangePartitionerFactory.java > 8ab692f > > giraph-core/src/main/java/org/apache/giraph/partition/SimpleLongRangePartitionerFactory.java > 2989598 > > giraph-core/src/main/java/org/apache/giraph/partition/SimplePartitionerFactory.java > 15b0756 > > giraph-core/src/main/java/org/apache/giraph/partition/SimpleWorkerPartitioner.java > 600d7a3 > giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java > aff7084 > > giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java > 828eac4 > > giraph-core/src/main/java/org/apache/giraph/worker/FullInputSplitCallable.java > PRE-CREATION > giraph-core/src/main/java/org/apache/giraph/worker/LocalData.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/worker/MappingInputSplitsCallable.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/worker/MappingInputSplitsCallableFactory.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java > e3e04d6 > > giraph-core/src/test/java/org/apache/giraph/partition/SimpleRangePartitionFactoryTest.java > 4e19cd2 > giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java > 603910b > > giraph-hive/src/main/java/org/apache/giraph/hive/common/GiraphHiveConstants.java > c7ad63b > giraph-hive/src/main/java/org/apache/giraph/hive/common/HiveUtils.java > 2388673 > > giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/AbstractHiveToMapping.java > PRE-CREATION > > giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveMappingInputFormat.java > PRE-CREATION > > giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveMappingReader.java > PRE-CREATION > > giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveToMapping.java > PRE-CREATION > > giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/SimpleHiveToMapping.java > PRE-CREATION > > giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/LongByteHiveToMapping.java > PRE-CREATION > > giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/LongInt2ByteHiveToMapping.java > PRE-CREATION > > giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/package-info.java > PRE-CREATION > > giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/package-info.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/22234/diff/ > > > Testing > ------- > > ran pagerank jobs multiple times > mvn clean verify > > > Thanks, > > Pavan Kumar Athivarapu > >
