----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23140/#review47023 -----------------------------------------------------------
I see a lot of the changes are related to aggregators, and you write them now from master, worker and MasterCompute - can't we write them just once and go through normal path of distributing them in the beginning of the superstep? - Maja Kabiljo On June 27, 2014, 8:48 p.m., Sergey Edunov wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23140/ > ----------------------------------------------------------- > > (Updated June 27, 2014, 8:48 p.m.) > > > Review request for giraph. > > > Repository: giraph-git > > > Description > ------- > > This fix merely makes checkpointing work again. > > > Diffs > ----- > > giraph-core/src/main/java/org/apache/giraph/aggregators/Aggregator.java > 514e470 > > giraph-core/src/main/java/org/apache/giraph/aggregators/AggregatorHandler.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/aggregators/AggregatorWrapper.java > 9613805 > > giraph-core/src/main/java/org/apache/giraph/aggregators/BasicAggregator.java > 07a4100 > giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java 2e35373 > giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java f0ecca2 > > giraph-core/src/main/java/org/apache/giraph/comm/aggregators/AllAggregatorServerData.java > 177e738 > giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java > 7d7ceb2 > giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java > ad7e045 > > giraph-core/src/main/java/org/apache/giraph/master/DefaultMasterCompute.java > bfb6f0e > > giraph-core/src/main/java/org/apache/giraph/master/MasterAggregatorHandler.java > 325d91f > giraph-core/src/main/java/org/apache/giraph/master/MasterCompute.java > d77a9b5 > > giraph-core/src/main/java/org/apache/giraph/master/WritableMasterAggregatorUsage.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/partition/BasicPartitionOwner.java > 545d1af > > giraph-core/src/main/java/org/apache/giraph/partition/HashMasterPartitioner.java > 240687e > > giraph-core/src/main/java/org/apache/giraph/partition/HashWorkerPartitioner.java > d833895 > > giraph-core/src/main/java/org/apache/giraph/partition/MasterGraphPartitioner.java > 50c750a > > giraph-core/src/main/java/org/apache/giraph/partition/PartitionBalancer.java > 3454d62 > giraph-core/src/main/java/org/apache/giraph/partition/PartitionOwner.java > 0ac74da > > giraph-core/src/main/java/org/apache/giraph/partition/SimpleMasterPartitioner.java > f128f34 > > giraph-core/src/main/java/org/apache/giraph/partition/SimpleWorkerPartitioner.java > 3c0de44 > > giraph-core/src/main/java/org/apache/giraph/partition/WorkerGraphPartitioner.java > 004ea81 > giraph-core/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java > 09dd46d > > giraph-core/src/main/java/org/apache/giraph/utils/io/ExtendedDataInputOutput.java > af45426 > giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java > 8dcf19a > > giraph-core/src/main/java/org/apache/giraph/worker/WorkerAggregatorHandler.java > 9bfd7b5 > giraph-core/src/main/java/org/apache/giraph/worker/WorkerContext.java > 17347db > > giraph-core/src/main/java/org/apache/giraph/worker/WorkerThreadAggregatorUsage.java > 194127e > > giraph-core/src/main/java/org/apache/giraph/worker/WritableWorkerAggregatorUsage.java > PRE-CREATION > > giraph-core/src/test/java/org/apache/giraph/partition/SimpleRangePartitionFactoryTest.java > 96bd5d7 > > giraph-examples/src/test/java/org/apache/giraph/aggregators/TestAggregatorsHandling.java > e2b611b > > Diff: https://reviews.apache.org/r/23140/diff/ > > > Testing > ------- > > I tested it running multiple different jobs. I run page rank on 2*10^9 > vertices on 200 workers and it seems to work just fine. It only takes 2 > minutes to save checkpoint. > > > Thanks, > > Sergey Edunov > >
