----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16402/ -----------------------------------------------------------
(Updated Feb. 3, 2014, 10:47 a.m.) Review request for helix, Zhen Zhang and Kishore Gopalakrishna. Changes ------- Updated the diff to reflect the final changes in the 0.6 version. Unfortunately the full diff here is much larger than the 0.6 version for the following reasons: - Need to make use of a global ClusterDataCache in the ClusterAccessor and the pipeline - Caching rebalancer instances - Fixing the issue where resource assignments were persisted unnecessarily and inefficiently - Reducing the number of conversions between typed and string maps Bugs: HELIX-345 Repository: helix-git Description ------- Optimized the following: - cached messages across pipeline runs - cached the HelixRebalancer instance for each resource - spawned threads to read resources and participants in parallel - started using caching and batching when persisting resource assignments - instrumented all pipeline stages with the timing logs commit 5758421ce71c37118423a543bf15569f4c697019 Author: Kanak Biscuitwala <[email protected]> Date: Thu Dec 19 13:52:56 2013 -0800 [HELIX-345] Speed up the controller pipelines for 0.7 :100644 100644 36c7aaa... 38da1b9... M helix-core/src/main/java/org/apache/helix/api/accessor/ClusterAccessor.java :100644 100644 ec812b2... fcbf45f... M helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java :100644 100644 532ecb5... 6cdd304... M helix-core/src/main/java/org/apache/helix/controller/stages/CompatibilityCheckStage.java :100644 100644 8235173... a888dc7... M helix-core/src/main/java/org/apache/helix/controller/stages/CurrentStateComputationStage.java :100644 100644 ef5a5fd... 97734a6... M helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationStage.java :100644 100644 966160c... 0ce132b... M helix-core/src/main/java/org/apache/helix/controller/stages/MessageSelectionStage.java :100644 100644 764b422... f941826... M helix-core/src/main/java/org/apache/helix/controller/stages/MessageThrottleStage.java :100644 100644 45fc355... fd37a2f... M helix-core/src/main/java/org/apache/helix/controller/stages/PersistAssignmentStage.java :100644 100644 7c2cc0f... ece4209... M helix-core/src/main/java/org/apache/helix/controller/stages/PersistContextStage.java :100644 100644 2279d76... 378eb61... M helix-core/src/main/java/org/apache/helix/controller/stages/ReadClusterDataStage.java :100644 100644 5b75535... f141aa3... M helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java :100644 100644 470de2c... fbc75be... M helix-core/src/main/java/org/apache/helix/controller/stages/ResourceValidationStage.java :100644 100644 c48f156... 5a4bd2d... M helix-core/src/main/java/org/apache/helix/controller/stages/StatsAggregationStage.java :100644 100644 471530c... a530836... M helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixDataAccessor.java :100644 100644 9ec1ef9... 2d577ca... M helix-core/src/main/java/org/apache/helix/tools/YAMLClusterSetup.java :100644 100644 919eb00... a89d27f... M helix-core/src/test/java/org/apache/helix/Mocks.java Diffs (updated) ----- helix-core/src/main/java/org/apache/helix/ConfigAccessor.java f46e537 helix-core/src/main/java/org/apache/helix/api/Cluster.java 98072d1 helix-core/src/main/java/org/apache/helix/api/accessor/ClusterAccessor.java 36c7aaa helix-core/src/main/java/org/apache/helix/api/accessor/ResourceAccessor.java 80c5b16 helix-core/src/main/java/org/apache/helix/api/config/ClusterConfig.java 22a1528 helix-core/src/main/java/org/apache/helix/controller/GenericHelixController.java 9fef2da helix-core/src/main/java/org/apache/helix/controller/rebalancer/FullAutoRebalancer.java 0c55d45 helix-core/src/main/java/org/apache/helix/controller/rebalancer/RebalancerRef.java 974222d helix-core/src/main/java/org/apache/helix/controller/rebalancer/config/CustomRebalancerConfig.java 73c3ccc helix-core/src/main/java/org/apache/helix/controller/rebalancer/config/FullAutoRebalancerConfig.java 828d509 helix-core/src/main/java/org/apache/helix/controller/rebalancer/config/PartitionedRebalancerConfig.java 2c9769d helix-core/src/main/java/org/apache/helix/controller/rebalancer/config/RebalancerConfigHolder.java 8581732 helix-core/src/main/java/org/apache/helix/controller/rebalancer/config/SemiAutoRebalancerConfig.java bfc3309 helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java ec812b2 helix-core/src/main/java/org/apache/helix/controller/stages/ClusterDataCache.java 6f09d26 helix-core/src/main/java/org/apache/helix/controller/stages/CompatibilityCheckStage.java 532ecb5 helix-core/src/main/java/org/apache/helix/controller/stages/CurrentStateComputationStage.java 8235173 helix-core/src/main/java/org/apache/helix/controller/stages/ExternalViewComputeStage.java 977b661 helix-core/src/main/java/org/apache/helix/controller/stages/MessageGenerationStage.java ef5a5fd helix-core/src/main/java/org/apache/helix/controller/stages/MessageSelectionStage.java 966160c helix-core/src/main/java/org/apache/helix/controller/stages/MessageThrottleStage.java 764b422 helix-core/src/main/java/org/apache/helix/controller/stages/PersistAssignmentStage.java 45fc355 helix-core/src/main/java/org/apache/helix/controller/stages/PersistContextStage.java 7c2cc0f helix-core/src/main/java/org/apache/helix/controller/stages/ReadClusterDataStage.java 2279d76 helix-core/src/main/java/org/apache/helix/controller/stages/ResourceComputationStage.java 5b75535 helix-core/src/main/java/org/apache/helix/controller/stages/ResourceValidationStage.java 470de2c helix-core/src/main/java/org/apache/helix/controller/stages/TaskAssignmentStage.java bc2ee50 helix-core/src/main/java/org/apache/helix/controller/strategy/AutoRebalanceStrategy.java bff7e46 helix-core/src/main/java/org/apache/helix/manager/zk/ZKHelixDataAccessor.java 471530c helix-core/src/main/java/org/apache/helix/model/IdealState.java 173e251 helix-core/src/main/java/org/apache/helix/model/ResourceAssignment.java 6da3dc2 helix-core/src/main/java/org/apache/helix/model/ResourceConfiguration.java 8c5b863 helix-core/src/main/java/org/apache/helix/task/TaskRebalancer.java d1bce56 helix-core/src/main/java/org/apache/helix/task/TaskUtil.java a9428c6 helix-core/src/main/java/org/apache/helix/tools/ClusterExternalViewVerifier.java a0959cc helix-core/src/main/java/org/apache/helix/tools/ClusterStateVerifier.java 1563769 helix-core/src/main/java/org/apache/helix/tools/YAMLClusterSetup.java 9ec1ef9 helix-core/src/test/java/org/apache/helix/Mocks.java 919eb00 helix-core/src/test/java/org/apache/helix/api/TestNewStages.java ab4ead0 helix-core/src/test/java/org/apache/helix/controller/stages/TestRebalancePipeline.java 2e17ad3 helix-core/src/test/java/org/apache/helix/controller/stages/TestResourceValidationStage.java ece46ff helix-core/src/test/java/org/apache/helix/integration/TestReelectedPipelineCorrectness.java PRE-CREATION helix-core/src/test/java/org/apache/helix/integration/TestSchedulerMessage.java 30f5807 helix-core/src/test/java/org/apache/helix/integration/TestUserDefRebalancerCompatibility.java 27004fe Diff: https://reviews.apache.org/r/16402/diff/ Testing ------- Tests pass locally. Reading the cluster is up to 30% faster. Thanks, Kanak Biscuitwala
