----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36471/ -----------------------------------------------------------
(Updated July 22, 2015, 7:06 p.m.) Review request for samza and Navina Ramesh. Repository: samza Description (updated) ------- This work is for SAMZA-719. Currently, a fixed number of containers is assigned to a job as an input configuration parameter. However, with this design jobs can fail due to lack of enough resources (such as memory), or they can become a bottleneck in a workflow containing many jobs. While auto-scaling is much broader term, the goal of this project will be to enable a Samza job to automatically scale its containers such that there is improved job performance. Based on the design, we need a profiler, analyser, optimizer and deployer module. -currently, all components added based on memory usage and the loop seems to work in inital test -tests not added for all components, and further testing is needed. Diffs (updated) ----- build.gradle 0852adc4e8e0c2816afd1ebf433f1af6b44852f7 checkstyle/import-control.xml 6654319392929857bb861d77763afd8a5ea7674c gradle/dependency-versions.gradle fb06e8ed393d1a38abfa1a48fe5244fc7f6c7339 samza-core/src/main/java/org/apache/samza/autoScaling/AutoScalingMode.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/AutoScalingSystem.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/analyzer/Analyzer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/analyzer/MemoryAnalyzer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/deployer/ConfigManager.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/deployer/Deployer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/deployer/RestartingJobDeployer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/metrics/MemoryMetrics.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/metrics/MetricsHistory.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/metrics/SlidingWindowMetric.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/CapacityBasedJobModel.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/Optimizer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/PerContainerMemoryOptimizer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/optimizer/PerContainerOptimizer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/profiler/Profiler.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/profiler/SnapshotReporterProfiler.java PRE-CREATION samza-core/src/main/java/org/apache/samza/autoScaling/stream/AutoScalingMetricsSystemConsumer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java b1078bdf7bddd16c9ccc6559b9efd40ca5ae67bc samza-core/src/main/scala/org/apache/samza/autoScaling/stream/AutoScalingMetricsSystemFactory.scala PRE-CREATION samza-core/src/main/scala/org/apache/samza/config/AutoScalingConfigRewriter.scala PRE-CREATION samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala e4b14f4da6649eb78753ba3b3f529373b6f2dbe4 samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 27b2517048ad5730762506426ee7578c66181db8 samza-core/src/main/scala/org/apache/samza/container/grouper/task/GroupByJobModel.scala PRE-CREATION samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala f62161108f19ad29f40318331419438be5fb97b7 samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 samza-core/src/main/scala/org/apache/samza/metrics/JvmMetrics.scala a95a0ecde300f6576fe46b37d5898e3d21634126 samza-core/src/main/scala/org/apache/samza/util/Util.scala 419452c0d48d8faa84279fcf02a03e87309821d8 samza-core/src/test/java/org/apache/samza/autoScaling/SlidingWindowMetricTest.java PRE-CREATION samza-core/src/test/java/org/apache/samza/autoScaling/analyzer/MockAnalyzer.java PRE-CREATION samza-core/src/test/java/org/apache/samza/autoScaling/optimizer/CapacityBasedJobModelTest.java PRE-CREATION samza-core/src/test/java/org/apache/samza/autoScaling/optimizer/MemoryOptimizerTest.java PRE-CREATION samza-core/src/test/java/org/apache/samza/autoScaling/optimizer/PerContainerMemoryOptimizerTest.java PRE-CREATION samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByJobModel.scala PRE-CREATION samza-shell/src/main/bash/run-auto-scaling.sh PRE-CREATION samza-shell/src/main/bash/run-config-manager.sh PRE-CREATION samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala ea702a919348305ff95ce0b4ca1996a13aff04ec samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala af42c6a6636953a95f79837fe372e0dbd735df70 samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMaster.scala 7b7d86a43c69e72c47eaa91f68be24e0f4022891 Diff: https://reviews.apache.org/r/36471/diff/ Testing ------- Thanks, Shadi A. Noghabi
