----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53027/ -----------------------------------------------------------
(Updated Oct. 19, 2016, 10:54 a.m.) Review request for samza and Xinyu Liu. Changes ------- Updated description. Bugs: SAMZA-1017 https://issues.apache.org/jira/browse/SAMZA-1017 Repository: samza Description (updated) ------- Added disk quota based throttling to AsyncRunLoop. Overview: Adds a Throttleable interface, implemented by RunLoop, AsyncRunLoop and ThrottlingExecutor When AsyncRunLoop is throttled, it delays the onComplete() callback from processAsync() by a delay amount appropriate for the desired work factor. This implementation has a couple of known issues: 1. Adding additional delay to process()/processAsync() callback will not throttle the run loop as long as task processing rate > message throughput. E.g., a low QPS stream with process() time < message inter-arrival time. If desirable, this can be addressed by delaying based on the total run loop time instead of just the process() time. 2. If throttled, users can increase their throughput back to original by increasing task.max.concurrency and redeploying their jobs. I don't have a simple solution for this, suggestions are welcome. Diffs ----- samza-core/src/main/java/org/apache/samza/container/RunLoopFactory.java a789d04 samza-core/src/main/java/org/apache/samza/container/disk/WatermarkDiskQuotaPolicy.java 21fbca2 samza-core/src/main/java/org/apache/samza/task/AsyncRunLoop.java a510bb0 samza-core/src/main/java/org/apache/samza/util/Throttleable.java PRE-CREATION samza-core/src/main/java/org/apache/samza/util/ThrottlingExecutor.java afcc4c5 samza-core/src/main/java/org/apache/samza/util/ThrottlingScheduler.java PRE-CREATION samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala 538ebb8 samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 05a996c samza-core/src/test/java/org/apache/samza/task/TestAsyncRunLoop.java ca913de samza-core/src/test/java/org/apache/samza/util/TestThrottlingScheduler.java PRE-CREATION samza-core/src/test/scala/org/apache/samza/container/TestRunLoop.scala aa1a8d6 samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cff6b96 Diff: https://reviews.apache.org/r/53027/diff/ Testing ------- Tested locally with a hello world app. Thanks, Prateek Maheshwari