-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53027/
-----------------------------------------------------------
(Updated Oct. 19, 2016, 10:54 a.m.)
Review request for samza and Xinyu Liu.
Changes
-------
Updated description.
Bugs: SAMZA-1017
https://issues.apache.org/jira/browse/SAMZA-1017
Repository: samza
Description (updated)
-------
Added disk quota based throttling to AsyncRunLoop.
Overview:
Adds a Throttleable interface, implemented by RunLoop, AsyncRunLoop and
ThrottlingExecutor
When AsyncRunLoop is throttled, it delays the onComplete() callback from
processAsync() by a delay amount appropriate for the desired work factor.
This implementation has a couple of known issues:
1. Adding additional delay to process()/processAsync() callback will not
throttle the run loop as long as task processing rate > message throughput.
E.g., a low QPS stream with process() time < message inter-arrival time. If
desirable, this can be addressed by delaying based on the total run loop time
instead of just the process() time.
2. If throttled, users can increase their throughput back to original by
increasing task.max.concurrency and redeploying their jobs. I don't have a
simple solution for this, suggestions are welcome.
Diffs
-----
samza-core/src/main/java/org/apache/samza/container/RunLoopFactory.java
a789d04
samza-core/src/main/java/org/apache/samza/container/disk/WatermarkDiskQuotaPolicy.java
21fbca2
samza-core/src/main/java/org/apache/samza/task/AsyncRunLoop.java a510bb0
samza-core/src/main/java/org/apache/samza/util/Throttleable.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/util/ThrottlingExecutor.java
afcc4c5
samza-core/src/main/java/org/apache/samza/util/ThrottlingScheduler.java
PRE-CREATION
samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala 538ebb8
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
05a996c
samza-core/src/test/java/org/apache/samza/task/TestAsyncRunLoop.java ca913de
samza-core/src/test/java/org/apache/samza/util/TestThrottlingScheduler.java
PRE-CREATION
samza-core/src/test/scala/org/apache/samza/container/TestRunLoop.scala
aa1a8d6
samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala
cff6b96
Diff: https://reviews.apache.org/r/53027/diff/
Testing
-------
Tested locally with a hello world app.
Thanks,
Prateek Maheshwari
