----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48080/ -----------------------------------------------------------
Review request for samza. Repository: samza Description ------- This change introduces a ThrottlingExecutor which is used to control the rate of execution in the main run loop. The DiskQuotaEnforcer houses the rules for switching from one DiskQuotaPolicy to the next as new disk usage samples arrive. By default, no throttling will occur. New policies can be added using the following form: ``` container.disk.quota.bytes=XXX container.disk.quota.policy.count=2 container.disk.quota.policy.0.lowWaterMark=0.4 container.disk.quota.policy.0.highWaterMark=0.5 container.disk.quota.policy.0.workFactor=0.5 container.disk.quota.policy.1.lowWaterMark=0.05 container.disk.quota.policy.1.highWaterMark=0.1 container.disk.quota.policy.1.workFactor=0.05 ``` See ThrottlingExecutor for details about how the work factor works and DiskQuotaPolicy for details about how the low and high water mark configuration work. Diffs ----- samza-core/src/main/java/org/apache/samza/container/disk/DiskQuotaEnforcer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/container/disk/DiskQuotaPolicy.java PRE-CREATION samza-core/src/main/java/org/apache/samza/container/disk/DiskSpaceMonitor.java 2a565be7858a4d3a6adbc49989b43b71ca3f6721 samza-core/src/main/java/org/apache/samza/util/HighResolutionClock.java PRE-CREATION samza-core/src/main/java/org/apache/samza/util/SystemHighResolutionClock.java PRE-CREATION samza-core/src/main/java/org/apache/samza/util/ThrottlingExecutor.java PRE-CREATION samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala 3f25eca6e3dffc57360e8bd8c435177c2a9a910a samza-core/src/main/scala/org/apache/samza/container/SameThreadExecutor.scala PRE-CREATION samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala cf3c4c0ab08a59760bc899c6f2027755e933b350 samza-core/src/main/scala/org/apache/samza/container/SamzaContainerMetrics.scala 9e6641c3628290dc05e1eb5537e86bff9d37f92c samza-core/src/main/scala/org/apache/samza/util/Util.scala fc3d085d7fff9f7dcec766ba48e550eb0052e99d samza-core/src/test/java/org/apache/samza/container/disk/TestDiskQuotaEnforcer.java PRE-CREATION samza-core/src/test/java/org/apache/samza/util/TestThrottlingExecutor.java PRE-CREATION samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala 05b4e5c37578340eefe6d412f5a76f540bec6fa6 Diff: https://reviews.apache.org/r/48080/diff/ Testing ------- - Added new unit tests - Ran existing tests with gradle test - Verified throttling behavior and instrumentation with local deployment - Verified average latency impact of feature to be < 150ns for Linux and OSX Thanks, Chris Pettitt