Re: Review Request 48109: SAMZA-957 Avoid unnecessary KV Store flushes (part 3)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48109/ --- (Updated June 1, 2016, 2:23 a.m.) Review request for samza, Boris Shkolnik, Chris Pettitt, Jake Maes, Navina Ramesh, Jagadish Venkatraman, Xinyu Liu, and Yi Pan (Data Infrastructure). Bugs: SAMZA-957 https://issues.apache.org/jira/browse/SAMZA-957 Repository: samza Description --- SAMZA-957 Avoid unnecessary KV Store flushes (part 3) Diffs - samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala c28f8db8cb59bd5415e78535877acc1e5bee0f67 samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala eee74473726cb2a36d0b75fe5c9df737440980bc Diff: https://reviews.apache.org/r/48109/diff/ Testing (updated) --- ./gradlew build Thanks, Jake Maes
Review Request 48109: SAMZA-957 Avoid unnecessary KV Store flushes (part 3)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48109/ --- Review request for samza, Boris Shkolnik, Chris Pettitt, Jake Maes, Navina Ramesh, Jagadish Venkatraman, Xinyu Liu, and Yi Pan (Data Infrastructure). Bugs: SAMZA-957 https://issues.apache.org/jira/browse/SAMZA-957 Repository: samza Description --- SAMZA-957 Avoid unnecessary KV Store flushes (part 3) Diffs - samza-kv/src/main/scala/org/apache/samza/storage/kv/CachedStore.scala c28f8db8cb59bd5415e78535877acc1e5bee0f67 samza-kv/src/test/scala/org/apache/samza/storage/kv/TestCachedStore.scala eee74473726cb2a36d0b75fe5c9df737440980bc Diff: https://reviews.apache.org/r/48109/diff/ Testing --- Thanks, Jake Maes
Review Request 48080: SAMZA-956: Disk Quotas: Add throttler and disk quota enforcement
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48080/ --- Review request for samza. Repository: samza Description --- This change introduces a ThrottlingExecutor which is used to control the rate of execution in the main run loop. The DiskQuotaEnforcer houses the rules for switching from one DiskQuotaPolicy to the next as new disk usage samples arrive. By default, no throttling will occur. New policies can be added using the following form: ``` container.disk.quota.bytes=XXX container.disk.quota.policy.count=2 container.disk.quota.policy.0.lowWaterMark=0.4 container.disk.quota.policy.0.highWaterMark=0.5 container.disk.quota.policy.0.workFactor=0.5 container.disk.quota.policy.1.lowWaterMark=0.05 container.disk.quota.policy.1.highWaterMark=0.1 container.disk.quota.policy.1.workFactor=0.05 ``` See ThrottlingExecutor for details about how the work factor works and DiskQuotaPolicy for details about how the low and high water mark configuration work. Diffs - samza-core/src/main/java/org/apache/samza/container/disk/DiskQuotaEnforcer.java PRE-CREATION samza-core/src/main/java/org/apache/samza/container/disk/DiskQuotaPolicy.java PRE-CREATION samza-core/src/main/java/org/apache/samza/container/disk/DiskSpaceMonitor.java 2a565be7858a4d3a6adbc49989b43b71ca3f6721 samza-core/src/main/java/org/apache/samza/util/HighResolutionClock.java PRE-CREATION samza-core/src/main/java/org/apache/samza/util/SystemHighResolutionClock.java PRE-CREATION samza-core/src/main/java/org/apache/samza/util/ThrottlingExecutor.java PRE-CREATION samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala 3f25eca6e3dffc57360e8bd8c435177c2a9a910a samza-core/src/main/scala/org/apache/samza/container/SameThreadExecutor.scala PRE-CREATION samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala cf3c4c0ab08a59760bc899c6f2027755e933b350 samza-core/src/main/scala/org/apache/samza/container/SamzaContainerMetrics.scala 9e6641c3628290dc05e1eb5537e86bff9d37f92c samza-core/src/main/scala/org/apache/samza/util/Util.scala fc3d085d7fff9f7dcec766ba48e550eb0052e99d samza-core/src/test/java/org/apache/samza/container/disk/TestDiskQuotaEnforcer.java PRE-CREATION samza-core/src/test/java/org/apache/samza/util/TestThrottlingExecutor.java PRE-CREATION samza-core/src/test/scala/org/apache/samza/util/TestUtil.scala 05b4e5c37578340eefe6d412f5a76f540bec6fa6 Diff: https://reviews.apache.org/r/48080/diff/ Testing --- - Added new unit tests - Ran existing tests with gradle test - Verified throttling behavior and instrumentation with local deployment - Verified average latency impact of feature to be < 150ns for Linux and OSX Thanks, Chris Pettitt
