Repository: samza Updated Branches: refs/heads/master c50dd080b -> df5aa6c4e
Disable flaky tests. Author: Shanthoosh Venkataraman <svenkatara...@linkedin.com> Reviewers: Navina Ramesh<nav...@apache.org> Closes #158 from shanthoosh/master Project: http://git-wip-us.apache.org/repos/asf/samza/repo Commit: http://git-wip-us.apache.org/repos/asf/samza/commit/df5aa6c4 Tree: http://git-wip-us.apache.org/repos/asf/samza/tree/df5aa6c4 Diff: http://git-wip-us.apache.org/repos/asf/samza/diff/df5aa6c4 Branch: refs/heads/master Commit: df5aa6c4eeb9b89a40780b19e08bb6d96b97730b Parents: c50dd08 Author: Shanthoosh Venkataraman <svenkatara...@linkedin.com> Authored: Wed May 10 09:27:57 2017 -0700 Committer: vjagadish1989 <jvenk...@linkedin.com> Committed: Wed May 10 09:27:57 2017 -0700 ---------------------------------------------------------------------- .../versioned/samza-async-user-guide.md | 25 ++++++++++++++++++++ .../samza/monitor/TestLocalStoreMonitor.java | 9 +++++-- 2 files changed, 32 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/samza/blob/df5aa6c4/docs/learn/tutorials/versioned/samza-async-user-guide.md ---------------------------------------------------------------------- diff --git a/docs/learn/tutorials/versioned/samza-async-user-guide.md b/docs/learn/tutorials/versioned/samza-async-user-guide.md index 30865a8..4396e61 100644 --- a/docs/learn/tutorials/versioned/samza-async-user-guide.md +++ b/docs/learn/tutorials/versioned/samza-async-user-guide.md @@ -118,6 +118,31 @@ task.max.concurrency=4 **NOTE:** In case of AsyncStreamTask, processAsync() is still invoked in the order of the message arrivals, but the completion can go out of order. In case of StreamTask with multithreading, process() can run out-of-order since they are dispatched to a thread pool. This option should **NOT** be used when strict ordering of the output is required. +### Non blocking commit + +By default in samza, concurrent execution of process and commit is not supported. This was designed with a motive to prevent commit flushing to disk and process updating the local store at the same time. + +In some scenarios, process calls will take a lot of time to complete. For instance, having high latency remote service calls in process will affect the execution frequency of commit(since only one can happen at a time). This incurs a additional delay in publishing the processed messages from a samza job to downstream samza jobs. + +Non blocking commit supports simultaneous processes and commits and is essential to prevent the starvation of commit from slow process completions. + +####Benefits. + +A) Reduces delay in publishing the processed messages by enabling the task to commit aggressively(a commit is no longer blocked by ongoing process calls). + +B) Increases the throughput of process calls, since process is not blocked by a ongoing commit. + +C) Decreases the time between fetching and processing the message. + +###NOTE: + +This should be used only when all the local store operations from a samza task are idempotent. Otherwise, during commit writes from messages in flight will be flushed to disk. + +{% highlight jproperties %} +# Turned off by default. Users should explicitly set following config to turn it on. +task.async.commit = true +{% endhighlight %} + ### Guaranteed Semantics In any of the scenarios, Samza guarantees the following semantics: http://git-wip-us.apache.org/repos/asf/samza/blob/df5aa6c4/samza-rest/src/test/java/org/apache/samza/monitor/TestLocalStoreMonitor.java ---------------------------------------------------------------------- diff --git a/samza-rest/src/test/java/org/apache/samza/monitor/TestLocalStoreMonitor.java b/samza-rest/src/test/java/org/apache/samza/monitor/TestLocalStoreMonitor.java index 2ad6935..971eaa4 100644 --- a/samza-rest/src/test/java/org/apache/samza/monitor/TestLocalStoreMonitor.java +++ b/samza-rest/src/test/java/org/apache/samza/monitor/TestLocalStoreMonitor.java @@ -97,8 +97,8 @@ public class TestLocalStoreMonitor { } } - // TODO fix in SAMZA-1243 - // @Test +// TODO: Fix in SAMZA-1243 +// @Test public void shouldDeleteLocalTaskStoreWhenItHasNoOffsetFile() throws Exception { localStoreMonitor.monitor(); assertTrue("Task store directory should not exist.", !taskStoreDir.exists()); @@ -148,8 +148,13 @@ public class TestLocalStoreMonitor { assertEquals(0, localStoreMonitorMetrics.diskSpaceFreedInBytes.getCount()); } +<<<<<<< HEAD // TODO fix in SAMZA-1243 // @Test +======= +// TODO: Fix in SAMZA-1243 +// @Test +>>>>>>> Disable flaky tests. public void shouldDeleteTaskStoreWhenTaskPreferredStoreIsNotLocalHost() throws Exception { Task task = new Task("notLocalHost", "test-task", "0", new ArrayList<>(), ImmutableList.of("test-store"));