Mike Percy has submitted this change and it was merged. Change subject: KUDU-2078: Sink failure if batch size > session's flush buffer size ......................................................................
KUDU-2078: Sink failure if batch size > session's flush buffer size The Flume sink uses manual flush mode, so if users set the sink's batch size parameter above the manual flush default buffer size, the sink could fail batches (over and over). This patch sets the session's buffer size (which is in terms of number of ops) to the same as the batch size, so this problem can no longer occur. I considered using AUTO_FLUSH_BACKGROUND for the flushing as well, but it can result in out-of-order writes, which might be unexpected semantics for Flume (as opposed to, say, Spark). Using AUTO_FLUSH_BACKGROUND with a high batch size would likely be more performant, but we can add that as an additional configuration later if the need arises. Change-Id: Id1c54bcecc3e13ae64dd90efe6cf53021517dcdf Reviewed-on: http://gerrit.cloudera.org:8080/7641 Tested-by: Kudu Jenkins Reviewed-by: Mike Percy <mpe...@apache.org> --- M java/kudu-flume-sink/src/main/java/org/apache/kudu/flume/sink/KuduSink.java 1 file changed, 6 insertions(+), 5 deletions(-) Approvals: Mike Percy: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/7641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id1c54bcecc3e13ae64dd90efe6cf53021517dcdf Gerrit-PatchSet: 3 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Will Berkeley <wdberke...@gmail.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy <mpe...@apache.org> Gerrit-Reviewer: Will Berkeley <wdberke...@gmail.com>
