[
https://issues.apache.org/jira/browse/KAFKA-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17250049#comment-17250049
]
Jason Gustafson commented on KAFKA-10652:
-----------------------------------------
[~sagarrao] Apologies for the delay.
> 1) this method maybeAppendBatches returns a timeToFlush which is used by the
> NetworkChannel to drain messages. If I were to add the size based flush here,
> then what do we return as timeToFlush from here?
Hmm... I am not sure it needs to change. Perhaps a simple idea is to let
`BatchAccumulator.timeUntilDrain` return 0 when there are any completed
batches. This will cause `needsDrain` to return true, which will cause the IO
thread to be woken up in `KafkaRaftClient.scheduleAppend`. Does that make sense?
> 2) There is a field called maxBatchSize which is configured in
> BatchAccumulator which seems to me as the max batch size. Currently, it is
> set to 2 ^ 20. We will need to add a cap on the min size shouldn't exceed
> certain values (or atleast not the maxBatchSize)?
Yeah, in line with the suggestion above, I think we can trigger a flush as soon
as we have any batches which are complete (which means they have reached
`maxBatchSize`). I'm not sure I follow the question about the min size though.
Can you clarify?
> Raft leader should flush accumulated writes after a min size is reached
> -----------------------------------------------------------------------
>
> Key: KAFKA-10652
> URL: https://issues.apache.org/jira/browse/KAFKA-10652
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Jason Gustafson
> Assignee: Sagar Rao
> Priority: Major
>
> In KAFKA-10601, we implemented linger semantics similar to the producer to
> let the leader accumulate a batch of writes before fsyncing them to disk.
> Currently the fsync is only based on the linger time, but it would be helpful
> to make it size-based as well. In other words, if we accumulate a
> configurable N bytes, then we should not wait for linger expiration and
> should just fsync immediately.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
