[
https://issues.apache.org/jira/browse/KAFKA-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576408#comment-13576408
]
Sriram Subramanian commented on KAFKA-756:
------------------------------------------
There are two parts to this change.
1. The change in FileMessageSet is the actual fix that prevents the infinite
loop.
2. The change in MultiSend is subtle and is more of an optimization. A
fetchresponse can typically take a while to transmit all the bytes to the
fetcher today (depends on the fetch size and number of topics). This blocks the
processor thread today and thereby all the responses in that processor queue.
We had two possible solutions for this.
a. A solution I had proposed is to have a configurable maxsize value that would
be used by Processor::write. Once the maxsize is reached, the write would
return and will try to write the remaining in the next processor thread
iteration. This would also enable other requests and responses to be handled
that are ready.
b. Jun's proposal is to fill up the socket buffer as much as possible and stop
the iteration when it cannot take more data. The patch v2 does that. It
essentially continues to write to the buffer till an incomplete write happens.
In such a scenario we return to the processor iteration loop and give a chance
to process the other requests and responses.
The 2nd change was made to get feedbacks from others on their thoughts. We are
testing the change as part of system tests and are running it currently in the
test cluster to monitor how it performs.
> Processor thread blocks due to infinite loop during fetch response send
> -----------------------------------------------------------------------
>
> Key: KAFKA-756
> URL: https://issues.apache.org/jira/browse/KAFKA-756
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Sriram Subramanian
> Assignee: Sriram Subramanian
> Priority: Blocker
> Labels: bugs, p1
> Attachments: KAFKA-756.patch, Kafka-756-v2.patch
>
>
> This looks to be because of an infinite loop during fetch response send. This
> happens because we try to send bytes from a log which has been truncated
> during send. The total size to send is calculated at the beginning of the
> iteration and it does not take into account the change in log size during
> send. When send happens, it uses the size calculated at the start and loops
> continuously hoping to send more data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
