[
https://issues.apache.org/jira/browse/KAFKA-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13975717#comment-13975717
]
Jun Rao commented on KAFKA-1409:
--------------------------------
The fix would be:
1. add an extra verification of the message size before decompression;
2. change the leader lock to be a read/write lock (only the leader updater will
hold the write lock) so that slow producer requests don't block the fetch
requests.
3. the byte-in rate includes over-sized messages, which is not consistent with
message-in rate.
> oversized messages can slow down the brokers
> --------------------------------------------
>
> Key: KAFKA-1409
> URL: https://issues.apache.org/jira/browse/KAFKA-1409
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2
> Reporter: Jun Rao
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> There are 2 main issues.
> 1. The broker first decompresses and then recompresses each message (to
> assign new offsets) before validating the message size (we have to do the
> validation after recompression since the message size could change). So, it
> can spend many secs to decompress/recompress an oversized message, only to be
> dropped later. While this was happening, a request thread was tied up, which
> reduced the capacity on the broker.
> 2. Both the fetch and producer requests need to hold a leader lock (per
> partition). So, if the producer is slow in appending the log, it will block
> other producer/fetch requests on the same partition.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
