[
https://issues.apache.org/jira/browse/KAFKA-598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Koshy updated KAFKA-598:
-----------------------------
Attachment: KAFKA-598-v1.patch
I spent the better part of a day rebasing - serves me right for letting this
patch sit for so long.
The basic idea is to keep track of incomplete partitions in a multi-fetch
request and reissue a fetch request with a higher fetch size for those
incomplete partitions.
I had considered the possibility of "pipelining" fetch requests for
incomplete partitions - i.e., without using an "upper" fetch size. That
would entail issuing fetch requests at increasing offsets (with the same
fetch size) until the message is complete - during which the (partial)
message would be buffered. With this approach we would probably add an
additional "maxFetchMem" config. However, with logical offsets we don't
have byte-addressability anymore - so it is not possible right now.
Furthermore, with a maxFetchMem param it becomes somewhat similar to the
upperFetchSize approach (in the sense that the client has to be prepared to
handle a certain amount of memory) - so we don't really gain much. The ideal
case would be to support streaming over a single fetch request but this is
obviously a much more complex feature to implement.
Also fixed a bug in the use of partitionMapLock - i.e., one line synchronized
on the reentrant lock instead of locking it.
BTW, for the ReplicaFetchTest change to make sense I could have it expect to
"fail" with a smaller upper fetch size, and then repeat with a higher upper
fetch size, but that would add to the test duration - and it's not mocked
out.
> decouple fetch size from max message size
> -----------------------------------------
>
> Key: KAFKA-598
> URL: https://issues.apache.org/jira/browse/KAFKA-598
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Attachments: KAFKA-598-v1.patch
>
>
> Currently, a consumer has to set fetch size larger than the max message size.
> This increases the memory footprint on the consumer, especially when a large
> number of topic/partition is subscribed. By decoupling the fetch size from
> max message size, we can use a smaller fetch size for normal consumption and
> when hitting a large message (hopefully rare), we automatically increase
> fetch size to max message size temporarily.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
