[ https://issues.apache.org/jira/browse/KAFKA-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536227#comment-13536227 ]
Joel Koshy commented on KAFKA-598: ---------------------------------- Thanks for taking the time to review. 30. I think what I have is correct but may be missing something - can you check this again and confirm? The passed-in value is (supposed to be) the fetch size per partition. Only the fetcher manager can compute the fetch size per partition (since it has access to the partition count for each fetcher thread) and that function is "wired in" through the constructor of the abstract fetcher thread. 31. Sure - will make that change. 32. It is true that map size is O(1). However, here I need to take each value of the map (which is a fetcher thread) and get the number of partitions that it owns and then sum up the counts. (This is required to compute the partition fetch size). The iteration over the map values seemed expensive (and it is done for each fetch request) so I went with the volatile int. Anyway, I think a better approach would be to maintain a Set of partitions that the fetcher manager is currently managing and just return the size of the set. > decouple fetch size from max message size > ----------------------------------------- > > Key: KAFKA-598 > URL: https://issues.apache.org/jira/browse/KAFKA-598 > Project: Kafka > Issue Type: Bug > Components: core > Affects Versions: 0.8 > Reporter: Jun Rao > Assignee: Joel Koshy > Priority: Blocker > Attachments: KAFKA-598-v1.patch, KAFKA-598-v2.patch, > KAFKA-598-v3.patch > > > Currently, a consumer has to set fetch size larger than the max message size. > This increases the memory footprint on the consumer, especially when a large > number of topic/partition is subscribed. By decoupling the fetch size from > max message size, we can use a smaller fetch size for normal consumption and > when hitting a large message (hopefully rare), we automatically increase > fetch size to max message size temporarily. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira