[
https://issues.apache.org/jira/browse/KAFKA-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14385375#comment-14385375
]
Jay Kreps commented on KAFKA-2045:
----------------------------------
The challenge with a rewrite is that it will be hard to know which
optimizations actually mattered. It will likely also be just a partial
implementation (i.e. just the fetch request to a single broker) so it will be
hard to judge what that would look like or what the performance would be if we
integrated it in the main client.
My recommendation would be to approach this in a data driven way instead. We
have a working client, let's look at where it actually spends time and then
improve things that use that time.
Example measurement:
1. % of time spent in decompression
2. % of time spent on CRC check
3. % of time spent on GC
4. Etc.
It would be easy to implement a slightly broken buffer reuse approach that just
preallocated a largish number of buffers bigger than the size needed for
experimental purposes and reused those. This would only work for the purposes
of the perf test but would let us determine the impact of a more complete
pooling implementation quickly.
> Memory Management on the consumer
> ---------------------------------
>
> Key: KAFKA-2045
> URL: https://issues.apache.org/jira/browse/KAFKA-2045
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Guozhang Wang
>
> We need to add the memory management on the new consumer like we did in the
> new producer. This would probably include:
> 1. byte buffer re-usage for fetch response partition data.
> 2. byte buffer re-usage for on-the-fly de-compression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
