[
https://issues.apache.org/jira/browse/KAFKA-808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neha Narkhede updated KAFKA-808:
--------------------------------
Attachment: kafka-808.patch
Added an optional queue.size config to the migration tool
> Migration tool internal queue between consumer and producer threads should be
> configurable
> ------------------------------------------------------------------------------------------
>
> Key: KAFKA-808
> URL: https://issues.apache.org/jira/browse/KAFKA-808
> Project: Kafka
> Issue Type: Improvement
> Components: tools
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Priority: Critical
> Labels: kafka-0.8, p2
> Attachments: kafka-808.patch
>
>
> Migration tool maintains an internal blocking queue to buffer messages
> between the 0.7 consumer and the 0.8 producer. It is fixed in size and it set
> to the number of producers. However, the consumer usually fetches data in
> large chunks, several thousands of messages. Also each producer's internal
> queue frees up in chunks of the producer's batch size. In performance
> profiling, I saw that the migration tool's consumer thread spent time waiting
> for space to free up in the internal queue. It seems that this internal queue
> should be configurable and much larger than the number of producers. This
> will improve the buffering and help bridge the gap between the consumer and
> producer's throughput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
