[
https://issues.apache.org/jira/browse/KAFKA-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472517#comment-13472517
]
Jay Kreps commented on KAFKA-543:
---------------------------------
That was what I originally started to do. I ended up going this route because:
(1) It was a more surgical change and (2) I feel we should avoid logic like
deduplication in the RPC format objects, these should stupidly turn objects
into bytes with processing done elsewhere. I thought the double copy was okay
since this is functionality off the main data path. Let me know what you think,
I can try it the other way if you like and we can compare.
> Metadata request from DefaultEventHandler.handle repeats same topic over and
> over
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-543
> URL: https://issues.apache.org/jira/browse/KAFKA-543
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Jay Kreps
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-543.patch
>
>
> It looks like we are calling BrokerPartitionInfo.updateInfo() with a list of
> the same topic repeated many times:
> Here is the line:
> Utils.swallowError(brokerPartitionInfo.updateInfo(outstandingProduceRequests.map(_.getTopic)))
> The outstandingProduceRequests can (and generally would) have many entries
> for the same topic.
> For example if I use the producer performance test with the default batch
> size on a topic "test" my metadata request will have the topic "test"
> repeated 200 times. On the server side we do several zk reads for each of
> these repetitions.
> This is causing the metadata api to timeout in my perf test periodically.
> I think the fix is simply to de-duplicate prior to the call (and perhaps
> again on the server in case of a misbehaving client).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
