[ https://issues.apache.org/jira/browse/KAFKA-332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-332: ----------------------------- Attachment: KAFKA-332-v2.patch Thanks for the review. - Captured throwable - Trace logging retry attempt - Additional doc on num.retries Circular iterator: I thought it would be a convenient pattern for round-robin selection and in fact makes the code simpler/clearer. If it is hard to read, we can just go with explicit modulo-based selection based on a counter - although IMO that is less clean. > Mirroring should use multiple producers; add producer retries to > DefaultEventHandler > ------------------------------------------------------------------------------------ > > Key: KAFKA-332 > URL: https://issues.apache.org/jira/browse/KAFKA-332 > Project: Kafka > Issue Type: Improvement > Components: core > Reporter: Joel Koshy > Assignee: Joel Koshy > Priority: Minor > Attachments: KAFKA-332-v1.patch, KAFKA-332-v2.patch > > > I'm clubbing these two together as these are both important for mirroring. > (1) Multiple producers: > Shallow iteration (KAFKA-315) helps improve mirroring throughput when > messages are compressed. With shallow iteration, the mirror-maker's consumer > does not do deep iteration over compressed messages. However, when its > embedded producer sends these messages to the target cluster's brokers, the > receiving broker does deep iteration to validate the messages before > appending to the log. > In the current (pre- KAFKA-48) request handling mechanism, one producer > effectively translates to one server-side thread for handling produce > requests, so there is still a bottleneck due to decompression (due to > message validation) on the target broker. > One way to work around this is to use broker.list with multiple brokers > specified per broker. E.g., > broker.list=0:localhost:9191,1:localhost:9191,2:localhost:9191,... which > effectively emulates multiple server-side threads. It would be better to > just add a num.producers option to the mirror-maker and instantiate that > many producers. > (2) Retries: > If the mirror-maker uses broker.list and one of the brokers is bounced for > any reason, messages can get lost. Message loss can be reduced/avoided if > the brokers are behind a VIP and if retries are supported. This option will > not work for the zk-based producer because the decision of which broker to > send to has already been made, so retries would go to the same (potentially > still down) broker. (With KAFKA-253 it would work for zk-based producers as > well, but that is only in 0.8). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira