Hi Kafka Dev Team, I would appreciate your feedback on moving producer partition selection from producer to Broker. Also, please do let me know what is correct process of collecting feedback from Kafka Dev team and/or community.
Thanks, Bhavesh On Tue, May 26, 2015 at 11:54 AM, Bhavesh Mistry <mistry.p.bhav...@gmail.com > wrote: > Hi Kafka Dev Team, > > I am sorry I am new to process of discussion and/or KIP. So, I had > commented other email voting chain. Please do let me know correct process > for collecting and staring discussion with Kafka Dev Group. > > Here is original message: > > I have had experience with both producer and consumer side. I have > different use case on this partition selection strategy. > > > > Problem : > > > We have heterogeneous environment of producers (by that I mean we have > node js, python, New Java & Old Scala Based producers to same topic). I > have seen that not all producers employ round-robing strategies for > non-keyed message like new producer does. Hence, it creates non uniform > data ingestion into partition and delay in overall message processing. > > How to address uniform distribution/message injection rate to all > partitions ? > > > > Propose Solution: > > > Let broker cluster decide the next partition for topic to send data rather > than producer itself with more intelligence. > > 1) When sending data to brokers (ProduceResponse) Kafka Protocol over > the wire send hint to client which partition to send based on following > logic (Or can be customizable) > > a. Based on overall data injection rate for topic and current > producer injection rate > > b. Ability rank partition based on consumer rate (Advance Use Case as > there may be many consumers so weighted average etc... ) > > > > Untimely, brokers will coordinate among thousand of producers and divert > data injection rate (out-of-box feature) and consumption rate (pluggable > interface implementation on brokers’ side). The goal here is to attain > uniformity and/or lower delivery rate to consumer. This is similar to > consumer coordination moving to brokers. The producer side partition > selection would also move to brokers. This will benefit both java and > non-java clients. > > > > Please let me know your feedback on this subject matter. I am sure lots > of you run Kafka in Enterprise Environment where you may have different > type of producers for same topic (e.g logging client in JavaScript, PHP, > Java and Python etc sending to log topic). I would really appreciate your > feedback on this. > > > > > > Thanks, > > > Bhavesh >
