Re: Requesting advice about Producer & Consumer Design

2014-03-20 Thread Neha Narkhede
1) The new producer and consumer is being designed to take care of auto balancing between partitions. Right ? That's correct. 2) with the current available producer and consumer, is my current setup(pls see attached file) a good design in terms of scalability. Your Kafka setup seems reasonable t

Re: Requesting advice about Producer & Consumer Design

2014-03-19 Thread Krishna Raj
Hi Jay, Thanks for taking time for the details ! Appreciate that. Just clarifying myself couple of things in the same line. 1) The new producer and consumer is being designed to take care of auto balancing between partitions. Right ? 2) with the current available producer and consumer, is my cu

Re: Requesting advice about Producer & Consumer Design

2014-03-19 Thread Jay Kreps
Hey Krishna, Let me clarify the current state of things a little: 1. Kafka offers a single producer interface as well as two consumer interfaces: the low-level "simple consumer" which just directly makes network requests, and the higher level interface which handles fault-tolerance, partition assi

Requesting advice about Producer & Consumer Design

2014-03-19 Thread Krishna Raj
Hello Experts & Kafka Team, Its existing to learn and work on using Kafka. I have been going through lot of pages and Q&A. We are building an infra & topology using Kafka for events processing in our application. We need some advice about designing the Producer and Consumer. *Please find attach