Producer not distributing across all partitions

2013-09-13 Thread prashant amar
I created a topic with 4 partitions and for some reason the producer is pushing only to one partition. This is consistently happening across all topics that I created ... Is there a specific configuration that I need to apply to ensure that load is evenly distributed across all partitions?

Multiple Processes Consuming from Same GroupID

2013-09-11 Thread prashant amar
A Design Question that needs verification: 1. Created a topic T with 'n' partitions. 2. Created a consumer group process with 'n + 1' threads subscribing from topic 'T' with a groupID 'y' 3. Added another consumer group process with 'n + 1' threads subscribing from same topic 'T' with same

Re: Multiple Processes Consuming from Same GroupID

2013-09-11 Thread prashant amar
%3F Let us know if we can improve the documentation to make it clearer. Thanks, Neha On Wed, Sep 11, 2013 at 5:28 PM, prashant amar amasin...@gmail.com wrote: Also attempted another pattern where 1. Created a topic T with 'n' partitions. 2. Created a consumer group process with 'n

Re: Multiple Processes Consuming from Same GroupID

2013-09-11 Thread prashant amar
somehow closed the socket connection. Anything in the broker log around the same time? Thanks, Jun On Wed, Sep 11, 2013 at 6:07 PM, prashant amar amasin...@gmail.com wrote: Also noticed another issue Specified below is the current configuration Topic1 - n Partitions - 2 Consumer

Re: Multiple Processes Consuming from Same GroupID

2013-09-11 Thread prashant amar
I usually get this exception when I define 2 partitions .. Current configuration : Single Topic - 4 partitions 1 Consumers Group - 10 Threads On Wed, Sep 11, 2013 at 10:24 PM, prashant amar amasin...@gmail.com wrote: From the broker log: INFO Reconnect due to socket error

Re: Message Encoding in Kafka 0.8

2013-07-20 Thread prashant amar
Can anybody please help with this issue indicated below. I have not heard from anyone with a solution. Thanks in advance Amar On Tue, Jul 16, 2013 at 11:35 PM, prashant amar amasin...@gmail.com wrote: Hello, Specified below is my code base where I am attempting to marshall a complex type

Message Encoding in Kafka 0.8

2013-07-20 Thread prashant amar
I noticed that ProducerData has been removed in 0.8 branch If I'd wish to send a complex message type (encoded message) , how would I do it in 0.8? In 0.7.X (a snippet from Neha's example) val producer = new Producer[Message, MemberRecord](config); // send a single message val

Re: Message Encoding in Kafka 0.8

2013-07-20 Thread prashant amar
Hello, Specified below is my code base where I am attempting to marshall a complex type and receiving the follow error. Am I missing anything here? Sending Encoded Messages .. [error] (run-main) java.lang.ClassCastException: java.lang.String cannot be cast to com.test.groups.MemberRecord

Sample Kafka Producer/Consumer example in Scala

2013-07-20 Thread prashant amar
Hello All I have been attempting to build an end-to-end example of producer with multiple consumers using scala. I managed to build a producer, but when I attempt to build a consumer the following statement causes the scala REPL to block/hang val consumerMap =