Re: Producer not distributing across all partitions

2013-09-13 Thread prashant amar
; > > > > > > > > > > > https://github.com/apache/kafka/commit/b71e6dc352770f22daec0c9a3682138666f032be > > > > > > It no longer assigns a random partition to data with a null partition > > key. > > > I had to change my code to generate

Re: Producer not distributing across all partitions

2013-09-13 Thread prashant amar
on at random. > Thanks, > Jun > > > On Thu, Sep 12, 2013 at 1:56 PM, prashant amar > wrote: > > > I created a topic with 4 partitions and for some reason the producer is > > pushing only to one partition. > > > > This is consistently happening across

Re: Producer not distributing across all partitions

2013-09-13 Thread prashant amar
the > topic.metadata.refresh.interval.ms is too high. Did you observe if the > distribution improves after topic.metadata.refresh.interval.ms has passed > ? > > Thanks > Neha > > > On Fri, Sep 13, 2013 at 4:47 AM, prashant amar > wrote: > > > I am using kafka 08 version ... > &

Producer not distributing across all partitions

2013-09-12 Thread prashant amar
I created a topic with 4 partitions and for some reason the producer is pushing only to one partition. This is consistently happening across all topics that I created ... Is there a specific configuration that I need to apply to ensure that load is evenly distributed across all partitions? Grou

Re: Multiple Processes Consuming from Same GroupID

2013-09-11 Thread prashant amar
I usually get this exception when I define > 2 partitions .. Current configuration : Single Topic - 4 partitions 1 Consumers Group - 10 Threads On Wed, Sep 11, 2013 at 10:24 PM, prashant amar wrote: > From the broker log: > > > INFO Reconnect due

Re: Multiple Processes Consuming from Same GroupID

2013-09-11 Thread prashant amar
omehow closed the socket connection. Anything in the > broker log around the same time? > > Thanks, > > Jun > > > On Wed, Sep 11, 2013 at 6:07 PM, prashant amar > wrote: > > > Also noticed another issue > > > > Specified below is the current configurati

Re: Multiple Processes Consuming from Same GroupID

2013-09-11 Thread prashant amar
groupneverreceiveanymessage%3F > > Let us know if we can improve the documentation to make it clearer. > > Thanks, > Neha > > > On Wed, Sep 11, 2013 at 5:28 PM, prashant amar > wrote: > > > Also attempted another pattern where > > > > 1. Created a

Re: Multiple Processes Consuming from Same GroupID

2013-09-11 Thread prashant amar
bscribing from same topic 'T' with same groupID 'z' (Note that 2 and 3 subscribe from same topic but different groups) Can a single topic with multiple partitions abetted with multiple consumer groups increase parallelism is consumption? On Wed, Sep 11, 2013 at 4:48

Multiple Processes Consuming from Same GroupID

2013-09-11 Thread prashant amar
A Design Question that needs verification: 1. Created a topic T with 'n' partitions. 2. Created a consumer group process with 'n + 1' threads subscribing from topic 'T' with a groupID 'y' 3. Added another consumer group process with 'n + 1' threads subscribing from same topic 'T' with same groupID

Re: Message Encoding in Kafka 0.8

2013-07-18 Thread prashant amar
Can anybody please help with this issue indicated below. I have not heard from anyone with a solution. Thanks in advance Amar On Tue, Jul 16, 2013 at 11:35 PM, prashant amar wrote: > Hello, > > Specified below is my code base where I am attempting to marshall a > complex type a

Re: Message Encoding in Kafka 0.8

2013-07-16 Thread prashant amar
Hello, Specified below is my code base where I am attempting to marshall a complex type and receiving the follow error. Am I missing anything here? Sending Encoded Messages .. [error] (run-main) java.lang.ClassCastException: java.lang.String cannot be cast to com.test.groups.MemberRecord java.la

Message Encoding in Kafka 0.8

2013-07-16 Thread prashant amar
I noticed that ProducerData has been removed in 0.8 branch If I'd wish to send a complex message type (encoded message) , how would I do it in 0.8? In 0.7.X (a snippet from Neha's example) val producer = new Producer[Message, MemberRecord](config); // send a single message val mes

Sample Kafka Producer/Consumer example in Scala

2013-07-16 Thread prashant amar
Hello All I have been attempting to build an end-to-end example of producer with multiple consumers using scala. I managed to build a producer, but when I attempt to build a consumer the following statement causes the scala REPL to block/hang val consumerMap = consumer.createMessageStreams(topicC