Kafka AWS deployment + UI console

2014-10-07 Thread Shafaq
://www.michael-noll.com/blog/2014/03/17/wirbelsturm-one-click-deploy-storm-kafka-clusters-with-vagrant-puppet/ -- Kind Regards, Shafaq

Topic Fetch metadata in producer

2013-10-09 Thread Shafaq
adata for topic: 37topic1381360774212 at kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:53) at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:185) -- Kind Regards, Shafaq

Re: Topic Fetch metadata in producer

2013-10-09 Thread Shafaq
e wrote: > Are you using 0.8 HEAD ? > > Can you send around the full stack trace since one of the common reasons > for failed topic metadata requests are socket timeouts. > > Thanks, > Neha > > > On Wed, Oct 9, 2013 at 4:30 PM, Shafaq wrote: > > > I run in

Re: Topic Fetch metadata in producer

2013-10-09 Thread Shafaq
hile fetching metadata %s for topic partition [%s,%d]: [%s]" > > > On Wed, Oct 9, 2013 at 5:49 PM, Shafaq wrote: > > > yes I'm using 0.8 head > > > > > > roducer starting with option to multi-thread > > compressed---true > > Exception in threa

Re: Topic Fetch metadata in producer

2013-10-09 Thread Shafaq
o.com,port:9092 (kafka.server.ReplicaManager) java.io.IOException: No such file or directory On Wed, Oct 9, 2013 at 10:25 PM, Shafaq wrote: > I could not see such messages (Error while fetching metadata [%s] for > topic [%s]: %s ") > > In the perf test, I create new topic for sending t

Questions about failure recovery, load balancing and producer ack/async mode

2013-10-23 Thread Shafaq
-level consumer) How is the faliover handlled 3. How do we configure producer for wait for leader ack and wait for commited act which Jun Rao talks about in his 2013 apachecon presentation -- Kind Regards, Shafaq

0.8 Head: Simple consumer no receiving messages

2013-10-29 Thread Shafaq
nd the problem. thanks! -- Kind Regards, Shafaq

Re: High Level Consumer commit offset

2013-11-04 Thread Shafaq
gt; > On Mon, Nov 4, 2013 at 3:32 PM, Guozhang Wang > > > wrote: > > > > > > > > > That is correct. If auto.commit.enable is set to faulse, the > offsets > > > will > > > > > not be committed at all unless the consumer calls the commit > function > > > > > explicitly. > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > On Mon, Nov 4, 2013 at 2:42 PM, Vadim Keylis < > vkeylis2...@gmail.com> > > > > > wrote: > > > > > > > > > > > Good afternoon. I was under impression if auto commit set to > false > > > > then > > > > > > once consumer is restarted then logs would be replayed from the > > > > > beginning. > > > > > > Is that correct? > > > > > > > > > > > > Thanks, > > > > > > Vadim > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > -- Kind Regards, Shafaq

0.8 High-level consumer commit offset question

2013-11-05 Thread Shafaq
achieve it? Also "auto.offset.reset"="smallest" need to be set inorder for consumer to consume messages which are already published by broker. Is that correct? Please comment. thanks! -- Kind Regards, Shafaq

Re: Message Width

2013-11-05 Thread Shafaq
The message is sent as serialized over wire so there is no concept of width but only length. The message size is specified in the broker config and default is 10 MB off the top of my head to prevent OOM. On Nov 5, 2013 1:06 PM, "Jason Pohl" wrote: > I am new to pub-sub architecture and am looki

Leader no available for partiton error

2013-11-07 Thread Shafaq
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42) at java.lang.Thread.run(Thread.java:724) -- Kind Regards, Shafaq

Re: Anyone working on a Kafka book?

2013-12-10 Thread Shafaq
nnecessary newlines. > >>>> > >>>> Thanks, > >>>> > >>>> Chris > >>>> > >>>> > >>>> On Mon, Mar 25, 2013 at 12:04 PM, Jun Rao wrote: > >>>> > >>>> Chris, > >>>>> > >>>>> This looks good. One thing about partitioning. Currently, if a > message > >>>>> doesn't have a key, we always use the random partitioner (regardless > of > >>>>> what "partitioner.class" is set to). > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Jun > >>>>> > >>>>> > >>>>> > >>>>> > >> > > > -- Kind Regards, Shafaq

Re: Anyone working on a Kafka book?

2013-12-10 Thread Shafaq
On Tue, Dec 10, 2013 at 10:22 AM, Steve Morin wrote: > Shafaq, > What does the architecture of what your building look like? > -Steve > > > On Tue, Dec 10, 2013 at 10:19 AM, Shafaq wrote: > >> Hey Guys, >>I would love to contribute to the book specially

Re: Anyone working on a Kafka book?

2013-12-13 Thread Shafaq
Thats already done. On Dec 13, 2013 9:28 AM, "Steve Morin" wrote: > Shafaq, > That's pretty cool, have you already connected Kafka to spark > RRD/DStream or is that something you have to figure out? > -Steve > > > On Tue, Dec 10, 2013 at 7:10 PM, Shafaq