Re: Using kafka in non million users environment

For operating kafka across multiple data centers have a look at https://kafka.apache.org/08/ops.html and MirrorMaker ( https://kafka.apache.org/08/tools.html) On 20 August 2014 04:09, Justin Maltat wrote: > Hi, > > As of today, our company IT is mainly composed of domain specific > software (pr

Re: Most common kafka client comsumer implementations?

Hi Jim, In one of the applications, we implemented option #1: messageList = getNext(1000) process(messageList) commit() In case of failure, this resulted in duplicate processing for at most 1000 records per partition. Regards, Anand On 1 August 2014 20:35, Jim wrote: > Thanks Guozhang, > >

Error when restarting consumer

Hi, I'm using High Level Consumer with kafka 0.7.2. The consumer is able to consume messages from kafka for the first time but when I restart the consumer process, I'm getting the following error: 2013-06-24 09:48:25 FetchRunnable-0 [ERROR] FetcherRunnable:89 - error in FetcherRunnable kafka.netw

Re: C/C++ Client

Hi Joel, Any updates on the c++ producer? Thanks, Anand On 3 April 2013 05:59, Joel Koshy wrote: > Yes - we would be interested in doing that. I have been spending most of my > time over the past couple weeks on the C++ library (currently, only for the > producer). It is reasonably stable, alt

Capacity planning for kafka

Hi, We are capacity planning for kafka deployment (Replication factor 3) in production environment, the producer is producing data at 1.5Gbps. Total number of producers will be around 500 and there will be 100 consumers. How many cores would be required to support them? And are there any known rep

Multiple log directories in 0.7.2

Hi, I'm aware of KAFKA-188 which implements multiple log directories in kafka 0.8. Is there a way to backport this functionality in 0.7.2? Thanks, Anand

Re: Kafka unable to process large number of messages

er then in case of multiple producers. Thanks, Anand On 29 March 2013 20:24, Jun Rao wrote: > This indicates that the messages sent to the broker are somehow corrupted. > Are you using a java producer? How many instances of producers do you have? > > Thanks, > > Jun > > On F

Kafka unable to process large number of messages

Hi, I'm running kafka in distributed mode with 2 nodes. It works fine with slow ingestion rates but when I increase the ingestion rate, both the nodes starts giving the following error: [2013-03-29 14:51:45,379] ERROR Closing socket for /192.168.145.183 because of error (kafka.network.Processor)