Re: Question about recovering from outage

I'm running Kafka 0.8. / Jonas 2014-05-18 23:45 GMT+02:00 Jonas Bergström luckysw...@gmail.com: Hi all, and thanks for a fantastic product. The other day our kafka node in our test environment went down due to full disc. I reconfigured kafka to save fewer messages, and restarted the node.

Re: ISR not updating

The value of under replicated partitions is 0 across the cluster. Thanks, Shone On Mon, May 19, 2014 at 12:23 AM, Jun Rao jun...@gmail.com wrote: What's the value of under replicated partitions JMX in each broker? Thanks, Jun On Sat, May 17, 2014 at 6:16 PM, Paul Mackles

RE: starting of at a small scale, single ec2 instance with 7.5 GB RAM with kafka

Hi, I like how kafka operates, but I'm wondering if it is possible to run everything on a single ec2 instance with 7.5 GB RAM. So that would be zookeeper and a single kafka broker. I would have a separate server to consume from the broker. Producers would be from my web servers. I don't want

Make kafka storage engine pluggable and provide a HDFS plugin?

Hi there, I recently started to use Kafka for our data analysis pipeline and it works very well. One problem to us so far is expanding our cluster when we need more storage space. Kafka provides some scripts for helping do this but the process wasn't smooth. To make it work perfectly, seems

Re: ISR not updating

Ok. That does indicate the ISR should include all replicas. Which version of ZK server are you using? Could you check the ZK server log to see if there if the ISR is being updated? Thanks, Jun On Mon, May 19, 2014 at 1:30 AM, Shone Sadler shone.sad...@gmail.comwrote: The value of under

Re: Question about recovering from outage

Do you think you could upgrade to 0.8.1.1? It fixed a bunch of corner cases in the controller. Thanks, Jun On Mon, May 19, 2014 at 12:00 AM, Jonas Bergström luckysw...@gmail.comwrote: I'm running Kafka 0.8. / Jonas 2014-05-18 23:45 GMT+02:00 Jonas Bergström luckysw...@gmail.com: Hi

Kafka Migration Tool

Hi all, I'm trying to migrate from Kafka 0.7.2-2.9.2 (with Zookeeper 3.3.4 from Cloudera) to Kafka 0.8.1.1-2.9.2 (with official Zookeeper 3.4.5 ) - However hitting brick walls with a very mysterious problem: 6) at kafka.tools.KafkaMigrationTool.main(KafkaMigrationTool.java:217) Caused by:

Re: What happens to Kafka when ZK lost its quorum?

Hi Weide/Connie, I have added this entry in the FAQ, please let me know if anything on the wiki is not clear to you. https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowareKafkabrokersdependonZookeeper ? Guozhang On Wed, May 14, 2014 at 8:15 AM, Neha Narkhede

Cause and Recovery

Hi, I got this from the log: [exec] 03:50:06.216 [ProducerSendThread-] [1;31mERROR [0;39m [1;35mk.producer.async.ProducerSendThread [0;39m - Error in handling batch of 100 events [exec] kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. [exec] at

Re: Cause and Recovery

Hi Mingtao, 1. Do you see any error/warn log before this entry? 2. The producer will re-try on sending when the previous trial is not successful. In your case it has exhausted all the retries and hence messages are dropped on this floor and lost. Guozhang On Mon, May 19, 2014 at 8:32 AM,

Re: Question about recovering from outage

Ok, I'll upgrade. Is there a way to see the status of a node that is recovering, e.g. in zookeeper or via jmx? / Jonas 2014-05-19 16:49 GMT+02:00 Jun Rao jun...@gmail.com: Do you think you could upgrade to 0.8.1.1? It fixed a bunch of corner cases in the controller. Thanks, Jun On

Consistent replication of an event stream into Kafka

Hello, We have a use case where we want to replicate an event stream which exists outside of kafka into a kafka topic (single partition). The event stream has sequence ids which always increase by 1. We want to preserve this ordering. The difficulty is that we want to be able to have the process

Re: Kafka Migration Tool

It seems that you may have set the zk connection string to one used by 0.8 Kafka brokers. Thanks, Jun On Mon, May 19, 2014 at 8:34 AM, Mo Firouz mo.fir...@visualdna.com wrote: Hi all, I'm trying to migrate from Kafka 0.7.2-2.9.2 (with Zookeeper 3.3.4 from Cloudera) to Kafka 0.8.1.1-2.9.2

Re: Question about recovering from outage

In trunk, we have a JMX monitoring the states of each broker. One of the states is log recovery. Thanks, Jun On Mon, May 19, 2014 at 11:15 AM, Jonas Bergström luckysw...@gmail.comwrote: Ok, I'll upgrade. Is there a way to see the status of a node that is recovering, e.g. in zookeeper or

Re: SocketServerStats not reporting bytes written or read

Is the problem with the jmx beans themselves or jmxstats? Thanks, Jun On Mon, May 19, 2014 at 2:48 PM, Xuyen On x...@ancestry.com wrote: Hi all, I have an intermittent problem with the JMX SocketServer stats on my 0.7.2 Kafka cluster. I'm collecting the SocketServerStats with jmxstats