Re: Mirror a partition of a topic

Flume could be an option with an Interceptor although the throughput could be less compared to Mirror Maker with compression and shallow iterator enabled. On Tue, Aug 25, 2015 at 10:28 PM tao xiao wrote: > In the trunk code mirror maker provides the ability to filter out messages > on demand by s

Re: Issue when enabling SSL on broker

Hi, Harsha, I appreciate you very much for your response and the bash script you provided to generate the keystores works for me and solve the problem. I was wondering it was caused by the cipher suite differences between openjdk and oracle-jdk, anyway it is not that case. Finally I got it worked

Re: Not able to find release version with patch jira-1591

Hi Amar, That JIRA has a patch attached, but it hasn't been committed yet. From the look of it, it's just waiting for a review/commit. -Ewen On Tue, Aug 25, 2015 at 4:18 PM, Amar wrote: > Hi, > > WE are seeing lot of un-necessary logging in server.log. As per Jira, it > got fixed in 1591. But

Re: Mirror a partition of a topic

In the trunk code mirror maker provides the ability to filter out messages on demand by supplying a message handler. https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L443 On Wed, 26 Aug 2015 at 11:35 Binh Nguyen Van wrote: > Hi all, > > Is there any t

Re: Issue when enabling SSL on broker

Hi,       Turns out to be a bug in the instructions in the wiki . I fixed it can you please retry generating the truststore and keystore https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka . checkout this section "All of the above steps in a bash script” to generate the keys

Re: Issue when enabling SSL on broker

Hi Xiang,          Did you try following the instructions here  https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka . Whats the output of openssl s_client and which version of java and OS are you using. Thanks, Harsha On August 25, 2015 at 8:42:18 PM, Xiang Zhou (Samuel) (

Issue when enabling SSL on broker

Hi, folks, I am just following up to see if anybody has clue on the following issue when I was following the guidelines provided in the step of running openssl to verify the SSL setup: [2015-08-25 22:29:15,385] WARN Error in I/O with connection to X (org.apache.kafka.common.network.Selector)

Mirror a partition of a topic

Hi all, Is there any tool that allows to mirror a partition of a topic? I looked at the mirror maker but it is used to mirror the whole topic and I don't see any option that allow me to configure partitions. I want to mirror our live data to staging environment but my staging environment can't ha

Not able to find release version with patch jira-1591

Hi, WE are seeing lot of un-necessary logging in server.log. As per Jira, it got fixed in 1591. But I couldn't able to find matched release. Can you please advise https://issues.apache.org/jira/browse/KAFKA-1591 Thanks Amar

Fwd: [kafka-clients] Re: Get kafka connection status

Can you help solving this question? Thanks! -- Forwarded message -- From: "yoel g" Date: Aug 25, 2015 16:07 Subject: [kafka-clients] Re: Get kafka connection status To: "kafka-clients" Cc: I forgot to mention that I'm using JAVA with the 0.8.1.1 kafka version library. On Tuesday

Re: kafka producer-perf-test.sh compression-codec not working

Hi Erik, Thanks for these inputs I'll implement it Regards, Prabhjot On Aug 25, 2015 11:53 PM, "Helleren, Erik" wrote: > Prabhjot, > You can’t do it with producer perf test, but its relatively simple to > implement. The message body includes a timestamp of when your producer > produces, and the

Re: kafka producer-perf-test.sh compression-codec not working

Prabhjot, You can’t do it with producer perf test, but its relatively simple to implement. The message body includes a timestamp of when your producer produces, and the consumer looks at the difference between between the timestamp in the body and the current timestamp. Or, if you were looking

Re: Kafka tools | zookeeper-shell.sh input args doesn't work

Well, it seems to be a zookeeper problem as thought. It has an issue on Jira: https://issues.apache.org/jira/browse/ZOOKEEPER-1897 ;) João Reis On 25/08/15 15:04, "João Reis" wrote: >Hi to all, > >I downloaded the latest stable kafka version 0.8.2.1 and got it working >on centos host. >

Re: kafka producer-perf-test.sh compression-codec not working

Hi Erik, Thanks for your inputs. How can we measure round trip latency using kafka-producer-perf-test.sh ? or any other tool ? Regards, Prabhjot On Tue, Aug 25, 2015 at 7:41 PM, Helleren, Erik wrote: > Prabhjot, > When no compression is being used, it should have only a tiny impact on > perf

Re: Metrics to monitor in Kafka

This document explains some important metrics http://docs.confluent.io/1.0/kafka/monitoring.html Also, you can check the kafka demo at https://apps.sematext.com/demo. As a side thing, as I have done, you can report every stat to Ganglia and then see, according to your domain, what are the most i

Storing offsets in ZooKeeper for a SimpleConsumer

Hi, We are writing our own consumer implementation and need to store offsets manually in ZooKeeper. Currently we are storing them in a top-level znode with a structure similar to what is found in /consumers. For example, we have /[consumer-name]/offsets/partition/[partition-id] In looking into s

Metrics to monitor in Kafka

Hi, What are the important metrics to monitor in Kafka (as explained here ) and raise alarms when the value reaches certain value? I am looking for some guidelines similar to this excellent monitoring doc

Re: kafka producer-perf-test.sh compression-codec not working

Prabhjot, When no compression is being used, it should have only a tiny impact on performance. But when it is enabled it will make it as though the message payload is small and nearly constant, regardless as to how large the configured message size is. I think that the answer is that this is r

Kafka tools | zookeeper-shell.sh input args doesn't work

Hi to all, I downloaded the latest stable kafka version 0.8.2.1 and got it working on centos host. I also have a zookeeper running on the localhost:2181. I can start and stop kafka without any problem and it is working as expected. My problem is with the zookeeper-shell.sh script. I am trying

Re: kafka producer-perf-test.sh compression-codec not working

Hi Erik, I have put my efforts on the produce side till now, Thanks for making me aware that consumer will decompress automatically. I'll also consider your point on creating real-life messages But, I have still have one confusion - Why would the current ProducerPerformance.scala compress an Ar

Re: kafka producer-perf-test.sh compression-codec not working

Hi Prabhjot, There are two important things to know about kafka compression: First uncompression happens automatically in the consumer (https://cwiki.apache.org/confluence/display/KAFKA/Compression) so you should see ascii returned on the consumer side. The best way to see if compression has happe

Re: Painfully slow kafka recovery

So okay, this is a little embarassing but the core of the issue was that max open files was not set correctly for kafka. It was not an oversight, but a few things together caused that the system configuration was not changed correctly, resulting in the default value. No wonder that kafka behav

kafka producer-perf-test.sh compression-codec not working

Hi, I have bene trying to use kafka-producer-perf-test.sh to arrive at certain benchmarks. When I try to run it with --compression-codec values of 1, 2 and 3, I notice increased throughput compared to NoCompressionCodec But, When I checked the Producerperformance.scala, I saw that the the `produc

0.8.x

Hi, We currently run 0.7.x on our clusters and are now finally getting around to upgrading to kafka latest. One thing that has been holding us back is that we can no longer use a VIP to front the clusters. I understand we could use a VIP for metadata lookups, but we have 100,000 + producers to at

RE: Topic partitions randomly failed on live system

This has happened again on the same system. Is anyone able to offer any pointers towards a possible cause, as we have no idea what is wrong or how to stop it happening again. We need to diagnose and fix this issue soon, as we can't have a live system that randomly fails due to unknown causes!