How to get consumer creating time and last consuming time

Hi all, In kafka0.8, I can get these infomations from zk.But how to get them in kafka0.10? Thanks Yuanjia Li

Re: Message order different each time stream is replayed?

Ah, That explains it. I have 3 partitions. On 1 Dec 2016 5:42 a.m., "Apurva Mehta" wrote: How many partitions do you have in that topic. Kafka only guarantees a total ordering of messages within a partition, not across partitions of a topic. If you want total ordering over

Re: Message order different each time stream is replayed?

How many partitions do you have in that topic. Kafka only guarantees a total ordering of messages within a partition, not across partitions of a topic. If you want total ordering over the entire topic, you need to create a topic with a single partition. On Wed, Nov 30, 2016 at 4:10 AM, Ali Akhtar

Re: Messages intermittently get lost

Hey Martin, Do you want to create a jira for this and report the issue? -Flavio > On 30 Nov 2016, at 18:33, Martin Gainty wrote: > > a shock when a zk script goes fubar > > > zookeeper devs can we get some help for sh zkServer.sh status? > > > Thanks! > > Martin >

Re: Writing a customized principal builder for authorization

this is a quick and dirty test you can use: org.apache.kafka.common.network.SSLSelectorTest: //Truststore needs to contain keystore/cert that contains the actual principal you will use File trustStoreFile = File.createTempFile("truststore", ".jks"); Map

FW: mirror and schema topics

Hello! I'm trying to mirror a kafka cluster, then run connect on the mirror. It seems the schemas are not getting moved in the mirror though, so I get the following error. Is this a configuration problem? Thank you for the help! Mirror>curl -X GET http://localhost:8081/subjects []

mirror and schema topics

Hello! I'm trying to mirror a kafka cluster, then run connect on the mirror. It seems the schemas are not getting moved in the mirror though, so I get the following error. Is this a configuration problem? Thank you for the help! Mirror>curl -X GET http://localhost:8081/subjects []

Save the date: ApacheCon Miami, May 15-19, 2017

Dear Apache enthusiast, ApacheCon and Apache Big Data will be held at the Intercontinental in Miami, Florida, May 16-18, 2017. Submit your talks, and register, at http://apachecon.com/ Talks aimed at the Big Data section of the event should go to

Re: Initializing StateStores takes *really* long for large datasets

+1 on this. Ara. > On Nov 30, 2016, at 5:18 AM, Mathieu Fenniak > wrote: > > I'd like to quickly reinforce Frank's opinion regarding the rocksdb memory > usage. I was also surprised by the amount of non-JVM-heap memory being > used and had to tune the 100 MB

Re: Messages intermittently get lost

a shock when a zk script goes fubar zookeeper devs can we get some help for sh zkServer.sh status? Thanks! Martin __ From: Zac Harvey Sent: Wednesday, November 30, 2016 10:25 AM To:

KStreams app in YARN

Hello, I was wondering if anyone has deployed a KStreams app to YARN, perhaps using Slider? If so, what was your experience? Thanks! -Aaron

Re: Writing a customized principal builder for authorization

How do you then modify Kafka's searchable classpath to pick up this new principal.builder.class classfile from a JAR somewhere on the filesystem? In other words, I change my server.properties to: principal.builder.class=com.example.kafkautils.MyCustomKafkaPrincipalBuilder How will Kafka be

Re: Writing a customized principal builder for authorization

Hi Kriti, You will have to implement the Principal Builder interface and provide the full class path in broker config. I don't remember the exact config name right now, but you can search for some config by name "principalbuilder.class" in the broker configs. Once you do this, Kafka will

Re: log.dirs balance?

Thanks for the report, Karolis. I have a potential theory for how this happened and I'm wondering if it's possibly valid: I have 9 partitions on a machine with 3 disks and they get assigned exactly as you'd expect: d1: t{1,2,3} d2: t{4,5,6} d3: t{7,8,9} Then, a disk fails or something

Re: question about property advertised.listeners

The advertised.listeners property would not help you to map a different port. If you need to talk to brokers at a different port you should also set the listeners property accordingly. Say if you want the broker to listen at port 10092 as an example, the server.properties needs to contain these

Writing a customized principal builder for authorization

Hi, Can anyone help me or point me to any resources that can be of help for writing a customized principal builder to use in Authorization using ACLs? I've enabled SSL authentication scheme for both clients and brokers but I would like to change the principal name to just the original name and

Re: Messages intermittently get lost

Hi Martin, makes sense. When I SSH into all 3 of my ZK nodes and run: sh zkServer.sh status All three of them give me the following output: JMX enabled by default zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "(" unexpected (expecting "fi") Looks like a bug in the ZK

Re: Convert log file in Avro json encoding to Avro binary encoding and send into Kafka

Hi Ewen, Thank you for your advise. I reached a following compromise: My application sends records in the Avro Binary encoding to Kafka directly using Confluent's serializers(io.confluent/kafka-avro-serializer), which integrated with schema registry, and also does logging in the Avro JSON

Re: Initializing StateStores takes *really* long for large datasets

Mathieu, You are absolutely right. We've written about the memory management strategy below: https://cwiki.apache.org/confluence/display/KAFKA/Discussion%3A+Memory+Management+in+Kafka+Streams

Release Kafka 0.9.0.2?

Hello Apache Kafka community, Would it be possible to release 0.9.0.2? It has few important fixes, like KAFKA-3594 and would be helpful for us that cannot upgrade to 0.10.x yet. Kind regards, Stevo Slavic.

Re: Initializing StateStores takes *really* long for large datasets

I'd like to quickly reinforce Frank's opinion regarding the rocksdb memory usage. I was also surprised by the amount of non-JVM-heap memory being used and had to tune the 100 MB default down considerably. It's also unfortunate that it's hard to estimate the memory requirements for a KS app

Re: Writing a customized principal builder for authorization using ACLs

i would suggest understanding SunJSSE security provider(s) implemented by Oracle https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#SunJSSE i would also suggest understanding complimentary security providers from BouncyCastle

Message order different each time stream is replayed?

While I was connected to console-consumer.sh, I posted a few messages to a Kafka topic, one message at a time, across a few hours. I'd post a message, see it arrive in console-consumer, a few mins later I'd post the next message, and so on. They all arrived in order. However, when I now try to

Re: OOM errors

My apologies. In fact the 'aggregate' step includes this: 'TimeWindows.of(20 * 60 * 1000L).advanceBy(60 * 1000L)' On Tue, Nov 29, 2016 at 9:12 PM, Guozhang Wang wrote: > Where does the "20 minutes" come from? I thought the "aggregate" operator > in your > >

Writing a customized principal builder for authorization using ACLs

Hi, Can anyone help me or point me to any resources that can be of help for writing a customized principal builder to use in Authorization using ACLs? I've enabled SSL authentication scheme for both clients and brokers but I would like to change the principal name to just the original name and