Re: Messages are repeating in kafka

@Abhimanyu 1) My guess is that topic offsets will remain for 30 days since that is the configuration you are explicitly setting and Kafka should respect that although I don't know for sure. 2) same as #1 offsets should remain to whatever time you specify. What is the problem with setting offset

Re: Messages are repeating in kafka

Hi Kant, After setting this configuration offsets.retention.minutes . I am in doubt about the two things 1. If I am deleting a topic will that topic offsets would also get deleted or will they present for 30 days? 2. What will happen if for some topics my log.retention.hours = 168 and offsets.re

Re: Question regarding buffer.memory, max.request.size and send.buffer.bytes

I am looking for Producer tuning as mentioned in the mail, all the properties are related to producer config. This is where the property is mentioned : https://kafka.apache.org/0100/documentation.html#producerconfigs Consumer in this case if KafkaSpout from Apache-Storm. On Tue, May 23, 2017

Re: Question regarding buffer.memory, max.request.size and send.buffer.bytes

This could be for various reasons: 1) Your consumer.property settings - if you have not been acknowledging automatically, you need to provide a sufficient polling time and commit in sync/async. 2) You are not consuming the messages how you think. I don't know how you got this buffer.memory prope

Re: Kafka Streams: "subscribed topics are not assigned to any members in the group"

Hi Dimity, this sounds like a bug to me. Can you share some more details. What is your program structure? How many partitions so you have per topic? How many threads/instances to you run? When does the issue occur exactly? -Matthias On 5/23/17 12:26 PM, Dmitry Minkovsky wrote: > Certain elemen

RE: MBean for records-lag-max metric always 1.0 or -Infinity

We just tried upgrading the jar to 10.2 version and it works! Also, I was misleading with the prior version. We were running 10.0.1 KafkaServer but using the 10.1 client jar. With the 10.2 client jar working against the 10.0.1 KafkaServer, the records-lag-max metrics is now reporting correctl

MBean for records-lag-max metric always 1.0 or -Infinity

Hello, We're researching autoscaling our microservices consuming from a Kafka Topic based on the records-lag-max metric. For the following MBean object: MBeans TAB: kafka.consumer-> consumer-fetcher-manager-metric -> consumer-1 -> Attributes -> records-lag-max In JConsole, we noticed that we

Kafka Streams: "subscribed topics are not assigned to any members in the group"

Certain elements of my streams app stopped working and I noticed that my logs contain: [2017-05-23 15:23:09,274] WARN (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:361) The following subscribed topics are not assigned to any members in the group user-service : [user-service-KSTR

Re: Server Property num.partitions and Specification of Topic Paritions on Console

The server property of num.partitions is the default value used when auto.create.topics.enable=true. If you are creating topics using a utility like kafka-topics, then what you provide with that utility is what is used for topic creation irrespective of whether it is less than or more than the serv

Question regarding buffer.memory, max.request.size and send.buffer.bytes

I have set the producer properties as follows (0.10.0.0) *"linger.ms "** : **"500"** ,* *"batch.size"** : **"1000"**,* *"buffer.memory"** :**"**1**"**,* *"send.buffer.bytes"** : **"512000"* *and default * * max.request.size = *1048576 If records are sent faster than

Re: Partition assignment with multiple topics

Hi No joins - they're all separate data flows. Having separate stream instances for subsets of topics would probably work. Is doesn't seem as clean. It's also slightly more tricky to distribute the load across separate processes. We'd have to have only one stream thread allocated to each topic in

Re: Kafka Authorization and ACLs Broken

Darshan, I have not yet successfully gotten the ACLs to work in Kafka. I am still looking for help. I will update this email thread if I do find. In case you get it working, please let me know. Thanks. R On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare < purandare.dars...@gmail.com> wrote: >

Server Property num.partitions and Specification of Topic Paritions on Console

Hello, I wanted to understand the relationship between the number of partitions in default specification (server.properties) and topic specification. If I do the following: kafka-topics --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic myTopic It will actually *o

Re: Streams error handling

That's great for the value but not the key On Thu, 13 Apr 2017 at 18:27, Sachin Mittal wrote: > We are also catching the exception in serde and returning null and then > filtering out null values downstream so as they are not included. > > Thanks > Sachin > > > On Thu, Apr 13, 2017 at 9:13 PM, M

Re: Streams error handling

Hi please open one for me thank you On Thu, 13 Apr 2017 at 17:04, Eno Thereska wrote: > Hi Mike, > > Thank you. Could you open a JIRA to capture this specific problem (a > copy-paste would suffice)? Alternatively we can open it, up to you. > > Thanks > Eno > > On 13 Apr 2017, at 08:43, Mike Goul

Re: Kafka Authorization and ACLs Broken

Raghav I saw few posts of yours around Kafka ACLs and the problems. I have seen similar issues where Writer has not been able to write to any topic. I have seen "leader not available" and sometimes "unknown topic or partition", and "topic_authorization_failed" error. Let me know if you find a val

Re: Partitions as mechanism to keep multitenant segregated data

It seems like you're trying to use the partitioning mechanism as a routing mechanism, which afaik is not really its objective. It may work but it is definitely not the best approach imo. 1. You're throwing away the parallelism capabilities of Kafka. You'll have a single "queue" per customer. By th

Kafka Authorization and ACLs Broken

Hello Kafka Users I am a new Kafka user and trying to make Kafka SSL work with Authorization and ACLs. I followed posts from Kafka and Confluent docs exactly to the point but my producer cannot write to kafka broker. I get "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors. Can s

Re: Partitions as mechanism to keep multitenant segregated data

That might be ok. If that's the case, you can probably just "precreate" all the partitions for them upfront and avoid any worry about having to futz with consumers. On Tue, May 23, 2017 at 4:33 PM, David Espinosa wrote: > Thanks for the answer Tom, > Indeed I will not have more than 10 or 20 cus

Re: Partitions as mechanism to keep multitenant segregated data

Thanks for the answer Tom, Indeed I will not have more than 10 or 20 customer per cluster, so that's also the maximum number of partitions possible per topic. Still a bad idea? 2017-05-23 16:48 GMT+02:00 Tom Crayford : > Hi there, > > I don't know about the consumer, but I'd *strongly* recommend

Re: APACHE LICENSES

Hi Lis, Yes, they are free software. The full terms of the licenses are available here: https://github.com/apache/kafka/blob/trunk/LICENSE and here: https://github.com/apache/zookeeper/blob/master/LICENSE.txt Mathieu On Tue, May 23, 2017 at 5:54 AM, LISBETH SANTAMARIA GUTIERREZ wrote: > Good

APACHE LICENSES

Good morning, The licenses Kafka & Zookeeper are free software? King regards *BBVA* *Lis Santamaría Gutiérrez* *CIB Engineering - CTO - Technical Systems & Solutions* E-mail: lisbeth.santamaria.contrac...@bbva.com *LAS TABLAS III - Isabel de Colbrand 4 - pl 1 – 28050 Madrid* -- "Es

Re: Partitions as mechanism to keep multitenant segregated data

Hi there, I don't know about the consumer, but I'd *strongly* recommend not designing your application around this. Kafka has severe and notable stability concerns with large numbers of partitions, and requiring "one partition per customer" is going to be limiting, unless you only ever expect to h

Re: Partitions as mechanism to keep multitenant segregated data

Hi there, I don't know about the consumer, but I'd *strongly* recommend not designing your application around this. Kafka has severe and notable stability concerns with large numbers of partitions, and requiring "one partition per customer" is going to be limiting, unless you only ever expect to h

Partitions as mechanism to keep multitenant segregated data

Hi, In order to keep separated (physically) the data from different customers in our application, we are using a custom partitioner to drive messages to a concrete partition of a topic. We know that we are loosing parallelism per topic this way, but our requirements regarding multitenancy are high

Re: Partition assignment with multiple topics

Hi Mike, Are you performing any operations (e.g. joins) across all topics? If so, I'd think increasing the number of partitions is indeed the way to go. Partition is the unit of parallelism per topic and all topics are bound together in your app in that case. If not, your other option is to

Partition assignment with multiple topics

Hi We have a couple of hundred topics - each carrying a similar but distinct message type but to keep the total partition count down each only has 3 partitions. If I start Kafka-streams consuming all topics only 3 threads ever get assigned any partitions. I think the first thread to start gets th

How to scripting Zookeeper - Kafka startup procedure as step by step

Hi, Working on a kafka project I'm trying to set up Integration test using docker to have Zookeeper and Kafka clusters and my client(s) program and some kafkacat clients on a docker network. To set up this work context I need to script each action and I guess I have a beginner problem about Zoo