Re: Kafka behind NAT

Try setting: advertised.listeners=EXTERNAL://:9093,INTERNAL://:9092 inter.broker.listener.name=INTERNAL listener.security.protocol.map=EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT Then you should be able to use :9093 as your bootstrap.servers from outside the network or :9092 from inside. Obviously th

Architectural patterns for full log replayability

Hello Kafka Users, we'd like to understand how you are designing systems based on Kafka so to be able to replay the full log. In particular, let's take the following example: - A product service streams products - A purchase service streams purchases - A recommendation service join the two and de

How does KStream transform performs repartitioning?

Hello users, we are performing a Transform so that out of a larger message we emit a new output record only if that specific field has changed. Since we introduced that to reduce the number of output records, our final Kstream - KStream windowed join is not ticking anymore, although the window i

Failed on publish to Kafka

Hi, I have 4 producers, writing to a kafka cluster with 12 partitions. Periodically (about once per hour or two), we get exceptions while trying to send messages to kafka. The exceptions are “Expiring 12 record(s) for -: 49193 ms has passed since batch creation plus linger time”. We get that for

Re: How does KStream transform performs repartitioning?

Hello Edmondo, If you have a join operator following the transform() operator, then the joining streams will be sent to a repartition topic, and the join operator's hosted thread will then read from that repartition topic. This is for "re-shuffling" the streams since the key of the stream record m

Re: Failed on publish to Kafka

Hi, You said "On all of those setting there was no change in the described behavior" - this is slightly confusing. Could you please clarify this? If there is no change, that means everything is working :) ? >From the provided exception stack, it seems as if you are waiting to batch a lot of reque

Re: Exactly Once, Producer configuration

From: thomas.k...@t-systems.com Sent: Monday, May 14, 2018 12:13 PM To: users@kafka.apache.org Subject: WG: Exactly Once, Producer configuration Hi, the following article describes how to configure Apache Kafka system to enforce exactly once approach: https:

Kafka consumer to unzip stream of .gz files and read

HI Team, Greeting! I am facing one issue on kafka consumer using python hope you guys help us to resolve this issue Kafka consumer to unzip stream of .gz files and read Kafka producer is sending .

Increase in Memory Cache After Upgrading to 1.1.0

*Hello All,We recently upgraded Kafka from 0.10.2 to 1.1.0 and we've seen an increase in the amount of cached memory used, resulting in the servers going into swap. Has anyone else seen this issue or would know why this would happen after the 1.1.0 upgrade? Thanks*

Frequent consumer rebalances, auto commit failures

We have a 3 broker Kafka 0.10.0.1 cluster. There we have 3 topics with 10 partitions each. We have an application which spawns threads as consumers. We spawn 5 consumers for each topic. I am observing that consider group randomly keeps rebalancing. Then many times we see logs saying "Revoking parti

Re: Failed on publish to Kafka

Hi, Thanks for the quick response. What I meant is, that had those exceptions on both versions (0.10.1 and 1.0.1), so upgrading didn't solve the problem. And the same for increasing the cluster size. Here are the properties we set: "key.serializer" = "org.apache.kafka.common.serialization.Stri

How to achieve exactly one semantic in Kafka consumer

Hi, Currently, I'm using kafka_2.11-0.10.2.0. In this working in almost once semantic in Kafka consumer, But I need to change that in exactly once semantics. -- With Regards, Karthick. K

Re: Failed on publish to Kafka

Can you try following? 1) reduce buffer size to exactly 2x of your batch size ? 2) back up and save you old properties file. 3) create a dummy topic 4) produce and consume messages using the new config. Let us know. On Wed, 23 May 2018 at 07:02, Omer Litov wrote: > Hi, > Thanks for the quic