Re: Streams reprocessing whole topic when deployed but not locally

Hi Guozhang, sorry, by "app" i mean the stream processor app, the one shown in pipeline.kt. The app reads a topic of data sent by a sensor each second and generates a 20 second window output to another topic. My "problem" is that when running locally with my local kafka setup, let's say I stop it

Re: Kafka 2.2.1 with Zookeeper 3.5.5

Thx Mark for clarification. That's what I had guessed but couldn't find anything on a quick google search... So far I've not seen any errors in my environment running the Kafka 2.2.1 with Zookeeper 3.5.5 On 06-Jun-19 8:47 AM, Mark Anderson wrote: Kafka has its own version of the zookeeper c

Re: Streams reprocessing whole topic when deployed but not locally

Hello Alessandro, What did you do for `restarting the app online`? I'm not sure I follow the difference between "restart the streams app" and "restart the app online" from your description. Guozhang On Wed, Jun 5, 2019 at 10:42 AM Alessandro Tagliapietra < tagliapietra.alessan...@gmail.com> wr

Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

Hello Pieter, It's normal to see offset increasing by 2: that is because each message on that partition is from a separate transaction which Streams EOS leverage on, and whenever a a txn commits a commit marker, written as a special message would be appended but not exposed to consumers. And hence

Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

I think increasing the segment size for repartition topic should mitigate the issues. -Matthias On 6/5/19 4:43 AM, Pieter Hameete wrote: > Hi Guozhang, > > Some additional finding: it seems to only happen on Kakfa Streams repartition > topics. We haven't seen this happening for any other topic

Re: Newb Trying to Publish via Kafka CLI

Hi, Leave the ’ssl://' part off your --broker-list argument and it should work. You only need ‘host:port’. — Peter Bukowinski > On Jun 5, 2019, at 12:41 PM, jbail...@gmail.com wrote: > > Hello, > > I am trying to connect to kafka via CLI to publish messages to a topic from a > windows box.

Re: Kafka 2.2.1 with Zookeeper 3.5.5

Kafka has its own version of the zookeeper client libraries that are still 3.4.13. I'd be interested to know if it is compatible with 3.5.x now that has a stable release. Mark On Wed, 5 Jun 2019, 21:27 Sebastian Schmitz, < sebastian.schm...@propellerhead.co.nz> wrote: > Hi, > > I am currently t

Newb Trying to Publish via Kafka CLI

Hello, I am trying to connect to kafka via CLI to publish messages to a topic from a windows box. I’ve read the docs, googled and asked others (everyone I know just uses the GUI). I’m clearly missing something. I’ve included the (masked) info I use to connect via Kafka Tool followed by a shor

Kafka 2.2.1 with Zookeeper 3.5.5

Hi, I am currently testing the new Kafka 2.2.1 with new Zookeeper 3.5.5 on my local machine to see if we can upgrade the other environments with it. During startup of Kafka it shows Zookeeper-Version as 3.4.13: [2019-06-05 20:16:41,486] INFO Client environment:zookeeper.version=3.4.13-2d71af

Re: Kafka ISR to server count questions

Hello Scott, as you see, the logs are telling you what is happening: The size of the current ISR Set(1) is insufficient to satisfy the min.isr requirement of 2 for partition . It means you either increase the replication factor > 2 or decrease the min.isr to 1. Since currently your replication

Kafka ISR to server count questions

I've have some questions regarding Kafka ISR's and the number of servers one should have. In our production environment we have a 6 node Kafka cluster with min.insync.replicas=2, and all of our topics have a ReplicationFactor: of 2 When we create topics, we do not pin them to a specific bro

Streams reprocessing whole topic when deployed but not locally

Hello everyone, I've a small streams app, the configuration and part of the code I'm using can be found here https://gist.github.com/alex88/6b7b31c2b008817a24f63246557099bc There's also the log when the app is started locally and when the app is started on our servers connecting to the confluent c

Re: SASL + SSL : authentication error in broker-to-broker communication

Hi, maybe the problem is zookeeper.connect setting: zookeeper.connect=zoo:2181/kafka Which means that you have to use /kafka path with kafka-configs.sh: bin/kafka-configs.sh --zookeeper localhost:2181/kafka --alter --add-config 'SCRAM-SHA-256=[iterations=4096,password=alice-secret],SCRAM-SHA-5

RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

Hi Guozhang, Some additional finding: it seems to only happen on Kakfa Streams repartition topics. We haven't seen this happening for any other topics so far. Best, Pieter -Oorspronkelijk bericht- Van: Pieter Hameete Verzonden: Wednesday, 5 June 2019 11:23 Aan: users@kafka.apache.org

RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

Hi Guozhang, Thanks for your reply! I noticed my original mail went out twice by accident, sorry for that. We currently have a small variety of keys so not all partitions are 'actively used' indeed. The strange thing is though is that the errors occur for the partitions that actively receive r

Re: Error building

Hi Landon, I tried your command on apache/kafka:trunk with HEAD at commit ba3dc494371145e8ad35d6b85f45b8fe1e44c21f and it worked. ./gradlew -v Gradle 5.1.1 Build time: 2019