Attempt to prove Kafka transactions work

2019-10-25 Thread Edward Capriolo
Hello all, I used to work in adtech. Adtech was great. CPM for ads is 1-$5 per thousand ad impression. If numbers are 5% off you can blame javascript click trackers. Now, I work in a non addtech industry and they are really, really serious about exactly once. So there is this blog: https://www.

Transaction error in Kafka producer perf test

2019-10-25 Thread Anindya Haldar
We are evaluating Kafka for some of our use cases. As part of that effort I am trying to run an experiment with a cluster we have set up, and using the producer perf test tool supplied with the binaries. Here’s the cluster info: Runs in Kubernetes, with 4 CPUs, 32 GB RAM, 100 GB log space allo

Re: Running a stand alone Mirror Maker 2.0 cluster and Issues

2019-10-25 Thread Vishal Santoshi
Yep, I increase the number of tasks and the umber of nodes executing the tasks, the above issues appears. It creates the topics ( so non issue reaching the clusters ) but does not create the MirrorSourceConnector/s . The process does launch the MirrorCheckPointConnector, one for each node. I took

kafka stream ktable with suppress operator

2019-10-25 Thread Tao Wang
When using suppress operator with windowed Ktable, it looks like restarting the kafka stream causes the aggregated messages from the SUPPRESS-STATE-STORE published again.. Here is the sudo code .. anything I am missing or anything can be done to avoid this .. KTable, String> test = .groupByK

Re: Kafka 2.2.1 with OpenJDK 11

2019-10-25 Thread Debraj Manna
Thanks for the pointers. Does kafka 2.2.1 support openjdk 8? On Fri 25 Oct, 2019, 3:51 PM M. Manna, wrote: > It’s in Github. Look for release tab it’s rx-2 for 2.3.1. You can download > and run it using Openjdk11 and see how it goes. > > I don’t see any Jenkins build for 2.3.1 with jdk11, but

Subject: [VOTE] 2.2.2 RC2

2019-10-25 Thread Randall Hauch
Hello all, we identified around three dozen bug fixes, including an update of a third party dependency, and wanted to release a patch release for the Apache Kafka 2.2.0 release. This is the *second* candidate for release of Apache Kafka 2.2.2. (RC1 did not include a fix for https://issues.apache.o

Re: Kafka 2.2.1 with OpenJDK 11

2019-10-25 Thread M. Manna
It’s in Github. Look for release tab it’s rx-2 for 2.3.1. You can download and run it using Openjdk11 and see how it goes. I don’t see any Jenkins build for 2.3.1 with jdk11, but there’s definitely a trunk build passing for jdl11 yesterday morning. On Fri, 25 Oct 2019 at 04:16, Debraj Manna w

Re: Monitor Kafka connect jobs

2019-10-25 Thread Robin Moffatt
You can use the REST API to monitor the status of connectors and their tasks. As of AK 2.3 it's been improved so you can poll http://localhost:8083/connectors?expand=info&expand=status and use the returned data to understand the status. You can parse it and do something like this: curl -s "http://

Kubernetes and Kafka

2019-10-25 Thread Pirow Engelbrecht
Hello, I am currently working on bringing up as simple as possible Kubernetes cluster with Kafka and Zookeeper for development purposes (using minikube). The cluster needs to expose a Kafka broker to the external networking environment (for interfacing to consumers and producers outside of the