Kafka connect rebalance problem

Hi All, We had done a POC with kafka s3 connect confluent platform and have been trying to move the setup to production. In the current setup we have 7 kafka topics and each topic drains the messages to a s3 bucket using s3 confluent connect. My setup: There are 8 workers distributed across 8 ubu

Kafka Streams 0.11.0.1 Rebalancing Delay

Hi, I'm upgrading a kafka streams application from 0.10.2.1 to 0.11.0.1, running against a kafka cluster with version 0.10.2.1. The application uses a couple of state stores. When stopping/starting the application prior to the upgrade (with 0.10.2.1 client) on 2 instances, it was up and running i

Re: [VOTE] 1.0.0 RC1

If you don't use the default Scala version, you have to set the SCALA_VERSION environment variable for the bin scripts to work. Ismael On 17 Oct 2017 1:30 am, "Vahid S Hashemian" wrote: Hi Guozhang, I'm not sure if this should be covered by "Java 9 support" in the RC note, but when I try to bu

Re: KIP-99 streams global ktable - slowly changing dimension type 2 supported?

Hello Chris, The global table described in KIP-99 will keep the most recent snapshot of the table when applying updates to the table, i.e. it is like type 1: overwrite. So when a table or stream is joined with the global table, it is always joined with the most recent values of the global table.

kafka replication all crashed

hi, all when a topic partition's all isr crashed, the partition is Unavailability, why don't like hdfs when a replication in isr crashed and auto move the rep to another alive one, what's the original intention of this design? iPhone

Re: Consumer Offsets partition skew on Kafka 0.10.1.1

So, we finally figured out why our __consumer_offsets topics was so skewed in volume across partitions. Turns out that the partition is chosen based on the consumer group_id (which makes sense), and we had a few extremely over-committing consumers in prod. A few of them were committing several th

Re: [VOTE] 1.0.0 RC1

Hi Guozhang, I'm not sure if this should be covered by "Java 9 support" in the RC note, but when I try to build jars from source using Java 9 (./gradlew -PscalaVersion=2.12 jar) even though the build reports as succeeded, it doesn't seem to have been successful: $ bin/zookeeper-server-start.sh

Re: [kafka-clients] Re: [VOTE] 1.0.0 RC1

Thanks Jun, I will roll out a new RC for this PR then. Guozhang On Mon, Oct 16, 2017 at 2:19 PM, Jun Rao wrote: > Hi, Guozhang, > > Onur found an existing performance bug in the controller when there are > lots of partitions. The fix is simple (https://github.com/apache/ > kafka/pull/4075) an

Re: [kafka-clients] Re: [VOTE] 1.0.0 RC1

Hi, Guozhang, Onur found an existing performance bug in the controller when there are lots of partitions. The fix is simple ( https://github.com/apache/kafka/pull/4075) and reduces the controlled shutdown time from 6.5 mins to 30 secs, with 25K partitions, RF=2 and 5 brokers. It would be useful t

KIP-99 streams global ktable - slowly changing dimension type 2 supported?

The streams global ktable wiki page [1] describes a data warehouse syle operation whereby dimension tables are joined to fact tables. I’m interested in whether this approach works for type 2 slowly changing dimensions [2]? In type 2 scd the dimension record history is preserved and the fact table

Re: Custom converter with Kafka Connect ?

Hi Jehan, I've run into the same issue last week and also got a "class could not be found" error. Konstantine Karantasis helpfully pointed me towards https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-6007 To test this, I simply copied my SMT jar to the folder of the connector I was u

Re: [VOTE] 1.0.0 RC1

Hi Tom, Thanks for pointing it out. I meant to say Oct. 17th, Tuesday, for a 72 hours period. That being said, we need to have a lazy majority to accept a release RC according to our bylaws ( https://cwiki.apache.org/confluence/display/KAFKA/Bylaws). And if we cannot achieve that via thorough tes

replication all crashed

hi, all when a topic partition's all isr crashed, the partition is Unavailability, why don't like hdfs when a replication in isr crashed and auto move the rep to another alive one, what's the original intention of this design?

kafka replication all crashed

hi, all when a topic partition's all isr crashed, the partition is Unavailability, why don't like hdfs when a replication in isr crashed and auto move the rep to another alive one, what's the original intention of this design?

Re: [VOTE] 1.0.0 RC1

Hi Guozhang, This says the due date on the testing is October 13th, which was the day this email was sent. Is that accurate, or is it meant to read October 17th, which is next Tuesday? I feel like this short a testing window for a 1.0 RC is a little low, as 1.0 is clearly a big announcement of st