Kafka Connect task re-balance repeatedly

2018-03-21 Thread Ziliang Chen
Hi, I have 2 Kafka Connect instances runs in 2 boxes which forms a Kafka Connect Cluster. One of the instance seems doing the re-balance repeatedly in a dead loop without running the actual Sink task, the other works fine. The following is the output message in the console. May I ask if you have

Kafka Connect high CPU / Memory usage

2018-01-23 Thread Ziliang Chen
Hi, I run Kafka Connect 0.11.2 (Kafka version) with only one instance and 6 tasks in it for a while, it then showed very high CPU usage ~250 % and memory consumption 10 Gb memory and it reported consume group rebalance warning. After the Kafka connect was killed and restarted, it showed the same

1 to N transformers in Kafka Connect

2017-12-27 Thread Ziliang Chen
Hi, May i ask if it is possible to do 1 kafka record to many Kafka Connect records transformation ? I know we have 1:1 transformation supported in Kafka Connect, but it appears to me there are quite some user cases which requires 1:N transformation Thank you very much ! -- Regards, Zi-Liang