Changing tumbling windows inclusion

Hello everyone, I'm trying to window a stream of machine production data, this use case needs a message with timestamp ending at the tumbling window end to be included in the current window not the next, because the message production amount refers to the previous x seconds. This doesn't work beca

Re: Best Practice Scaling Consumers

1. Yes, you may have to overprovision the number of partitions to handle the load peaks. Refer this document to choose the no. of partitions. 2. KIP-429

Re: SELECT * from table shows no data until new messages come in

Thank you , Steve, for the suggestion - I did already set this property to earliest: Just to make sure - I set it again - below is the output from the CLI: ksql> SET 'auto.offset.reset' = 'earliest'; Successfully changed local property 'auto.offset.reset' from 'earliest' to 'earliest'. ksql> s

Re: SELECT * from table shows no data until new messages come in

Sorry, I hit send too quickly. On Mon, May 6, 2019 at 4:42 PM Steve Howard wrote: > Hi Marina, > > Try... > > set 'auto.offset.reset'='earliest'; > > > Thanks, > > > Steve > > On Mon, May 6, 2019 at 4:40 PM Marina Popova > wrote: > >> Hi, >> Trying to understand if I'm missing something... >> I

Re: SELECT * from table shows no data until new messages come in

Hi Marina, Try... set 'auto.offset.reset'='earliest'; Thanks, Steve On Mon, May 6, 2019 at 4:40 PM Marina Popova wrote: > Hi, > Trying to understand if I'm missing something... > I'm using Confluent KSQL setup (running all components in Docker > containers). I've created a Table from a Str

SELECT * from table shows no data until new messages come in

Hi, Trying to understand if I'm missing something... I'm using Confluent KSQL setup (running all components in Docker containers). I've created a Table from a Stream, as following: CREATE TABLE rc_ip_key_agg_ptable AS SELECT remote_addr, windowStart(), windowEnd(), count(*) FROM

Question related to Kafka Connect

Hi There, We currently have the Kafka connect cluster set up with 4 nodes and registered the connectors to read the data from Kafka topics and write to S3. When we were trying to collect the JMX metrics using MBeans using java API we are able to get the metrics on the node on which the conn

RE: Kafka hardware requirements

Thank you Kurt for your insight. Jean-Marc. Sent from Mail for Windows 10 From: Kurt Rudolph Sent: Monday, May 6, 2019 9:22:24 AM To: users@kafka.apache.org Subject: Re: Kafka hardware requirements I am curr

Best Practice Scaling Consumers

Hi all, I’m new to Kafka and have a very basic question: We build a cloud-scale platform and evaluate if we can use Kafka for pub-sub messaging between our services. Most of our services scale dynamically based on load (number of requests, CPU load etc.). In our current architecture, services

Re: Kafka hardware requirements

I am currently running a few clusters on virtual machines connected to a VSAN. I have not had any major ssues. My clusters are small; 3 virtual machines (4 cpu and 8 GB memory each). I am routinely processing 20K msg/sec with surges of over 35K msg/sec. Based on my performance testing, I don't

RE: Mirror Maker tool is not running

Dear Team Members Please find these below configurations for mirror maker tool scripts source-cluster.config bootstrap.servers=192.168.175.128:9092,192.168.175.129:9092,192.168.175.130:9092 group.id=mirror-maker-consumer exclude.internal.topics=true client.id=mirror_maker_consumer secu

Re: Restart process after adding control.plane.listener.name config in Kafka 2.2.0

Thanks a lot Kamal for the reference. Cheers! -- Jonathan On Mon, May 6, 2019 at 11:52 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Migration plan for this change is mentioned in the KIP-291 > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-291%3A+Separating+c

Re: Restart process after adding control.plane.listener.name config in Kafka 2.2.0

Migration plan for this change is mentioned in the KIP-291 . Take a look at the migration plan. Migration plan: 2 rounds of rolling upgrades are needed to pick up the >