[VOTE] 2.3.0 RC3

2019-06-19 Thread Colin McCabe
Hi all, We discovered some problems with the second release candidate (RC2) of 2.3.0. Specifically, KAFKA-8564. I've created a new RC which includes the fix for this issue. Check out the release notes for the 2.3.0 release here: https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/RELEASE_NOTES.h

Re: Can kafka internal state be purged ?

2019-06-19 Thread Parthasarathy, Mohan
We do explicitly set the grace period to zero. I am going to try the new version -mohan On 6/19/19, 12:50 PM, "Parthasarathy, Mohan" wrote: Thanks. We will give it a shot. On 6/19/19, 12:42 PM, "Bruno Cadonna" wrote: Hi Mohan, I realized that my pre

Re: Can kafka internal state be purged ?

2019-06-19 Thread Parthasarathy, Mohan
Thanks. We will give it a shot. On 6/19/19, 12:42 PM, "Bruno Cadonna" wrote: Hi Mohan, I realized that my previous statement was not clear. With a grace period of 12 hour, suppress would wait for late events until stream time has advanced 12 hours before a result would be e

Re: Can kafka internal state be purged ?

2019-06-19 Thread Bruno Cadonna
Hi Mohan, I realized that my previous statement was not clear. With a grace period of 12 hour, suppress would wait for late events until stream time has advanced 12 hours before a result would be emitted. Best, Bruno On Wed, Jun 19, 2019 at 9:21 PM Bruno Cadonna wrote: > > Hi Mohan, > > if you

Re: Can kafka internal state be purged ?

2019-06-19 Thread Bruno Cadonna
Hi Mohan, if you do not set a grace period, the grace period defaults to 12 hours. Hence, suppress would wait for an event that occurs 12 hour later before it outputs a result. Try to explicitly set the grace period to 0 and let us know if it worked. If it still does not work, upgrade to version

Re: Can kafka internal state be purged ?

2019-06-19 Thread Parthasarathy, Mohan
No, I have not set any grace period. Is that mandatory ? Have you seen problems with suppress and windows expiring ? Thanks Mohan On 6/19/19, 12:41 AM, "Bruno Cadonna" wrote: Hi Mohan, Did you set a grace period on the window? Best, Bruno On Tue, Jun 18, 201

redistribute data partition

2019-06-19 Thread andrea sposito
I'm relative new to Kafka and probably I'd have a miss understand on how rebalancing works, in my case I've created a topic with 4 partition and then I've publish some data. To test our product we start to consume data with 4 clients with the same consumer group. For increase consumer throughput

Re: retention.ms not working for a single topic

2019-06-19 Thread Tom Vernon
This is my topic setup: Topic:mytopic PartitionCount:6 ReplicationFactor:3 Configs:retention.ms =60480,cleanup.policy=delete,retention.bytes=580 Topic: mytopic Partition: 0 Leader: 1 Replicas: 1,3,2 Isr: 3,2,1 Topic: mytopic Partition: 1 Leader: 2 Replicas: 2,1,3 Isr: 3,2,1 Topic: myto

Re: Can kafka internal state be purged ?

2019-06-19 Thread Bruno Cadonna
Hi Mohan, Did you set a grace period on the window? Best, Bruno On Tue, Jun 18, 2019 at 2:04 AM Parthasarathy, Mohan wrote: > > On further debugging, what we are seeing is that windows are expiring rather > randomly as new messages are being processed. . We tested with new key for > every new