Leah Thomas created KAFKA-13811:
-----------------------------------
Summary: Investigate sliding windows performance
Key: KAFKA-13811
URL: https://issues.apache.org/jira/browse/KAFKA-13811
Project: Kafka
Issue Type: Task
Components: streams
Reporter: Leah Thomas
We recently fixed a bug in sliding windows so that a grace period of 0ms is
properly calculated, see https://issues.apache.org/jira/browse/KAFKA-13739.
Before this patch, sliding windows with a grace period of 0ms would just skip
all records so nothing would get put into the store.
When we ran benchmarks for the 3.2 release we saw a significant drop in
performance for sliding windows on both the 3.2 and trunk branches, see the
`sliding windows` results
[here|[http://kstreams-benchmark-results.s3-website-us-west-2.amazonaws.com/summaries/process-cumulative-rate/graph.html].]
These benchmarks use a sliding window with a 0ms grace period, which means
until now we weren't sending any values to the state store when running these
benchmarks.
I ran benchmarks on the
[commit|https://github.com/apache/kafka/commit/430f9c99012d1585aa544d4dadf449963296c1fd]
before KAFKA-13739 and changed the grace period to 2 seconds to see if the bug
fix changed anything. The performance was still low for [this
run|[http://kstreams-benchmark-results.s3-website-us-west-2.amazonaws.com/experiments/sliding1min-3-5-3-3-0-430f9c9901-leah-20220408084241-streamsbench/].]
Given this, it seems like the performance for sliding windows has always been
low but we didn't realize it because the bug fixed in KAFKA-13739 was present
in the benchmarks we were running. We should investigate why the current
algorithm is slow and see if improvements can be made
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
