[
https://issues.apache.org/jira/browse/KAFKA-7591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17015513#comment-17015513
]
Sophie Blee-Goldman commented on KAFKA-7591:
--------------------------------------------
{quote}Shouldn't you need to reset the app if the window size changes?
{quote}
What I mean is, isn't it semantically incorrect to suddenly change the window
size (without resetting)? For example the current WindowStore implementations
store the window start time but not the end time, and don't persist the window
size in any way. If the window size is suddenly changed from 1 minute to 5 min,
all the currently stored results will be inaccurate as they only contain data
within 1 min, but Streams will now interpret it containing all the data within
that 5 minute period. We could in theory recompute all the windows, but only if
we knew the old window size (and even then that would be quite painful). If it
was originally a 1 min tumbling window, and it changes to a 5 min tumbling
window, we will lose everything in every 4/5 windows, because Streams will only
lookup the windows starting at 0min, 5min, 10min, etc (whereas before we had
windows starting at each 1min interval)
> Changelog retention period doesn't synchronise with window-store size
> ---------------------------------------------------------------------
>
> Key: KAFKA-7591
> URL: https://issues.apache.org/jira/browse/KAFKA-7591
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Jon Bates
> Priority: Major
>
> When a new windowed state store is created, the associated changelog topic's
> `retention.ms` value is set to `window-size +
> CHANGELOG_ADDITIONAL_RETENTION_MS`
> h3. Expected Behaviour
> If the window-size is updated, the changelog topic's `retention.ms` config
> should be updated to reflect the new size
> h3. Actual Behaviour
> The changelog-topic's `retention.ms` setting is not amended, resulting in
> possible loss of data upon application restart
>
> n.b. Although it is easy to update changelog topic config, I logged this as
> `major` due to the potential for data-loss for any user of Kafka-Streams who
> may not be intimately aware of the relationship between a windowed store and
> the changelog config
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
