[
https://issues.apache.org/jira/browse/KAFKA-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-6535:
-
Description:
After KIP-220 / KIP-204, repartition topics in Streams are transient, so it is
better to set its default retention to infinity to allow any records be pushed
to it with old timestamps (think: bootstrapping, re-processing) and just rely
on the purging API to keeping its storage small.
More specifically, in {{RepartitionTopicConfig}} we have a few default
overrides for repartition topic configs, we should just add the override for
{{TopicConfig.RETENTION_MS_CONFIG}} to set it to Long.MAX_VALUE. This still
allows users to override themselves if they want via
{{StreamsConfig.TOPIC_PREFIX}}. We need to add unit test to verify this update
takes effect.
In addition to the code change, we also need to have doc changes in
streams/upgrade_guide.html specifying this default value change.
was:After KIP-220 / KIP-204, repartition topics in Streams are transient, so
it is better to set its default retention to infinity to allow any records be
pushed to it with old timestamps (think: bootstrapping, re-processing) and just
rely on the purging API to keeping its storage small.
> Set default retention ms for Streams repartition topics to Long.MAX_VALUE
> -
>
> Key: KAFKA-6535
> URL: https://issues.apache.org/jira/browse/KAFKA-6535
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Guozhang Wang
>Priority: Major
> Labels: needs-kip, newbie
>
> After KIP-220 / KIP-204, repartition topics in Streams are transient, so it
> is better to set its default retention to infinity to allow any records be
> pushed to it with old timestamps (think: bootstrapping, re-processing) and
> just rely on the purging API to keeping its storage small.
> More specifically, in {{RepartitionTopicConfig}} we have a few default
> overrides for repartition topic configs, we should just add the override for
> {{TopicConfig.RETENTION_MS_CONFIG}} to set it to Long.MAX_VALUE. This still
> allows users to override themselves if they want via
> {{StreamsConfig.TOPIC_PREFIX}}. We need to add unit test to verify this
> update takes effect.
> In addition to the code change, we also need to have doc changes in
> streams/upgrade_guide.html specifying this default value change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
