xinyuiscool commented on a change in pull request #1190: SAMZA-2352: Use
min.compaction.lag.ms to avoid compacting the Kafka changelog topic
URL: https://github.com/apache/samza/pull/1190#discussion_r336124114
##########
File path: samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala
##########
@@ -338,6 +340,15 @@ class KafkaConfig(config: Config) extends
ScalaMapConfig(config) {
kafkaChangeLogProperties.setProperty("segment.bytes",
KafkaConfig.CHANGELOG_DEFAULT_SEGMENT_SIZE)
kafkaChangeLogProperties.setProperty("delete.retention.ms",
String.valueOf(new StorageConfig(config).getChangeLogDeleteRetentionInMs(name)))
+
+ // To enable transactional state, we will need to avoid the head of the
changelog
+ // (the messages after last checkpoint) being log-compacted so we can trim
the rest of the updates.
+ // We use min.compaction.log.ms to control the compaction time.
+ if (new TaskConfig(this).getTransactionalStateRestoreEnabled) {
+ kafkaChangeLogProperties.setProperty(KafkaConfig.MIN_COMPACTION_LAG_MS,
+ String.valueOf(StorageConfig.DEFAULT_CHANGELOG_MIN_COMPACTION_LAG_MS))
Review comment:
no, the override is already happening by appending all the properties
specified after stores.x.changelog.kafka.xxxxx. It's in the line of code below.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
