We would find this KIP very useful. Our particular use case falls into the "application mistakes" portion of the KIP.
We are storing source of truth data in log compacted topics, similar to the Confluent Schema Registry. One situation we had recently was a misbehaving application. It sent data into its log-compacted source-of-truth topic, with the same keys but with corrupted message bodies. It took several days for us to notice this and by that time, log compaction had occurred. We lost data. If we had been able to specify that log compaction would not occur for say 14 days, this would give us 2 weeks to notice and recover from this situation. We could not have prevented the application from appending bad data after the good data, but at least the good data would still be present in the not-yet-compacted portion of the log, and we could write some manual recovery process to extract the good data out and do with it whatever we needed. Because we do not have this facility, we currently have a rather complicated "backup" procedure whereby we use mirrormaker to mirror from our log compacted topic to a time-retention based topic. We have 14 day retention on the time-based topic, thus allowing us 14 days to recover from any data corruption on our log compacted topic. There is also a (imho) clever thing we do where we restart mirroring from the beginning of the log-compacted topic every 7 days. This allows us to make sure that even for very old non-changing keys, we always have a copy of them in the time-based topic. Gwen helped us come up with that backup plan many months ago (Thanks Gwen!) but having a configurable "do not compact the last X days" setting would likely be enough to satisfy our backup needs. -James > On May 16, 2016, at 9:20 PM, Jay Kreps <j...@confluent.io> wrote: > > Yeah I think I gave a scenario but that is not the same as a concrete use > case. I think the question you have is how common is it that people care > about this and what concrete things would you build where you had this > requirement? I think that would be good to figure out. > > I think the issue with the current state is that it really gives no SLA at > all, the last write to a segment is potentially compacted immediately so > even a few seconds of lag (if your segment size is small) would cause this. > > -Jay > > On Mon, May 16, 2016 at 9:05 PM, Gwen Shapira <g...@confluent.io> wrote: > >> I agree that log.cleaner.min.compaction.lag.ms gives slightly more >> flexibility for potentially-lagging consumers than tuning >> segment.roll.ms for the exact same scenario. >> >> If more people think that the use-case of "consumer which must see >> every single record, is running on a compacted topic, and is lagging >> enough that tuning segment.roll.ms won't help" is important enough >> that we need to address, I won't object to proceeding with the KIP >> (i.e. I'm probably -0 on this). It is easy to come up with a scenario >> in which a feature is helpful (heck, I do it all the time), I'm just >> not sure there is a real problem that cannot be addressed using >> Kafka's existing behavior. >> >> I do think that it will be an excellent idea to revisit the log >> compaction configurations and see whether they make sense to users. >> For example, if "log.cleaner.min.compaction.lag.X" can replace >> "log.cleaner.min.cleanable.ratio" as an easier-to-tune alternative, >> I'll be more excited about the replacement, even without a strong >> use-case for a specific compaction lag. >> >> Gwen >> >> On Mon, May 16, 2016 at 7:46 PM, Jay Kreps <j...@confluent.io> wrote: >>> I think it would be good to hammer out some of the practical use cases--I >>> definitely share your disdain for adding more configs. Here is my sort of >>> theoretical understanding of why you might want this. >>> >>> As you say a consumer bootstrapping itself in the compacted part of the >> log >>> isn't actually traversing through valid states globally. i.e. if you have >>> written the following: >>> offset, key, value >>> 0, k0, v0 >>> 1, k1, v1 >>> 2, k0, v2 >>> it could be compacted to >>> 1, k1, v1 >>> 2, k0, v2 >>> Thus at offset 1 in the compacted log, you would have applied k1, but not >>> k0. So even though k0 and k1 both have valid values they get applied out >> of >>> order. This is totally normal, there is obviously no way to both compact >>> and retain every valid state. >>> >>> For many things this is a non-issue since they treat items only on a >>> per-key basis without any global notion of consistency. >>> >>> But let's say you want to guarantee you only traverse valid states in a >>> caught-up real-time consumer, how can you do this? It's actually a bit >>> tough. Generally speaking since we don't compact the active segment a >>> real-time consumer should have this property but this doesn't really >> give a >>> hard SLA. With a small segment size and a lagging consumer you could >>> imagine the compactor potentially getting ahead of the consumer. >>> >>> So effectively what this config would establish is a guarantee that as >> long >>> as you consume all messages in log.cleaner.min.compaction.lag.ms you >> will >>> get every single produced record. >>> >>> -Jay >>> >>> >>> >>> >>> >>> On Mon, May 16, 2016 at 6:42 PM, Gwen Shapira <g...@confluent.io> wrote: >>> >>>> Hi Eric, >>>> >>>> Thank you for submitting this improvement suggestion. >>>> >>>> Do you mind clarifying the use-case for me? >>>> >>>> Looking at your gist: >>>> https://gist.github.com/ewasserman/f8c892c2e7a9cf26ee46 >>>> >>>> If my consumer started reading all the CDC topics from the very >>>> beginning in which they were created, without ever stopping, it is >>>> obviously guaranteed to see every single consistent state of the >>>> database. >>>> If my consumer joined late (lets say after Tq got clobbered by Tr) it >>>> will get a mixed state, but if it will continue listening on those >>>> topics, always following the logs to their end, it is guaranteed to >>>> see a consistent state as soon a new transaction commits. Am I missing >>>> anything? >>>> >>>> Basically, I do not understand why you claim: "However, to recover all >>>> the tables at the same checkpoint, with each independently compacting, >>>> one may need to move to an even more recent checkpoint when a >>>> different table had the same read issue with the new checkpoint. Thus >>>> one could never be assured of this process terminating." >>>> >>>> I mean, it is true that you need to continuously read forward in order >>>> to get to a consistent state, but why can't you be assured of getting >>>> there? >>>> >>>> We are doing something very similar in KafkaConnect, where we need a >>>> consistent view of our configuration. We make sure that if the current >>>> state is inconsistent (i.e there is data that are not "committed" >>>> yet), we continue reading to the log end until we get to a consistent >>>> state. >>>> >>>> I am not convinced the new functionality is necessary, or even helpful. >>>> >>>> Gwen >>>> >>>> On Mon, May 16, 2016 at 4:07 PM, Eric Wasserman >>>> <eric.wasser...@gmail.com> wrote: >>>>> I would like to begin discussion on KIP-58 >>>>> >>>>> The KIP is here: >>>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable >>>>> >>>>> Jira: https://issues.apache.org/jira/browse/KAFKA-1981 >>>>> >>>>> Pull Request: https://github.com/apache/kafka/pull/1168 >>>>> >>>>> Thanks, >>>>> >>>>> Eric >>>> >>
