Hi Colt, it's a good point that we do not need to separate RocksDBs for keeping the timestamp index.
I'd support implementing a TTL in Kafka Streams this way if we know that the compaction filter idea won't fly. Cheers, Lucas On Sun, Oct 12, 2025 at 12:53 AM Colt McNealy <[email protected]> wrote: > > Hi all—just chiming in here since we built something like this inside > LittleHorse. > > Another alternative option to implement TTL which does not require scanning > through all stores is to "hijack" the `.put()` implementation to insert two > keys: > > 1. nativeStore.put("values/" + key, value); > 2. nativeStore.put("timestamps/" + timestamp, key); > > Putting the `timestamp` can be as simple as just printing out the > leading-zero-padded epoch time in milliiseconds. > > Every punctuation interval (eg. 10s, 1m, whatever you want—should be > configurable—), you simply do a range scan from ["timestamp/0", > "timestamp/" + System.currentTimeMillis() or record.timestamp() ) and > delete each key that is returned. That avoids scanning the full store, and > in fact bounds the rate of things processed in the punctuation to roughly > the rate at which you are processing them from the input topics. > > We do something like this in LittleHorse, actually, and it works quite well > (note that we have a separate rocksdb for the timestamps but that's a > really long story which is irrelevant to this discussion). > > As for the actual store API, I'd recommend making the constructor / builder > of the RocksDBTTLStore contain a configurable punctuation interval. > > May the Force be with you! > > Colt McNealy > Founder and Managing Member > https://littlehorse.io > > On 2025/10/07 05:25:32 Ankur Sinha wrote: > > Hello, > > > > I’d like to start the discussion 🙂 for KIP-1225: Add Optional TTL > Support to Kafka Streams State Stores< > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1225%3A+Add+Optional+TTL+Support+to+Kafka+Streams+State+Stores > > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-19759< > https://issues.apache.org/jira/browse/KAFKA-19759> > > KIP Wiki: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1225%3A+Add+Optional+TTL+Support+to+Kafka+Streams+State+Stores > > > > Regards, > > Ankur Sinha > > > > ________________________________ > > From: Matthias J. Sax <[email protected]> > > Sent: Tuesday, October 7, 2025 1:04 AM > > To: [email protected] <[email protected]> > > Subject: [EXT] Re: [DISCUSS] KIP-19759: Add built-in TTL (Time-to-Live) > support for Kafka Streams State Stores > > > > Warning External sender Do not click on any links or open any attachments > unless you trust the sender and know the content is safe. > > > > If you want to start a KIP, you need to write one :) A Jira ticket is > > not a KIP > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > < > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > > -Matthias > > > > This email was screened for spam and malicious content but exercise > caution anyway. > > > > > > > > On 10/6/25 10:26 AM, Ankur Sinha wrote: > > > Hi all, > > > > > > I’d like to start the discussion for KAFKA-19759: Add built-in TTL > (Time-to-Live) support for Kafka Streams State Stores. > > > > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-19759< > https://issues.apache.org/jira/browse/KAFKA-19759> > > > KIP Wiki: (will be created shortly) > > > > > > Streams users often need per-key TTL functionality (e.g., for > cache-like state, per-key deduplication, or automatic cleanup). > > > Currently, this requires manual punctuators and tombstones. > > > > > > This KIP proposes adding a built-in `withTtl(Duration ttl)` option to > state stores, > > > which would handle key expiration and changelog consistency > automatically. > > > > > > Please see the JIRA for full details and general analysis. > > > Looking forward for a discussion and talk. > > > > > > Best regards, > > > Ankur Sinha > > > > >
