Hey Jan, although I believe your case is much more complicated, but would time based retention work for you at all? If yes, time window store is like the best option.
If no, streams has no out-of-box solution for invalidating the aggregation record. It seems at least we could provide an API to inject tombstone records for aggregation logic so that they don't get ignored eventually. This sounds like a good future work. Boyang On Thu, Jan 2, 2020 at 1:47 PM Jan Bols <janb...@telenet.be> wrote: > Hi, > I have a rather complicated kafka streams application involving multiple > joins, aggregates, maps etc. At a certain point, parts of the data needs to > be removed throughout the entire streams topology, both in the topics, the > changelogs and the rocksdb state stores. > > Managing this requires a lot of effort and things get very complex. F.e. > when a KStream has a null value and is aggregated, you first need to > convert it into some optional value instead b/c aggregates ignore nulls. > > Is there a better way or a way that does not impact all the existing > streaming logic? > > I was thinking about having an out-of-bound process that sends null values > to all topics with the correct keys. I could then filter out all null > values before doing the rest of the existing stream logic. > Would that make sense? > > I can send null values to all my topics, but how do I get the changelog > topics created by kafka-streams. And what about the state store? > > Best regards > Jan >
