Hi everyone, another question for the list :) I'm creating a cluster of KTable (and GlobalKTable) based off the same input stream K,V.
It has a number of secondary indices (think like a RDBMS)
K1 -> K
K2 -> K
etc
These are all based off of trivial mappings from my main stream that also
feeds the K -> V StateStore. Think one liners like v -> v.getSecondaryKey1()
Currently, for each one it seems that I have to do
builder.stream(mainTopic)
.mapValues(...)
.to(secondaryIndex1)
builder.globalTable(secondaryIndex1, secondaryIndexStore1);
Unfortunately the intermediate "secondaryIndex1" topic is relatively
low value. In a case where my state stores are lost, I already have to
read through the mainTopic to recover the main state store. While it's doing
that, I'd much rather it rebuild the GlobalKTable instance from that data
directly. Then I could skip having this index in Kafka at all, it's entirely
redundant. The data is already loaded and deserialized for the benefit of
another Processor.
Any thoughts? Happy Friday,
Steven
signature.asc
Description: Message signed with OpenPGP using GPGMail
