Frederic Tardif created KAFKA-7397:
--------------------------------------
Summary: Ability to apply DSL stateless transformation on a global
table
Key: KAFKA-7397
URL: https://issues.apache.org/jira/browse/KAFKA-7397
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Frederic Tardif
When consuming a globalKTable (with the expectation of caching all the data of
a topic in a consumer store), we can't apply any stateless transformation
(filter, map), prior to materializing. To achieve this, while ensure to consume
the records of all the partitions, we must first run a stream app that
pre-process the ingress topic into an exact K,V egress topic as we want to
store in our GlobalKTable. This looks unnecessarily complex, and causes to
double the storage of the topic, while the only goal is to adapt statelessly
the data prior to storing (rockDB) at the receiving end.
See discussion on
:https://stackoverflow.com/questions/50993292/kafka-streams-shared-changelog-topic
As a workaround, I have used `new Builder().addGlobalStore(....)` with a Custom
Processor able to filter and map prior to store (see attached). Although this
seem to work, I believe this functionality should be part of the basic dsl api
when working with a globalTable (`new
StreamsBuilder().globalTable().filter(...).map()... `).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
