[
https://issues.apache.org/jira/browse/KAFKA-21009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18112809#comment-18112809
]
sanghyeok An commented on KAFKA-21009:
--------------------------------------
[~mjsax]
Sorry for the delayed response.
Thanks for pointing this out. After reviewing KAFKA-7718, I agree that this
ticket substantially overlaps with it and is likely a duplicate.
Also, to clarify what I meant by a “policy”: I was referring to the behavior an
application may want for result headers, such as dropping, preserving, or
merging them. I did not necessarily intend to propose a fixed enum or a global
configuration.
I agree that a user-provided callback, similar to Aggregator or ValueJoiner,
would be a more direct and flexible API mechanism for implementing such
behavior.
Common behaviors such as drop, preserve, or merge could potentially be provided
as callback utilities, but the callback itself would be the primary extension
point.
For example, we can provide default method based on callback.
* HeaderJoiners.drop()
* HeaderJoiners.preserveLeft()
* HeaderJoiners.preserveRight()
* HeaderJoiners.preserveTrigger()
* HeaderJoiners.merge()
KIP-1271 and KIP-1285 have provided the state-store foundation for preserving
headers.
Are you planning to work on a KIP for the callback-based approach?
If not, I would be interested in writing the KIP and leading the discussion.
> Consider introducing record header handling policies for Kafka Streams
> stateful operations
> ------------------------------------------------------------------------------------------
>
> Key: KAFKA-21009
> URL: https://issues.apache.org/jira/browse/KAFKA-21009
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
> Labels: needs-kip
>
> Stateful operations such as joins and aggregations may combine or retain
> records from multiple inputs. Unlike one-to-one stateless operations, there
> may not be an obvious set of headers to use for the output record.
>
> Currently, header handling is determined by the behavior of each operator,
> and applications do not have an explicit policy for controlling it.
>
> We should consider whether Kafka Streams stateful operations could support
> header handling policies such as:
> * Retaining the existing behavior
> * Ignoring or dropping headers
> * Preserving headers from a selected input record
> * Merging headers from multiple input records
> This Jira is intended to capture the use case and evaluate whether such
> policies would be useful. If pursued, the affected operators, exact policy
> set, defaults, conflict handling, and consistency across caching and
> restoration should be defined through a KIP.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
