[
https://issues.apache.org/jira/browse/KAFKA-16956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Chen updated KAFKA-16956:
------------------------------
Labels: need-kip (was: )
> Broker-side ability to subscribe to record delete events
> --------------------------------------------------------
>
> Key: KAFKA-16956
> URL: https://issues.apache.org/jira/browse/KAFKA-16956
> Project: Kafka
> Issue Type: Improvement
> Reporter: Luke Chen
> Priority: Major
> Labels: need-kip
>
> In some cases it would be useful for systems outside Kafka to have the
> ability to know when Kafka deletes records (tombstoning or retention).
> In general the use-case is where there is a desire to link the lifecycle of a
> record in a third party system (database or filesystem etc) to the lifecycle
> of the record in Kafka.
> A concrete use-case: a system using Kafka to distribute video clips +
> metadata. The binary content is too big to store in Kafka so the publishing
> application caches the content in cloud storage and publishes a record
> containing a S3 url to the video clip. The desire is to have a mechanism to
> remove the clip from cloud storage at the same time the record is expunged
> from Kafka by retention or tombstoning. Currently there is no practical way
> to achieve this.
> h2. Desired solution
> A pluggable broker-side mechanism that is informed as records are being
> compacted away or deleted. The API would expose the topic from which the
> record is being deleted, the record key, record headers, timestamp and
> (possibly) record value.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
