Luke Chen created KAFKA-16956:
---------------------------------
Summary: Broker-side ability to subscribe to record delete events
Key: KAFKA-16956
URL: https://issues.apache.org/jira/browse/KAFKA-16956
Project: Kafka
Issue Type: Improvement
Reporter: Luke Chen
In some cases it would be useful for systems outside Kafka to have the ability
to know when Kafka deletes records (tombstoning or retention).
In general the use-case is where there is a desire to link the lifecycle of a
record in a third party system (database or filesystem etc) to the lifecycle of
the record in Kafka.
A concrete use-case: a system using Kafka to distribute video clips +
metadata. The binary content is too big to store in Kafka so the publishing
application caches the content in cloud storage and publishes a record
containing a S3 url to the video clip. The desire is to have a mechanism to
remove the clip from cloud storage at the same time the record is expunged from
Kafka by retention or tombstoning. Currently there is no practical way to
achieve this.
h2. Desired solution
A pluggable broker-side mechanism that is informed as records are being
compacted away or deleted. The API would expose the topic from which the
record is being deleted, the record key, record headers, timestamp and
(possibly) record value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
