[ https://issues.apache.org/jira/browse/KAFKA-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710078#comment-17710078 ]
Sagar Rao commented on KAFKA-14873: ----------------------------------- Hi [~malthe] , thanks for reporting this. I think your point makes sense in general terms. That's why, if you see all the3 backing stores are implemented as interfaces for example => `OffsetBackingStore`, `StatusBackingStore` and `ConfigBackingStore`. The implementations that we see for Kafka are just implementors of these interfaces. Typically, there is a `MemoryBased` equivalent to these backing stores which IIUC is used in standalone mode. Do you have any KV based backing stores in mind that you would want to see being added? I see Azure Table Storage has been cited as an example in OP. If you think something like this is needed, then it would need to go through a KIP process since a new public interface is being added. Let me know if you want to take it further via a KIP. > Pluggable storage for Kafka Connect internal topics > --------------------------------------------------- > > Key: KAFKA-14873 > URL: https://issues.apache.org/jira/browse/KAFKA-14873 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect > Reporter: Malthe Borch > Priority: Major > Labels: needs-kip > > The Kafka Connect framework relies on compacted topics to store config, > offset and status information for each connector. > This conflates two kinds of data, control and content, which some people > disagree with. Notably, [Azure Event > Hub|https://learn.microsoft.com/en-us/azure/event-hubs/log-compaction] does > not (or _did not_, because there's currently a preview release out which does > have support for compacted topics albeit only at the more expensive premium > tiers). > In some deployments, it may be desirable to use a different backend for these > control settings (which essentially take a key/value form), for example > [Azure Table > Storage|https://learn.microsoft.com/en-us/rest/api/storageservices/table-service-rest-api] > – basically any key/value store that provides the Write-If-Matches primitive > to update a key only if the current value matches a known value. -- This message was sent by Atlassian Jira (v8.20.10#820010)