Hi Kamal, Thanks for your reply. We have now published the consumer-side KIP-1254 [1] which addresses these topics in detail. To summarize:
1. The existing consumer transaction filtering logic (CompletedFetch.isBatchAborted()) already handles read_committed isolation. Aborted transaction records are filtered out and not returned to the application. For remote fetches, the broker populates the existing abortedTransactions field in FetchResponse from its cached TransactionIndex, and the consumer reuses the same filtering logic. No changes to transaction semantics are needed. See the Transactions section [2] for details. 2. This depends on the RemoteStorageFetcher plugin implementation, but we provide guidance in the Authorization section [3]. Our preferred approach is pre-signed URLs: the broker generates a temporary URL granting access for that specific request, passed via CustomMetadata in FetchResponse. This ensures broker ACL changes are instantly reflected for consumers, and consumers need no long-term credentials. 3. Not supported in this KIP - planned for a subsequent KIP. Currently, the broker's SharePartition determines which records go to which consumer. To support direct remote fetch, we'd adapt this to return offset range metadata + segment location instead of actual records, allowing consumers to fetch directly from remote storage. See the Share Groups section [4]. Would you prefer we have the Share Groups sub-KIP fully drafted before moving forward with this KIP? Happy to write it up if that would be helpful. Thanks, Tom [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-1254%3A+Kafka+Consumer+Support+for+Remote+Tiered+Storage+Fetch [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=399279678#KIP1254:KafkaConsumerSupportforRemoteTieredStorageFetch-Transactions [3] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=399279678#KIP1254:KafkaConsumerSupportforRemoteTieredStorageFetch-Authorization [4] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=399279678#KIP1254:KafkaConsumerSupportforRemoteTieredStorageFetch-ShareGroups(Queues) On Tue, Dec 2, 2025 at 8:05 AM Kamal Chandraprakash < [email protected]> wrote: > Hi Haiying, > > Thanks for the KIP! > > 1. Do you plan to add support for transactional consumers? Currently, the > consumer doesn't return the aborted transaction records to the handler. > 2. To access the remote storage directly, the client might need additional > certificates / keys. How do you plan to expose those configs on the client? > 3. Will it support the Queues for Kafka feature KIP-932 > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka > >? > And so on. > > -- > Kamal > > On Tue, Dec 2, 2025 at 10:29 AM Haiying Cai via dev <[email protected]> > wrote: > > > For some reason, the KIP link was truncated in the original email. Here > > is the link again: > > > > KIP: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1248%3A+Allow+consumer+to+fetch+from+remote+tiered+storage > > > > Henry Haiying Cai > > > > On 2025/12/02 04:34:39 Henry Haiying Cai via dev wrote: > > > > > > Hi all, > > > > > > > > > > > > > > > I would like to start discussion on KIP-1248: Allow consumer to fetch > > from remote tiered storage > > > > > > > > > > > > KIP link: KIP-1248: Allow consumer to fetch from remote tiered storage > - > > Apache Kafka - Apache Software Foundation > > > > > > | > > > | > > > | | > > > KIP-1248: Allow consumer to fetch from remote tiered storage - Apache > > Ka... > > > > > > > > > | > > > > > > | > > > > > > | > > > > > > > > > > > > > > > The KIP proposes to allow consumer clients to fetch from remote tiered > > storage directly to avoid hitting broker's network capacity and cache > > performance. This is very useful to serve large backfill requests from a > > new or fallen-off consumer. > > > > > > > > > > > > > > > Any feedback is appreciated. > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Henry >
