Greg/Stan, KIP-1150 is aiming for both diskless and leaderless. I am not sure why it is a good idea trying to solve both in one KIP. This significantly increase the scope of this project with the complexity of solving many related issues (transaction, queue support, data locality) in the same project.
I think with KIP-392 (Fetch from follower) and KIP-1123 (Rack aware partitioning for Kafka Producer), both producer and consumer can read/write to a broker other than the original leader broker. The leader broker is no longer the blocker component in the read/write paths. With the KIP-405 tiered storage, the leader state is also much smaller than before. By removing the leader broker in the revised KIP-1150 design, you would need to move many logic (e.g. offset assignment, transaction control) which was originally in the leader to the batch coordinator, this adds to the latency and complicated the logic (those logic are easier to implement in a single leader broker). In the current Kafka design, the control is distributed in many leader brokers with no hotspots for the whole cluster. But KIP-1150 is moving the distributed control into a central component: batch coordinator which adds a hotspot/bottleneck for the cluster; And this batch coordinator needs to be involved in transaction control as well which adds more bottleneck/latency in a critical control flow. On 2025/08/06 02:13:32 Xinyu Zhou wrote: > Hi Luke, > > Thank you for creating this dedicated thread; we definitely need a space to > discuss future steps for these topics. I apologize for my delay on KIP-1183 > and will provide more details in the coming weeks. > > I agree with Stanislav that we should first focus on the community's > direction. Specifically, should we consider introducing a leaderless > architecture to Kafka, given that it currently relies on a partitioned, > leader-based model? > > From my own perspective, I’m particularly interested in how Leaderless and > Leader-based architectures differ when it comes to handling data > locality—which directly affects batching and fetch efficiency—and in the > way core features are implemented. For instance, ordering, compaction, > transactions, idempotent producers, and queues all have to be realized on > the Coordinator in a Leaderless design, whereas in a Leader-based design > they are handled by the Leader Partition. > > If the community plans to adopt a leaderless architecture, will the focus > be on a complete transition to leaderless, or will both architectures > coexist in the long term? > > I welcome discussions on this topic and am eager to hear diverse opinions. > > Regards, > Xinyu > > On Wed, Aug 6, 2025 at 3:05 AM Stanislav Kozlovski < > stanislavkozlov...@apache.org> wrote: > > > Thank you Luke for this wonderful summary and taking initiative. > > > > To me, it seems like a large differentiator from KIP-1150 and others is > > the leaderless design. The other two don’t allow for it. > > > > It sounds productive to focus the discussion on whether the leaderless > > design is worth it on top of the replication cost savings. > > > > I’m of the opinion that it’s worth pursuing - both for the truly zero > > network cost (no producer cross az) but perhaps even more importantly the > > zero state architecture that promises to significantly simplify operations, > > including auto scaling brokers and scaling throughput per partition > > > > It would be great if the folks at Aiven could address the concerns > > regarding queue and transactions support. I’m not of the opinion that these > > things need to ship with v1, but it would be wise to ensure nothing in the > > architecture blocks these features from being shipped in the future > > > > KIP-1176 is also very cool, addressing the acks=1 case will still be > > necessary. I think it’s a necessary feature to implement, but I’d be > > disappointed if that’s the only diskless solution the community agrees on. > > > > A good path, if possible, may be to merge KIP-1150 and KIP-1176. > > > > If instead the community decides leaderless isn’t necessary, then KIP-1183 > > seems fit. > > > > That’s my opinion. Happy to hear if anyone disagrees. > > > > On 2025/08/05 14:30:45 Josep Prat wrote: > > > Hi Luke and community! > > > > > > Luke, thank you for being proactive and caring about this topic! > > > > > > In the meantime we have been keeping ourselves busy pushing our > > > implementation of KIP-1150 to production to validate our assumptions and > > > confirm its strengths while discovering its weaknesses. > > > Now, after gathering some experience running it, we are (as I'm writing > > > this, gathered in the same room) working on an improved proposal for > > > KIP-1150 that also addresses the concerns from the community. > > > We expect to share the updated KIP in the next couple of weeks. > > > > > > We apologize for the recent period of silence and are committed to more > > > regular communication as we move forward. > > > > > > Best, > > > > > > > > > On Tue, Aug 5, 2025 at 10:31 AM Luke Chen <sh...@gmail.com> wrote: > > > > > > > Hi all, > > > > > > > > The Kafka community is currently seeing an unprecedented situation with > > > > three KIPs (KIP-1150, IP-1176, KIP-1183) simultaneously addressing the > > same > > > > challenge of high replication costs when running Kafka across multiple > > > > cloud availability zones. Each KIP offers a different solution to this > > > > issue. While diversity of innovative ideas is a key strength of > > open-source > > > > projects, it creates a burden for reviewers and users who must compare > > and > > > > comment on multiple proposals simultaneously. Furthermore, discussion > > > > around the three KIPs has stalled for over two months now. This could > > be > > > > due to the authors being hesitant to proceed due to the existence of > > > > alternative, potentially conflicting, solutions. Addressing replication > > > > cost is a key concern of Kafka’s userbase and we should try to move the > > > > conversation forward if we can. > > > > > > > > From what I understand, these three KIPs are not mutually exclusive. > > But > > > > adopting all three KIPs in the community might not be what we expect. > > Thus, > > > > I would like to *start a discussion on how we could move the > > conversation > > > > forward*. > > > > > > > > To save time for the KIP readers/reviewers, I have created this > > document > > > > < > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/The+Path+Forward+for+Saving+Cross-AZ+Replication+Costs+KIPs > > > > >[1] > > > > to help summarize each of the KIPs and describe their current status. > > *Hope > > > > to get some suggestions/feedback from the community*. > > > > > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/The+Path+Forward+for+Saving+Cross-AZ+Replication+Costs+KIPs > > > > > > > > KIP-1150: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics > > > > KIP-1176 > > > > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+TopicsKIP-1176 > > > > > > > : > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1176%3A+Tiered+Storage+for+Active+Log+Segment > > > > KIP-1183 > > > > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1176%3A+Tiered+Storage+for+Active+Log+SegmentKIP-1183 > > > > > > > : > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1183%3A+Unified+Shared+Storage > > > > > > > > > > > > Thank you. > > > > Luke > > > > > > > > > > > > > -- > > > [image: Aiven] <https://www.aiven.io> > > > > > > *Josep Prat* > > > Sr. Engineering Director, Streaming Services, *Aiven* > > > josep.p...@aiven.io | +491715557497 > > > aiven.io <https://www.aiven.io> | < > > https://www.facebook.com/aivencloud> > > > <https://www.linkedin.com/company/aiven/> < > > https://twitter.com/aiven_io> > > > *Aiven Deutschland GmbH* > > > Alexanderufer 3-7, 10117 Berlin > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > > > > > Kenneth Chen > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > >
