Hey Andrew! Kudos on the proposal. It is greatly written - a joy to read. It is definitely an interesting solution to the queueing problem - I would not have guessed we could solve it like this. Thank you for working on this.
Happy to get the discussion started - I have a few comments/questions on first read: 1. Tiered Storage I notice no mention of Tiered Storage (KIP-405). Does that complicate the design, especially when fetching historical data? It would be good to have at least one sentence mentioning it, even if it doesn't impact it. Right now I'm unsure if it was considered. 2. SSO initialized to the latest offset > "By default, the SSO for each share-partition is initialized to the latest offset for the corresponding topic-partitions." Have we considered allowing this to be configurable to latest/earliest? This would be consistent with the auto.offset.reset config of vanilla consumer groups. Thinking from a user's perspective, it sounds valid to want to start from the start of a topic when starting a share group. Historical processing comes to mind 3. Durable Storage The KIP mentions that "The cluster records this information durably", which implies that it saves it somewhere. Does the ShareCoordinator have its own topic? Would it be compacted? In particular, I am interested in what such a topic's retention would be like. The vanilla consumer offsets topic has some special retention semantics (KIP-211) where we start counting the retention after the consumer group becomes empty (inactive) - the default being 7 days. Need to make sure the retention here isn't too short either, as the offsets topic originally had 24 hours of retention and that proved problematic. In general, some extra detail about the persistence would be greatly appreciated! 4. Batch Acknowledgement > "In the situation where some records in a batch have been released or rejected separately, subsequent fetches of those records are more likely to have gaps." Can we expand a bit more on this edge case? I am interested in learning what gets returned on subsequent fetch requests. In particular, - how does this work with compression? As far as I remember, we can compress the whole batch there, which might make individual record filtering tricky. 5. Member Management How is consumer group member management handled? I didn't see any specific mention - is it the same as a vanilla group? In particular - how will bad consumers be handled? I guess I see two cases: 1. bad consumer that doesn't even heartbeat 2. bad consumer that heartbeats well but for some reason every message processing times out. e.g imagine it was network partitioned from some third-party system that is a critical part of its message processing loop One evident problem I can foresee in production systems is one (or a few) slow consumer applications bringing the SSO/SEO advancement down to a crawl. Imagine an example where the same consumer app always hits the timeout limit - what would the behavior be in such a case? Do we keep that consumer app indefinitely (if so, do we run the risk of having it invalidate completely valid messages)? Are there any equivalents to the consumer group rebalances which fence off such bad consumers? 6. Processing Semantics (exactly once) > The delivery counts are only maintained approximately and the Acquired state is not persisted. Does this introduce the risk of zombie consumers on share-partition-leader failure? i.e restarting and giving another consumer the acquired state for the same record I notice that the KIP says: > Finally, this KIP does not include support for acknowledging delivery using transactions for exactly-once semantics. at the very end. It would be helpful to address this earlier in the example, as one of the key points. And it would be good to be clearer on what the processing semantics are. They seem to be *at-least-once* to me. 7. nit: Acronyms I feel like SSO and SEO may be bad acronyms. Googling "Kafka SEO" is bound to return weird results. What do we think about the tradeoff of using more-unique acronyms (like SGEO, SSGO) at the expense of one extra letter? Again - thanks for working on this! I think it's a great initiative. I'm excited to see us perfect this proposal and enable a brand new use case in Kafka! Best, Stanislav On Mon, May 15, 2023 at 2:55 PM Andrew Schofield <andrew_schofi...@live.com> wrote: > Hi, > I would like to start a discussion thread on KIP-932: Queues for Kafka. > This KIP proposes an alternative to consumer groups to enable cooperative > consumption by consumers without partition assignment. You end up with > queue semantics on top of regular Kafka topics, with per-message > acknowledgement and automatic handling of messages which repeatedly fail to > be processed. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka > > Please take a look and let me know what you think. > > Thanks. > Andrew -- Best, Stanislav
