Hi Nick, a bit late to the party, sorry. I recently spent some time looking into this / a similar issue [1]. After some investigation and playing around with settings I think that the benefit that could be gained from this is somewhat limited and probably outweighed by the implementation effort.
The consumer internal are already geared towards treating partitions fairly so that no partition has to wait an undue amount of time and this can be further tuned for latency over throughput. Additionally, if this is a large issue for someone, there is always the option of having a dedicated consumer reading only from the control topic, which would mean that messages from that topic are received "immediately". For a Kafka Streams job it would probably make sense to create two input streams and then merging those as a first step. I think with these knobs a fairly large amount of flexibility can be achieved so that there is no urgent need to implement priorities. So my personal preference would be to set this KIP to dormant for now. Best regards, Sönke [1] https://lists.apache.org/thread.html/07b5a9f1232ea139e01c477481a9d77208d8e8b2e55d8608a0271417@%3Cdev.kafka.apache.org%3E On Fri, Jan 25, 2019 at 1:21 AM <n...@afshartous.com> wrote: > > > Hi Colin, > > > On Jan 24, 2019, at 12:14 PM, Colin McCabe <cmcc...@apache.org> wrote: > > > > Users almost always like the idea of new features, whatever they are. But > > that doesn't mean that the feature would necessarily work well or be > > necessary. > > Yes, though we should certainly consider the responses on the user list as > input (Subject: Prioritized Topics for Kafka). > > > If you still want to pursue this, then I suggest gathering a set of > > use-cases that can't be addressed through the means we discussed here > > previously. So, something that can't effectively be addressed through > > using the pause and resume API. > > We’ve discussed this point before. I accept you point that a user could > implement this behavior with pause and resume. This KIP is about creating a > higher-level API to make it easier to do so. > > > Then come up with a concrete proposal that addresses all the questions we > > have, including about starvation, incremental fetch requests, and so on. > > To me it seems like there’s only one outstanding issue here (incremental > fetch), and we could just pick one of the options. Starvation is by design. > I’m not sure what “and so on” references. > > > This could be a lot of work. If you're looking for a way to make more > > contributions, I'd recommend getting started with something easier. > > > Yes it does. And after 6 months of (sometimes circular) discussion I’d like > to either move towards a vote or set the status of this KIP to dormant until > if and when someone else picks up it up. > > Does anybody else have input on either having a vote or setting the KIP > dormant ? > > Cheers, > -- > Nick > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
