On Mon, Oct 8, 2018, at 12:35, Thomas Becker wrote: > Well my (perhaps flawed) understanding of topic priorities is that lower > priority topics are not consumed as long as higher priority ones have > unconsumed messages (which means our position < HW). So if I'm doing > this manually, I have to make some determination as to whether my high > priority topic partitions are at the HW before I can decide if I want to > poll the lower priority ones. Right?
Hi Thomas, You could periodically check the last committed position of various partitions using KafkaConsumer#committed. But this would be very inefficient. For one thing, you'd have to keep waking up your consumer thread all the time to do this. The two-consumer solution that I suggested earlier just implies that you have two consumers, one for the control data and one for the non-control data. In that case, as long as control data is available, your consumer will always try to read it. It doesn't involve the caller checking committed position using KafkaConsumer#committed at any point. Usually, consumers are reading data that is relatively recent. If the consumer is too slow to keep up with the incoming messages over the long term, the system usually gets into a bad state. I think this is one reason why it's hard to think of use-cases for this feature. If you had a control partition and data partition, the data partition wouldn't really block you from getting the control messages in a timely fashion. You almost certainly need to be able to keep up with both partitions anyway. Also, if you have to do some very expensive processing on data messages, you should be offloading that processing to another thread, rather than doing the expensive thing in your consumer thread. And you can mute a partition while you're processing an expensive message from that partition, so it doesn't really block the processing of other partitions anyway. Maybe there's some really cool use-case that I haven't thought of. But so far I can't really think of any time I would need topic priorities if I was muting topics and offloading blocking operations in a reasonable way. It would be good to identify use-cases because it would motivate choices like how many priorities do we want (2? 256? 4 billion?) and what the API would be like, etc. best, Colin > > On Fri, 2018-10-05 at 11:34 -0700, Colin McCabe wrote: > > On Fri, Oct 5, 2018, at 10:58, Thomas Becker wrote: > > Colin, > > Would you mind sharing your vision for how this looks with multiple > > consumers? I'm still getting my bearings with the new consumer but it's > > not immediately obvious to me how this would work. > > > Hi Thomas, > > > I was just responding to the general idea that you would have some kind > of control topic that you wanted to read with very low latency, and some > kind of set of data topics where the latency requirements are less > strict. In that case, you can just have two consumers: one for the low- > latency topic, and one for the less low-latency topics. > > > There's a lot of things in this picture that are unclear. Does the data > in one set of topics have any relation to the data in the other? Why do > we want a control channel distinct from the data channel? That's why I > asked for clarification on the use-case. > > > In particular, it doesn't seem particularly easy to know when you are at > the high > > watermark of a topic. > > > KafkaConsumer#committed will return the last committed offset for a > partition. However, I'm not sure I understand why you want this > information in this case-- can you expand a bit on this? > > > best, > > Colin > > > > > -Tommy > > > On Mon, 2018-10-01 at 13:43 -0700, Colin McCabe wrote: > > > Hi all, > > > > I feel like the DISCUSS thread didn't really come to a conclusion, so a > > vote would be premature here. > > > > In particular, I still don't really understand the use-case for this > > feature. Can someone give a concrete scenario where you would need > > this? The control plane / data plane example that is listed in the KIP > > doesn't require this feature. You can just have one consumer for the > > control plane, and one for the data plane, and do priority that way. > > The discussion feels kind of unfocused since we haven't identified even > > one concrete use-case that needs this feature. > > > > Unfortunately, this is a feature which consumes server-side memory. We > > have to store the priorities somehow when doing incremental fetch > > requests. If we go with an int as suggested, then this is at least 4 > > bytes per partition per incremental fetch request. It also makes it > > more complex and potentially slower to maintain the linked list of > > partitions in the fetch requests. Before we think about this, I'd like > > to have a concrete use-case in mind, so that we can evaluate the costs > > versus benefits. > > > > best, > > > Colin > > > > > On Mon, Oct 1, 2018, at 07:47, Dongjin Lee wrote: > > > Great. +1 (non-binding) > > > > On Mon, Oct 1, 2018 at 4:23 AM Matthias J. Sax > > <matth...@confluent.io<mailto:matth...@confluent.io><mailto:matth...@confluent.io<mailto:matth...@confluent.io>>> > > > wrote: > > > > +1 (binding) > > > > As Dongjin pointed out, the community is working on upcoming 2.1 > > > release, and thus it might take some time until people find time to > > > follow up on this an vote. > > > > > -Matthias > > > > On 9/30/18 11:11 AM, > n...@afshartous.com<mailto:n...@afshartous.com><mailto:n...@afshartous.com<mailto:n...@afshartous.com>> > > wrote: > > > > On Sep 30, 2018, at 5:16 AM, Dongjin Lee > > <dong...@apache.org<mailto:dong...@apache.org><mailto:dong...@apache.org<mailto:dong...@apache.org>>> > > wrote: > > > > 1. Your KIP document > > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics > > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics > > > > lacks hyperlink to the discussion thread. And I couldn`t find the > > > discussion thread from the mailing archive. > > > > > Hi Dongjin, > > > > There has been a discussion thread. I added this link as a reference > > > > https://lists.apache.org/list.html?dev@kafka.apache.org:lte=1M:kip-349 > > > <https://lists.apache.org/list.html?dev@kafka.apache.org:lte=1M:kip-349> > > > > to the KIP-349 page > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics > > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-349:+Priorities+for+Source+Topics > > > > > Best, > > > -- > > > Nick > > > > > > > > -- > > > *Dongjin Lee* > > > > *A hitchhiker in the mathematical world.* > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > > www.slideshare.net/ > dongjinleekr<http://www.slideshare.net/dongjinleekr<http://www.slideshare.net/dongjinleekr<http://www.slideshare.net/dongjinleekr>> > > > <http://www.slideshare.net/dongjinleekr>* > > > ________________________________ > > > This email and any attachments may contain confidential and privileged > > material for the sole use of the intended recipient. Any review, > > copying, or distribution of this email (or any attachments) by others is > > prohibited. If you are not the intended recipient, please contact the > > sender immediately and permanently delete this email and any > > attachments. No employee or agent of TiVo Inc. is authorized to conclude > > any binding agreement on behalf of TiVo Inc. by email. Binding > > agreements with TiVo Inc. may only be made by a signed written > > agreement. > > ________________________________ > > This email and any attachments may contain confidential and privileged > material for the sole use of the intended recipient. Any review, > copying, or distribution of this email (or any attachments) by others is > prohibited. If you are not the intended recipient, please contact the > sender immediately and permanently delete this email and any > attachments. No employee or agent of TiVo Inc. is authorized to conclude > any binding agreement on behalf of TiVo Inc. by email. Binding > agreements with TiVo Inc. may only be made by a signed written > agreement.
