On Thu, Oct 25, 2018, at 18:16, n...@afshartous.com wrote: > > The reporter of KAFKA-6690 (Bala) replied in the JIra ticket to my > question to elaborate about his use-case. I don’t think he’s on the dev > list. Here’s his response: > > Bala: Sorry about the delay in reply. We use Kafka to process the > asynchronous events of our Document Management System such as preview > generation, indexing for search etc. The traffic gets generated via Web > and Desktop Sync application. In such cases, we had to prioritize the > traffic from web and consume them first. But this might lead to the > starvation of events from sync if the consumer speed is slow and the > event rate is high from web. A solution to handle the starvation with a > timeout after which the events are consumed normally for a specified > period of time would be great and help us use our resources effectively.
Priorities won't help for this use-case, right? If the "web" partition has a higher priority, and data is always available, there will *never* be any events reported for "sync". Priorities don't prevent starvation-- they cause starvation by design, because the high priority partition always takes priority. In general the best solution would probably be to have a work queue between the consumer and the event handler, and manage the backpressure as appropriate. This could be done with pause and resume, as Streams does. best, Colin > > -- > Nick > > > > > > On Oct 18, 2018, at 12:23 PM, n...@afshartous.com wrote: > > > >> On Oct 12, 2018, at 5:06 PM, Colin McCabe <cmcc...@apache.org> wrote: > >> > >> Maybe there's some really cool use-case that I haven't thought of. But so > >> far I can't really think of any time I would need topic priorities if I > >> was muting topics and offloading blocking operations in a reasonable way. > >> It would be good to identify use-cases > > > > > > Hi Colin, > > > > How about the use-case where there are multiple streams/topics, and the > > intent is to have a single consumer interleave the messages so that higher > > priority messages are processed first ? > > That seems to be what the reporter of the associated Jira ticket > > > > https://issues.apache.org/jira/browse/KAFKA-6690 > > <https://issues.apache.org/jira/browse/KAFKA-6690> > > > > has identified as a use-case he frequently encounters. I’ve asked him to > > elaborate on the dev list though he has not responded yet. > > > > Best, > > -- > > Nick > > > > > > > > > > >
