Hi, thank you for the reply!
I'm developing system where producers are spending money every time a request arrives. Consumers account money spent using the data from producers as well as few other sources. Consumers are also responsible to calculatate statistics that affect policies used by producers to spend money. As a result, if consumers are temporary lagging then producres are neither know if they can spend money, nor they know how to better do it. There is some lag values that producers can tolerate, but if lag is growing further then producers have to stop. Thanks, Filipp. On Tue, Feb 19, 2019 at 7:44 PM Javier Arias Losada <[email protected]> wrote: > > Hi, > > could you please be more specific on your use case? > One of the theoretical advantages of a system like kafka is that you can > decouple producers and consumers, so you don't need to to do backpressure. > A different topic is how to handle lagging consumers, in that scenario you > could scale up your service, etc. > > Best. > > El mar., 19 feb. 2019 a las 15:43, Filipp Zhinkin (<[email protected]>) > escribió: > > > Hi! > > > > I'm trying to implement backpressure mechanism that asks producers to > > stop doing any work when consumers are not able to process all > > messages in time (producers require statistics calculated by consumers > > in order to answer client requests, when consumers are lagging behind > > we have to stop producers from making any responses). > > > > I see several ways to implement it: > > - compute lag on consumer side and store it somewhere (zk, some db, etc); > > - use separate service like Burrow; > > - compute lag on every producer by getting commited and end offsets > > for every partition via consumer API. > > > > Are there any downsides of the latter approach? Would is negatively > > impact brokers performance? > > > > Thanks, > > Filipp. > >
