My only point was, if the broker tries to manage more than 10M unack message ranges (per subscription) then the broker has to face many negative consequences in terms of memory and CPU, and then it will require to build additional debt to solve such problems. Therefore, I consider 10M unack message ranges are more than enough for any application, if not then we should prevent those applications to not go beyond such thresholds to handle the back-pressure of unack messages and not allowing applications to build a cache at broker-side.
We have seen multiple issues in the past due to large numbers of unack messages in memory and to handle such abuse, brokers have configurations to allow max unack messages per consumer (maxUnackedMessagesPerConsumer: default 50K) and subscription (maxUnackedMessagesPerSubscription: default 200K). So, brokers already have large enough limits to store unack messages and going beyond that limit causes scaling issues in brokers, so better to keep the system stable and safe. Thanks, Rajan On Thu, Jan 28, 2021 at 9:50 AM Sijie Guo <guosi...@gmail.com> wrote: > Agreed with Lin. I think we should try to abstract this into an interface > and allow different implementations. > > Rajan - what is your real concern making it abstract? > > - Sijie > > On Wed, Jan 27, 2021 at 7:37 PM Lin Lin <lin...@apache.org> wrote: > > > Hi Rajan, > > Thank you for your PR. > > The main difference lies in whether 10MB is enough and memory doubling > > problem, which is caused by different business scenarios. > > In some business scenario, the QPS of 20k/s is considered to be very low, > > and requests exceeding this order of magnitude are common. > > If it is only increased to 10MB, the time exceeding the threshold only > > changes from 30 seconds to 60 seconds, and the problems in PIP are still > > not solved. > > "large enough" may be base on your scenario, and in some scenario, it is > > not enough in most cases... > > Because the problem has not been solved, I suggest to abstract, so that > > different people can choose. > > Your PR is an improvement to the current performance, there is no > conflict > > between them. > > > > Thanks > > > > On 2021/01/27 03:50:07, Rajan Dhabalia <rdhaba...@apache.org> wrote: > > > I have created a PR which should allow brokers to store up to 10M > > > unack-message ranges. I think it should be large enough for any > usecases > > > and probably now, we might not need to introduce abstraction for ack > > > management to avoid any further complexity in message acknowledgement > > path > > > as well. > > > https://github.com/apache/pulsar/pull/9292 > > > > > > Thanks, > > > Rajan > > > > >
