> otherwise the write will timeout internally and it will get replayed to a new bookie. If Qa is met and the writes of Qw-Qa fail after we send the success to the client, why would the write replayed on a new bookie?
On Fri, Jan 8, 2021 at 1:47 PM Matteo Merli <mme...@apache.org> wrote: > On Fri, Jan 8, 2021 at 8:27 AM Enrico Olivelli <eolive...@gmail.com> > wrote: > > > > Hi Matteo, > > in this comment you are talking about an issue you saw when WQ is > greater that AQ > > https://github.com/apache/bookkeeper/issues/2497#issuecomment-734423246 > > > > IIUC you are saying that if one bookie is slow the client continues to > accumulate references to the entries that still have not received the > confirmation from it. > > I think that this is correct. > > > > Have you seen problems in production related to this scenario ? > > Can you tell more about them ? > > Yes, for simplicity, assume e=3, w=3, a=2. > > If one bookie is slow (not down, just slow), the BK client will the > acks to the user that the entries are written after the first 2 acks. > In the meantime, it will keep waiting for the 3rd bookie to respond. > If the bookie responds within the timeout, the entries can now be > dropped from memory, otherwise the write will timeout internally and > it will get replayed to a new bookie. > > In both cases, the amount of memory used in the client will max at > "throughput" * "timeout". This can be a large amount of memory and > easily cause OOM errors. > > Part of the problem is that it cannot be solved from outside the BK > client, since there's no visibility on what entries have 2 or 3 acks > and therefore it's not possible to apply backpressure. Instead, > there should be a backpressure mechanism in the BK client itself to > prevent this kind of issue. > One possibility there could be to use the same approach as described > in > https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits > , > giving a max memory limit per BK client instance and throttling > everything after the quota is reached. > > > Matteo > -- Jvrao --- First they ignore you, then they laugh at you, then they fight you, then you win. - Mahatma Gandhi
