Matteo, Rajan, Joe, Addison, Penghui, and Jia have met today to discuss
this issue. We have come up with next steps:
In general, we have agreed to proceed with the PIP.
1. Penghui is going to update the PIP to clarify terminology and diagrams
in the PIP. He will also write more details about how
@Rajan Dhabalia
We discussed this PIP in the community meeting and agreed that we need to
evaluate this PIP one more time to see if we can combine this PIP together
with the bitmap proposal together as the final solution.
We should think about more use cases before pushing the bitmap approach.
H
Rajan - I understand your concern about memory. Lin, Penghui, and I also
acknowledged that none of the implementations solve every use case. Each
implementation has its limitation and concerns.
I am trying to find a way for both parties (You and Lin) can explore
different implementations. An abstr
My only point was, if the broker tries to manage more than 10M unack
message ranges (per subscription) then the broker has to face many negative
consequences in terms of memory and CPU, and then it will require to build
additional debt to solve such problems. Therefore, I consider 10M unack
message
Agreed with Lin. I think we should try to abstract this into an interface
and allow different implementations.
Rajan - what is your real concern making it abstract?
- Sijie
On Wed, Jan 27, 2021 at 7:37 PM Lin Lin wrote:
> Hi Rajan,
> Thank you for your PR.
> The main difference lies in whether
Hi Rajan,
Thank you for your PR.
The main difference lies in whether 10MB is enough and memory doubling problem,
which is caused by different business scenarios.
In some business scenario, the QPS of 20k/s is considered to be very low, and
requests exceeding this order of magnitude are common.
I
Hi Rajan,
10M unack-message ranges can respond to most use-cases
but there are still some scenarios that cannot be dealt with.
We can make an abstraction for users who want to extend the implementation.
I think this will not conflict with your improvement.
This PIP not just to improve the persist
I have created a PR which should allow brokers to store up to 10M
unack-message ranges. I think it should be large enough for any usecases
and probably now, we might not need to introduce abstraction for ack
management to avoid any further complexity in message acknowledgement path
as well.
https:/
Thanks for all your suggestion, LinLin and I will work on abstract this
acknowledgment management as an
interface.
- Penghui
Sijie Guo 于2021年1月25日周一 上午11:14写道：
> Rajan,
>
> Thank you for sharing the prototype! That looks great to me.
>
> In order for the community to evolve and experiment with
Rajan,
Thank you for sharing the prototype! That looks great to me.
In order for the community to evolve and experiment with different
approaches, how about we abstract this acknowledgment management as an
interface?
If this approach works for you, maybe Lin Lin can focus on abstracting this
int
*Hi, >> Delayed messages or certain user logic can introduce a lot of
message-holes. We have seen this issue in quite a lot of customers'
production environment.I agree that we need a larger buffer to store and
recover individually deleted messages, and the existing 150K limit might
not be enough f
Joe - Delayed messages or certain user logic can introduce a lot of message
holes. We have seen this issue in quite a lot of customers' production
environment. Hence we need to find a solution for solving these problems.
If you are skeptical of an implementation like that, how about us making
curso
Let me take a step back and explain how I am looking at this from a
high-level
design viewpoint
Bookkeeper (BK) is like an LSM implementation of a KV store. Writes to all
keys are appended to a single file; deletes are logical. Compaction
reclaims space. An Index is used locate entries, tracki
We can look at ManagedCursorImpl.buildIndividualDeletedMessageRanges
What is saved in the entry is not a bitSet, but a messageRange one by one,
which contains information such as ledgerId and entryId. BitSet only exists
in the memory and is used to quickly determine whether it already exists.
In a
Thank you very much, I will add these points to the document
1
Now the persistence of ack is also asynchronous, not every ack will be
persisted immediately, and the lost part will be re-consumed.
When the entry is successfully written but the marker fails to be written,
we will consider the whole t
I have a simpler question. Just storing the message-ids raw will fit ~300K
entries in one ledger entry. With the bitmap changes, we can store a
couple of million within one 5MB ledger entry. So can you tell us what
numbers of unacked messages are creating a problem? What exactly are the
issues
Hi Lin,
Thanks you and Penghui for drafting this! We have seen a lot of pain points
of `managedLedgerMaxUnackedRangesToPersist` when enabling delayed messages.
Glad that you and Penghui are spending time on resolving this!
Overall the proposal looks good. But I have a couple of questions about th
Hi, community:
Recently we encountered some problems when using individual
acknowledgments, such as:
when the amount of acknowledgment is large, entry writing fails; a large
amount of cache causes OOM, etc.
So I drafted a PIP in `
https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBc
18 matches
Mail list logo
