> Penghui suggested instead of communicating via Admin API we use the same
> methodology used in the new load balancer - using a compacted topic as a
> queue:
> The system topic you create will have compaction enabled.
> You write the pending ledger delete, and the key is ledger-ID and
>
On Sun, Mar 12, 2023 at 7:21 PM Yan Zhao wrote:
> > > In the LedgerDeletionService start, it will create a producer for
> sending
> > > pending delete ledger. When deleting a ledger, the producer sends
> > > PendingDeleteLedgerInfo to the system-topic. If the sent succeeds,
> delete
> > > the
> > In the LedgerDeletionService start, it will create a producer for sending
> > pending delete ledger. When deleting a ledger, the producer sends
> > PendingDeleteLedgerInfo to the system-topic. If the sent succeeds, delete
> > the ledger id from the whole ledgers and update the ledgers into the
Hi, Asaf, Tune the pip https://github.com/apache/pulsar/issues/16569, please
help to review it again, thanks!
Hi, Asaf. Tune the pip-186(https://github.com/apache/pulsar/issues/16569),
please help to view it again. thanks!
On 2023/02/16 13:50:54 Yan Zhao wrote:
> > If understood correctly, every broker will have a consumer right? You will
> > use a fail-over subscription? The retry-topic is consumed by
> If understood correctly, every broker will have a consumer right? You will
> use a fail-over subscription? The retry-topic is consumed by the same
> subscription, same consumer?
Yes, you are right, there is the case you mention. The deletion is idempotent,
I'm not sure if it's worth making it
On Tue, Feb 14, 2023 at 1:02 PM Yan Zhao wrote:
> > Shouldn't you specify the changes you are going to make in
> > `PulsarApi.proto`?
> We didn't change the proto file, we use the Restful API(AdminAPI).
> > Also, shouldn't you change the wording in this sentence in the PIP to
> > clarify you
> Shouldn't you specify the changes you are going to make in
> `PulsarApi.proto`?
We didn't change the proto file, we use the Restful API(AdminAPI).
> Also, shouldn't you change the wording in this sentence in the PIP to
> clarify you will use AdminAPI / Pulsar RPC?
We use AdminAPI, I will tune
On Mon, Feb 6, 2023 at 12:02 PM Yan Zhao wrote:
>
>
> On 2023/02/05 14:40:10 Asaf Mesika wrote:
> > On Fri, Feb 3, 2023 at 1:48 PM Yan Zhao wrote:
> >
> > > > If you persisted the message successfully by the producer and the
> broker
> > > > was terminated before being able to delete the ledger
On 2023/02/07 18:41:20 Heesung Sohn wrote:
> On Sun, Feb 5, 2023 at 2:26 AM Yan Zhao wrote:
>
> > On 2023/02/03 18:14:53 Heesung Sohn wrote:
> > > There are some cases to trigger it.
> > > 1. A cursor be removed.
> > > 2. Close the current ledger and create a new ledger.
> > > 3. Consumer ack
On Sun, Feb 5, 2023 at 2:26 AM Yan Zhao wrote:
> On 2023/02/03 18:14:53 Heesung Sohn wrote:
> > There are some cases to trigger it.
> > 1. A cursor be removed.
> > 2. Close the current ledger and create a new ledger.
> > 3. Consumer ack the message, the slowest cursor move forward.
> > 4. User
On 2023/02/05 14:40:10 Asaf Mesika wrote:
> On Fri, Feb 3, 2023 at 1:48 PM Yan Zhao wrote:
>
> > > If you persisted the message successfully by the producer and the broker
> > > was terminated before being able to delete the ledger from the metadata
> > > service?
> > If the broker is
On Fri, Feb 3, 2023 at 1:48 PM Yan Zhao wrote:
> > If you persisted the message successfully by the producer and the broker
> > was terminated before being able to delete the ledger from the metadata
> > service?
> If the broker is terminated, the consumer won't ack the message, the
> message
On 2023/02/03 20:05:59 Heesung Sohn wrote:
> For these internal requesters,
> 1. A cursor be removed.
> 2. Close the current ledger and create a new ledger.
> 3. Consumer ack the message, the slowest cursor move forward.
>
> How do we prevent these callers from requesting duplicate requests for
On 2023/02/03 18:14:53 Heesung Sohn wrote:
> There are some cases to trigger it.
> 1. A cursor be removed.
> 2. Close the current ledger and create a new ledger.
> 3. Consumer ack the message, the slowest cursor move forward.
> 4. User trigger truncateTopic by admin.
>
> I see that this pip is
For these internal requesters,
1. A cursor be removed.
2. Close the current ledger and create a new ledger.
3. Consumer ack the message, the slowest cursor move forward.
How do we prevent these callers from requesting duplicate requests for the
same resources(ledgers) (how do we handle racing
There are some cases to trigger it.
1. A cursor be removed.
2. Close the current ledger and create a new ledger.
3. Consumer ack the message, the slowest cursor move forward.
4. User trigger truncateTopic by admin.
I see that this pip is for the internal ledger deletion cases(1-3 above),
and it
> If we don't want to add more pressure on the metadata store from this
> feature as a requirement,
> I think we can use a system topic as well. Just to confirm, is this the
> direction we are agreeing on?
The design is too generic, I'm afraid not it is not suitable for ledger
deletion.
>
>
Heesung, thanks. That's a good point for multi steps works. But I'm afraid it
will increase the zk pressure and it also needs to handle some corner cases. I
prefer to use a system topic to handle it.
On 2023/01/31 19:05:34 Heesung Sohn wrote:
> On Tue, Jan 31, 2023 at 6:43 AM Yan Zhao wrote:
>
> If you persisted the message successfully by the producer and the broker
> was terminated before being able to delete the ledger from the metadata
> service?
If the broker is terminated, the consumer won't ack the message, the message
will be re-consume later.
> I recommend having the logic
On Thu, Feb 2, 2023 at 6:42 AM Asaf Mesika wrote:
> Heesung, there are users out there which delete 2M topics in a day. The
> stress we apply on ZK today due to the amount of topics is more than
> enough. I don't think we should rely on ZK to also manage the deletion of
> those ledgers if that
Heesung, there are users out there which delete 2M topics in a day. The
stress we apply on ZK today due to the amount of topics is more than
enough. I don't think we should rely on ZK to also manage the deletion of
those ledgers if that means adding more nodes / more reads and writes to it.
I
Heesung points out really good points, which actually made me add one more:
Can you please describe all the triggers for ledger deletion? Some are user
facing, some not.
On Mon, Jan 30, 2023 at 8:19 PM Heesung Sohn
wrote:
> Hi,
>
>
> I assume the deletion APIs are async(when a user requests
>
> When pulsar wants to delete a ledger, ManagedLedger uses
> ledgerDeletionService to send a message, the message content contains the
> waiting delete ledger info. After sending success, delete the ledger id
> from the metadata store.
> The consumer receives the message, it will use
On Tue, Jan 31, 2023 at 6:43 AM Yan Zhao wrote:
> > - Have we considered a metadata store to persist and dedup deletion
> > requests instead of the system topic? Why is the system topic the better
> > choice than a metadata store for this problem?
> If we use the metadata store to store the
> - Have we considered a metadata store to persist and dedup deletion
> requests instead of the system topic? Why is the system topic the better
> choice than a metadata store for this problem?
If we use the metadata store to store the middle step ledger, we need to
operate the metadata store
Hi,
I assume the deletion APIs are async(when a user requests deletion, pulsar
first returns success to the user if the request is persisted. Then, Pulsar
asynchronously runs the deletion flow)
- Have we considered a metadata store to persist and dedup deletion
requests instead of the system
> Couples of notes:
>
> 1.
>
> > In the LedgerDeletionService start, it will create a producer to send
> > pending delete ledger.
> > When delete a ledger, a PendingDeleteLedgerInfo msg with 1 min delay (the
> > delay is for consumer side, if send it immediately, maybe the metadata
> > din't
Couples of notes:
1.
> In the LedgerDeletionService start, it will create a producer to send
> pending delete ledger.
> When delete a ledger, a PendingDeleteLedgerInfo msg with 1 min delay (the
> delay is for consumer side, if send it immediately, maybe the metadata
> din't change when
We need more eyes and votes. Thanks.
Good idea.
> Good idea. I suggest the naming endings with `Seconds` like
> sendDelayOfTwoPhaseDeletionInSeconds,`
> reconsumeLaterOfTwoPhaseDeletionInSeconds`.
> >private int sendDelaySecondsOfTwoPhaseDeletion = 60;
> >private int reconsumeLaterSecondsOfTwoPhaseDeletion = 600;
Good idea. I suggest the naming endings with `Seconds` like
sendDelayOfTwoPhaseDeletionInSeconds,`
reconsumeLaterOfTwoPhaseDeletionInSeconds`.
>private int sendDelaySecondsOfTwoPhaseDeletion = 60;
>private int reconsumeLaterSecondsOfTwoPhaseDeletion = 600;
Yan Zhao 于2022年8月12日周五 10:45写道：
> I suggest to include 'topic' in the flag, we have too many entities in
> Pulsar
Thanks, change it to `topicTwoPhaseDeletionEnabled`.
Il Sab 23 Lug 2022, 03:26 Yan Zhao ha scritto:
> Yes, wo define `twoPhaseDeletionEnabled` in the Service Configuration.
>
Thanks.
I suggest to include 'topic' in the flag, we have too many entities in
Pulsar
Enrico
>
Yes, wo define `twoPhaseDeletionEnabled` in the Service Configuration.
Il Mer 20 Lug 2022, 18:05 Yan Zhao ha scritto:
> Hi, Enrico. If we bind the the topic with per-tenant, when tenant be
> deleted or the tenant not be load anymore, we data in the tennat system
> topic can't be consumed before the tenant next load.
>
This is a good point.
So let's go with the
Hi, Enrico. If we bind the the topic with per-tenant, when tenant be deleted or
the tenant not be load anymore, we data in the tennat system topic can't be
consumed before the tenant next load.
On 2022/07/14 15:35:16 Enrico Olivelli wrote:
> This is very interesting.
>
> I have only one
If we use per-tenant system topic or per-namespace. and there is no active
topic in the tenant anymore, how we consume the msg in this tenant.
This is the concern I cared about, so I prefer to use system topic.
This is very interesting.
I have only one concern.
I think that we should at least use a per-tenant system topic, or,
better, per-namespace.
There is no need to create the deletion topic if there is nothing to delete.
I am used to dealing with Pulsar clusters in which Tenants are
strictly
Hi Pulsar community: I open a pip to discuss "Introduce two phase deletion
protocol based on system topic" Proposal Link:
https://github.com/apache/pulsar/issues/16569
---
## Motivation
Original issue: #13238
In current ledger deletion, we divided it into two separate steps. It
happens in
40 matches
Mail list logo
