Re: [VOTE] PIP-315: Configurable max delay limit for delayed delivery

[Qiang Zhao]

+1 (binding)

Best,
Mattison

On 2023/11/15 03:59:46 Kevin Lu wrote:
> Hi All,
> 
> This thread is to start a vote for PIP-315.
> 
> PIP: https://github.com/apache/pulsar/pull/21490
> Discussion thread:
> https://lists.apache.org/thread/285nm08842or324rxc2zy83wxgqxtcjp
> 
> Regards,
> Kevin
> 

Re: [VOTE] PIP-280 : Refactor CLI Argument Parsing Logic for Measurement Units using JCommander's custom converter

[Qiang Zhao]

+1(binding)

Best,
Mattison

On 2023/07/07 09:25:22 Joo Hyuk Kim wrote:
> Hi community,
> 
> This PIP has received a couple of approvals in github PR link [1]
> So I thought it's time to vote.
> 
> ## Motivation
> 
> In the current Pulsar codebase, the logic to parse CLI arguments for
> measurement units like time and bytes is
> 
> scattered across various CLI classes. Each value read has its distinct
> parsing implementation, leading to a lack of code
> 
> reuse.
> 
> 
> ## Goals
> 
> 
> This PIP is to refactor the argument parsing logic to leverage the
> `@Parameter.converter`
> 
> functionality provided by JCommander [link 3]. This will isolate the
> measurement-specific parsing logic and increase
> 
> code
> 
> reusability.
> 
> 
> ### In Scope
> 
> 
> - Refactor all `Cmd` classes to utilize the converter functionality of
> JCommander. This will streamline the parsing
> 
>   logic and simplify the codebase.
> 
> - Refer to bottom section "Concrete Example", before "Links"
> 
> - Or on-going PR with small use case in
> https://github.com/apache/pulsar/pull/20663
> 
> 
> ## links
> 
> 
> [1] PR : https://github.com/apache/pulsar/pull/20691
> 
> 
> 
> Best regards,
> 
> JooHyukKim (Vince)
> 

Re: [DISCUSS] Allow the producer to connect to the topic with producer_request_hold backlog policy

[Qiang Zhao]

t when the topic
> > > no longer exceeds its quota.
> > >
> > > > It looks like we do not support the `producer_request_hold` semantics.
> > > We just easily use the same behaviour like `producer_exception`. Maybe 
> > > it's
> > > a missing feature.
> > >
> > > I agree that the only references to the
> > > RetentionPolicy.producer_request_hold enum have to do with disallowing
> > > producer creation or with disconnecting the producer when the backlog
> > > is exceeded [0].
> > >
> > > However, I think the feature does work if we look closer. The
> > > implementation is in the client (at least it is in the Java client).
> > > First, note that the only functional difference between
> > > producer_exception and producer_request_hold comes here [1] where two
> > > different exceptions are sent to the producer. Then, see that the
> > > producer handles these exceptions differently [2]. The producer fails
> > > the pending messages when the policy is producer_exception and the
> > > producer does nothing when the policy is producer_request_hold. In the
> > > second case, the producer will attempt to reconnect to the broker and
> > > will resend the messages that have been "held".
> > >
> > > It seems relevant to point out that the backlog quota's state is only
> > > changed on a 60 second interval by default (see
> > > backlogQuotaCheckIntervalInSeconds) and the default send timeout is 30
> > > seconds. Therefore, many sends will likely timeout on the client side
> > > before the broker updates the topic's state to "writable" and lets the
> > > producer reconnect. To use this feature meaningfully, it might make
> > > sense to increase the send timeout.
> > >
> > > The primary advantage for this solution is that the broker does not
> > > need to hold a producer's messages in memory for some undefined time.
> > >
> > > I just checked, and we do not have this behavior documented in the
> > > pulsar binary protocol spec [3]. We should update the spec to indicate
> > > how this feature is supposed to work, assuming we keep it this way.
> > >
> > > Thanks,
> > > Michael
> > >
> > > [0]
> > > https://github.com/apache/pulsar/blob/d24d82780fd27a98c6cdbee28d756ee7d419495f/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BacklogQuotaManager.java#L104-L107
> > > [1]
> > > https://github.com/apache/pulsar/blob/4c6989c4da6c0b18c9b0196630e03daf437cea68/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java#L1383-L1391
> > > [2]
> > > https://github.com/apache/pulsar/blob/955dcd10ce28b996811e194c9ad852b06ab30aee/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L1735-L1753
> > > [3] https://pulsar.apache.org/docs/next/developing-binary-protocol/
> > >
> > >
> > > On Thu, Aug 11, 2022 at 6:52 AM Qiang Zhao 
> > > wrote:
> > > >
> > > > Hi, Penghui
> > > >
> > > > I support your opinion.
> > > >
> > > > It looks like we do not support the `producer_request_hold` semantics.
> > > We just easily use the same behaviour like `producer_exception`. Maybe 
> > > it's
> > > a missing feature.
> > > >
> > > > Best,
> > > > Mattison
> > > >
> > > > On 2022/08/11 05:28:25 PengHui Li wrote:
> > > > > Hi all,
> > > > >
> > > > > Pulsar has a backlog quota policy `producer_request_hold` which will
> > > hold
> > > > > the message
> > > > > publish request. It is very useful for some data sync scenarios. It
> > > looks
> > > > > like the producer
> > > > > waiting for the consumer to process the data.
> > > > >
> > > > > But the new producer is not allowed to connect to the topic after
> > > reaching
> > > > > the max backlog
> > > > > limitation with producer_request_hold policy. The producer will get
> > > > > `ProducerBlockedQuotaExceededError` error. This caused inconvenience 
> > > > > to
> > > > > users, they need
> > > > > to have some retry to logic to try to create the producer again until
> > > the
> > > > > consumer acked more
> > > > > messages
> > > > >
> > > > > IMO, we should allow the producer to connect the topic. To keep the
> > > > > behavior consistent with
> > > > > the producer is already connected.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Best Regards,
> > > > > Penghui
> > > > >
> > >
> 

Re: [DISCUSS] Allow the producer to connect to the topic with producer_request_hold backlog policy

[Qiang Zhao]

Hi, Penghui

I support your opinion. 

It looks like we do not support the `producer_request_hold` semantics. We just 
easily use the same behaviour like `producer_exception`. Maybe it's a missing 
feature.

Best,
Mattison

On 2022/08/11 05:28:25 PengHui Li wrote:
> Hi all,
> 
> Pulsar has a backlog quota policy `producer_request_hold` which will hold
> the message
> publish request. It is very useful for some data sync scenarios. It looks
> like the producer
> waiting for the consumer to process the data.
> 
> But the new producer is not allowed to connect to the topic after reaching
> the max backlog
> limitation with producer_request_hold policy. The producer will get
> `ProducerBlockedQuotaExceededError` error. This caused inconvenience to
> users, they need
> to have some retry to logic to try to create the producer again until the
> consumer acked more
> messages
> 
> IMO, we should allow the producer to connect the topic. To keep the
> behavior consistent with
> the producer is already connected.
> 
> WDYT?
> 
> Best Regards,
> Penghui
> 

Re: [DISCUSSION] PIP-156: Enable system topic by default

[Qiang Zhao]

On 2022/04/26 12:07:32 guo jiwei wrote:
> Hi community:
>In PIP-39
> ,
> we have introduced namespace event to support topic level policy. This
> feature was released in Pulsar 2.6.
> 
>Then PIP-92
> ,
> we support topic policy to replicate across clusters and release in Pulsar
> 2.10.
> 
> And with version iteration, the corresponding topic policy CLI and
> functions are mature and stable, but users need to enable system topic to
> support these features.  So it's better to enable system topic by default.
> 
> 
> So I have opened apache/pulsar#15333
>  to discuss this.
> 
> 
> Regards
> Jiwei Guo (Tboy)
>