Re: [DISCUSS] Cherry-pick PIP-359 Support custom message listener executor for specific subscription

2024-09-02 Thread xiangying meng
Hi all Cherry-picked. Thanks On Mon, Aug 26, 2024 at 10:52 AM xiangying meng wrote: > > Hi yubiao > Oh, sorry. I used your cherry-pick email template. It looks really > professional and is very informative. > Unfortunately, I forgot to change the PIP number somewhere. Th

Re: {DISCUSS] PIP-374: Visibility of messages in receiverQueue for the consumers

2024-09-01 Thread xiangying meng
Good Work! This interceptor can be useful in many ways. Thanks On Mon, Sep 2, 2024 at 10:14 AM Cong Zhao wrote: > > Good work > > Thanks > Cong > > On 2024/08/30 04:57:01 "Bhat, Vinay" wrote: > > Hi All, > > > > I drafted a proposal for Visibility of messages in receiverQueue for the > > consum

Re: [VOTE] PIP-375 Expose the Admin client configs: readTimeout, requestTimeout, and connectionTimeout

2024-09-01 Thread xiangying meng
+1 (non-binding) On Fri, Aug 30, 2024 at 1:14 AM Apurva Telang wrote: > > +1 (non-binding) > > On Thu, Aug 29, 2024 at 4:38 AM Lari Hotari wrote: > > > +1 (binding) > > > > -Lari > > > > On 2024/08/28 04:45:53 Yubiao Feng wrote: > > > Hi all > > > > > > I want to start a vote on PIP-375: expose

Re: [DISCUSS] PIP-375 Expose the Admin client configs: readTimeout, requestTimeout, and connectionTimeout

2024-08-26 Thread xiangying meng
+1, this would be very useful. Thanks, Xiangying Meng On Mon, Aug 26, 2024 at 6:12 PM Yubiao Feng wrote: > > Hi all > > I drafted a proposal to expose the Admin client configs: `readTimeout`, > `requestTimeout`, and `connectionTimeout` > > https://github.com/apache/puls

Re: [DISCUSS] Cherry-pick PIP-359 Support custom message listener executor for specific subscription

2024-08-25 Thread xiangying meng
t; > BTW. there is a mistake in your email content: The "PIP-xxx" includes the > following PRs > > On Mon, Aug 26, 2024 at 10:39 AM xiangying meng > wrote: > > > Hi all > > > > I would like to start a discussion to cherry-pick PIP-359 into `branch-3.0`

[DISCUSS] Cherry-pick PIP-359 Support custom message listener executor for specific subscription

2024-08-25 Thread xiangying meng
Xiangying Meng

Re: [DISCUSS] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before

2024-08-23 Thread xiangying meng
-considered, and you are welcome to point it out. Thanks, Xiangying Meng On Fri, Aug 23, 2024 at 6:02 PM xiangying meng wrote: > > Hi Yubiao > If it's just for compatibility, wouldn't it be better to add a global > property? There is no need to add it for every topic. > > Thanks

Re: [DISCUSS] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before

2024-08-23 Thread xiangying meng
t contain > TXN messages. > > Thanks > Yubiao Feng > > On Fri, Aug 23, 2024 at 4:47 PM xiangying meng wrote: > > > Hi yubiao > > Then return to your plan, must we set the system property to false for > > topics that do not use transactions? I understand that it

Re: [DISCUSS] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before

2024-08-23 Thread xiangying meng
Hi yubiao Then return to your plan, must we set the system property to false for topics that do not use transactions? I understand that it is enough to only set the system properties of the topics that used the transaction. Thanks, Xiangying Meng On Fri, Aug 23, 2024 at 4:16 PM Yubiao Feng

Re: [DISCUSS] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before

2024-08-22 Thread xiangying meng
Good idea. I still have some questions about this. Since we can add data to the topic attribute, why not directly put the snapshot or snapshot segment position into the topic position? This method has two advantages: 1. System properties of topics that do not use transactions are not added. It wil

Re: [VOTE] PIP-367 Propose a Contributor Repository for Pulsar

2024-08-19 Thread xiangying meng
: > > Sorry, I would like to know more about it. > PIP-371(https://github.com/apache/pulsar/pull/23194) > > Can this be contributed to this repo? I think the repo is just a plug-in > implementation of the interface? > > > Thanks, > sinan > > > xiangying meng 于2024年

Re: [VOTE] PIP-367 Propose a Contributor Repository for Pulsar

2024-08-19 Thread xiangying meng
in/java/org/apache/pulsar/client/api/impl > > [6] execution plan: > > https://github.com/StevenLuMT/pulsar-java-contrib/issues/7 > > [7] VOTE for PIP-367: > > https://lists.apache.org/thread/td0j8l1c3l93nny0m5smnsdmb91j1n2y > > > > On Sat, Aug 3, 2024 at 5:25 PM

Re: [VOTE] PIP-368: Support lookup based on the lookup properties

2024-08-04 Thread xiangying meng
+1 nonbinding BR, Xiangying On Mon, Aug 5, 2024 at 10:02 AM Yunze Xu wrote: > > +1 (binding) > > Thanks, > Yunze > > On Thu, Aug 1, 2024 at 9:39 PM Jie crossover wrote: > > > > +1 nonbinding > > -- > > Best Regards! > > crossoverJie > > > > > > Tao Jiuming 于2024年8月1日周四 19:38写道: > > > > > +1 no

Re: [DISCUSS] Propose a Contributor Repository for Pulsar

2024-08-03 Thread xiangying meng
tart a separate > VOTE thread. > > Assuming that the PIP passes then we can create the new repository by > following your example and adjust the files to best fit policies. > > Best Regards, > Dave > > > On Jul 29, 2024, at 7:45 PM, xiangying meng wrote: > > &

[VOTE] PIP-367 Propose a Contributor Repository for Pulsar

2024-08-03 Thread xiangying meng
Hi all, I want to start a vote on PIP-367: Propose a Contributor Repository for Pulsar You can find the proposal at https://github.com/apache/pulsar/pull/23061 and the discussion thread at https://lists.apache.org/thread/75y70j6pqw15271b0tq8j637tb02mrwo The vote will stay open for at least 48 hou

Re: [DISCUSS] Propose a Contributor Repository for Pulsar

2024-07-29 Thread xiangying meng
4 at 2:59 PM Aurora Twinkle < > >> foreverlove...@gmail.com > >>>> > >>>> wrote: > >>>> > >>>>> Hi: > >>>>> I think this is a very good idea. > >>>>> > >>>>> In the nati

Re: [VOTE] PIP-366: Support to specify different config for Configuration and Local Metadata Store

2024-07-24 Thread xiangying meng
+1 (no-binding) Thanks Xiangying On Thu, Jul 25, 2024 at 9:50 AM Yubiao Feng wrote: > > +1 (binding) > > Thanks > Yubiao Feng > > On Mon, Jul 22, 2024 at 9:46 AM Kai Wang wrote: > > > Hi all, I want to start a vote on PIP-366: Support to specify different > > config for Configuration and Local

Re: [DISCUSS] Propose a Contributor Repository for Pulsar

2024-07-22 Thread xiangying meng
an reduce development costs. I think some people > > will be happy to use them. > > > > As for the other functions in the document, maintained in personal > > repositories, they can be used as a reference and provided to > > companies that are capable of solving securi

Re: [DISCUSS] Propose a Contributor Repository for Pulsar

2024-07-22 Thread xiangying meng
rovided to companies that are capable of solving security issues and bugs. Best regards, Xiangying On Tue, Jul 23, 2024 at 12:03 AM Enrico Olivelli wrote: > > Il giorno lun 22 lug 2024 alle ore 17:34 xiangying meng < > xiangyingme...@gmail.com> ha scritto: > > > >thanks for

Re: [DISCUSS] Propose a Contributor Repository for Pulsar

2024-07-22 Thread xiangying meng
Twinkle <https://github.com/AuroraTwinkle> @StevenLuMT > > <https://github.com/StevenLuMT> @cai152 <https://github.com/cai152>) will > > submit a few to pulsar-java-contrib > > <https://github.com/StevenLuMT/pulsar-java-contrib>( > > https://github.co

[DISCUSS] Propose a Contributor Repository for Pulsar

2024-07-22 Thread xiangying meng
Hello Pulsar Community, I hope this message finds you well. I'm reaching out to propose the establishment of a Pulsar Contributor Repository. This new initiative is designed to provide a dedicated space for experimental and community-driven features that complement our core offerings. This repos

Re: [VOTE] PIP-359: Support custom message listener executor for specific subscription

2024-06-18 Thread xiangying meng
+1 (no-binding) Thanks, Xiangying On Wed, Jun 19, 2024 at 2:14 PM Lari Hotari wrote: > +1 (binding) > > -Lari > > On 2024/06/18 16:44:46 Aurora Twinkle wrote: > > Hi, Pulsar Community: I would like to start the voting thread for > > PIP-359: Support > > custom message listener executor for spec

RE: RE: Re: [DISCUSS] PIP-359: Support custom message listener executor for specific subscription

2024-06-13 Thread xiangying meng
Good work! Same as Lari. Thanks Xiangying meng On 2024/06/14 03:39:01 xiangying meng wrote: > > > On 2024/06/14 03:13:03 Yubiao Feng wrote: > > Same as Lari > > > > Thanks > > Yubiao Feng > > > > On Thu, Jun 13, 2024 at 8:05 PM Aurora Twinkl

RE: Re: [DISCUSS] PIP-359: Support custom message listener executor for specific subscription

2024-06-13 Thread xiangying meng
On 2024/06/14 03:13:03 Yubiao Feng wrote: > Same as Lari > > Thanks > Yubiao Feng > > On Thu, Jun 13, 2024 at 8:05 PM Aurora Twinkle > wrote: > > > Hi, Pulsar Community. > > I open a new PIP for support custom message listener executor for specific > > subscription to avoid individual subscr

[DISCUSS] cherry-pick #22034 Create new ledger after the current ledger is closed

2024-04-07 Thread Xiangying Meng
Hi all, I want to start a discussion to cherry-pick #22034 [0] to release branches. The PR creates a new ledger after the current one is full. This is a bug fix to resolve the issue where the last ledger could not be deleted after expiration. Moreover, since there is no need to create a new ledger

Re: [DISCUSS] GEO-replication issues on topic level

2024-03-28 Thread Xiangying Meng
Hi zixuan, Thanks for your work in improving this geo-replication issue. In my opinion, this should be a mistake when implementing pulsar geo-replication at the topic level. As we know, after a user configures replication policies at the namespace level, the topics under the namespace will be crea

Re: [DISCUSS] Optimizing the Method of Estimating Message Backlog Size in Pulsar

2024-03-27 Thread Xiangying Meng
Agree. While the name might be misleading, it indeed accurately reflects the actual disk usage situation. BR On Wed, Mar 27, 2024 at 3:48 PM Girish Sharma wrote: > > Hi Xiangying, > > > > In the current implementation, the backlog size is estimated from the > > mark delete position to the last

Re: [DISCUSS] Optimize the Acktimeout Mechanism in Pulsar Client

2024-03-27 Thread Xiangying Meng
d ack requests. Thanks, On Wed, Mar 27, 2024 at 5:37 PM ZhangJian He wrote: > > Hi, Xiangying. Have you ever considered the `isAckReceiptEnabled` param? > > Thanks > ZhangJian He > > > On Wed, Mar 27, 2024 at 3:33 PM Xiangying Meng wrote: > > > Dear Pulsar Communi

[DISCUSS] Optimize the Acktimeout Mechanism in Pulsar Client

2024-03-27 Thread Xiangying Meng
Dear Pulsar Community, I would like to initiate a discussion regarding the optimization of the acktimeout mechanism on the client side. As we all know, the Pulsar consumer has a configuration for ack timeout that automatically redelivers unacknowledged messages after a certain period. The workflow

[DISCUSS] Optimizing the Method of Estimating Message Backlog Size in Pulsar

2024-03-26 Thread Xiangying Meng
Dear Pulsar Community, I would like to initiate a discussion regarding the optimization of the method used for estimating the message backlog size. In the current implementation, the backlog size is estimated from the mark delete position to the last confirm position, whereas the backlog message

[ANNOUNCE] Apache Pulsar 2.10.6 released

2024-03-08 Thread Xiangying Meng
The Apache Pulsar team is proud to announce Apache Pulsar version 2.10.6. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subs

Re: [VOTE] Pulsar Release 2.10.6 Candidate 2

2024-03-08 Thread Xiangying Meng
gt; Jiwei Guo (Tboy) > > > > > > On Fri, Mar 8, 2024 at 3:48 PM Lari Hotari wrote: > > > > > +1 (binding) > > > > > > - Built from source > > > - Checked the signatures of the source and binary release artifacts > > > - Run standal

[VOTE] Pulsar Release 2.10.6 Candidate 2

2024-03-07 Thread Xiangying Meng
This is the first release candidate for Apache Pulsar, version 2.10.6. It fixes the following issues: https://github.com/apache/pulsar/pulls?q=is:pr+label:cherry-picked/branch-2.10+label:release/2.10.6+is:closed *** Please download, test and vote on this release. This vote will stay open for at l

Re: [VOTE] Pulsar Release 2.10.6 Candidate 1

2024-03-07 Thread Xiangying Meng
Close this candidate, as we have a new fix, #22202, that needs to be contained in the release. I will raise a new candidate soon. Regards On Fri, Mar 8, 2024 at 10:06 AM Xiangying Meng wrote: > > This right. > Maybe we need some discussion and update our release policy. > > Tha

Re: [VOTE] Pulsar Release 2.10.6 Candidate 1

2024-03-07 Thread Xiangying Meng
ly when a fatal security issue occurs can we trigger a release of a new > version, but we also need to clarify the maintenance cycle, otherwise this > maintenance is endless. > > Thanks, > Zixuan > > Xiangying Meng 于2024年3月6日周三 16:45写道: > > > Dear Zixuan, > > >

Re: [VOTE] Pulsar Release 2.10.6 Candidate 1

2024-03-06 Thread Xiangying Meng
the 2.10, and update our doc. > > - [0] https://pulsar.apache.org/contribute/release-policy/ > > Thanks, > Zixuan > > > Xiangying Meng 于2024年3月6日周三 11:15写道: > > > This is the first release candidate for Apache Pulsar, version 2.10.6. > > > > It fixes th

[VOTE] Pulsar Release 2.10.6 Candidate 1

2024-03-05 Thread Xiangying Meng
This is the first release candidate for Apache Pulsar, version 2.10.6. It fixes the following issues: https://github.com/apache/pulsar/pulls?q=is:pr+label:cherry-picked/branch-2.10+label:release/2.10.6+is:closed *** Please download, test and vote on this release. This vote will stay open for at l

Re: [DISCUSS] Deletion of Current Ledger upon Rollover

2024-02-06 Thread Xiangying Meng
> > Il giorno mar 6 feb 2024 alle ore 13:01 Xiangying Meng > ha scritto: > > > > Dear Community, > > > > I hope this message finds you well. I am writing to discuss a > modification > > to the behavior of deleting the current ledger. As you may know, in &g

[DISCUSS] Deletion of Current Ledger upon Rollover

2024-02-06 Thread Xiangying Meng
Dear Community, I hope this message finds you well. I am writing to discuss a modification to the behavior of deleting the current ledger. As you may know, in Pulsar, the current ledger cannot be deleted because it may still be written to. However, there is an exception. When the current ledger is

Re: [DISCUSS] PIP-335: Oxia metadata support

2024-01-31 Thread Xiangying Meng
+1 Thanks, Xiangying On Thu, Feb 1, 2024 at 11:20 AM Yubiao Feng wrote: > +1 > > Thanks > Yubiao Feng > > On Thu, Feb 1, 2024 at 7:58 AM Matteo Merli > wrote: > > > https://github.com/apache/pulsar/pull/22009 > > > > === > > > > # PIP-335: Supporty Oxia metadata store plugin > > > > #

Re: [VOTE]PIP-321: Split the responsibilities of namespace replication-clusters

2024-01-19 Thread Xiangying Meng
Close this vote with three bindings. - Jiwei Guo (Tboy) - Penghui - Yubiao Feng On Fri, Jan 19, 2024 at 6:31 PM Yubiao Feng wrote: > +1 (binding) > > Thanks > Yubiao Feng > > On Tue, Jan 16, 2024 at 2:41 PM Xiangying Meng > wrote: > > > Dear Pulsar community, &g

[VOTE]PIP-321: Split the responsibilities of namespace replication-clusters

2024-01-15 Thread Xiangying Meng
Dear Pulsar community, I am initiating a voting thread for "PIP-321: Split the responsibilities of namespace replication-clusters". Here is the pull request for PIP-321: https://github.com/apache/pulsar/pull/21648 And the discussion thread: https://lists.apache.org/thread/87qfp8ht5s0fvw2y4t3j9yz

Re: [DISSCUSS] PIP-325: Add command to abort transaction

2023-12-25 Thread Xiangying Meng
Hi, Ruihong This proposal looks good to me. BR, Xiangying On Tue, Dec 19, 2023 at 8:13 PM Xiangying Meng wrote: > Hi, Ruihong > > Thanks for your proposal. > I wonder whether we should abort all the transactions one client creates > when the client crushes. > For example,

Re: [DISSCUSS] PIP-325: Add command to abort transaction

2023-12-19 Thread Xiangying Meng
Hi, Ruihong Thanks for your proposal. I wonder whether we should abort all the transactions one client creates when the client crushes. For example, a process builds a Pulsar client and creates a transaction by this client to do some operations. If the process crushes, the transaction cannot be co

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

2023-12-17 Thread Xiangying Meng
Hi Penghui >I'm sorry, I don't fully understand your point here. What is the "support replication on message and topic level"? >As I understand, are the `allowed-clusters` and `replication-clusters` more concise options? Pulsar support set replication-cluster for per message. After this proposal

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

2023-12-17 Thread Xiangying Meng
s` with cluster1, cluster2, cluster3, and > cluster4. The broker side will validate the topic or message replication > clusters from the `allowed-cluster.` > In this way, we can simplify more codes and logic here. > For *`topic-policy-synchronized-clusters` *we also add examples in

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

2023-12-07 Thread Xiangying Meng
el replication becomes valuable. Therefore, I lean towards the first option, to solve this problem and make it a truly expected topic policy. On Thu, Dec 7, 2023 at 12:45 PM Girish Sharma wrote: > Hello Xiangying, > > > On Thu, Dec 7, 2023 at 6:32 AM Xiangying Meng > wrot

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

2023-12-06 Thread Xiangying Meng
independently. Isn't this also a poor design for Pulsar? On Thu, Dec 7, 2023 at 2:28 AM Girish Sharma wrote: > Hello, replies inline. > > On Wed, Dec 6, 2023 at 5:28 PM Xiangying Meng > wrote: > > > Hi Girish, > > > > Thank you for your explanation. Becaus

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

2023-12-06 Thread Xiangying Meng
policy {replication clusters {cluster1, cluster2}} ``` [1] https://pulsar.apache.org/docs/3.1.x/administration-geo/#enable-geo-replication On Wed, Dec 6, 2023 at 7:57 PM Xiangying Meng wrote: > Hi Girish, > > Thank you for your explanation. Because Joe's email referenced the c

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

2023-12-06 Thread Xiangying Meng
This brings me to a very basic question - What's the use case that you > are > > trying to solve that needs these changes? Because, then what's stopping > us > > from bringing every construct that's at a namespace level (bundling, > > hardware affinity, etc)

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

2023-12-05 Thread Xiangying Meng
cies that applies to all > topics. The exceptions would be if a topic had a need for a deficit, The > case of one topic in the namespace sticking out of the namespace policy > umbrella is bad application design in my opinion > > -Joe. > > > > On Sun, Dec 3, 2023 at 6:00 PM

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

2023-12-03 Thread Xiangying Meng
work. > And users can anyway go and update the namespace's cluster list to add the > missing cluster. > > As Rajan also mentioned, allowed-clusters field has a different > meaning/purpose. > Regards > > On Thu, Nov 30, 2023 at 10:56 AM Xiangying Meng > wrote: > >

[DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

2023-11-29 Thread Xiangying Meng
Hi, Pulsar Community I drafted a proposal to make the configuration of clusters at the namespace level clearer. This helps solve the problem of geo-replication not working correctly at the topic level. https://github.com/apache/pulsar/pull/21648 I'm looking forward to hearing from you. BR Xiang

Re: [ANNOUNCE] Yubiao Feng as new PMC member in Apache Pulsar

2023-11-13 Thread Xiangying Meng
Congrats! Yubiao. Thanks, Xiangying On Mon, Nov 13, 2023 at 8:15 PM Kai Wang wrote: > Congrats! > > Thanks, > Kai >

[DISCUSS] PIP-311 Modify the signature of the `newMultiTransactionMessageAck` method

2023-10-23 Thread Xiangying Meng
Hi dev, Currently, the public API `newMultiTransactionMessageAck` in the `Commands.java` creates the ack commands without request ID and hopes to wait for the response. This causes all the requests created by `newMultiTransactionMessageAck` to be timed out. I drafted a proposal [0] to modify it. L

Re: [VOTE] PIP-302 Introduce refreshAsync API for TableView

2023-10-18 Thread Xiangying Meng
n hasReachedLatest()` > > method could do the trick. > > > > Thanks, > > Yunze > > > > On Sun, Oct 8, 2023 at 2:12 PM 太上玄元道君 wrote: > > > > > > +1 (no-binding) > > > > > > > > > Xiangying Meng 于2023年9月27日 周三15:

Re: [VOTE] PIP-302 Add new API readAllExistingMessages for TableView

2023-09-27 Thread Xiangying Meng
Close this via https://lists.apache.org/thread/vox93tmj33mms026wt52l92h1wffctbk On Mon, Sep 25, 2023 at 6:34 PM Xiangying Meng wrote: > > Thank you for your reminder. In our discussion, there were several > changes to the specific plan and method names, which resulted in the > PR tit

[VOTE] PIP-302 Introduce refreshAsync API for TableView

2023-09-27 Thread Xiangying Meng
Hi dev, This thread is to start a vote for PIP-302 Add new API refreshAsync for TableView. Discuss thread: https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q PIP: https://github.com/apache/pulsar/pull/21166 BR, Xiangying

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

2023-09-25 Thread Xiangying Meng
tion about this use case and how to use it. > > Sent from my iPhone > > > On Sep 25, 2023, at 6:23 PM, Xiangying Meng wrote: > > > > Hi Dave, > > The uncommitted transactions do not impact actual users' bank accounts. > > Business Processing System E o

Re: [DISCUSS] PIP-300: Add custom dynamic configuration for plugins

2023-09-25 Thread Xiangying Meng
Hi Zixuan, This is really a great feature. I support it. Regarding cherry-pick, as far as I know, we have cherry-picked some configuration items and interfaces into branch-2.10. But that should be mentioned in a separate discussion and provide sufficient reasons why we have to do it. Cherry-pick

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

2023-09-25 Thread Xiangying Meng
Sep 20, 2023, at 12:50 AM, Xiangying Meng wrote: > > > > Hi, all, > > > > Let's consider another example: > > > > **System**: Financial Transaction System > > > > **Operations**: Large volume of deposit and withdrawal operations, a >

Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar

2023-09-25 Thread Xiangying Meng
al. And I agree that we need a new PIP > for this change. > > BR, > Zike Yang > > On Mon, Sep 25, 2023 at 3:54 PM Xiangying Meng wrote: > > > > Hi Donglai, Heesung > > > > >brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute=60 is the same as >

Re: [VOTE] PIP-302 Add new API readAllExistingMessages for TableView

2023-09-25 Thread Xiangying Meng
ew to Read Latest Values” > > BR, > Zike Yang > > On Mon, Sep 25, 2023 at 3:25 PM Xiangying Meng wrote: > > > > Hi dev, > >This thread is to start a vote for PIP-302 Add new API > > readAllExistingMessages for TableView. > > Discuss thread: > &

Re: [DISCUSS] Unload Rate Limiting during Graceful Shutdown of Pulsar

2023-09-25 Thread Xiangying Meng
Hi Donglai, Heesung >brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute=60 is the same as brokerShutdownMaxNumberOfGracefulBundleUnloadPerSec=1 So, the "per-min" config can be more granular. I have some doubts about introducing the `brokerShutdownMaxNumberOfGracefulBundleUnloadPerMinute` conf

[VOTE] PIP-302 Add new API readAllExistingMessages for TableView

2023-09-25 Thread Xiangying Meng
Hi dev, This thread is to start a vote for PIP-302 Add new API readAllExistingMessages for TableView. Discuss thread: https://lists.apache.org/thread/o085y2314o0fymvx0x8pojmgjwcwn59q PIP: https://github.com/apache/pulsar/pull/21166 BR, Xiangying

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

2023-09-20 Thread Xiangying Meng
Hi, all, Let's consider another example: **System**: Financial Transaction System **Operations**: Large volume of deposit and withdrawal operations, a small number of transfer operations. **Roles**: - **Client A1** - **Client A2** - **User Account B1** - **User Account B2** - **Request Topic C

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

2023-09-18 Thread Xiangying Meng
g about these isolation levels. > > There is really no information in those Paimon issues. How is Paimon’s > ability to support Pulsar broken by this edge case? > > Best, > Dave > > Sent from my iPhone > > > On Sep 18, 2023, at 7:26 AM, Xiangying Meng wrote: >

Re: [DISSCUSS] PIP-298: Consumer supports specifying consumption isolation level

2023-09-18 Thread Xiangying Meng
Hi Dave, This is an external request. Paimon has added support for Kafka but has not yet incorporated support for Pulsar. Therefore, the Paimon community desires to integrate Pulsar. Furthermore, when integrating Pulsar into Paimon, it is desired to enable the ability to configure isolation levels

[DISCUSS] PIP-302 Add new API readAllExistingMessages for TableView

2023-09-11 Thread Xiangying Meng
Hi dev, I proposed a PIP, accessible via https://github.com/apache/pulsar/pull/21166, to introduce an API that allows us to wait until all data has been fully retrieved before accessing the value corresponding to the desired key. Please take a look and give your feedback. Best Regards, Xiangying

Re: [VOTE] PIP 296: Introduce the `getLastMessageIds` API to Reader

2023-08-31 Thread Xiangying Meng
Close this vote with 4 binding - Penghui - Mattison - Hang - tison On Mon, Aug 28, 2023 at 10:41 AM Zili Chen wrote: > > +1 (binding) > > Thanks for driving the proposal! > > On 2023/08/25 06:52:38 Xiangying Meng wrote: > > Hi Pulsar Community, > > > >

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-27 Thread Xiangying Meng
s for their contributions. This has been a successful collaboration. Best regards, Xiangying On Mon, Aug 28, 2023 at 10:13 AM Xiangying Meng wrote: > > Hi Penghui, > > >From my understanding. > >The message deduplication should only check the last chunk of the message. > >

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-27 Thread Xiangying Meng
. > Any subsequent messages with the same sequence ID and producer name will be > treated as > duplicates, no matter whether the sequence ID is generated by the producer > or specified by users. > > Regards, > Penghui > > On Sat, Aug 26, 2023 at 5:55 PM Xiangying Meng wr

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-26 Thread Xiangying Meng
alChunks, chunkedMsgBuffer)); ``` And the new solution we adopted in the PR [0] is to add a timestamp in the uuid. Thank Heesung for providing this idea again. [0] https://github.com/apache/pulsar/pull/20948 On Sat, Aug 26, 2023 at 5:20 PM Xiangying Meng wrote: > > Hi Zike, > > PR [0] ha

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-26 Thread Xiangying Meng
.com/apache/pulsar/pull/21070 > > For the behavior before Pulsar 3.0.0. The consumer should assemble the > message using 3,4,5. > > Thanks for pointing this out. > > BR, > Zike Yang > > On Sat, Aug 26, 2023 at 3:58 PM Xiangying Meng wrote: > > > > >&

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-26 Thread Xiangying Meng
e(e.g. maxMessageSize) ? > > uuid=1, sid=0, cid=0 > > uuid=1, sid=0, cid=1 > > uuid=2, sid=0, cid=0 > > uuid=2, sid=0, cid=1 > > > > We could refine what to track and algo logic on the broker side more, but > > do we agree that the broker chunk dedup

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-26 Thread Xiangying Meng
to track and algo logic on the broker side more, but > do we agree that the broker chunk dedup logic is needed? > > I will continue to think more next week. Have a nice weekend. > > > > > On Fri, Aug 25, 2023 at 9:14 PM Xiangying Meng wrote: > > > Hi Heesung, > &

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-25 Thread Xiangying Meng
is trying to resend the same msg(so m2) with the > same sequence id after restarting? > > > > > > On Fri, Aug 25, 2023 at 8:22 PM Xiangying Meng wrote: > > > Hi Heesung, > > > > In this case, the consumer only can receive m1. > > > > But it has

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-25 Thread Xiangying Meng
licate msg, m2 > s1, c0 > s1, c1 > s1, c2 //complete > > Do consumers receive m1 and m2(no dedup)? > > > > On Fri, Aug 25, 2023 at 6:55 PM Xiangying Meng wrote: > > > Hi Heesung, > > > > >I think this means, for the PIP, the broker side's

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-25 Thread Xiangying Meng
On Sat, Aug 26, 2023 at 9:34 AM Heesung Sohn wrote: > > I think this means, for the PIP, the broker side's chunk deduplication. > I think brokers probably need to track map to dedup > chunks on the broker side. > > > > > On Fri, Aug 25, 2023 at 6:16 PM Xiangying Men

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-25 Thread Xiangying Meng
the producer application jvm restarts in the middle of chunking and > resends the message chunks from the beginning with the previous sequence id? > > > > On Fri, Aug 25, 2023 at 5:15 PM Xiangying Meng wrote: > > > Hi Heesung > > > > It is a good idea to cover this

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-25 Thread Xiangying Meng
t; Why don't we add a chunk session id suffix to identify the ongoing chunking > uniquely? > > Currently, > > chunking uuid = producer + sequence_id > > Proposal > chunking uuid = producer + sequence_id + chunkingSessionId > > * chunkingSessionId could be a timestam

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-25 Thread Xiangying Meng
nk it's OK to persist them all in the topic. Is there any issue > with doing that? > > > There is another point. The resend of the chunk message has a bug that > I shared with you, and you fixed in PR [0]. It will make this case > happen in another way. > > If the use

[VOTE] PIP 296: Introduce the `getLastMessageIds` API to Reader

2023-08-24 Thread Xiangying Meng
Hi Pulsar Community, This is the vote thread for PIP 296: https://github.com/apache/pulsar/pull/21052 This PIP will help to improve the flexibility of Reader usage. Thanks, Xiangying

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-24 Thread Xiangying Meng
ingMessages`) 4. SequenceID: 0, ChunkID: 2 -> (Resend op1) 5. SequenceID: 0, ChunkID: 2 -> (Resend op2) 6. SequenceID: 0, ChunkID: 2 -> (Send op3) BR, Xiangying [0] - https://github.com/apache/pulsar/pull/21048 On Thu, Aug 24, 2023 at 8:09 PM Xiangying Meng wrote: > > >> This

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-24 Thread Xiangying Meng
gt; > For the existing behavior, the consumer assembles messages 3,4,5 into > > > > the original large message. But the changes brought about by this PIP > > > > will cause the consumer to use messages 1,2,5 for assembly. There is > > > > no

Re: [DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-23 Thread Xiangying Meng
e message after receiving the last chunk. We don't need to check > the deduplication for all previous chunks. Also by doing this we only > need bug fixes, we don't need to introduce a new PIP. > > BR, > Zike Yang > > On Fri, Aug 18, 2023 at 7:54 PM Xiangying Meng w

[DISCUSS]PIP-296 Introduce the `getLastMessageIds` API to Reader

2023-08-23 Thread Xiangying Meng
Hi, community, I would like to bring attention to the current absence of the `getLastMessageIds` method within the Reader interface, which has caused various inconveniences. To address this issue, I have prepared a proposal [1] to incorporate this API into the Reader interface. Please review the p

[DISCUSS]PIP-295: Fixing Chunk Message Duplication Issue

2023-08-18 Thread Xiangying Meng
Dear Community, I hope this email finds you well. I'd like to address an important issue related to Apache Pulsar and discuss a solution I've proposed on GitHub. The problem pertains to the handling of Chunk Messages after enabling deduplication. In the current version of Apache Pulsar, all chunk

Re: [VOTE] Pulsar Release 2.10.5 Candidate 1

2023-08-10 Thread Xiangying Meng
> > > ./pulsar-client-cpp/lib/CMakeFiles/PULSAR_OBJECT_LIB.dir/c/c_ClientConfiguration.cc.o > > > > ./pulsar-client-cpp/lib/CMakeFiles/PULSAR_OBJECT_LIB.dir/c/c_MessageRouter.cc.o > > ./pulsar-client-cpp/lib/CMakeFiles/PULSAR_OBJECT_LIB.dir/c/c_Client.cc.o > &g

[ANNOUNCE] Apache Pulsar 2.10.5 released

2023-08-01 Thread Xiangying Meng
The Apache Pulsar team is proud to announce Apache Pulsar version 2.10.5. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subs

Re: [VOTE] Pulsar Release 2.10.5 Candidate 1

2023-07-27 Thread Xiangying Meng
. > > > > > > guo jiwei 于2023年7月25日周二 20:53写道: > > > > > +1 (binding) > > > > > > Checked the signature > > > - Build from source > > > - Start standalone > > > - Publish and Consume > > > - Verified Cassandra connec

[VOTE] Pulsar Release 2.10.5 Candidate 1

2023-07-22 Thread Xiangying Meng
This is the first release candidate for Apache Pulsar, version 2.10.5. This release contains 128 commits by 48 contributors. https://github.com/apache/pulsar/compare/v2.10.4...v2.10.5-candidate-1 *** Please download, test, and vote on this release. This vote will stay open for at least 72 hours *

[DISCUSS] Apache Pulsar 2.10.5 release

2023-06-26 Thread Xiangying Meng
Hello, community: It has been more than 2 months since the release of 2.10.4. During this period, we have 64 fixes. https://github.com/apache/pulsar/compare/v2.10.4...branch-2.10 I suggest releasing 2.10.5. If you have no comments, I will check the existing PR that needs to cherry-pick to th

Re: [VOTE] PIP-251 Enhancing Transaction Buffer Stats and Introducing TransactionBufferInternalStats API

2023-05-21 Thread Xiangying Meng
king more from a user perspective > > > > > > > > > Nicolò Boschi > > > > > > > > > Il giorno mer 10 mag 2023 alle ore 10:58 Xiangying Meng < > > > xiangy...@apache.org> ha scritto: > > > > > > > Hello Pulsar

[VOTE] PIP-251 Enhancing Transaction Buffer Stats and Introducing TransactionBufferInternalStats API

2023-05-10 Thread Xiangying Meng
Hello Pulsar community, This thread is to start a vote for PIP-251: Enhancing Transaction Buffer Stats and Introducing TransactionBufferInternalStats API. Discussion thread: https://lists.apache.org/thread/jsh2rod208xg28mojxwrod84p5zt1nrw Issue: https://github.com/apache/pulsar/issues/20291 Voti

[VOTE] PIP-266: Support batch deletion of tenants, namespaces, topics, and subscriptions using input files and regex in Pulsar CLI

2023-05-04 Thread Xiangying Meng
Hello Pulsar community, This thread is to start a vote for PIP-266: Support batch deletion of tenants, namespaces, topics, and subscriptions using input files and regex in Pulsar CLI. Discussion thread: https://lists.apache.org/thread/bcw7wdbll6d85z9sry4g1mskfnn0nwrx Issue: https://github.com/apa

Re: [Discuss] Suggestion for a "clear" parameter in Pulsar-admin to simplify tenant and namespace cleanup

2023-04-25 Thread Xiangying Meng
for deletion. This would cater to situations where an arbitrary list of topics/namespaces needs to be deleted, providing a more comprehensive solution. Let me know if you have any further questions or suggestions. Best regards, Xiangying Meng On Tue, Apr 25, 2023 at 5:23 PM Zike Yang wrote

Re: [Discuss] Suggestion for a "clear" parameter in Pulsar-admin to simplify tenant and namespace cleanup

2023-04-24 Thread Xiangying Meng
me know if you have any further questions or suggestions. Best regards, Xiangying Meng On Mon, Apr 24, 2023 at 6:23 PM Girish Sharma wrote: > > Hello Yubiao, > As per my understanding, this feature suggestion is intended to delete the > topics from all replicated clusters under the nam

[DISCUSS] Pulsar Transaction Buffer Stats Enhancements and New API Proposal

2023-04-22 Thread Xiangying Meng
Dear Pulsar community, We would like to initiate a discussion on a proposal to enhance Pulsar's Transaction Buffer Stats and introduce a new API to improve visibility and troubleshooting capabilities. The proposal aims to provide more detailed information about the snapshot stats and system topic

Re: ANNOUNCE] Apache Pulsar 2.10.4 released

2023-04-20 Thread Xiangying Meng
t; Regards > > On Wed, Apr 19, 2023 at 2:09 PM Zike Yang wrote: > > > Hi, Xiangying > > > > Thanks for the announcement. > > I think we also need to send this email to us...@pulsar.apache.org and > > annou...@apache.org. > > > > BR, > > Zike Ya

  1   2   >