Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

[Girish Sharma]

Hi Xiangying,

Shouldn't the solution to the issue mentioned in #21564 [0] mostly be
around validating that topic level replication clusters are subset of
namespace level replication clusters?
It would be a completely compatible change as even today the case where a
topic has a cluster not defined in namespaces's replication-clusters
doesn't really work.
And users can anyway go and update the namespace's cluster list to add the
missing cluster.

As Rajan also mentioned, allowed-clusters field has a different
meaning/purpose.
Regards

On Thu, Nov 30, 2023 at 10:56 AM Xiangying Meng 
wrote:

> Hi, Pulsar Community
>
> I drafted a proposal to make the configuration of clusters at the namespace
> level clearer. This helps solve the problem of geo-replication not working
> correctly at the topic level.
>
> https://github.com/apache/pulsar/pull/21648
>
> I'm looking forward to hearing from you.
>
> BR
> Xiangying
>


-- 
Girish Sharma

Re: [Discuss] PIP-319 Unblock stuck Key_Shared subscription after consumer reconnect

[Rajan Dhabalia]

Hi Nikolai,

Thanks for bringing this to the attention. I have seen multiple PIP and bug
fixes for the Key-Shared subscription and I feel Key_Shared subscription
dispatcher is developed with multiple fundamental issues which required to
introduce scheduled-task to unblock stuck reads, introducing stuck delivery
flag, and now we are seeing at least 2-3 PIP to add few more hacks to
control such stuck dispatch. After reviewing code, it seems
PersistentStickyKeyDispatcherMultipleConsumers::getRestrictedMaxEntriesForConsumer(..)
function can still return 0 dispatch messages with multiple different edge
cases and fundamental dispatch is not clean at all. This is the only
dispatcher introduced after open sourcing Pulsar and unfortunately it was
not developed with a cleaner approach. I understand the pain of such stuck
dispatch and even the current release is having a bugs where dispatch is
getting stuck and many production systems are struggling with it and you
must be suggesting this PIP to get work around for this stuck issue. But I
don't think this is the right approach to follow.
I think the correct approach is to get rid of this "stuck" keyword from the
dispatcher because having "stuck" concept in dispatcher means it has a
known flow and bug which we are not solving but we are adding more debt to
perform the work around. We never had any such concept in other dispatchers
which was introduced when Pulsar was introduced originally and it just
works with clear semantics and permit flow mechanism , and we should follow
the same practice for this subscription.

Thanks,
Rajan

On Thu, Nov 30, 2023 at 9:18 AM Nikolai  wrote:

> Hello!
>
> I submitted a new PIP to add a configuration option which allows to skip
> blocking recently joined consumers for Key_Shared subscriptions
>
> It introduces additional memory consumption, so it will be disabled by
> default. It would fix issues like
> https://github.com/apache/pulsar/issues/21199
>
> Link to the PIP: https://github.com/apache/pulsar/pull/21615
>
> PR with the PIP implementation:
> https://github.com/apache/pulsar/pull/21579
>
>
> Best Regards,
>
> Nikolai
>
>

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

[Rajan Dhabalia]

Hi,

I don't agree with your assumptions about allowed clusters and
replication-cluster definition. Allowed clusters defined at tenant level
will restrict tenants to create namespaces in regions/clusters where they
are not allowed. and replication clusters will help tenant to set up full
mesh replication for that namespace across the clusters defined in
replication-clusters. So, each configuration has a different meaning to
serve. If anyone wants to setup different replication clusters then either
those topics can be created under different namespaces or defined at topic
level policy. Pulsar is being used by many legacy systems and changing its
semantics for specific usecases without considering consequences are
creating a lot of pain and incompatibility problems for other existing
systems and let's avoid doing it as we are struggling with such changes and
breaking compatibility or changing semantics are just not acceptable.

Thanks,
Rajan

On Wed, Nov 29, 2023 at 9:26 PM Xiangying Meng  wrote:

> Hi, Pulsar Community
>
> I drafted a proposal to make the configuration of clusters at the namespace
> level clearer. This helps solve the problem of geo-replication not working
> correctly at the topic level.
>
> https://github.com/apache/pulsar/pull/21648
>
> I'm looking forward to hearing from you.
>
> BR
> Xiangying
>

Re: [VOTE] Pulsar Release 2.11.3 Candidate 1

[Baodi Shi]

Hi, yubiao.

Thanks for your feedback.

After discussing with Penghui, we need to wait for the Bookkeeper 4.15.5
release and then upgrade it on Pulsar.

So, close this Vote first.

Thanks,
Baodi Shi


On Nov 29, 2023 at 12:09:00, Yubiao Feng
 wrote:

> Hi Penghui, Baodi
>
> - Run standalone and check the produce, consume
> - Verified Cassandra connector
> - Verified stateful function
>
> But the result of `unzip -t
> ./lib/org.apache.bookkeeper-circe-checksum-*.jar | grep lib` is not
> expected. I only got the logs below
>
> ```
>
> unzip -t ./lib/org.apache.bookkeeper-circe-checksum-*.jar | grep lib
>
>
>
> Archive:  ./lib/org.apache.bookkeeper-circe-checksum-4.15.4.jar
>
> No errors detected in compressed data of
> ./lib/org.apache.bookkeeper-circe-checksum-4.15.4.jar.
> ```
>
> I can not find `testing: lib/libcirce-checksum.so   OK` in the logs.
>
> By the way, the check above is newly added to the document by
> https://github.com/apache/pulsar-site/pull/740
>
> Thanks
> Yubiao Feng
>
> On Tue, Nov 21, 2023 at 9:25 PM Baodi Shi  wrote:
>
> This is the first release candidate for Apache Pulsar, version 2.11.3.
>
>
> It fixes the following issues:
>
>
>
> https://github.com/apache/pulsar/pulls?q=is%3Apr+label%3Arelease%2F2.11.3+is%3Aclosed
>
>
> *** Please download, test and vote on this release. This vote will stay
>
> open for at least 72 hours ***
>
>
> Note that we are voting upon the source (tag), binaries are provided for
>
> convenience.
>
>
> Source and binary files:
>
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.11.3-candidate-1/
>
>
> SHA-512 checksums:
>
>
>
>
> 1104ce10ee55f99f162f71487922d9883201516754936feab07a631b25b8f76bc2443735b4bdff17d821d62ca20f605d386ce9ca2e0450ce8d2555ca07fd8dd
>
>  ./apache-pulsar-2.11.3-bin.tar.gz
>
>
>
>
> bdf2579d718d25def297538def0c237974c856f63aea00db30e61b10683eec29a52a354b61daa6eda5ffe5bdfda78e6a83d473f8d7f44104fc5d715ffb1892fc
>
>  ./apache-pulsar-2.11.3-src.tar.gz
>
>
>
> Maven staging repo:
>
> https://repository.apache.org/content/repositories/orgapachepulsar-1251
>
>
> The tag to be voted upon:
>
> v2.11.3-candidate-1 (aa7082efcafb58b1fc4b7bb1bc68c6e22f7bc2d3)
>
> https://github.com/apache/pulsar/releases/tag/v2.11.3-candidate-1
>
>
> Pulsar’s KEYS file containing PGP keys you use to sign the release:
>
> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>
>
> Docker images:
>
> docker pull wudixiaobaozi/pulsar-all:2.11.3
>
> docker pull wudixiaobaozi/pulsar
>
>
> Please download the source package, and follow the README to build
>
> and run the Pulsar standalone service.
>
>
> Thanks,
>
> Baodi Shi
>
>
>

Re: [VOTE] PIP-315: Configurable max delay limit for delayed delivery

[Kevin Lu]

Hi All,

I am closing the vote as we have 4 binding +1's:
- Yubiao Feng
- Lari Hotari
- Matteo Merli
- Qiang Zhao

And 3 non-binding +1's:
- Apurva Telang
- Cong Zhao
- Kai Wang

Thanks everyone!

Regards,
Kevin

On Wed, Nov 29, 2023 at 4:25 PM Kai Wang  wrote:

> +1 (non-binding)
>
> Thanks,
> Kai Wang
>
> On 2023/11/15 03:59:46 Kevin Lu wrote:
> > Hi All,
> >
> > This thread is to start a vote for PIP-315.
> >
> > PIP: https://github.com/apache/pulsar/pull/21490
> > Discussion thread:
> > https://lists.apache.org/thread/285nm08842or324rxc2zy83wxgqxtcjp
> >
> > Regards,
> > Kevin
> >
>

Re: [Discuss] PIP-319 Unblock stuck Key_Shared subscription after consumer reconnect

[Dave Fisher]

Hi Nikolai - Your email was stuck in moderation. Please subscribe to the dev 
list by sending an email to dev-subscr...@pulsar.apache.org and respond to the 
CONFIRM email.

Welcome to the Pulsar community.

Best,
Dave

> On Nov 28, 2023, at 10:30 PM, Nikolai  wrote:
> 
> Hello!
> 
> I submitted a new PIP to add a configuration option which allows to skip 
> blocking recently joined consumers for Key_Shared subscriptions
> 
> It introduces additional memory consumption, so it will be disabled by 
> default. It would fix issues like 
> https://github.com/apache/pulsar/issues/21199
> 
> Link to the PIP: https://github.com/apache/pulsar/pull/21615
> 
> PR with the PIP implementation: https://github.com/apache/pulsar/pull/21579
> 
> 
> Best Regards,
> 
> Nikolai
> 

[Discuss] PIP-319 Unblock stuck Key_Shared subscription after consumer reconnect

[Nikolai]

Hello!

I submitted a new PIP to add a configuration option which allows to skip 
blocking recently joined consumers for Key_Shared subscriptions


It introduces additional memory consumption, so it will be disabled by 
default. It would fix issues like 
https://github.com/apache/pulsar/issues/21199


Link to the PIP: https://github.com/apache/pulsar/pull/21615

PR with the PIP implementation: https://github.com/apache/pulsar/pull/21579


Best Regards,

Nikolai

[Bug] std::system_error thrown during/after client->subscribe(...) when using configuration setUnAckedMessagesTimeoutMs. #358

[Tolfree, Jack]

Hello,

I have filed the following issue which is an unexpected and unhandled exception 
that terminates our application, full details in the issue.
>From looking at the code it appears the likely location for the exception is 
>the std::recursive_mutex.
It is not clear to us why this exception would occur and any insight into 
avoiding this would be greatly appreciated.
I have filed the issue already as I do believe there is room for improvement 
around the handling of the exception even if the exception could have been 
avoided.

[Bug] std::system_error thrown during/after client->subscribe(...) when using 
configuration setUnAckedMessagesTimeoutMs. * Issue #358 * 
apache/pulsar-client-cpp 
(github.com)

We will try a different approach that does not use 
'setUnAckedMessagesTimeotuMs', but it would be great to get this issue 
resolved, so that we can use this configuration.

Thanks in advance for your assistance.

Best regards,
Jack Tolfree
Lead Software Engineer
jack.tolf...@softwareag.com

[cid:image001.png@01DA2387.92713220]
[Icon  Description automatically 
generated]
[cid:image003.png@01DA2387.92713220]
[cid:image010.png@01DA2399.1691BFF0]
[cid:image006.png@01DA2387.92713220]
[cid:image007.png@01DA2387.92713220]



[cid:image008.png@01DA2387.92713220]

[cid:image009.gif@01DA2387.92713220]

This communication contains information which is confidential and may also be 
privileged. It is for the exclusive use of the intended recipient(s). If you 
are not the intended recipient(s), please note that any distribution, copying, 
or use of this communication or the information in it, is strictly prohibited. 
If you have received this communication in error please notify us by e-mail and 
then delete the e-mail and any copies of it.
Software AG (UK) Limited Registered in England & Wales 1310740 - 
http://www.softwareag.com/uk

RE: [ANNOUNCE] Apache Pulsar Client C++ 3.2.0 released

[Karthikeyan A]

Hi Team,

Any updates on including transaction support on C++ client ?

Thanks & Regards
Karthikeyan A

-Original Message-
From: Yunze Xu  
Sent: 17 May 2023 08:33
To: dev@pulsar.apache.org
Subject: Re: [ANNOUNCE] Apache Pulsar Client C++ 3.2.0 released

I have a plan to support the transaction on C++ client next month (June). It 
might be included in the 3.3.0 release.

Thanks,
Yunze

On Tue, May 16, 2023 at 10:09 PM Karthikeyan A 
 wrote:
>
> Hi,
>
> We have a requirement where the message has been consumed from one topic, 
> processed it and written to another topic in single transaction.
> Looks like transaction support related client APIs in C++ is still not 
> available and that is in roadmap of pulsar delivery.
>
> Please let us know the details on when the C++ client library for Transaction 
> Support will be available for use.
>
> Thanks & Regards
> Karthikeyan A
>
>
> -Original Message-
> From: Yunze Xu 
> Sent: 16 May 2023 18:17
> To: dev@pulsar.apache.org; us...@pulsar.apache.org; 
> annou...@apache.org
> Subject: [ANNOUNCE] Apache Pulsar Client C++ 3.2.0 released
>
> The Apache Pulsar team is proud to announce Apache Pulsar Client C++ version 
> 3.2.0.
>
> Pulsar is a highly scalable, low latency messaging platform running on 
> commodity hardware. It provides simple pub-sub semantics over topics, 
> guaranteed at-least-once delivery of messages, automatic cursor management 
> for subscribers, and cross-datacenter replication.
>
> For Pulsar release details and downloads, visit:
>
> https://pulsar.apache.org/download
>
> Release Notes are at:
> https://pulsar.apache.org/release-notes/client-cpp/
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The Pulsar Team