RE: [ANNOUNCE] Apache Pulsar Client C++ 3.4.0 released
Hi, We have a requirement where the message has been consumed from one topic, processed it and written to another topic in single transaction. Looks like transaction support related client APIs in C++ is still not available and that is in roadmap of pulsar delivery. Please let us know the details on when the C++ client library for Transaction Support will be available for use. Thanks & Regards Karthikeyan A -Original Message- From: Yunze Xu Sent: 13 November 2023 19:03 To: dev@pulsar.apache.org; us...@pulsar.apache.org; annou...@apache.org Subject: [ANNOUNCE] Apache Pulsar Client C++ 3.4.0 released The Apache Pulsar team is proud to announce Apache Pulsar Client C++ version 3.4.0. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication. For Pulsar C++ client release details and downloads, visit: https://pulsar.apache.org/download/#pulsar-c-client Release Notes are at: https://pulsar.apache.org/release-notes/client-cpp API documents are at: https://pulsar.apache.org/api/cpp/3.4.x/index.html We would like to thank the contributors that made the release possible. Regards, The Pulsar Team
[DISCUSS] Release Pulsar Python clieng 3.4.0
Hi all, I would like to propose releasing the Pulsar Python client 3.4.0. It's over three months since the release of 3.3.0, and there are 46 new commits: https://github.com/apache/pulsar-client-python/compare/v3.0.0...main The Pulsar C++ client 3.4.1 has been released. I have upgraded the C++ client for the Python client to 3.4.1: https://github.com/apache/pulsar-client-python/pull/167 We need to cut a new release. Please let me know if you have any important fixes that need to be included in Pulsar Python client 3.4.0. Thanks, Zike Yang
RE: Re: Re: [Discuss] PIP-319 Unblock stuck Key_Shared subscription after consumer reconnect
Hi Rajan, Thanks for sharing the link! I will follow the PR. Best Regards, Nikolai On 2023/12/02 01:21:04 Rajan Dhabalia wrote: > Hi Nikolai, > > I completely understand your concern and pain. But let's understand > ordering semantics for exclusive/fail-over subscription. This subscription > makes sure to dispatch new messages which the broker is reading using > readPosition must be in order but that doesn't mean the broker will never > handle redelivery of unack message and broker will not stop dispatching by > considering the end of the world. However, key-Shared sub thinks it's the > end of the world. key-shared sub extends the shared sub behavior with > additional speciality of consumer dispatch affinity based on message key. > Broker tries its best to perform this responsibility but it's still not > guaranteed as it will be really costly to give such guarantee and if one > requires it then we can use a failover sub. In shared or even in other > subscriptions, brokers don't restrict or stop dispatching of redelivery > messages. > However, in key-shared sub, broker stops forever dispatching new and > redelivery messages in certain conditions and additionally broker goes into > an infinite iterative sequence of reading duplicate cold messages and > discarding them which causes IO and CPU impact both on broker and bookie. > Pulsar must not have such broken and incorrect semantic behavior in the > system. I have created an issue: > https://github.com/apache/pulsar/issues/21656 which has a simple unit test > case to reproduce this behavior and that's what is happening in multiple > production Pulsar systems and users are not happy with it. > > Therefore, we should not stop dispatching and restricting redeliver > messages which will automatically address almost all stuck issues and the > broker can still follow the ordering guarantee for new messages which it > promises for other subs as well. > > Thanks, > Rajan > > On Fri, Dec 1, 2023 at 12:03 AM Nikolai wrote: > > > Hi Rajan, > > > > Thanks for the review! > > > > It looks like the root cause of "stuck" keyword is the nature of > > Key_Shared subscription which must guarantee messages ordering. I > > understand it is really hard to achieve such guarantees without having > > separate physical partitions but this feature is very essential for many > > use cases. I also understand that the solution I've provided is more a > > hack than a solid solution. I was trying to touch as little code as I > > can and, unfortunately, I see no other options to avoid stuck dispatches > > in the current implementation approach. > > > > I also want to highlight that the current implementation leads entire > > system to hang due to a single message corruption in case consumers > > disconnects. This breaks one of a big Pulsar advantage: independent keys > > processing. > > > > Do you have plans for Key_Shared dispatcher rework? I would be happy to > > contribute to such a project if I can help somehow. > > > > > > Best Regards, > > > > Nikolai > > > > > > > > On 2023/12/01 06:34:59 Rajan Dhabalia wrote: > > > Hi Nikolai, > > > > > > Thanks for bringing this to the attention. I have seen multiple PIP > > and bug > > > fixes for the Key-Shared subscription and I feel Key_Shared subscription > > > dispatcher is developed with multiple fundamental issues which > > required to > > > introduce scheduled-task to unblock stuck reads, introducing stuck > > delivery > > > flag, and now we are seeing at least 2-3 PIP to add few more hacks to > > > control such stuck dispatch. After reviewing code, it seems > > > > > > > PersistentStickyKeyDispatcherMultipleConsumers::getRestrictedMaxEntriesForConsumer(..) > > > function can still return 0 dispatch messages with multiple different > > edge > > > cases and fundamental dispatch is not clean at all. This is the only > > > dispatcher introduced after open sourcing Pulsar and unfortunately it > > was > > > not developed with a cleaner approach. I understand the pain of such > > stuck > > > dispatch and even the current release is having a bugs where dispatch is > > > getting stuck and many production systems are struggling with it and you > > > must be suggesting this PIP to get work around for this stuck issue. > > But I > > > don't think this is the right approach to follow. > > > I think the correct approach is to get rid of this "stuck" keyword > > from the > > > dispatcher because having "stuck" concept in dispatcher means it has a > > > known flow and bug which we are not solving but we are adding more > > debt to > > > perform the work around. We never had any such concept in other > > dispatchers > > > which was introduced when Pulsar was introduced originally and it just > > > works with clear semantics and permit flow mechanism , and we should > > follow > > > the same practice for this subscription. > > > > > > Thanks, > > > Rajan > > > > > > On Thu, Nov 30, 2023 at 9:18 AM Nikolai wrote: > > > >
Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters
>if users want to change the replication policy for topic-n and do not change the replication policy of other topics, they need to change all the topic policy under this namespace. This PIP unfortunately flows from attempting to solve bad application design A namespace is supposed to represent an application, and the namespace policy is an umbrella for a similar set of policies that applies to all topics. The exceptions would be if a topic had a need for a deficit, The case of one topic in the namespace sticking out of the namespace policy umbrella is bad application design in my opinion -Joe. On Sun, Dec 3, 2023 at 6:00 PM Xiangying Meng wrote: > Hi Rajan and Girish, > Thanks for your reply. About the question you mentioned, there is some > information I want to share with you. > >If anyone wants to setup different replication clusters then either > >those topics can be created under different namespaces or defined at topic > >level policy. > > >And users can anyway go and update the namespace's cluster list to add the > >missing cluster. > Because the replication clusters also mean the clusters where the topic can > be created or loaded, the topic-level replication clusters can only be the > subset of namespace-level replication clusters. > Just as Girish mentioned, the users can update the namespace's cluster list > to add the missing cluster. > But there is a problem because the replication clusters as the namespace > level will create a full mesh replication for that namespace across the > clusters defined in > replication-clusters if users want to change the replication policy for > topic-n and do not change the replication policy of other topics, they need > to change all the topic policy under this namespace. > > > Pulsar is being used by many legacy systems and changing its > >semantics for specific usecases without considering consequences are > >creating a lot of pain and incompatibility problems for other existing > >systems and let's avoid doing it as we are struggling with such changes > and > >breaking compatibility or changing semantics are just not acceptable. > > This proposal will not introduce an incompatibility problem, because the > default value of the namespace policy of allowed-clusters and > topic-policy-synchronized-clusters are the replication-clusters. > > >Allowed clusters defined at tenant level > >will restrict tenants to create namespaces in regions/clusters where they > >are not allowed. > >As Rajan also mentioned, allowed-clusters field has a different > meaning/purpose. > > Allowed clusters defined at the tenant level will restrict tenants from > creating namespaces in regions/clusters where they are not allowed. > Similarly, the allowed clusters defined at the namespace level will > restrict the namespace from creating topics in regions/clusters where they > are not allowed. > What's wrong with this? > > Regards, > Xiangying > > On Fri, Dec 1, 2023 at 2:35 PM Girish Sharma > wrote: > > > Hi Xiangying, > > > > Shouldn't the solution to the issue mentioned in #21564 [0] mostly be > > around validating that topic level replication clusters are subset of > > namespace level replication clusters? > > It would be a completely compatible change as even today the case where a > > topic has a cluster not defined in namespaces's replication-clusters > > doesn't really work. > > And users can anyway go and update the namespace's cluster list to add > the > > missing cluster. > > > > As Rajan also mentioned, allowed-clusters field has a different > > meaning/purpose. > > Regards > > > > On Thu, Nov 30, 2023 at 10:56 AM Xiangying Meng > > wrote: > > > > > Hi, Pulsar Community > > > > > > I drafted a proposal to make the configuration of clusters at the > > namespace > > > level clearer. This helps solve the problem of geo-replication not > > working > > > correctly at the topic level. > > > > > > https://github.com/apache/pulsar/pull/21648 > > > > > > I'm looking forward to hearing from you. > > > > > > BR > > > Xiangying > > > > > > > > > -- > > Girish Sharma > > >
Re: Pulsar 3.0.2 docker image only supports arm64 platform
We need to use the following commands to push the images: ```sh regctl image copy 9947090/pulsar:3.0.2-12c92fe apachepulsar/pulsar:3.0.2 regctl image copy 9947090/pulsar-all:3.0.2-12c92fe apachepulsar/pulsar-all:3.0.2 ``` It's missing in the release doc. I will update it later. Thanks, Zike Yang On Mon, Dec 4, 2023 at 10:31 AM Yunze Xu wrote: > > I found Matteo has updated 3.1.1 as the latest image: > https://hub.docker.com/layers/apachepulsar/pulsar/latest/images/sha256-21e8bf1571e4ab559a51b3f6e524d725cffabe3c6836101f9d7ea7eb1e2bf62c?context=explore > > But the 3.0.2 still lacks the image for the amd64 platform. > > Thanks, > Yunze > > On Mon, Dec 4, 2023 at 12:51 AM Yunze Xu wrote: > > > > Hi all, > > > > When I ran unit tests for the pulsar-client-cpp repo, which uses > > `apachepulsar/pulsar:latest` image to start a standalone, I found the > > tests failed and the following error [1]: > > > > ``` > > standalone The requested image's platform (linux/arm64) does not match > > the detected host platform (linux/amd64/v3) and no specific platform > > was requested > > ``` > > > > It seems the 3.0.2 image [2] was only built for the arm64 platform. I > > think there is something wrong with the steps to build the image. It > > breaks the CI for pulsar-client-cpp repo, as well as any other repo > > that depends on the latest image. We should fix the image and push it > > again. > > > > [1] > > https://github.com/apache/pulsar-client-cpp/actions/runs/7078043224/job/19263093582?pr=360 > > [2] > > https://hub.docker.com/layers/apachepulsar/pulsar/3.0.2/images/sha256-5af72da140f3901bccc0a5b56de8764cd60f0d351011fd1b90c484a30889fbf8?context=explore > > > > Thanks, > > Yunze
Re: Pulsar 3.0.2 docker image only supports arm64 platform
I found Matteo has updated 3.1.1 as the latest image: https://hub.docker.com/layers/apachepulsar/pulsar/latest/images/sha256-21e8bf1571e4ab559a51b3f6e524d725cffabe3c6836101f9d7ea7eb1e2bf62c?context=explore But the 3.0.2 still lacks the image for the amd64 platform. Thanks, Yunze On Mon, Dec 4, 2023 at 12:51 AM Yunze Xu wrote: > > Hi all, > > When I ran unit tests for the pulsar-client-cpp repo, which uses > `apachepulsar/pulsar:latest` image to start a standalone, I found the > tests failed and the following error [1]: > > ``` > standalone The requested image's platform (linux/arm64) does not match > the detected host platform (linux/amd64/v3) and no specific platform > was requested > ``` > > It seems the 3.0.2 image [2] was only built for the arm64 platform. I > think there is something wrong with the steps to build the image. It > breaks the CI for pulsar-client-cpp repo, as well as any other repo > that depends on the latest image. We should fix the image and push it > again. > > [1] > https://github.com/apache/pulsar-client-cpp/actions/runs/7078043224/job/19263093582?pr=360 > [2] > https://hub.docker.com/layers/apachepulsar/pulsar/3.0.2/images/sha256-5af72da140f3901bccc0a5b56de8764cd60f0d351011fd1b90c484a30889fbf8?context=explore > > Thanks, > Yunze
Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters
Hi Rajan and Girish, Thanks for your reply. About the question you mentioned, there is some information I want to share with you. >If anyone wants to setup different replication clusters then either >those topics can be created under different namespaces or defined at topic >level policy. >And users can anyway go and update the namespace's cluster list to add the >missing cluster. Because the replication clusters also mean the clusters where the topic can be created or loaded, the topic-level replication clusters can only be the subset of namespace-level replication clusters. Just as Girish mentioned, the users can update the namespace's cluster list to add the missing cluster. But there is a problem because the replication clusters as the namespace level will create a full mesh replication for that namespace across the clusters defined in replication-clusters if users want to change the replication policy for topic-n and do not change the replication policy of other topics, they need to change all the topic policy under this namespace. > Pulsar is being used by many legacy systems and changing its >semantics for specific usecases without considering consequences are >creating a lot of pain and incompatibility problems for other existing >systems and let's avoid doing it as we are struggling with such changes and >breaking compatibility or changing semantics are just not acceptable. This proposal will not introduce an incompatibility problem, because the default value of the namespace policy of allowed-clusters and topic-policy-synchronized-clusters are the replication-clusters. >Allowed clusters defined at tenant level >will restrict tenants to create namespaces in regions/clusters where they >are not allowed. >As Rajan also mentioned, allowed-clusters field has a different meaning/purpose. Allowed clusters defined at the tenant level will restrict tenants from creating namespaces in regions/clusters where they are not allowed. Similarly, the allowed clusters defined at the namespace level will restrict the namespace from creating topics in regions/clusters where they are not allowed. What's wrong with this? Regards, Xiangying On Fri, Dec 1, 2023 at 2:35 PM Girish Sharma wrote: > Hi Xiangying, > > Shouldn't the solution to the issue mentioned in #21564 [0] mostly be > around validating that topic level replication clusters are subset of > namespace level replication clusters? > It would be a completely compatible change as even today the case where a > topic has a cluster not defined in namespaces's replication-clusters > doesn't really work. > And users can anyway go and update the namespace's cluster list to add the > missing cluster. > > As Rajan also mentioned, allowed-clusters field has a different > meaning/purpose. > Regards > > On Thu, Nov 30, 2023 at 10:56 AM Xiangying Meng > wrote: > > > Hi, Pulsar Community > > > > I drafted a proposal to make the configuration of clusters at the > namespace > > level clearer. This helps solve the problem of geo-replication not > working > > correctly at the topic level. > > > > https://github.com/apache/pulsar/pull/21648 > > > > I'm looking forward to hearing from you. > > > > BR > > Xiangying > > > > > -- > Girish Sharma >
Pulsar 3.0.2 docker image only supports arm64 platform
Hi all, When I ran unit tests for the pulsar-client-cpp repo, which uses `apachepulsar/pulsar:latest` image to start a standalone, I found the tests failed and the following error [1]: ``` standalone The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested ``` It seems the 3.0.2 image [2] was only built for the arm64 platform. I think there is something wrong with the steps to build the image. It breaks the CI for pulsar-client-cpp repo, as well as any other repo that depends on the latest image. We should fix the image and push it again. [1] https://github.com/apache/pulsar-client-cpp/actions/runs/7078043224/job/19263093582?pr=360 [2] https://hub.docker.com/layers/apachepulsar/pulsar/3.0.2/images/sha256-5af72da140f3901bccc0a5b56de8764cd60f0d351011fd1b90c484a30889fbf8?context=explore Thanks, Yunze
