Re: (Apache committer criteria) [ANNOUNCE] New Committer: Asaf Mesika

> > know I’m not trying to be disrespectful, but it’s not respectful to be > biased and act like an expert during the reviews, while you’ve contributed > just for documentation PRs. When I talk about experience, I’m talking about > reviewers who don’t contribute to the project, they ask questions t

Re: (Apache committer criteria) [ANNOUNCE] New Committer: Asaf Mesika

I have not seen any specific asks or examples that give rise to these gripes, so it's all hypotheticals as of now. Please, as Enrico said, bring up specific issues. (1) How many reviewers (less than 1) have been involved in PIP review outside of the streamnative provider and how many of them have

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

n clusters should solely be used to establish full mesh > > replication for that specific namespace, without having any other > > definitions or functionalities. > > > > BR, > > Xiangying > > > > > > On Mon, Dec 4, 2023 at 1:52 PM Joe F wrote: > &g

Re: [DISCUSS] PIP-321 Split the responsibilities of namespace replication-clusters

>if users want to change the replication policy for topic-n and do not change the replication policy of other topics, they need to change all the topic policy under this namespace. This PIP unfortunately flows from attempting to solve bad application design A namespace is supposed to represent

Re: [DISCUSS] PIP-247: Notifications for partitions update

Why is this needed when we have notifications on regex sub changes? Aren't the partition names a well-defined regex? Joe On Thu, Feb 23, 2023 at 8:52 PM houxiaoyu wrote: > Hi Asaf, > thanks for your reminder. > > ## Changing > I have updated the following changes to make sure the notification a

Re: [DISCUSS] PIP-240 A new API to unload subscriptions

Inclined to agree with Enrico. If it's a hard problem, it will repeat, and this is not helping. If it's some race on the client, it will occur randomly and rarely, and this unload sub will get programmed in as a way of life. >If you don't think unloading the subscription can't help anything. Unl

Re: [DISCUSS] The use of consumer redeliverUnacknowledgedMessages method

I am not familiar with all the changes since it's original implementation, nor can I speak to for all the changes that went after The original concept was simple and rigorous. For shared sub, all unacked messages will be redelivered, and for Exclusive subs, the cursor was rewound and everything af

Re: [DISCUSS] PIP-221: Make TableView support read the non-persistent topic

he new load manager. It doesn't require strong consistency > ensure, and no need persistence. > > > Thanks, > Kai > > On 2022/11/14 23:03:13 Joe F wrote: > > I am not sure about the semantics of TableView on a non-persistent topic. > > > > Exactly how doe

Re: [DISCUSS] PIP-221: Make TableView support read the non-persistent topic

I am not sure about the semantics of TableView on a non-persistent topic. Exactly how does this work? What happens if the client crashes? What is the base state for the table? What exactly can I expect as a user from this? Joe On Sun, Nov 13, 2022 at 8:57 PM Kai Wang wrote: > Hi, pulsar-d

Re: [DISCUSSION] Redesign the MessageId interface

Id. It would > be easy to know what "ledger is 0, entry id is 1" means, users only > need to know the concepts > of ledger id and entry id. But it would be harder to know a tuple like > "0:1:-1:-1" means. > > Thanks, > Yunze > > On Tue, Nov 8, 2022 at

Re: [DISCUSSION] Redesign the MessageId interface

>Maybe this design is to hidden some details, but if users don't know the details like ledger id and entry id, how could you know what does "0:0:-1:0" mean? Abstractions exist for a reason. Ledgerid and entryid are implementation details, and an application should not be interpreting that at all

Re: [DISCUSS] Planning for Apache Pulsar 3.0

I would prefer that we avoid using the term “breaking changes”, which is too vague to convey any specific meaning. So let me try to bring some clarity. There have been many changes to implementations, APIs and data storage formats in Pulsar (and book keeper also). I have deployed many of these ch

Re: [DISCUSS] PIP-204: Reactive Java client for Apache Pulsar

+1 On Tue, Aug 30, 2022 at 9:37 AM Matteo Merli wrote: > +1 > > > -- > Matteo Merli > > > On Mon, Aug 29, 2022 at 5:56 AM Lari Hotari wrote: > > > > Hi all, > > > > I have drafted PIP-204: Reactive Java client for Apache Pulsar. > > > > PIP link: > > https://github.com/apache/pulsar/issues/173

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

Hi Rajan, > > > For consumers, the message will be sent when there are no more messages > to > > read? > > > Yes. I think it would be helpful to add a bit more detail on the consumer switch (sp. Shared consumers). There is a window of transition where "dispatched-but-not-acked messages" might ne

Re: [DISCUSS] PIP-181: Pulsar Shell

Why are there 2 PIPs being discussed as PIP-181? There's another discussion on dev with [DISCUSS] PIP-181: Provide new load balance placement strategy implementation for ModularLoadManagerStrategy On Sat, Jul 2, 2022 at 1:13 AM Haiting Jiang wrote: > +1, Great feature. > > Thanks, > Haiting >

Re: [DISCUSS] [PIP-152] Support subscription level dispatch rate limiter setting.

-1 The rate limits that are currently in place are there to protect the Pulsar service, to manage multi-tenancy on the broker. This is not meant as a feature to manage demand side throttling. In my opinion, this is best done as a client side feature. There is no need to add complexity to t

Re: [VOTE] PIP-136: Sync Pulsar policies across multiple clouds

es. > > I agree, Pulsar should have provision to prevent such scenarios where > > changes from one tenant in a cluster can impact other clusters. This PIP > > considers the tenant/admin will be the same at both the ends but that can > > not be true in all cases. We can add an

Re: [DISCUSS] Use the non-persistent topic to sync bundle load data

To clarify, historical profile that is persisted in ZK On Thu, Mar 17, 2022 at 11:54 AM Joe F wrote: > IIRC, there is a historical load profile for topic that feeds into > decisions by the load balancer. > > What happens during a cluster startup, with this new proposal? > >

Re: [DISCUSS] Use the non-persistent topic to sync bundle load data

IIRC, there is a historical load profile for topic that feeds into decisions by the load balancer. What happens during a cluster startup, with this new proposal? On Thu, Mar 17, 2022 at 7:50 AM PengHui Li wrote: > > But which brokers will own that topic ? > in a Pulsar cluster with a high l

Re: [DISCUSS] PIP-136: Sync Pulsar policies across multiple clouds

>On my first reading, it wasn't clear if there was only one topic required for this feature. I now see that the topic is not tied to a specific tenant or namespace. As such, we can avoid complicated authorization questions by putting the required event topic(s) into a "system" tenant and namespace

Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd

+1 (binding) On Sat, Jan 15, 2022 at 4:46 AM Enrico Olivelli wrote: > Il Sab 15 Gen 2022, 09:10 tamer Abdlatif ha > scritto: > > > Will that affect the existing ZK metadata in a pulsar instance , When we > > upgrade from a lower version to 2.10 ? In other words, > > Do we need a metadate migra

Re: [DISCUSSION] PIP-135: Include MetadataStore backend for Etcd

+1 On Wed, Jan 12, 2022 at 3:52 PM Aloys Zhang wrote: > +1 > > 陳智弘 于2022年1月12日周三 10:19写道： > > > +1 > > > > Haiting Jiang 於 2022年1月12日 週三 09:50 寫道： > > > > > +1 > > > > > > On 2022/01/12 01:44:21 PengHui Li wrote: > > > > +1 > > > > > > > > Penghui > > > > > > > > On Wed, Jan 12, 2022 at 8:39 A

Re: [PIP] Broker extensions to provide operators of enterprise-wide clusters better control and flexibility

Agree with Enrico. I am not clear on how this (allow interception of write and read operations of a managed ledger and modify the payload) would work with e2e encryption. That is literally a MITM proposal. Joe On Thu, Nov 18, 2021 at 9:04 AM Enrico Olivelli wrote: > Madhavan, > Thanks for sha

Re: Correct semantics of producer close

Correct semantics of producer close > > >>>> > > >>>> > > >>>> > > >>>> > but equally they might be > > >>>> > surprised when closeAsync doesn't complete because the pending >

Re: Correct semantics of producer close

> > capturing references to the async callbacks and then making business > > logic decisions based on the results of those callbacks. > > > > Thanks, > > Michael > > > > On Tue, Sep 28, 2021 at 4:58 AM Yunze Xu > wrote: > >> > >&

Re: Correct semantics of producer close

Clients should not depend on any of this behaviour, since the broker is at the other end of an unreliable network connection. The semantic differences are kind of meaningless from a usability point, since flushing on close =/= published. What exactly does "graceful" convey here? Flush the buffe

Re: PIP-95 Live migration of producer consumer from one cluster to another.

Is this mostly addressing a topic cutover or a whole cluster cutover? What is "pulsar-admin clusters set-generation --generation " That seems to dangle in the doc, without any explanation On Thu, Sep 16, 2021 at 11:19 AM Prashant Kumar < prashant.kumar.si...@gmail.com> wrote: > Dear Pulsar

Re: Topic metadata

I don't think pulsar should increase its dependency on ZK. Can we store it somewhere else? On Wed, Sep 22, 2021, 3:30 PM Matteo Merli wrote: > We already have a mechanism to store custom properties in managed > ledgers. So we don't need to store it separately. > -- > Matteo Merli > > > On Wed,

Re: PIP-93 Pulsar Proxy Protocol Handlers

Enrico, my initial comment when you brought up PH was in relation to the larger question about proxying, rather than looking at this in a limited fashion on how to make it easy to add new PH in the proxy. But specifically with this, here are my comments. Two very distinct abstractions are being

Re: Protocol Handlers in Pulsar Proxy

To give some history and context, Pulsar proxy was meant to be a barebones TCP proxy when it was built. It's sole reason to exist was to forward network traffic to the right host. Discovery/authn/z was a dependency . The way it came around, it was for a narrow use case (and it was not for k8s),

Re: Proposing a meetup organizing committee

The conversation here seems incoherent because of a few factors. One is by the use of "community'' and "project" interchangeably, as required to support this proposal - community in one context to support holding meetups/conferences and at the same time asking the PMC to manage this effort un

Re: Lack of retries on TooManyRequests

Suppose you have about a million topics and your Pulsar cluster goes down (say, ZK down). ..many millions of producers and consumers are now anxiously awaiting the cluster to comeback. .. fun experience for the first broker that comes up. Don't ask me how I know, ref blame ServerCnx.java#L429

Re: [Discuss] PIP to add system topic for topic creation/deletion events

I would be very careful about implementing such a feature, because of introducing undesirable interdependencies. Broker processes only talk to the metadata store or data store. This keeps brokers isolated from each other - one broker is not dependent on the functioning of another broker. A broke

Re: [PIP-78] Split the individual acknowledgments into multiple entries

I have a simpler question. Just storing the message-ids raw will fit ~300K entries in one ledger entry. With the bitmap changes, we can store a couple of million within one 5MB ledger entry. So can you tell us what numbers of unacked messages are creating a problem? What exactly are the issues

Re: Proposal for Consumer Filtering in Pulsar brokers

We have had discussions in the community list on server side logic previously. I would like to keep the specific proposal in this PIP aside, and address what this PIP is implicitly changing in core Pulsar design. I want to have an explicit discussion on that topic: what is the path for server-sid

Re: ReadOnly Topic Ownership Support

Very useful feature. I would like the proposers to think just beyond scaling consumers. If done right, this has the potential to open up a lot of use cases in ML, where you need to reprocess old/archived data. Being able to spin up a read-only broker, ( dedicated brokers that read from tiered s

Re: Reschedule Pulsar Summit SF to August/September

Thank you Sijie. Given what we know, this is the right thing to do at this time Joe On Tue, Mar 10, 2020 at 10:38 AM Sijie Guo wrote: > Dear all, > > In light of growing concern and the further spread of the COVID-19 > (Corona) virus and after close consultation with event stakeholders and the

Re: [DISCUESS] PIP 57: Improve Broker's Zookeeper Session Timeout Handling

sion expire time > as possible, since they don't impact failover time. > > So the main discussion should be about 1) - whether we are safe to > re-establish a zookeeper session to re-acquire bundles after the previous > session is expired. Hence we need to draw an agreement on und

Re: [DISCUESS] PIP 57: Improve Broker's Zookeeper Session Timeout Handling

On Sat, Feb 22, 2020 at 6:28 PM PengHui Li wrote: > Hi, joe > > The fundamental correctness is guaranteed by the fencing mechanism > provided by Apache BookKeeper and the CAS operation provided by the > metadata storage. Both fencing and CAS operations will prevent two owners > updating data or m

Re: [DISCUESS] PIP 57: Improve Broker's Zookeeper Session Timeout Handling

My concerns are listed in the PR comments. A broker is allowed to operate on a (resource) bundle under a lock. When a broker loses its session, the lock ownership COULD be lost. The right thing at this point is to give up the resource and re-acquire it. ( In fact, shutdown is just a shortcut to do

Re: [VOTE] Accept DotPulsar as part of Apache Pulsar project

+1 On Tue, Jan 14, 2020 at 6:36 AM Yu Liu wrote: > +1 👏 > > On Tue, Jan 14, 2020 at 10:19 AM Guangning E wrote: > > > +1, 👏👏👏 > > > > Thanks, > > Guangning > > > > xiaolong ran 于2020年1月14日周二 上午10:15写道： > > > > > Thanks Danske contribution, LGTM +1 > > > > > > -- > > > Thanks > > > Xiaolong Ran

Re: PIP 49: Permission levels and inheritance

; > https://github.com/apache/pulsar/wiki/PIP-49:-Permission-levels-and-inheritance > > > > > > > In this PIP, I only fix the unreasonable permissions in the current > > command and shown in bold. > > About specific permission of command, if there are different o

Re: PIP 49: Permission levels and inheritance

ng about the logical implication of this > dichotomy. For any object’s attributes some ought to be controlled by > object level permissions and others by sysadmin permissions. How can > developers tell? > > Best Regards, > Dave > > > Sent from my iPhone > > &

Re: PIP 49: Permission levels and inheritance

This proposal has the same issues as the previous one . In general it is not correct to think of commands as being controlled by owner of the object on which the command operates. Eg: It will be an error to assing control of all namespace commands to the namespace owner For eg: set subscription

Re: PIP 49: Permission levels and inheritance

It is good that we are looking at revamping the system. But the proposal as it is, is thin. First I would like this proposal to be split into two. One for inheritance and another for changes from existing controls. They are completely orthogonal and independent issues. Second, both of them (inher

Re: PIP 36: Max message size

I think 5MB is too large and it should be reduced. :-) . I am with Kafka on this one.. 1MB is good enough. Have we completely ruled out a split/join abstraction on the Pulsar produce/consumer APIs? No backward compatibility, no client-server capability mismatches, now that we have dedup, split

Re: [DISCUSS] PIP 33: Replicated subscriptions

ved similar to the existing proposal. On Wed, May 1, 2019 at 4:10 PM Matteo Merli wrote: > On Mon, Apr 29, 2019 at 1:57 PM Joe F wrote: > > > > I have suggestions for an alternate solution. > > > > If source message-ids were known for replicated messages, a composite &

Re: [DISCUSS] PIP 33: Replicated subscriptions

I have suggestions for an alternate solution. If source message-ids were known for replicated messages, a composite cursor can be maintained for replicated subscriptions as an n-tuple. Since messages are ordered from a source, it would be possible to restart from a known cursor n-tuple in any clu

Re: [DISCUSSION] Delayed message delivery

Delayed subscription is simpler, and probably worth doing in the broker IF done right. How is this different from a subscription running behind? Why does supporting that require this complex a change in the dispatcher, when we already support backlogged subscribers? I am extremely wary of change

Re: PIP 28: Improve pulsar proxy log messages

I run Pulsar proxy in production, and the same concern here. I don't think we can get any of these metrics unless we start parsing protocol, and definitely its going to make everything slower, and create additional memory and GC pressures. Joe On Wed, Jan 30, 2019 at 11:08 AM Matteo Merli wrot

Re: [DISCUSS] Skip tests for documentation related changes

+1 to Sanjeev's suggestion On Mon, Jan 28, 2019 at 12:15 PM Sanjeev Kulkarni wrote: > If developers are in charge of checking the checkbox, it might lead to > errors. Any way to make it automatic? Since docs are restricted to certain > areas of repo, maybe we can have some rules around that? > >

Re: [DISCUSSION] Delayed message delivery

To me this discussion presupposes that a streaming system should provide a service like a database. Before we discuss about how to implement this, we should look at whether this is something that fits into what is the core of Pulsar. I still have the same concerns against doing this in the brok

Re: [DISCUSS] PIP 26: Delayed Message Delivery

I am not a fan of adding complexity to the dispatch path, and I will always have serious concerns about proposals that do so, including this one. In general, I would prefer Pulsar to keep the dispatch path simple and efficient, and avoid server side implementations of business logic. Streaming at

Subject: [ANNOUNCE] Apache Pulsar 2.2.0 released

The Apache Pulsar team is proud to announce Apache Pulsar version 2.2.0. This is the first release of Apache Pulsar as an Apache Top Level Project. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guarante

Re: [VOTE] Pulsar Release 2.2.0 Candidate 2

ient using RPM/DEB packages > > -- > Masahiro Sakamoto > Yahoo Japan Corp. > E-mail: massa...@yahoo-corp.jp > -- > > > -Original Message- > > From: Joe F [mailto:j...@apache.org] > > Sent: Wednesday, October 17, 2018 3:34 AM > > To: dev@pulsar.apache.org

[VOTE] Pulsar Release 2.2.0 Candidate 2

This is the Second release candidate for Apache Pulsar, version 2.2.0 Release 2.2.0 adds new features and also fixes for various issues from 2.1.1 * Pulsar Java Client Interceptors * Integration of functions and io with schema registry * Dead Letter Topic * Flink Source connector * JDBC Sin

Release 2.2.0 is blocked for archive size limit

Issue INFRA-17151 - Increase SVN size limits for archive Our packages have busted the current size limit Joe

[CANCEL] [VOTE] Pulsar Release 2.2.0 Candidate 1

Joe F wrote: > This is the first release candidate for Apache Pulsar, version 2.2.0 > > It adds new features and also fixes for various issues from 2.1.1 > > * Pulsar Java Client Interceptors > * Integration of functions and io with schema registry > * Dead Letter To

[VOTE] Pulsar Release 2.2.0 Candidate 1

This is the first release candidate for Apache Pulsar, version 2.2.0 It adds new features and also fixes for various issues from 2.1.1 * Pulsar Java Client Interceptors * Integration of functions and io with schema registry * Dead Letter Topic * Flink Source connector * JDBC Sink Connector

Re: Pulsar release 2.2

Update on 2.2 release --- found some issues with schema enforcement when testing yesterday, and Sijie is working on fixing them. Joe On Wed, Oct 3, 2018 at 10:13 AM Joe F wrote: > Yes, I know. -- I spend quite some time debugging this in 2.2, and found > that it was fixed in master a

Re: Pulsar release 2.2

with #2693 <https://github.com/apache/pulsar/pull/2693>. So, > rebasing with mater will fix it. > > Thanks, > Rajan > > On Tue, Oct 2, 2018 at 7:16 PM, Joe F wrote: > > > Please, if you know of a problem that is bound to cause issue in release > > testin

Re: Pulsar release 2.2

d.com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1155) at org.apache.pulsar.functions.runtime.shaded.com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:338) at org.apache.pulsar.functions.runtime.JavaInstanceMain.start(JavaInstanceMain.java:114) On Tue, Oct 2, 201

Re: Pulsar release 2.2

i, Sep 21, 2018 at 5:49 PM Dave Fisher > > >> wrote: > > >>> > > >>>> BTW - the Infra team just now moved the repository out of the > > >> incubator! > > >>>> > > >>>> Check the release sc

Re: Pulsar release 2.2

; > https://github.com/apache/incubator-pulsar/pull/2543>) as far as > possible. > > Thanks. > > > 在 2018年9月18日，02:30，Joe F 写道： > > > > Now that 2.1.1 is completed, I intend to start the Pulsar 2.2 release > > process in a few days, and would like to free