date:20230226

Re: [ANNOUNCE] New committer: Yuri Mizushima

2023-02-26 Thread Horizon

Congrats!

Re: [DISCUSS] Change PIP template

2023-02-26 Thread Enrico Olivelli

+1 great suggestions

Enrico

Il giorno lun 27 feb 2023 alle ore 02:54  ha scritto:
>
> +1
>
> Best,
> Mattison
> On Feb 26, 2023, 23:02 +0800, Dave Fisher , wrote:
> > Excellent proposal.
> >
> > Inline below.
> >
> > Sent from my iPhone
> >
> > > On Feb 26, 2023, at 3:19 AM, Asaf Mesika  wrote:
> > >
> > > Hi,
> > >
> > > I would like to suggest two changes I'd like to make to the PIP design
> > > template:
> > > 1. Remove the form - just have a markdown template fill the issue body as
> > > it is created.
> > > 2. Change the PIP template structure
> > >
> > > == Removing the form
> > >
> > > Today, when you want to submit a PIP, you are required to fill out a form
> > > with boxes composed of 3-4 lines length.
> > > It's not good because:
> > > * It broadcasts to the author: we want a very small PIP, something that
> > > fits those small boxes.
> > > * It makes the PIP look like a bug, where you fill out fields.
> > > * It doesn't allow having H2 headings, only H1 headings, thus limiting the
> > > structure.
> > >
> > > A PIP is a design essentially, something 1-3 pages long. Thus,
> > > people take the time to write it down. Preferably, they copy paste the 
> > > body
> > > of the PIP issue, and use it to fill in sections.
> > >
> > > My suggestion is to define an issue template using only markdown, without 
> > > a
> > > form.
> > >
> > > == Changing PIP Structure
> > >
> > > Today the structure of the PIP doc (pasted below), is missing a section 
> > > and
> > > generally aims to jump directly into API changes / code / implementation.
> > > This results in lots of back and forth emails in an attempt to get the
> > > following essentials:
> > > * All required background knowledge to understand the proposal
> > > * A high level overview of the proposed solution
> > > * Understanding how this proposal will be monitored
> > > * What steps exactly I need to take if I revert to the previous version.
> > >
> > > The structure I propose below aims to reduce that friction and get all PIP
> > > aligned to provide that information.
> > >
> > > === Today's structure
> > >
> > > # Motivation
> > > * "Explain why this change is needed, what benefits it would bring to
> > > Apache Pulsar and what problem it's trying to solve."
> > > # Goal
> > > * "Define the scope of this proposal. Given the motivation stated above,
> > > what are the problems that this proposal is addressing and what other 
> > > items
> > > will be considering out of scope, perhaps to be left to a different PIP."
> > > # API Changes
> > > * "Illustrate all the proposed changes to the API or wire protocol, with
> > > examples of all the newly added classes/methods, including Javadoc"
> >
> > Yes this is important as api and similar changes are what triggers 
> > requiring small pips.
> >
> > > # Implementation
> > > * "This should be a detailed description of all the changes that are
> > > expected to be made. It should be detailed enough that any developer that
> > > is familiar with Pulsar internals would be able to understand all the 
> > > parts
> > > of the code changes for this proposal."
> > > * "This should also serve as documentation for any person that is trying 
> > > to
> > > understand or debug the behavior of a certain feature."
> > > # Alernatives
> > > * "If there are alternatives that were already considered by the authors
> > > or, after the discussion, by the community, and were rejected, please list
> > > them here along with the reason why they were rejected"
> > > # Anything else?
> > >
> > >
> > > === My suggestion
> > >
> > > # Motivation and Background information
> > > * Give a high level explanation on all concepts you will be using
> > > throughout this document. For example, if you want to talk about 
> > > Persistent
> > > Subscriptions, explain briefly (1 paragraph) what this is. If you're going
> > > to talk about Transaction Buffer, explain briefly what this is. If you're
> > > going to change something specific, that goes into a bit more detail about
> > > it and how it works. The Litmus test: I can read the design document and
> > > understand the problem statement and what you plan to change *without*
> > > resorting to a couple of hours of code reading just to start having a high
> > > level understanding of the change.
> > > * Provide links where possible if a person wants to dig deeper into the
> > > background information.
> > > * Explain what is the problem you're trying to solve - current situation.
> > > * This section is the "Why" of your proposal.
> > >
> > > # Goals
> > > ## Scope
> > > * Describe the goals of your proposal, and why it benefits Apache Pulsar
> > > ## Out of Scope
> > > * Describe what you have decided to keep out of scope, perhaps left for a
> > > different PIP/s.
> > >
> > > # High-level Design
> > > * Describe in high level, end-to-end, the solution. This should be a few
> > > paragraphs long as a guideline.
> > > * Reading this would allow me to understand the solution from a

Re: [ANNOUNCE] New committer: Yuri Mizushima

2023-02-26 Thread Enrico Olivelli

Congrats Yuri

well deserved !

Enrico

Il giorno lun 27 feb 2023 alle ore 08:47 Nozomi Kurihara
 ha scritto:
>
> Hi everyone,
>
> The Project Management Committee (PMC) for Apache Pulsar
> has invited Yuri Mizushima (https://github.com/equanz) to become a
> committer and we are pleased
> to announce that he has accepted.
>
> Yuri has been an active contributor for over 3 years. He has constantly
> sent Pull-Requests for Pulsar and its clients. Moreover he proposed PIP-79
> and gave a talk at Pulsar Summit NA 2021.
>
> Please join us in congratulating and welcoming Yuri onboard!
>
> Thanks,
> Nozomi

[ANNOUNCE] New committer: Yuri Mizushima

2023-02-26 Thread Nozomi Kurihara

Hi everyone,

The Project Management Committee (PMC) for Apache Pulsar
has invited Yuri Mizushima (https://github.com/equanz) to become a
committer and we are pleased
to announce that he has accepted.

Yuri has been an active contributor for over 3 years. He has constantly
sent Pull-Requests for Pulsar and its clients. Moreover he proposed PIP-79
and gave a talk at Pulsar Summit NA 2021.

Please join us in congratulating and welcoming Yuri onboard!

Thanks,
Nozomi

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

2023-02-26 Thread SiNan Liu

Hi! I updated the PIP issue again. This time I've added some background and
some explanations.

The compatibility check rules are already written in the Implementation.
ProtoBufNative implements the same canRead method as Apache Avro.
It does this by checking whether the schema for writing and reading is
compatible. I also indicate whether the writtenSchema and readSchema of the
Backward, Forward, and Full strategies are the old or the new version of
the schema.

Thanks,
sinan

Asaf Mesika  于2023年2月26日周日 23:24写道：

> I'm sorry, but this PIP lacks a lot of background knowledge, so you need to
> add IMO for people to understand it. You don't need to explain the entire
> pulsar in this PIP, but at the very least a few paragraphs detailing all
> you need to know, to put you in context:
>
>
>- Start by saying Pulsar as a built-in schema registry inside Pulsar
>broker.
>   - Every time the client updates the schema, it uploads it to the
>   broker. When that happens, it has a feature which validates if the
> new
>   schema version is compatible with the previous versions. There
> are 4 types
>   of compatibility: Full, ... (complete and explain each one briefly)
>- Also explain Pulsar Schema registry supports various schema
>protocols:  Avro, protobuf native, ... (complete the rest), each
> protocol
>has a schema which dictates how to serialize and deserialize the message
>content into typed object.
>- Explain in short what is protobuf native (compare protobuf non-native)
>- Please don't paste code instead of explaining.
>   - Explain that protobuf native current validation check is only
>   composed of checking the root message name is the same between
> the current
>   schema version and the new version.
>  - Explain briefly what is a root message and its name.
>   - Explain the problem (list scenarios) that we have because protobuf
>   native schema only supports FULL compatibility validation.
>
>
> Regarding high level design - as in what you plan to do.
> I suggest you add "High Level Design" and in it detail how you plan to
> validate, per protobuf version, per compatibility check (backward, forward,
> full,...).
> I tried reading the implementation - for me , it's all over the place. Can
> you please list in order what I wrote above, and list the validation rules
> with a good explanation why you validate it like that?
>
> Lastly, one you have all the validation rules clearly stated, you can use
> it to document it properly so users can know what validation to expect.
>
> Thanks,
>
> Asaf
>
>
> On Wed, Feb 22, 2023 at 5:10 PM SiNan Liu  wrote:
>
> > Sorry, my mistake. I removed the code and described the design to improve
> > the PROTOBUF_NATIVE schema compatibility checks. You can have a look. 
> >
> > Asaf Mesika  于2023年2月22日周三 21:16写道：
> >
> > > I read it but you're almost directly diving into the code - it will
> take
> > me
> > > hours just to reverse engineer your design.
> > >
> > > Can you please include a "High Level Design" section in which you
> explain
> > > how you plan to tackle any issue?
> > > If I can read that section and explain to someone else how this will
> > work,
> > > it means the section is complete.
> > >
> > > Let's leave the code to the PRs.
> > >
> > >
> > > On Sun, Feb 19, 2023 at 2:59 PM SiNan Liu 
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I made a PIP to discuss:
> https://github.com/apache/pulsar/issues/19565
> > .
> > > >
> > > > We can talk about the current design here. Especially for the field
> > type
> > > > change check rules, please give your valuable advice.
> > > >
> > > > Thanks,
> > > > Sinan
> > > >
> > >
> >
>

Re: [DISCUSS] PIP-247: Notifications for partitions update

2023-02-26 Thread houxiaoyu

Hi Joe and Michael,

I think I misunderstood what you replied before. Now I understand and
explain it again.

Besides the reasons what Asaf mentioned above, there are also some limits
for using topic list watcher.  For example the `topicsPattern.pattern` must
less that `maxSubscriptionPatternLeng` [0]. If the consumer subscribes
multi partitioned-topics, the `topicsPattern.pattern` maybe very long.

So I think that it's better to have a separate notification implementation
for partition update.

[0]
https://github.com/apache/pulsar/blob/5d6932137d76d544f939bef27df25f61b4a4d00d/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java#L115-L126

Thanks,
Xiaoyu Hou

houxiaoyu  于2023年2月27日周一 10:56写道：

> Hi Michael,
>
> >  I think we just need the client to "subscribe" to a topic notification
> for
> >  "-partition-[0-9]+" to eliminate the polling
>
> If pulsar users want to pub/sub a partitioned-topic, I think most of the
> users would like to create a simple producer or consumer like following:
> ```
> Producer producer = client.newProducer().topic(topic).create();
> producer.sendAsync(msg);
> ```
> ```
> client.newConsumer()
> .topic(topic)
> .subscriptionName(subscription)
> .subscribe();
> ```
> I think there is no reason for users to use `topicsPattern` if a pulsar
> just wants to subscribe a partitioned-topic. In addition, `topicsPattern`
> couldn't be used for producers.
>
> So I think PIP-145 [0] will benefit for regex subscriptions.  And this PIP
> [1] will benefit for the common partitioned-topic pub/sub scenario.
>
> [0] https://github.com/apache/pulsar/issues/14505
> [1] https://github.com/apache/pulsar/issues/19596
>
> Thanks
> Xiaoyu Hou
>
> Michael Marshall  于2023年2月25日周六 01:29写道：
>
>> > Just the way to implements partitioned-topic metadata
>> > notification mechanism is much like notifications on regex sub changes
>>
>> Why do we need a separate notification implementation? The regex
>> subscription feature is about discovering topics (not subscriptions)
>> that match a regular expression. As Joe mentioned, I think we just
>> need the client to "subscribe" to a topic notification for
>> "-partition-[0-9]+" to eliminate the polling.
>>
>> Building on PIP 145, the work for this PIP would be in implementing a
>> different `TopicsChangedListener` [1] so that the result of an added
>> topic is to add a producer/consumer to the new partition.
>>
>> I support removing polling in our streaming platform, but I'd prefer
>> to limit the number of notification systems we implement.
>>
>> Thanks,
>> Michael
>>
>> [0] https://github.com/apache/pulsar/pull/16062
>> [1]
>> https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PatternMultiTopicsConsumerImpl.java#L169-L175
>>
>>
>>
>> On Fri, Feb 24, 2023 at 1:57 AM houxiaoyu  wrote:
>> >
>> > Hi Joe,
>> >
>> > When we use PartitionedProducerImpl or MultiTopicsConsumerImpl,  there
>> is a
>> > poll task to fetch the metadata of the partitioned-topic regularly for
>> the
>> > number of partitions updated.  This PIP wants to use a
>> > notification mechanism to replace the metadata poll task.
>> >
>> > Just the way to implements partitioned-topic metadata
>> > notification mechanism is much like notifications on regex sub changes
>> >
>> > Joe F  于2023年2月24日周五 13:37写道：
>> >
>> > > Why is this needed when we have notifications on regex sub changes?
>> Aren't
>> > > the partition names a well-defined regex?
>> > >
>> > > Joe
>> > >
>> > > On Thu, Feb 23, 2023 at 8:52 PM houxiaoyu 
>> wrote:
>> > >
>> > > > Hi Asaf,
>> > > > thanks for your reminder.
>> > > >
>> > > > ## Changing
>> > > > I have updated the following changes to make sure the notification
>> > > arrived
>> > > > successfully:
>> > > > 1. The watch success response `CommandWatchPartitionUpdateSuccess`
>> will
>> > > > contain all the concerned topics of this watcher
>> > > > 2. The notification `CommandPartitionUpdate` will always contain
>> all the
>> > > > concerned topics of this watcher.
>> > > > 3. The notification `CommandPartitionUpdate`contains a monotonically
>> > > > increased version.
>> > > > 4. A map
>> `PartitonUpdateWatcherService#inFlightUpdate> > > > Pair>` will keep track of the updating
>> > > > 5. A timer will check the updating timeout through `inFlightUpdate`
>> > > > 6. The client acks `CommandPartitionUpdateResult` to broker when it
>> > > > finishes updating.
>> > > >
>> > > > ## Details
>> > > >
>> > > > The following mechanism could make sure the newest notification
>> arrived
>> > > > successfully, copying the description from GH:
>> > > >
>> > > > A new class, `org.apache.pulsar.PartitonUpdateWatcherService` will
>> keep
>> > > > track of watchers and will listen to the changes in the metadata.
>> > > Whenever
>> > > > a topic partition updates it checks if any watchers should be
>> notified
>> > > and
>> > > > sends an update for all topics

Re: [questions] Which jakartaEE version we plan for pulsar 3.0.0?

2023-02-26 Thread ZhangJian He

Hi @Enrico Olivelli  @Zixuan Liu

I'm concerned that this change may create some complex compatibility issues
between our client and Springboot. Springboot 2.x uses the package name
"javax" while Springboot 3.x uses "jakarta". I'm still not entirely sure
about the extent of its impact.

Thanks
ZhangJian He


On Mon, 27 Feb 2023 at 13:12, Zixuan Liu  wrote:

> > IIUC the main pain point is for Custom Servlet developers who will
> have to migrate to the jakarta.xxx packages
> is this correct ?
>
> Sure!
>
> Our web service and some http client depend on the jersey. I checked on
> jetty and our project. Looks like we can only use the Jakarta EE9.
>
> We neet to upgrade the jetty to the 11.0.x from the 9.x and jersey to
> 3.0.x. from 2.x.
>
> jetty 11.0.x - JakartaEE 9 - Java 11, you can see this in the
> https://www.eclipse.org/jetty/
> jersey 3.0.x - JakartaEE 9 - Java 8+(It looks like it supports Java 8, not
> sure how to implementation), you can see
> https://eclipse-ee4j.github.io/jersey/ and
>
> https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest30x/modules-and-dependencies.html#d0e560
>
> Thanks,
> Zixuan
>
> Enrico Olivelli  于2023年2月26日周日 20:48写道：
>
> > +1 to upgrade.
> >
> > IIUC the main pain point is for Custom Servlet developers who will
> > have to migrate to the jakarta.xxx packages
> > is this correct ?
> >
> > Enrico
> >
> > Il giorno sab 25 feb 2023 alle ore 06:46 Zixuan Liu
> >  ha scritto:
> > >
> > > Sounds greate that migrating to jersey 3.x from jersey 2.x.
> > >
> > > Thanks,
> > > Zixuan
> > >
> > > ZhangJian He  于2023年2月25日周六 11:01写道：
> > >
> > > > Hello, community. I want to know Which jakartaEE version we plan for
> > pulsar
> > > > 3.0.0. Now we use jersey 2.x, which uses Jakarta EE8.
> > > >
> > > > Maybe we will need to update the jersey.
> > > > Eclipse Jersey (eclipse-ee4j.github.io)
> > > > 
> > > > 2.x is for Jakarta EE8. 3.0.x is for Jakarta EE9 and 3.1.x is for
> > Jakarta
> > > > EE10.
> > > >
> > > > Thank you for your time and consideration.
> > > >
> > > > Thanks
> > > > ZhangJian He
> > > >
> >
>

Re: Introducer Pulsar admin api to pulsar-client-go

2023-02-26 Thread ZhangJian He

> Another idea is whether it is possible to generate clients for various
languages based on the current pulsar rest swagger files
Yes. But I don't think this is a suitable choice for the go client. Due to
the existing swagger file not fully complying with the specification, it
cannot be used directly to generate client code. Personally, I don't think
fixing it is particularly easy and there is currently no one working on it.
For the go client, I would still recommend creating a new repository and
having everyone contribute to it from there. WDYT?

Also, I would appreciate it if others could let me know your thoughts on
this matter. @PengHuiLi @Enrico Olivelli 

Thanks
ZhangJian He


On Sun, 26 Feb 2023 at 21:04, Guangning E  wrote:

> Another idea is whether it is possible to generate clients for various
> languages based on the current pulsar rest swagger files?
>
> Thanks,
> Guangning
>
> On Sat, Feb 25, 2023 at 1:44 PM Zixuan Liu  wrote:
>
> > > I would like to suggest creating a new repository called
> > "pulsar-admin-go"
> >
> > +1, I agreed with you.
> >
> > Thanks,
> > Zixuan
> >
> > ZhangJian He  于2023年2月25日周六 11:05写道：
> >
> > > Following our previous discussions via email, I would like to suggest
> > > creating a new repository called "pulsar-admin-go". This repository
> will
> > > serve as a platform for all of us to contribute our code. Pulsarctl[0]
> go
> > > first.
> > >
> > > Please let me know your thoughts on this matter.
> > >
> > > [0] - https://github.com/streamnative/pulsarctl
> > >
> > > Thanks
> > > ZhangJian He
> > >
> > >
> > > On Fri, 17 Feb 2023 at 23:24, ZhangJian He  wrote:
> > >
> > > > Thank for StreamNative for willing to donate this project. This means
> > we
> > > > don't have to develop and maintain a set of HTTP code from scratch.
> My
> > > idea
> > > > aligns with Yunze's, and separating it into a standalone
> > pulsar-admin-go
> > > > project would be better. The **pulsarctl** repo contains bookkeeper
> > http
> > > > call too. Maybe we can have a project bookkeeper-admin-go ?（it's a
> > liitle
> > > > going off-topic ）
> > > >
> > > > Thanks
> > > > ZhangJian He
> > > >
> > > >
> > > > On Fri, 17 Feb 2023 at 20:29, PengHui Li 
> > > wrote:
> > > >
> > > >> Hi Yunze,
> > > >>
> > > >> Yes, we can split it.
> > > >> Both one repo with two modules or two repos works for me.
> > > >>
> > > >> The pulsarctl already have the admin API and CLI.
> > > >> So I think we don’t need to develop another one.
> > > >>
> > > >> Best,
> > > >> Penghui
> > > >>
> > > >> > On Feb 17, 2023, at 17:44, Yunze Xu  >
> > > >> wrote:
> > > >> >
> > > >> > Hi PengHui,
> > > >> >
> > > >> > Now I changed my mind a bit. Even if the pulsarctl was contributed
> > to
> > > >> > the Apache Foundation, I think we should also avoid adding it as
> the
> > > >> > dependency. What we need is an API layer but not the CLI, while
> > > >> > pulsarctl couples the API and CLI.
> > > >> >
> > > >> > At the moment, my expectation is:
> > > >> > 1. Use a separate repo (e.g. pulsar-admin-go) to implement the
> admin
> > > >> > APIs in Golang.
> > > >> > 2. Depend this new repo in pulsarctl.
> > > >> >
> > > >> > Then we will have three Go projects:
> > > >> > - pulsar-client-go: The Pulsar Go client APIs
> > > >> > - pulsar-admin-go: The Pulsar Go admin APIs
> > > >> > - pulsarctl: The admin CLI tool written in Go
> > > >> >
> > > >> > Thanks,
> > > >> > Yunze
> > > >> >
> > > >> > On Fri, Feb 17, 2023 at 4:22 PM PengHui Li 
> > > wrote:
> > > >> >>
> > > >> >> I checked with Sijie today.
> > > >> >> StreamNative can contribute the pulsarctl project to Apache
> > > Foundation.
> > > >> >>
> > > >> >> Regards,
> > > >> >> Penghui
> > > >> >>
> > > >> >> On Fri, Feb 17, 2023 at 4:02 PM Enrico Olivelli <
> > eolive...@gmail.com
> > > >
> > > >> wrote:
> > > >> >>
> > > >> >>> I agree to add an admin API to the go client, this would be very
> > > >> helpful.
> > > >> >>>
> > > >> >>> Il giorno ven 17 feb 2023 alle ore 08:44 Zixuan Liu
> > > >> >>>  ha scritto:
> > > >> 
> > > >>  Hi Zhangjian,
> > > >> 
> > > >>  This is a good idea to write the admin client by golang, but I
> > > don't
> > > >>  suggest add the admin features to pulsar-go-client, it's better
> > to
> > > >> use a
> > > >>  new repository to do that to  separate dependencies.
> > > >> 
> > > >>  BTW, StreamNative has a pulsarctl [0] tool, which includes the
> > > admin
> > > >> api.
> > > >> 
> > > >> >> It's better to reuse existing code rather than reinventing
> the
> > > >> wheel.
> > > >> 
> > > >>  I aggred this point. If possible, we can integrate the
> pulsarctl
> > to
> > > >> this
> > > >>  new project.
> > > >> >>>
> > > >> >>> We are talking about adding a client that calls a
> > > >> >>> well defined and maintained REST API.
> > > >> >>> It is better to have our implementation and not rely on third
> > > parties
> > > >> >>> when it is possible.
> > > >> >>> If there is a security issue

Re: [questions] Which jakartaEE version we plan for pulsar 3.0.0?

2023-02-26 Thread Zixuan Liu

> IIUC the main pain point is for Custom Servlet developers who will
have to migrate to the jakarta.xxx packages
is this correct ?

Sure!

Our web service and some http client depend on the jersey. I checked on
jetty and our project. Looks like we can only use the Jakarta EE9.

We neet to upgrade the jetty to the 11.0.x from the 9.x and jersey to
3.0.x. from 2.x.

jetty 11.0.x - JakartaEE 9 - Java 11, you can see this in the
https://www.eclipse.org/jetty/
jersey 3.0.x - JakartaEE 9 - Java 8+(It looks like it supports Java 8, not
sure how to implementation), you can see
https://eclipse-ee4j.github.io/jersey/ and
https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest30x/modules-and-dependencies.html#d0e560

Thanks,
Zixuan

Enrico Olivelli  于2023年2月26日周日 20:48写道：

> +1 to upgrade.
>
> IIUC the main pain point is for Custom Servlet developers who will
> have to migrate to the jakarta.xxx packages
> is this correct ?
>
> Enrico
>
> Il giorno sab 25 feb 2023 alle ore 06:46 Zixuan Liu
>  ha scritto:
> >
> > Sounds greate that migrating to jersey 3.x from jersey 2.x.
> >
> > Thanks,
> > Zixuan
> >
> > ZhangJian He  于2023年2月25日周六 11:01写道：
> >
> > > Hello, community. I want to know Which jakartaEE version we plan for
> pulsar
> > > 3.0.0. Now we use jersey 2.x, which uses Jakarta EE8.
> > >
> > > Maybe we will need to update the jersey.
> > > Eclipse Jersey (eclipse-ee4j.github.io)
> > > 
> > > 2.x is for Jakarta EE8. 3.0.x is for Jakarta EE9 and 3.1.x is for
> Jakarta
> > > EE10.
> > >
> > > Thank you for your time and consideration.
> > >
> > > Thanks
> > > ZhangJian He
> > >
>

Re: [DISCUSS] PIP-247: Notifications for partitions update

2023-02-26 Thread houxiaoyu

Hi Asaf,

>  Prefer something more explicit: onTopicPartitionsCountChanged
> Maybe we should mark "PartitionsChangedListener" deprecated?
> 60 seconds seems like *a lot* to miss an update. Maybe 5-10 sec

It looks good to me

> Let's think about the case of endless retries

We will remove the watcher from broker if `channelInactive` invoked like
PIP-145.  Endless retries should not happen if we do so I think.

> Can you elaborate on which broker you'll be doing that conversation with?
> Partitioned topic are spread across several brokers, so how can one broker
> know the count of the partitioned topic?

The client side  `PartitionUpdateWatcher` will extends `HandlerState`, It
could get connection to any broker services. The broker knows the count of
the partitioned topic throught metaDataStore, and it also could knows the
partitioned changed event from watching metaDataStore.

Thanks,
Xiaoyu Hou


Asaf Mesika  于2023年2月27日周一 00:31写道：

>  Michael,
>
> I think the current suggested protocol is more optimized compared to
> existing regex mechanism:
>
>- If the broker sends notification and it's lost due network issues,
>you'll only know about it due to the client doing constant polling,
> using
>its hash to minimize response.
>   - In the suggested mechanism, there's no constant polling at all.
>   Just think of introducing polling to *all* multiple partition
> producers and
>   consumers (Which likely all pulsar users). The suggested mechanism
> has a
>   configured timeout allowing it to resend on timeout, but only
> when it fails
>   as opposed to constant polling mechanisms.
>- Each time meta-update you'll need to run it through regular
>expression, on all topics hosted on the broker, for any given client.
>That's a lot of CPU.
>   - Suggested mechanism mainly cares about the count of partitions, so
>   it's a lot more efficient.
>
>
> Xiaoyu,
>
> I don't like the "onTopicExtend" method name. Prefer something more
> explicit: onTopicPartitionsCountChanged.
>
> Let's think about the case of endless retries - you know, the client is
> acting out, and never acknowledges. Need to to think about it.
> 60 seconds seems like *a lot* to miss an update. Maybe 5-10 sec?
>
> Can you elaborate on which broker you'll be doing that conversation with?
> Partitioned topic are spread across several brokers, so how can one broker
> know the count of the partitioned topic?
>
> Maybe we should mark "PartitionsChangedListener" deprecated? Having two
> interfaces is confusing.
>
>
>
> On Fri, Feb 24, 2023 at 7:29 PM Michael Marshall 
> wrote:
>
> > > Just the way to implements partitioned-topic metadata
> > > notification mechanism is much like notifications on regex sub changes
> >
> > Why do we need a separate notification implementation? The regex
> > subscription feature is about discovering topics (not subscriptions)
> > that match a regular expression. As Joe mentioned, I think we just
> > need the client to "subscribe" to a topic notification for
> > "-partition-[0-9]+" to eliminate the polling.
> >
> > Building on PIP 145, the work for this PIP would be in implementing a
> > different `TopicsChangedListener` [1] so that the result of an added
> > topic is to add a producer/consumer to the new partition.
> >
> > I support removing polling in our streaming platform, but I'd prefer
> > to limit the number of notification systems we implement.
> >
> > Thanks,
> > Michael
> >
> > [0] https://github.com/apache/pulsar/pull/16062
> > [1]
> >
> https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PatternMultiTopicsConsumerImpl.java#L169-L175
> >
> >
> >
> > On Fri, Feb 24, 2023 at 1:57 AM houxiaoyu  wrote:
> > >
> > > Hi Joe,
> > >
> > > When we use PartitionedProducerImpl or MultiTopicsConsumerImpl,  there
> > is a
> > > poll task to fetch the metadata of the partitioned-topic regularly for
> > the
> > > number of partitions updated.  This PIP wants to use a
> > > notification mechanism to replace the metadata poll task.
> > >
> > > Just the way to implements partitioned-topic metadata
> > > notification mechanism is much like notifications on regex sub changes
> > >
> > > Joe F  于2023年2月24日周五 13:37写道：
> > >
> > > > Why is this needed when we have notifications on regex sub changes?
> > Aren't
> > > > the partition names a well-defined regex?
> > > >
> > > > Joe
> > > >
> > > > On Thu, Feb 23, 2023 at 8:52 PM houxiaoyu 
> wrote:
> > > >
> > > > > Hi Asaf,
> > > > > thanks for your reminder.
> > > > >
> > > > > ## Changing
> > > > > I have updated the following changes to make sure the notification
> > > > arrived
> > > > > successfully:
> > > > > 1. The watch success response `CommandWatchPartitionUpdateSuccess`
> > will
> > > > > contain all the concerned topics of this watcher
> > > > > 2. The notification `CommandPartitionUpdate` will always contain
> all
> > the
> > > > > concerned topics

Re: [DISCUSS] PIP-247: Notifications for partitions update

2023-02-26 Thread houxiaoyu

Hi Michael,

>  I think we just need the client to "subscribe" to a topic notification
for
>  "-partition-[0-9]+" to eliminate the polling

If pulsar users want to pub/sub a partitioned-topic, I think most of the
users would like to create a simple producer or consumer like following:
```
Producer producer = client.newProducer().topic(topic).create();
producer.sendAsync(msg);
```
```
client.newConsumer()
.topic(topic)
.subscriptionName(subscription)
.subscribe();
```
I think there is no reason for users to use `topicsPattern` if a pulsar
just wants to subscribe a partitioned-topic. In addition, `topicsPattern`
couldn't be used for producers.

So I think PIP-145 [0] will benefit for regex subscriptions.  And this PIP
[1] will benefit for the common partitioned-topic pub/sub scenario.

[0] https://github.com/apache/pulsar/issues/14505
[1] https://github.com/apache/pulsar/issues/19596

Thanks
Xiaoyu Hou

Michael Marshall  于2023年2月25日周六 01:29写道：

> > Just the way to implements partitioned-topic metadata
> > notification mechanism is much like notifications on regex sub changes
>
> Why do we need a separate notification implementation? The regex
> subscription feature is about discovering topics (not subscriptions)
> that match a regular expression. As Joe mentioned, I think we just
> need the client to "subscribe" to a topic notification for
> "-partition-[0-9]+" to eliminate the polling.
>
> Building on PIP 145, the work for this PIP would be in implementing a
> different `TopicsChangedListener` [1] so that the result of an added
> topic is to add a producer/consumer to the new partition.
>
> I support removing polling in our streaming platform, but I'd prefer
> to limit the number of notification systems we implement.
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar/pull/16062
> [1]
> https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PatternMultiTopicsConsumerImpl.java#L169-L175
>
>
>
> On Fri, Feb 24, 2023 at 1:57 AM houxiaoyu  wrote:
> >
> > Hi Joe,
> >
> > When we use PartitionedProducerImpl or MultiTopicsConsumerImpl,  there
> is a
> > poll task to fetch the metadata of the partitioned-topic regularly for
> the
> > number of partitions updated.  This PIP wants to use a
> > notification mechanism to replace the metadata poll task.
> >
> > Just the way to implements partitioned-topic metadata
> > notification mechanism is much like notifications on regex sub changes
> >
> > Joe F  于2023年2月24日周五 13:37写道：
> >
> > > Why is this needed when we have notifications on regex sub changes?
> Aren't
> > > the partition names a well-defined regex?
> > >
> > > Joe
> > >
> > > On Thu, Feb 23, 2023 at 8:52 PM houxiaoyu  wrote:
> > >
> > > > Hi Asaf,
> > > > thanks for your reminder.
> > > >
> > > > ## Changing
> > > > I have updated the following changes to make sure the notification
> > > arrived
> > > > successfully:
> > > > 1. The watch success response `CommandWatchPartitionUpdateSuccess`
> will
> > > > contain all the concerned topics of this watcher
> > > > 2. The notification `CommandPartitionUpdate` will always contain all
> the
> > > > concerned topics of this watcher.
> > > > 3. The notification `CommandPartitionUpdate`contains a monotonically
> > > > increased version.
> > > > 4. A map
> `PartitonUpdateWatcherService#inFlightUpdate > > > Pair>` will keep track of the updating
> > > > 5. A timer will check the updating timeout through `inFlightUpdate`
> > > > 6. The client acks `CommandPartitionUpdateResult` to broker when it
> > > > finishes updating.
> > > >
> > > > ## Details
> > > >
> > > > The following mechanism could make sure the newest notification
> arrived
> > > > successfully, copying the description from GH:
> > > >
> > > > A new class, `org.apache.pulsar.PartitonUpdateWatcherService` will
> keep
> > > > track of watchers and will listen to the changes in the metadata.
> > > Whenever
> > > > a topic partition updates it checks if any watchers should be
> notified
> > > and
> > > > sends an update for all topics the watcher concerns through the
> > > ServerCnx.
> > > > Then we will record this request into a map,
> > > > `PartitonUpdateWatcherService#inFlightUpdate > > Pair > > > long/*timestamp*/>>`.  A timer will check this update timeout through
> > > > inFlightUpdate .  We will query all the concerned topics's partition
> if
> > > > this watcher has sent an update timeout and will resend it.
> > > >
> > > > The client acks `CommandPartitionUpdateResult` to broker when it
> finishes
> > > > updating.  The broker handle `CommandPartitionUpdateResult` request:
> > > >  - If CommandPartitionUpdateResult#version <
> > > > PartitonUpdateWatcherService#inFlightUpdate.get(watcherID).version,
> > > broker
> > > > ignores this ack.
> > > >  -  If CommandPartitionUpdateResult#version ==
> > > > PartitonUpdateWatcherService#inFlightUpdate.get(watcherID).version
> > > > - If

Re: [DISCUSS] Change PIP template

2023-02-26 Thread mattisonchao

+1

Best,
Mattison
On Feb 26, 2023, 23:02 +0800, Dave Fisher , wrote:
> Excellent proposal.
>
> Inline below.
>
> Sent from my iPhone
>
> > On Feb 26, 2023, at 3:19 AM, Asaf Mesika  wrote:
> >
> > Hi,
> >
> > I would like to suggest two changes I'd like to make to the PIP design
> > template:
> > 1. Remove the form - just have a markdown template fill the issue body as
> > it is created.
> > 2. Change the PIP template structure
> >
> > == Removing the form
> >
> > Today, when you want to submit a PIP, you are required to fill out a form
> > with boxes composed of 3-4 lines length.
> > It's not good because:
> > * It broadcasts to the author: we want a very small PIP, something that
> > fits those small boxes.
> > * It makes the PIP look like a bug, where you fill out fields.
> > * It doesn't allow having H2 headings, only H1 headings, thus limiting the
> > structure.
> >
> > A PIP is a design essentially, something 1-3 pages long. Thus,
> > people take the time to write it down. Preferably, they copy paste the body
> > of the PIP issue, and use it to fill in sections.
> >
> > My suggestion is to define an issue template using only markdown, without a
> > form.
> >
> > == Changing PIP Structure
> >
> > Today the structure of the PIP doc (pasted below), is missing a section and
> > generally aims to jump directly into API changes / code / implementation.
> > This results in lots of back and forth emails in an attempt to get the
> > following essentials:
> > * All required background knowledge to understand the proposal
> > * A high level overview of the proposed solution
> > * Understanding how this proposal will be monitored
> > * What steps exactly I need to take if I revert to the previous version.
> >
> > The structure I propose below aims to reduce that friction and get all PIP
> > aligned to provide that information.
> >
> > === Today's structure
> >
> > # Motivation
> > * "Explain why this change is needed, what benefits it would bring to
> > Apache Pulsar and what problem it's trying to solve."
> > # Goal
> > * "Define the scope of this proposal. Given the motivation stated above,
> > what are the problems that this proposal is addressing and what other items
> > will be considering out of scope, perhaps to be left to a different PIP."
> > # API Changes
> > * "Illustrate all the proposed changes to the API or wire protocol, with
> > examples of all the newly added classes/methods, including Javadoc"
>
> Yes this is important as api and similar changes are what triggers requiring 
> small pips.
>
> > # Implementation
> > * "This should be a detailed description of all the changes that are
> > expected to be made. It should be detailed enough that any developer that
> > is familiar with Pulsar internals would be able to understand all the parts
> > of the code changes for this proposal."
> > * "This should also serve as documentation for any person that is trying to
> > understand or debug the behavior of a certain feature."
> > # Alernatives
> > * "If there are alternatives that were already considered by the authors
> > or, after the discussion, by the community, and were rejected, please list
> > them here along with the reason why they were rejected"
> > # Anything else?
> >
> >
> > === My suggestion
> >
> > # Motivation and Background information
> > * Give a high level explanation on all concepts you will be using
> > throughout this document. For example, if you want to talk about Persistent
> > Subscriptions, explain briefly (1 paragraph) what this is. If you're going
> > to talk about Transaction Buffer, explain briefly what this is. If you're
> > going to change something specific, that goes into a bit more detail about
> > it and how it works. The Litmus test: I can read the design document and
> > understand the problem statement and what you plan to change *without*
> > resorting to a couple of hours of code reading just to start having a high
> > level understanding of the change.
> > * Provide links where possible if a person wants to dig deeper into the
> > background information.
> > * Explain what is the problem you're trying to solve - current situation.
> > * This section is the "Why" of your proposal.
> >
> > # Goals
> > ## Scope
> > * Describe the goals of your proposal, and why it benefits Apache Pulsar
> > ## Out of Scope
> > * Describe what you have decided to keep out of scope, perhaps left for a
> > different PIP/s.
> >
> > # High-level Design
> > * Describe in high level, end-to-end, the solution. This should be a few
> > paragraphs long as a guideline.
> > * Reading this would allow me to understand the solution from a bird's eye
> > view, end to end.
> > * DON'T put all the design in a Google Doc and share the link here, as it
> > won't last the test of time.
>
> +1000! A Google doc hides details and obfuscates the design and motivation. 
> Google docs are not searchable within GitHub.
>
> We should reject all new PIPs that include Google Docs.
>
> Best,
> Dave
>
> >
> >

Re: [DISCUSS] PIP-247: Notifications for partitions update

2023-02-26 Thread Asaf Mesika

 Michael,

I think the current suggested protocol is more optimized compared to
existing regex mechanism:

   - If the broker sends notification and it's lost due network issues,
   you'll only know about it due to the client doing constant polling, using
   its hash to minimize response.
  - In the suggested mechanism, there's no constant polling at all.
  Just think of introducing polling to *all* multiple partition
producers and
  consumers (Which likely all pulsar users). The suggested mechanism has a
  configured timeout allowing it to resend on timeout, but only
when it fails
  as opposed to constant polling mechanisms.
   - Each time meta-update you'll need to run it through regular
   expression, on all topics hosted on the broker, for any given client.
   That's a lot of CPU.
  - Suggested mechanism mainly cares about the count of partitions, so
  it's a lot more efficient.


Xiaoyu,

I don't like the "onTopicExtend" method name. Prefer something more
explicit: onTopicPartitionsCountChanged.

Let's think about the case of endless retries - you know, the client is
acting out, and never acknowledges. Need to to think about it.
60 seconds seems like *a lot* to miss an update. Maybe 5-10 sec?

Can you elaborate on which broker you'll be doing that conversation with?
Partitioned topic are spread across several brokers, so how can one broker
know the count of the partitioned topic?

Maybe we should mark "PartitionsChangedListener" deprecated? Having two
interfaces is confusing.



On Fri, Feb 24, 2023 at 7:29 PM Michael Marshall 
wrote:

> > Just the way to implements partitioned-topic metadata
> > notification mechanism is much like notifications on regex sub changes
>
> Why do we need a separate notification implementation? The regex
> subscription feature is about discovering topics (not subscriptions)
> that match a regular expression. As Joe mentioned, I think we just
> need the client to "subscribe" to a topic notification for
> "-partition-[0-9]+" to eliminate the polling.
>
> Building on PIP 145, the work for this PIP would be in implementing a
> different `TopicsChangedListener` [1] so that the result of an added
> topic is to add a producer/consumer to the new partition.
>
> I support removing polling in our streaming platform, but I'd prefer
> to limit the number of notification systems we implement.
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar/pull/16062
> [1]
> https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PatternMultiTopicsConsumerImpl.java#L169-L175
>
>
>
> On Fri, Feb 24, 2023 at 1:57 AM houxiaoyu  wrote:
> >
> > Hi Joe,
> >
> > When we use PartitionedProducerImpl or MultiTopicsConsumerImpl,  there
> is a
> > poll task to fetch the metadata of the partitioned-topic regularly for
> the
> > number of partitions updated.  This PIP wants to use a
> > notification mechanism to replace the metadata poll task.
> >
> > Just the way to implements partitioned-topic metadata
> > notification mechanism is much like notifications on regex sub changes
> >
> > Joe F  于2023年2月24日周五 13:37写道：
> >
> > > Why is this needed when we have notifications on regex sub changes?
> Aren't
> > > the partition names a well-defined regex?
> > >
> > > Joe
> > >
> > > On Thu, Feb 23, 2023 at 8:52 PM houxiaoyu  wrote:
> > >
> > > > Hi Asaf,
> > > > thanks for your reminder.
> > > >
> > > > ## Changing
> > > > I have updated the following changes to make sure the notification
> > > arrived
> > > > successfully:
> > > > 1. The watch success response `CommandWatchPartitionUpdateSuccess`
> will
> > > > contain all the concerned topics of this watcher
> > > > 2. The notification `CommandPartitionUpdate` will always contain all
> the
> > > > concerned topics of this watcher.
> > > > 3. The notification `CommandPartitionUpdate`contains a monotonically
> > > > increased version.
> > > > 4. A map
> `PartitonUpdateWatcherService#inFlightUpdate > > > Pair>` will keep track of the updating
> > > > 5. A timer will check the updating timeout through `inFlightUpdate`
> > > > 6. The client acks `CommandPartitionUpdateResult` to broker when it
> > > > finishes updating.
> > > >
> > > > ## Details
> > > >
> > > > The following mechanism could make sure the newest notification
> arrived
> > > > successfully, copying the description from GH:
> > > >
> > > > A new class, `org.apache.pulsar.PartitonUpdateWatcherService` will
> keep
> > > > track of watchers and will listen to the changes in the metadata.
> > > Whenever
> > > > a topic partition updates it checks if any watchers should be
> notified
> > > and
> > > > sends an update for all topics the watcher concerns through the
> > > ServerCnx.
> > > > Then we will record this request into a map,
> > > > `PartitonUpdateWatcherService#inFlightUpdate > > Pair > > > long/*timestamp*/>>`.  A timer will check this update timeout through
> > > > inFlightUpdate .

Re: [DISCUSS]Add an internal class to `TransactionBufferStats` to record the snapshot status uniformly.

2023-02-26 Thread Asaf Mesika

On Fri, Feb 24, 2023 at 5:33 AM Xiangying Meng  wrote:

> Hi Asaf,
> > How are those stats exposed to the user?
> IMO, these stats should be exposed to the user.
> The users can use it to check if the transaction snapshot is
> running correctly.
>
>
My question was "how" - prometheus, Pulsar REST API?

I'm asking because this stats doesn't look very "metrics" in term of
structure / naming

> Thanks,
> Xiangying
>
> On Wed, Feb 22, 2023 at 9:06 PM Asaf Mesika  wrote:
>
> > How are those stats exposed to the user?
> >
> >
> > On Mon, Feb 20, 2023 at 6:01 AM Xiangying Meng 
> > wrote:
> >
> > > Hi, Community,
> > > We plan to add an internal class to `TransactionBufferStats` to record
> > the
> > > snapshot status uniformly.
> > > As we all know, the current transaction buffer(TB) filters the messages
> > > sent using the aborted transaction by storing the aborted ID in TB.
> > > Then TB will periodically store these aborted txn IDs in a bookie entry
> > in
> > > the form of snapshots so that TB can recover faster when recovering.
> > > But as more and more people use transactions, we found that in some
> > extreme
> > > cases, a bookie entry may not be able to store all aborted transaction
> > IDs.
> > > So in PIP196 , we
> > > implemented the multiple-snapshot function.
> > > As the transaction buffer snapshot mechanism becomes increasingly
> > complex,
> > > the only information related to the transaction snapshot is
> > > `lastSnapshotTimestamps`; That is not enough, we need to add more info
> to
> > > record the snapshot stats.
> > > So I suggest adding an internal class SnapshotStats to
> > > TransactionBufferStats to record the snapshot status uniformly.
> > >
> > > The modification could be :
> > > ```java
> > > public class TransactionBufferStats {
> > > ...
> > > public long lastSnapshotTimestamps;
> > > ...
> > > }
> > > ```
> > > ```java
> > > public class TransactionBufferStats {
> > > ...
> > > //public long lastSnapshotTimestamps;
> > > ...
> > > public SnapshotStats snapshotStats;
> > >
> > > public static class SnapshotStats {
> > > public long segmentsSize;
> > >
> > > public long unsealedAbortTxnIDs;
> > >
> > >
> > > public long lastSnapshotTimestamps;
> > > }
> > > }
> > >
> > > ```
> > > Thanks.
> > > Xiangying
> > >
> >
>

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

2023-02-26 Thread Asaf Mesika

I'm sorry, but this PIP lacks a lot of background knowledge, so you need to
add IMO for people to understand it. You don't need to explain the entire
pulsar in this PIP, but at the very least a few paragraphs detailing all
you need to know, to put you in context:


   - Start by saying Pulsar as a built-in schema registry inside Pulsar
   broker.
  - Every time the client updates the schema, it uploads it to the
  broker. When that happens, it has a feature which validates if the new
  schema version is compatible with the previous versions. There
are 4 types
  of compatibility: Full, ... (complete and explain each one briefly)
   - Also explain Pulsar Schema registry supports various schema
   protocols:  Avro, protobuf native, ... (complete the rest), each protocol
   has a schema which dictates how to serialize and deserialize the message
   content into typed object.
   - Explain in short what is protobuf native (compare protobuf non-native)
   - Please don't paste code instead of explaining.
  - Explain that protobuf native current validation check is only
  composed of checking the root message name is the same between
the current
  schema version and the new version.
 - Explain briefly what is a root message and its name.
  - Explain the problem (list scenarios) that we have because protobuf
  native schema only supports FULL compatibility validation.


Regarding high level design - as in what you plan to do.
I suggest you add "High Level Design" and in it detail how you plan to
validate, per protobuf version, per compatibility check (backward, forward,
full,...).
I tried reading the implementation - for me , it's all over the place. Can
you please list in order what I wrote above, and list the validation rules
with a good explanation why you validate it like that?

Lastly, one you have all the validation rules clearly stated, you can use
it to document it properly so users can know what validation to expect.

Thanks,

Asaf


On Wed, Feb 22, 2023 at 5:10 PM SiNan Liu  wrote:

> Sorry, my mistake. I removed the code and described the design to improve
> the PROTOBUF_NATIVE schema compatibility checks. You can have a look. 
>
> Asaf Mesika  于2023年2月22日周三 21:16写道：
>
> > I read it but you're almost directly diving into the code - it will take
> me
> > hours just to reverse engineer your design.
> >
> > Can you please include a "High Level Design" section in which you explain
> > how you plan to tackle any issue?
> > If I can read that section and explain to someone else how this will
> work,
> > it means the section is complete.
> >
> > Let's leave the code to the PRs.
> >
> >
> > On Sun, Feb 19, 2023 at 2:59 PM SiNan Liu 
> wrote:
> >
> > > Hi all,
> > >
> > > I made a PIP to discuss: https://github.com/apache/pulsar/issues/19565
> .
> > >
> > > We can talk about the current design here. Especially for the field
> type
> > > change check rules, please give your valuable advice.
> > >
> > > Thanks,
> > > Sinan
> > >
> >
>

Re: [Discuss] PIP-248: Add backlog eviction metric

2023-02-26 Thread Asaf Mesika

Hi,

Pulsar has 2 configurations for the backlog eviction:
> backlogQuotaDefaultLimitBytes and backlogQuotaDefaultLimitSecond, if
> topic backlog reaches the threshold of any item, backlog eviction will be
> triggered.

This seems like default values, not the actual values. Can you please
provide an explanation in the PIP and link to read more:
1. Where do you define the backlog quota exactly? What is the granularity
(subscription?)
2.  Is the backlog quota on by default? If so, what are the default values?

*Notes*
1. When the backlog quota limit is defined in Bytes, and you wish to know
how close a subscription is to its bytes limit, you need to calculate the
backlog size in bytes. From my understanding, there is an accurate
calculation (which is costly in terms of I/O) and there is an estimate of
it. I presume you would want to use the estimated one, is that correct?
The backlog quota itself, uses the accurate or the estimated when it starts
evicting entries (i.e. marking them as acknowledged)?

2. For the backlog limit specifying in time units, there is no estimate, as
it must be calculated all the time (earliest unacknowledged message
distance from now). How do you plan to calculate the current age of the
earliest message without bearing that I/O cost on each metric calculation?

3. In the Goal section, you specify that your goal is to add a "proximity"
metric.
a) You must define that - what is proximity metric exactly? What are its
units? How are you planning to calculate it?
b) Proximity is not a good term IMO. I personally have never seen this term
used in software systems, unless it's in the aviation/space industry. Once
you explain (a) I hope I can help provide alternative names.

4. Maybe we should provide the used quota percentage for both limits,
instead of one per both, since it's easier to act upon the alert when you
need which one triggered it.

5. I didn't understand the "slowest_subscription" label used when
describing the metric label. Can you please provide an explanation?

6. I suggest writing a "High Level Design" section, and add everything you
need to know for this proposal, so I don't need to read the
implementation details below (code).

Thanks,

Asaf

On Wed, Feb 22, 2023 at 4:52 PM 太上玄元道君  wrote:

> Hi all,
>
> I've started a PIP to discuss: PIP-248 Add backlog eviction metric
>
> ### Motivation:
>
> Pulsar has 2 configurations for the backlog eviction:
> `backlogQuotaDefaultLimitBytes` and `backlogQuotaDefaultLimitSecond`, if
> topic backlog reaches the threshold of any item, backlog eviction will be
> triggered.
>
> Before backlog eviction happens, we don't have a metric to monitor how long
> that it can reaches the threshold.
>
> We can provide a progress bar metric to tell users some topics is about to
> trigger backlog eviction. And users can subscribe the alert to schedule
> consumers.
>
> For more details, please read the PIP at
> https://github.com/apache/pulsar/issues/19601
>
> Thanks,
> Tao Jiuming
>

Re: [DISCUSS] Change PIP template

2023-02-26 Thread Dave Fisher

Excellent proposal.

Inline below.

Sent from my iPhone

> On Feb 26, 2023, at 3:19 AM, Asaf Mesika  wrote:
> 
> Hi,
> 
> I would like to suggest two changes I'd like to make to the PIP design
> template:
> 1. Remove the form - just have a markdown template fill the issue body as
> it is created.
> 2. Change the PIP template structure
> 
> == Removing the form
> 
> Today, when you want to submit a PIP, you are required to fill out a form
> with boxes composed of 3-4 lines length.
> It's not good because:
> * It broadcasts to the author: we want a very small PIP, something that
> fits those small boxes.
> * It makes the PIP look like a bug, where you fill out fields.
> * It doesn't allow having H2 headings, only H1 headings, thus limiting the
> structure.
> 
> A PIP is a design essentially, something 1-3 pages long. Thus,
> people take the time to write it down. Preferably, they copy paste the body
> of the PIP issue, and use it to fill in sections.
> 
> My suggestion is to define an issue template using only markdown, without a
> form.
> 
> == Changing PIP Structure
> 
> Today the structure of the PIP doc (pasted below), is missing a section and
> generally aims to jump directly into API changes / code / implementation.
> This results in lots of back and forth emails in an attempt to get the
> following essentials:
> * All required background knowledge to understand the proposal
> * A high level overview of the proposed solution
> * Understanding how this proposal will be monitored
> * What steps exactly I need to take if I revert to the previous version.
> 
> The structure I propose below aims to reduce that friction and get all PIP
> aligned to provide that information.
> 
> === Today's structure
> 
> # Motivation
> * "Explain why this change is needed, what benefits it would bring to
> Apache Pulsar and what problem it's trying to solve."
> # Goal
> * "Define the scope of this proposal. Given the motivation stated above,
> what are the problems that this proposal is addressing and what other items
> will be considering out of scope, perhaps to be left to a different PIP."
> # API Changes
> * "Illustrate all the proposed changes to the API or wire protocol, with
> examples of all the newly added classes/methods, including Javadoc"

Yes this is important as api and similar changes are what triggers requiring 
small pips.

> # Implementation
> * "This should be a detailed description of all the changes that are
> expected to be made. It should be detailed enough that any developer that
> is familiar with Pulsar internals would be able to understand all the parts
> of the code changes for this proposal."
> * "This should also serve as documentation for any person that is trying to
> understand or debug the behavior of a certain feature."
> # Alernatives
> * "If there are alternatives that were already considered by the authors
> or, after the discussion, by the community, and were rejected, please list
> them here along with the reason why they were rejected"
> # Anything else?
> 
> 
> === My suggestion
> 
> # Motivation and Background information
> * Give a high level explanation on all concepts you will be using
> throughout this document. For example, if you want to talk about Persistent
> Subscriptions, explain briefly (1 paragraph) what this is. If you're going
> to talk about Transaction Buffer, explain briefly what this is. If you're
> going to change something specific, that goes into a bit more detail about
> it and how it works. The Litmus test: I can read the design document and
> understand the problem statement and what you plan to change *without*
> resorting to a couple of hours of code reading just to start having a high
> level understanding of the change.
> * Provide links where possible if a person wants to dig deeper into the
> background information.
> * Explain what is the problem you're trying to solve - current situation.
> * This section is the "Why" of your proposal.
> 
> # Goals
> ## Scope
> * Describe the goals of your proposal, and why it benefits Apache Pulsar
> ## Out of Scope
> * Describe what you have decided to keep out of scope, perhaps left for a
> different PIP/s.
> 
> # High-level Design
> * Describe in high level, end-to-end, the solution. This should be a few
> paragraphs long as a guideline.
> * Reading this would allow me to understand the solution from a bird's eye
> view, end to end.
> * DON'T put all the design in a Google Doc and share the link here, as it
> won't last the test of time.

+1000! A Google doc hides details and obfuscates the design and motivation. 
Google docs are not searchable within GitHub.

We should reject all new PIPs that include Google Docs.

Best,
Dave

> 
> # Detailed Design
> * Describe in detail what you plan to do to achieve your high level design
> * It should include the following if applicable:
>  * REST API Changes
>  * Protocol Changes
> 
> # Monitoring
> * Describe exactly what you will add to Pulsar allowing you to
>

Re: Introducer Pulsar admin api to pulsar-client-go

2023-02-26 Thread Guangning E

Another idea is whether it is possible to generate clients for various
languages based on the current pulsar rest swagger files?

Thanks,
Guangning

On Sat, Feb 25, 2023 at 1:44 PM Zixuan Liu  wrote:

> > I would like to suggest creating a new repository called
> "pulsar-admin-go"
>
> +1, I agreed with you.
>
> Thanks,
> Zixuan
>
> ZhangJian He  于2023年2月25日周六 11:05写道：
>
> > Following our previous discussions via email, I would like to suggest
> > creating a new repository called "pulsar-admin-go". This repository will
> > serve as a platform for all of us to contribute our code. Pulsarctl[0] go
> > first.
> >
> > Please let me know your thoughts on this matter.
> >
> > [0] - https://github.com/streamnative/pulsarctl
> >
> > Thanks
> > ZhangJian He
> >
> >
> > On Fri, 17 Feb 2023 at 23:24, ZhangJian He  wrote:
> >
> > > Thank for StreamNative for willing to donate this project. This means
> we
> > > don't have to develop and maintain a set of HTTP code from scratch. My
> > idea
> > > aligns with Yunze's, and separating it into a standalone
> pulsar-admin-go
> > > project would be better. The **pulsarctl** repo contains bookkeeper
> http
> > > call too. Maybe we can have a project bookkeeper-admin-go ?（it's a
> liitle
> > > going off-topic ）
> > >
> > > Thanks
> > > ZhangJian He
> > >
> > >
> > > On Fri, 17 Feb 2023 at 20:29, PengHui Li 
> > wrote:
> > >
> > >> Hi Yunze,
> > >>
> > >> Yes, we can split it.
> > >> Both one repo with two modules or two repos works for me.
> > >>
> > >> The pulsarctl already have the admin API and CLI.
> > >> So I think we don’t need to develop another one.
> > >>
> > >> Best,
> > >> Penghui
> > >>
> > >> > On Feb 17, 2023, at 17:44, Yunze Xu 
> > >> wrote:
> > >> >
> > >> > Hi PengHui,
> > >> >
> > >> > Now I changed my mind a bit. Even if the pulsarctl was contributed
> to
> > >> > the Apache Foundation, I think we should also avoid adding it as the
> > >> > dependency. What we need is an API layer but not the CLI, while
> > >> > pulsarctl couples the API and CLI.
> > >> >
> > >> > At the moment, my expectation is:
> > >> > 1. Use a separate repo (e.g. pulsar-admin-go) to implement the admin
> > >> > APIs in Golang.
> > >> > 2. Depend this new repo in pulsarctl.
> > >> >
> > >> > Then we will have three Go projects:
> > >> > - pulsar-client-go: The Pulsar Go client APIs
> > >> > - pulsar-admin-go: The Pulsar Go admin APIs
> > >> > - pulsarctl: The admin CLI tool written in Go
> > >> >
> > >> > Thanks,
> > >> > Yunze
> > >> >
> > >> > On Fri, Feb 17, 2023 at 4:22 PM PengHui Li 
> > wrote:
> > >> >>
> > >> >> I checked with Sijie today.
> > >> >> StreamNative can contribute the pulsarctl project to Apache
> > Foundation.
> > >> >>
> > >> >> Regards,
> > >> >> Penghui
> > >> >>
> > >> >> On Fri, Feb 17, 2023 at 4:02 PM Enrico Olivelli <
> eolive...@gmail.com
> > >
> > >> wrote:
> > >> >>
> > >> >>> I agree to add an admin API to the go client, this would be very
> > >> helpful.
> > >> >>>
> > >> >>> Il giorno ven 17 feb 2023 alle ore 08:44 Zixuan Liu
> > >> >>>  ha scritto:
> > >> 
> > >>  Hi Zhangjian,
> > >> 
> > >>  This is a good idea to write the admin client by golang, but I
> > don't
> > >>  suggest add the admin features to pulsar-go-client, it's better
> to
> > >> use a
> > >>  new repository to do that to  separate dependencies.
> > >> 
> > >>  BTW, StreamNative has a pulsarctl [0] tool, which includes the
> > admin
> > >> api.
> > >> 
> > >> >> It's better to reuse existing code rather than reinventing the
> > >> wheel.
> > >> 
> > >>  I aggred this point. If possible, we can integrate the pulsarctl
> to
> > >> this
> > >>  new project.
> > >> >>>
> > >> >>> We are talking about adding a client that calls a
> > >> >>> well defined and maintained REST API.
> > >> >>> It is better to have our implementation and not rely on third
> > parties
> > >> >>> when it is possible.
> > >> >>> If there is a security issue in pulsarctl, how would we handle
> that
> > ?
> > >> >>> Also the Pulsar community maintains the Pulsar API and this is the
> > >> >>> place where it is easier to keep the client up-to-date with the
> new
> > >> >>> APIs that we will develop,
> > >> >>> we can't wait for a third party project to implement our own APIs
> > and
> > >> >>> wait for an upgrade (even if it is OSS, we cannot cut releases or
> > have
> > >> >>> control over the release cycle)
> > >> >>>
> > >> >>>
> > >> >>> Enrico
> > >> >>>
> > >> >>>
> > >> >>>
> > >> 
> > >>  [0] - https://github.com/streamnative/pulsarctl
> > >> 
> > >>  Thanks,
> > >>  Zixuan
> > >> 
> > >> 
> > >>  ZhangJian He  于2023年2月17日周五 13:47写道：
> > >> 
> > >> > Separating dependencies is better. For example, I think
> > >> >>> Pulsar-admin-go can
> > >> > only have golang standard tls and http dependencies.
> > >> > But it seems impossible to have two go modules when publishing
> > >> packages
> > >> > using github.
>

Re: [DISCUSS] Change PIP template

2023-02-26 Thread Girish Sharma

Good proposal Asaf.
I've also wondered why the PIP creation and discussion process is so
separated. The PIP discussion and voting starts off as a GitHub issue, but
all of its discussion happens here on the mailing list. Is there scope of
improvement in that process as well?

Regards

On Sun, Feb 26, 2023 at 6:16 PM tison  wrote:

> Hi Asaf,
>
> I agree that, generally, a PIP is written as a whole and paste as the body.
> So +1 for your proposal.
>
> Additionally, I'm thinking of moving the doc of procedure (wiki/PIP.md) to
> the contributions guide and use the new markdown template to supersede the
> wiki/PIP-template.md. Then we don't need to hold the wiki folder.
>
> It can be an extended version to your proposal, so let's keep on your
> proposal in this thread. Just for your reference.
>
> Best,
> tison.
>
>
> Asaf Mesika  于2023年2月26日周日 19:18写道：
>
> > Hi,
> >
> > I would like to suggest two changes I'd like to make to the PIP design
> > template:
> > 1. Remove the form - just have a markdown template fill the issue body as
> > it is created.
> > 2. Change the PIP template structure
> >
> > == Removing the form
> >
> > Today, when you want to submit a PIP, you are required to fill out a form
> > with boxes composed of 3-4 lines length.
> > It's not good because:
> > * It broadcasts to the author: we want a very small PIP, something that
> > fits those small boxes.
> > * It makes the PIP look like a bug, where you fill out fields.
> > * It doesn't allow having H2 headings, only H1 headings, thus limiting
> the
> > structure.
> >
> > A PIP is a design essentially, something 1-3 pages long. Thus,
> > people take the time to write it down. Preferably, they copy paste the
> body
> > of the PIP issue, and use it to fill in sections.
> >
> > My suggestion is to define an issue template using only markdown,
> without a
> > form.
> >
> > == Changing PIP Structure
> >
> > Today the structure of the PIP doc (pasted below), is missing a section
> and
> > generally aims to jump directly into API changes / code / implementation.
> > This results in lots of back and forth emails in an attempt to get the
> > following essentials:
> > * All required background knowledge to understand the proposal
> > * A high level overview of the proposed solution
> > * Understanding how this proposal will be monitored
> > * What steps exactly I need to take if I revert to the previous version.
> >
> > The structure I propose below aims to reduce that friction and get all
> PIP
> > aligned to provide that information.
> >
> > === Today's structure
> >
> > # Motivation
> > * "Explain why this change is needed, what benefits it would bring to
> > Apache Pulsar and what problem it's trying to solve."
> > # Goal
> > * "Define the scope of this proposal. Given the motivation stated above,
> > what are the problems that this proposal is addressing and what other
> items
> > will be considering out of scope, perhaps to be left to a different PIP."
> > # API Changes
> > * "Illustrate all the proposed changes to the API or wire protocol, with
> > examples of all the newly added classes/methods, including Javadoc"
> > # Implementation
> > * "This should be a detailed description of all the changes that are
> > expected to be made. It should be detailed enough that any developer that
> > is familiar with Pulsar internals would be able to understand all the
> parts
> > of the code changes for this proposal."
> > * "This should also serve as documentation for any person that is trying
> to
> > understand or debug the behavior of a certain feature."
> > # Alernatives
> > * "If there are alternatives that were already considered by the authors
> > or, after the discussion, by the community, and were rejected, please
> list
> > them here along with the reason why they were rejected"
> > # Anything else?
> >
> >
> > === My suggestion
> >
> > # Motivation and Background information
> > * Give a high level explanation on all concepts you will be using
> > throughout this document. For example, if you want to talk about
> Persistent
> > Subscriptions, explain briefly (1 paragraph) what this is. If you're
> going
> > to talk about Transaction Buffer, explain briefly what this is. If you're
> > going to change something specific, that goes into a bit more detail
> about
> > it and how it works. The Litmus test: I can read the design document and
> > understand the problem statement and what you plan to change *without*
> > resorting to a couple of hours of code reading just to start having a
> high
> > level understanding of the change.
> > * Provide links where possible if a person wants to dig deeper into the
> > background information.
> > * Explain what is the problem you're trying to solve - current situation.
> > * This section is the "Why" of your proposal.
> >
> > # Goals
> > ## Scope
> > * Describe the goals of your proposal, and why it benefits Apache Pulsar
> > ## Out of Scope
> > * Describe what you have decided to keep out of scope, perhaps

Re: [questions] Which jakartaEE version we plan for pulsar 3.0.0?

2023-02-26 Thread Enrico Olivelli

+1 to upgrade.

IIUC the main pain point is for Custom Servlet developers who will
have to migrate to the jakarta.xxx packages
is this correct ?

Enrico

Il giorno sab 25 feb 2023 alle ore 06:46 Zixuan Liu
 ha scritto:
>
> Sounds greate that migrating to jersey 3.x from jersey 2.x.
>
> Thanks,
> Zixuan
>
> ZhangJian He  于2023年2月25日周六 11:01写道：
>
> > Hello, community. I want to know Which jakartaEE version we plan for pulsar
> > 3.0.0. Now we use jersey 2.x, which uses Jakarta EE8.
> >
> > Maybe we will need to update the jersey.
> > Eclipse Jersey (eclipse-ee4j.github.io)
> > 
> > 2.x is for Jakarta EE8. 3.0.x is for Jakarta EE9 and 3.1.x is for Jakarta
> > EE10.
> >
> > Thank you for your time and consideration.
> >
> > Thanks
> > ZhangJian He
> >

Re: [DISCUSS] Change PIP template

2023-02-26 Thread tison

Hi Asaf,

I agree that, generally, a PIP is written as a whole and paste as the body.
So +1 for your proposal.

Additionally, I'm thinking of moving the doc of procedure (wiki/PIP.md) to
the contributions guide and use the new markdown template to supersede the
wiki/PIP-template.md. Then we don't need to hold the wiki folder.

It can be an extended version to your proposal, so let's keep on your
proposal in this thread. Just for your reference.

Best,
tison.


Asaf Mesika  于2023年2月26日周日 19:18写道：

> Hi,
>
> I would like to suggest two changes I'd like to make to the PIP design
> template:
> 1. Remove the form - just have a markdown template fill the issue body as
> it is created.
> 2. Change the PIP template structure
>
> == Removing the form
>
> Today, when you want to submit a PIP, you are required to fill out a form
> with boxes composed of 3-4 lines length.
> It's not good because:
> * It broadcasts to the author: we want a very small PIP, something that
> fits those small boxes.
> * It makes the PIP look like a bug, where you fill out fields.
> * It doesn't allow having H2 headings, only H1 headings, thus limiting the
> structure.
>
> A PIP is a design essentially, something 1-3 pages long. Thus,
> people take the time to write it down. Preferably, they copy paste the body
> of the PIP issue, and use it to fill in sections.
>
> My suggestion is to define an issue template using only markdown, without a
> form.
>
> == Changing PIP Structure
>
> Today the structure of the PIP doc (pasted below), is missing a section and
> generally aims to jump directly into API changes / code / implementation.
> This results in lots of back and forth emails in an attempt to get the
> following essentials:
> * All required background knowledge to understand the proposal
> * A high level overview of the proposed solution
> * Understanding how this proposal will be monitored
> * What steps exactly I need to take if I revert to the previous version.
>
> The structure I propose below aims to reduce that friction and get all PIP
> aligned to provide that information.
>
> === Today's structure
>
> # Motivation
> * "Explain why this change is needed, what benefits it would bring to
> Apache Pulsar and what problem it's trying to solve."
> # Goal
> * "Define the scope of this proposal. Given the motivation stated above,
> what are the problems that this proposal is addressing and what other items
> will be considering out of scope, perhaps to be left to a different PIP."
> # API Changes
> * "Illustrate all the proposed changes to the API or wire protocol, with
> examples of all the newly added classes/methods, including Javadoc"
> # Implementation
> * "This should be a detailed description of all the changes that are
> expected to be made. It should be detailed enough that any developer that
> is familiar with Pulsar internals would be able to understand all the parts
> of the code changes for this proposal."
> * "This should also serve as documentation for any person that is trying to
> understand or debug the behavior of a certain feature."
> # Alernatives
> * "If there are alternatives that were already considered by the authors
> or, after the discussion, by the community, and were rejected, please list
> them here along with the reason why they were rejected"
> # Anything else?
>
>
> === My suggestion
>
> # Motivation and Background information
> * Give a high level explanation on all concepts you will be using
> throughout this document. For example, if you want to talk about Persistent
> Subscriptions, explain briefly (1 paragraph) what this is. If you're going
> to talk about Transaction Buffer, explain briefly what this is. If you're
> going to change something specific, that goes into a bit more detail about
> it and how it works. The Litmus test: I can read the design document and
> understand the problem statement and what you plan to change *without*
> resorting to a couple of hours of code reading just to start having a high
> level understanding of the change.
> * Provide links where possible if a person wants to dig deeper into the
> background information.
> * Explain what is the problem you're trying to solve - current situation.
> * This section is the "Why" of your proposal.
>
> # Goals
> ## Scope
> * Describe the goals of your proposal, and why it benefits Apache Pulsar
> ## Out of Scope
> * Describe what you have decided to keep out of scope, perhaps left for a
> different PIP/s.
>
> # High-level Design
> * Describe in high level, end-to-end, the solution. This should be a few
> paragraphs long as a guideline.
> * Reading this would allow me to understand the solution from a bird's eye
> view, end to end.
> * DON'T put all the design in a Google Doc and share the link here, as it
> won't last the test of time.
>
> # Detailed Design
> * Describe in detail what you plan to do to achieve your high level design
> * It should include the following if applicable:
>   * REST API Changes
>   * Protocol Changes
>
>

[DISCUSS] Change PIP template

2023-02-26 Thread Asaf Mesika

Hi,

I would like to suggest two changes I'd like to make to the PIP design
template:
1. Remove the form - just have a markdown template fill the issue body as
it is created.
2. Change the PIP template structure

== Removing the form

Today, when you want to submit a PIP, you are required to fill out a form
with boxes composed of 3-4 lines length.
It's not good because:
* It broadcasts to the author: we want a very small PIP, something that
fits those small boxes.
* It makes the PIP look like a bug, where you fill out fields.
* It doesn't allow having H2 headings, only H1 headings, thus limiting the
structure.

A PIP is a design essentially, something 1-3 pages long. Thus,
people take the time to write it down. Preferably, they copy paste the body
of the PIP issue, and use it to fill in sections.

My suggestion is to define an issue template using only markdown, without a
form.

== Changing PIP Structure

Today the structure of the PIP doc (pasted below), is missing a section and
generally aims to jump directly into API changes / code / implementation.
This results in lots of back and forth emails in an attempt to get the
following essentials:
* All required background knowledge to understand the proposal
* A high level overview of the proposed solution
* Understanding how this proposal will be monitored
* What steps exactly I need to take if I revert to the previous version.

The structure I propose below aims to reduce that friction and get all PIP
aligned to provide that information.

=== Today's structure

# Motivation
* "Explain why this change is needed, what benefits it would bring to
Apache Pulsar and what problem it's trying to solve."
# Goal
* "Define the scope of this proposal. Given the motivation stated above,
what are the problems that this proposal is addressing and what other items
will be considering out of scope, perhaps to be left to a different PIP."
# API Changes
* "Illustrate all the proposed changes to the API or wire protocol, with
examples of all the newly added classes/methods, including Javadoc"
# Implementation
* "This should be a detailed description of all the changes that are
expected to be made. It should be detailed enough that any developer that
is familiar with Pulsar internals would be able to understand all the parts
of the code changes for this proposal."
* "This should also serve as documentation for any person that is trying to
understand or debug the behavior of a certain feature."
# Alernatives
* "If there are alternatives that were already considered by the authors
or, after the discussion, by the community, and were rejected, please list
them here along with the reason why they were rejected"
# Anything else?


=== My suggestion

# Motivation and Background information
* Give a high level explanation on all concepts you will be using
throughout this document. For example, if you want to talk about Persistent
Subscriptions, explain briefly (1 paragraph) what this is. If you're going
to talk about Transaction Buffer, explain briefly what this is. If you're
going to change something specific, that goes into a bit more detail about
it and how it works. The Litmus test: I can read the design document and
understand the problem statement and what you plan to change *without*
resorting to a couple of hours of code reading just to start having a high
level understanding of the change.
* Provide links where possible if a person wants to dig deeper into the
background information.
* Explain what is the problem you're trying to solve - current situation.
* This section is the "Why" of your proposal.

# Goals
## Scope
* Describe the goals of your proposal, and why it benefits Apache Pulsar
## Out of Scope
* Describe what you have decided to keep out of scope, perhaps left for a
different PIP/s.

# High-level Design
* Describe in high level, end-to-end, the solution. This should be a few
paragraphs long as a guideline.
* Reading this would allow me to understand the solution from a bird's eye
view, end to end.
* DON'T put all the design in a Google Doc and share the link here, as it
won't last the test of time.

# Detailed Design
* Describe in detail what you plan to do to achieve your high level design
* It should include the following if applicable:
  * REST API Changes
  * Protocol Changes

# Monitoring
* Describe exactly what you will add to Pulsar allowing you to
monitor/observe this proposal?
  * If those are metrics, provide the names, description, labels and units
  * Explain what constitutes abnormal that I should pay attention to

# Backward Compatibility
* Describe exact instructions if someone needs to revert from a version
containing it to a previous version

# Alternatives
* Describe alternative design decisions and why you have not opted for them

# General notes
* Any general notes you wish to make

# Links (Updated afterwards)
* Mailing List discussion thread:
* Mailing List voting thread:

==
Would love to hear what you think about it, before opening a PR

Re: [ANNOUNCE] New committer: Yuri Mizushima

Re: [DISCUSS] Change PIP template

Re: [ANNOUNCE] New committer: Yuri Mizushima

[ANNOUNCE] New committer: Yuri Mizushima

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

Re: [DISCUSS] PIP-247: Notifications for partitions update

Re: [questions] Which jakartaEE version we plan for pulsar 3.0.0?

Re: Introducer Pulsar admin api to pulsar-client-go

Re: [questions] Which jakartaEE version we plan for pulsar 3.0.0?

Re: [DISCUSS] PIP-247: Notifications for partitions update

Re: [DISCUSS] PIP-247: Notifications for partitions update

Re: [DISCUSS] Change PIP template

Re: [DISCUSS] PIP-247: Notifications for partitions update

Re: [DISCUSS]Add an internal class to `TransactionBufferStats` to record the snapshot status uniformly.

Re: [DISCUSS] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf

Re: [Discuss] PIP-248: Add backlog eviction metric

Re: [DISCUSS] Change PIP template

Re: Introducer Pulsar admin api to pulsar-client-go

Re: [DISCUSS] Change PIP template

Re: [questions] Which jakartaEE version we plan for pulsar 3.0.0?

Re: [DISCUSS] Change PIP template

[DISCUSS] Change PIP template

22 matches

Site Navigation

Mail list logo

Footer information