Re: [VOTE] PIP-268: Add support of topic stats/stats-internal using

[PengHui Li]

Hi Rajan,

Thanks for the explanation

> This feature helps them to avoid multiple different extra
efforts

If I understand correctly. You want to say users don't want to
add the pulsar-admin dependency or the the cluster don't want
to expose the REST API to external (not the cluster admin or
tenant admin)?

If so, I think it could be a good reason for introducing binary
protocol support here. For the security sensitive users like financial
application.

Usually they will try to reduce the dependencies (less dependencies, less
CVEs)
and the exposed service endpoints. For example, the flink connector
also have pulsar-admin dependency but some of the users want to
remove it.

I want to say that from the perspective of improving performance,
it may not be more convincing than the above reason.

Thanks,
Penghui


On Mon, Jun 26, 2023 at 3:37 PM Rajan Dhabalia  wrote:

> > I do not deny that binary protocol has performance advantages. But maybe
> the bottleneck is not the protocol level for now.
>
> Well, sure. We had serious issue with performance in past over http but
> this feature we would mainly like to introduce it now for the applications
> to enhance api user accessibility experience where we have these multiple
> usecases where applications with large number of topics and high fanout
> consumers would like to fetch stats and stats-internal to retrieve various
> metadata for application startup and managing application state based on
> managed-ledge states. You can think of producer/consumer stats api which is
> used by many usecases in different scenarios of monitoring or state
> management. This feature helps them to avoid multiple different extra
> efforts and performance considerations, which helps to give clean and easy
> experience for their application.
> I am open to hear about alternative of json string and response schema
> definition but keeping similar response-schema as admin-api response for
> stats/stats-internal helps to give consistent view of stats across all APIs
> and transferring json format helps to skip transformation stats definition
> and we don't have to make any wire protocol changes whenever we add any new
> field or state in response which makes this protocol response agnostic and
> do not require maintenance for any changes in stats. So, we can consider
> any suggestion but we have multiple usecaes which need this API and we
> should implement it in a way where it can provide a consistent definition
> view across other APIs without maintenance efforts while making any stat
> response changes.
>
> Thanks,
> Rajan
>
> On Sat, Jun 24, 2023 at 5:51 PM PengHui Li  wrote:
>
> > > >> Main reasons are performance (you will see improvement due to http
> > overhead compare to tcp, limited threads on http-server side and
> definitely
> > performance compare to netty-tcp vs http-jetty server), user feasibility
> > (same reason why we have producer/consumer stats using binary protocol
> > where user can get the stats from the same producer/consumer entity
> without
> > having additional client creation for the stats), scale (allowing large
> > number of requests over binary protocol compare to http is more
> scalable).
> >
> > I do not deny that binary protocol has performance advantages.
> > But maybe the bottleneck is not the protocol level for now.
> >
> > IMO, we should have a clear performance expectation for getting the topic
> > stats
> > If you want to achieve 10k of requests per second.
> > Maybe the protocol is not the bottleneck. Of course, if you want to reach
> > 100k or
> > even millions of requests on a single server per second.
> > Exactly, the binary protocol will help. I'm curious whether it is a real
> > case.
> > Unless you have to query topic stats every second and the broker owned
> > 100k topics.
> >
> > And I also saw some benchmark results for the HTTP servers.
> >
> >
> https://cwiki.apache.org/confluence/display/HTTPCOMPONENTS/HttpCoreBenchmark
> > It's not refreshed, but it's instructive. Try other HTTP servers should
> > also be considered.
> >
> > The binary protocol for topic stats is an option. But it also has
> > disadvantages and negative effects.
> >
> > - Make the binary protocol more complicated
> > - All the clients need to catch up with the new protocol
> >
> > > >> SNI is layer-4 tcp layer protocol and does not support over the
> HTTP.
> >
> > But you can just have an HTTP proxy to handle the redirect?
> > Why SNI is required for getting topic stats?
> > I'm not sure about the whole story here.
> >
> > > Well JSON would be the better option to avoid data conversion and copy
> > for
> > complex stats/internal-stats complex data-structures at broker and client
> > side, maintaining consistency between stats/internal-stats pojo-schema,
> and
> > accessing stats with consistent pojo definition.
> >
> > This is from the perspective of developers who are already very familiar
> > with Pulsar.
> > If you don't know it, just want to create a client based 

Re: [VOTE] PIP-278: Support pluggable topic compaction service

[PengHui Li]

+1 (binding)

Regards,
Penghui

On Mon, Jun 26, 2023 at 12:19 PM Yunze Xu  wrote:

> I have left some comments in the proposal so please hold this vote
> until the comments are addressed.
>
> Thanks,
> Yunze
>
> On Mon, Jun 26, 2023 at 10:47 AM  wrote:
> >
> > +1 (binding)
> >
> > Best,
> > Mattison
> > On 26 Jun 2023 at 10:01 +0800, Cong Zhao , wrote:
> > > Hello, community:
> > >
> > > This thread is to start a vote for PIP-278: Support pluggable topic
> > > compaction service.
> > >
> > > Discussion thread:
> > > https://lists.apache.org/thread/ox2bot3p9j9fydqkw3v5gt5twc8jslvd
> > >
> > > PIP PR: https://github.com/apache/pulsar/pull/20624
> > >
> > > Sincerely
> > > Cong Zhao
>

Re: [VOTE] PIP-276: Add metric `pulsar_topic_load_times

[Hang Chen]

+1 (binding)

Thanks,
Hang

Yunze Xu  于2023年6月26日周一 19:59写道：
>
> +1 (binding)
>
> Thanks,
> Yunze
>
>
>
>
> > On Jun 20, 2023, at 10:53, mattisonc...@gmail.com wrote:
> >
> > +1(binding)
> >
> > Best,
> > Mattison
> > On 20 Jun 2023 at 10:45 +0800, PengHui Li , wrote:
> >> +1 (binding)
> >>
> >> Thanks,
> >> Penghui
> >>
> >> On Tue, Jun 20, 2023 at 10:40 AM Yubiao Feng
> >>  wrote:
> >>
> >>> Voting +1 (non-binding)
> >>>
> >>> Thanks
> >>> Yubiao Feng
> >>>
> >>> On Mon, Jun 19, 2023 at 5:21 PM Asaf Mesika  wrote:
> >>>
> > Voting +1 (non-binding)
> >
> > On Fri, Jun 16, 2023 at 12:23 PM guo jiwei  wrote:
> >
> >>> @Asaf Thanks, I have addressed the comment.
> >>>
> >>> Regards
> >>> Jiwei Guo (Tboy)
> >>>
> >>>
> >>> On Fri, Jun 16, 2023 at 3:55 AM Asaf Mesika 
> > wrote:
> >>>
> > -1 (non-binding)
> >
> > I'm perfectly ok with the idea; just please fix the document. It
> >>> looks
> >>> too
> > messy. Even 1 paragraph changes can look neat and clean.
> > I left notes in the draft PR you opened for the pip.
> >
> > I'll change my non-binding vote once that's done.
> >
> > On Thu, Jun 15, 2023 at 11:07 AM guo jiwei 
> > wrote:
> >
> >>> Hi, community:
> >>> The metrics are all started with `pulsar_`, so that both users
> > and
> >>> operators can quickly find the metrics of the entire system 
> >>> through
> >>> this prefix. However, due to some other reasons, it was found that
> >>> `topic_load_times` was missing the prefix, so want to get it 
> >>> right.
> >>> In the master branch :
> >>> * `pulsar_topic_load_times`: Add this new metric which has the
> >>> same
> >>> meaning as `topic_load_times`
> >>> * `topic_load_times`: Mark this metric as deprecated and
> >>> remove
> >>> it
> > in
> >>> the next version
> >>>
> >>> PIP: https://github.com/apache/pulsar/pull/20518
> >>>
> >>> Regards
> >>> Jiwei Guo (Tboy)
> >>>
> >
> >>>
> >
> >>>
>

Re: [VOTE] PIP-276: Add metric `pulsar_topic_load_times

[Yunze Xu]

+1 (binding)

Thanks,
Yunze




> On Jun 20, 2023, at 10:53, mattisonc...@gmail.com wrote:
> 
> +1(binding)
> 
> Best,
> Mattison
> On 20 Jun 2023 at 10:45 +0800, PengHui Li , wrote:
>> +1 (binding)
>> 
>> Thanks,
>> Penghui
>> 
>> On Tue, Jun 20, 2023 at 10:40 AM Yubiao Feng
>>  wrote:
>> 
>>> Voting +1 (non-binding)
>>> 
>>> Thanks
>>> Yubiao Feng
>>> 
>>> On Mon, Jun 19, 2023 at 5:21 PM Asaf Mesika  wrote:
>>> 
> Voting +1 (non-binding)
> 
> On Fri, Jun 16, 2023 at 12:23 PM guo jiwei  wrote:
> 
>>> @Asaf Thanks, I have addressed the comment.
>>> 
>>> Regards
>>> Jiwei Guo (Tboy)
>>> 
>>> 
>>> On Fri, Jun 16, 2023 at 3:55 AM Asaf Mesika 
> wrote:
>>> 
> -1 (non-binding)
> 
> I'm perfectly ok with the idea; just please fix the document. It
>>> looks
>>> too
> messy. Even 1 paragraph changes can look neat and clean.
> I left notes in the draft PR you opened for the pip.
> 
> I'll change my non-binding vote once that's done.
> 
> On Thu, Jun 15, 2023 at 11:07 AM guo jiwei 
> wrote:
> 
>>> Hi, community:
>>> The metrics are all started with `pulsar_`, so that both users
> and
>>> operators can quickly find the metrics of the entire system through
>>> this prefix. However, due to some other reasons, it was found that
>>> `topic_load_times` was missing the prefix, so want to get it right.
>>> In the master branch :
>>> * `pulsar_topic_load_times`: Add this new metric which has the
>>> same
>>> meaning as `topic_load_times`
>>> * `topic_load_times`: Mark this metric as deprecated and
>>> remove
>>> it
> in
>>> the next version
>>> 
>>> PIP: https://github.com/apache/pulsar/pull/20518
>>> 
>>> Regards
>>> Jiwei Guo (Tboy)
>>> 
> 
>>> 
> 
>>> 

Re: [DISCUSS] Apache Pulsar 2.10.5 release

[Enrico Olivelli]

Xiangying,


Il giorno lun 26 giu 2023 alle ore 11:47 Xiangying Meng
 ha scritto:
>
> Hello, community:
>It has been more than 2 months since the release of 2.10.4. During this
> period, we have 64 fixes.
> https://github.com/apache/pulsar/compare/v2.10.4...branch-2.10
>I suggest releasing  2.10.5. If you have no comments, I will check the
> existing PR that needs to cherry-pick to the 2.10 branch.

I agree

Enrico

>If you have a PR that needs to cherry-pick, please leave a message here.
>
> Regards
> Xiangying

[DISCUSS] Apache Pulsar 2.10.5 release

[Xiangying Meng]

Hello, community:
   It has been more than 2 months since the release of 2.10.4. During this
period, we have 64 fixes.
https://github.com/apache/pulsar/compare/v2.10.4...branch-2.10
   I suggest releasing  2.10.5. If you have no comments, I will check the
existing PR that needs to cherry-pick to the 2.10 branch.
   If you have a PR that needs to cherry-pick, please leave a message here.

Regards
Xiangying

Re: [DISCUSS] About cherry-picking `Use Ubuntu 22.04 for Pulsar images (#20475)`

[Enrico Olivelli]

My understanding is that Ubuntu 20 is EOL.

I agree that we should cherry-pick that change (without upgrading the
Python client) to 2.10, 2.11 and also to 3.0

Enrico

Il giorno lun 26 giu 2023 alle ore 11:10 guo jiwei
 ha scritto:
>
> Hello community:
>For this patch Use Ubuntu 22.04 for Pulsar images
> , we need some ideas about
> cherry-picking to other branches.
>
>
> Regards
> Jiwei Guo (Tboy)

[DISCUSS] About cherry-picking `Use Ubuntu 22.04 for Pulsar images (#20475)`

[guo jiwei]

Hello community:
   For this patch Use Ubuntu 22.04 for Pulsar images
, we need some ideas about
cherry-picking to other branches.


Regards
Jiwei Guo (Tboy)

Re: [DISCUSS] PIP-279: Reformat property in generateResponseWithEntry

[houxiaoyu]

+1 non-binding

Xiaoyu Hou

Enrico Olivelli  于2023年6月26日周一 14:57写道：

> +1 (binding)
>
> Enrico
>
> Il giorno lun 26 giu 2023 alle ore 07:53 guo jiwei
>  ha scritto:
> >
> > +1 (binding)
> >
> > This is a bug and we need this fix.
> >
> >
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
> >
> > On Wed, Jun 21, 2023 at 11:23 AM steven lu 
> wrote:
> >
> > > # Motivation
> > >
> > > reformat property,for a http header name cannot contain the following
> > > prohibited characters: =,;: \t\r\n\v\f
> > >
> > > for example:
> > > {"city=shanghai":"tag"}
> > > when we run `bin/pulsar-admin topics get-message-by-id `, it will
> > > throw exception, the exception is:
> > > `Reason: java.util.concurrent.CompletionException:
> > >
> > >
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> > > Could not complete the operation. Number of retries has been
> > > exhausted. Failed reason: a header name cannot contain the following
> > > prohibited characters: =,;: \t\r\n\v\f: =`
> > >
> > >  > > src="
> > >
> https://github.com/StevenLuMT/pulsar/assets/42990025/973d95b9-4ac2-4977-b160-162c4b53a613
> > > ">
> > >
> > > # High Level Design
> > >
> > > In master branch,
> > > in an http
> > >
> request:getMessageById("/{tenant}/{namespace}/{topic}/ledger/{ledgerId}/entry/{entryId}"),
> > > replace `"X-Pulsar-PROPERTY-" + msgProperties.getKey()` with
> > > `"X-Pulsar-PROPERTY"`
> > >
> > > After release-3.1.0, this feature begins to take effect.
> > >
>

Re: [VOTE] PIP-268: Add support of topic stats/stats-internal using

[Rajan Dhabalia]

> I do not deny that binary protocol has performance advantages. But maybe
the bottleneck is not the protocol level for now.

Well, sure. We had serious issue with performance in past over http but
this feature we would mainly like to introduce it now for the applications
to enhance api user accessibility experience where we have these multiple
usecases where applications with large number of topics and high fanout
consumers would like to fetch stats and stats-internal to retrieve various
metadata for application startup and managing application state based on
managed-ledge states. You can think of producer/consumer stats api which is
used by many usecases in different scenarios of monitoring or state
management. This feature helps them to avoid multiple different extra
efforts and performance considerations, which helps to give clean and easy
experience for their application.
I am open to hear about alternative of json string and response schema
definition but keeping similar response-schema as admin-api response for
stats/stats-internal helps to give consistent view of stats across all APIs
and transferring json format helps to skip transformation stats definition
and we don't have to make any wire protocol changes whenever we add any new
field or state in response which makes this protocol response agnostic and
do not require maintenance for any changes in stats. So, we can consider
any suggestion but we have multiple usecaes which need this API and we
should implement it in a way where it can provide a consistent definition
view across other APIs without maintenance efforts while making any stat
response changes.

Thanks,
Rajan

On Sat, Jun 24, 2023 at 5:51 PM PengHui Li  wrote:

> > >> Main reasons are performance (you will see improvement due to http
> overhead compare to tcp, limited threads on http-server side and definitely
> performance compare to netty-tcp vs http-jetty server), user feasibility
> (same reason why we have producer/consumer stats using binary protocol
> where user can get the stats from the same producer/consumer entity without
> having additional client creation for the stats), scale (allowing large
> number of requests over binary protocol compare to http is more scalable).
>
> I do not deny that binary protocol has performance advantages.
> But maybe the bottleneck is not the protocol level for now.
>
> IMO, we should have a clear performance expectation for getting the topic
> stats
> If you want to achieve 10k of requests per second.
> Maybe the protocol is not the bottleneck. Of course, if you want to reach
> 100k or
> even millions of requests on a single server per second.
> Exactly, the binary protocol will help. I'm curious whether it is a real
> case.
> Unless you have to query topic stats every second and the broker owned
> 100k topics.
>
> And I also saw some benchmark results for the HTTP servers.
>
> https://cwiki.apache.org/confluence/display/HTTPCOMPONENTS/HttpCoreBenchmark
> It's not refreshed, but it's instructive. Try other HTTP servers should
> also be considered.
>
> The binary protocol for topic stats is an option. But it also has
> disadvantages and negative effects.
>
> - Make the binary protocol more complicated
> - All the clients need to catch up with the new protocol
>
> > >> SNI is layer-4 tcp layer protocol and does not support over the HTTP.
>
> But you can just have an HTTP proxy to handle the redirect?
> Why SNI is required for getting topic stats?
> I'm not sure about the whole story here.
>
> > Well JSON would be the better option to avoid data conversion and copy
> for
> complex stats/internal-stats complex data-structures at broker and client
> side, maintaining consistency between stats/internal-stats pojo-schema, and
> accessing stats with consistent pojo definition.
>
> This is from the perspective of developers who are already very familiar
> with Pulsar.
> If you don't know it, just want to create a client based on Pulsar's API.
> How to do it? For public API, I think a clear definition should be the
> first consideration.
>
> Thanks,
> Penghui
>
>
> On Thu, Jun 22, 2023 at 9:13 AM Rajan Dhabalia 
> wrote:
>
> > Please find the response inline.
> >
> > On Wed, Jun 21, 2023 at 5:53 PM PengHui Li  wrote:
> >
> > > > However, stats retrieval over HTTP API doesn’t work well in use cases
> > > when users would like to access this API at a higher scale when a large
> > > number of application nodes would like to use it over HTTP which could
> > > overload brokers and sometimes makes broker irresponsive and impact
> admin
> > > API performance. It’s also become difficult when Pulsar is deployed in
> > the
> > > cloud behind the SNI proxy and applications also want to access
> > large-scale
> > > stats information periodically over different HTTP port instead it
> would
> > be
> > > better if applications can fetch stats over on same binary protocol for
> > > scalability and accessibility reasons.
> > >
> > > From the motivation of the 

Re: Improving the usability of the bookie Isolation feature

[Enrico Olivelli]

Il giorno lun 26 giu 2023 alle ore 09:21 Yubiao Feng
 ha scritto:
>
> Hi Yan,Asaf
>
> > I want to add only one step to your plan.
> > If you introduce this flag in Y.X, then in Y.(X+1),
> > let's remove this flag
> > and keep the "true" value as the behavior.
>
> I agree with Asaf
+1

Enrico

>
> Thanks
> Yubiao Feng
>
> On Mon, Jun 19, 2023 at 9:57 AM horizonzy  wrote:
>
> > Background
> >
> > In the Pulsar, it has two features:
> >
> >-
> >
> >The first feature allows users to set group and rack information for
> >bookies using pulsar-admin bookies set-bookie-rack.
> >
> > Here, users set bookie1 to bookie5 to the default group and bookie6 to
> > bookie10 to the share group using commands, they don't care about rack
> > information, they only care about which group the bookie belongs to.
> >
> > default={bookie1:3181=BookieInfoImpl(rack=default-rack,
> > hostname=null), bookie2:3181=BookieInfoImpl(rack=default-rack,
> > hostname=null), bookie3:3181=BookieInfoImpl(rack=default-rack,
> > hostname=null), bookie4:3181=BookieInfoImpl(rack=default-rack,
> > hostname=null), bookie5:3181=BookieInfoImpl(rack=default-rack,
> > hostname=null)}
> >
> > _shared_={bookie6:3181=BookieInfoImpl(rack=default-rack,
> > hostname=null), bookie7:3181=BookieInfoImpl(rack=default-rack,
> > hostname=null), bookie8:3181=BookieInfoImpl(rack=default-rack,
> > hostname=null), bookie9:3181=BookieInfoImpl(rack=default-rack,
> > hostname=null), bookie10:3181=BookieInfoImpl(rack=default-rack,
> > hostname=null)}
> >
> >
> >-
> >
> >The second feature allows users to set the priority of traffic for a
> >namespace, where traffic is directed to the primary group first and
> > then to
> >the secondary group. Users can set this priority using pulsar-admin
> >ns-isolation-policy set --namespaces public/default --primary "group"
> >--secondary "group".
> >
> > Here, users set the primary group of the /public/default namespace to
> > "share" using a command.
> >
> > {
> >   "bookkeeperAffinityGroupPrimary" : "share"
> > }
> >
> > After this work is completed, all traffic under the /public/default
> > namespace will be directed to bookie6-10 in the "share" group.
> >
> > Drawbacks
> >
> > After a period of time, users added some new bookies [bk11, bk12, bk13,
> > bk14, bk15] to the bookie cluster, they found that some traffic under the
> > /public/default namespace was directed to the newly added machines. After
> > investigation, we eventually found that this was a defect in the working
> > mechanism of bookkeeperAffinityGroupPrimary.
> >
> > *bookkeeperAffinityGroupPrimary work mechanism*
> >
> > All bookies in the cluster: bk1-bk15.
> >
> > Here are the steps of the broker pick bookies.
> >
> >1.
> >
> >Get the bookie rack info config default: [bk1, bk2, bk3, bk4, bk5];
> > share:
> >[bk6, bk7, bk8, bk9, bk10]
> >2.
> >
> >Exclude the bookies which are not the bookkeeperAffinityGroupPrimary
> >(share).
> >3.
> >
> >Exclude the default group bookies [bk1, bk2, bk3, bk4, bk5].
> >4.
> >
> >Pick bookies from the remaining bookies [bk6, bk7, bk8, bk9, bk10, bk11,
> >bk12, bk13, bk14, bk15]
> >
> > Therefore, some traffic may go to bk11-bk15, which is not what the users
> > expect. The reason is that the new bookies, bk11 to bk15, did not have rack
> > information set and were not part of any group.
> >
> > We provided a workaround for users to set the rack information for bk11 to
> > bk15 in advance using the command pulsar-admin bookies set-bookie-rack
> > before starting them. After user adopting this workaround, the traffic
> > worked as expected.
> >
> > For user, it may be a bit inconvenient as they need to set rack information
> > in advance before bringing new bookies online. In scenarios where there are
> > strict limitations on traffic, if the bookie operation and maintenance
> > personnel overlook this step, it could cause problems.
> >
> > Improvement
> >
> > I would like to introduce a new configuration strict for
> > bookkeeperAffinityGroupPrimary. The default value for this configuration is
> > false, which means that for old users upgrading to the new version, the
> > logic will remain the same and bookies without rack information will not be
> > constrained.
> >
> > If users manually set strict to true using the command pulsar-admin
> > ns-isolation-policy set --namespaces public/default --primary "group"
> > --secondary "group" --strict true, when the broker selects a bookie, it
> > will only choose from the bookies in the primary group. If there are not
> > enough bookies in the primary group, it will choose from the bookies in the
> > secondary group. If there are not enough bookies in either group, an
> > exception will be thrown. Bookies without rack information set using
> > pulsar-admin
> > bookies set-bookie-rack will not be selected.
> >
> > Compatibility
> >
> > When users upgrade from the old version to the new version, the working
> > 

Re: Improving the usability of the bookie Isolation feature

[Yubiao Feng]

Hi Yan,Asaf

> I want to add only one step to your plan.
> If you introduce this flag in Y.X, then in Y.(X+1),
> let's remove this flag
> and keep the "true" value as the behavior.

I agree with Asaf

Thanks
Yubiao Feng

On Mon, Jun 19, 2023 at 9:57 AM horizonzy  wrote:

> Background
>
> In the Pulsar, it has two features:
>
>-
>
>The first feature allows users to set group and rack information for
>bookies using pulsar-admin bookies set-bookie-rack.
>
> Here, users set bookie1 to bookie5 to the default group and bookie6 to
> bookie10 to the share group using commands, they don't care about rack
> information, they only care about which group the bookie belongs to.
>
> default={bookie1:3181=BookieInfoImpl(rack=default-rack,
> hostname=null), bookie2:3181=BookieInfoImpl(rack=default-rack,
> hostname=null), bookie3:3181=BookieInfoImpl(rack=default-rack,
> hostname=null), bookie4:3181=BookieInfoImpl(rack=default-rack,
> hostname=null), bookie5:3181=BookieInfoImpl(rack=default-rack,
> hostname=null)}
>
> _shared_={bookie6:3181=BookieInfoImpl(rack=default-rack,
> hostname=null), bookie7:3181=BookieInfoImpl(rack=default-rack,
> hostname=null), bookie8:3181=BookieInfoImpl(rack=default-rack,
> hostname=null), bookie9:3181=BookieInfoImpl(rack=default-rack,
> hostname=null), bookie10:3181=BookieInfoImpl(rack=default-rack,
> hostname=null)}
>
>
>-
>
>The second feature allows users to set the priority of traffic for a
>namespace, where traffic is directed to the primary group first and
> then to
>the secondary group. Users can set this priority using pulsar-admin
>ns-isolation-policy set --namespaces public/default --primary "group"
>--secondary "group".
>
> Here, users set the primary group of the /public/default namespace to
> "share" using a command.
>
> {
>   "bookkeeperAffinityGroupPrimary" : "share"
> }
>
> After this work is completed, all traffic under the /public/default
> namespace will be directed to bookie6-10 in the "share" group.
>
> Drawbacks
>
> After a period of time, users added some new bookies [bk11, bk12, bk13,
> bk14, bk15] to the bookie cluster, they found that some traffic under the
> /public/default namespace was directed to the newly added machines. After
> investigation, we eventually found that this was a defect in the working
> mechanism of bookkeeperAffinityGroupPrimary.
>
> *bookkeeperAffinityGroupPrimary work mechanism*
>
> All bookies in the cluster: bk1-bk15.
>
> Here are the steps of the broker pick bookies.
>
>1.
>
>Get the bookie rack info config default: [bk1, bk2, bk3, bk4, bk5];
> share:
>[bk6, bk7, bk8, bk9, bk10]
>2.
>
>Exclude the bookies which are not the bookkeeperAffinityGroupPrimary
>(share).
>3.
>
>Exclude the default group bookies [bk1, bk2, bk3, bk4, bk5].
>4.
>
>Pick bookies from the remaining bookies [bk6, bk7, bk8, bk9, bk10, bk11,
>bk12, bk13, bk14, bk15]
>
> Therefore, some traffic may go to bk11-bk15, which is not what the users
> expect. The reason is that the new bookies, bk11 to bk15, did not have rack
> information set and were not part of any group.
>
> We provided a workaround for users to set the rack information for bk11 to
> bk15 in advance using the command pulsar-admin bookies set-bookie-rack
> before starting them. After user adopting this workaround, the traffic
> worked as expected.
>
> For user, it may be a bit inconvenient as they need to set rack information
> in advance before bringing new bookies online. In scenarios where there are
> strict limitations on traffic, if the bookie operation and maintenance
> personnel overlook this step, it could cause problems.
>
> Improvement
>
> I would like to introduce a new configuration strict for
> bookkeeperAffinityGroupPrimary. The default value for this configuration is
> false, which means that for old users upgrading to the new version, the
> logic will remain the same and bookies without rack information will not be
> constrained.
>
> If users manually set strict to true using the command pulsar-admin
> ns-isolation-policy set --namespaces public/default --primary "group"
> --secondary "group" --strict true, when the broker selects a bookie, it
> will only choose from the bookies in the primary group. If there are not
> enough bookies in the primary group, it will choose from the bookies in the
> secondary group. If there are not enough bookies in either group, an
> exception will be thrown. Bookies without rack information set using
> pulsar-admin
> bookies set-bookie-rack will not be selected.
>
> Compatibility
>
> When users upgrade from the old version to the new version, the working
> mechanism of bookkeeperAffinityGroupPrimary remains the same as before.
> When users upgrade to the new version and set strict to true using the
> command pulsar-admin ns-isolation-policy set --namespaces public/default
> --primary "group" --secondary "group" --strict true, and then roll back to
> the old version, the broker should 

[DISCUSS] RFR - ns-isolation-policy support hostport pattern

[tison]

Hi devs,

The pulsar-client CLI has a ns-isolation-policy command whose primary and
secondary regex pattern currently verify against hostname only, while we
return broker list in hostport pattern, we may support the regex pattern
test against the whole URL (or a combine of host:port but that can be a bit
tricky.

The direction is still TBD so I am posting this email for ask an explicit
consensus on whether it's a valid usability improvement.

Here is the PR[1] and the issue[2] is back to the last year when the author
claims there is a rough (offline) consensus on the community meeting.


Best,
tison.

[1] https://github.com/apache/pulsar/pull/20256
[2] https://github.com/apache/pulsar/issues/15795

Re: [DISCUSS] PIP-279: Reformat property in generateResponseWithEntry

[Enrico Olivelli]

+1 (binding)

Enrico

Il giorno lun 26 giu 2023 alle ore 07:53 guo jiwei
 ha scritto:
>
> +1 (binding)
>
> This is a bug and we need this fix.
>
>
>
>
> Regards
> Jiwei Guo (Tboy)
>
>
> On Wed, Jun 21, 2023 at 11:23 AM steven lu  wrote:
>
> > # Motivation
> >
> > reformat property,for a http header name cannot contain the following
> > prohibited characters: =,;: \t\r\n\v\f
> >
> > for example:
> > {"city=shanghai":"tag"}
> > when we run `bin/pulsar-admin topics get-message-by-id `, it will
> > throw exception, the exception is:
> > `Reason: java.util.concurrent.CompletionException:
> >
> > org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> > Could not complete the operation. Number of retries has been
> > exhausted. Failed reason: a header name cannot contain the following
> > prohibited characters: =,;: \t\r\n\v\f: =`
> >
> >  > src="
> > https://github.com/StevenLuMT/pulsar/assets/42990025/973d95b9-4ac2-4977-b160-162c4b53a613
> > ">
> >
> > # High Level Design
> >
> > In master branch,
> > in an http
> > request:getMessageById("/{tenant}/{namespace}/{topic}/ledger/{ledgerId}/entry/{entryId}"),
> > replace `"X-Pulsar-PROPERTY-" + msgProperties.getKey()` with
> > `"X-Pulsar-PROPERTY"`
> >
> > After release-3.1.0, this feature begins to take effect.
> >

Re: [DISCUSS] Apache Pulsar 2.11.2 release

[Enrico Olivelli]

Jiwei,

Il giorno lun 26 giu 2023 alle ore 08:09 guo jiwei
 ha scritto:
>
> Hello, community:
>It has been more than 2 months since the release of 2.11.1. During this
> period, we have more than 80 fixes.
> https://github.com/apache/pulsar/compare/v2.11.1...branch-2.11
>I suggest releasing  2.11.2. If you have no comments, I will check the
> existing PR that needs to cherry-pick to the 2.11 branch.
>If you have a PR that needs to cherry-pick, please leave a message here.


Probably we want to wait for a BK release, apart from that I am +1

Enrico

>
> Regards
> Jiwei Guo (Tboy)

[DISCUSS] Apache Pulsar 2.11.2 release

[guo jiwei]

Hello, community:
   It has been more than 2 months since the release of 2.11.1. During this
period, we have more than 80 fixes.
https://github.com/apache/pulsar/compare/v2.11.1...branch-2.11
   I suggest releasing  2.11.2. If you have no comments, I will check the
existing PR that needs to cherry-pick to the 2.11 branch.
   If you have a PR that needs to cherry-pick, please leave a message here.

Regards
Jiwei Guo (Tboy)