Re: [DISCUSS] Release Pulsar Go Client 0.10.0
Hi, Frank I will start the release next Monday. BR, Zike Yang On Sat, Mar 18, 2023 at 2:52 AM Frank Kelly wrote: > > Hello, > Any updates on this? > > Thanks in advance > > -Frank > > On Fri, Mar 3, 2023 at 6:49 AM Frank Kelly wrote: > > > I too am excited for the 0.10.0 release - thank you for driving this! > > > > -Frank Kelly > > > > On Wed, Mar 1, 2023 at 10:07 PM Zike Yang wrote: > > > >> Hi, Baodi, Yunze > >> > >> Thanks. Sure, I will wait for them. > >> > >> BR, > >> Zike Yang > >> > >> On Wed, Mar 1, 2023 at 9:02 PM Yunze Xu > >> wrote: > >> > > >> > Please wait for a performance fix for the case when batch index ACK is > >> > enabled, I'm working on it. Currently, the throughput cannot exceed > >> > even 20MB/s when it's enabled. > >> > > >> > Thanks, > >> > Yunze > >> > > >> > On Wed, Mar 1, 2023 at 8:39 PM Baodi Shi wrote: > >> > > > >> > > Hi, zike. > >> > > > >> > > The current pulsar-client-go master branch has some flay-test. There > >> may be > >> > > some internal bugs, I think we need to wait for them to be fixed. > >> > > > >> > >- https://github.com/apache/pulsar-client-go/issues/971 > >> > > > >> > > > >> > > Thanks, > >> > > Baodi Shi > >> > > > >> > > > >> > > 在 2023年3月1日 20:26:10 上，Zike Yang 写道： > >> > > > >> > > > I will include this PR > >> > > > https://github.com/apache/pulsar-client-go/pull/968 to this release > >> > > > since it's an important performance improvement. > >> > > > > >> > > > BR, > >> > > > Zike Yang > >> > > > > >> > > > On Wed, Mar 1, 2023 at 8:25 PM Zike Yang wrote: > >> > > > > >> > > > > >> > > > Hi everyone, > >> > > > > >> > > > > >> > > > I would like to propose releasing the Pulsar Go Client 0.10.0. > >> > > > > >> > > > > >> > > > It has been several months since the last release. And there are > >> > > > > >> > > > several new features and bug fixes in the master branch[0]. It’s > >> time > >> > > > > >> > > > to release a new version. > >> > > > > >> > > > > >> > > > Please let me know if you have any PRs that need to be included in > >> 0.10.0 > >> > > > > >> > > > > >> > > > [0] > >> https://github.com/apache/pulsar-client-go/compare/v0.9.0...master > >> > > > > >> > > > > >> > > > BR, > >> > > > > >> > > > Zike Yang > >> > > > > >> > > > > >> > >
Re: [DISCUSS] Release Pulsar Go Client 0.10.0
Hello, Any updates on this? Thanks in advance -Frank On Fri, Mar 3, 2023 at 6:49 AM Frank Kelly wrote: > I too am excited for the 0.10.0 release - thank you for driving this! > > -Frank Kelly > > On Wed, Mar 1, 2023 at 10:07 PM Zike Yang wrote: > >> Hi, Baodi, Yunze >> >> Thanks. Sure, I will wait for them. >> >> BR, >> Zike Yang >> >> On Wed, Mar 1, 2023 at 9:02 PM Yunze Xu >> wrote: >> > >> > Please wait for a performance fix for the case when batch index ACK is >> > enabled, I'm working on it. Currently, the throughput cannot exceed >> > even 20MB/s when it's enabled. >> > >> > Thanks, >> > Yunze >> > >> > On Wed, Mar 1, 2023 at 8:39 PM Baodi Shi wrote: >> > > >> > > Hi, zike. >> > > >> > > The current pulsar-client-go master branch has some flay-test. There >> may be >> > > some internal bugs, I think we need to wait for them to be fixed. >> > > >> > >- https://github.com/apache/pulsar-client-go/issues/971 >> > > >> > > >> > > Thanks, >> > > Baodi Shi >> > > >> > > >> > > 在 2023年3月1日 20:26:10 上，Zike Yang 写道： >> > > >> > > > I will include this PR >> > > > https://github.com/apache/pulsar-client-go/pull/968 to this release >> > > > since it's an important performance improvement. >> > > > >> > > > BR, >> > > > Zike Yang >> > > > >> > > > On Wed, Mar 1, 2023 at 8:25 PM Zike Yang wrote: >> > > > >> > > > >> > > > Hi everyone, >> > > > >> > > > >> > > > I would like to propose releasing the Pulsar Go Client 0.10.0. >> > > > >> > > > >> > > > It has been several months since the last release. And there are >> > > > >> > > > several new features and bug fixes in the master branch[0]. It’s >> time >> > > > >> > > > to release a new version. >> > > > >> > > > >> > > > Please let me know if you have any PRs that need to be included in >> 0.10.0 >> > > > >> > > > >> > > > [0] >> https://github.com/apache/pulsar-client-go/compare/v0.9.0...master >> > > > >> > > > >> > > > BR, >> > > > >> > > > Zike Yang >> > > > >> > > > >> >
Re: [DISCUSS] Change PIP template
Thanks for this initiative, Asaf. As part of this process, I would like for us to add a security and a multi-tenancy section to the PIP template. As you suggest, the template conveys what the community values, and these two sections must always be considered when changing Pulsar in fundamental ways. (Thanks for already adding the security section to your template!) Thanks, Michael On Thu, Mar 16, 2023 at 2:58 AM Asaf Mesika wrote: > > Here's the PR to remove the form and add a new issue template in Markdown > containing the suggested structure and description for each section. > > https://github.com/apache/pulsar/pull/19832 > > > On Wed, Mar 1, 2023 at 3:43 PM Elliot West > wrote: > > > +1 Asaf > > > > I'd also suggest that we encourage the submission of relevant diagrams. > > This is trivial to do with the GitHub markdown editor, but I suspect is > > often neglected because users do not know the feature exists. > > > > On Wed, 1 Mar 2023 at 13:22, Asaf Mesika wrote: > > > > > Ok. > > > > > > I'll draft a PR and link it here when I'm done. Thanks! > > > > > > On Tue, Feb 28, 2023 at 7:08 AM PengHui Li wrote: > > > > > > > +1 > > > > > > > > Penghui > > > > > > > > On Mon, Feb 27, 2023 at 9:24 PM Asaf Mesika > > > wrote: > > > > > > > > > Mails don't support things like markdown diagrams or images and are > > > > > generally less easy to read. > > > > > My proposal includes a required section called Links in which you > > need > > > to > > > > > fill in the discussion thread in DEV mailing list and vote thread. > > > > > > > > > > > > > > > On Mon, Feb 27, 2023 at 3:08 PM Girish Sharma < > > scrapmachi...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > Hi Asaf, > > > > > > I was referring to the PIP process, as a whole, as explained in > > > > > > https://github.com/apache/pulsar/blob/master/wiki/proposals/PIP.md > > > > > > Someone looking at GitHub ticket would find and almost empty PIP GH > > > > issue > > > > > > while the same PIP has had many discussions over here in the ML. > > > > > > There is scope of improvement in the process where we either remove > > > the > > > > > > first step to create the PIP over at GitHub and directly present > > the > > > > PIP > > > > > in > > > > > > the first mail of the thread here, or we do all discussions in GH. > > > > > > Both the ML and GH are searchable and linkable for tracking > > purposes. > > > > > > > > > > > > Regards > > > > > > > > > > > > On Mon, Feb 27, 2023 at 6:23 PM Asaf Mesika > > > > > > > wrote: > > > > > > > > > > > > > On Sun, Feb 26, 2023 at 2:49 PM Girish Sharma < > > > > scrapmachi...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Good proposal Asaf. > > > > > > > > I've also wondered why the PIP creation and discussion process > > is > > > > so > > > > > > > > separated. The PIP discussion and voting starts off as a GitHub > > > > > issue, > > > > > > > but > > > > > > > > all of its discussion happens here on the mailing list. Is > > there > > > > > scope > > > > > > of > > > > > > > > improvement in that process as well? > > > > > > > > > > > > > > > > > > > > > > Not sure I follow. Can you outline the problem exactly? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > On Sun, Feb 26, 2023 at 6:16 PM tison > > wrote: > > > > > > > > > > > > > > > > > Hi Asaf, > > > > > > > > > > > > > > > > > > I agree that, generally, a PIP is written as a whole and > > paste > > > as > > > > > the > > > > > > > > body. > > > > > > > > > So +1 for your proposal. > > > > > > > > > > > > > > > > > > Additionally, I'm thinking of moving the doc of procedure > > > > > > (wiki/PIP.md) > > > > > > > > to > > > > > > > > > the contributions guide and use the new markdown template to > > > > > > supersede > > > > > > > > the > > > > > > > > > wiki/PIP-template.md. Then we don't need to hold the wiki > > > folder. > > > > > > > > > > > > > > > > > > It can be an extended version to your proposal, so let's keep > > > on > > > > > your > > > > > > > > > proposal in this thread. Just for your reference. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > tison. > > > > > > > > > > > > > > > > > > > > > > > > > > > Asaf Mesika 于2023年2月26日周日 19:18写道： > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > I would like to suggest two changes I'd like to make to the > > > PIP > > > > > > > design > > > > > > > > > > template: > > > > > > > > > > 1. Remove the form - just have a markdown template fill the > > > > issue > > > > > > > body > > > > > > > > as > > > > > > > > > > it is created. > > > > > > > > > > 2. Change the PIP template structure > > > > > > > > > > > > > > > > > > > > == Removing the form > > > > > > > > > > > > > > > > > > > > Today, when you want to submit a PIP, you are required to > > > fill > > > > > out > > > > > > a > > > > > > > > form > > > > > > > > > > with boxes composed of 3-4 lines length. > > > > > > > > >
Re: [DISCUSS] PIP-259: Make the config httpMaxRequestHeaderSize of the pulsar web server to configurable
+1 Hi, Yubiao : Thanks for your explanation. That makes sense to me. Thanks, Bo Yubiao Feng 于2023年3月17日周五 16:29写道： > > Hi Bo > > > I have a question, why we need `httpClientRequestBufferSize ` in > > proxy, can you explain in detail? > > Since The Pulsar-Proxy uses the tool `jetty-client` to forward HTTP > requests from users to The Pulsar-Broker, if the proxy receives a request > like this: > > ``` > GET /admin/v2/public/default/tp.long..long/stats HTTP/1.1 > ``` > > The internal client with forward this request like this: > > ``` > ByteBuf buf = allocate( config.httpClientRequestBufferSize ) > buf.write(requestLine); // (Highlight) we will get > a BufferOverflowException if the request line is too long. > ``` > > Therefore, in addition to ensuring that the proxy server can receive a long > request line, the internal client must also process a long request line. > And this problem can be solved by making configuration > `httpClientRequestBufferSize` configurable. > > > Thanks > Yubiao Feng > > > On Thu, Mar 16, 2023 at 8:12 PM 丛搏 wrote: > > > hi yubiao : > > > > I have a question, why we need `httpClientRequestBufferSize ` in > > proxy, can you explain in detail? > > > > Thanks, > > Bo > > > > Yubiao Feng 于2023年3月16日周四 00:11写道： > > > > > > > > Hi community > > > > > > I am starting a DISCUSS for "PIP-259: Make the config > > > httpMaxRequestHeaderSize of the pulsar web server configurable". > > > > > > ### Motivation > > > > > > We have two ways to manage pulsar's resources: > > > - By client API (Can manage some resources, such as `create topic`, > > `create > > > subscriber`, and so on) > > > - By admin API (Can manage all the resources) > > > > > > The `client API` has no limit on the request length. And the `admin API` > > > has a limit on the request length(such as HTTP request line and HTTP > > > request headers), this restriction is done by the built-in web container > > > Jetty. > > > > > > Almost resources can be created by two APIs, but can only be modified and > > > deleted by `admin API`. This causes us to be unable to modify or delete > > > resources created by `client API` with too long a name because it exceeds > > > Jetty's default HTTP request URI length limit. > > > > > > ### Goal > > > > > > 1. For web servers > > > Provide a way to modify Jetty's `httpMaxRequestHeaderSize` configuration > > > (involves two servers: the web server in pulsar and the web server in > > > pulsar-proxy) > > > > > > 2.For the internal client in pulsar-proxy > > > Provide a way to modify Jetty-client's `httpClientRequestBufferSize` > > > configuration. > > > > > > Since the pulsar-proxy handles HTTP requests like this： `pulsar-admin.sh` > > > -> `proxy web server` -> `(highlight) internal client in proxy` -> > > `pulsar > > > web server`. > > > > > > When the internal client forwards a request, it forwards the request > > header > > > and the request body, and all the data passes through a buffer( we call > > it > > > Buf ), like this: > > > - Receive a request > > > - Put the request line and request headers input to the Buf. > > > - (highlight)Flush the Buf ( If the data in the request > > > line and request header exceeds the length of the buf, an error is > > reported > > > ) > > > - Put the request body input to the Buf. > > > - Flush the Buf if it is full. > > > > > > So we need a config to set the `buff size` of the Buf: > > > `pulsar-proxy.conf.httpClientRequestBufferSize` -> `buf size of the > > > internal client`. > > > > > > ### API Changes > > > > > > ServiceConfiguration.java > > > ```java > > >@FieldContext( > > > category = CATEGORY_HTTP, > > > doc = """ > > > The maximum size in bytes of the request header. > > > Larger headers will allow for more and/or larger cookies > > > plus larger form content encoded in a URL. > > > However, larger headers consume more memory and can make > > a > > > server more vulnerable to denial of service > > > attacks. > > > """ > > > ) > > >private int httpMaxRequestHeaderSize = 8 * 1024; > > > ``` > > > > > > ProxyConfiguration.java > > > > > > ```java > > > @FieldContext( > > > minValue = 1, > > > category = CATEGORY_HTTP, > > > doc = """ > > > The maximum size in bytes of the request header. > > > Larger headers will allow for more and/or larger cookies > > > plus larger form content encoded in a URL. > > > However, larger headers consume more memory and can make > > a > > > server more vulnerable to denial of service > > > attacks. > > > """ > > > ) > > > private int httpMaxRequestHeaderSize = 8 * 1024; > > > > > > @FieldContext( > > > minValue = 1, > > > category = CATEGORY_HTTP, > > > doc = """ > > > the size of the buffer used to write requests to
Re: New Pulsar docs (API + CLI) are available!
Yu , Thanks for driving this. It looks great! Best, Max Xu On Thu, Mar 16, 2023 at 8:06 PM Yu wrote: > Hi community, > > To improve the developer experience and take Pulsar content to the next > level, recently we (+@hangc0276) have added some new docs as below. > > === > Fresh New Content > === > > - Pulsar API > (https://pulsar.apache.org/docs/next/pulsar-api-overview/) > > - Pulsar admin API > → Overview (https://pulsar.apache.org/docs/next/admin-api-overview/) > → Use cases (https://pulsar.apache.org/docs/next/admin-api-use-cases/） > → Features (https://pulsar.apache.org/docs/next/admin-api-features/) > → Tools (https://pulsar.apache.org/docs/next/admin-api-tools/) > → Get started (https://pulsar.apache.org/docs/next/admin-api-get-started/) > > - Pulsar client API > (https://pulsar.apache.org/docs/next/client-api-overview/) > > - Pulsar CLI > → pulsar-admin: Get started ( > https://pulsar.apache.org/docs/next/get-started-pulsar-admin/) > > Enjoy reading! > Here [1] are related doc PRs. > We'll add more docs on these topics in the near future. > Feel free to leave your suggestions, TIA! > > === > Acknowledgment > === > > @hangc0276 Huge shout out to you for providing technical input and > guidance! > > @BewareMyPower @horizonzy A big thank you for your technical support! > > === > > [1] > - https://github.com/apache/pulsar-site/pull/462 > - https://github.com/apache/pulsar-site/pull/403 > - https://github.com/apache/pulsar/pull/19002 > - https://github.com/apache/pulsar/pull/18680 > > Yu >
Re: Re: Introducer Pulsar admin api to pulsar-client-go
Thanks Yu and Eric for your follow-up.
Yes, we're working on that. I believe we will soon see pulsar-admin-go :)
Best,
Max Xu
On Wed, Mar 15, 2023 at 2:36 PM Yu wrote:
> Hi Eric,
>
> Thanks for sharing the updates!
>
> I discussed this project with Max previously. We would like to contribute
> docs for it, and I've already added "Go admin API (coming soon)" to the new
> Pulsar API docs already [1], which brings many benefits, such as:
>
> - Build excitement before launch.
> - Generate a targeted list of early adopters interested in this brand-new
> tool.
> - Improve SEO by adding relevant keywords.
>
> [1] https://github.com/apache/pulsar-site/pull/462#discussion_r1130770643
>
> Yu
>
>
> On Wed, Mar 15, 2023 at 11:56 AM Shen Eric
> wrote:
>
> > Hi Zhangjian,
> >
> > We can share some progress on our side:
> >
> > we have completed:
> >
> > * Extracted the pulsar-admin-go library from the pulsarctl and moved
> > it to a separate repo under StreamNative (now it is private)
> > * Updated the project layout and rename some pkg names like the cli
> pkg
> >
> > what we will do next:
> >
> > * Add some pulsar-admin-go library documentation
> > * Setup some CI/CD infra configurations for the repo
> > * Then we will release the 0.1.0 and open source this repo
> >
> > The release of the 0.1.0 and repo open-source we wanna make it in March
> > and will announce this to the community.
> >
> > The timeline to donate this project to Apache is undecided now but we
> will
> > listen to the feedback from the community and donate it when most of us
> > think it's qualified and stable.
> >
> > On 2023/03/03 04:14:33 ZhangJian He wrote:
> > > Hi, Eric. Thank you very much for the work you have done. I have no
> > > objections to the process, and it would be even better if there could
> be
> > a
> > > rough timeline.
> > >
> > > Thanks
> > > ZhangJian He
> > >
> > >
> > > On Fri, 3 Mar 2023 at 09:06, Shen Eric wrote:
> > >
> > > > Hi Zhangjian,
> > > >
> > > > I am a PM from StreamNative and we also had some internal discussions
> > > > related to this topic. Let me share our ongoing planning:
> > > >
> > > > * We will extract the pulsar admin pkg from the pulsarctl to a
> > > > separate open repo which will be called pulsar-admin-go under
> > StreamNative.
> > > > * Will iterate the pulsar-admin-go library by adding more tests,
> > > > documentation and may also update or fix the existing APIs.
> > > > * After the pulsar-admin-go library is stable, we will contribute
> this
> > > > project to Apache Foundation.
> > > >
> > > > Do you think this plan works for you?
> > > >
> > > > On 2023/02/17 01:47:26 ZhangJian He wrote:
> > > > > I would like to express that the current Pulsar client for Go
> > > > > (pulsar-client-go) is missing the pulsar Admin API. As such, I
> would
> > like
> > > > > to propose that we work towards adding this feature to
> > pulsar-client-go.
> > > > >
> > > > > I believe that this new feature would be a valuable addition to
> > > > > pulsar-client-go, and I am excited to work to make it happen.
> > > > >
> > > > > I have submitted a PR:
> > > > https://github.com/apache/pulsar-client-go/pull/959
> > > > > The full api is not currently available, but we are adding.
> > > > >
> > > > > Below is a simple example about how to use
> > > > >
> > > > > ## usage
> > > > >
> > > > > ```go
> > > > > package main
> > > > >
> > > > > import (
> > > > > "fmt"
> > > > > "github.com/apache/pulsar-client-go/padmin"
> > > > > )
> > > > >
> > > > > func main() {
> > > > > admin, err := padmin.NewDefaultPulsarAdmin()
> > > > > if err != nil {
> > > > > panic(err)
> > > > > }
> > > > > // get namespace topic list
> > > > > topics, err := admin.PersistentTopics.ListNamespaceTopics("tenant",
> > > > > "namespace")
> > > > > if err != nil {
> > > > > panic(err)
> > > > > }
> > > > > fmt.Println(topics)
> > > > > }
> > > > > ```
> > > > >
> > > > > Thanks
> > > > > ZhangJian He
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > >
> > > > Eric Shen 沈瑀昊
> > > >
> > >
> >
> >
> > --
> >
> > Best regards,
> >
> > Eric Shen 沈瑀昊
> >
>
Re: [DISCUSS] PIP-259: Make the config httpMaxRequestHeaderSize of the pulsar web server to configurable
Hi Bo > I have a question, why we need `httpClientRequestBufferSize ` in > proxy, can you explain in detail? Since The Pulsar-Proxy uses the tool `jetty-client` to forward HTTP requests from users to The Pulsar-Broker, if the proxy receives a request like this: ``` GET /admin/v2/public/default/tp.long..long/stats HTTP/1.1 ``` The internal client with forward this request like this: ``` ByteBuf buf = allocate( config.httpClientRequestBufferSize ) buf.write(requestLine); // (Highlight) we will get a BufferOverflowException if the request line is too long. ``` Therefore, in addition to ensuring that the proxy server can receive a long request line, the internal client must also process a long request line. And this problem can be solved by making configuration `httpClientRequestBufferSize` configurable. Thanks Yubiao Feng On Thu, Mar 16, 2023 at 8:12 PM 丛搏 wrote: > hi yubiao : > > I have a question, why we need `httpClientRequestBufferSize ` in > proxy, can you explain in detail? > > Thanks, > Bo > > Yubiao Feng 于2023年3月16日周四 00:11写道： > > > > > Hi community > > > > I am starting a DISCUSS for "PIP-259: Make the config > > httpMaxRequestHeaderSize of the pulsar web server configurable". > > > > ### Motivation > > > > We have two ways to manage pulsar's resources: > > - By client API (Can manage some resources, such as `create topic`, > `create > > subscriber`, and so on) > > - By admin API (Can manage all the resources) > > > > The `client API` has no limit on the request length. And the `admin API` > > has a limit on the request length(such as HTTP request line and HTTP > > request headers), this restriction is done by the built-in web container > > Jetty. > > > > Almost resources can be created by two APIs, but can only be modified and > > deleted by `admin API`. This causes us to be unable to modify or delete > > resources created by `client API` with too long a name because it exceeds > > Jetty's default HTTP request URI length limit. > > > > ### Goal > > > > 1. For web servers > > Provide a way to modify Jetty's `httpMaxRequestHeaderSize` configuration > > (involves two servers: the web server in pulsar and the web server in > > pulsar-proxy) > > > > 2.For the internal client in pulsar-proxy > > Provide a way to modify Jetty-client's `httpClientRequestBufferSize` > > configuration. > > > > Since the pulsar-proxy handles HTTP requests like this： `pulsar-admin.sh` > > -> `proxy web server` -> `(highlight) internal client in proxy` -> > `pulsar > > web server`. > > > > When the internal client forwards a request, it forwards the request > header > > and the request body, and all the data passes through a buffer( we call > it > > Buf ), like this: > > - Receive a request > > - Put the request line and request headers input to the Buf. > > - (highlight)Flush the Buf ( If the data in the request > > line and request header exceeds the length of the buf, an error is > reported > > ) > > - Put the request body input to the Buf. > > - Flush the Buf if it is full. > > > > So we need a config to set the `buff size` of the Buf: > > `pulsar-proxy.conf.httpClientRequestBufferSize` -> `buf size of the > > internal client`. > > > > ### API Changes > > > > ServiceConfiguration.java > > ```java > >@FieldContext( > > category = CATEGORY_HTTP, > > doc = """ > > The maximum size in bytes of the request header. > > Larger headers will allow for more and/or larger cookies > > plus larger form content encoded in a URL. > > However, larger headers consume more memory and can make > a > > server more vulnerable to denial of service > > attacks. > > """ > > ) > >private int httpMaxRequestHeaderSize = 8 * 1024; > > ``` > > > > ProxyConfiguration.java > > > > ```java > > @FieldContext( > > minValue = 1, > > category = CATEGORY_HTTP, > > doc = """ > > The maximum size in bytes of the request header. > > Larger headers will allow for more and/or larger cookies > > plus larger form content encoded in a URL. > > However, larger headers consume more memory and can make > a > > server more vulnerable to denial of service > > attacks. > > """ > > ) > > private int httpMaxRequestHeaderSize = 8 * 1024; > > > > @FieldContext( > > minValue = 1, > > category = CATEGORY_HTTP, > > doc = """ > > the size of the buffer used to write requests to Broker. > > if "httpMaxRequestHeaderSize" is large than > > "httpClientRequestBufferSize", will set > > "httpClientRequestBufferSize" to the value of > > "httpMaxRequestHeaderSize" > > """ > > ) > > private int httpClientRequestBufferSize = httpMaxRequestHeaderSize; > > ``` > > > > ### Anything else? > > > > This change should cherry-pick
Re: [DISCUSS] PIP-258: Deprecation of the consumer subscribeTopicMode configuration
LGTM. +1
It will make the API clearer without bringing breaking changes.
> After the configuration is removed in subsequent versions, it will be clearer
Should we state in the PIP which version it will be removed?
Thanks,
Zike Yang
On Fri, Mar 17, 2023 at 8:50 AM Baodi Shi wrote:
>
> Hi, Any ideas please discuss, thanks.
>
> Thanks,
> Baodi Shi
>
>
> 在 2023年3月13日 22:24:09 上，Baodi Shi 写道：
>
> > Hi all,
> >
> > I've started a PIP to discuss: PIP-258: Deprecation of the consumer
> > subscribeTopicMode configuration
> >
> > ### Motivation
> >
> > About pattern subscribes of consumers, the `topicsPattern` and
> > `subscribeTopicMode` configurations are contradictory.
> >
> > For example, the `topicsPattern` represents only subscription to
> > `persistent topic`, but the `subscriptionTopicsMode` represents
> > subscription to `all topic`.
> >
> > ``` java
> > Pattern pattern =
> > Pattern.compile("persistent://my-property/my-ns/pattern-topic.*");
> > Consumer consumer = pulsarClient.newConsumer()
> > .topicsPattern(pattern)
> > .subscriptionTopicsMode(RegexSubscriptionMode.AllTopics)
> > .build();
> > ```
> >
> > Finally, `all topics` are subscribed. It's very confusing.
> >
> >
> > For more details, please read the PIP at
> > https://github.com/apache/pulsar/issues/19798
> >
> >
> >
> > Thanks,
> > Baodi Shi
> >
Re: [DISCUSS] PIP-255: Assign topic partitions to bundle by round robin
Thanks for your review. > Could you clarify the limitation of the current logic? The current logic cannot guarantee that the traffic of each bundle is the same, and must be balanced through split. However, the load of the topic is not always the same, and the traffic of the business changes with time, so the load of bundle will continue to change. If we rely on split+unload to balance, the number of bundles will eventually reach the upper limit. In order to avoid frequent split and unload, the current logic has many thresholds, allowing Broker to tolerate load imbalance, which is one of the reasons why the load gap between different nodes of the Pulsar cluster is large > For this issue, the community introduced a new assignment strategy, > LeastResourceUsageWithWeight, which better randomizes assignments. Yes, but LeastResourceUsageWithWeight still cannot completely solve the current problem, only alleviate it. We also optimized based on this implementation, but we will discuss this optimization in the following PIP, The current pip is not covered. > If each partition has the same load, then having the same number of topics per bundle should lead to the load balance. Then, I wonder how the current way, "hashing" does not achieve the goal here. We think that the loads of different partitions under a same topic are the same, but the loads of partitions of different topics are different. Bundles are shared by all topics in the entire namespace. If we guarantee each bundle has the same number of partitions, but these partitions may come from different topics, resulting in different loads for each bundle. If we split bundle according to load, the load of each topic may be different in different time periods, and it is impossible to keep the load of each Bundle the same. Using the round robin strategy, we can ensure that the number of partitions from a same Topic on each Bundle is roughly consistent, so that the load of each Bundle is also consistent. > happens if the leader restarts? how do we guarantee this mappingpersistence? 1）First of all, we need to find the starting bundle. partition-0 finds a bundle through consistent hashing, so as long as the number of bundles remains the same, the starting bundle is the same every time, and then other partitions 1, 2, 3, 4 ... is assigned the same result every time. 2）If the number of bundles changes, i.e. triggering split, the bundles of the entire namespace will be forced to be unloaded and all reassigned > It is unclear how RoundRobinPartitionAssigner will work with the existing > code. The specific implementation has been refined, please check the latest PIP issue On 2023/03/16 18:20:35 Heesung Sohn wrote: > Hi, > > Thank you for sharing this. > In general, I think this can be another good option for Pulsar load > assignment logic. > However, I have some comments below.
