[GitHub] [pulsar] tisonkun added a comment to the discussion: [Design] Pulsar All Releases Page

[GitBox]

GitHub user tisonkun added a comment to the discussion: [Design] Pulsar All 
Releases Page

I think the release manager could be included in the blog and we don't need to 
highlight it. Highlight the release date is useful for users, though.

Generally the picture looks good :)

GitHub link: 
https://github.com/apache/pulsar/discussions/17310#discussioncomment-3505745


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar-site] tisonkun commented on pull request #187: [fix][site] turn on all related code highlight

[GitBox]

tisonkun commented on PR #187:
URL: https://github.com/apache/pulsar-site/pull/187#issuecomment-1231164990

   cc @Anonymitaet 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] Pulsar Release 2.7.5 Candidate 3

[Michael Marshall]

When I build from the source (both for the git tag
`v2.7.5-candidate-3` and the extracted src tarball
`apache-pulsar-2.7.5-src.tar.gz`), I am getting an error when I run
the license check:

```
$ src/check-binary-license
distribution/server/target/apache-pulsar-2.7.5-bin.tar.gz
com.sun.activation-jakarta.activation-1.2.2.jar unaccounted for in LICENSE
jakarta.activation-jakarta.activation-api-1.2.2.jar unaccounted for in LICENSE
jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar unaccounted for in LICENSE
jakarta.activation-jakarta.activation-api-1.2.1.jar mentioned in
LICENSE, but not bundled
jakarta.xml.bind-jakarta.xml.bind-api-2.3.2.jar mentioned in LICENSE,
but not bundled
jakarta.activation-1.2.2.jar unaccounted for in lib/presto/LICENSE
jakarta.activation-1.2.2.jar unaccounted for in lib/presto/LICENSE
jakarta.activation-api-1.2.2.jar unaccounted for in lib/presto/LICENSE
jakarta.xml.bind-api-2.3.3.jar unaccounted for in lib/presto/LICENSE
jetty-alpn-java-client-9.4.27.v20200227.jar unaccounted for in
lib/presto/LICENSE

It looks like there are issues with the LICENSE/NOTICE.
```

My understanding is that the missing license attributions are a
blocker for the release candidate. Is someone able to confirm that
this is an issue? I ran this command as the release process doc
indicates here [0].

Before finding the above issue, I successfully ran the following checks:
- Verified checksums and signatures on all 45 artifacts
- Compiled from source (apache-pulsar-2.7.5-src.tar.gz) using `mvn
clean install -DskipTest`
- Ran `mvn apache-rat:check` successfully

Thanks,
Michael

[0] 
https://github.com/apache/pulsar/blob/db16e440622c3eed225c4da5380c70b16a583787/wiki/release/release-process.md#3-build-and-inspect-the-artifacts

On Mon, Aug 29, 2022 at 9:20 AM Nicolò Boschi  wrote:
>
> +1 (non binding)
>
> Checks:
>
> - Checksum and signatures
>
> - Apache Rat check passes
>
> - Compile from source
>
> - Run Pulsar standalone and produce-consume from CLI
>
>
> Nicolò Boschi
>
>
> Il giorno sab 27 ago 2022 alle ore 17:12 Qiang Huang <
> qiang.huang1...@gmail.com> ha scritto:
>
> > +1(non-binding)
> > - Checked signatures/checksums
> > - Checked the license headers
> > - Ran the standalone
> > - Validate Connectors
> > - Validate Pub/Sub and Java Functions
> > - Validate Stateful Functions
> >
> > Haiting Jiang  于2022年8月27日周六 06:40写道：
> >
> > > This is the third release candidate for Apache Pulsar, version 2.7.5.
> > >
> > > It fixes the following issues:
> > > https://github.com/apache/pulsar/compare/v2.7.4...v2.7.5-candidate-3
> > >
> > > *** Please download, test and vote on this release. This vote will stay
> > > open
> > > for at least 72 hours ***
> > >
> > > Note that we are voting upon the source (tag), binaries are provided for
> > > convenience.
> > >
> > > Source and binary files:
> > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.5-candidate-3/
> > >
> > > SHA-512 checksums:
> > >
> > >
> > >
> > 6aa74cb742c411edd40de2199eb70d774d0ce8cb7dbfc2ce33fce7e87949aafa73f93a1e154261b58a1df4ad10160bf43fc5a93b1aa9a0466fcf2a3ba1a6e385
> > > apache-pulsar-2.7.5-bin.tar.gz
> > >
> > >
> > facb1698e394b5428a0b9b6d71b891013f3c5b473cf72dd68a75ca729800710f2c6c476b1c47f7804c1b1c0941f79a6a9f334c7c703f66f6bac8f399386c5ad6
> > > apache-pulsar-2.7.5-src.tar.gz
> > >
> > > Maven staging repo:
> > > https://repository.apache.org/content/repositories/orgapachepulsar-1172/
> > >
> > > The tag to be voted upon:
> > > v2.7.5-candidate-3 (8eae5b8d572861e49c40d456b1f3cbc5d414afe1)
> > > https://github.com/apache/pulsar/releases/tag/v2.7.5-candidate-3
> > >
> > > Docker images:
> > >
> > >
> > >
> > https://hub.docker.com/layers/jason918/pulsar/2.7.5/images/sha256-ec77603bd943f1a56065d155428d6edccb2bf2ec57fafab8112c4b73c7bfcb6e
> > >
> > >
> > https://hub.docker.com/layers/pulsar-all/jason918/pulsar-all/latest/images/sha256-0792a0ec75ecbc16c9a42af127a68b3dea2331be3f2f0599f5fb1f4cd78def25
> > >
> > > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > >
> > > Please download the source package, and follow the README to build
> > > and run the Pulsar standalone service.
> > >
> > > Thanks,
> > > Haiting
> > >
> >
> >
> > --
> > BR,
> > Qiang Huang
> >

[GitHub] [pulsar] Anonymitaet edited a discussion: [Design] Pulsar All Releases Page

[GitBox]

GitHub user Anonymitaet edited a discussion: [Design] Pulsar All Releases Page

Hi team,

1. How about providing more useful content to the [Pulsar Release Notes > All 
Releases](https://pulsar.apache.org/release-notes/) page as 
[below](https://docs.google.com/spreadsheets/d/1XHmzno-1kbHvPp7eogri8XAiShH-EgpEG40svte-qxM/edit#gid=1677243891)?

https://user-images.githubusercontent.com/50226895/187336497-95a68232-36e5-4fc7-a0c0-7b3788ba78a6.png;>


2. This table should be updated automatically. 

Info | Info can be pulled from
---|---
Release note URL | https://pulsar.apache.org/release-notes/
Release blogs name and URL | https://pulsar.apache.org/blog
Documentation URL | [Versions page](https://pulsar.apache.org/versions), so 
this page can be removed since the info is consolidated into the `All Releases` 
page.
Release date  Release manager| https://github.com/apache/pulsar/releases.

☘️☘️☘️

Hi @tisonkun @urfreespace @SignorMercurio thoughts?


GitHub link: https://github.com/apache/pulsar/discussions/17310


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] Anonymitaet edited a discussion: [Design] Pulsar All Releases Page

[GitBox]

GitHub user Anonymitaet edited a discussion: [Design] Pulsar All Releases Page

Hi team,

Currently, we do not have a place showing the timeline for all Pulsar releases.

How about creating a Pulsar [Release Notes > 
Timeline](https://pulsar.apache.org/release-notes/timeline) page along with the 
following content?

1. Add the [Pulsar release timeline 
table](https://docs.google.com/spreadsheets/d/1XHmzno-1kbHvPp7eogri8XAiShH-EgpEG40svte-qxM/edit#gid=1677243891)
 as below.

https://user-images.githubusercontent.com/50226895/187153499-0689fdce-1ab9-40bd-8115-b4b816dd1f92.png;>

2. The timeline table should be updated automatically. Data (release date + 
release manager) can be pulled from https://github.com/apache/pulsar/releases.

3. [Versions page](https://pulsar.apache.org/versions) info is consolidated 
into the timeline page, so the Versions page can be removed.

☘️☘️☘️

Hi @tisonkun @urfreespace @SignorMercurio thoughts?


GitHub link: https://github.com/apache/pulsar/discussions/17310


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar-site] tisonkun commented on a diff in pull request #187: [fix][site] turn on all related code highlight

[GitBox]

tisonkun commented on code in PR #187:
URL: https://github.com/apache/pulsar-site/pull/187#discussion_r957944903


##
site2/website-next/docusaurus.config.js:
##
@@ -349,10 +349,17 @@ module.exports = {
   Copyright © ${new Date().getFullYear()} The Apache Software 
Foundation. All Rights Reserved. Apache, Pulsar, Apache Pulsar, and the Apache 
feather logo are trademarks or registered trademarks of The Apache Software 
Foundation.`,
 },
 prism: {
-  // theme: lightCodeTheme,
-  // darkTheme: darkCodeTheme,
   theme: require("prism-react-renderer/themes/dracula"),
-  additionalLanguages: ["powershell", "java", "go", "c", "cpp", "python"],
+  additionalLanguages: [
+"csharp",
+"groovy",
+"java",
+"ini",
+"powershell",
+"properties",
+"protobuf",
+"yaml",
+  ],

Review Comment:
   Related docusaurus guide:
   
   * 
https://docusaurus.io/docs/markdown-features/code-blocks#supported-languages
   * 
https://github.com/FormidableLabs/prism-react-renderer/blob/c1430679ee6cb0fb652d3a1b63f62f0c270ad631/src/vendor/prism/includeLangs.js
   * https://prismjs.com/#supported-languages



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] Anonymitaet commented on pull request #186: [fix][doc]Fix Elasticsearch compatibilityMode configurations

[GitBox]

Anonymitaet commented on PR #186:
URL: https://github.com/apache/pulsar-site/pull/186#issuecomment-1231064979

   @mendonk thanks for your contribution! Please submit your PR to 
https://github.com/apache/pulsar rather than here.
   Details see 
https://docs.google.com/document/d/1-1uJyd1k9_h56xiiVRVOnrLcCnTmg9n7SrHhNVNEEi4/edit#bookmark=id.6wysf15w7d4g


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] tisonkun commented on a diff in pull request #187: [fix][site] turn on all related code highlight

[GitBox]

tisonkun commented on code in PR #187:
URL: https://github.com/apache/pulsar-site/pull/187#discussion_r957942811


##
site2/website-next/docusaurus.config.js:
##
@@ -349,10 +349,17 @@ module.exports = {
   Copyright © ${new Date().getFullYear()} The Apache Software 
Foundation. All Rights Reserved. Apache, Pulsar, Apache Pulsar, and the Apache 
feather logo are trademarks or registered trademarks of The Apache Software 
Foundation.`,
 },
 prism: {
-  // theme: lightCodeTheme,
-  // darkTheme: darkCodeTheme,
   theme: require("prism-react-renderer/themes/dracula"),
-  additionalLanguages: ["powershell", "java", "go", "c", "cpp", "python"],
+  additionalLanguages: [
+"csharp",
+"groovy",
+"java",
+"ini",
+"powershell",
+"properties",
+"protobuf",
+"yaml",
+  ],

Review Comment:
   For the upstream changes, see https://github.com/apache/pulsar/pull/17342.



##
site2/website-next/docusaurus.config.js:
##
@@ -349,10 +349,17 @@ module.exports = {
   Copyright © ${new Date().getFullYear()} The Apache Software 
Foundation. All Rights Reserved. Apache, Pulsar, Apache Pulsar, and the Apache 
feather logo are trademarks or registered trademarks of The Apache Software 
Foundation.`,
 },
 prism: {
-  // theme: lightCodeTheme,
-  // darkTheme: darkCodeTheme,
   theme: require("prism-react-renderer/themes/dracula"),
-  additionalLanguages: ["powershell", "java", "go", "c", "cpp", "python"],
+  additionalLanguages: [
+"csharp",
+"groovy",
+"java",
+"ini",
+"powershell",
+"properties",
+"protobuf",
+"yaml",
+  ],

Review Comment:
   For the main repo changes, see https://github.com/apache/pulsar/pull/17342.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] tisonkun commented on a diff in pull request #187: [fix][site] turn on all related code highlight

[GitBox]

tisonkun commented on code in PR #187:
URL: https://github.com/apache/pulsar-site/pull/187#discussion_r957941579


##
site2/website-next/docusaurus.config.js:
##
@@ -349,10 +349,17 @@ module.exports = {
   Copyright © ${new Date().getFullYear()} The Apache Software 
Foundation. All Rights Reserved. Apache, Pulsar, Apache Pulsar, and the Apache 
feather logo are trademarks or registered trademarks of The Apache Software 
Foundation.`,
 },
 prism: {
-  // theme: lightCodeTheme,
-  // darkTheme: darkCodeTheme,
   theme: require("prism-react-renderer/themes/dracula"),
-  additionalLanguages: ["powershell", "java", "go", "c", "cpp", "python"],
+  additionalLanguages: [
+"csharp",
+"groovy",
+"java",
+"ini",
+"powershell",
+"properties",
+"protobuf",
+"yaml",
+  ],

Review Comment:
   @dave2wave this change should be in this repo.
   
   I pull in other changes so that reviewers can easily preview the final 
result. We carry changes under `docs` to the main repo to prevent regression by 
synching.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] tisonkun opened a new pull request, #187: [fix][site] turn on all related code highlight

[GitBox]

tisonkun opened a new pull request, #187:
URL: https://github.com/apache/pulsar-site/pull/187

   All changes to the `docs` folder should be carried to the main repo also.
   
   cc @urfreespace @dave2wave 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] urfreespace closed issue #177: Migrate all docs from `/tools` to `/reference`

[GitBox]

urfreespace closed issue #177: Migrate all docs from `/tools` to `/reference`
URL: https://github.com/apache/pulsar-site/issues/177


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] urfreespace merged pull request #184: Migrate all docs from /tools to /reference

[GitBox]

urfreespace merged PR #184:
URL: https://github.com/apache/pulsar-site/pull/184


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] urfreespace merged pull request #183: [build][site] cleanup previous content before copy new ones

[GitBox]

urfreespace merged PR #183:
URL: https://github.com/apache/pulsar-site/pull/183


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] mendonk opened a new pull request, #186: [fix][doc]Fix Elasticsearch compatibilityMode configurations

[GitBox]

mendonk opened a new pull request, #186:
URL: https://github.com/apache/pulsar-site/pull/186

   Corrects Elasticsearch `compatibilityMode` config values. 
   
   Motivation
   Clarifies the values that a user expects from compatibilityMode
   
   Modifications
   required=false and default value=AUTO
   
   Documentation
   [x ] doc
   (Your PR contains doc changes)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] michaeljmarshall commented on pull request #182: Generate 2.9.2, 2.9.3, 2.10.1 python docs

[GitBox]

michaeljmarshall commented on PR #182:
URL: https://github.com/apache/pulsar-site/pull/182#issuecomment-1230861282

   Thank you for catching my mistake @tisonkun. I fixed the script here 
https://github.com/apache/pulsar-site/pull/185.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] michaeljmarshall merged pull request #185: Make generated python docs go to website-next directory

[GitBox]

michaeljmarshall merged PR #185:
URL: https://github.com/apache/pulsar-site/pull/185


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] michaeljmarshall commented on pull request #185: Make generated python docs go to website-next directory

[GitBox]

michaeljmarshall commented on PR #185:
URL: https://github.com/apache/pulsar-site/pull/185#issuecomment-1230860667

   I manually tested this change to verify its correctness.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] michaeljmarshall opened a new pull request, #185: Make generated python docs go to website-next directory

[GitBox]

michaeljmarshall opened a new pull request, #185:
URL: https://github.com/apache/pulsar-site/pull/185

   Put generated python docs into the `website-next` directory, as @tisonkun 
explains in this comment 
https://github.com/apache/pulsar-site/pull/182#issuecomment-1230199254.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[DISCUSS] Introduce FlowControl to metrics endpoint

[Jiuming Tao]

Hi Lari,

I had considered about the `remote write API`, but there will be some problems:
1. Does all the Metrics Systems support Prometheus format remote write protocol
2. How to resolve extra label(podName etc.), I don’t know if the extra labels 
will be added atomically when  write to TSDB

Thanks,
Tao Jiuming

> 下面是被转发的邮件：
> 
> 发件人: Lari Hotari 
> 主题: 回复：[DISCUSS] Introduce FlowControl to metrics endpoint
> 日期: 2022年8月29日 GMT+8 下午6:40:38
> 收件人: dev@pulsar.apache.org
> 回复－收件人: dev@pulsar.apache.org
> 
> Good reiteration of the problem and good points, Asaf.
> 
> I'd like to add a new aspect to the proposal: there might be other
> solutions that would be useful in the case of large amount of topics in a
> Pulsar cluster.
> Rate limiting on the /metrics endpoint doesn't sound like the correct
> approach.
> 
> When there's a huge about of metrics, instead of scraping the metrics, it
> could be more useful to ingest the metrics to Prometheus using the "Remote
> write API".
> There's a recording of a talk explaining remote write in
> https://www.youtube.com/watch?v=vMeCyX3Y3HY .
> The specification is
> https://docs.google.com/document/d/1LPhVRSFkGNSuU1fBd81ulhsCPR4hkSZyyBj1SZ8fWOM/edit#
> .
> The benefit of this could be that /metrics endpoint wouldn't be a
> bottleneck and there wouldn't be a need to do any hacks to support a high
> number of metrics.
> There might be need to route the metrics for different namespaces/topics to
> different destinations. This could be handled in the implementation that
> uses the Remote write API for pushing metrics.
> 
> Regards,
> 
> -Lari
> 
> 
> On Mon, Aug 29, 2022 at 1:12 PM Asaf Mesika  wrote:
> 
>> Hi Jiuming,
>> 
>> I would reiterate the problem statement to make it clear (at least for me):
>> 
>> There are cases where a very large amount of topics exists (> 10k per
>> broker) and are used in Pulsar. Those topics usually have multiple
>> producers and multiple consumers.
>> There are metrics that are in the granularity of topics and also in
>> topic/producers and topic/consumers granularity.
>> When that happens, the amount of unique metrics is severely high which
>> causes the response size of /metrics endpoint (the Prometheus Exposition
>> Format endpoint) to be substantially high - 200MB - 500MB.
>> 
>> Every time the metrics are scraped (30sec, or 1 min), the network gets
>> surged up due to the /metrics response, thereby causing latency to messages
>> produced or consumed from that broker.
>> 
>> The solution proposed is to throttle the /metrics response based on a
>> pre-configured rate limit.
>> 
>> Points to consider for this discussion from the participants:
>> 
>> 1. Did you happen to experience such difficulties in your clusters?
>> 2. When that happened, did you experience the bottleneck also on the TSDB
>> be it metrics ingestion or querying?
>> 
>> Thanks,
>> 
>> Asaf
>> 
>> 
>> On Thu, Aug 18, 2022 at 7:40 PM Jiuming Tao >> 
>> wrote:
>> 
>>> bump
>>> 
>>> Jiuming Tao  于2022年8月8日周一 18:19写道：
>>> 
 Hi Pulsar community,
 
 In the situation of expose metrics data which has a big size, it will
>>> lead
 to:
 1. A sudden increase of network usage
 2. The latency of pub/sub rising
 
 For the purpose of resolving these problems, I’d opened a PR:
 https://github.com/apache/pulsar/pull/16452
 
 Please feel free to help review/discuss about it.
 
 Thanks,
 Tao Jiuming
 
>>> 
>> 

Re: [DISCUSS] Move PIPs to the codebase?

[Lari Hotari]

Is there a specific reason to put the PIPs in the apache/pulsar repository?

I think that it will add unnecessary cruft to our core source code repository.

Could we instead create a separate repository to hold the PIP files? The 
example of Rust lang has a separate repository too, 
https://github.com/rust-lang/rfcs.

I'd suggest that we create https://github.com/apache/pulsar-pips to hold the 
PIPs.

The benefit of this is that we won't be triggering workflows when modifying the 
PIP files. 
This would also open new opportunities like publishing a static website that 
include the rendered view of the PIP files etc.

-Lari


On 2022/08/29 15:38:46 PengHui Li wrote:
> ```
> A sub-team makes final decisions about RFCs after the benefits and
> drawbacks are well understood. These decisions can be made at any time, but
> the sub-team will regularly issue decisions. When a decision is made, the
> RFC pull request will either be merged or closed. In either case, if the
> reasoning is not clear from the discussion in thread, the sub-team will add
> a comment describing the rationale for the decision.
> ```
> from https://github.com/rust-lang/rfcs#reviewing-rfcs
> 
> I think they merged all the approved proposals?
> 
> https://github.com/rust-lang/rfcs/pulls?q=is%3Apr+is%3Amerged+
> 
> And I can also find the md files
> 
> https://github.com/rust-lang/rfcs/tree/master/text
> 
> Thanks,
> Penghui
> 
> On Mon, Aug 29, 2022 at 11:33 PM PengHui Li  wrote:
> 
> > > The only point worth attending to while drafting the new process is how
> > to
> > avoid neglected out-of-date proposals. If a proposal was declined, who is
> > in charge to update its status in the readme?
> >
> > That is a good point. IMO, everyone is able to correct the proposal status.
> > It wasn't easy for us before, only the committer could update the wiki
> > page.
> > Of course, in most cases, it is done by the author. But this is not 100%
> > guaranteed.
> > We should add this part to the proposal process.
> >
> > > There is something appealing in not merging a PR proposal, as this status
> > update takes care of itself (borrowing from rust RFC).
> >
> > The problem I thought of not merging the PR proposal is the proposal looks
> > like the proposed changes never end. The author can continue to update
> > after approved. I don't want to say all the changes to the proposal need
> > to be voted on
> > the mailing list, but we should get a chance to review the changes.
> >
> > Thanks,
> > Penghui
> >
> > On Mon, Aug 29, 2022 at 10:45 PM Asaf Mesika 
> > wrote:
> >
> >> I like the idea of keeping the suggestions as files in the repo since as
> >> you mentioned, it makes the review process using PRs much more
> >> streamlined.
> >>
> >> I think keeping the status in an MD file and only there (having a single
> >> source of truth) will make it less error-prone (people might forget to
> >> move
> >> between directories) , and also easier to have a single page to view all
> >> proposals.
> >>
> >> The only point worth attending to while drafting the new process is how to
> >> avoid neglected out-of-date proposals. If a proposal was declined, who is
> >> in charge to update its status in the readme?
> >> There is something appealing in not merging a PR proposal, as this status
> >> update takes care of itself (borrowing from rust RFC).
> >>
> >> One bonus item is that I learned through this long discussion thread about
> >> Zulip chat :)
> >>
> >>
> >>
> >> On Mon, Aug 29, 2022 at 1:55 PM PengHui Li  wrote:
> >>
> >> > Hi all,
> >> >
> >> > I will merge https://github.com/apache/pulsar/pull/17270 first and
> >> draft a
> >> > proposal for
> >> > the detailed PIP process changes. And will start a new VOTE thread for
> >> the
> >> > PIP process change.
> >> >
> >> > After the proposal(for PIP process change) is approved. I will go back
> >> here
> >> > to discuss
> >> > how to migrate the old PIPs to the codebase(because we need to follow
> >> > the new format of PIPs).
> >> >
> >> > Thanks,
> >> > Penghui
> >> >
> >> > On Fri, Aug 26, 2022 at 11:20 PM PengHui Li  wrote:
> >> >
> >> > > > Using a directory structure to organize PIP status might make the
> >> git
> >> > > history a bit less direct because changing state is equivalent with a
> >> > > file move instead of a line updated in a file. However, if we do it
> >> > > that way, we could have a README.md file organizing PIP metadata and
> >> > > linking to the actual PIP file in the directory structure. That
> >> > > README.md would also serve as the source of truth for PIP numbers
> >> > > because each PIP pointer would get its associated line number. Then,
> >> > > concurrent PIPs would need to resolve merge conflicts just before
> >> > > merging and only then would they acquire their number.
> >> > >
> >> > > Oh, get your point, Michael. I think this solution looks better. We
> >> can
> >> > > also
> >> > > add something in README.md and users can also get the complete
> >> > > proposal list here. In 

Re: Pulsar CI congested, master branch build broken

[Lari Hotari]

master branch is broken once again. Here's the fix:
https://github.com/apache/pulsar/pull/17339

Please review and merge

-Lari

On 2022/08/26 12:00:20 Lari Hotari wrote:
> Hi,
> 
> GitHub Actions builds have been piling up in the build queue in the last few 
> days.
> I posted on bui...@apache.org 
> https://lists.apache.org/thread/6lbqr0f6mqt9s8ggollp5kj2nv7rlo9s and created 
> INFRA ticket https://issues.apache.org/jira/browse/INFRA-23633 about this 
> issue.
> There's also a thread on the-asf slack, 
> https://the-asf.slack.com/archives/CBX4TSBQ8/p1661512133913279 . 
> 
> It seems that our build queue is finally getting picked up, but it would be 
> great to see if we hit quota and whether that is the cause of pauses. 
> 
> Another issue is that the master branch broke after merging 2 conflicting 
> PRs. 
> The fix is in https://github.com/apache/pulsar/pull/17300 . 
> 
> Merging PRs will be slow until we have these 2 problems solved and existing 
> PRs rebased over the changes. Let's prioritize merging #17300 before pushing 
> more changes.
> 
> I'd like to point out that a good way to get build feedback before sending a 
> PR, is to run builds on your personal GitHub Actions CI. The benefit of this 
> is that it doesn't consume the shared quota and builds usually start 
> instantly.
> There are instructions in the contributors guide about this. 
> https://pulsar.apache.org/contributing/#ci-testing-in-your-fork
> You simply open PRs to your own fork of apache/pulsar to run builds on your 
> personal GitHub Actions CI.
> 
> BR,
> 
> Lari
> 
> 
> 
> 
> 
> 
> 
> 

Re: [DISCUSS] Move PIPs to the codebase?

[PengHui Li]

```
A sub-team makes final decisions about RFCs after the benefits and
drawbacks are well understood. These decisions can be made at any time, but
the sub-team will regularly issue decisions. When a decision is made, the
RFC pull request will either be merged or closed. In either case, if the
reasoning is not clear from the discussion in thread, the sub-team will add
a comment describing the rationale for the decision.
```
from https://github.com/rust-lang/rfcs#reviewing-rfcs

I think they merged all the approved proposals?

https://github.com/rust-lang/rfcs/pulls?q=is%3Apr+is%3Amerged+

And I can also find the md files

https://github.com/rust-lang/rfcs/tree/master/text

Thanks,
Penghui

On Mon, Aug 29, 2022 at 11:33 PM PengHui Li  wrote:

> > The only point worth attending to while drafting the new process is how
> to
> avoid neglected out-of-date proposals. If a proposal was declined, who is
> in charge to update its status in the readme?
>
> That is a good point. IMO, everyone is able to correct the proposal status.
> It wasn't easy for us before, only the committer could update the wiki
> page.
> Of course, in most cases, it is done by the author. But this is not 100%
> guaranteed.
> We should add this part to the proposal process.
>
> > There is something appealing in not merging a PR proposal, as this status
> update takes care of itself (borrowing from rust RFC).
>
> The problem I thought of not merging the PR proposal is the proposal looks
> like the proposed changes never end. The author can continue to update
> after approved. I don't want to say all the changes to the proposal need
> to be voted on
> the mailing list, but we should get a chance to review the changes.
>
> Thanks,
> Penghui
>
> On Mon, Aug 29, 2022 at 10:45 PM Asaf Mesika 
> wrote:
>
>> I like the idea of keeping the suggestions as files in the repo since as
>> you mentioned, it makes the review process using PRs much more
>> streamlined.
>>
>> I think keeping the status in an MD file and only there (having a single
>> source of truth) will make it less error-prone (people might forget to
>> move
>> between directories) , and also easier to have a single page to view all
>> proposals.
>>
>> The only point worth attending to while drafting the new process is how to
>> avoid neglected out-of-date proposals. If a proposal was declined, who is
>> in charge to update its status in the readme?
>> There is something appealing in not merging a PR proposal, as this status
>> update takes care of itself (borrowing from rust RFC).
>>
>> One bonus item is that I learned through this long discussion thread about
>> Zulip chat :)
>>
>>
>>
>> On Mon, Aug 29, 2022 at 1:55 PM PengHui Li  wrote:
>>
>> > Hi all,
>> >
>> > I will merge https://github.com/apache/pulsar/pull/17270 first and
>> draft a
>> > proposal for
>> > the detailed PIP process changes. And will start a new VOTE thread for
>> the
>> > PIP process change.
>> >
>> > After the proposal(for PIP process change) is approved. I will go back
>> here
>> > to discuss
>> > how to migrate the old PIPs to the codebase(because we need to follow
>> > the new format of PIPs).
>> >
>> > Thanks,
>> > Penghui
>> >
>> > On Fri, Aug 26, 2022 at 11:20 PM PengHui Li  wrote:
>> >
>> > > > Using a directory structure to organize PIP status might make the
>> git
>> > > history a bit less direct because changing state is equivalent with a
>> > > file move instead of a line updated in a file. However, if we do it
>> > > that way, we could have a README.md file organizing PIP metadata and
>> > > linking to the actual PIP file in the directory structure. That
>> > > README.md would also serve as the source of truth for PIP numbers
>> > > because each PIP pointer would get its associated line number. Then,
>> > > concurrent PIPs would need to resolve merge conflicts just before
>> > > merging and only then would they acquire their number.
>> > >
>> > > Oh, get your point, Michael. I think this solution looks better. We
>> can
>> > > also
>> > > add something in README.md and users can also get the complete
>> > > proposal list here. In the future, maybe we can show it on the
>> website.
>> > >
>> > > Best,
>> > > Penghui
>> > >
>> > > On Fri, Aug 26, 2022 at 12:16 PM Michael Marshall <
>> mmarsh...@apache.org>
>> > > wrote:
>> > >
>> > >> > We should move to the codebase 100% the same as the original
>> > >> documentation.
>> > >> > And use another PR to make improvements for it. If it is needed, we
>> > >> should
>> > >> > start with an email.
>> > >> > We need to have historical records.
>> > >>
>> > >> +1 I think this is a great idea. It makes sense to copy them verbatim
>> > >> and then worry about updating them in a second step.
>> > >>
>> > >> Using a directory structure to organize PIP status might make the git
>> > >> history a bit less direct because changing state is equivalent with a
>> > >> file move instead of a line updated in a file. However, if we do it
>> > >> that way, we could have a 

Re: [DISCUSS] Move PIPs to the codebase?

[PengHui Li]

> The only point worth attending to while drafting the new process is how to
avoid neglected out-of-date proposals. If a proposal was declined, who is
in charge to update its status in the readme?

That is a good point. IMO, everyone is able to correct the proposal status.
It wasn't easy for us before, only the committer could update the wiki page.
Of course, in most cases, it is done by the author. But this is not 100%
guaranteed.
We should add this part to the proposal process.

> There is something appealing in not merging a PR proposal, as this status
update takes care of itself (borrowing from rust RFC).

The problem I thought of not merging the PR proposal is the proposal looks
like the proposed changes never end. The author can continue to update
after approved. I don't want to say all the changes to the proposal need to
be voted on
the mailing list, but we should get a chance to review the changes.

Thanks,
Penghui

On Mon, Aug 29, 2022 at 10:45 PM Asaf Mesika  wrote:

> I like the idea of keeping the suggestions as files in the repo since as
> you mentioned, it makes the review process using PRs much more streamlined.
>
> I think keeping the status in an MD file and only there (having a single
> source of truth) will make it less error-prone (people might forget to move
> between directories) , and also easier to have a single page to view all
> proposals.
>
> The only point worth attending to while drafting the new process is how to
> avoid neglected out-of-date proposals. If a proposal was declined, who is
> in charge to update its status in the readme?
> There is something appealing in not merging a PR proposal, as this status
> update takes care of itself (borrowing from rust RFC).
>
> One bonus item is that I learned through this long discussion thread about
> Zulip chat :)
>
>
>
> On Mon, Aug 29, 2022 at 1:55 PM PengHui Li  wrote:
>
> > Hi all,
> >
> > I will merge https://github.com/apache/pulsar/pull/17270 first and
> draft a
> > proposal for
> > the detailed PIP process changes. And will start a new VOTE thread for
> the
> > PIP process change.
> >
> > After the proposal(for PIP process change) is approved. I will go back
> here
> > to discuss
> > how to migrate the old PIPs to the codebase(because we need to follow
> > the new format of PIPs).
> >
> > Thanks,
> > Penghui
> >
> > On Fri, Aug 26, 2022 at 11:20 PM PengHui Li  wrote:
> >
> > > > Using a directory structure to organize PIP status might make the git
> > > history a bit less direct because changing state is equivalent with a
> > > file move instead of a line updated in a file. However, if we do it
> > > that way, we could have a README.md file organizing PIP metadata and
> > > linking to the actual PIP file in the directory structure. That
> > > README.md would also serve as the source of truth for PIP numbers
> > > because each PIP pointer would get its associated line number. Then,
> > > concurrent PIPs would need to resolve merge conflicts just before
> > > merging and only then would they acquire their number.
> > >
> > > Oh, get your point, Michael. I think this solution looks better. We can
> > > also
> > > add something in README.md and users can also get the complete
> > > proposal list here. In the future, maybe we can show it on the website.
> > >
> > > Best,
> > > Penghui
> > >
> > > On Fri, Aug 26, 2022 at 12:16 PM Michael Marshall <
> mmarsh...@apache.org>
> > > wrote:
> > >
> > >> > We should move to the codebase 100% the same as the original
> > >> documentation.
> > >> > And use another PR to make improvements for it. If it is needed, we
> > >> should
> > >> > start with an email.
> > >> > We need to have historical records.
> > >>
> > >> +1 I think this is a great idea. It makes sense to copy them verbatim
> > >> and then worry about updating them in a second step.
> > >>
> > >> Using a directory structure to organize PIP status might make the git
> > >> history a bit less direct because changing state is equivalent with a
> > >> file move instead of a line updated in a file. However, if we do it
> > >> that way, we could have a README.md file organizing PIP metadata and
> > >> linking to the actual PIP file in the directory structure. That
> > >> README.md would also serve as the source of truth for PIP numbers
> > >> because each PIP pointer would get its associated line number. Then,
> > >> concurrent PIPs would need to resolve merge conflicts just before
> > >> merging and only then would they acquire their number.
> > >>
> > >> - Michael
> > >>
> > >> On Thu, Aug 25, 2022 at 9:15 PM PengHui Li 
> wrote:
> > >> >
> > >> > Hi Xiangying,
> > >> >
> > >> > > We can classify the pips under these folders according to the
> pulsar
> > >> > modules, instead of just placing these pips under these folders in
> an
> > >> > incrementing sequence number.
> > >> >
> > >> > I think it's not easy to do this. A proposal can have changes
> related
> > to
> > >> > multiple
> > >> > modules(Broker, Metadata, Client).
> > >> 

Re: [VOTE] Pulsar Release 2.8.4 Candidate 1

[Nicolò Boschi]

+1 (non binding)

Checks:

- Checksum and signatures

- Apache Rat check passes

- Compile from source

- Run Pulsar standalone and produce-consume from CLI

- Tested K8S installation with Datastax Pulsar helm chart and verified TLS,
offloads and ElasticSearch sink



Nicolò Boschi


Il giorno lun 29 ago 2022 alle ore 17:20 Yunze Xu
 ha scritto:

> Hi everyone, there are already two binding +1. Could anyone else help
> verify it?
>
> Thanks,
> Yunze
>
>
>
>
> > 2022年8月23日 21:26，PengHui Li  写道：
> >
> > +1 (binding)
> >
> > - Start the standalone service
> > - Publish and consume messages
> > - Run the Cassandra connector
> > - Validate the stateful function
> >
> > Thanks,
> > Penghui
> >
> > On Tue, Aug 23, 2022 at 9:57 AM guo jiwei  wrote:
> >
> >> +1 (binding)
> >>
> >> - Checked checksums and signatures
> >> - Checked license headers using Apache Rat
> >> - Compiled the source by JDK11
> >> - Ran the standalone server
> >> - Confirmed that producer and consumer work properly
> >> - Validated functions, connectors, and stateful functions
> >>
> >>
> >> Regards
> >> Jiwei Guo (Tboy)
> >>
> >>
> >> On Mon, Aug 15, 2022 at 10:18 AM Qiang Huang  >
> >> wrote:
> >>
> >>> Got it. Thx.
> >>>
> >>> Yunze Xu  于2022年8月14日周日 23:22写道：
> >>>
>  You can see
>  https://lists.apache.org/thread/rg1g083c06ozm5go6zo1jophg9y9zl2f
>  for more details about the LTS release.
> 
>  Thanks,
>  Yunze
> 
> 
> 
> 
> > 2022年8月14日 11:00，Qiang Huang  写道：
> >
> > +1 (non-binding)
> > Is 2.8.4 a long term support release?
> >
> > Yunze Xu  于2022年8月12日周五 16:20写道：
> >
> >> This is the first release candidate for Apache Pulsar, version
> >> 2.8.4.
> >>
> >> It fixes the following issues:
> >>
> >>
> 
> >>>
> >>
> https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.8.4
> >>
> >> *** Please download, test and vote on this release. This vote will
> >>> stay
> >> open
> >> for at least 72 hours ***
> >>
> >> Note that we are voting upon the source (tag), binaries are provided
> >>> for
> >> convenience.
> >>
> >> Source and binary files:
> >>
> >>>
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.4-candidate-1/
> >>
> >> SHA-512 checksums:
> >>
> >>
> 
> >>>
> >>
> c3d26704f2cfb3365c29d4110612ca7351084f8bee3c306d5e906b3f9b22c7557cc5baf12f74f8c222baccae3310691419eda5b47fdf9cd6c5281b70134ab5eb
> >> apache-pulsar-2.8.4-bin.tar.gz
> >>
> 
> >>>
> >>
> 28160ee94dccfb74dfb56e0e5d0e08870c6612659507333a52b5660ecbf060a75d1eed667cffd8596f9468de95055b78916b932db0e0d4c2745868d55429ee98
> >> apache-pulsar-2.8.4-src.tar.gz
> >>
> >> Maven staging repo:
> >>
> 
> >>
> https://repository.apache.org/content/repositories/orgapachepulsar-1170/
> >>
> >> The tag to be voted upon:
> >> v2.8.4-candidate-1 (02ee5616866d4eda8dd94f85d9d9b71c459f248d)
> >> https://github.com/apache/pulsar/releases/tag/v2.8.4-candidate-1
> >>
> >> Pulsar's KEYS file containing PGP keys we use to sign the release:
> >> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >>
> >> Docker images:
> >>
> >>
> >>
> 
> >>>
> >>
> https://hub.docker.com/layers/pulsar/bewaremypower/pulsar/2.8.4/images/sha256-fba51a75c0f2ca79fbff7b254f80f641fcda661fd702f8149bbfdd5994078e3a
> >>
> >>
> >>
> 
> >>>
> >>
> https://hub.docker.com/layers/pulsar-all/bewaremypower/pulsar-all/2.8.4/images/sha256-42d4b41e5869edc6242bb49d6a1687bd6d191a6385637122edc5c7b2c44ee46f
> >>
> >> Please download the source package, and follow the Release Candidate
> >> Validation[1] to validate the release
> >>
> >> [1]
> >>> https://github.com/apache/pulsar/wiki/Release-Candidate-Validation
> >>
> >> Thanks,
> >> Yunze
> >>
> >>
> >>
> >>
> >>
> >
> > --
> > BR,
> > Qiang Huang
> 
> 
> >>>
> >>> --
> >>> BR,
> >>> Qiang Huang
> >>>
> >>
>
>

Re: [VOTE] Pulsar Release 2.8.4 Candidate 1

[Yunze Xu]

Hi everyone, there are already two binding +1. Could anyone else help verify it?

Thanks,
Yunze




> 2022年8月23日 21:26，PengHui Li  写道：
> 
> +1 (binding)
> 
> - Start the standalone service
> - Publish and consume messages
> - Run the Cassandra connector
> - Validate the stateful function
> 
> Thanks,
> Penghui
> 
> On Tue, Aug 23, 2022 at 9:57 AM guo jiwei  wrote:
> 
>> +1 (binding)
>> 
>> - Checked checksums and signatures
>> - Checked license headers using Apache Rat
>> - Compiled the source by JDK11
>> - Ran the standalone server
>> - Confirmed that producer and consumer work properly
>> - Validated functions, connectors, and stateful functions
>> 
>> 
>> Regards
>> Jiwei Guo (Tboy)
>> 
>> 
>> On Mon, Aug 15, 2022 at 10:18 AM Qiang Huang 
>> wrote:
>> 
>>> Got it. Thx.
>>> 
>>> Yunze Xu  于2022年8月14日周日 23:22写道：
>>> 
 You can see
 https://lists.apache.org/thread/rg1g083c06ozm5go6zo1jophg9y9zl2f
 for more details about the LTS release.
 
 Thanks,
 Yunze
 
 
 
 
> 2022年8月14日 11:00，Qiang Huang  写道：
> 
> +1 (non-binding)
> Is 2.8.4 a long term support release?
> 
> Yunze Xu  于2022年8月12日周五 16:20写道：
> 
>> This is the first release candidate for Apache Pulsar, version
>> 2.8.4.
>> 
>> It fixes the following issues:
>> 
>> 
 
>>> 
>> https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.8.4
>> 
>> *** Please download, test and vote on this release. This vote will
>>> stay
>> open
>> for at least 72 hours ***
>> 
>> Note that we are voting upon the source (tag), binaries are provided
>>> for
>> convenience.
>> 
>> Source and binary files:
>> 
>>> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.8.4-candidate-1/
>> 
>> SHA-512 checksums:
>> 
>> 
 
>>> 
>> c3d26704f2cfb3365c29d4110612ca7351084f8bee3c306d5e906b3f9b22c7557cc5baf12f74f8c222baccae3310691419eda5b47fdf9cd6c5281b70134ab5eb
>> apache-pulsar-2.8.4-bin.tar.gz
>> 
 
>>> 
>> 28160ee94dccfb74dfb56e0e5d0e08870c6612659507333a52b5660ecbf060a75d1eed667cffd8596f9468de95055b78916b932db0e0d4c2745868d55429ee98
>> apache-pulsar-2.8.4-src.tar.gz
>> 
>> Maven staging repo:
>> 
 
>> https://repository.apache.org/content/repositories/orgapachepulsar-1170/
>> 
>> The tag to be voted upon:
>> v2.8.4-candidate-1 (02ee5616866d4eda8dd94f85d9d9b71c459f248d)
>> https://github.com/apache/pulsar/releases/tag/v2.8.4-candidate-1
>> 
>> Pulsar's KEYS file containing PGP keys we use to sign the release:
>> https://dist.apache.org/repos/dist/dev/pulsar/KEYS
>> 
>> Docker images:
>> 
>> 
>> 
 
>>> 
>> https://hub.docker.com/layers/pulsar/bewaremypower/pulsar/2.8.4/images/sha256-fba51a75c0f2ca79fbff7b254f80f641fcda661fd702f8149bbfdd5994078e3a
>> 
>> 
>> 
 
>>> 
>> https://hub.docker.com/layers/pulsar-all/bewaremypower/pulsar-all/2.8.4/images/sha256-42d4b41e5869edc6242bb49d6a1687bd6d191a6385637122edc5c7b2c44ee46f
>> 
>> Please download the source package, and follow the Release Candidate
>> Validation[1] to validate the release
>> 
>> [1]
>>> https://github.com/apache/pulsar/wiki/Release-Candidate-Validation
>> 
>> Thanks,
>> Yunze
>> 
>> 
>> 
>> 
>> 
> 
> --
> BR,
> Qiang Huang
 
 
>>> 
>>> --
>>> BR,
>>> Qiang Huang
>>> 
>> 

Re: [DISCUSS] Move PIPs to the codebase?

[Asaf Mesika]

I like the idea of keeping the suggestions as files in the repo since as
you mentioned, it makes the review process using PRs much more streamlined.

I think keeping the status in an MD file and only there (having a single
source of truth) will make it less error-prone (people might forget to move
between directories) , and also easier to have a single page to view all
proposals.

The only point worth attending to while drafting the new process is how to
avoid neglected out-of-date proposals. If a proposal was declined, who is
in charge to update its status in the readme?
There is something appealing in not merging a PR proposal, as this status
update takes care of itself (borrowing from rust RFC).

One bonus item is that I learned through this long discussion thread about
Zulip chat :)



On Mon, Aug 29, 2022 at 1:55 PM PengHui Li  wrote:

> Hi all,
>
> I will merge https://github.com/apache/pulsar/pull/17270 first and draft a
> proposal for
> the detailed PIP process changes. And will start a new VOTE thread for the
> PIP process change.
>
> After the proposal(for PIP process change) is approved. I will go back here
> to discuss
> how to migrate the old PIPs to the codebase(because we need to follow
> the new format of PIPs).
>
> Thanks,
> Penghui
>
> On Fri, Aug 26, 2022 at 11:20 PM PengHui Li  wrote:
>
> > > Using a directory structure to organize PIP status might make the git
> > history a bit less direct because changing state is equivalent with a
> > file move instead of a line updated in a file. However, if we do it
> > that way, we could have a README.md file organizing PIP metadata and
> > linking to the actual PIP file in the directory structure. That
> > README.md would also serve as the source of truth for PIP numbers
> > because each PIP pointer would get its associated line number. Then,
> > concurrent PIPs would need to resolve merge conflicts just before
> > merging and only then would they acquire their number.
> >
> > Oh, get your point, Michael. I think this solution looks better. We can
> > also
> > add something in README.md and users can also get the complete
> > proposal list here. In the future, maybe we can show it on the website.
> >
> > Best,
> > Penghui
> >
> > On Fri, Aug 26, 2022 at 12:16 PM Michael Marshall 
> > wrote:
> >
> >> > We should move to the codebase 100% the same as the original
> >> documentation.
> >> > And use another PR to make improvements for it. If it is needed, we
> >> should
> >> > start with an email.
> >> > We need to have historical records.
> >>
> >> +1 I think this is a great idea. It makes sense to copy them verbatim
> >> and then worry about updating them in a second step.
> >>
> >> Using a directory structure to organize PIP status might make the git
> >> history a bit less direct because changing state is equivalent with a
> >> file move instead of a line updated in a file. However, if we do it
> >> that way, we could have a README.md file organizing PIP metadata and
> >> linking to the actual PIP file in the directory structure. That
> >> README.md would also serve as the source of truth for PIP numbers
> >> because each PIP pointer would get its associated line number. Then,
> >> concurrent PIPs would need to resolve merge conflicts just before
> >> merging and only then would they acquire their number.
> >>
> >> - Michael
> >>
> >> On Thu, Aug 25, 2022 at 9:15 PM PengHui Li  wrote:
> >> >
> >> > Hi Xiangying,
> >> >
> >> > > We can classify the pips under these folders according to the pulsar
> >> > modules, instead of just placing these pips under these folders in an
> >> > incrementing sequence number.
> >> >
> >> > I think it's not easy to do this. A proposal can have changes related
> to
> >> > multiple
> >> > modules(Broker, Metadata, Client).
> >> >
> >> > Thanks,
> >> > Penghui
> >> >
> >> > On Fri, Aug 26, 2022 at 9:20 AM Xiangying Meng 
> >> wrote:
> >> >
> >> > > Hi Penghui
> >> > > Thanks for you start this discussion. IMO, It is also a good way for
> >> > > beginners to learn the design and implementation of each module of
> >> Pulsar.
> >> > > > 3. /wiki/pip/accepted - for PIPs that have been accepted and are
> >> works in
> >> > > progress
> >> > > > 4. /wiki/pip/complete - for PIPs that have been completed.
> >> > > > 5. /wiki/pip/rejected - for PIPs that were proposed, but then
> >> rejected or
> >> > > abandoned.
> >> > > We can classify the pips under these folders according to the pulsar
> >> > > modules, instead of just placing these pips under these folders in
> an
> >> > > incrementing sequence number.
> >> > >
> >> > > In this way, readers can create a new local branch dedicated to
> >> reading and
> >> > > annotating proposals for themselves to read proposals they are
> >> interested
> >> > > in and write their own understanding and comments anytime and
> >> anywhere.
> >> > > Thanks,
> >> > > Xiangying Meng
> >> > >
> >> > > On Fri, Aug 26, 2022 at 12:23 AM PengHui Li 
> >> wrote:
> >> > >
> >> > > > Hi Dave,
> >> > > 

[GitHub] [pulsar] syhily edited a comment on the discussion: Flink Connector Exception: Failed to list subscribed topic partitions due to

[GitBox]

GitHub user syhily edited a comment on the discussion: Flink Connector 
Exception: Failed to list subscribed topic partitions due to

@BohanZhang0222 I think the root cause should be some performance issues on 
Pulsar side. Since flink connector requests a lot of metadata informations in 
an extremely short period, the 5xx error could happen easily. We will add rate 
limit and retry logic on connector side. But the core fix should be 
accomplished on the Pulsar side.

You can track the fix progress on the connector here. 
https://github.com/streamnative/flink/issues/162

cc @codelipenghui

GitHub link: 
https://github.com/apache/pulsar/discussions/17200#discussioncomment-3500901


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] syhily added a comment to the discussion: Flink Connector Exception: Failed to list subscribed topic partitions due to

[GitBox]

GitHub user syhily added a comment to the discussion: Flink Connector 
Exception: Failed to list subscribed topic partitions due to

@BohanZhang0222 I think the root cause should be some performance issues on 
Pulsar side. Since flink connector requests a lot of metadata informations in 
an extremely short period, the 5xx error could happen easily. We will add rate 
limit and retry logic on connector side. But the core fix should be 
accomplished on the Pulsar side.

cc @codelipenghui

GitHub link: 
https://github.com/apache/pulsar/discussions/17200#discussioncomment-3500901


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

Re: [VOTE] Pulsar Release 2.7.5 Candidate 3

[Nicolò Boschi]

+1 (non binding)

Checks:

- Checksum and signatures

- Apache Rat check passes

- Compile from source

- Run Pulsar standalone and produce-consume from CLI


Nicolò Boschi


Il giorno sab 27 ago 2022 alle ore 17:12 Qiang Huang <
qiang.huang1...@gmail.com> ha scritto:

> +1(non-binding)
> - Checked signatures/checksums
> - Checked the license headers
> - Ran the standalone
> - Validate Connectors
> - Validate Pub/Sub and Java Functions
> - Validate Stateful Functions
>
> Haiting Jiang  于2022年8月27日周六 06:40写道：
>
> > This is the third release candidate for Apache Pulsar, version 2.7.5.
> >
> > It fixes the following issues:
> > https://github.com/apache/pulsar/compare/v2.7.4...v2.7.5-candidate-3
> >
> > *** Please download, test and vote on this release. This vote will stay
> > open
> > for at least 72 hours ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.5-candidate-3/
> >
> > SHA-512 checksums:
> >
> >
> >
> 6aa74cb742c411edd40de2199eb70d774d0ce8cb7dbfc2ce33fce7e87949aafa73f93a1e154261b58a1df4ad10160bf43fc5a93b1aa9a0466fcf2a3ba1a6e385
> > apache-pulsar-2.7.5-bin.tar.gz
> >
> >
> facb1698e394b5428a0b9b6d71b891013f3c5b473cf72dd68a75ca729800710f2c6c476b1c47f7804c1b1c0941f79a6a9f334c7c703f66f6bac8f399386c5ad6
> > apache-pulsar-2.7.5-src.tar.gz
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachepulsar-1172/
> >
> > The tag to be voted upon:
> > v2.7.5-candidate-3 (8eae5b8d572861e49c40d456b1f3cbc5d414afe1)
> > https://github.com/apache/pulsar/releases/tag/v2.7.5-candidate-3
> >
> > Docker images:
> >
> >
> >
> https://hub.docker.com/layers/jason918/pulsar/2.7.5/images/sha256-ec77603bd943f1a56065d155428d6edccb2bf2ec57fafab8112c4b73c7bfcb6e
> >
> >
> https://hub.docker.com/layers/pulsar-all/jason918/pulsar-all/latest/images/sha256-0792a0ec75ecbc16c9a42af127a68b3dea2331be3f2f0599f5fb1f4cd78def25
> >
> > Pulsar's KEYS file containing PGP keys we use to sign the release:
> > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> >
> > Please download the source package, and follow the README to build
> > and run the Pulsar standalone service.
> >
> > Thanks,
> > Haiting
> >
>
>
> --
> BR,
> Qiang Huang
>

Re: [VOTE] PIP-200 Package Pulsar Trino distro and config in a dedicated folder

[Nicolò Boschi]

+1 (non binding)
Thanks for taking care of this

Nicolò Boschi


Il giorno lun 29 ago 2022 alle ore 15:26 tison  ha
scritto:

> Hi devs,
>
> This is the official thread VOTE for PIP-200 Package Pulsar Trino distro
> and config in a dedicated folder.
>
> Here is the PIP issue: https://github.com/apache/pulsar/issues/17137
> Here is the discussion thread:
> https://lists.apache.org/thread/s985ypf0r0hzcm0mx653n5h2rt7n273v
> Here is the link to the draft PR:
> https://github.com/apache/pulsar/pull/17062
>
> Voting will stay open for at least 72h.
>
> Best,
> tison.
>

[GitHub] [pulsar] syhily added a comment to the discussion: Flink Connector Exception: Failed to list subscribed topic partitions due to

[GitBox]

GitHub user syhily added a comment to the discussion: Flink Connector 
Exception: Failed to list subscribed topic partitions due to

It's tracked here. https://github.com/streamnative/flink/issues/162

GitHub link: 
https://github.com/apache/pulsar/discussions/17200#discussioncomment-3500858


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

Re: [DISCUSS] PIP-200 Package Pulsar Trino distro and config in a dedicated folder

[tison]

Here is the vote thread:
https://lists.apache.org/thread/9stw26rrp9hm6l2xd2p78gpogw9bywgc

You're welcome to vote on the thread.

Best,
tison.


tison  于2022年8月26日周五 09:06写道：

> Thanks for your input!
>
> I update the prototype as described in the proposal:
> https://github.com/apache/pulsar/pull/17062
>
> Will initialize a vote for PIP-200 soon.
>
> Best,
> tison.
>
>
> Zhengxin Cai  于2022年8月26日周五 08:36写道：
>
>> +1 on this,
>> I think helm chart should not be a big issue, we can just upgrade the
>> chart
>> and provide necessary doc on how to upgrade.
>>
>> tison  于2022年8月24日周三 16:41写道：
>>
>> > Hi Enrico and Qiang,
>> >
>> > The issue description was updated. Picked here:
>> >
>> > > upgrade and downgrade doc
>> >
>> > This change should not affect those who use Pulsar with the entry point
>> > script, but it changes the layout of the release artifact.
>> >
>> > I'm going to write a release note about this change and also post it on
>> the
>> > Pulsar SQL overview page as a caveat. Draft here:
>> >
>> > # Caveat
>> >
>> > If you're upgrading Pulsar SQL from 2.11 or early, you should copy the
>> > related configs from `conf/presto` to `trino/conf`, and `lib/presto` to
>> > `trino`. If you're downgrading Pulsar SQL to 2.11 or early from 2.12, do
>> > verse visa.
>> >
>> > > Pulsar Helm Chart
>> >
>> > From what I understand, Pulsar Helm Chart is a wrapper of pulsar-all
>> docker
>> > image. I don't find any reference to presto/trino/sql effectively in the
>> > codebase, so I think it's currently transparent for its users and the
>> > caveat under Pulsar SQL overview page should be enough.
>> >
>> > Best,
>> > tison.
>> >
>> >
>> > tison  于2022年8月17日周三 19:40写道：
>> >
>> > > Thanks for your feedback!
>> > >
>> > > 1. According to the upgrade and downgrade doc, I think the minimum
>> > > requirements are a release note. Describe the layout change and how
>> users
>> > > should move the folder. I'll elaborate on the issue and notify you
>> here.
>> > > 2. It seems Pulsar Helm Chart support 2.9.3 now. I'll investigate how
>> it
>> > > can be relevant in days. If someone who maintains the Chart can
>> provide
>> > > some input, it will help!
>> > > 3. "There are 3 issues". It's a description about the
>> > > background/motivation, while we handle the first issue in this PIP.
>> > > Although, subtasks can be divided into packaging changes, possible doc
>> > > changes, and possible Chart changes.
>> > >
>> > > Best,
>> > > tison.
>> > >
>> > >
>> > > Qiang Huang  于2022年8月17日周三 19:25写道：
>> > >
>> > >> Looks good. I have two points:
>> > >> 1. It is necessary to supplement the upgrade and downgrade
>> documentation
>> > >> in
>> > >> Pulsar.
>> > >> 2. There are 3 issues mentioned in the PIP, should we split it into 3
>> > >> small
>> > >> issues?
>> > >>
>> > >> Enrico Olivelli  于2022年8月17日周三 17:30写道：
>> > >>
>> > >> > I generally agree with the PIP
>> > >> >
>> > >> > Can you please explain the interactions with the Pulsar Helm chart
>> ?
>> > >> > also we have to draw a migration path, because users that will
>> upgrade
>> > >> > Pulsar will have to move the configuration files in another
>> location
>> > >> >
>> > >> > Enrico
>> > >> >
>> > >> > Il giorno mer 17 ago 2022 alle ore 11:15 tison <
>> wander4...@gmail.com>
>> > >> > ha scritto:
>> > >> > >
>> > >> > > Hello,
>> > >> > >
>> > >> > > This is a PIP to package the Pulsar Trino distro and config in a
>> > >> > dedicated
>> > >> > > folder.
>> > >> > >
>> > >> > > Link: https://github.com/apache/pulsar/issues/17137
>> > >> > > Prototype: https://github.com/apache/pulsar/pull/17062
>> > >> > >
>> > >> > > Below you can find the proposal (I will amend the GH issue while
>> we
>> > >> > discuss
>> > >> > > it).
>> > >> > >
>> > >> > > Best,
>> > >> > > tison.
>> > >> > >
>> > >> > > Motivation
>> > >> > > 
>> > >> > >
>> > >> > > After https://github.com/apache/pulsar/pull/16683 merged, we
>> > upgrade
>> > >> > > PrestoSQL dependency in Pulsar SQL to the first several Trino
>> > >> version. To
>> > >> > > handle the name change cases and gradually refactor Pulsar SQL
>> as a
>> > >> > > self-contained module so that we can move it into a standalone
>> > >> > repository,
>> > >> > > I find that there're three major issues to resolve.
>> > >> > >
>> > >> > > 1. Configs of Pulsar SQL go under the `conf/` folder and mix with
>> > >> other
>> > >> > > Pulsar configs.
>> > >> > > 2. Pulsar Docker images (base and all) bundle Pulsar SQL.
>> > >> > > 3. Integration tests of Pulsar SQL are tightly coupled with the
>> main
>> > >> repo
>> > >> > > (test infra).
>> > >> > >
>> > >> > > This proposal is aimed at resolving the first issue to package
>> > Pulsar
>> > >> > Trino
>> > >> > > distro and config in a dedicated folder; that is, to make it
>> > >> > self-contained.
>> > >> > >
>> > >> > > Goal
>> > >> > > 
>> > >> > >
>> > >> > > I have already prepared a draft to perform the changes as
>> > >> > > https://github.com/apache/pulsar/pull/17062. 

[VOTE] PIP-200 Package Pulsar Trino distro and config in a dedicated folder

[tison]

Hi devs,

This is the official thread VOTE for PIP-200 Package Pulsar Trino distro
and config in a dedicated folder.

Here is the PIP issue: https://github.com/apache/pulsar/issues/17137
Here is the discussion thread:
https://lists.apache.org/thread/s985ypf0r0hzcm0mx653n5h2rt7n273v
Here is the link to the draft PR:
https://github.com/apache/pulsar/pull/17062

Voting will stay open for at least 72h.

Best,
tison.

Re: [DISCUSS] PIP-204: Reactive Java client for Apache Pulsar

[Lari Hotari]

I updated it to be PIP-205 since there was a previous reference of PIP-204. :) 

-Lari

On 2022/08/29 12:55:43 Lari Hotari wrote:
> Hi all,
> 
> I have drafted PIP-204: Reactive Java client for Apache Pulsar.
> 
> PIP link:
> https://github.com/apache/pulsar/issues/17335
> 
> Here's a copy of the contents of the GH issue for your references:
> 
> Motivation
> 
> There's a need to "go reactive from end-to-end" when building modern
> reactive applications with platforms such as Spring Reactive.
> There are ways to adapt the Apache Pulsar Java client async API calls to
> Reactive Streams with a few lines of code.
> However, a lot will be missing and achieving the complete solution will
> require much more effort.
> 
> A better solution would be to have first-class support Reactive Streams in
> Apache Pulsar.
> 
> Reactive Streams  is an interoperability
> specification and there are multiple implementations for the JVM.
> It's not about a single programming language.
> For example, a Reactive client for Apache Pulsar supporting Reactive
> Streams can be used together with Project Reactor / Spring Reactive, Akka
> Streams, RxJava 3, Vert.x, SmallRye Mutiny (RedHat/Quarkus) and others.
> Goal
> 
> Provide Reactive Java client for Apache Pulsar
> 
> The Reactive Java client for Apache Pulsar exposes a Reactive Streams
> compatible Reactive client API for Apache Pulsar.
> Reactive programming is about non-blocking applications that are
> asynchronous and event-driven and require a small number of threads to
> scale. The Reactive Java client for Apache Pulsar supports non-blocking
> reactive asynchronous back pressure for producing and consuming messages so
> that the producing or consuming pipeline doesn't get overwhelmed by
> producing or consuming.
> Libraries that support Reactive Streams provide a programming model that is
> efficient and optimal for message producing and consuming (processing) use
> cases.
> API Changes
> 
> Establish a Reactive Streams compatible client API for Apache Pulsar.
> This client will be published in Maven central as a library.
> Implementation
> 
> There's an existing proof-of-concept available at
> https://github.com/datastax/pulsar .
> This implementation will be used as a reference for an entirely new
> implementation that is started as a new repository under the Apache Pulsar
> project.
> 
> The proposal for the repository location is
> https://github.com/apache/pulsar-client-reactive .
> The Maven central group Id is "org.apache.pulsar" and the main artifact id
> is "pulsar-client-reactive".
> The root package name is "org.apache.pulsar.reactive.client".
> 
> The implementation will provide an interface module that abstracts the
> Reactive client API.
> This interface is implemented by wrapping the current Apache Pulsar Java
> client and adapts the existing async Java API to the the Reactive client
> API.
> The reason for this particular detail is that it is possible to provide a
> native Reactive client later while having the possibility to start
> developing applications immediately using the Reactive client API.
> Applications depending on the API will be able to migrate to use the native
> Reactive client with minor or no changes when it becomes available.
> Anything else?
> 
> By having an official Reactive Java client for Apache Pulsar, it will
> provide a way to contribute and improve the official client.
> Other opensource projects might want to provide support for using Apache
> Pulsar within reactive application frameworks. Without an official reactive
> client, this becomes hard, since open source projects would like to use
> stable client dependencies instead of a hobby project provided by an
> individual.
> There are several members within the existing Apache Pulsar contributors
> and committers that have expressed the desire to contribute to a Reactive
> client for Apache Pulsar and are willing to maintain the new repository.
> With the new repository and sub-project we will most likely see new active
> contributors and could possibly appoint new Apache Pulsar committers to the
> project to empower the developers working on this new sub-project.
> 
> I'm looking forward to the discussion.
> 
> 
> BR,
> 
> 
> Lari
> 

[DISCUSS] PIP-204: Reactive Java client for Apache Pulsar

[Lari Hotari]

Hi all,

I have drafted PIP-204: Reactive Java client for Apache Pulsar.

PIP link:
https://github.com/apache/pulsar/issues/17335

Here's a copy of the contents of the GH issue for your references:

Motivation

There's a need to "go reactive from end-to-end" when building modern
reactive applications with platforms such as Spring Reactive.
There are ways to adapt the Apache Pulsar Java client async API calls to
Reactive Streams with a few lines of code.
However, a lot will be missing and achieving the complete solution will
require much more effort.

A better solution would be to have first-class support Reactive Streams in
Apache Pulsar.

Reactive Streams  is an interoperability
specification and there are multiple implementations for the JVM.
It's not about a single programming language.
For example, a Reactive client for Apache Pulsar supporting Reactive
Streams can be used together with Project Reactor / Spring Reactive, Akka
Streams, RxJava 3, Vert.x, SmallRye Mutiny (RedHat/Quarkus) and others.
Goal

Provide Reactive Java client for Apache Pulsar

The Reactive Java client for Apache Pulsar exposes a Reactive Streams
compatible Reactive client API for Apache Pulsar.
Reactive programming is about non-blocking applications that are
asynchronous and event-driven and require a small number of threads to
scale. The Reactive Java client for Apache Pulsar supports non-blocking
reactive asynchronous back pressure for producing and consuming messages so
that the producing or consuming pipeline doesn't get overwhelmed by
producing or consuming.
Libraries that support Reactive Streams provide a programming model that is
efficient and optimal for message producing and consuming (processing) use
cases.
API Changes

Establish a Reactive Streams compatible client API for Apache Pulsar.
This client will be published in Maven central as a library.
Implementation

There's an existing proof-of-concept available at
https://github.com/datastax/pulsar .
This implementation will be used as a reference for an entirely new
implementation that is started as a new repository under the Apache Pulsar
project.

The proposal for the repository location is
https://github.com/apache/pulsar-client-reactive .
The Maven central group Id is "org.apache.pulsar" and the main artifact id
is "pulsar-client-reactive".
The root package name is "org.apache.pulsar.reactive.client".

The implementation will provide an interface module that abstracts the
Reactive client API.
This interface is implemented by wrapping the current Apache Pulsar Java
client and adapts the existing async Java API to the the Reactive client
API.
The reason for this particular detail is that it is possible to provide a
native Reactive client later while having the possibility to start
developing applications immediately using the Reactive client API.
Applications depending on the API will be able to migrate to use the native
Reactive client with minor or no changes when it becomes available.
Anything else?

By having an official Reactive Java client for Apache Pulsar, it will
provide a way to contribute and improve the official client.
Other opensource projects might want to provide support for using Apache
Pulsar within reactive application frameworks. Without an official reactive
client, this becomes hard, since open source projects would like to use
stable client dependencies instead of a hobby project provided by an
individual.
There are several members within the existing Apache Pulsar contributors
and committers that have expressed the desire to contribute to a Reactive
client for Apache Pulsar and are willing to maintain the new repository.
With the new repository and sub-project we will most likely see new active
contributors and could possibly appoint new Apache Pulsar committers to the
project to empower the developers working on this new sub-project.

I'm looking forward to the discussion.


BR,


Lari

[GitHub] [pulsar] liangyuanpeng added a comment to the discussion: I encountered a port error when using pulsar to integrate etcd

[GitBox]

GitHub user liangyuanpeng added a comment to the discussion: I encountered a 
port error when using pulsar to integrate etcd

If you can see some log like `Cluster metadata for 'xxx' setup correctly` , 
your cluster metadata is init successed. Then you can ignore this log.

GitHub link: 
https://github.com/apache/pulsar/discussions/17240#discussioncomment-3500145


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] liangyuanpeng added a comment to the discussion: I encountered a port error when using pulsar to integrate etcd

[GitBox]

GitHub user liangyuanpeng added a comment to the discussion: I encountered a 
port error when using pulsar to integrate etcd

If you can see some log like `Cluster metadata for 'xxx' setup correctly` , 
your cluster metadata is init successed. Then you can ignore this log.

GitHub link: 
https://github.com/apache/pulsar/discussions/17240#discussioncomment-3500139


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar-manager] jakiuncle closed issue #483: I want use casdoor to login,when i change code, happen some problem

[GitBox]

jakiuncle closed issue #483: I want use casdoor to login,when i change code, 
happen some problem
URL: https://github.com/apache/pulsar-manager/issues/483


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-site] tisonkun commented on pull request #182: Generate 2.9.2, 2.9.3, 2.10.1 python docs

[GitBox]

tisonkun commented on PR #182:
URL: https://github.com/apache/pulsar-site/pull/182#issuecomment-1230199254

   @michaeljmarshall we have api docs for python under 
https://github.com/apache/pulsar-site/tree/main/site2/website-next/static/api/python.
   
   `site2/website` can be the folder for legacy pulsar website 
(http://pulsar.staged.apache.org/ now), how can we make some changes on those 
folders?
   
   cc @urfreespace @Anonymitaet 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-test-infra] nodece commented on pull request #69: Refactor docbot and add tests

[GitBox]

nodece commented on PR #69:
URL: https://github.com/apache/pulsar-test-infra/pull/69#issuecomment-1230166815

   Covert to draft for testing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-test-infra] nodece commented on a diff in pull request #69: Refactor docbot and add tests

[GitBox]

nodece commented on code in PR #69:
URL: https://github.com/apache/pulsar-test-infra/pull/69#discussion_r957210713


##
docbot/action.yml:
##
@@ -11,12 +11,12 @@ runs:
 - name: Checkout
   uses: actions/checkout@v3
   with:
-repository: apache/pulsar-test-infra
+repository: nodece/pulsar-test-infra

Review Comment:
   When this PR is ready for review, I will reset this config.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-test-infra] nodece commented on pull request #69: Refactor docbot and add tests

[GitBox]

nodece commented on PR #69:
URL: https://github.com/apache/pulsar-test-infra/pull/69#issuecomment-1230164873

   Hi @tisonkun and @maxsxu, could help review this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [pulsar-test-infra] nodece opened a new pull request, #69: Refactor docbot and add tests

[GitBox]

nodece opened a new pull request, #69:
URL: https://github.com/apache/pulsar-test-infra/pull/69

   ### Motivation
   
   1. Fix #59 
   2. Add tests to cover this feature
   
   ### Verify
   
   I verified this on my repo, please help verify the following case on your 
repo if you have time, thanks!
   
   1. Checked single checkbox
   2. Checked multiple checkboxes
   3. Unchecked any checkboxes, waited for CI to have done, then checked one 
checkbox
   
   Workflow:
   ```
   name: Documentation Bot
   
   on:
 pull_request_target :
   types:
 - opened
 - edited
 - labeled
 - unlabeled
   
   concurrency:
 group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.number }}
 cancel-in-progress: true
   
   jobs:
 label:
   permissions:
 pull-requests: write 
   runs-on: ubuntu-latest
   steps:
 - name: Labeling
   uses: nodece/pulsar-test-infra/docbot@master
   env:
 GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 LABEL_WATCH_LIST: 'doc,doc-required,doc-not-needed,doc-complete'
 LABEL_MISSING: 'doc-label-missing'
   ```
   
   You also need to setting your `Workflow permissions`, see 
https://github.com/$username/$repo/settings/actions.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Move PIPs to the codebase?

[PengHui Li]

Hi all,

I will merge https://github.com/apache/pulsar/pull/17270 first and draft a
proposal for
the detailed PIP process changes. And will start a new VOTE thread for the
PIP process change.

After the proposal(for PIP process change) is approved. I will go back here
to discuss
how to migrate the old PIPs to the codebase(because we need to follow
the new format of PIPs).

Thanks,
Penghui

On Fri, Aug 26, 2022 at 11:20 PM PengHui Li  wrote:

> > Using a directory structure to organize PIP status might make the git
> history a bit less direct because changing state is equivalent with a
> file move instead of a line updated in a file. However, if we do it
> that way, we could have a README.md file organizing PIP metadata and
> linking to the actual PIP file in the directory structure. That
> README.md would also serve as the source of truth for PIP numbers
> because each PIP pointer would get its associated line number. Then,
> concurrent PIPs would need to resolve merge conflicts just before
> merging and only then would they acquire their number.
>
> Oh, get your point, Michael. I think this solution looks better. We can
> also
> add something in README.md and users can also get the complete
> proposal list here. In the future, maybe we can show it on the website.
>
> Best,
> Penghui
>
> On Fri, Aug 26, 2022 at 12:16 PM Michael Marshall 
> wrote:
>
>> > We should move to the codebase 100% the same as the original
>> documentation.
>> > And use another PR to make improvements for it. If it is needed, we
>> should
>> > start with an email.
>> > We need to have historical records.
>>
>> +1 I think this is a great idea. It makes sense to copy them verbatim
>> and then worry about updating them in a second step.
>>
>> Using a directory structure to organize PIP status might make the git
>> history a bit less direct because changing state is equivalent with a
>> file move instead of a line updated in a file. However, if we do it
>> that way, we could have a README.md file organizing PIP metadata and
>> linking to the actual PIP file in the directory structure. That
>> README.md would also serve as the source of truth for PIP numbers
>> because each PIP pointer would get its associated line number. Then,
>> concurrent PIPs would need to resolve merge conflicts just before
>> merging and only then would they acquire their number.
>>
>> - Michael
>>
>> On Thu, Aug 25, 2022 at 9:15 PM PengHui Li  wrote:
>> >
>> > Hi Xiangying,
>> >
>> > > We can classify the pips under these folders according to the pulsar
>> > modules, instead of just placing these pips under these folders in an
>> > incrementing sequence number.
>> >
>> > I think it's not easy to do this. A proposal can have changes related to
>> > multiple
>> > modules(Broker, Metadata, Client).
>> >
>> > Thanks,
>> > Penghui
>> >
>> > On Fri, Aug 26, 2022 at 9:20 AM Xiangying Meng 
>> wrote:
>> >
>> > > Hi Penghui
>> > > Thanks for you start this discussion. IMO, It is also a good way for
>> > > beginners to learn the design and implementation of each module of
>> Pulsar.
>> > > > 3. /wiki/pip/accepted - for PIPs that have been accepted and are
>> works in
>> > > progress
>> > > > 4. /wiki/pip/complete - for PIPs that have been completed.
>> > > > 5. /wiki/pip/rejected - for PIPs that were proposed, but then
>> rejected or
>> > > abandoned.
>> > > We can classify the pips under these folders according to the pulsar
>> > > modules, instead of just placing these pips under these folders in an
>> > > incrementing sequence number.
>> > >
>> > > In this way, readers can create a new local branch dedicated to
>> reading and
>> > > annotating proposals for themselves to read proposals they are
>> interested
>> > > in and write their own understanding and comments anytime and
>> anywhere.
>> > > Thanks,
>> > > Xiangying Meng
>> > >
>> > > On Fri, Aug 26, 2022 at 12:23 AM PengHui Li 
>> wrote:
>> > >
>> > > > Hi Dave,
>> > > >
>> > > > > Let’s outline how PIPs are currently working and then discuss
>> changes.
>> > > >
>> > > > Yes, I will prepare for the changes.
>> > > > This is the documentation for how PIPs are currently working:
>> > > >
>> > > >
>> > >
>> https://github.com/apache/pulsar/pull/17270/files#diff-a56445d038f8a3df4c74724076c62437915f091437b4e26a1c5aada7184dcffd
>> > > > The mailing list discussion:
>> > > > https://lists.apache.org/thread/m8dr0hz7qn7rkd48bcp430lcq2q3674g
>> > > >
>> > > > Anyway, I will start a new discussion with the new changes to the
>> current
>> > > > process.
>> > > >
>> > > > > I’m not sure what is meant by putting the PIP into the “codebase”.
>> > > > > Is the proposal to create PIPs here?
>> > > > https://github.com/apache/pulsar/tree/master/wiki
>> > > >
>> > > > Just move out from
>> https://github.com/apache/pulsar/tree/master/wiki to
>> > > > Pulsar codebase /wiki/proposals
>> > > >
>> > > > > I think that a directory structure / convention could be used for
>> pip
>> > > > status:
>> > > >
>> > > > > 1. 

Re: [VOTE] PIP-195: New bucket based delayed message tracker

[PengHui Li]

+1

I have completed the proposal review on the gdoc.
But unfortunately, that is an internal google driver which not able to
create
public links.

Cong has copied to a new one.

Thanks,
Penghui

On Mon, Aug 29, 2022 at 6:37 PM Cong Zhao  wrote:

> Hi Pulsar Community,
>
> In order to facilitate community review, I copied this proposal to Google
> Docs, please review it and complete the vote.
>
> PIP:
> https://docs.google.com/document/d/17HE88w4WuEsLoz0DExPpRN555UC_Tc3MEIKckVWksDc/edit
>
> On 2022/08/08 06:51:31 Cong Zhao wrote:
> >  Hi Pulsar Community,
> >
> > I would like to start a VOTE on "New bucket based delayed message
> tracker"
> > (PIP-195).
> >
> > The proposal can be read at
> https://github.com/apache/pulsar/issues/16763
> > and the discussion thread is available at
> > https://lists.apache.org/thread/1krdhrvs803kb6rqzdh17q0f199nroz4
> >
> > Voting will stay open for at least 48h.
> >
> > Thanks,
> > Cong Zhao
> >
>

Re: [DISCUSS] Introduce FlowControl to metrics endpoint

[Lari Hotari]

Good reiteration of the problem and good points, Asaf.

I'd like to add a new aspect to the proposal: there might be other
solutions that would be useful in the case of large amount of topics in a
Pulsar cluster.
Rate limiting on the /metrics endpoint doesn't sound like the correct
approach.

When there's a huge about of metrics, instead of scraping the metrics, it
could be more useful to ingest the metrics to Prometheus using the "Remote
write API".
There's a recording of a talk explaining remote write in
https://www.youtube.com/watch?v=vMeCyX3Y3HY .
The specification is
https://docs.google.com/document/d/1LPhVRSFkGNSuU1fBd81ulhsCPR4hkSZyyBj1SZ8fWOM/edit#
.
The benefit of this could be that /metrics endpoint wouldn't be a
bottleneck and there wouldn't be a need to do any hacks to support a high
number of metrics.
There might be need to route the metrics for different namespaces/topics to
different destinations. This could be handled in the implementation that
uses the Remote write API for pushing metrics.

Regards,

-Lari


On Mon, Aug 29, 2022 at 1:12 PM Asaf Mesika  wrote:

> Hi Jiuming,
>
> I would reiterate the problem statement to make it clear (at least for me):
>
> There are cases where a very large amount of topics exists (> 10k per
> broker) and are used in Pulsar. Those topics usually have multiple
> producers and multiple consumers.
> There are metrics that are in the granularity of topics and also in
> topic/producers and topic/consumers granularity.
> When that happens, the amount of unique metrics is severely high which
> causes the response size of /metrics endpoint (the Prometheus Exposition
> Format endpoint) to be substantially high - 200MB - 500MB.
>
> Every time the metrics are scraped (30sec, or 1 min), the network gets
> surged up due to the /metrics response, thereby causing latency to messages
> produced or consumed from that broker.
>
> The solution proposed is to throttle the /metrics response based on a
> pre-configured rate limit.
>
> Points to consider for this discussion from the participants:
>
> 1. Did you happen to experience such difficulties in your clusters?
> 2. When that happened, did you experience the bottleneck also on the TSDB
> be it metrics ingestion or querying?
>
> Thanks,
>
> Asaf
>
>
> On Thu, Aug 18, 2022 at 7:40 PM Jiuming Tao  >
> wrote:
>
> > bump
> >
> > Jiuming Tao  于2022年8月8日周一 18:19写道：
> >
> > > Hi Pulsar community,
> > >
> > > In the situation of expose metrics data which has a big size, it will
> > lead
> > > to:
> > > 1. A sudden increase of network usage
> > > 2. The latency of pub/sub rising
> > >
> > > For the purpose of resolving these problems, I’d opened a PR:
> > > https://github.com/apache/pulsar/pull/16452
> > >
> > > Please feel free to help review/discuss about it.
> > >
> > > Thanks,
> > > Tao Jiuming
> > >
> >
>

Re: [VOTE] PIP-195: New bucket based delayed message tracker

[Cong Zhao]

Hi Pulsar Community,

In order to facilitate community review, I copied this proposal to Google Docs, 
please review it and complete the vote.

PIP: 
https://docs.google.com/document/d/17HE88w4WuEsLoz0DExPpRN555UC_Tc3MEIKckVWksDc/edit

On 2022/08/08 06:51:31 Cong Zhao wrote:
>  Hi Pulsar Community,
> 
> I would like to start a VOTE on "New bucket based delayed message tracker"
> (PIP-195).
> 
> The proposal can be read at https://github.com/apache/pulsar/issues/16763
> and the discussion thread is available at
> https://lists.apache.org/thread/1krdhrvs803kb6rqzdh17q0f199nroz4
> 
> Voting will stay open for at least 48h.
> 
> Thanks,
> Cong Zhao
> 

Re: [DISCUSS] Introduce FlowControl to metrics endpoint

[Asaf Mesika]

Hi Jiuming,

I would reiterate the problem statement to make it clear (at least for me):

There are cases where a very large amount of topics exists (> 10k per
broker) and are used in Pulsar. Those topics usually have multiple
producers and multiple consumers.
There are metrics that are in the granularity of topics and also in
topic/producers and topic/consumers granularity.
When that happens, the amount of unique metrics is severely high which
causes the response size of /metrics endpoint (the Prometheus Exposition
Format endpoint) to be substantially high - 200MB - 500MB.

Every time the metrics are scraped (30sec, or 1 min), the network gets
surged up due to the /metrics response, thereby causing latency to messages
produced or consumed from that broker.

The solution proposed is to throttle the /metrics response based on a
pre-configured rate limit.

Points to consider for this discussion from the participants:

1. Did you happen to experience such difficulties in your clusters?
2. When that happened, did you experience the bottleneck also on the TSDB
be it metrics ingestion or querying?

Thanks,

Asaf


On Thu, Aug 18, 2022 at 7:40 PM Jiuming Tao 
wrote:

> bump
>
> Jiuming Tao  于2022年8月8日周一 18:19写道：
>
> > Hi Pulsar community,
> >
> > In the situation of expose metrics data which has a big size, it will
> lead
> > to:
> > 1. A sudden increase of network usage
> > 2. The latency of pub/sub rising
> >
> > For the purpose of resolving these problems, I’d opened a PR:
> > https://github.com/apache/pulsar/pull/16452
> >
> > Please feel free to help review/discuss about it.
> >
> > Thanks,
> > Tao Jiuming
> >
>

[DISCUSS]Do not delete all schema data when one version is non recoverable

[Yubiao Feng]

=== Motivation
Now, when a version-data of schema is lost, all schema data for that topic
will be deleted. see:
https://github.com/apache/pulsar/blob/0e149ef0484755d42e214f03003237299433c89a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/SchemaRegistryServiceImpl.java#L527-L565

This will cause the following problems:
- It is possible that the missing version is no longer in use, but the
other versions of the schema data have also been deleted.
- After all the schema data is deleted, the version counter will again
start counting from 0. However, the old Message(already send and still in
use) and the new Message(create with the new schema) will possibly use the
same version of schemas, but the old Message cannot be parsed using the new
schema.

In my opinion:
- Delete only the invalid record from Schema Meta Data.
- The Schema version counter is always monotonically increased and is not
used cyclically.

[GitHub] [pulsar] Anonymitaet added a comment to the discussion: How to Make Contributions to Pulsar Documentation and Website?

[GitBox]

GitHub user Anonymitaet added a comment to the discussion: How to Make 
Contributions to Pulsar Documentation and Website?

Thanks for raising this up!

The [Pulsar Contribution Guide](https://pulsar.apache.org/contributing/) lists 
all the following guides explaining how to make contributions to Pulsar 
documentation and website.

☘️☘️☘️

 **Before writing documentation** 

- [Pulsar Documentation Contribution 
Guide](https://docs.google.com/document/d/1-1uJyd1k9_h56xiiVRVOnrLcCnTmg9n7SrHhNVNEEi4/edit#heading=h.f4ftjqe2pezi)

☘️☘️☘️

✏️ **Writing documentation** 

- [Pulsar Documentation Writing Style 
Guide](https://docs.google.com/document/d/1lc5j4RtuLIzlEYCBo97AC8-U_3Erzs_lxpkDuseU0n4/edit#)
- [Pulsar Documentation Writing Syntax 
Guide](https://docs.google.com/document/d/12De2btkDHQVaqUlHjTqERmroMLKGhHdiC7rEttFTyqc/edit#)
 
- [Pulsar Design Style 
Guide](https://docs.google.com/document/d/16Hp7Sc86MQtL0m8fc2w_TrcKXAuglwRwHmdmwfk00mI/edit#heading=h.b8ogodj5sj0)
- [Pulsar API Documentation 
Guide](https://docs.google.com/document/d/1-I1oQp1_HUaQopqilU-JdC-ksrLAgYNi93FZVnECwV8/edit#heading=h.wu6ygjne8e35)
- [Pulsar Release Notes 
Guide](https://docs.google.com/document/d/1cwNkBefKyV6OPbEXnUrcCdVZi0i2BezqL6vAL7VqVC0/edit#)

☘️☘️☘️ 

離 **Testing documentation**

- [Pulsar Documentation Preview 
Guide](https://docs.google.com/document/d/1wszdtMRo6MhKbVaggPK7_bnKaC4TewuT--GWZZxJNGg/edit#heading=h.tng8x5vsxlpi)

☘️☘️☘️ 

 **Preparing to submit a documentation PR** 

- [Pulsar Pull Request Naming Convention 
Guide](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#heading=h.wu6ygjne8e35)
- [Pulsar Documentation Label 
Guide](https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0/edit#)

☘️☘️☘️

 **Tip** 
- These guides are in drafts and continuously updated. More specific guides are 
on the way. Feel free to comment!
- Do not worry about the accessibility. They will be **moved to the Pulsar 
website** after they are finalized. 



GitHub link: 
https://github.com/apache/pulsar/discussions/17329#discussioncomment-3497978


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] Anonymitaet edited a discussion: [Design] Pulsar Release Timeline Page

[GitBox]

GitHub user Anonymitaet edited a discussion: [Design] Pulsar Release Timeline 
Page

Hi team,

Currently, we do not have a place showing the timeline for all Pulsar releases.

How about creating a Pulsar [Release Notes > 
Timeline](https://pulsar.apache.org/release-notes/timeline) page along with the 
following content?

1. Add the [Pulsar release timeline 
table](https://docs.google.com/spreadsheets/d/1XHmzno-1kbHvPp7eogri8XAiShH-EgpEG40svte-qxM/edit#gid=1677243891)
 as below.

https://user-images.githubusercontent.com/50226895/187153499-0689fdce-1ab9-40bd-8115-b4b816dd1f92.png;>

2. The timeline table should be updated automatically. Data (release date + 
release manager) can be pulled from https://github.com/apache/pulsar/releases.

3. [Versions page](https://pulsar.apache.org/versions) info is consolidated 
into the timeline page, so the Versions page can be removed.

☘️☘️☘️

Hi @tisonkun @urfreespace @SignorMercurio thoughts?


GitHub link: https://github.com/apache/pulsar/discussions/17310


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] Anonymitaet edited a discussion: [Design] Pulsar Release Timeline Page

[GitBox]

GitHub user Anonymitaet edited a discussion: [Design] Pulsar Release Timeline 
Page

Hi team,

Currently, we do not have a place showing the timeline for all Pulsar releases.

How about creating a Pulsar [Release Notes > 
Timeline](https://pulsar.apache.org/release-notes/timeline) page along with the 
following content?

1. Add the [Pulsar release timeline 
table](https://docs.google.com/spreadsheets/d/1XHmzno-1kbHvPp7eogri8XAiShH-EgpEG40svte-qxM/edit#gid=1677243891)
 as below.

https://user-images.githubusercontent.com/50226895/187023651-f6d5303d-a864-45b0-89c3-095e8cab2a47.png;>

2. The timeline table should be updated automatically. Data (release date + 
release manager) can be pulled from https://github.com/apache/pulsar/releases.

3. [Versions page](https://pulsar.apache.org/versions) info is consolidated 
into the timeline page, so the Versions page can be removed.

☘️☘️☘️

Hi @tisonkun @urfreespace @SignorMercurio thoughts?


GitHub link: https://github.com/apache/pulsar/discussions/17310


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

Re: [DISCUSS] Pulsar 3.0 brainstorming: Going beyond PIP-45 Pluggable metadata interface

[Lari Hotari]

Hi all,

In case someone missed the first message in this thread, it can be read at
https://lists.apache.org/thread/tvco1orf0hsyt59pjtfbwoq0vf6hfrcj .

The intention is to start discussions about improving Apache Pulsar design
in Pulsar 3.0 . This seems to be the first time that we are discussing
Pulsar 3.0 design in the community on the dev-mailing list.
I haven't asked permission to start this discussion, since there is no need
for that. In Apache projects, all individuals, regardless of committer or
PMC member status, are given the opportunity to participate in discussions
and make proposals without asking permission to do so.
In the Apache way, private decisions on project direction are indeed
disallowed [1], which could be surprising for those who are new to the
Apache way.
I'd like to clarify that there is no decision about Pulsar 3.0 or any
timelines related to it. This is the beginning of the discussion and no
decisions have been made.

The current PIP process in the Apache Pulsar project seems to be a lot
about documenting design decisions, and it's hard to make changes
that would require reversing some previous design decision.
In some cases, some problems or challenges could be better solved by
reversing or replacing some element of the current design.

One example of a fundamental foundation of the current Pulsar design is the
concept of "namespace bundles". "Namespace bundles" are defined in Pulsar
reference document in this way:
"The assignment of topics to brokers is not done at the topic level but at
the bundle level (a higher level). Instead of individual topic assignments,
each broker takes ownership of a subset of the topics for a namespace. This
subset is called a bundle and effectively this subset is a sharding
mechanism."
"Topics are assigned to a particular bundle by taking the hash of the topic
name and checking in which bundle the hash falls."
(source:
https://pulsar.apache.org/docs/administration-load-balance#dynamic-assignments
)
There's also an excellent blog post explaining how Pulsar load balancing
currently works in detail, it's
https://streamnative.io/blog/engineering/2022-07-21-achieving-broker-load-balancing-with-apache-pulsar/
.

There has been a clear reason in the current design. With namespace
bundles, topic load balancing operations can be handled as groups (batches)
to minimize the overhead of coordination and load balancing operations.
However, this design also causes essential limitations: a topic cannot be
freely assigned to a bundle or a broker. In the current namespace bundles
design, a namespace bundle is determined by calculating a hash of the
topic's name and there are hash ranges to define bundle boundaries.
There is no explicit way to assign a topic to a bundle. Splitting a bundle
is the only way to impact bundle assignment.
This adds a lot of unnecessary complexity to load balancing decisions and
prevents optimal topic placement.

The current Pulsar load balancing makes load balancing decisions
dynamically, although there are features where placement with predefined
rules is used (broker isolation, anti-affinity namespaces across failure
domains).
The current load balancing actions are heavily impacted and limited by the
namespace bundles.

There might be future requirements of having affinity and anti-affinity
rules for topic placement. It might be useful to be able to have placement
rules to take availability zones (or data center racks) into account.
Free form topic placement would also be useful for capacity management.
Rate limits could be used as part of load balancing decisions, and there
could be rules for how much total possible throughput is allowed on a
specific broker when considering the rate limits of assigned topics. Rule
based placement could also be useful when it is known that the topic
traffic pattern is bursty (common in batch oriented processing) and maximum
throughput is desired while the traffic burst is being processed. Dynamic
load balancing is simply too slow in reacting to such changes in traffic
patterns. In these cases, it might be useful to be able to pre-assign the
different partitions of a partitioned topic to be balanced across available
brokers in the cluster with anti-affinity rules instead of making this
decision dynamically after the traffic is running. The traffic burst could
be over when the load balancer reacts.

Broker isolation is another use case for free form topic placement. In the
case of capacity issues or noisy neighbor performance issues in achieving
SLAs, it would be useful to be able to assign a specific topic to a
dedicate set of brokers. Broker isolation is currently possible at
namespace level, but having this possibility at topic level would increase
flexibility.
On a multi-tenant SaaS platform, this would give more possibilities in
meeting QoS/SLA by having better ways for ensuring guaranteed throughput
and latency with better capacity management.
Achieving autoscaling requires better capacity 

[GitHub] [pulsar] Huanli-Meng created a discussion: How to Make Contributions to Pulsar Documentation and Website?

[GitBox]

GitHub user Huanli-Meng created a discussion: How to Make Contributions to 
Pulsar Documentation and Website?

Hi team,
I want to update Pulsar documentation and website, are there any guides to 
reference or standards to follow?
Thank you!

GitHub link: https://github.com/apache/pulsar/discussions/17329


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org

[GitHub] [pulsar] Anonymitaet created a discussion: [New Site Design] Pulsar Reference + Pulsar CLI + Pulsar Configuration

[GitBox]

GitHub user Anonymitaet created a discussion: [New Site Design] Pulsar 
Reference + Pulsar CLI + Pulsar Configuration

Hi team,

We're implementing [PIP 78: Generate Docs from Code 
Automatically](https://docs.google.com/spreadsheets/d/1P6tePGmU1lEhGKY_hE3JWgJ1rUTBCmW7AshDICJZr3o/edit#gid=1030826289)
 and [replacing brodocs with 
docsify](https://lists.apache.org/thread/1g8nxqg4gdw3obq4wfgv442hvspk7ovy).  

Along with this change, 3 new sites will be created with the following design:

- [[Design] Pulsar Reference - Home 
Page](https://docs.google.com/spreadsheets/d/1P6tePGmU1lEhGKY_hE3JWgJ1rUTBCmW7AshDICJZr3o/edit#gid=1310268188)
- [[Design] Pulsar CLI tools 
Page](https://docs.google.com/spreadsheets/d/1P6tePGmU1lEhGKY_hE3JWgJ1rUTBCmW7AshDICJZr3o/edit#gid=523560491)
- [[Design] Pulsar Configuration 
Page](https://docs.google.com/spreadsheets/d/1P6tePGmU1lEhGKY_hE3JWgJ1rUTBCmW7AshDICJZr3o/edit#gid=1186957953)

Do these mockups look good to you? Any suggestions? Thank you!

/ cc @urfreespace @SignorMercurio  

GitHub link: https://github.com/apache/pulsar/discussions/17328


This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org