[GitHub] [pulsar-test-infra] nodece closed issue #67: Simplify invocation of docbot action
nodece closed issue #67: Simplify invocation of docbot action URL: https://github.com/apache/pulsar-test-infra/issues/67 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-test-infra] nodece merged pull request #68: [improve][docbot] Move the preceding steps into the action
nodece merged PR #68: URL: https://github.com/apache/pulsar-test-infra/pull/68 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [DISCUSS] Move PIPs to the codebase?
Hi Tison, > But any proposer can take their proposal number once proposed and a committer clicks the merge button. Still, I'm concerned about adding such a step to exclusively allocate a proposal number. Here is the previous discussion https://lists.apache.org/thread/vw25onjyo18trfysd7x9k9wrs2tyqfok about this part. Without the request, approval steps for the PIP number. I didn't think of any other better way to help us manage the proposal number better. The proposal number should be solemn and not very casual. > I would suggest you collect the suggestions in this discussion thread Yes, I agree. This is exactly what I'm currently doing. Sorry, the PR https://github.com/apache/pulsar/pull/17270 introduced the confusion. Have this one just for that we are able to review detailed changes based on the old one. So I pushed the PR. Best, Penghui On Thu, Aug 25, 2022 at 2:22 PM PengHui Li wrote: > Hi all, > > I have pushed out a PR https://github.com/apache/pulsar/pull/17270 > which wants to copy the PIP and proposal template to the codebase. > > The PR doesn't want to make some changes to the proposal process > before we have approval. Since we will make some changes to the > current PIP process, I want to copy the old one to the codebase so that > all the subsequent updates will have historical records. > > Finally, the PIP process changes should be done with a new VOTE email. > With https://github.com/apache/pulsar/pull/17270, we can have a PR to show > all the details which can help the reviewer to easily review. > > Thanks, > Penghui > > > On Thu, Aug 25, 2022 at 12:57 PM tison wrote: > >> Hi Penghui, >> >> > IMO, the final status of the proposal should be summarized in the >> proposal. >> >> Agree. It's not exclusive. What I propose is that if you want to track >> proposals in a project, track issues. But it seems your opinion is not to >> do so. Then it's a nonissue. Flink FLIP tracks proposal status with the >> proposal content and it works well. Although, I'd still highlight that >> maintainer efforts are required to maintain the metadata. >> >> > As I know, if we have an issue tracking all the proposal status, not all >> the contributors make contributions. Only the committers and the author? >> >> I don't propose to track the status _within_ the issue. But if you want to >> track in a project, only committer have the privilege :) >> >> > if the discussion of the motivation doesn't have objections and is >> discussed enough. >> >> This is quite vague and hard to follow. >> >> Rust RFCs have a nonmandatory pre-RFC stage. I think we can encourage the >> proposer to discuss on the mailing list or any channel with committers >> first. But any proposer can take their proposal number once proposed and a >> committer clicks the merge button. Still, I'm concerned about adding such >> a >> step to exclusively allocate a proposal number. >> >> I would suggest you collect the suggestions in this discussion thread, and >> propose a process as shown in >> >> * https://github.com/rust-lang/rfcs#readme >> * https://github.com/pingcap/tidb/tree/master/docs/design#readme >> >> So that we can have a vote. Instead of rushing into making several PRs to >> move content fragmentedly. >> >> Best, >> tison. >> >> >> PengHui Li 于2022年8月25日周四 10:59写道： >> >> > Hi Tison, >> > >> > Thanks for providing more context. >> > >> > > Rust RFCs hold in a separate repo so that they can use the PR number >> as >> > the >> > proposal number to prevent duplicates. It's a bit overhead for Pulsar to >> > hold a separate repo. So I'd prefer to hold proposals in the same repo >> > under the dev/proposals folder and resolve the PR number duplicates >> issue >> > as described above. Although, it can still be a bit awkward. But at >> least >> > we can quickly notice duplicate numbers and resolve them. Using the PR >> > number as the proposal number keeps it unique, but the number will be >> > discontinuous. Especially if we're proposing in the main repo, the >> number >> > will grow quickly. >> > >> > The PR number can provide unique numbers. Everyone can create PRs, maybe >> > not related to PIP, but we are not able to revert it. >> > >> > > In this direction, we can associate a proposal with a tracking issue >> and >> > manipulate the proposal status represented by their tracking issue. It >> > doesn't mean we keep the current issue discussion process. You can >> regard >> > it as an umbrella ticket for the proposal. See >> > https://github.com/pingcap/tidb/issues/26022 as an example. >> > >> > > In this direction, we should always ensure that there're several eyes >> on >> > the project. Otherwise, we can easily end up with an out-of-sync >> project. >> > >> > IMO, the final status of the proposal should be summarized in the >> proposal. >> > We can have some metadata for md file which allow us to have a page that >> > can >> > show all the proposals like the bookkeeper does >> > https://bookkeeper.apache.org/community/bookkeeper-propo
Re: [DISCUSS] Catch up reads with many subscriptions - saving reads from BK
Thank you folks for your feedback Il giorno gio 25 ago 2022 alle ore 05:56 Haiting Jiang ha scritto: > > Here is my idea of handing the read position shift if we want to go a bit > further with this solution. > > Instead of only checking the exact match of entry range match > (PendingReadKey in the PR), > we can check if the requesting range is covered by several ongoing read > requests, and merge > the result of these requests. > > For example, currently there are already inflight request reads of entries > [10,20] and [20,30], > and now comes a new request [15,25], it can just merge the result from > [10,20] and [20,30], > and avoids sending the request to BK. > > As for the complexity checking procedure, I think we can sort all the > inflight requests by startEntryId > and apply a binary search here, to reduce the time complexity to O(logN) > where N is the > max entry size in one ledger. This is an interesting idea. We are doing more testing to empirically understand how frequently we have overlapping ranges. Because if it is not frequent to have such overlapping ranges then we will waste resources. Currently my explanation about this scenario is that groups of subscriptions are close to each other and nobody is able to leverage the entries cached by the others and they try to read the same entries. By default we have this default of 100 in dispatcherMaxReadBatchSize and that limits the ranges to be up to 100 entries. When there are no errors in reads and the consumers work well all the subscriptions quickly reach the max value (100). So if you are lucky they read the same ranges. If you are not lucky, because they start from different positions then this feature does not help. By the way this means that it is likely that you have: 100...200,200...300,180...280 but not 100...200, 120...180 So it is unlikely that you see a range that is totally included in a pending read. a note about "merging": the main problem is that we have to choose the "best way" of combining what's going on and what you can do: A) use parts of the pending reads (how many reads do you want to concatenate?) B) add one (or two) new read to fill the missing entries and merge C) start a new read from scratch Also, if you go for B) and not for C) other new readers may miss a chance to leverage your range I will do further testing in order to understand if merging two existing pending reads improves the situation. Enrico > > Thanks, > Haiting > > On Thu, Aug 25, 2022 at 11:40 AM Haiting Jiang > wrote: > > > +1 for this optimization for the specific scenario. > > > > This simple patch is really effective, in some tests it reduces a lot > >> (from 100K reads/s to 250 reads/s) the concurrent reads to BK. > >> You may say that it is unlikely that the broker sends concurrent > >> requests for exactly the same range of entries but empirically I have > >> observed that it happens quite often, at least in the scenario I drew > >> above. > > > > > > This is quite an interesting test result. My guess is that all the > > subscriptions > > have exactly the same backlog and all the messages are consumed instantly > > and successfully. So that they can read the same range of entries. > > Once there are some consuming delays or retries due to failures, I believe > > there will be read position shifts and the BK read requests will go up. > > > > Thanks, > > Haiting > > > > On Thu, Aug 25, 2022 at 12:12 AM Michael Marshall > > wrote: > > > >> +1 I support merging the temporary solution. > >> > >> > Just one point, please make sure the change will not introduce too much > >> > heap memory overhead. > >> > >> If we see that there is extra pressure, I wonder if we can dynamically > >> determine when to deduplicate requests. A broker only needs this > >> feature when a topic has multiple subscriptions. I haven't double > >> checked if this information is easily accessible to the cache. > >> > >> One of the interesting benefits of this optimization is that it will > >> decrease direct memory pressure. When we do not deduplicate reads, the > >> same entry will be duplicated in the bookie's netty direct memory > >> pool. With this design, an entry will be materialized once with many > >> different ByteBuffer's objects pointing to it. > >> > >> Thanks, > >> Michael > >> > >> On Wed, Aug 24, 2022 at 11:00 AM PengHui Li wrote: > >> > > >> > +1 for we can have a short-term solution first. > >> > > >> > Just one point, please make sure the change will not introduce too much > >> > heap memory overhead. > >> > > >> > Thanks, > >> > Penghui > >> > > >> > On Wed, Aug 24, 2022 at 11:54 PM Enrico Olivelli > >> > wrote: > >> > > >> > > Hello folks, > >> > > These days I am working on a use case in which I have many > >> > > subscriptions on the same topic (~100). > >> > > If you get to a situation with a big backlog, the consumers are no > >> > > more able to leverage the broker cache then the broker starts to read > >> > > from BookKeeper. > >> > >
Re: [DISCUSS] Move PIPs to the codebase?
Hi all, I have pushed out a PR https://github.com/apache/pulsar/pull/17270 which wants to copy the PIP and proposal template to the codebase. The PR doesn't want to make some changes to the proposal process before we have approval. Since we will make some changes to the current PIP process, I want to copy the old one to the codebase so that all the subsequent updates will have historical records. Finally, the PIP process changes should be done with a new VOTE email. With https://github.com/apache/pulsar/pull/17270, we can have a PR to show all the details which can help the reviewer to easily review. Thanks, Penghui On Thu, Aug 25, 2022 at 12:57 PM tison wrote: > Hi Penghui, > > > IMO, the final status of the proposal should be summarized in the > proposal. > > Agree. It's not exclusive. What I propose is that if you want to track > proposals in a project, track issues. But it seems your opinion is not to > do so. Then it's a nonissue. Flink FLIP tracks proposal status with the > proposal content and it works well. Although, I'd still highlight that > maintainer efforts are required to maintain the metadata. > > > As I know, if we have an issue tracking all the proposal status, not all > the contributors make contributions. Only the committers and the author? > > I don't propose to track the status _within_ the issue. But if you want to > track in a project, only committer have the privilege :) > > > if the discussion of the motivation doesn't have objections and is > discussed enough. > > This is quite vague and hard to follow. > > Rust RFCs have a nonmandatory pre-RFC stage. I think we can encourage the > proposer to discuss on the mailing list or any channel with committers > first. But any proposer can take their proposal number once proposed and a > committer clicks the merge button. Still, I'm concerned about adding such a > step to exclusively allocate a proposal number. > > I would suggest you collect the suggestions in this discussion thread, and > propose a process as shown in > > * https://github.com/rust-lang/rfcs#readme > * https://github.com/pingcap/tidb/tree/master/docs/design#readme > > So that we can have a vote. Instead of rushing into making several PRs to > move content fragmentedly. > > Best, > tison. > > > PengHui Li 于2022年8月25日周四 10:59写道： > > > Hi Tison, > > > > Thanks for providing more context. > > > > > Rust RFCs hold in a separate repo so that they can use the PR number as > > the > > proposal number to prevent duplicates. It's a bit overhead for Pulsar to > > hold a separate repo. So I'd prefer to hold proposals in the same repo > > under the dev/proposals folder and resolve the PR number duplicates issue > > as described above. Although, it can still be a bit awkward. But at least > > we can quickly notice duplicate numbers and resolve them. Using the PR > > number as the proposal number keeps it unique, but the number will be > > discontinuous. Especially if we're proposing in the main repo, the number > > will grow quickly. > > > > The PR number can provide unique numbers. Everyone can create PRs, maybe > > not related to PIP, but we are not able to revert it. > > > > > In this direction, we can associate a proposal with a tracking issue > and > > manipulate the proposal status represented by their tracking issue. It > > doesn't mean we keep the current issue discussion process. You can regard > > it as an umbrella ticket for the proposal. See > > https://github.com/pingcap/tidb/issues/26022 as an example. > > > > > In this direction, we should always ensure that there're several eyes > on > > the project. Otherwise, we can easily end up with an out-of-sync project. > > > > IMO, the final status of the proposal should be summarized in the > proposal. > > We can have some metadata for md file which allow us to have a page that > > can > > show all the proposals like the bookkeeper does > > https://bookkeeper.apache.org/community/bookkeeper-proposals > > From my personal view, I prefer this way of the bookkeeper. This will > allow > > all the users find the proposals on the website, instead of going to > search > > in the > > issue list. > > > > And the md file is the source of the truth. I think we can only provide > one > > way to > > update the proposal status (from a PR). So that all the contributors can > > contribute. > > As I know, if we have an issue tracking all the proposal status, not all > > the contributors > > make contributions. Only the committers and the author? Maybe the author > > can change > > the policy but still can't involve more people to review. > > > > > It seems the confusion comes from we have to merge one PR first to > > prevent > > duplicate numbers. If it's a certain concern, see also my comment above > > using the PR number as the proposal, or even host a separate proposal > repo > > as rust-lang/rfcs does and keep the number growing normally. I can > provide > > experience to maintain this sort of resource. > > > > Yes, it looks to re
[GitHub] [pulsar] Nintorac edited a comment on the discussion: Presto connector schema issue
GitHub user Nintorac edited a comment on the discussion: Presto connector schema issue And for logs in that folder I only see `logs/pulsar-standalone.log` and `logs/gc.log` GitHub link: https://github.com/apache/pulsar/discussions/17234#discussioncomment-3471296 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Nintorac added a comment to the discussion: Presto connector schema issue
GitHub user Nintorac added a comment to the discussion: Presto connector schema issue And for logs I only see `logs/pulsar-standalone.log` and `logs/gc.log` GitHub link: https://github.com/apache/pulsar/discussions/17234#discussioncomment-3471296 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] Nintorac added a comment to the discussion: Presto connector schema issue
GitHub user Nintorac added a comment to the discussion: Presto connector schema issue [pulsar-trino.tar.gz](https://github.com/apache/pulsar/files/9421624/pulsar-trino.tar.gz) Here is the setup -- all in docker compose. It is missing the pulsar presto connctor plugin folder found in the release for version `apache-pulsar-2.10.1`. I also changed the pulsar docker image to match `2.10.1` (it is `2.9.1` in the archive) and tried again but same error GitHub link: https://github.com/apache/pulsar/discussions/17234#discussioncomment-3471265 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar-site] Anonymitaet closed issue #172: Dark/Light mode toggle cannot be clicked for develop-binary-protocol page
Anonymitaet closed issue #172: Dark/Light mode toggle cannot be clicked for develop-binary-protocol page URL: https://github.com/apache/pulsar-site/issues/172 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [DISCUSS] PIP-Enable read-only partition, add read and write control for a single partition.
Where we store the metadata? I suggest we use the system topic to store the metadata instead of zk On 2022/08/25 02:36:56 baomingyu_5 wrote: > Hi all, > > In the exclusive consumption mode of exclusive, failover, reader, etc. if a > large number of messages accumulate due to limited consumption capacity or > consumption logic problems, expanding partitions or modifying consumption > logic cannot reduce the consumption of new messages in time. Influence, a > certain proportion of newly added messages will still be produced in the > accumulated paritition, and new messages cannot be quickly consumed by > restarted consumers or new consumers. In this way, the time affected by the > accumulation factor may very long. > > In order to reduce the impact on the business, through the control of read > and write permissions, configure some partitions as read-only, so that newly > produced messages can be sent to the partitions with less accumulation or > newly expanded, reducing the impact on the business caused by the > accumulation of messages. Reduce the overall time-consuming of > production--consumption.
[GitHub] [pulsar] gaoran10 added a comment to the discussion: Presto connector schema issue
GitHub user gaoran10 added a comment to the discussion: Presto connector schema issue Could you try to reproduce this problem in standalone mode? You should find log file `$PULSAR_HOME/lib/presto/var/log/server.log`. If there are reproduce steps, it's better. GitHub link: https://github.com/apache/pulsar/discussions/17234#discussioncomment-3471125 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
Re: [Discussion] PIP 198 - How to define [type] and [scope]?
Anyway, it's a separate topic to discuss. If you want to discuss issue types and whether to label components, please start another thread. Best, tison. tison 于2022年8月25日周四 13:12写道： > From current issue templates, we already sort issues into bug reports, > improvements, doc changes, flaky tests, and PIPs. They're types. [type] and > [component] described here are applied to commit messages, not for issues. > > For components, we may encourage contributors to try their best to sort > out related components, but it's generally hard to do. I report a bug, how > can I know which components are related? It is required I have to dig it > out? > > Best, > tison. > > > tison 于2022年8月25日周四 13:08写道： > >> > Agree with applying the same rules ( [type] [scope] summary) for >> writing issue titles. >> >> It cannot be guarded by check so I think it only increases contributors' >> overhead. >> >> Instead, we can try to find out some integration if we can use the GitHub >> issue forms dropdown widget to allow contributors to tag the issue. >> >> I don't know whether it's possible, but it's better than setting up title >> rules. I can foresee that it's seldomly followed. >> >> Best, >> tison. >> >> >> Liu Yu 于2022年8月25日周四 12:59写道： >> >>> Thanks Max! >>> >>> Agree with applying the same rules ( [type] [scope] summary) for writing >>> issue titles. >>> >>> On 2022/08/25 02:48:51 Max Xu wrote: >>> > LGTM. >>> > >>> > And I think we should also update our issue templates. >>> > >>> > Best, >>> > Max Xu >>> > >>> > >>> > On Tue, Aug 23, 2022 at 6:04 PM Yu wrote: >>> > >>> > > Hi team, >>> > > >>> > > Many thanks for your feedback! We've adjusted the convention based >>> on your >>> > > suggestions! >>> > > >>> > > Below is a brief summary of what we have reached a consensus on: >>> > > >>> > > >>> > > >>> > > 1. Convention >>> > > >>> > > Continue to follow our existing convention (it's customized on >>> Agular) [1] >>> > > >>> > > >>> > > >>> > > 2. Definition >>> > > >>> > > [type] must be one of the following: >>> > > - feat (abbr for "feature") >>> > > - improve >>> > > - fix >>> > > - cleanup >>> > > - refactor >>> > > - revert >>> > > >>> > > [scope] must be one of the following: >>> > > - admin >>> > > - broker >>> > > - cli (changes to CLI tools) >>> > > - io >>> > > - fn (abbr for "function") >>> > > - meta (abbr for "metadata") >>> > > - monitor >>> > > - proxy >>> > > - schema >>> > > - sec (abbr for "security") >>> > > - sql >>> > > - storage >>> > > - offload (changes to tiered storage) >>> > > - txn >>> > > - java >>> > > - cpp >>> > > - py >>> > > - ws (changes to WebSocket) >>> > > - test (changes to code tests) >>> > > - ci (changes to CI workflow) >>> > > - build (changes to dependencies, docker, build or release script) >>> > > - misc >>> > > - doc >>> > > - blog >>> > > - site >>> > > >>> > > For full details, see [Guide] Pulsar Pull Request Naming Convention >>> [2] >>> > > >>> > > >>> > > >>> > > If you have any concerns, feel free to comment before 13:00 August >>> 25 (UTC >>> > > +8). >>> > > >>> > > We'll start implementing it if there is no objection after that time. >>> > > >>> > > Thank you! >>> > > >>> > > >>> > > >>> > > [1] https://lists.apache.org/thread/90rcjf1dv0fbkb5hm31kmgr65fj0nfnn >>> > > [2] >>> > > >>> > > >>> https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.y8943h392zno >>> > > >>> > > Yu and mangoGoForward >>> > > >>> > > On Tue, Aug 23, 2022 at 5:59 PM Yu wrote: >>> > > >>> > > > Hi Jiuming, Yunze, tison, >>> > > > Thanks for your vote! >>> > > > >>> > > > >>> > > > >>> > > > Hi tison, >>> > > > >>> > > > > "packaging logics" >>> > > > > For example, build the docker image, build & publish shell >>> scripts. >>> > > > >>> > > > If you refer to these changes, they belong to [build] scope. >>> > > > >>> > > > Yu and Zixuan >>> > > > >>> > > > On Tue, Aug 23, 2022 at 1:25 PM tison >>> wrote: >>> > > > >>> > > >> Hi Yu, >>> > > >> >>> > > >> Reply inline: >>> > > >> >>> > > >> > Besides, the existing scope, [tool], refers to Pulsar CLI tools >>> [1]. >>> > > >> > We're considering to rename it to [cli] since: >>> > > >> >>> > > >> Make sense. >>> > > >> >>> > > >> > "deployment logic" If so, can we ignore this? >>> > > >> >>> > > >> I saw you already remove [deploy] scope. No comment here. It >>> should be >>> > > >> fine. >>> > > >> >>> > > >> > "packaging logics" >>> > > >> >>> > > >> For example, build the docker image, build & publish shell >>> scripts. >>> > > >> >>> > > >> > How about defining [build] refer to the following? >>> > > >> >>> > > >> Make sense. >>> > > >> >>> > > >> > Two quick questions need your vote! >>> > > >> >>> > > >> To save letters, B & A. >>> > > >> >>> > > >> Best, >>> > > >> tison. >>> > > >> >>> > > > >>> > > >>> > >>> >>
Re: [Discussion] PIP 198 - How to define [type] and [scope]?
>From current issue templates, we already sort issues into bug reports, improvements, doc changes, flaky tests, and PIPs. They're types. [type] and [component] described here are applied to commit messages, not for issues. For components, we may encourage contributors to try their best to sort out related components, but it's generally hard to do. I report a bug, how can I know which components are related? It is required I have to dig it out? Best, tison. tison 于2022年8月25日周四 13:08写道： > > Agree with applying the same rules ( [type] [scope] summary) for > writing issue titles. > > It cannot be guarded by check so I think it only increases contributors' > overhead. > > Instead, we can try to find out some integration if we can use the GitHub > issue forms dropdown widget to allow contributors to tag the issue. > > I don't know whether it's possible, but it's better than setting up title > rules. I can foresee that it's seldomly followed. > > Best, > tison. > > > Liu Yu 于2022年8月25日周四 12:59写道： > >> Thanks Max! >> >> Agree with applying the same rules ( [type] [scope] summary) for writing >> issue titles. >> >> On 2022/08/25 02:48:51 Max Xu wrote: >> > LGTM. >> > >> > And I think we should also update our issue templates. >> > >> > Best, >> > Max Xu >> > >> > >> > On Tue, Aug 23, 2022 at 6:04 PM Yu wrote: >> > >> > > Hi team, >> > > >> > > Many thanks for your feedback! We've adjusted the convention based on >> your >> > > suggestions! >> > > >> > > Below is a brief summary of what we have reached a consensus on: >> > > >> > > >> > > >> > > 1. Convention >> > > >> > > Continue to follow our existing convention (it's customized on >> Agular) [1] >> > > >> > > >> > > >> > > 2. Definition >> > > >> > > [type] must be one of the following: >> > > - feat (abbr for "feature") >> > > - improve >> > > - fix >> > > - cleanup >> > > - refactor >> > > - revert >> > > >> > > [scope] must be one of the following: >> > > - admin >> > > - broker >> > > - cli (changes to CLI tools) >> > > - io >> > > - fn (abbr for "function") >> > > - meta (abbr for "metadata") >> > > - monitor >> > > - proxy >> > > - schema >> > > - sec (abbr for "security") >> > > - sql >> > > - storage >> > > - offload (changes to tiered storage) >> > > - txn >> > > - java >> > > - cpp >> > > - py >> > > - ws (changes to WebSocket) >> > > - test (changes to code tests) >> > > - ci (changes to CI workflow) >> > > - build (changes to dependencies, docker, build or release script) >> > > - misc >> > > - doc >> > > - blog >> > > - site >> > > >> > > For full details, see [Guide] Pulsar Pull Request Naming Convention >> [2] >> > > >> > > >> > > >> > > If you have any concerns, feel free to comment before 13:00 August 25 >> (UTC >> > > +8). >> > > >> > > We'll start implementing it if there is no objection after that time. >> > > >> > > Thank you! >> > > >> > > >> > > >> > > [1] https://lists.apache.org/thread/90rcjf1dv0fbkb5hm31kmgr65fj0nfnn >> > > [2] >> > > >> > > >> https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.y8943h392zno >> > > >> > > Yu and mangoGoForward >> > > >> > > On Tue, Aug 23, 2022 at 5:59 PM Yu wrote: >> > > >> > > > Hi Jiuming, Yunze, tison, >> > > > Thanks for your vote! >> > > > >> > > > >> > > > >> > > > Hi tison, >> > > > >> > > > > "packaging logics" >> > > > > For example, build the docker image, build & publish shell >> scripts. >> > > > >> > > > If you refer to these changes, they belong to [build] scope. >> > > > >> > > > Yu and Zixuan >> > > > >> > > > On Tue, Aug 23, 2022 at 1:25 PM tison wrote: >> > > > >> > > >> Hi Yu, >> > > >> >> > > >> Reply inline: >> > > >> >> > > >> > Besides, the existing scope, [tool], refers to Pulsar CLI tools >> [1]. >> > > >> > We're considering to rename it to [cli] since: >> > > >> >> > > >> Make sense. >> > > >> >> > > >> > "deployment logic" If so, can we ignore this? >> > > >> >> > > >> I saw you already remove [deploy] scope. No comment here. It >> should be >> > > >> fine. >> > > >> >> > > >> > "packaging logics" >> > > >> >> > > >> For example, build the docker image, build & publish shell scripts. >> > > >> >> > > >> > How about defining [build] refer to the following? >> > > >> >> > > >> Make sense. >> > > >> >> > > >> > Two quick questions need your vote! >> > > >> >> > > >> To save letters, B & A. >> > > >> >> > > >> Best, >> > > >> tison. >> > > >> >> > > > >> > > >> > >> >
Re: [Discussion] PIP 198 - How to define [type] and [scope]?
> Agree with applying the same rules ( [type] [scope] summary) for writing issue titles. It cannot be guarded by check so I think it only increases contributors' overhead. Instead, we can try to find out some integration if we can use the GitHub issue forms dropdown widget to allow contributors to tag the issue. I don't know whether it's possible, but it's better than setting up title rules. I can foresee that it's seldomly followed. Best, tison. Liu Yu 于2022年8月25日周四 12:59写道： > Thanks Max! > > Agree with applying the same rules ( [type] [scope] summary) for writing > issue titles. > > On 2022/08/25 02:48:51 Max Xu wrote: > > LGTM. > > > > And I think we should also update our issue templates. > > > > Best, > > Max Xu > > > > > > On Tue, Aug 23, 2022 at 6:04 PM Yu wrote: > > > > > Hi team, > > > > > > Many thanks for your feedback! We've adjusted the convention based on > your > > > suggestions! > > > > > > Below is a brief summary of what we have reached a consensus on: > > > > > > > > > > > > 1. Convention > > > > > > Continue to follow our existing convention (it's customized on Agular) > [1] > > > > > > > > > > > > 2. Definition > > > > > > [type] must be one of the following: > > > - feat (abbr for "feature") > > > - improve > > > - fix > > > - cleanup > > > - refactor > > > - revert > > > > > > [scope] must be one of the following: > > > - admin > > > - broker > > > - cli (changes to CLI tools) > > > - io > > > - fn (abbr for "function") > > > - meta (abbr for "metadata") > > > - monitor > > > - proxy > > > - schema > > > - sec (abbr for "security") > > > - sql > > > - storage > > > - offload (changes to tiered storage) > > > - txn > > > - java > > > - cpp > > > - py > > > - ws (changes to WebSocket) > > > - test (changes to code tests) > > > - ci (changes to CI workflow) > > > - build (changes to dependencies, docker, build or release script) > > > - misc > > > - doc > > > - blog > > > - site > > > > > > For full details, see [Guide] Pulsar Pull Request Naming Convention [2] > > > > > > > > > > > > If you have any concerns, feel free to comment before 13:00 August 25 > (UTC > > > +8). > > > > > > We'll start implementing it if there is no objection after that time. > > > > > > Thank you! > > > > > > > > > > > > [1] https://lists.apache.org/thread/90rcjf1dv0fbkb5hm31kmgr65fj0nfnn > > > [2] > > > > > > > https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.y8943h392zno > > > > > > Yu and mangoGoForward > > > > > > On Tue, Aug 23, 2022 at 5:59 PM Yu wrote: > > > > > > > Hi Jiuming, Yunze, tison, > > > > Thanks for your vote! > > > > > > > > > > > > > > > > Hi tison, > > > > > > > > > "packaging logics" > > > > > For example, build the docker image, build & publish shell scripts. > > > > > > > > If you refer to these changes, they belong to [build] scope. > > > > > > > > Yu and Zixuan > > > > > > > > On Tue, Aug 23, 2022 at 1:25 PM tison wrote: > > > > > > > >> Hi Yu, > > > >> > > > >> Reply inline: > > > >> > > > >> > Besides, the existing scope, [tool], refers to Pulsar CLI tools > [1]. > > > >> > We're considering to rename it to [cli] since: > > > >> > > > >> Make sense. > > > >> > > > >> > "deployment logic" If so, can we ignore this? > > > >> > > > >> I saw you already remove [deploy] scope. No comment here. It should > be > > > >> fine. > > > >> > > > >> > "packaging logics" > > > >> > > > >> For example, build the docker image, build & publish shell scripts. > > > >> > > > >> > How about defining [build] refer to the following? > > > >> > > > >> Make sense. > > > >> > > > >> > Two quick questions need your vote! > > > >> > > > >> To save letters, B & A. > > > >> > > > >> Best, > > > >> tison. > > > >> > > > > > > > > > >
[VOTE RESULT] PIP 198: Standardize PR Naming Convention using GitHub Actions
Hi team, As discussed previously [1], we'll start implementing PIP 198: Standardize PR Naming Convention using GitHub Actions [1] based on the results voted by the community. Below is a brief summary of the vote results. 1. Convention Continue to follow our existing convention (it's customized on Agular) [3] 2. Definition [type] must be one of the following: - feat (abbr for "feature") - improve - fix - cleanup - refactor - revert [scope] must be one of the following: - admin - broker - cli (changes to CLI tools) - io - fn (abbr for "function") - meta (abbr for "metadata") - monitor - proxy - schema - sec (abbr for "security") - sql - storage - offload (changes to tiered storage) - txn - java - cpp - py - ws (changes to WebSocket) - test (changes to code tests) - ci (changes to CI workflow) - build (changes to dependencies, docker, build or release script) - misc - doc - blog - site For full details, see [Guide] Pulsar Pull Request Naming Convention [4] [1] https://lists.apache.org/thread/11hpppllkh68zodromd7pzn173gb9hgl [2] https://docs.google.com/document/d/1sJlUNAHnYAbvu9UtEgCrn_oVTnVc1M5nHC19x1bFab4/edit?pli=1 [3] https://lists.apache.org/thread/90rcjf1dv0fbkb5hm31kmgr65fj0nfnn [4] https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.y8943h392zno Yu, mangoGoForward, Max
Re: [Discussion] PIP 198 - How to define [type] and [scope]?
Thanks Max! Agree with applying the same rules ( [type] [scope] summary) for writing issue titles. On 2022/08/25 02:48:51 Max Xu wrote: > LGTM. > > And I think we should also update our issue templates. > > Best, > Max Xu > > > On Tue, Aug 23, 2022 at 6:04 PM Yu wrote: > > > Hi team, > > > > Many thanks for your feedback! We've adjusted the convention based on your > > suggestions! > > > > Below is a brief summary of what we have reached a consensus on: > > > > > > > > 1. Convention > > > > Continue to follow our existing convention (it's customized on Agular) [1] > > > > > > > > 2. Definition > > > > [type] must be one of the following: > > - feat (abbr for "feature") > > - improve > > - fix > > - cleanup > > - refactor > > - revert > > > > [scope] must be one of the following: > > - admin > > - broker > > - cli (changes to CLI tools) > > - io > > - fn (abbr for "function") > > - meta (abbr for "metadata") > > - monitor > > - proxy > > - schema > > - sec (abbr for "security") > > - sql > > - storage > > - offload (changes to tiered storage) > > - txn > > - java > > - cpp > > - py > > - ws (changes to WebSocket) > > - test (changes to code tests) > > - ci (changes to CI workflow) > > - build (changes to dependencies, docker, build or release script) > > - misc > > - doc > > - blog > > - site > > > > For full details, see [Guide] Pulsar Pull Request Naming Convention [2] > > > > > > > > If you have any concerns, feel free to comment before 13:00 August 25 (UTC > > +8). > > > > We'll start implementing it if there is no objection after that time. > > > > Thank you! > > > > > > > > [1] https://lists.apache.org/thread/90rcjf1dv0fbkb5hm31kmgr65fj0nfnn > > [2] > > > > https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.y8943h392zno > > > > Yu and mangoGoForward > > > > On Tue, Aug 23, 2022 at 5:59 PM Yu wrote: > > > > > Hi Jiuming, Yunze, tison, > > > Thanks for your vote! > > > > > > > > > > > > Hi tison, > > > > > > > "packaging logics" > > > > For example, build the docker image, build & publish shell scripts. > > > > > > If you refer to these changes, they belong to [build] scope. > > > > > > Yu and Zixuan > > > > > > On Tue, Aug 23, 2022 at 1:25 PM tison wrote: > > > > > >> Hi Yu, > > >> > > >> Reply inline: > > >> > > >> > Besides, the existing scope, [tool], refers to Pulsar CLI tools [1]. > > >> > We're considering to rename it to [cli] since: > > >> > > >> Make sense. > > >> > > >> > "deployment logic" If so, can we ignore this? > > >> > > >> I saw you already remove [deploy] scope. No comment here. It should be > > >> fine. > > >> > > >> > "packaging logics" > > >> > > >> For example, build the docker image, build & publish shell scripts. > > >> > > >> > How about defining [build] refer to the following? > > >> > > >> Make sense. > > >> > > >> > Two quick questions need your vote! > > >> > > >> To save letters, B & A. > > >> > > >> Best, > > >> tison. > > >> > > > > > >
Re: [DISCUSS] Move PIPs to the codebase?
Hi Penghui, > IMO, the final status of the proposal should be summarized in the proposal. Agree. It's not exclusive. What I propose is that if you want to track proposals in a project, track issues. But it seems your opinion is not to do so. Then it's a nonissue. Flink FLIP tracks proposal status with the proposal content and it works well. Although, I'd still highlight that maintainer efforts are required to maintain the metadata. > As I know, if we have an issue tracking all the proposal status, not all the contributors make contributions. Only the committers and the author? I don't propose to track the status _within_ the issue. But if you want to track in a project, only committer have the privilege :) > if the discussion of the motivation doesn't have objections and is discussed enough. This is quite vague and hard to follow. Rust RFCs have a nonmandatory pre-RFC stage. I think we can encourage the proposer to discuss on the mailing list or any channel with committers first. But any proposer can take their proposal number once proposed and a committer clicks the merge button. Still, I'm concerned about adding such a step to exclusively allocate a proposal number. I would suggest you collect the suggestions in this discussion thread, and propose a process as shown in * https://github.com/rust-lang/rfcs#readme * https://github.com/pingcap/tidb/tree/master/docs/design#readme So that we can have a vote. Instead of rushing into making several PRs to move content fragmentedly. Best, tison. PengHui Li 于2022年8月25日周四 10:59写道： > Hi Tison, > > Thanks for providing more context. > > > Rust RFCs hold in a separate repo so that they can use the PR number as > the > proposal number to prevent duplicates. It's a bit overhead for Pulsar to > hold a separate repo. So I'd prefer to hold proposals in the same repo > under the dev/proposals folder and resolve the PR number duplicates issue > as described above. Although, it can still be a bit awkward. But at least > we can quickly notice duplicate numbers and resolve them. Using the PR > number as the proposal number keeps it unique, but the number will be > discontinuous. Especially if we're proposing in the main repo, the number > will grow quickly. > > The PR number can provide unique numbers. Everyone can create PRs, maybe > not related to PIP, but we are not able to revert it. > > > In this direction, we can associate a proposal with a tracking issue and > manipulate the proposal status represented by their tracking issue. It > doesn't mean we keep the current issue discussion process. You can regard > it as an umbrella ticket for the proposal. See > https://github.com/pingcap/tidb/issues/26022 as an example. > > > In this direction, we should always ensure that there're several eyes on > the project. Otherwise, we can easily end up with an out-of-sync project. > > IMO, the final status of the proposal should be summarized in the proposal. > We can have some metadata for md file which allow us to have a page that > can > show all the proposals like the bookkeeper does > https://bookkeeper.apache.org/community/bookkeeper-proposals > From my personal view, I prefer this way of the bookkeeper. This will allow > all the users find the proposals on the website, instead of going to search > in the > issue list. > > And the md file is the source of the truth. I think we can only provide one > way to > update the proposal status (from a PR). So that all the contributors can > contribute. > As I know, if we have an issue tracking all the proposal status, not all > the contributors > make contributions. Only the committers and the author? Maybe the author > can change > the policy but still can't involve more people to review. > > > It seems the confusion comes from we have to merge one PR first to > prevent > duplicate numbers. If it's a certain concern, see also my comment above > using the PR number as the proposal, or even host a separate proposal repo > as rust-lang/rfcs does and keep the number growing normally. I can provide > experience to maintain this sort of resource. > > Yes, it looks to request a PIP number through a PR. IMO, it is necessary. > The reviewer can only merge the PR (request the PIP number) if the > discussion > of the motivation doesn't have objections and is discussed enough. > Without this step, maybe only the author thinks that "I can create a > proposal for now" (maybe > no response on the mailing list). > > Best, > Penghui > > On Thu, Aug 25, 2022 at 10:31 AM PengHui Li wrote: > > > > So that we can review the details and improve these 2 documents (I see > > the last update > > for these two documents are 2021 and 2020) > > > > Sorry, I think it's not a good idea. > > We should move to the codebase 100% the same as the original > documentation. > > And use another PR to make improvements for it. If it is needed, we > should > > start with an email. > > We need to have historical records. > > > > Penghui > > > > > > On Thu, Aug 2
Re: [VOTE] [PIP-201] Extensions mechanism for Pulsar Admin CLI tools
+1 (non-binding) It's a cool idea! Best, Max Xu On Wed, Aug 24, 2022 at 9:46 PM PengHui Li wrote: > +1 (binding) > > Best, > Penghui > > On Wed, Aug 24, 2022 at 9:39 PM Dave Fisher wrote: > > > +1 (binding) > > > > Best, > > Dave > > > > Sent from my iPhone > > > > > On Aug 24, 2022, at 12:43 AM, Yunze Xu > > wrote: > > > > > > +1 (non binding) > > > > > > Thanks, > > > Yunze > > > > > > > > > > > > > > >> 2022年8月24日 15:38，Nicolò Boschi 写道： > > >> > > >> +1 (non binding) > > >> Nicolò Boschi > > >> > > >> > > >>> Il giorno mer 24 ago 2022 alle ore 09:11 Enrico Olivelli < > > >>> eolive...@gmail.com> ha scritto: > > >>> > > >>> Hello, > > >>> this is the official thread VOTE for PIP-201 Extensions mechanism for > > >>> Pulsar Admin CLI tools > > >>> > > >>> This is the PIP link https://github.com/apache/pulsar/issues/17155 > > >>> This is the discussion: > > >>> https://lists.apache.org/thread/287ft8twc11cp4s1y4qkcx5nmh451cyo > > >>> > > >>> I am still working on the PR, that is the subject of the VOTE. > > >>> > > >>> Best regards > > >>> Enrico Olivelli > > >>> > > > > > > > >
[GitHub] [pulsar] liangyuanpeng added a comment to the discussion: I encountered a port error when using pulsar to integrate etcd
GitHub user liangyuanpeng added a comment to the discussion: I encountered a port error when using pulsar to integrate etcd I have tests on the docker image of `docker pull lypgcs/pulsar:2022-08-23` and it's working for etcd cluster. You need Change `etcd:node01:2379,node02:2379,node03:2379` to `etcd:http://node01:2379,http://node02:2379,http://node03:2379` GitHub link: https://github.com/apache/pulsar/discussions/17240#discussioncomment-3471019 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar-site] Anonymitaet commented on pull request #176: Generate client config docs from source code
Anonymitaet commented on PR #176: URL: https://github.com/apache/pulsar-site/pull/176#issuecomment-1226745537 @urfreespace Could you please review this PR from a technical perspective? Thank you! 😊 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-test-infra] maxsxu commented on a diff in pull request #68: [improve][docbot] Move the preceding steps into the action
maxsxu commented on code in PR #68: URL: https://github.com/apache/pulsar-test-infra/pull/68#discussion_r954477706 ## docbot/action.yml: ## @@ -8,6 +8,16 @@ branding: runs: using: composite steps: -- run: go run main.go +- name: Checkout + uses: actions/checkout@v3 + with: +repository: pulsar/pulsar-test-infra +ref: master Review Comment: Removed, let ref uses the default branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-test-infra] Anonymitaet closed pull request #36: [Docbot] Add doc-added label
Anonymitaet closed pull request #36: [Docbot] Add doc-added label URL: https://github.com/apache/pulsar-test-infra/pull/36 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-test-infra] Anonymitaet commented on pull request #36: [Docbot] Add doc-added label
Anonymitaet commented on PR #36: URL: https://github.com/apache/pulsar-test-infra/pull/36#issuecomment-1226743227 Thanks for your contribution! Already resolved https://github.com/apache/pulsar/blob/b21f72886dc20362efe6808d690ddca32c5abd1e/.github/workflows/ci-documentbot.yml#L56 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [DISCUSS] Catch up reads with many subscriptions - saving reads from BK
Here is my idea of handing the read position shift if we want to go a bit further with this solution. Instead of only checking the exact match of entry range match (PendingReadKey in the PR), we can check if the requesting range is covered by several ongoing read requests, and merge the result of these requests. For example, currently there are already inflight request reads of entries [10,20] and [20,30], and now comes a new request [15,25], it can just merge the result from [10,20] and [20,30], and avoids sending the request to BK. As for the complexity checking procedure, I think we can sort all the inflight requests by startEntryId and apply a binary search here, to reduce the time complexity to O(logN) where N is the max entry size in one ledger. Thanks, Haiting On Thu, Aug 25, 2022 at 11:40 AM Haiting Jiang wrote: > +1 for this optimization for the specific scenario. > > This simple patch is really effective, in some tests it reduces a lot >> (from 100K reads/s to 250 reads/s) the concurrent reads to BK. >> You may say that it is unlikely that the broker sends concurrent >> requests for exactly the same range of entries but empirically I have >> observed that it happens quite often, at least in the scenario I drew >> above. > > > This is quite an interesting test result. My guess is that all the > subscriptions > have exactly the same backlog and all the messages are consumed instantly > and successfully. So that they can read the same range of entries. > Once there are some consuming delays or retries due to failures, I believe > there will be read position shifts and the BK read requests will go up. > > Thanks, > Haiting > > On Thu, Aug 25, 2022 at 12:12 AM Michael Marshall > wrote: > >> +1 I support merging the temporary solution. >> >> > Just one point, please make sure the change will not introduce too much >> > heap memory overhead. >> >> If we see that there is extra pressure, I wonder if we can dynamically >> determine when to deduplicate requests. A broker only needs this >> feature when a topic has multiple subscriptions. I haven't double >> checked if this information is easily accessible to the cache. >> >> One of the interesting benefits of this optimization is that it will >> decrease direct memory pressure. When we do not deduplicate reads, the >> same entry will be duplicated in the bookie's netty direct memory >> pool. With this design, an entry will be materialized once with many >> different ByteBuffer's objects pointing to it. >> >> Thanks, >> Michael >> >> On Wed, Aug 24, 2022 at 11:00 AM PengHui Li wrote: >> > >> > +1 for we can have a short-term solution first. >> > >> > Just one point, please make sure the change will not introduce too much >> > heap memory overhead. >> > >> > Thanks, >> > Penghui >> > >> > On Wed, Aug 24, 2022 at 11:54 PM Enrico Olivelli >> > wrote: >> > >> > > Hello folks, >> > > These days I am working on a use case in which I have many >> > > subscriptions on the same topic (~100). >> > > If you get to a situation with a big backlog, the consumers are no >> > > more able to leverage the broker cache then the broker starts to read >> > > from BookKeeper. >> > > >> > > In 2.11 we have a first mechanism of catch-up reads cache for >> > > backlogged consumers (when you have a few backlogged cursors we start >> > > putting in the cache the entries that you read). >> > > >> > > That is not enough to deal with my case because if you have many >> > > consumers (one per partition) that connect all together, for instance >> > > after a major problem in the application, the cache is empty and the >> > > broker starts to overwhelm the bookies with reads. >> > > >> > > As a short term solution I have created this patch >> > > https://github.com/apache/pulsar/pull/17241 >> > > >> > > The patch basically saves reads from BK by preventing concurrent reads >> > > for the same entries (actually "ranges of entries"). >> > > If the broker starts a read for a range and then comes another read >> > > for the same range before the read completes, then we skip sending the >> > > new read and we use the result of the BK request. >> > > >> > > This simple patch is really effective, in some tests it reduces a lot >> > > (from 100K reads/s to 250 reads/s) the concurrent reads to BK. >> > > You may say that it is unlikely that the broker sends concurrent >> > > requests for exactly the same range of entries but empirically I have >> > > observed that it happens quite often, at least in the scenario I drew >> > > above. >> > > >> > > My proposal is to commit my patch as a short term solution. >> > > I have already talked with PengHui and a better solution would be to >> > > introduce explicitly a "Catch Up reads cache", but that is a mid term >> > > implementation. >> > > >> > > WDYT ? >> > > >> > > >> > > Enrico >> > > >> >
[GitHub] [pulsar-test-infra] Anonymitaet commented on pull request #60: Remove unused legacy docbot
Anonymitaet commented on PR #60: URL: https://github.com/apache/pulsar-test-infra/pull/60#issuecomment-1226742311 Hi @tisonkun @nodece thanks for your feedback! We (+ @maxsxu) prefer not to tag it. Reasons: 1. We've searched and found that nearly no one uses this code. 2. The quality of this code is not high. I can merge this PR if there are no more comments tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-test-infra] maxsxu commented on pull request #36: [Docbot] Add doc-added label
maxsxu commented on PR #36: URL: https://github.com/apache/pulsar-test-infra/pull/36#issuecomment-1226741194 Already addressed in new docbot. /cc @Anonymitaet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [DISCUSS] Catch up reads with many subscriptions - saving reads from BK
+1 for this optimization for the specific scenario. This simple patch is really effective, in some tests it reduces a lot > (from 100K reads/s to 250 reads/s) the concurrent reads to BK. > You may say that it is unlikely that the broker sends concurrent > requests for exactly the same range of entries but empirically I have > observed that it happens quite often, at least in the scenario I drew > above. This is quite an interesting test result. My guess is that all the subscriptions have exactly the same backlog and all the messages are consumed instantly and successfully. So that they can read the same range of entries. Once there are some consuming delays or retries due to failures, I believe there will be read position shifts and the BK read requests will go up. Thanks, Haiting On Thu, Aug 25, 2022 at 12:12 AM Michael Marshall wrote: > +1 I support merging the temporary solution. > > > Just one point, please make sure the change will not introduce too much > > heap memory overhead. > > If we see that there is extra pressure, I wonder if we can dynamically > determine when to deduplicate requests. A broker only needs this > feature when a topic has multiple subscriptions. I haven't double > checked if this information is easily accessible to the cache. > > One of the interesting benefits of this optimization is that it will > decrease direct memory pressure. When we do not deduplicate reads, the > same entry will be duplicated in the bookie's netty direct memory > pool. With this design, an entry will be materialized once with many > different ByteBuffer's objects pointing to it. > > Thanks, > Michael > > On Wed, Aug 24, 2022 at 11:00 AM PengHui Li wrote: > > > > +1 for we can have a short-term solution first. > > > > Just one point, please make sure the change will not introduce too much > > heap memory overhead. > > > > Thanks, > > Penghui > > > > On Wed, Aug 24, 2022 at 11:54 PM Enrico Olivelli > > wrote: > > > > > Hello folks, > > > These days I am working on a use case in which I have many > > > subscriptions on the same topic (~100). > > > If you get to a situation with a big backlog, the consumers are no > > > more able to leverage the broker cache then the broker starts to read > > > from BookKeeper. > > > > > > In 2.11 we have a first mechanism of catch-up reads cache for > > > backlogged consumers (when you have a few backlogged cursors we start > > > putting in the cache the entries that you read). > > > > > > That is not enough to deal with my case because if you have many > > > consumers (one per partition) that connect all together, for instance > > > after a major problem in the application, the cache is empty and the > > > broker starts to overwhelm the bookies with reads. > > > > > > As a short term solution I have created this patch > > > https://github.com/apache/pulsar/pull/17241 > > > > > > The patch basically saves reads from BK by preventing concurrent reads > > > for the same entries (actually "ranges of entries"). > > > If the broker starts a read for a range and then comes another read > > > for the same range before the read completes, then we skip sending the > > > new read and we use the result of the BK request. > > > > > > This simple patch is really effective, in some tests it reduces a lot > > > (from 100K reads/s to 250 reads/s) the concurrent reads to BK. > > > You may say that it is unlikely that the broker sends concurrent > > > requests for exactly the same range of entries but empirically I have > > > observed that it happens quite often, at least in the scenario I drew > > > above. > > > > > > My proposal is to commit my patch as a short term solution. > > > I have already talked with PengHui and a better solution would be to > > > introduce explicitly a "Catch Up reads cache", but that is a mid term > > > implementation. > > > > > > WDYT ? > > > > > > > > > Enrico > > > >
[GitHub] [pulsar-test-infra] tisonkun commented on a diff in pull request #68: [improve][docbot] Move the preceding steps into the action
tisonkun commented on code in PR #68: URL: https://github.com/apache/pulsar-test-infra/pull/68#discussion_r954466400 ## docbot/action.yml: ## @@ -8,6 +8,16 @@ branding: runs: using: composite steps: -- run: go run main.go +- name: Checkout + uses: actions/checkout@v3 + with: +repository: pulsar/pulsar-test-infra +ref: master Review Comment: Fair enough. If we remove this config, the default branch is also master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-test-infra] maxsxu commented on a diff in pull request #68: [improve][docbot] Move the preceding steps into the action
maxsxu commented on code in PR #68: URL: https://github.com/apache/pulsar-test-infra/pull/68#discussion_r954462336 ## docbot/action.yml: ## @@ -8,6 +8,16 @@ branding: runs: using: composite steps: -- run: go run main.go +- name: Checkout + uses: actions/checkout@v3 + with: +repository: pulsar/pulsar-test-infra +ref: master Review Comment: The current branch ref is master: https://github.com/apache/pulsar-test-infra/tree/master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] SignorMercurio opened a new issue, #177: Migrate all docs from `/tools` to `/reference`
SignorMercurio opened a new issue, #177: URL: https://github.com/apache/pulsar-site/issues/177 According to https://apache-pulsar.slack.com/archives/C03MG1H2B3N/p1661220553497289, we can start working on migrate all docs from `/tools` to `/reference`. For instance, https://pulsar.apache.org/tools/pulsar-admin/2.11.0-SNAPSHOT/ will be moved to https://pulsar.apache.org/reference/ and can be viewed in the `Pulsar Admin` section of the side navbar, just like `Pulsar Configuration` right now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-test-infra] maxsxu commented on a diff in pull request #68: [improve][docbot] Move the preceding steps into the action
maxsxu commented on code in PR #68: URL: https://github.com/apache/pulsar-test-infra/pull/68#discussion_r954459313 ## docbot/action.yml: ## @@ -8,6 +8,16 @@ branding: runs: using: composite steps: -- run: go run main.go +- name: Checkout + uses: actions/checkout@v3 + with: +repository: pulsar/pulsar-test-infra Review Comment: Fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [Vote] PIP-192 New Pulsar Broker Load Balancer
+1 (binding) Awesome job! Thanks, Hang Heesung Sohn 于2022年8月24日周三 05:29写道： > > Hi, Pulsar dev community, > > This vote has been open for 23 days, and we are closing this vote today as > this PIP received three binding +1s. > > We are moving forward and starting implementation -- there will be a series > of PRs and sub-PIPs(if further design discussions are required). > > Thank you, > Heesung > > On Tue, Aug 9, 2022 at 4:56 AM guo jiwei wrote: > > > +1 > > > > Regards > > Jiwei Guo (Tboy) > > > > > > On Tue, Aug 9, 2022 at 3:12 PM Qiang Huang > > wrote: > > > > > +1 (non-binding) > > > > > > Kai Wang 于2022年8月5日周五 10:18写道： > > > > > > > +1 (non-binding) > > > > > > > > Thanks, > > > > Kai > > > > > > > > Heesung Sohn 于2022年8月2日周二 > > > 08:50写道： > > > > > > > > > Dear Pulsar Community, > > > > > > > > > > Please review and vote on this PIP. > > > > > > > > > > PIP link: https://github.com/apache/pulsar/issues/16691 > > > > > > > > > > Thank you, > > > > > -Heesung > > > > > > > > > > > > > > > > > > -- > > > BR, > > > Qiang Huang > > > > >
Re: [DISCUSS] Move PIPs to the codebase?
Hi Tison, Thanks for providing more context. > Rust RFCs hold in a separate repo so that they can use the PR number as the proposal number to prevent duplicates. It's a bit overhead for Pulsar to hold a separate repo. So I'd prefer to hold proposals in the same repo under the dev/proposals folder and resolve the PR number duplicates issue as described above. Although, it can still be a bit awkward. But at least we can quickly notice duplicate numbers and resolve them. Using the PR number as the proposal number keeps it unique, but the number will be discontinuous. Especially if we're proposing in the main repo, the number will grow quickly. The PR number can provide unique numbers. Everyone can create PRs, maybe not related to PIP, but we are not able to revert it. > In this direction, we can associate a proposal with a tracking issue and manipulate the proposal status represented by their tracking issue. It doesn't mean we keep the current issue discussion process. You can regard it as an umbrella ticket for the proposal. See https://github.com/pingcap/tidb/issues/26022 as an example. > In this direction, we should always ensure that there're several eyes on the project. Otherwise, we can easily end up with an out-of-sync project. IMO, the final status of the proposal should be summarized in the proposal. We can have some metadata for md file which allow us to have a page that can show all the proposals like the bookkeeper does https://bookkeeper.apache.org/community/bookkeeper-proposals >From my personal view, I prefer this way of the bookkeeper. This will allow all the users find the proposals on the website, instead of going to search in the issue list. And the md file is the source of the truth. I think we can only provide one way to update the proposal status (from a PR). So that all the contributors can contribute. As I know, if we have an issue tracking all the proposal status, not all the contributors make contributions. Only the committers and the author? Maybe the author can change the policy but still can't involve more people to review. > It seems the confusion comes from we have to merge one PR first to prevent duplicate numbers. If it's a certain concern, see also my comment above using the PR number as the proposal, or even host a separate proposal repo as rust-lang/rfcs does and keep the number growing normally. I can provide experience to maintain this sort of resource. Yes, it looks to request a PIP number through a PR. IMO, it is necessary. The reviewer can only merge the PR (request the PIP number) if the discussion of the motivation doesn't have objections and is discussed enough. Without this step, maybe only the author thinks that "I can create a proposal for now" (maybe no response on the mailing list). Best, Penghui On Thu, Aug 25, 2022 at 10:31 AM PengHui Li wrote: > > So that we can review the details and improve these 2 documents (I see > the last update > for these two documents are 2021 and 2020) > > Sorry, I think it's not a good idea. > We should move to the codebase 100% the same as the original documentation. > And use another PR to make improvements for it. If it is needed, we should > start with an email. > We need to have historical records. > > Penghui > > > On Thu, Aug 25, 2022 at 10:28 AM PengHui Li wrote: > >> Hi Michael, >> >> > This raises the question of how to handle historical PIPs. We could do >> some work to move PIPs out of the wiki and out of issues into the code >> base. The wiki is just another git repo, so any committer should be >> able to help migrate PIPs in the wiki to the apache/pulsar git repo. >> This might also be an opportunity to double check the status on old >> PIPs. >> >> Yes, I think it's a good idea. IMO, we can migrate them one by one. >> To provide more context for each one like the discussion email, vote email >> , and the approvals (Of course, very early proposals did not require >> approval). >> So that all the contributors can help with the migration process. >> >> I think I can move [0] [1] to the codebase first. >> >> [0] >> https://github.com/apache/pulsar/wiki/Pulsar-Improvement-Proposal-Template >> [1] >> https://github.com/apache/pulsar/wiki/Pulsar-Improvement-Proposal-%28PIP%29 >> >> So that we can review the details and improve these 2 documents (I see >> the last update >> for these two documents are 2021 and 2020) >> >> Thanks, >> Penghui >> >> On Thu, Aug 25, 2022 at 9:44 AM tison wrote: >> >>> Hi Penghui, >>> >>> Thanks for starting this discussion! +1 for holding PIPs in the codebase. >>> >>> I'd like to share some experience working on open-source projects holding >>> proposals as code. >>> >>> 1. Rust RFCs: codebase https://github.com/rust-lang/rfcs >>> 2. TiDB Proposals: >>> https://github.com/pingcap/tidb/tree/master/docs/design#readme >>> >>> Reflect on topics in this thread: >>> >>> > any reviews should happen with a PR first >>> >>> The experience should be similar to reviewing on Google Docs while
[GitHub] [pulsar-test-infra] tisonkun commented on a diff in pull request #68: [improve][docbot] Move the preceding steps into the action
tisonkun commented on code in PR #68: URL: https://github.com/apache/pulsar-test-infra/pull/68#discussion_r954452284 ## docbot/action.yml: ## @@ -8,6 +8,16 @@ branding: runs: using: composite steps: -- run: go run main.go +- name: Checkout + uses: actions/checkout@v3 + with: +repository: pulsar/pulsar-test-infra Review Comment: ```suggestion repository: apache/pulsar-test-infra ``` ? ## docbot/action.yml: ## @@ -8,6 +8,16 @@ branding: runs: using: composite steps: -- run: go run main.go +- name: Checkout + uses: actions/checkout@v3 + with: +repository: pulsar/pulsar-test-infra +ref: master Review Comment: ```suggestion ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] SignorMercurio opened a new pull request, #176: Generate client config docs from source code
SignorMercurio opened a new pull request, #176: URL: https://github.com/apache/pulsar-site/pull/176 Related PR: apache/pulsar#17198. @urfreespace PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [Discussion] PIP 198 - How to define [type] and [scope]?
LGTM. And I think we should also update our issue templates. Best, Max Xu On Tue, Aug 23, 2022 at 6:04 PM Yu wrote: > Hi team, > > Many thanks for your feedback! We've adjusted the convention based on your > suggestions! > > Below is a brief summary of what we have reached a consensus on: > > > > 1. Convention > > Continue to follow our existing convention (it's customized on Agular) [1] > > > > 2. Definition > > [type] must be one of the following: > - feat (abbr for "feature") > - improve > - fix > - cleanup > - refactor > - revert > > [scope] must be one of the following: > - admin > - broker > - cli (changes to CLI tools) > - io > - fn (abbr for "function") > - meta (abbr for "metadata") > - monitor > - proxy > - schema > - sec (abbr for "security") > - sql > - storage > - offload (changes to tiered storage) > - txn > - java > - cpp > - py > - ws (changes to WebSocket) > - test (changes to code tests) > - ci (changes to CI workflow) > - build (changes to dependencies, docker, build or release script) > - misc > - doc > - blog > - site > > For full details, see [Guide] Pulsar Pull Request Naming Convention [2] > > > > If you have any concerns, feel free to comment before 13:00 August 25 (UTC > +8). > > We'll start implementing it if there is no objection after that time. > > Thank you! > > > > [1] https://lists.apache.org/thread/90rcjf1dv0fbkb5hm31kmgr65fj0nfnn > [2] > > https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.y8943h392zno > > Yu and mangoGoForward > > On Tue, Aug 23, 2022 at 5:59 PM Yu wrote: > > > Hi Jiuming, Yunze, tison, > > Thanks for your vote! > > > > > > > > Hi tison, > > > > > "packaging logics" > > > For example, build the docker image, build & publish shell scripts. > > > > If you refer to these changes, they belong to [build] scope. > > > > Yu and Zixuan > > > > On Tue, Aug 23, 2022 at 1:25 PM tison wrote: > > > >> Hi Yu, > >> > >> Reply inline: > >> > >> > Besides, the existing scope, [tool], refers to Pulsar CLI tools [1]. > >> > We're considering to rename it to [cli] since: > >> > >> Make sense. > >> > >> > "deployment logic" If so, can we ignore this? > >> > >> I saw you already remove [deploy] scope. No comment here. It should be > >> fine. > >> > >> > "packaging logics" > >> > >> For example, build the docker image, build & publish shell scripts. > >> > >> > How about defining [build] refer to the following? > >> > >> Make sense. > >> > >> > Two quick questions need your vote! > >> > >> To save letters, B & A. > >> > >> Best, > >> tison. > >> > > >
[DISCUSS] PIP-Enable read-only partition, add read and write control for a single partition.
Hi all,
In the exclusive consumption mode of exclusive, failover, reader, etc. if a
large number of messages accumulate due to limited consumption capacity or
consumption logic problems, expanding partitions or modifying consumption logic
cannot reduce the consumption of new messages in time. Influence, a certain
proportion of newly added messages will still be produced in the accumulated
paritition, and new messages cannot be quickly consumed by restarted consumers
or new consumers. In this way, the time affected by the accumulation factor may
very long.
In order to reduce the impact on the business, through the control of read and
write permissions, configure some partitions as read-only, so that newly
produced messages can be sent to the partitions with less accumulation or newly
expanded, reducing the impact on the business caused by the accumulation of
messages. Reduce the overall time-consuming of production--consumption.
This feature needs to be changed as follows：
1、PB protocl
message CommandPartitionedTopicMetadataResponse {
enum LookupType {
Success = 0;
Failed = 1;
}
optional uint32 partitions= 1; // Optional in case of error
required uint64 request_id= 2;
optional LookupType response = 3;
optional ServerError error= 4;
optional string message = 5;
repeated KeyValue metadata = 6；// key parition index，value：r、w、rw
}
2、Modify choosePartition method in all implement class of MessageRouter, and it
will not choose read-only partition to send msg.
3、ProducerImpl adds permission attributes, and if permission is read-only will
not be used sending msg.
4、Modify admin interface updatePartitionedTopic to update metadata of topic
Thanks,
Mingyu.
[DISCUSS] PIP-Enable read-only partition, add read and write control for a single partition.
Hi all, In the exclusive consumption mode of exclusive, failover, reader, etc. if a large number of messages accumulate due to limited consumption capacity or consumption logic problems, expanding partitions or modifying consumption logic cannot reduce the consumption of new messages in time. Influence, a certain proportion of newly added messages will still be produced in the accumulated paritition, and new messages cannot be quickly consumed by restarted consumers or new consumers. In this way, the time affected by the accumulation factor may very long. In order to reduce the impact on the business, through the control of read and write permissions, configure some partitions as read-only, so that newly produced messages can be sent to the partitions with less accumulation or newly expanded, reducing the impact on the business caused by the accumulation of messages. Reduce the overall time-consuming of production--consumption.
Re: [DISCUSS] Move PIPs to the codebase?
> So that we can review the details and improve these 2 documents (I see the last update for these two documents are 2021 and 2020) Sorry, I think it's not a good idea. We should move to the codebase 100% the same as the original documentation. And use another PR to make improvements for it. If it is needed, we should start with an email. We need to have historical records. Penghui On Thu, Aug 25, 2022 at 10:28 AM PengHui Li wrote: > Hi Michael, > > > This raises the question of how to handle historical PIPs. We could do > some work to move PIPs out of the wiki and out of issues into the code > base. The wiki is just another git repo, so any committer should be > able to help migrate PIPs in the wiki to the apache/pulsar git repo. > This might also be an opportunity to double check the status on old > PIPs. > > Yes, I think it's a good idea. IMO, we can migrate them one by one. > To provide more context for each one like the discussion email, vote email > , and the approvals (Of course, very early proposals did not require > approval). > So that all the contributors can help with the migration process. > > I think I can move [0] [1] to the codebase first. > > [0] > https://github.com/apache/pulsar/wiki/Pulsar-Improvement-Proposal-Template > [1] > https://github.com/apache/pulsar/wiki/Pulsar-Improvement-Proposal-%28PIP%29 > > So that we can review the details and improve these 2 documents (I see the > last update > for these two documents are 2021 and 2020) > > Thanks, > Penghui > > On Thu, Aug 25, 2022 at 9:44 AM tison wrote: > >> Hi Penghui, >> >> Thanks for starting this discussion! +1 for holding PIPs in the codebase. >> >> I'd like to share some experience working on open-source projects holding >> proposals as code. >> >> 1. Rust RFCs: codebase https://github.com/rust-lang/rfcs >> 2. TiDB Proposals: >> https://github.com/pingcap/tidb/tree/master/docs/design#readme >> >> Reflect on topics in this thread: >> >> > any reviews should happen with a PR first >> >> The experience should be similar to reviewing on Google Docs while keeping >> all conversations on GitHub and thus synced on the mailing list. You can >> preview can some examples: >> >> * https://github.com/pingcap/tidb/pull/23673 >> * https://github.com/rust-lang/rfcs/pull/42 >> >> Furthermore, continuous follow-up like PIP-198 can discuss implementation >> details and update the proposal content one PR by another. The downside is >> that even trivial typo fixes should go through a PR. >> >> > Certainly, all the votes should happen on the mailing list. >> >> Yes. We should have an explicit [VOTE] and [RESULT][VOTE] thread on the >> mailing list while keeping a reference within the proposal content. We can >> port the current PIP issue template to a proposal template as Rust RFCs to >> highlight it. >> >> > We can just add a link that points to the PIPs dir to the contribution >> guide >> or README. >> >> Rust RFCs hold in a separate repo so that they can use the PR number as >> the >> proposal number to prevent duplicates. It's a bit overhead for Pulsar to >> hold a separate repo. So I'd prefer to hold proposals in the same repo >> under the dev/proposals folder and resolve the PR number duplicates issue >> as described above. Although, it can still be a bit awkward. But at least >> we can quickly notice duplicate numbers and resolve them. Using the PR >> number as the proposal number keeps it unique, but the number will be >> discontinuous. Especially if we're proposing in the main repo, the number >> will grow quickly. >> >> > Should we track the status of PIPs as a project? >> >> In this direction, we can associate a proposal with a tracking issue and >> manipulate the proposal status represented by their tracking issue. It >> doesn't mean we keep the current issue discussion process. You can regard >> it as an umbrella ticket for the proposal. See >> https://github.com/pingcap/tidb/issues/26022 as an example. >> >> In this direction, we should always ensure that there're several eyes on >> the project. Otherwise, we can easily end up with an out-of-sync project. >> >> To @Rajan @Penghui >> >> It seems the confusion comes from we have to merge one PR first to prevent >> duplicate numbers. If it's a certain concern, see also my comment above >> using the PR number as the proposal, or even host a separate proposal repo >> as rust-lang/rfcs does and keep the number growing normally. I can provide >> experience to maintain this sort of resource. >> >> >> Best, >> tison. >> >
Re: [DISCUSS] Move PIPs to the codebase?
Hi Michael, > This raises the question of how to handle historical PIPs. We could do some work to move PIPs out of the wiki and out of issues into the code base. The wiki is just another git repo, so any committer should be able to help migrate PIPs in the wiki to the apache/pulsar git repo. This might also be an opportunity to double check the status on old PIPs. Yes, I think it's a good idea. IMO, we can migrate them one by one. To provide more context for each one like the discussion email, vote email , and the approvals (Of course, very early proposals did not require approval). So that all the contributors can help with the migration process. I think I can move [0] [1] to the codebase first. [0] https://github.com/apache/pulsar/wiki/Pulsar-Improvement-Proposal-Template [1] https://github.com/apache/pulsar/wiki/Pulsar-Improvement-Proposal-%28PIP%29 So that we can review the details and improve these 2 documents (I see the last update for these two documents are 2021 and 2020) Thanks, Penghui On Thu, Aug 25, 2022 at 9:44 AM tison wrote: > Hi Penghui, > > Thanks for starting this discussion! +1 for holding PIPs in the codebase. > > I'd like to share some experience working on open-source projects holding > proposals as code. > > 1. Rust RFCs: codebase https://github.com/rust-lang/rfcs > 2. TiDB Proposals: > https://github.com/pingcap/tidb/tree/master/docs/design#readme > > Reflect on topics in this thread: > > > any reviews should happen with a PR first > > The experience should be similar to reviewing on Google Docs while keeping > all conversations on GitHub and thus synced on the mailing list. You can > preview can some examples: > > * https://github.com/pingcap/tidb/pull/23673 > * https://github.com/rust-lang/rfcs/pull/42 > > Furthermore, continuous follow-up like PIP-198 can discuss implementation > details and update the proposal content one PR by another. The downside is > that even trivial typo fixes should go through a PR. > > > Certainly, all the votes should happen on the mailing list. > > Yes. We should have an explicit [VOTE] and [RESULT][VOTE] thread on the > mailing list while keeping a reference within the proposal content. We can > port the current PIP issue template to a proposal template as Rust RFCs to > highlight it. > > > We can just add a link that points to the PIPs dir to the contribution > guide > or README. > > Rust RFCs hold in a separate repo so that they can use the PR number as the > proposal number to prevent duplicates. It's a bit overhead for Pulsar to > hold a separate repo. So I'd prefer to hold proposals in the same repo > under the dev/proposals folder and resolve the PR number duplicates issue > as described above. Although, it can still be a bit awkward. But at least > we can quickly notice duplicate numbers and resolve them. Using the PR > number as the proposal number keeps it unique, but the number will be > discontinuous. Especially if we're proposing in the main repo, the number > will grow quickly. > > > Should we track the status of PIPs as a project? > > In this direction, we can associate a proposal with a tracking issue and > manipulate the proposal status represented by their tracking issue. It > doesn't mean we keep the current issue discussion process. You can regard > it as an umbrella ticket for the proposal. See > https://github.com/pingcap/tidb/issues/26022 as an example. > > In this direction, we should always ensure that there're several eyes on > the project. Otherwise, we can easily end up with an out-of-sync project. > > To @Rajan @Penghui > > It seems the confusion comes from we have to merge one PR first to prevent > duplicate numbers. If it's a certain concern, see also my comment above > using the PR number as the proposal, or even host a separate proposal repo > as rust-lang/rfcs does and keep the number growing normally. I can provide > experience to maintain this sort of resource. > > > Best, > tison. >
[GitHub] [pulsar-test-infra] maxsxu commented on pull request #68: [improve][docbot] Move the preceding steps into the action
maxsxu commented on PR #68: URL: https://github.com/apache/pulsar-test-infra/pull/68#issuecomment-1226695489 /cc @tisonkun @Anonymitaet @lhotari -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-test-infra] maxsxu opened a new pull request, #68: [improve][docbot] Move the preceding steps into the action
maxsxu opened a new pull request, #68: URL: https://github.com/apache/pulsar-test-infra/pull/68 Fixes #67 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-test-infra] maxsxu opened a new issue, #67: Simplify invocation of docbot action
maxsxu opened a new issue, #67:
URL: https://github.com/apache/pulsar-test-infra/issues/67
### Motivation
The [Current invocation of
docbot](https://github.com/apache/pulsar/blob/46b1239a23fd2a250b009df1e9b4a65341d299a7/.github/workflows/ci-documentbot.yml):
```
jobs:
label:
steps:
- name: Checkout action
uses: actions/checkout@v3
with:
repository: apache/pulsar-test-infra
ref: master
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: 1.18
- name: Labeling
uses: ./docbot
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
LABEL_WATCH_LIST: 'doc,doc-required,doc-not-needed,doc-complete'
LABEL_MISSING: 'doc-label-missing'
```
These steps before Labeling should be organized into the action itself, the
desired invocation should be illustrated below:
```
jobs:
label:
steps:
- name: Labeling
uses: pulsar/pulsar-test-infra/docbot@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
LABEL_WATCH_LIST: 'doc,doc-required,doc-not-needed,doc-complete'
LABEL_MISSING: 'doc-label-missing'
```
inspired from
https://github.com/apache/pulsar-test-infra/issues/58#issuecomment-1225889058
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [DISCUSS] Enable message deduplication for replicator by default
+1 Thanks, Baodi Shi > On Aug 24, 2022, at 20:1312, Qiang Huang wrote: > > +1 > > Zike Yang 于2022年8月22日周一 15:32写道： > >> +1 >> >> Thanks, >> Zike Yang >> >> On Mon, Aug 22, 2022 at 3:16 PM mattison chao >> wrote: >>> >>> +1 >>> >>> Best, >>> Mattison >>> >>> On Fri, 19 Aug 2022 at 01:40, Enrico Olivelli >> wrote: >>> I agree Enrico Il Gio 18 Ago 2022, 18:23 PengHui Li ha scritto: > Hi all, > > When I tried to fix a problem related to replicator > https://github.com/apache/pulsar/pull/17154 > It surprised me that the message deduplication will not work by >> default > with the replicator. > I always thought it was enabled for replicators by default. Details >> to see > [0]. > > I think we should enable the deduplication for the replicator. >> Otherwise, > we will see duplicated > messages on the remote cluster. And the producer of the replicator >> always > has a fixed producer > name, this will make the message deduplication work properly. > > The test introduced in https://github.com/apache/pulsar/pull/17154 >> will > check the message > replication ordering. Without the message deduplication enabled, the >> test > is flaky with received > duplicated messages. After enabling, everything is fine. > > Best, > Penghui > > [0] >> https://github.com/apache/pulsar/pull/17154#discussion_r948736894 > >> > > > -- > BR, > Qiang Huang
Re: [DISCUSS] Move PIPs to the codebase?
Hi Penghui, Thanks for starting this discussion! +1 for holding PIPs in the codebase. I'd like to share some experience working on open-source projects holding proposals as code. 1. Rust RFCs: codebase https://github.com/rust-lang/rfcs 2. TiDB Proposals: https://github.com/pingcap/tidb/tree/master/docs/design#readme Reflect on topics in this thread: > any reviews should happen with a PR first The experience should be similar to reviewing on Google Docs while keeping all conversations on GitHub and thus synced on the mailing list. You can preview can some examples: * https://github.com/pingcap/tidb/pull/23673 * https://github.com/rust-lang/rfcs/pull/42 Furthermore, continuous follow-up like PIP-198 can discuss implementation details and update the proposal content one PR by another. The downside is that even trivial typo fixes should go through a PR. > Certainly, all the votes should happen on the mailing list. Yes. We should have an explicit [VOTE] and [RESULT][VOTE] thread on the mailing list while keeping a reference within the proposal content. We can port the current PIP issue template to a proposal template as Rust RFCs to highlight it. > We can just add a link that points to the PIPs dir to the contribution guide or README. Rust RFCs hold in a separate repo so that they can use the PR number as the proposal number to prevent duplicates. It's a bit overhead for Pulsar to hold a separate repo. So I'd prefer to hold proposals in the same repo under the dev/proposals folder and resolve the PR number duplicates issue as described above. Although, it can still be a bit awkward. But at least we can quickly notice duplicate numbers and resolve them. Using the PR number as the proposal number keeps it unique, but the number will be discontinuous. Especially if we're proposing in the main repo, the number will grow quickly. > Should we track the status of PIPs as a project? In this direction, we can associate a proposal with a tracking issue and manipulate the proposal status represented by their tracking issue. It doesn't mean we keep the current issue discussion process. You can regard it as an umbrella ticket for the proposal. See https://github.com/pingcap/tidb/issues/26022 as an example. In this direction, we should always ensure that there're several eyes on the project. Otherwise, we can easily end up with an out-of-sync project. To @Rajan @Penghui It seems the confusion comes from we have to merge one PR first to prevent duplicate numbers. If it's a certain concern, see also my comment above using the PR number as the proposal, or even host a separate proposal repo as rust-lang/rfcs does and keep the number growing normally. I can provide experience to maintain this sort of resource. Best, tison.
[jira] [Closed] (PULSAR-22) flink消费pulsar时出现错误，但是仍可消费到数据
[
https://issues.apache.org/jira/browse/PULSAR-22?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
海洋饼干 closed PULSAR-22.
--
Resolution: Cannot Reproduce
> flink消费pulsar时出现错误，但是仍可消费到数据
>
>
> Key: PULSAR-22
> URL: https://issues.apache.org/jira/browse/PULSAR-22
> Project: Pulsar
> Issue Type: Bug
> Environment: kafka(json) -> flink(Stream) -> custom
> Serializer -> kop -> pulsar -> flink(stream) keyshared consum(问题所在区域)
>Reporter: 海洋饼干
>Priority: Major
>
> [Source Data Fetcher for Source: kafkaSource (4/4)#0] ERROR
> org.apache.flink.connector.pulsar.source.reader.split.PulsarPartitionSplitReaderBase
> - Error in polling message from pulsar consumer.
> java.util.concurrent.ExecutionException:
> org.apache.pulsar.client.api.PulsarClientException$TransactionConflictException:
>
> \{"errorMsg":"org.apache.pulsar.transaction.common.exception.TransactionConflictException:
>
> [persistent://public/default/HB0002_rdwBaseCtr_aaamore5005-test-pulsar-partition-0][flink-source]
> Transaction:(1,12) try to ack batch message:810:24 in pending ack
> status.","reqId":4357026755663389273,
> "remote":"tk01-bd-test-pulsar-7-139/192.168.7.139:6650",
> "local":"/192.168.34.54:9756"}
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.connector.pulsar.source.reader.split.PulsarUnorderedPartitionSplitReader.pollMessage(PulsarUnorderedPartitionSplitReader.java:98)
> at
> org.apache.flink.connector.pulsar.source.reader.split.PulsarPartitionSplitReaderBase.fetch(PulsarPartitionSplitReaderBase.java:110)
> at
> org.apache.flink.connector.pulsar.source.reader.split.PulsarUnorderedPartitionSplitReader.fetch(PulsarUnorderedPartitionSplitReader.java:56)
> at
> org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58)
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:105)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by:
> org.apache.pulsar.client.api.PulsarClientException$TransactionConflictException:
>
> \{"errorMsg":"org.apache.pulsar.transaction.common.exception.TransactionConflictException:
>
> [persistent://public/default/HB0002_rdwBaseCtr_aaamore5005-test-pulsar-partition-0][flink-source]
> Transaction:(1,12) try to ack batch message:810:24 in pending ack
> status.","reqId":4357026755663389273,
> "remote":"tk01-bd-test-pulsar-7-139/192.168.7.139:6650",
> "local":"/192.168.34.54:9756"}
> at
> org.apache.pulsar.client.impl.ClientCnx.getPulsarClientException(ClientCnx.java:1177)
> at
> org.apache.pulsar.client.impl.ClientCnx.handleAckResponse(ClientCnx.java:431)
> at
> org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:150)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
> at
> org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.pu
Re: [DISCUSS] Move PIPs to the codebase?
+1 from me. (I've always been a supporter of having PIPs in codebase) Le mer. 24 août 2022 à 18:04, Michael Marshall a écrit : > I support moving PIPs to the code base. > > When we discussed this option during last year's redesign of the PIP > process, an objection was that "merging" a PIP could be construed as > acceptance of the PIP. As long as we are clear that we merge all > proposed PIP PRs because it is good to have the historical record, I > think we'll be able to prevent confusion. > > > Also we have old PIPs that have been abandoned or that never completed > VOTEs > > This raises the question of how to handle historical PIPs. We could do > some work to move PIPs out of the wiki and out of issues into the code > base. The wiki is just another git repo, so any committer should be > able to help migrate PIPs in the wiki to the apache/pulsar git repo. > This might also be an opportunity to double check the status on old > PIPs. > > Thanks, > Michael > > > On Wed, Aug 24, 2022 at 10:23 AM 丛搏 wrote: > > > > Hi all, > > > > I also think it's a good idea. Not only can it prevent the duplication > of PIP > > numbers, but most importantly, it is easier to annotate, in other words, > > It can review the PIP just like the review code, it can make the problem > > more focused, and it is easier for the viewer to annotate at any time. > > It didn't make a huge difference, just from ISSUE to PR. We have no > > reason not to change it, having it will greatly improve the review > > efficiency of PIP. > > > > Thanks, > > Bo > > > > PengHui Li 于2022年8月24日周三 10:01写道： > > > > > > Hi Rajan, > > > > > > > I didn't understand this part. You want one to create a PR before > > > submitting a proposal? That's clearly not a good idea because if the > PIP > > > approach will change then the entire development effort will be wasted > and > > > that's the whole purpose of PIP. I guess creating PIP into an issue and > > > discussing the issue is definitely working and it's an easier way to > > > discuss quickly rather than discussing over email threads. > > > > > > Sorry, I mean create a PR to add the proposal, not the implementation. > > > Before adding the proposal, we should discuss the motivation on the > mailing > > > list first. > > > > > > > Let's not change this practice without good discussion and agreement > from > > > the community. > > > > > > I don't think this essentially changes this part, it just provides a > way > > > for reviewers to > > > review the details of the proposal more efficiently. For a quick > > > discussion, this will not change > > > it. Let me try to clarify the procedure after we applied this idea > > > > > > > > >1. Send the email first to discuss the motivation of your proposal > > >2. Discuss the motivation under the mailing list and request a PIP > > >number if there are no objections > > >3. Create a PR to add the detailed proposal > > >4. *Send out the VOTE email with the PR link* > > >5. *Review the proposal(The PR) on Github and vote under the mailing > > >list.* > > >6. Merge the PR of the proposal after the proposal get 3 (+1) > bindings > > > > > > The only difference is step 4 and step 5. Currently, we share the issue > > > link to ask for a review. > > > > > > Thanks, > > > Penghui > > > > > > On Wed, Aug 24, 2022 at 1:22 AM Rajan Dhabalia > > > wrote: > > > > > > > Hi, > > > > > > > > >>> I think we can move all the PIPs to the codebase and the new > proposal > > > > and proposal without any reviews should happen with a PR first. So > that we > > > > can review and comment easily. > > > > > > > > I didn't understand this part. You want one to create a PR before > > > > submitting a proposal? That's clearly not a good idea because if the > PIP > > > > approach will change then the entire development effort will be > wasted and > > > > that's the whole purpose of PIP. I guess creating PIP into an issue > and > > > > discussing the issue is definitely working and it's an easier way to > > > > discuss quickly rather than discussing over email threads. > > > > > > > > Let's not change this practice without good discussion and agreement > from > > > > the community. > > > > > > > > Thanks, > > > > Rajan > > > > > > > > On Thu, Aug 18, 2022 at 8:27 AM PengHui Li > wrote: > > > > > > > > > Hi all, > > > > > > > > > > Currently, the new proposal will be added to the issue list and > then > > > > shared > > > > > link in the email > > > > > to request the proposal review. It's really hard to review a long > > > > proposal > > > > > if you want to comment > > > > > in detail. > > > > > > > > > > Here is an example: > > > > > > https://github.com/apache/pulsar/issues/16763#issuecomment-1219606491 > > > > > This seems very unintuitive. > > > > > > > > > > I think we can move all the PIPs to the codebase and the new > proposal and > > > > > proposal without > > > > > any reviews should happen with a PR first. So that we can review > and > > > > > comment eas
[GitHub] [pulsar-site] tisonkun commented on pull request #170: fix version replace
tisonkun commented on PR #170: URL: https://github.com/apache/pulsar-site/pull/170#issuecomment-1226029231 Cool! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun opened a new issue, #175: Remove all content before sync
tisonkun opened a new issue, #175: URL: https://github.com/apache/pulsar-site/issues/175 @urfreespace @Anonymitaet Otherwise, we can suffer from some files deleted from upstream but still remain in this repo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun closed pull request #174: Sync with main repo rename develop-binary-protocol
tisonkun closed pull request #174: Sync with main repo rename develop-binary-protocol URL: https://github.com/apache/pulsar-site/pull/174 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun commented on pull request #174: Sync with main repo rename develop-binary-protocol
tisonkun commented on PR #174:
URL: https://github.com/apache/pulsar-site/pull/174#issuecomment-1225988594
Closed. It seems I can use the following snippet to redirect. Should submit
a pr on the main repo:
```mdx
---
id: develop-binary-protocol
title: Pulsar binary protocol specification2
sidebar_label: "Binary protocol2"
---
import {Redirect} from '@docusaurus/router';
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [DISCUSS] Catch up reads with many subscriptions - saving reads from BK
+1 I support merging the temporary solution. > Just one point, please make sure the change will not introduce too much > heap memory overhead. If we see that there is extra pressure, I wonder if we can dynamically determine when to deduplicate requests. A broker only needs this feature when a topic has multiple subscriptions. I haven't double checked if this information is easily accessible to the cache. One of the interesting benefits of this optimization is that it will decrease direct memory pressure. When we do not deduplicate reads, the same entry will be duplicated in the bookie's netty direct memory pool. With this design, an entry will be materialized once with many different ByteBuffer's objects pointing to it. Thanks, Michael On Wed, Aug 24, 2022 at 11:00 AM PengHui Li wrote: > > +1 for we can have a short-term solution first. > > Just one point, please make sure the change will not introduce too much > heap memory overhead. > > Thanks, > Penghui > > On Wed, Aug 24, 2022 at 11:54 PM Enrico Olivelli > wrote: > > > Hello folks, > > These days I am working on a use case in which I have many > > subscriptions on the same topic (~100). > > If you get to a situation with a big backlog, the consumers are no > > more able to leverage the broker cache then the broker starts to read > > from BookKeeper. > > > > In 2.11 we have a first mechanism of catch-up reads cache for > > backlogged consumers (when you have a few backlogged cursors we start > > putting in the cache the entries that you read). > > > > That is not enough to deal with my case because if you have many > > consumers (one per partition) that connect all together, for instance > > after a major problem in the application, the cache is empty and the > > broker starts to overwhelm the bookies with reads. > > > > As a short term solution I have created this patch > > https://github.com/apache/pulsar/pull/17241 > > > > The patch basically saves reads from BK by preventing concurrent reads > > for the same entries (actually "ranges of entries"). > > If the broker starts a read for a range and then comes another read > > for the same range before the read completes, then we skip sending the > > new read and we use the result of the BK request. > > > > This simple patch is really effective, in some tests it reduces a lot > > (from 100K reads/s to 250 reads/s) the concurrent reads to BK. > > You may say that it is unlikely that the broker sends concurrent > > requests for exactly the same range of entries but empirically I have > > observed that it happens quite often, at least in the scenario I drew > > above. > > > > My proposal is to commit my patch as a short term solution. > > I have already talked with PengHui and a better solution would be to > > introduce explicitly a "Catch Up reads cache", but that is a mid term > > implementation. > > > > WDYT ? > > > > > > Enrico > >
[GitHub] [pulsar-site] tisonkun commented on pull request #174: Sync with main repo rename develop-binary-protocol
tisonkun commented on PR #174: URL: https://github.com/apache/pulsar-site/pull/174#issuecomment-1225936104 Or we can change main repo files `develop-binary-protocol` to redirect to `developing-binary-protocol ` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun opened a new pull request, #174: Sync with main repo rename develop-binary-protocol
tisonkun opened a new pull request, #174: URL: https://github.com/apache/pulsar-site/pull/174 This closes #172. cc @urfreespace perhaps we should set up a redirect from develop-binary-protocol to developing-binary-protocol. Although, I don't know how to achieve it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [DISCUSS] Move PIPs to the codebase?
I support moving PIPs to the code base. When we discussed this option during last year's redesign of the PIP process, an objection was that "merging" a PIP could be construed as acceptance of the PIP. As long as we are clear that we merge all proposed PIP PRs because it is good to have the historical record, I think we'll be able to prevent confusion. > Also we have old PIPs that have been abandoned or that never completed VOTEs This raises the question of how to handle historical PIPs. We could do some work to move PIPs out of the wiki and out of issues into the code base. The wiki is just another git repo, so any committer should be able to help migrate PIPs in the wiki to the apache/pulsar git repo. This might also be an opportunity to double check the status on old PIPs. Thanks, Michael On Wed, Aug 24, 2022 at 10:23 AM 丛搏 wrote: > > Hi all, > > I also think it's a good idea. Not only can it prevent the duplication of PIP > numbers, but most importantly, it is easier to annotate, in other words, > It can review the PIP just like the review code, it can make the problem > more focused, and it is easier for the viewer to annotate at any time. > It didn't make a huge difference, just from ISSUE to PR. We have no > reason not to change it, having it will greatly improve the review > efficiency of PIP. > > Thanks, > Bo > > PengHui Li 于2022年8月24日周三 10:01写道： > > > > Hi Rajan, > > > > > I didn't understand this part. You want one to create a PR before > > submitting a proposal? That's clearly not a good idea because if the PIP > > approach will change then the entire development effort will be wasted and > > that's the whole purpose of PIP. I guess creating PIP into an issue and > > discussing the issue is definitely working and it's an easier way to > > discuss quickly rather than discussing over email threads. > > > > Sorry, I mean create a PR to add the proposal, not the implementation. > > Before adding the proposal, we should discuss the motivation on the mailing > > list first. > > > > > Let's not change this practice without good discussion and agreement from > > the community. > > > > I don't think this essentially changes this part, it just provides a way > > for reviewers to > > review the details of the proposal more efficiently. For a quick > > discussion, this will not change > > it. Let me try to clarify the procedure after we applied this idea > > > > > >1. Send the email first to discuss the motivation of your proposal > >2. Discuss the motivation under the mailing list and request a PIP > >number if there are no objections > >3. Create a PR to add the detailed proposal > >4. *Send out the VOTE email with the PR link* > >5. *Review the proposal(The PR) on Github and vote under the mailing > >list.* > >6. Merge the PR of the proposal after the proposal get 3 (+1) bindings > > > > The only difference is step 4 and step 5. Currently, we share the issue > > link to ask for a review. > > > > Thanks, > > Penghui > > > > On Wed, Aug 24, 2022 at 1:22 AM Rajan Dhabalia > > wrote: > > > > > Hi, > > > > > > >>> I think we can move all the PIPs to the codebase and the new proposal > > > and proposal without any reviews should happen with a PR first. So that we > > > can review and comment easily. > > > > > > I didn't understand this part. You want one to create a PR before > > > submitting a proposal? That's clearly not a good idea because if the PIP > > > approach will change then the entire development effort will be wasted and > > > that's the whole purpose of PIP. I guess creating PIP into an issue and > > > discussing the issue is definitely working and it's an easier way to > > > discuss quickly rather than discussing over email threads. > > > > > > Let's not change this practice without good discussion and agreement from > > > the community. > > > > > > Thanks, > > > Rajan > > > > > > On Thu, Aug 18, 2022 at 8:27 AM PengHui Li wrote: > > > > > > > Hi all, > > > > > > > > Currently, the new proposal will be added to the issue list and then > > > shared > > > > link in the email > > > > to request the proposal review. It's really hard to review a long > > > proposal > > > > if you want to comment > > > > in detail. > > > > > > > > Here is an example: > > > > https://github.com/apache/pulsar/issues/16763#issuecomment-1219606491 > > > > This seems very unintuitive. > > > > > > > > I think we can move all the PIPs to the codebase and the new proposal > > > > and > > > > proposal without > > > > any reviews should happen with a PR first. So that we can review and > > > > comment easily. > > > > Certainly, all the votes should happen on the mailing list. And we can > > > also > > > > discuss the > > > > proposal on the mailing list. > > > > > > > > Following this way, we don't need to sync the PIPs from the issue to the > > > > wiki page. > > > > We can just add a link that points to the PIPs dir to the contribution > > > > guide or README. > > > > > > > > We have
Re: [DISCUSS] Catch up reads with many subscriptions - saving reads from BK
+1 for we can have a short-term solution first. Just one point, please make sure the change will not introduce too much heap memory overhead. Thanks, Penghui On Wed, Aug 24, 2022 at 11:54 PM Enrico Olivelli wrote: > Hello folks, > These days I am working on a use case in which I have many > subscriptions on the same topic (~100). > If you get to a situation with a big backlog, the consumers are no > more able to leverage the broker cache then the broker starts to read > from BookKeeper. > > In 2.11 we have a first mechanism of catch-up reads cache for > backlogged consumers (when you have a few backlogged cursors we start > putting in the cache the entries that you read). > > That is not enough to deal with my case because if you have many > consumers (one per partition) that connect all together, for instance > after a major problem in the application, the cache is empty and the > broker starts to overwhelm the bookies with reads. > > As a short term solution I have created this patch > https://github.com/apache/pulsar/pull/17241 > > The patch basically saves reads from BK by preventing concurrent reads > for the same entries (actually "ranges of entries"). > If the broker starts a read for a range and then comes another read > for the same range before the read completes, then we skip sending the > new read and we use the result of the BK request. > > This simple patch is really effective, in some tests it reduces a lot > (from 100K reads/s to 250 reads/s) the concurrent reads to BK. > You may say that it is unlikely that the broker sends concurrent > requests for exactly the same range of entries but empirically I have > observed that it happens quite often, at least in the scenario I drew > above. > > My proposal is to commit my patch as a short term solution. > I have already talked with PengHui and a better solution would be to > introduce explicitly a "Catch Up reads cache", but that is a mid term > implementation. > > WDYT ? > > > Enrico >
[DISCUSS] Catch up reads with many subscriptions - saving reads from BK
Hello folks, These days I am working on a use case in which I have many subscriptions on the same topic (~100). If you get to a situation with a big backlog, the consumers are no more able to leverage the broker cache then the broker starts to read from BookKeeper. In 2.11 we have a first mechanism of catch-up reads cache for backlogged consumers (when you have a few backlogged cursors we start putting in the cache the entries that you read). That is not enough to deal with my case because if you have many consumers (one per partition) that connect all together, for instance after a major problem in the application, the cache is empty and the broker starts to overwhelm the bookies with reads. As a short term solution I have created this patch https://github.com/apache/pulsar/pull/17241 The patch basically saves reads from BK by preventing concurrent reads for the same entries (actually "ranges of entries"). If the broker starts a read for a range and then comes another read for the same range before the read completes, then we skip sending the new read and we use the result of the BK request. This simple patch is really effective, in some tests it reduces a lot (from 100K reads/s to 250 reads/s) the concurrent reads to BK. You may say that it is unlikely that the broker sends concurrent requests for exactly the same range of entries but empirically I have observed that it happens quite often, at least in the scenario I drew above. My proposal is to commit my patch as a short term solution. I have already talked with PengHui and a better solution would be to introduce explicitly a "Catch Up reads cache", but that is a mid term implementation. WDYT ? Enrico
[GitHub] [pulsar-manager] jakiuncle opened a new issue, #483: I want use casdoor to login,when i change code, happen some problem
jakiuncle opened a new issue, #483: URL: https://github.com/apache/pulsar-manager/issues/483  why it jump to  How can i aviod it. Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-test-infra] maxsxu commented on issue #58: Use Docker container action for docbot
maxsxu commented on issue #58: URL: https://github.com/apache/pulsar-test-infra/issues/58#issuecomment-1225889058 > Although, we can still try to let the docbot action to be used as barely `apache/pulsar-test-infra/docbot@main`. Yes, one thing we can be sure of now is to simplify the current ci-documentbot.yaml workflow file like this https://github.com/open-github/pulsar/blob/master/.github/workflows/ci-documentbot.yml. which is both applicable for whether composite or docker. This will send PR in both pulsar-test-infra and pulsar repo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-test-infra] tisonkun closed issue #58: Use Docker container action for docbot
tisonkun closed issue #58: Use Docker container action for docbot URL: https://github.com/apache/pulsar-test-infra/issues/58 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-test-infra] tisonkun commented on issue #58: Use Docker container action for docbot
tisonkun commented on issue #58: URL: https://github.com/apache/pulsar-test-infra/issues/58#issuecomment-1225879730 Cool! Then I'll close this issue as won't do. Although, we can still try to let the docbot action to be used as barely `apache/pulsar-test-infra/docbot@main`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [DISCUSS] Move PIPs to the codebase?
Hi all, I also think it's a good idea. Not only can it prevent the duplication of PIP numbers, but most importantly, it is easier to annotate, in other words, It can review the PIP just like the review code, it can make the problem more focused, and it is easier for the viewer to annotate at any time. It didn't make a huge difference, just from ISSUE to PR. We have no reason not to change it, having it will greatly improve the review efficiency of PIP. Thanks, Bo PengHui Li 于2022年8月24日周三 10:01写道： > > Hi Rajan, > > > I didn't understand this part. You want one to create a PR before > submitting a proposal? That's clearly not a good idea because if the PIP > approach will change then the entire development effort will be wasted and > that's the whole purpose of PIP. I guess creating PIP into an issue and > discussing the issue is definitely working and it's an easier way to > discuss quickly rather than discussing over email threads. > > Sorry, I mean create a PR to add the proposal, not the implementation. > Before adding the proposal, we should discuss the motivation on the mailing > list first. > > > Let's not change this practice without good discussion and agreement from > the community. > > I don't think this essentially changes this part, it just provides a way > for reviewers to > review the details of the proposal more efficiently. For a quick > discussion, this will not change > it. Let me try to clarify the procedure after we applied this idea > > >1. Send the email first to discuss the motivation of your proposal >2. Discuss the motivation under the mailing list and request a PIP >number if there are no objections >3. Create a PR to add the detailed proposal >4. *Send out the VOTE email with the PR link* >5. *Review the proposal(The PR) on Github and vote under the mailing >list.* >6. Merge the PR of the proposal after the proposal get 3 (+1) bindings > > The only difference is step 4 and step 5. Currently, we share the issue > link to ask for a review. > > Thanks, > Penghui > > On Wed, Aug 24, 2022 at 1:22 AM Rajan Dhabalia > wrote: > > > Hi, > > > > >>> I think we can move all the PIPs to the codebase and the new proposal > > and proposal without any reviews should happen with a PR first. So that we > > can review and comment easily. > > > > I didn't understand this part. You want one to create a PR before > > submitting a proposal? That's clearly not a good idea because if the PIP > > approach will change then the entire development effort will be wasted and > > that's the whole purpose of PIP. I guess creating PIP into an issue and > > discussing the issue is definitely working and it's an easier way to > > discuss quickly rather than discussing over email threads. > > > > Let's not change this practice without good discussion and agreement from > > the community. > > > > Thanks, > > Rajan > > > > On Thu, Aug 18, 2022 at 8:27 AM PengHui Li wrote: > > > > > Hi all, > > > > > > Currently, the new proposal will be added to the issue list and then > > shared > > > link in the email > > > to request the proposal review. It's really hard to review a long > > proposal > > > if you want to comment > > > in detail. > > > > > > Here is an example: > > > https://github.com/apache/pulsar/issues/16763#issuecomment-1219606491 > > > This seems very unintuitive. > > > > > > I think we can move all the PIPs to the codebase and the new proposal and > > > proposal without > > > any reviews should happen with a PR first. So that we can review and > > > comment easily. > > > Certainly, all the votes should happen on the mailing list. And we can > > also > > > discuss the > > > proposal on the mailing list. > > > > > > Following this way, we don't need to sync the PIPs from the issue to the > > > wiki page. > > > We can just add a link that points to the PIPs dir to the contribution > > > guide or README. > > > > > > We have another pain point about the duplicated PIP number. We can > > maintain > > > a file, a list of > > > all the proposal contains the approved, in-review, drafting. Before > > > creating a proposal, we should > > > have a discussion first on the mailing list, just get feedback on the > > > motivation. If there are no objections, > > > the proposal owner can add a line to the file with the PIP number > > through a > > > PR, like PIP-123: xxx (Under Discussion). > > > So that we can prevent the duplicated PIP number(which will conflict if > > > someone merged first). > > > After the PR is merged, we can send out a new PR to add the proposal. > > > > > > Thanks, > > > Penghui > > > > >
[GitHub] [pulsar-test-infra] maxsxu commented on issue #58: Use Docker container action for docbot
maxsxu commented on issue #58: URL: https://github.com/apache/pulsar-test-infra/issues/58#issuecomment-1225869732 @tisonkun You have raised a very valuable question. I do curious about the real performance between composite action and docker action. So I did a quick test and the results show that composite action takes only about half the execution time of docker action. - [Comparison](https://github.com/open-github/pulsar/actions/workflows/ci-documentbot.yml) as shown below: > https://user-images.githubusercontent.com/15963141/186450802-7b95a180-6ee5-4f19-8653-697846bbc1de.png";> - [Details of composite action](https://github.com/open-github/pulsar/runs/7995320623?check_suite_focus=true): > https://user-images.githubusercontent.com/15963141/186451495-04591dfd-e813-4896-9465-de000309847b.png";> - [Details of docker action:](https://github.com/open-github/pulsar/runs/7997522614?check_suite_focus=true) > https://user-images.githubusercontent.com/15963141/186451851-c94ea119-7076-41b7-a129-003a8dc0468d.png";> For short operations like labeling, we need to run our action quickly. So execution speed is the primary requirement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [VOTE] PIP-193 : Sink preprocessing Function
Thanks to you all, Closing this vote with 8 +1s (4 bindings and 4 non-bindings) and no -1s. We'll adjust the naming during the implementation. Thanks, Christophe Le mar. 23 août 2022 à 07:04, Michael Marshall a écrit : > +1 (binding) > > I have some thoughts on naming, too. > > My preference is to call it a “mapper function”, though I think > “decorator function” is a good name too. My concern with the > preprocessing (and the correlated future postprocessing) name is that > it's not clear to me what the "processing" step is. Converting a > message to another message seems like the processing step for the > source/sink. > > Either way, these are minor details. The feature looks good to me. > > Thanks, > Michael > > > > > On Tue, Aug 16, 2022 at 12:55 PM Lari Hotari wrote: > > > > +1 (binding) > > > > Good proposal. My only comment is about the naming. Instead of calling > it a "preprocessing function" or "extra function", I'd propose calling it > "decorator function". > > > > -Lari > > > > On 2022/07/28 10:39:35 Christophe Bornet wrote: > > > Hi, Pulsar community, > > > > > > I'd like to start a vote on PIP-193 : Sink preprocessing Function > > > > > > You can find the proposal at > https://github.com/apache/pulsar/issues/16739 and > > > the discussion thread at > > > https://lists.apache.org/thread/qn59jwn47w9ngxpkvq3kswbl1y882jth. > > > > > > The vote will stay open for at least 48 hours. > > > > > > Best regards. > > > > > > Christophe Bornet > > > >
Re: [VOTE] [PIP-201] Extensions mechanism for Pulsar Admin CLI tools
+1 (binding) Best, Penghui On Wed, Aug 24, 2022 at 9:39 PM Dave Fisher wrote: > +1 (binding) > > Best, > Dave > > Sent from my iPhone > > > On Aug 24, 2022, at 12:43 AM, Yunze Xu > wrote: > > > > +1 (non binding) > > > > Thanks, > > Yunze > > > > > > > > > >> 2022年8月24日 15:38，Nicolò Boschi 写道： > >> > >> +1 (non binding) > >> Nicolò Boschi > >> > >> > >>> Il giorno mer 24 ago 2022 alle ore 09:11 Enrico Olivelli < > >>> eolive...@gmail.com> ha scritto: > >>> > >>> Hello, > >>> this is the official thread VOTE for PIP-201 Extensions mechanism for > >>> Pulsar Admin CLI tools > >>> > >>> This is the PIP link https://github.com/apache/pulsar/issues/17155 > >>> This is the discussion: > >>> https://lists.apache.org/thread/287ft8twc11cp4s1y4qkcx5nmh451cyo > >>> > >>> I am still working on the PR, that is the subject of the VOTE. > >>> > >>> Best regards > >>> Enrico Olivelli > >>> > > > >
Re: [VOTE] [PIP-201] Extensions mechanism for Pulsar Admin CLI tools
+1 (binding) Best, Dave Sent from my iPhone > On Aug 24, 2022, at 12:43 AM, Yunze Xu wrote: > > +1 (non binding) > > Thanks, > Yunze > > > > >> 2022年8月24日 15:38，Nicolò Boschi 写道： >> >> +1 (non binding) >> Nicolò Boschi >> >> >>> Il giorno mer 24 ago 2022 alle ore 09:11 Enrico Olivelli < >>> eolive...@gmail.com> ha scritto: >>> >>> Hello, >>> this is the official thread VOTE for PIP-201 Extensions mechanism for >>> Pulsar Admin CLI tools >>> >>> This is the PIP link https://github.com/apache/pulsar/issues/17155 >>> This is the discussion: >>> https://lists.apache.org/thread/287ft8twc11cp4s1y4qkcx5nmh451cyo >>> >>> I am still working on the PR, that is the subject of the VOTE. >>> >>> Best regards >>> Enrico Olivelli >>> >
[GitHub] [pulsar] abcMH1966 added a comment to the discussion: I encountered a port error when using pulsar to integrate etcd
GitHub user abcMH1966 added a comment to the discussion: I encountered a port error when using pulsar to integrate etcd It doesn't work. Occur the same error!  My JDK version is 18.0.2.1 GitHub link: https://github.com/apache/pulsar/discussions/17240#discussioncomment-3465717 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
Re: [DISCUSS] Use Github project to track the multi clients development
I think that the main problem is that in our community we have plenty of Java developers and there are a few committers that review the patches on non-Java client. The same problem is also for other projects like the Helm Chart. Also, the Java client is used internally by the broker so it is usually more used and it needs more work That said I agree that with the proposal in this thread Enrico Il giorno mer 24 ago 2022 alle ore 14:32 Qiang Huang ha scritto: > > It makes sense to me. I'm thinking, if an issue is encountered in a Java > client, it is possible to be in other clients. If so, the Java client board > is also needed. > > mattison chao 于2022年8月22日周一 16:00写道： > > > +1 > > > > Best, > > Mattison > > > > On Mon, 22 Aug 2022 at 15:52, Baodi Shi > > wrote: > > > > > Nice idea! > > > > > > Thanks, > > > Baodi Shi > > > > > > > On Aug 22, 2022, at 12:4250, Zike Yang wrote: > > > > > > > > Hi, all > > > > > > > > Currently, many features or bug fixes of some clients like the C++ > > > > client and Python client have not caught up with the Java client. For > > > > better tracking of these features or bug fixes, I have created a > > > > Github project called `Apache Pulsar / Multi Clients`[0] under the > > > > pulsar main repo. It can be removed if we don't like it this way. > > > > > > > > There are six columns in this Github project: > > > > * Backlog: Low priority issues or issues(or cards) that have not been > > > > initially investigated. > > > > * Todo: Newly added issues and issues suitable for contributors to pick > > > first. > > > > * In progress: Work in progress issues or PRs. > > > > * Review in progress: The PRs or issues that are waiting for review. > > > > * Reviewer approved: The PRs or issues that are waiting to merge. > > > > * Done: Merged PRs and Solved issues. > > > > > > > > This not only helps to track the status of other clients' features and > > > > bug fixes but also helps contributors who want to participate in other > > > > language client feature development to pick the task. > > > > > > > > This Project is currently only for clients maintained by the main > > > > pulsar repository, such as C++ and Python clients. > > > > > > > > I have created a PR [1] to add C++ and Python client issues and PRs to > > > > the Multi Clients project. > > > > > > > > Please let me know what you think. Thanks! > > > > > > > > [0] https://github.com/apache/pulsar/projects/12 > > > > [1] https://github.com/apache/pulsar/pull/17201 > > > > > > > > Best, > > > > Zike Yang > > > > > > > > > > > -- > BR, > Qiang Huang
Re: [DISCUSS] Apache Pulsar 2.10.2 release
+1 Enrico Il giorno mer 24 ago 2022 alle ore 14:49 Qiang Huang ha scritto: > > +1 > > PengHui Li 于2022年8月23日周二 23:42写道： > > > +1 > > > > And, I have added the release/2.10.2 label, Lan Liang. > > > > Thanks, > > Penghui > > > > On Tue, Aug 23, 2022 at 11:02 PM Lan Liang > > wrote: > > > > > Hi,Haiting: > > > > > >Thanks for your work of release. Could you help to release PR-16309 > > > [0] to 2.10.2, It resolved pulsar can not working with etcd cluster. > > > > > >Some users are wondering about this part [1]. > > > > > > [0] https://github.com/apache/pulsar/pull/16309 > > > [1] https://github.com/apache/pulsar/discussions/17240 > > > > > > > > > > > > > > > Best Regards, > > > Lan Liang > > > Replied Message > > > | From | Haiting Jiang | > > > | Date | 8/23/2022 22:10 | > > > | To | | > > > | Subject | [DISCUSS] Apache Pulsar 2.10.2 release | > > > Hello, Pulsar community: > > > > > > I'd like to propose to release Apache Pulsar 2.10.2 > > > > > > Currently, we have 189 commits [0] labeled with `release/2.10.2`, > > > And 39 of them are not cherry-picked yet [1]. > > > > > > And there are 39 open PRs [2], I will follow them to make sure that > > > the important fixes could be contained in 2.10.2. > > > > > > If you have any important fixes or any questions, > > > please reply to this email, we will evaluate whether to > > > include it in 2.10.2 > > > > > > [0] > > > > > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.10.2+ > > > [1] > > > > > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.10.2+-label%3Acherry-picked%2Fbranch-2.10+ > > > [2] > > > > > https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.10.2+ > > > > > > Best Regards > > > Haiting Jiang > > > > > > > > -- > BR, > Qiang Huang
[GitHub] [pulsar] abcMH1966 added a comment to the discussion: I encountered a port error when using pulsar to integrate etcd
GitHub user abcMH1966 added a comment to the discussion: I encountered a port error when using pulsar to integrate etcd > [This PR](https://github.com/apache/pulsar/pull/16309) had resolve pulsar > working with etcd cluster and seems like have not any release version of > pulsar include it. and i have build a version of pulsar from master, could > you try it? `docker pull ghcr.io/pulsar-sigs/pulsar:2022-08-23` or `docker > pull lypgcs/pulsar:2022-08-23` **something goes wrong：** Error: LinkageError occurred while loading main class org.apache.pulsar.PulsarClusterMetadataSetup java.lang.UnsupportedClassVersionError: org/apache/pulsar/PulsarClusterMetadataSetup has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 Can you tell me what's your jdk version? GitHub link: https://github.com/apache/pulsar/discussions/17240#discussioncomment-3465565 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
Re: [DISCUSS] Deprecate KeyValue schema factory methods with Class parameters
Nice catch. LGTM. Yunze Xu 于2022年8月24日周三 16:29写道： > Hi folks, > > Recently I'm looking into the KeyValue schema and found **FOUR** > static methods in `Schema` to create a `KeyValueSchema` object: > > 1. KeyValue(Class key, Class value); > 2. KeyValue(Class key, Class value, SchemaType type); > 3. KeyValue(Schema key, Schema value); > 4. KeyValue(Schema key, Schema value, KeyValueEncodingType > keyValueEncodingType); > > IMO, having too many overloads is not user-friendly. I turned to the > official website and found overload 4 is used. The overload 3 is just > a simple version of overload 4 whose encoding type is `INLINE`, but > it's not clear. I opened a PR > https://github.com/apache/pulsar/pull/17256 to make it more clear. > > However, for overload 1 and 2, I can only find two references in unit > tests `testAllowNullAvroSchemaCreate` and > `testAllowNullJsonSchemaCreate`. From the very simple JavaDocs, it > looks like they are only available for JSON and AVRO schemas. > > Then I found the original PR to introduce these overloads: > https://github.com/apache/pulsar/pull/2885 > > Code has been changed a lot since that. It looks other codes are not > much related to these two overloads now. IMO, we should not encourage > users to use these two overloads, so I suggest marking them as > deprecated and might remove them in future releases. > > > Thanks, > Yunze > > > > > -- BR, Qiang Huang
Re: [VOTE] [PIP-201] Extensions mechanism for Pulsar Admin CLI tools
+1 (non binding) Yunze Xu 于2022年8月24日周三 15:44写道： > +1 (non binding) > > Thanks, > Yunze > > > > > > 2022年8月24日 15:38，Nicolò Boschi 写道： > > > > +1 (non binding) > > Nicolò Boschi > > > > > > Il giorno mer 24 ago 2022 alle ore 09:11 Enrico Olivelli < > > eolive...@gmail.com> ha scritto: > > > >> Hello, > >> this is the official thread VOTE for PIP-201 Extensions mechanism for > >> Pulsar Admin CLI tools > >> > >> This is the PIP link https://github.com/apache/pulsar/issues/17155 > >> This is the discussion: > >> https://lists.apache.org/thread/287ft8twc11cp4s1y4qkcx5nmh451cyo > >> > >> I am still working on the PR, that is the subject of the VOTE. > >> > >> Best regards > >> Enrico Olivelli > >> > > -- BR, Qiang Huang
Re: [VOTE] [PIP-169] Support split bundle by flow or qps
+1(non-binding) Enrico Olivelli 于2022年8月24日周三 15:04写道： > +1 (binding) > > > Enrico > > Il giorno mer 24 ago 2022 alle ore 07:13 Aloys Zhang > ha scritto: > > > > +1 > > > > Haiting Jiang 于2022年8月24日周三 12:35写道： > > > > > +1 (non) > > > > > > Thanks, > > > Haiting > > > > > > On Wed, Aug 24, 2022 at 10:19 AM guo jiwei > wrote: > > > > > > > +1 (binding) > > > > > > > > > > > > Regards > > > > Jiwei Guo (Tboy) > > > > > > > > > > > > On Wed, Aug 24, 2022 at 1:01 AM Heesung Sohn > > > > wrote: > > > > > > > > > +1 (non-binding) > > > > > > > > > > Heesung > > > > > > > > > > On Tue, Aug 23, 2022 at 10:00 AM Anon Hxy > wrote: > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > Thanks, > > > > > > Xiaoyu Hou > > > > > > > > > > > > lordcheng10 <1572139...@qq.com.invalid> 于2022年8月23日周二 15:10写道： > > > > > > > > > > > > > Hi Pulsar Community, I would like to start a VOTE on "Support > split > > > > > bundle > > > > > > > by flow or qps."(PIP-169) > > > > > > > Here is the design detail: > > > > > https://github.com/apache/pulsar/issues/16782 > > > > > > > > > > > > > > > > > > > > > and the discussion thread: > > > > > > > > https://lists.apache.org/thread/cshyt10fwcjjxs93g8yf0svgwcgnshmg > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > lordcheng10 > > > > > > > > > > > > > -- BR, Qiang Huang
Re: [DISCUSS] Minor breaking change to 2 method signatures in RangeCache class
Great work! It can tune the range cache better. Michael Marshall 于2022年8月24日周三 11:11写道： > Hi All, > > I'd like to add some more metrics to the broker entry cache. In order > to do so, I needed to update RangeCache#clear and > RangeCache#evictLEntriesBeforeTimestamp so they return Pair Long> instead of long. This is technically a breaking change, but > since it is an internal broker class, it would only break broker > extensions, if they used the return value for these methods. > > Please let me know if you think the changes will be a problem. > > Here is the PR: https://github.com/apache/pulsar/pull/17248. > > Thanks, > Michael > -- BR, Qiang Huang
Re: [DISCUSS] Apache Pulsar 2.10.2 release
+1 PengHui Li 于2022年8月23日周二 23:42写道： > +1 > > And, I have added the release/2.10.2 label, Lan Liang. > > Thanks, > Penghui > > On Tue, Aug 23, 2022 at 11:02 PM Lan Liang > wrote: > > > Hi,Haiting: > > > >Thanks for your work of release. Could you help to release PR-16309 > > [0] to 2.10.2, It resolved pulsar can not working with etcd cluster. > > > >Some users are wondering about this part [1]. > > > > [0] https://github.com/apache/pulsar/pull/16309 > > [1] https://github.com/apache/pulsar/discussions/17240 > > > > > > > > > > Best Regards, > > Lan Liang > > Replied Message > > | From | Haiting Jiang | > > | Date | 8/23/2022 22:10 | > > | To | | > > | Subject | [DISCUSS] Apache Pulsar 2.10.2 release | > > Hello, Pulsar community: > > > > I'd like to propose to release Apache Pulsar 2.10.2 > > > > Currently, we have 189 commits [0] labeled with `release/2.10.2`, > > And 39 of them are not cherry-picked yet [1]. > > > > And there are 39 open PRs [2], I will follow them to make sure that > > the important fixes could be contained in 2.10.2. > > > > If you have any important fixes or any questions, > > please reply to this email, we will evaluate whether to > > include it in 2.10.2 > > > > [0] > > > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.10.2+ > > [1] > > > https://github.com/apache/pulsar/pulls?q=is%3Amerged+is%3Apr+label%3Arelease%2F2.10.2+-label%3Acherry-picked%2Fbranch-2.10+ > > [2] > > > https://github.com/apache/pulsar/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F2.10.2+ > > > > Best Regards > > Haiting Jiang > > > -- BR, Qiang Huang
Re: [DISCUSS] Make `Pulsar CI / CI - Unit - Brokers - Flaky` test group to be required
Failure flaky tests reduce the passion of contributors. I suggest making stable tests as required. I have 3 points. 1. Find the existing failure flaky tests and fix them. I believe that there is already a project to record them. 2. Move the failure flaky tests out of the flaky test group. 3. Add them into a new group and fix. And make it required if they are stable. PengHui Li 于2022年8月22日周一 17:27写道： > I agree with moving the tests out of the flaky test group. > I just checking some new PRs > > https://github.com/apache/pulsar/pull/17195 > https://github.com/apache/pulsar/pull/17201 > https://github.com/apache/pulsar/pull/17193 > https://github.com/apache/pulsar/pull/17204 > > The `Pulsar CI / CI - Unit - Brokers - Flaky` looks more stable than other > groups such as > > `Pulsar CI / CI - System - Pulsar Connectors - Thread` > `Pulsar CI / CI - Unit - Brokers - Broker Group 2` > `Pulsar CI / CI - Unit - Brokers - Broker Group 1` > > It looks like we can change the test group for now and make it required. > To a new test group, or move them to `Pulsar CI / CI - Unit - Brokers - * ` > > Thanks, > Penghui > > On Mon, Aug 22, 2022 at 4:33 PM tison wrote: > > > While agree to require more tests to pass, here are my two coins: > > > > 1. This group is named "flaky tests" so I regard it as flaky tests > > literally. NOT require these tests to pass could be by design. Besides, > > IIRC some developers keep investigating tests in the flaky test group, > try > > to make them stable, and move out of the flaky test group. This seems the > > desired approach to resolve flaky tests. > > > > 2. Instead of barely "require broker-flaky" test group, do you have a > list > > of tests that fail frequently? Otherwise, we just go back to the > situation > > where we want to exclude it from the required status - it's quite > unstable. > > > > Best, > > tison. > > > > > > mattison chao 于2022年8月22日周一 15:56写道： > > > > > Hi all > > > > > > Recently, some tests in the `broker-flaky` test group always failed, > but > > > since it doesn't block CI, no one cared for a long time. > > > > > > This behaviour causes some test scenarios to go unchecked and risk > > > introducing some regressions, and I think we need to make this test > group > > > set required. > > > > > > e.g. https://github.com/apache/pulsar/pull/17163 > > > > > > WDYT? > > > > > > Best, > > > Mattison > > > > > > -- BR, Qiang Huang
Re: [DISCUSS] Extended auth operation
Nice catch! +1 Zixuan Liu 于2022年8月22日周一 16:49写道： > Hi all, > > I noticed we added the TenantOperation, NamespaceOperation, and > PolicyOperation in https://github.com/apache/pulsar/pull/6428, which > provide the "real" authz abilities for each resource. This PR only adds > check permissions method in authorization provider, no grant, revoke, and > get action, so I want to add this feature. > > Best, > Zixuan > -- BR, Qiang Huang
Re: [DISCUSS] Use Github project to track the multi clients development
It makes sense to me. I'm thinking, if an issue is encountered in a Java client, it is possible to be in other clients. If so, the Java client board is also needed. mattison chao 于2022年8月22日周一 16:00写道： > +1 > > Best, > Mattison > > On Mon, 22 Aug 2022 at 15:52, Baodi Shi > wrote: > > > Nice idea! > > > > Thanks, > > Baodi Shi > > > > > On Aug 22, 2022, at 12:4250, Zike Yang wrote: > > > > > > Hi, all > > > > > > Currently, many features or bug fixes of some clients like the C++ > > > client and Python client have not caught up with the Java client. For > > > better tracking of these features or bug fixes, I have created a > > > Github project called `Apache Pulsar / Multi Clients`[0] under the > > > pulsar main repo. It can be removed if we don't like it this way. > > > > > > There are six columns in this Github project: > > > * Backlog: Low priority issues or issues(or cards) that have not been > > > initially investigated. > > > * Todo: Newly added issues and issues suitable for contributors to pick > > first. > > > * In progress: Work in progress issues or PRs. > > > * Review in progress: The PRs or issues that are waiting for review. > > > * Reviewer approved: The PRs or issues that are waiting to merge. > > > * Done: Merged PRs and Solved issues. > > > > > > This not only helps to track the status of other clients' features and > > > bug fixes but also helps contributors who want to participate in other > > > language client feature development to pick the task. > > > > > > This Project is currently only for clients maintained by the main > > > pulsar repository, such as C++ and Python clients. > > > > > > I have created a PR [1] to add C++ and Python client issues and PRs to > > > the Multi Clients project. > > > > > > Please let me know what you think. Thanks! > > > > > > [0] https://github.com/apache/pulsar/projects/12 > > > [1] https://github.com/apache/pulsar/pull/17201 > > > > > > Best, > > > Zike Yang > > > > > -- BR, Qiang Huang
Re: [DISCUSS] Enable message deduplication for replicator by default
+1 Zike Yang 于2022年8月22日周一 15:32写道： > +1 > > Thanks, > Zike Yang > > On Mon, Aug 22, 2022 at 3:16 PM mattison chao > wrote: > > > > +1 > > > > Best, > > Mattison > > > > On Fri, 19 Aug 2022 at 01:40, Enrico Olivelli > wrote: > > > > > I agree > > > > > > Enrico > > > > > > Il Gio 18 Ago 2022, 18:23 PengHui Li ha scritto: > > > > > > > Hi all, > > > > > > > > When I tried to fix a problem related to replicator > > > > https://github.com/apache/pulsar/pull/17154 > > > > It surprised me that the message deduplication will not work by > default > > > > with the replicator. > > > > I always thought it was enabled for replicators by default. Details > to > > > see > > > > [0]. > > > > > > > > I think we should enable the deduplication for the replicator. > Otherwise, > > > > we will see duplicated > > > > messages on the remote cluster. And the producer of the replicator > always > > > > has a fixed producer > > > > name, this will make the message deduplication work properly. > > > > > > > > The test introduced in https://github.com/apache/pulsar/pull/17154 > will > > > > check the message > > > > replication ordering. Without the message deduplication enabled, the > test > > > > is flaky with received > > > > duplicated messages. After enabling, everything is fine. > > > > > > > > Best, > > > > Penghui > > > > > > > > [0] > https://github.com/apache/pulsar/pull/17154#discussion_r948736894 > > > > > > > > -- BR, Qiang Huang
Re: [DISCUSS] Enable non-mandatory updating PR branches
It works now. It helps. Zixuan Liu 于2022年8月22日周一 10:38写道： > +1 > > Best, > Zixuan > > On 2022/08/18 13:01:58 tison wrote: > > Hello, > > > > The short version > > = > > > > Vote if you agree on enabling the non-mandatory updating PR branches > > button, i.e., the "Always suggest updating pull request branches" GitHub > > settings. > > > > The full version > > > > > > Pulsar is under rapid development and numerous fixes are pushed to master > > every time. Since we are still suffering from quite a few flaky tests, > > merge master and retest is a hotspot to verify the patch once more. > > > > However, we should pull the nightly master locally, check out the PR > > branch, perform the merge and push to remote. It's a bit awkward > especially > > when a developer works on multiple branches. > > > > GitHub provides a button "Always suggest updating pull request branches" > > with the description "Whenever there are new changes available in the > base > > branch, present an “update branch” option in the pull request."[1] > > > > It can simplify the workflow with one button click. > > > > To clarify, this is different from the branch protection rule "Require > > branches to be up to date before merging" - it's non-mandatory and just > > provides the "update branch" button. It means we don't force every PR to > > catch up with the latest master before merged, which can cause > exextremely > > high unnecessary traffic. Since we already allow PR authors to retrigger > > tests with the pulsorbot (or even contributors can push an empty commit), > > providing such a button does no harm. > > > > I post this thread here to collect feedback, especially from the PMC > > members. Previously I asked the INFRA team to turn on this option for > > Apache Kvrocks (Incubating)[2] and I believe the INFRA team would be > happy > > to see an explicit community consensus. > > > > Best, > > tison. > > > > [1] > > > https://github.blog/changelog/2022-02-03-more-ways-to-keep-your-pull-request-branch-up-to-date/ > > [2] https://issues.apache.org/jira/browse/INFRA-23432 > > -- BR, Qiang Huang
Re: Enable "Update branch" Github button in pull requests
+1. It can help a lot. Zixuan Liu 于2022年8月22日周一 10:38写道： > +1 > > Best, > Zixuan > > On 2022/07/12 16:06:43 Nicolò Boschi wrote: > > Hi all, > > > > I'd like to propose to enable the Github button "update branch" in the > pull > > requests. > > > > The main reason is that it helps when you want to rebase your pull to the > > current master. > > For instance today a fix for a failing job in the CI has been committed > and > > you were forced to close/reopen the pull or manual rebase your branch. > With > > this button is much easier and it will retrigger the CI as well. > > > > This is the guide to enable it: > > > https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/keeping-your-pull-request-in-sync-with-the-base-branch > > > > This is an example where I activated the button in my own Pulsar fork: > > https://github.com/nicoloboschi/pulsar/pull/4 > > > > Thanks, > > Nicolò Boschi > > -- BR, Qiang Huang
[GitHub] [pulsar-site] urfreespace merged pull request #169: Move config docs to /reference and improve docs gen scripts
urfreespace merged PR #169: URL: https://github.com/apache/pulsar-site/pull/169 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Anonymitaet commented on issue #173: Improve color shades for dark mode
Anonymitaet commented on issue #173: URL: https://github.com/apache/pulsar-site/issues/173#issuecomment-1225476599 Hi designers, @gancunjie @DaveDuggins, is it a good practice to set all the texts in the dark color background to `white`? https://user-images.githubusercontent.com/50226895/186384398-a6b3c13f-b49b-457a-9895-64740cfe3c48.png";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [Discussion] PIP 78: Replace brodocs with docsify?
Hi Dave, thanks for your great reminder! We'll add those rules. Yu, signormercurio, urfreespace On Tue, Aug 23, 2022 at 10:33 PM Dave Fisher wrote: > If page urls change then we should also add http redirect rules to help > users who bookmarked these docs. > > Sent from my iPhone > > > On Aug 23, 2022, at 5:53 AM, Yu wrote: > > > > Hi team, > > > > Now is 3 working days after the initial discussion > > > > We'll start replacing brodocs with docsify for docs since there is no > > objection or technical suggestions. > > > > Yu, signormercurio, urfreespace > > > >> On Fri, Aug 19, 2022 at 5:09 PM Yu wrote: > >> > >> Thanks Dave! > >> > >>> How is this search organized on the server and how would this integrate > >> with algolia? > >> > >> Search is performed at the front end, and it does not need to integrate > >> with Algolia. > >> > >>> Google Analytics will be disallowed on ASF project sites very soon due > >> to Privacy concerns with tracking data. I’ve warned about this several > >> times. Please eliminate GA. If we need to track then we have options. > >> > >> Listing GA here just shows docsify's capabilities. > >> As we discussed before [1], GA will no longer process new data in > standard > >> properties beginning July 1, 2023. > >> Matteo is being considered to replace it. > >> > >> [1] https://github.com/apache/pulsar/issues/15664 > >> > >> Yu and signormercurio > >> > >> > >> > >> > >> > >> > >>> On Thu, Aug 18, 2022 at 9:46 PM Dave Fisher > wrote: > >>> > >>> > >>> > >>> Sent from my iPhone > >>> > On Aug 18, 2022, at 4:55 AM, Yu wrote: > > Hi team, > > To reduce manual work, we’re constantly implementing PIP 78: Generate > >>> Docs > from Code Automatically [1]. > > Docs here refer to the following： > - Connector configurations [2] > - Client configurations [3] > - Pulsar CLI tools [4] > - Pulsar configurations [5] > - pulsar-admin [6] > > # Question > > Currently, some of the docs above [7] are generated using brodocs [8]. > We’re wondering if it makes sense to replace brodocs with docsify? > > # docsify benefits > > ## For users > - Enjoy new features (eg. search, copy to clipboard, language > >>> highlighting, > tabs, Disqus, etc.) > >>> > >>> How is this search organized on the server and how would this integrate > >>> with algolia? > >>> > - More clear and organized layout > > ## For doc maintainers > - Simple and lightweight > - No statically built HTML files. Instead, it smartly loads and parses > >>> .md > files and displays them as a website. > - Allow feature customization > - Able to work with Google Analytics to track data > >>> > >>> Google Analytics will be disallowed on ASF project sites very soon due > to > >>> Privacy concerns with tracking data. I’ve warned about this several > times. > >>> Please eliminate GA. If we need to track then we have options. > >>> > >>> Also we never discuss GA anywhere within the PMC or developer > community. > >>> What value does it have? > >>> > >>> Dave > >>> > > # Demo > > We’ve created a demo here [9]. > This demo is only for demonstration purposes, so we just add a few > improvements. It can be iterated later. > > > > If docsify is better, we’ll use it for all the docs above to provide a > consistent content experience. > > Feel free to check and comment, thank you! > > [1] https://lists.apache.org/thread/638q6kmq81z2qjmyhgkqzp00sllh4x0w > [2] https://pulsar.apache.org/docs/next/io-connectors (Configurations > >>> for > each connector) > [3] https://pulsar.apache.org/docs/next/client-libraries-java > (Configuration for each client. Here takes Java client as an example, > it > refers to configurations of client, producer, consumer, and reader) > [4] https://pulsar.apache.org/docs/next/reference-cli-tools > [5] https://pulsar.apache.org/docs/next/reference-configuration > [6] https://pulsar.apache.org/tools/pulsar-admin/ > [7] https://pulsar.apache.org/tools/ > [8] https://github.com/Birdrock/brodocs > [9] https://pulsar.apache.org/tools/pulsar-config/2.11.0-SNAPSHOT/#/ > > Yu, signormercurio, urfreespace > >>> > >>> > >
Re: [DISCUSS] Move PIPs to the codebase?
Hi Penghui, Thanks for raising this up! > we should have a discussion first on the mailing list, just get feedback on the > motivation. If there are no objections, > the proposal owner can add a line to the file with the PIP number through > a PR, like PIP-123: xxx (Under Discussion). > So that we can prevent the duplicated PIP number This is a good idea to ensure no NO. is re-used or jumped over. As you explained, we can act like this: 1. Send the PIP idea to the community without using a NO. 2. After the idea is approved, add "PIP 123 (under discussion)" to the "PIP metadata" file. 3. ... Yu On Wed, Aug 24, 2022 at 10:01 AM PengHui Li wrote: > Hi Rajan, > > > I didn't understand this part. You want one to create a PR before > submitting a proposal? That's clearly not a good idea because if the PIP > approach will change then the entire development effort will be wasted and > that's the whole purpose of PIP. I guess creating PIP into an issue and > discussing the issue is definitely working and it's an easier way to > discuss quickly rather than discussing over email threads. > > Sorry, I mean create a PR to add the proposal, not the implementation. > Before adding the proposal, we should discuss the motivation on the mailing > list first. > > > Let's not change this practice without good discussion and agreement from > the community. > > I don't think this essentially changes this part, it just provides a way > for reviewers to > review the details of the proposal more efficiently. For a quick > discussion, this will not change > it. Let me try to clarify the procedure after we applied this idea > > >1. Send the email first to discuss the motivation of your proposal >2. Discuss the motivation under the mailing list and request a PIP >number if there are no objections >3. Create a PR to add the detailed proposal >4. *Send out the VOTE email with the PR link* >5. *Review the proposal(The PR) on Github and vote under the mailing >list.* >6. Merge the PR of the proposal after the proposal get 3 (+1) bindings > > The only difference is step 4 and step 5. Currently, we share the issue > link to ask for a review. > > Thanks, > Penghui > > On Wed, Aug 24, 2022 at 1:22 AM Rajan Dhabalia > wrote: > > > Hi, > > > > >>> I think we can move all the PIPs to the codebase and the new proposal > > and proposal without any reviews should happen with a PR first. So that > we > > can review and comment easily. > > > > I didn't understand this part. You want one to create a PR before > > submitting a proposal? That's clearly not a good idea because if the PIP > > approach will change then the entire development effort will be wasted > and > > that's the whole purpose of PIP. I guess creating PIP into an issue and > > discussing the issue is definitely working and it's an easier way to > > discuss quickly rather than discussing over email threads. > > > > Let's not change this practice without good discussion and agreement from > > the community. > > > > Thanks, > > Rajan > > > > On Thu, Aug 18, 2022 at 8:27 AM PengHui Li wrote: > > > > > Hi all, > > > > > > Currently, the new proposal will be added to the issue list and then > > shared > > > link in the email > > > to request the proposal review. It's really hard to review a long > > proposal > > > if you want to comment > > > in detail. > > > > > > Here is an example: > > > https://github.com/apache/pulsar/issues/16763#issuecomment-1219606491 > > > This seems very unintuitive. > > > > > > I think we can move all the PIPs to the codebase and the new proposal > and > > > proposal without > > > any reviews should happen with a PR first. So that we can review and > > > comment easily. > > > Certainly, all the votes should happen on the mailing list. And we can > > also > > > discuss the > > > proposal on the mailing list. > > > > > > Following this way, we don't need to sync the PIPs from the issue to > the > > > wiki page. > > > We can just add a link that points to the PIPs dir to the contribution > > > guide or README. > > > > > > We have another pain point about the duplicated PIP number. We can > > maintain > > > a file, a list of > > > all the proposal contains the approved, in-review, drafting. Before > > > creating a proposal, we should > > > have a discussion first on the mailing list, just get feedback on the > > > motivation. If there are no objections, > > > the proposal owner can add a line to the file with the PIP number > > through a > > > PR, like PIP-123: xxx (Under Discussion). > > > So that we can prevent the duplicated PIP number(which will conflict if > > > someone merged first). > > > After the PR is merged, we can send out a new PR to add the proposal. > > > > > > Thanks, > > > Penghui > > > > > >
Re: [DISCUSS] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
Hi Enrico and Qiang, The issue description was updated. Picked here: > upgrade and downgrade doc This change should not affect those who use Pulsar with the entry point script, but it changes the layout of the release artifact. I'm going to write a release note about this change and also post it on the Pulsar SQL overview page as a caveat. Draft here: # Caveat If you're upgrading Pulsar SQL from 2.11 or early, you should copy the related configs from `conf/presto` to `trino/conf`, and `lib/presto` to `trino`. If you're downgrading Pulsar SQL to 2.11 or early from 2.12, do verse visa. > Pulsar Helm Chart >From what I understand, Pulsar Helm Chart is a wrapper of pulsar-all docker image. I don't find any reference to presto/trino/sql effectively in the codebase, so I think it's currently transparent for its users and the caveat under Pulsar SQL overview page should be enough. Best, tison. tison 于2022年8月17日周三 19:40写道： > Thanks for your feedback! > > 1. According to the upgrade and downgrade doc, I think the minimum > requirements are a release note. Describe the layout change and how users > should move the folder. I'll elaborate on the issue and notify you here. > 2. It seems Pulsar Helm Chart support 2.9.3 now. I'll investigate how it > can be relevant in days. If someone who maintains the Chart can provide > some input, it will help! > 3. "There are 3 issues". It's a description about the > background/motivation, while we handle the first issue in this PIP. > Although, subtasks can be divided into packaging changes, possible doc > changes, and possible Chart changes. > > Best, > tison. > > > Qiang Huang 于2022年8月17日周三 19:25写道： > >> Looks good. I have two points: >> 1. It is necessary to supplement the upgrade and downgrade documentation >> in >> Pulsar. >> 2. There are 3 issues mentioned in the PIP, should we split it into 3 >> small >> issues? >> >> Enrico Olivelli 于2022年8月17日周三 17:30写道： >> >> > I generally agree with the PIP >> > >> > Can you please explain the interactions with the Pulsar Helm chart ? >> > also we have to draw a migration path, because users that will upgrade >> > Pulsar will have to move the configuration files in another location >> > >> > Enrico >> > >> > Il giorno mer 17 ago 2022 alle ore 11:15 tison >> > ha scritto: >> > > >> > > Hello, >> > > >> > > This is a PIP to package the Pulsar Trino distro and config in a >> > dedicated >> > > folder. >> > > >> > > Link: https://github.com/apache/pulsar/issues/17137 >> > > Prototype: https://github.com/apache/pulsar/pull/17062 >> > > >> > > Below you can find the proposal (I will amend the GH issue while we >> > discuss >> > > it). >> > > >> > > Best, >> > > tison. >> > > >> > > Motivation >> > > >> > > >> > > After https://github.com/apache/pulsar/pull/16683 merged, we upgrade >> > > PrestoSQL dependency in Pulsar SQL to the first several Trino >> version. To >> > > handle the name change cases and gradually refactor Pulsar SQL as a >> > > self-contained module so that we can move it into a standalone >> > repository, >> > > I find that there're three major issues to resolve. >> > > >> > > 1. Configs of Pulsar SQL go under the `conf/` folder and mix with >> other >> > > Pulsar configs. >> > > 2. Pulsar Docker images (base and all) bundle Pulsar SQL. >> > > 3. Integration tests of Pulsar SQL are tightly coupled with the main >> repo >> > > (test infra). >> > > >> > > This proposal is aimed at resolving the first issue to package Pulsar >> > Trino >> > > distro and config in a dedicated folder; that is, to make it >> > self-contained. >> > > >> > > Goal >> > > >> > > >> > > I have already prepared a draft to perform the changes as >> > > https://github.com/apache/pulsar/pull/17062. Generally, we move the >> > config >> > > files under `PRESTO_HOME` and correspondingly update scripts. >> > > >> > > In this way, all Trino distro artifacts are under the same home path, >> so >> > > that we can later move it out as a whole. >> > > >> > > This change should not affect those who use Pulsar with the entry >> point >> > > script, but it changes the layout of the release artifact, so I'd >> prefer >> > to >> > > perform a PIP process. >> > > >> > > Implementation >> > > >> > > >> > > It's straightforward to inline in the "Goal" section. >> > > >> > > However, the name of the folder (`presto` or `trino`) and the level of >> > the >> > > folder (`lib/presto/` or `trino/`) is open to discussion. I think both >> > are >> > > fine and will try `trino/` first. >> > > >> > > To minimize unnecessary changes, I tend to keep the modules name >> > > `pulsar-presto-xxx` as is. >> > > >> > > Alternatives >> > > = >> > > >> > > I don't make a completed proposal to resolve all three issues listed >> > above. >> > > Because I'm still unfamiliar with the latter two topics yet and I'd >> > prefer >> > > to implement these improvements one by one since they're naturally >> > > independent. If I try to make a completed
[DISCUSS] Deprecate KeyValue schema factory methods with Class parameters
Hi folks, Recently I'm looking into the KeyValue schema and found **FOUR** static methods in `Schema` to create a `KeyValueSchema` object: 1. KeyValue(Class key, Class value); 2. KeyValue(Class key, Class value, SchemaType type); 3. KeyValue(Schema key, Schema value); 4. KeyValue(Schema key, Schema value, KeyValueEncodingType keyValueEncodingType); IMO, having too many overloads is not user-friendly. I turned to the official website and found overload 4 is used. The overload 3 is just a simple version of overload 4 whose encoding type is `INLINE`, but it's not clear. I opened a PR https://github.com/apache/pulsar/pull/17256 to make it more clear. However, for overload 1 and 2, I can only find two references in unit tests `testAllowNullAvroSchemaCreate` and `testAllowNullJsonSchemaCreate`. From the very simple JavaDocs, it looks like they are only available for JSON and AVRO schemas. Then I found the original PR to introduce these overloads: https://github.com/apache/pulsar/pull/2885 Code has been changed a lot since that. It looks other codes are not much related to these two overloads now. IMO, we should not encourage users to use these two overloads, so I suggest marking them as deprecated and might remove them in future releases. Thanks, Yunze
[GitHub] [pulsar-site] tisonkun commented on issue #173: Improve color shades for dark mode
tisonkun commented on issue #173: URL: https://github.com/apache/pulsar-site/issues/173#issuecomment-1225347135 @Anonymitaet for this specific issue, I think we need UI experts to help with selecting colors series. The related code is https://github.com/apache/pulsar-site/blob/e1785ad30a9bd02047ccd31e85e9c89fa1c37c6a/site2/website-next/src/css/custom.css while is hosted primary in this repo instead of synced from `pulsar`. @urfreespace suggests I report site-related topics here and content-related topics in the main repo. Although, I agree on the "High visibility". I'm OK to just disable the issue tab in this repo and report all issues in `pulsar` - [SkyWalking](https://github.com/apache/skywalking) follows this pattern. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [VOTE] [PIP-201] Extensions mechanism for Pulsar Admin CLI tools
+1 (non binding) Thanks, Yunze > 2022年8月24日 15:38，Nicolò Boschi 写道： > > +1 (non binding) > Nicolò Boschi > > > Il giorno mer 24 ago 2022 alle ore 09:11 Enrico Olivelli < > eolive...@gmail.com> ha scritto: > >> Hello, >> this is the official thread VOTE for PIP-201 Extensions mechanism for >> Pulsar Admin CLI tools >> >> This is the PIP link https://github.com/apache/pulsar/issues/17155 >> This is the discussion: >> https://lists.apache.org/thread/287ft8twc11cp4s1y4qkcx5nmh451cyo >> >> I am still working on the PR, that is the subject of the VOTE. >> >> Best regards >> Enrico Olivelli >>
[GitHub] [pulsar-site] Anonymitaet commented on issue #173: Improve color shades for dark mode
Anonymitaet commented on issue #173: URL: https://github.com/apache/pulsar-site/issues/173#issuecomment-1225320929 @tisonkun thanks for raising this up! Even though the Pulsar website is built in the `pulsar-site` repo and many website-related data are hosted here, I still suggest reporting website-related issues in the `pulsar` repo. Reasons: - High visibility, which means the issue can get more attention since the majority of ppl go to `pulsar`. Only a few ppl know `pulsar-site`. - From past to now, we nearly track all website-related issues in `pulsar`, it's convenient for us to triage and check them all in one place. Thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [VOTE] [PIP-201] Extensions mechanism for Pulsar Admin CLI tools
+1 (non binding) Nicolò Boschi Il giorno mer 24 ago 2022 alle ore 09:11 Enrico Olivelli < eolive...@gmail.com> ha scritto: > Hello, > this is the official thread VOTE for PIP-201 Extensions mechanism for > Pulsar Admin CLI tools > > This is the PIP link https://github.com/apache/pulsar/issues/17155 > This is the discussion: > https://lists.apache.org/thread/287ft8twc11cp4s1y4qkcx5nmh451cyo > > I am still working on the PR, that is the subject of the VOTE. > > Best regards > Enrico Olivelli >
[GitHub] [pulsar-site] Anonymitaet merged pull request #171: Update the website banner
Anonymitaet merged PR #171: URL: https://github.com/apache/pulsar-site/pull/171 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] tisonkun opened a new issue, #173: Improve color shades for dark mode
tisonkun opened a new issue, #173: URL: https://github.com/apache/pulsar-site/issues/173 Currently, in dark mode, the contents on several pages are hard to read. For example, https://pulsar.apache.org/docs/next/tiered-storage-filesystem#configure-filesystem-offloader-driver. https://user-images.githubusercontent.com/18818196/186356215-8b6e3d0f-0ded-4db4-9be7-d63c518f54bc.png";> cc @Anonymitaet @urfreespace -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[VOTE] [PIP-201] Extensions mechanism for Pulsar Admin CLI tools
Hello, this is the official thread VOTE for PIP-201 Extensions mechanism for Pulsar Admin CLI tools This is the PIP link https://github.com/apache/pulsar/issues/17155 This is the discussion: https://lists.apache.org/thread/287ft8twc11cp4s1y4qkcx5nmh451cyo I am still working on the PR, that is the subject of the VOTE. Best regards Enrico Olivelli
Re: [DISCUSS] [PIP-201] Extensions mechanism for Pulsar Admin CLI tools
Yunze, I have updated the PIP with the code of a sample command. Thank you everyone for your comments, I will start a VOTE. Enrico Il giorno mar 23 ago 2022 alle ore 05:26 Michael Marshall ha scritto: > > This proposal looks good to me. I think it will be a helpful addition > to the pulsar admin tool. I like that the design avoids an unnecessary > dependency on JCommander. > > Thanks, > Michael > > > On Mon, Aug 22, 2022 at 1:22 PM Enrico Olivelli wrote: > > > > Yunze, > > > > > > > > Il Lun 22 Ago 2022, 20:20 Enrico Olivelli ha scritto: > > > > > > > > > > > Il Lun 22 Ago 2022, 17:07 Yunze Xu ha > > > scritto: > > > > > >> I will take a look. But I think we should also add a trivial example > > >> (or a test) in the Apache repo > > > > > > > > I will add integration tests for sure. > > Currently the PR is only a prototype without tests. > > > > Enrico > > > > , e.g. just print some messages for an > > >> extended command. > > > > > > > > > Sure. I will update the PIP > > > > > > And the JavaDocs of these interfaces should be > > >> complete and more clear. > > >> > > > > > > > > > I will leave that to the implementation PR > > > > > > Enrico > > > > > > > > >> Thanks, > > >> Yunze > > >> > > >> > > >> > > >> > > >> > 2022年8月22日 18:37，Enrico Olivelli 写道： > > >> > > > >> > Yunze, > > >> > > > >> > Il giorno lun 22 ago 2022 alle ore 08:06 Yunze Xu > > >> > ha scritto: > > >> >> > > >> >> The motivation and goal LGTM, but the API changes look very simple and > > >> >> hard to use. Do we have to implement all these interfaces for an admin > > >> >> extension? If yes, could you show an example in the proposal as a > > >> >> guidance? > > >> >> > > >> >> For example, if I just want to implement a simple command: > > >> >> > > >> >> ```bash > > >> >> ./bin/pulsar-admin kafka create-topic --partitions > > >> > > >> >> ``` > > >> >> > > >> >> How should I implement these interfaces? > > >> > > > >> > This is a example for the implementation that I am going to do for JMS > > >> > > > >> https://github.com/datastax/pulsar-jms/pull/53/files#diff-9afaac9c7dc4b3d674e0623cd3d76348b01537c6095e9b5b8e804f59a481cceeR31 > > >> > > > >> > it is only a mock command at the moment, but it is good to showcase the > > >> feature > > >> > > > >> > Enrico > > >> > > > >> > > > >> >> > > >> >> Thanks, > > >> >> Yunze > > >> >> > > >> >> > > >> >> > > >> >> > > >> >>> 2022年8月18日 16:23，Enrico Olivelli 写道： > > >> >>> > > >> >>> Hello, > > >> >>> I have drafted a PIP around this proposal. > > >> >>> > > >> >>> PIP link: https://github.com/apache/pulsar/issues/17155 > > >> >>> > > >> >>> I am preparing an official PR, I already have a working prototype. > > >> >>> > > >> >>> Copy of the contents of the GH issue is attached for discussion here > > >> >>> on the Mailing list. > > >> >>> > > >> >>> Motivation > > >> >>> > > >> >>> There are many projects that are in the Pulsar ecosystem like > > >> >>> Protocol > > >> >>> Handlers (Kafka, MQTT, RabbitMQ) and libraries (JMS…) that need > > >> >>> additional tools for operating Pulsar following specific conventions > > >> >>> adopted by each project and to handle custom domain objects (like JMS > > >> >>> queues, Kafka Consumer Groups...). > > >> >>> > > >> >>> Some examples: > > >> >>> > > >> >>> Kafka: tools for inspecting internal systems, SchemaRegistry, > > >> >>> Transaction Manager, Consumers Groups > > >> >>> JMS: tools to handling Subscriptions and Selectors > > >> >>> RabbitMQ: tools to handle Pulsar topics and subscription following > > >> >>> the > > >> >>> convention > > >> >>> > > >> >>> This is very important as it is hard to follow the conventions of > > >> >>> each > > >> >>> project using pulsar-admin and the administrator may inadvertently > > >> >>> break the system. > > >> >>> > > >> >>> This feature will enhance the UX of the Pulsar Admin CLI tools for > > >> >>> the > > >> >>> benefit of the whole ecosystem and users. > > >> >>> > > >> >>> Goal > > >> >>> > > >> >>> As we do for many other components in Pulsar, we need a way to > > >> >>> enhance > > >> >>> the CLI tools, pulsar-admin and pulsar-shell, with additional > > >> >>> commands > > >> >>> that are specific to the additional features. > > >> >>> > > >> >>> The proposal here is to add an extension mechanism to the > > >> >>> pulsar-admin > > >> >>> (and so pulsar-shell) tool. > > >> >>> Following the usual conventions for extensions the extension will be > > >> >>> bundled in a .nar file that may contain additional third party > > >> >>> libraries. > > >> >>> > > >> >>> The extension will be able to provide new top level commands > > >> >>> > > >> >>> pulsar-admin my-command-group my-command arguments… > > >> >>> > > >> >>> The extension will be able to access the PulsarAdmin API provided by > > >> >>> the environment. > > >> >>> > > >> >>> The extension must not depend directly on the JCommander library but > > >> >>> we will provide an API to declare the parameters and the other > > >> >>> met
Re: [VOTE] [PIP-169] Support split bundle by flow or qps
+1 (binding) Enrico Il giorno mer 24 ago 2022 alle ore 07:13 Aloys Zhang ha scritto: > > +1 > > Haiting Jiang 于2022年8月24日周三 12:35写道： > > > +1 (non) > > > > Thanks, > > Haiting > > > > On Wed, Aug 24, 2022 at 10:19 AM guo jiwei wrote: > > > > > +1 (binding) > > > > > > > > > Regards > > > Jiwei Guo (Tboy) > > > > > > > > > On Wed, Aug 24, 2022 at 1:01 AM Heesung Sohn > > > wrote: > > > > > > > +1 (non-binding) > > > > > > > > Heesung > > > > > > > > On Tue, Aug 23, 2022 at 10:00 AM Anon Hxy wrote: > > > > > > > > > > +1 (non-binding) > > > > > > > > > > Thanks, > > > > > Xiaoyu Hou > > > > > > > > > > lordcheng10 <1572139...@qq.com.invalid> 于2022年8月23日周二 15:10写道： > > > > > > > > > > > Hi Pulsar Community, I would like to start a VOTE on "Support split > > > > bundle > > > > > > by flow or qps."(PIP-169) > > > > > > Here is the design detail: > > > > https://github.com/apache/pulsar/issues/16782 > > > > > > > > > > > > > > > > > > and the discussion thread: > > > > > > https://lists.apache.org/thread/cshyt10fwcjjxs93g8yf0svgwcgnshmg > > > > > > > > > > > > > > > > > > Thanks, > > > > > > lordcheng10 > > > > > > > > >
