Re: Can we archive https://github.com/apache/pulsar-release?
>> Pulsar SQL >> objections >It's not objections the blocker, but volunteers. IIRC, there were objections to moving pulsar sql to its own repo because there was an effort to contribute it to trino. Given that the trino contribution has stalled, I think we should move pulsar sql out of the apache/pulsar code base. It takes a lot of resources to test. Thanks, Michael On Tue, Jun 27, 2023 at 3:26 PM tison wrote: > > While, I visit tests/integration recently that it can be clear how to > factor out the testcontainers-based integration utils. Let me file an issue > to track the work so that we have more visibility about the progress and > hopefully a nudge to the target - of course separating those code is a good > idea. > > Back to apache/pulsar-release, it seems the separation of connectors is > part of the prerequisite. > > Best, > tison. > > > tison 于2023年6月28日周三 04:15写道： > > > > Pulsar SQL > > > objections > > > > It's not objections the blocker, but volunteers. > > > > A comparable example is that Flink firstly tried to move out connectors in > > 2018 and finally made it happen in 2022 (specifically, Kafka connector > > _today_[1]). That indicates such initiatives require heavy development. > > > > For Pulsar SQL separation, there are two major works I don't find time to > > finish - > > > > 1. Factor out integration utils to shared libs and release them. Flink > > finally has flink-connector-shared-utils[2] > > and flink-test-utils-junit, flink-connector-test-utils, > > flink-architecture-tests-test, and so on. > > 2. Work out a solution to handle Docker image build and release after > > Pulsar SQL moved out. > > > > Best, > > tison. > > > > [1] https://lists.apache.org/thread/vt4746r8kbnmcq7llcsvyjk4gor7gops > > [2] https://github.com/apache/flink-connector-shared-utils/branches > > > > > > Lari Hotari 于2023年6月28日周三 01:12写道： > > > >> I hope we could finally resume the work on PIP-62 and continue with > >> the plan that was set in April 2020. > >> You can find old discussion threads about PIP-62 with this query: > >> https://lists.apache.org/list?dev@pulsar.apache.org:lte=10y:PIP-62 > >> > >> There was a discussion in July 2022 about moving Pulsar SQL to a > >> separate repository. [1] > >> Could we finally make this happen? Any objections? > >> > >> -Lari > >> > >> [1] https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8 > >> > >> On Tue, Jun 27, 2023 at 1:35 PM Enrico Olivelli > >> wrote: > >> > > >> > Yes we can. > >> > > >> > It was a tentative goal of creating a new workflow to build releases, > >> > especially for the pulsar-all docker image, that depends both on > >> > pulsar main repo code and on Pulsar SQL, > >> > that we wanted to remove from the base repository. > >> > > >> > Enrico > >> > > >> > Il giorno mar 27 giu 2023 alle ore 08:27 Asaf Mesika > >> > ha scritto: > >> > > > >> > > Seems quite neglected > >> > >
Re: [BLOCKING] Master branch always fails on testElasticSearch8Sink
> I think it's better to disable (or quarantine) a test if it blocks master and > there's no immediate solution. It is regrettable to disable a test, but given that this test is passing locally, I agree with quarantining the test. Thanks, Michael On Wed, Jun 28, 2023 at 12:10 AM Lari Hotari wrote: > > I think it's better to disable (or quarantine) a test if it blocks master and > there's no immediate solution. > > -Lari > > On 2023/06/28 01:52:14 tison wrote: > > See also https://github.com/apache/pulsar/issues/20661 > > > > Enrico and I both verified that it works well locally, so that can be an > > env issue or unstable dependency - I checked the ES image not changed, > > though. > > > > If we cannot locate the cause quickly, perhaps disable the test to unblock > > other PRs first? > > > > I tried to read the code, but there is no trivial cause (even the test > > passed locally). The log indicates that statistics received one message > > instead of 20 expected, but as other test cases passed, it may not be a > > kernel logic issue. > > > > Best, > > tison. > >
Re: [BLOCKING] Master branch always fails on testElasticSearch8Sink
I think it's better to disable (or quarantine) a test if it blocks master and there's no immediate solution. -Lari On 2023/06/28 01:52:14 tison wrote: > See also https://github.com/apache/pulsar/issues/20661 > > Enrico and I both verified that it works well locally, so that can be an > env issue or unstable dependency - I checked the ES image not changed, > though. > > If we cannot locate the cause quickly, perhaps disable the test to unblock > other PRs first? > > I tried to read the code, but there is no trivial cause (even the test > passed locally). The log indicates that statistics received one message > instead of 20 expected, but as other test cases passed, it may not be a > kernel logic issue. > > Best, > tison. >
Re: [DISCUSS] Make system topics non-partitioned by default
> Why not disable auto-creation for system topics? and then we can create it > before the broker start. which can avoid introduce complex meaning of > `allowAutoTopicCreationType` configuration. A broker cannot know all of the system topics to create at startup. For example, some system topics are created for each namespace. The current paradigm implicitly relies on auto creation as a form of automatic retry to ensure eventual creation of the namespace's system topic. > why we chose the non-partitioned topic? This is likely a deeper question about the role of system topics. I posit that a system topic should never exceed the throughput capabilities of a non-partitioned topic. Further, some system topics implicitly relied on it a single topic to ensure ordering, like the load balancing topics. This change likely improves stability for those features. > maybe we can just create one partition? This would work, but would only be necessary if we plan to increase the number of partitions later. Thanks, Michael On Tue, Jun 27, 2023 at 9:12 PM wrote: > > Hi, Michael > > I've got several questions and hope you can help answer. > > Why not disable auto-creation for system topics? and then we can create it > before the broker start. which can avoid introduce complex meaning of > `allowAutoTopicCreationType` configuration. > > Plus, why we chose the non-partitioned topic? maybe we can just create one > partition? because the non-partitioned topic can't scale by updating the > partition number. > > Best, > Mattison > On 28 Jun 2023 at 06:45 +0800, Michael Marshall , wrote: > > Hi Pulsar Community, > > > > After several recent issues regarding incorrectly auto created topics, > > I think we should default to creating system topics as non-partitioned > > topics when auto created. > > > > Lari put together a PR here > > https://github.com/apache/pulsar/pull/20397. The PR has relevant > > context. > > > > Let me know what you think. > > > > Thanks, > > Michael
[DISCUSS] Release Pulsar Node.js client v1.9.0
Hi all, I would like to propose releasing the Pulsar Node.js client v1.9.0 It has been over 2 months since the last release (1.8.2). There have been many new features and bug fixes since then. It's time to release a new version. - https://github.com/apache/pulsar-client-node/milestone/14?closed=1 Please let me know if you have any PRs that need to be included in 1.9.0. Thanks, Baodi Shi
Re: [VOTE] PIP-279: Reformat property in generateResponseWithEntry
+1 (non-binding)
Xiaoyu Hou
steven lu 于2023年6月28日周三 09:40写道：
> Hi, community:
>
> # Motivation
>
> reformat property,for a http header name cannot contain the following
> prohibited characters: =,;: \t\r\n\v\f
>
> for example:
> {"city=shanghai":"tag"}
> when we run `bin/pulsar-admin topics get-message-by-id `, it will
> throw exception, the exception is:
> `Reason: java.util.concurrent.CompletionException:
>
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been
> exhausted. Failed reason: a header name cannot contain the following
> prohibited characters: =,;: \t\r\n\v\f: =`
>
> # High Level Design
>
> In master branch,
> in an http
> request:getMessageById("/{tenant}/{namespace}/{topic}/ledger/{ledgerId}/entry/{entryId}"),
> replace `"X-Pulsar-PROPERTY-" + msgProperties.getKey()` with
> `"X-Pulsar-PROPERTY"`
>
> After release-3.1.0, this feature begins to take effect.
>
>
> PIP: https://github.com/apache/pulsar/pull/20627
>
> PR: https://github.com/apache/pulsar/pull/20481
>
Re: [DISCUSS] Make system topics non-partitioned by default
Hi, Michael I've got several questions and hope you can help answer. Why not disable auto-creation for system topics? and then we can create it before the broker start. which can avoid introduce complex meaning of `allowAutoTopicCreationType` configuration. Plus, why we chose the non-partitioned topic? maybe we can just create one partition? because the non-partitioned topic can't scale by updating the partition number. Best, Mattison On 28 Jun 2023 at 06:45 +0800, Michael Marshall , wrote: > Hi Pulsar Community, > > After several recent issues regarding incorrectly auto created topics, > I think we should default to creating system topics as non-partitioned > topics when auto created. > > Lari put together a PR here > https://github.com/apache/pulsar/pull/20397. The PR has relevant > context. > > Let me know what you think. > > Thanks, > Michael
Re: [DISCUSS] PIP-279: Reformat property in generateResponseWithEntry
the DISCUSS is passed,I have send the VOTE for this PIP
https://lists.apache.org/thread/g354684m9h495o3p0kmzb7fh7vfxhddx
On 2023/06/21 03:22:52 steven lu wrote:
> # Motivation
>
> reformat property,for a http header name cannot contain the following
> prohibited characters: =,;: \t\r\n\v\f
>
> for example:
> {"city=shanghai":"tag"}
> when we run `bin/pulsar-admin topics get-message-by-id `, it will
> throw exception, the exception is:
> `Reason: java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been
> exhausted. Failed reason: a header name cannot contain the following
> prohibited characters: =,;: \t\r\n\v\f: =`
>
> src="https://github.com/StevenLuMT/pulsar/assets/42990025/973d95b9-4ac2-4977-b160-162c4b53a613;>
>
> # High Level Design
>
> In master branch,
> in an http
> request:getMessageById("/{tenant}/{namespace}/{topic}/ledger/{ledgerId}/entry/{entryId}"),
> replace `"X-Pulsar-PROPERTY-" + msgProperties.getKey()` with
> `"X-Pulsar-PROPERTY"`
>
> After release-3.1.0, this feature begins to take effect.
>
[BLOCKING] Master branch always fails on testElasticSearch8Sink
See also https://github.com/apache/pulsar/issues/20661 Enrico and I both verified that it works well locally, so that can be an env issue or unstable dependency - I checked the ES image not changed, though. If we cannot locate the cause quickly, perhaps disable the test to unblock other PRs first? I tried to read the code, but there is no trivial cause (even the test passed locally). The log indicates that statistics received one message instead of 20 expected, but as other test cases passed, it may not be a kernel logic issue. Best, tison.
[VOTE] PIP-279: Reformat property in generateResponseWithEntry
Hi, community:
# Motivation
reformat property,for a http header name cannot contain the following
prohibited characters: =,;: \t\r\n\v\f
for example:
{"city=shanghai":"tag"}
when we run `bin/pulsar-admin topics get-message-by-id `, it will
throw exception, the exception is:
`Reason: java.util.concurrent.CompletionException:
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
Could not complete the operation. Number of retries has been
exhausted. Failed reason: a header name cannot contain the following
prohibited characters: =,;: \t\r\n\v\f: =`
# High Level Design
In master branch,
in an http
request:getMessageById("/{tenant}/{namespace}/{topic}/ledger/{ledgerId}/entry/{entryId}"),
replace `"X-Pulsar-PROPERTY-" + msgProperties.getKey()` with
`"X-Pulsar-PROPERTY"`
After release-3.1.0, this feature begins to take effect.
PIP: https://github.com/apache/pulsar/pull/20627
PR: https://github.com/apache/pulsar/pull/20481
[DISCUSS] Make system topics non-partitioned by default
Hi Pulsar Community, After several recent issues regarding incorrectly auto created topics, I think we should default to creating system topics as non-partitioned topics when auto created. Lari put together a PR here https://github.com/apache/pulsar/pull/20397. The PR has relevant context. Let me know what you think. Thanks, Michael
Re: [DISCUSS] Provide consistent concept around partitioned topic and topic partition/internal topic
I agree that the terminology is confusing. On a practical note, that confusion likely caused this bug [0]. > I'd like to say "a partitioned topic is a topic that consists of > multiple non-partitioned topics". It's better not to use the "internal > topic" concept. +1 In general, I think the equivocation for partitioned topic and non-partitioned topic is meant to simplify the abstraction for users. Partitioned topics are generally thought of as a single topic. However, advanced use cases can leverage the fact that a partitioned topic is composed of non-partitioned topics. That being said, when we document what a partitioned topic is, we should say that it is made of non-partitioned topics. In looking at your GitHub issue, I agree that the method names are not ideal for the TopicName class. We can certainly deprecate methods and add new ones. One additional problem is that we have too many string equality checks to determine if one topic is equal to another. There are many method parameters in the Pulsar code base that are named "topicName" and are of type String. These checks led to this bug [0]. I think it would be better to use the TopicName object more and to rely on the TopicName methods to implement equality/comparison checks. Thanks, Michael [0] https://github.com/apache/pulsar/pull/20392 Further, in pulsar, a "topic name" can be either a partitioned or a non partitioned topic. On Sun, Jun 25, 2023 at 8:13 AM Yunze Xu wrote: > > I'd like to say "a partitioned topic is a topic that consists of > multiple non-partitioned topics". It's better not to use the "internal > topic" concept. > > Thanks, > Yunze > > On Sun, Jun 25, 2023 at 4:31 PM Joo Hyuk Kim wrote: > > > > Hi community: > > > > Currently, it seems like there is no good definition of what "partitioned > > topic" and "non-partitioned topic" mean and some places topic > > partition and internal > > topic interchangeably. > > > > I would like to propose to vote (or discuss) to come up with solid > > definition of what composes "partitioned topic", then apply to pulsar-site, > > javadoc, and TopicName API accordingly. > > > > Let me know what you think > > > > > > # ISSUE https://github.com/apache/pulsar/issues/20622 Regards Joo Hyuk, > > Kim (Vince)
Re: [DISCUSS] which versions are EOL?
I removed the "release/2.9.6" label from 50 PRs: https://github.com/apache/pulsar/pull/20533 https://github.com/apache/pulsar/pull/20513 https://github.com/apache/pulsar/pull/20482 https://github.com/apache/pulsar/pull/20416 https://github.com/apache/pulsar/pull/20363 https://github.com/apache/pulsar/pull/20346 https://github.com/apache/pulsar/pull/20341 https://github.com/apache/pulsar/pull/20326 https://github.com/apache/pulsar/pull/20288 https://github.com/apache/pulsar/pull/20244 https://github.com/apache/pulsar/pull/20233 https://github.com/apache/pulsar/pull/20230 https://github.com/apache/pulsar/pull/20176 https://github.com/apache/pulsar/pull/20122 https://github.com/apache/pulsar/pull/20055 https://github.com/apache/pulsar/pull/20046 https://github.com/apache/pulsar/pull/20043 https://github.com/apache/pulsar/pull/20037 https://github.com/apache/pulsar/pull/20030 https://github.com/apache/pulsar/pull/20025 https://github.com/apache/pulsar/pull/19975 https://github.com/apache/pulsar/pull/19972 https://github.com/apache/pulsar/pull/19957 https://github.com/apache/pulsar/pull/19934 https://github.com/apache/pulsar/pull/19815 https://github.com/apache/pulsar/pull/19775 https://github.com/apache/pulsar/pull/19735 https://github.com/apache/pulsar/pull/19727 https://github.com/apache/pulsar/pull/19700 https://github.com/apache/pulsar/pull/19696 https://github.com/apache/pulsar/pull/19662 https://github.com/apache/pulsar/pull/17095 https://github.com/apache/pulsar/pull/18092 https://github.com/apache/pulsar/pull/17820 https://github.com/apache/pulsar/pull/18007 https://github.com/apache/pulsar/pull/18688 https://github.com/apache/pulsar/pull/17338 https://github.com/apache/pulsar/pull/19661 https://github.com/apache/pulsar/pull/17751 https://github.com/apache/pulsar/pull/19581 https://github.com/apache/pulsar/pull/19031 https://github.com/apache/pulsar/pull/16502 https://github.com/apache/pulsar/pull/15628 https://github.com/apache/pulsar/pull/15363 https://github.com/apache/pulsar/pull/15852 https://github.com/apache/pulsar/pull/15494 https://github.com/apache/pulsar/pull/14327 https://github.com/apache/pulsar/pull/13298 https://github.com/apache/pulsar/pull/14287 https://github.com/apache/pulsar/pull/14641 On Tue, Jun 27, 2023 at 4:37 PM Michael Marshall wrote: > > In light of this thread, I think we have a (lazy) consensus that 2.8 > and 2.9 are EOL and we will continue to maintain 2.10. > > Would someone be able to update the website to say that 2.10 is not EOL? > > I will update the GitHub labels to indicate the 2.9 label should no > longer be used. > > Thanks, > Michael > > On Fri, Jun 9, 2023 at 4:33 AM Yunze Xu wrote: > > > > > This is the latest release that runs on JDK11 > > > > This point makes sense to me. Actually I think it's the latest release > > that runs on JDK 8 though the recommended JDK for 2.10 is 11. The only > > question from me is when should 2.10 be EOL? > > > > Thanks, > > Yunze > > > > On Fri, Jun 9, 2023 at 3:15 PM Enrico Olivelli wrote: > > > > > > Micheal, > > > > > > Il giorno ven 9 giu 2023 alle ore 07:54 Michael Marshall > > > ha scritto: > > > > > > > > Hi Pulsar Community, > > > > > > > > I recently noticed this page on our website: > > > > https://pulsar.apache.org/contribute/release-policy/#supported-versions > > > > > > > > It shows that only 2.11 and 3.0 are active and security support. > > > > > > > > I am guessing the timelines were taken from a strict reading of PIP > > > > 47. However, we haven't historically followed the EOL timelines for > > > > PIP 47 strictly, so I want to check here. > > > > > > > > For 2.8, I think we should declare it EOL or perform once last release. > > > > > > I agree > > > > > > > > > > > For 2.9, I think we could do the same as 2.8, but I am not sure if > > > > that would surprise users. > > > > > > I am not sure, but as far as I know, 2.9 was kind of a transitional > > > release, and I agree > > > > > > > > > > > For 2.10, I think we should not consider it EOL. > > > > > > This is the latest release that runs on JDK11, we really cannot drop > > > support for this. > > > It would be worth declaring this LTS, as long as JDK11 is still widely > > > used. > > > > > > This issue was discussed many times when we decided to move to JDK17 on > > > 2.11. > > > > > > Cheers > > > Enrico > > > > > > > > > > > > > > Let me know what you think. > > > > > > > > (Reminder: all feedback is welcome, especially from users!) > > > > > > > > Thanks! > > > > Michael
Re: [DISCUSS] which versions are EOL?
In light of this thread, I think we have a (lazy) consensus that 2.8 and 2.9 are EOL and we will continue to maintain 2.10. Would someone be able to update the website to say that 2.10 is not EOL? I will update the GitHub labels to indicate the 2.9 label should no longer be used. Thanks, Michael On Fri, Jun 9, 2023 at 4:33 AM Yunze Xu wrote: > > > This is the latest release that runs on JDK11 > > This point makes sense to me. Actually I think it's the latest release > that runs on JDK 8 though the recommended JDK for 2.10 is 11. The only > question from me is when should 2.10 be EOL? > > Thanks, > Yunze > > On Fri, Jun 9, 2023 at 3:15 PM Enrico Olivelli wrote: > > > > Micheal, > > > > Il giorno ven 9 giu 2023 alle ore 07:54 Michael Marshall > > ha scritto: > > > > > > Hi Pulsar Community, > > > > > > I recently noticed this page on our website: > > > https://pulsar.apache.org/contribute/release-policy/#supported-versions > > > > > > It shows that only 2.11 and 3.0 are active and security support. > > > > > > I am guessing the timelines were taken from a strict reading of PIP > > > 47. However, we haven't historically followed the EOL timelines for > > > PIP 47 strictly, so I want to check here. > > > > > > For 2.8, I think we should declare it EOL or perform once last release. > > > > I agree > > > > > > > > For 2.9, I think we could do the same as 2.8, but I am not sure if > > > that would surprise users. > > > > I am not sure, but as far as I know, 2.9 was kind of a transitional > > release, and I agree > > > > > > > > For 2.10, I think we should not consider it EOL. > > > > This is the latest release that runs on JDK11, we really cannot drop > > support for this. > > It would be worth declaring this LTS, as long as JDK11 is still widely used. > > > > This issue was discussed many times when we decided to move to JDK17 on > > 2.11. > > > > Cheers > > Enrico > > > > > > > > > > Let me know what you think. > > > > > > (Reminder: all feedback is welcome, especially from users!) > > > > > > Thanks! > > > Michael
Re: Can we archive https://github.com/apache/pulsar-release?
While, I visit tests/integration recently that it can be clear how to factor out the testcontainers-based integration utils. Let me file an issue to track the work so that we have more visibility about the progress and hopefully a nudge to the target - of course separating those code is a good idea. Back to apache/pulsar-release, it seems the separation of connectors is part of the prerequisite. Best, tison. tison 于2023年6月28日周三 04:15写道： > > Pulsar SQL > > objections > > It's not objections the blocker, but volunteers. > > A comparable example is that Flink firstly tried to move out connectors in > 2018 and finally made it happen in 2022 (specifically, Kafka connector > _today_[1]). That indicates such initiatives require heavy development. > > For Pulsar SQL separation, there are two major works I don't find time to > finish - > > 1. Factor out integration utils to shared libs and release them. Flink > finally has flink-connector-shared-utils[2] > and flink-test-utils-junit, flink-connector-test-utils, > flink-architecture-tests-test, and so on. > 2. Work out a solution to handle Docker image build and release after > Pulsar SQL moved out. > > Best, > tison. > > [1] https://lists.apache.org/thread/vt4746r8kbnmcq7llcsvyjk4gor7gops > [2] https://github.com/apache/flink-connector-shared-utils/branches > > > Lari Hotari 于2023年6月28日周三 01:12写道： > >> I hope we could finally resume the work on PIP-62 and continue with >> the plan that was set in April 2020. >> You can find old discussion threads about PIP-62 with this query: >> https://lists.apache.org/list?dev@pulsar.apache.org:lte=10y:PIP-62 >> >> There was a discussion in July 2022 about moving Pulsar SQL to a >> separate repository. [1] >> Could we finally make this happen? Any objections? >> >> -Lari >> >> [1] https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8 >> >> On Tue, Jun 27, 2023 at 1:35 PM Enrico Olivelli >> wrote: >> > >> > Yes we can. >> > >> > It was a tentative goal of creating a new workflow to build releases, >> > especially for the pulsar-all docker image, that depends both on >> > pulsar main repo code and on Pulsar SQL, >> > that we wanted to remove from the base repository. >> > >> > Enrico >> > >> > Il giorno mar 27 giu 2023 alle ore 08:27 Asaf Mesika >> > ha scritto: >> > > >> > > Seems quite neglected >> >
Re: Can we archive https://github.com/apache/pulsar-release?
> Pulsar SQL > objections It's not objections the blocker, but volunteers. A comparable example is that Flink firstly tried to move out connectors in 2018 and finally made it happen in 2022 (specifically, Kafka connector _today_[1]). That indicates such initiatives require heavy development. For Pulsar SQL separation, there are two major works I don't find time to finish - 1. Factor out integration utils to shared libs and release them. Flink finally has flink-connector-shared-utils[2] and flink-test-utils-junit, flink-connector-test-utils, flink-architecture-tests-test, and so on. 2. Work out a solution to handle Docker image build and release after Pulsar SQL moved out. Best, tison. [1] https://lists.apache.org/thread/vt4746r8kbnmcq7llcsvyjk4gor7gops [2] https://github.com/apache/flink-connector-shared-utils/branches Lari Hotari 于2023年6月28日周三 01:12写道： > I hope we could finally resume the work on PIP-62 and continue with > the plan that was set in April 2020. > You can find old discussion threads about PIP-62 with this query: > https://lists.apache.org/list?dev@pulsar.apache.org:lte=10y:PIP-62 > > There was a discussion in July 2022 about moving Pulsar SQL to a > separate repository. [1] > Could we finally make this happen? Any objections? > > -Lari > > [1] https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8 > > On Tue, Jun 27, 2023 at 1:35 PM Enrico Olivelli > wrote: > > > > Yes we can. > > > > It was a tentative goal of creating a new workflow to build releases, > > especially for the pulsar-all docker image, that depends both on > > pulsar main repo code and on Pulsar SQL, > > that we wanted to remove from the base repository. > > > > Enrico > > > > Il giorno mar 27 giu 2023 alle ore 08:27 Asaf Mesika > > ha scritto: > > > > > > Seems quite neglected >
Re: Can we archive https://github.com/apache/pulsar-release?
I hope we could finally resume the work on PIP-62 and continue with the plan that was set in April 2020. You can find old discussion threads about PIP-62 with this query: https://lists.apache.org/list?dev@pulsar.apache.org:lte=10y:PIP-62 There was a discussion in July 2022 about moving Pulsar SQL to a separate repository. [1] Could we finally make this happen? Any objections? -Lari [1] https://lists.apache.org/thread/mflm0pb5235jjk80vol0vs7v0hvowkq8 On Tue, Jun 27, 2023 at 1:35 PM Enrico Olivelli wrote: > > Yes we can. > > It was a tentative goal of creating a new workflow to build releases, > especially for the pulsar-all docker image, that depends both on > pulsar main repo code and on Pulsar SQL, > that we wanted to remove from the base repository. > > Enrico > > Il giorno mar 27 giu 2023 alle ore 08:27 Asaf Mesika > ha scritto: > > > > Seems quite neglected
Re: Pulsar doesn't support cgroup v2 which became the default in Kubernetes v1.25+ (AKS v1.25, GKE v1.26, EKS v1.26)
Il giorno mar 27 giu 2023 alle ore 12:58 Lari Hotari ha scritto: > > I take my words back about branch-2.9 . It's also using Java 11 at runtime, > so it should be possible to apply https://github.com/apache/pulsar/pull/20659 > to branch-2.9 too so that we get cgroup v2 support for Pulsar 2.9.x. > > -Lari > > On 2023/06/27 10:52:02 Lari Hotari wrote: > > We don't have any released stable version of Pulsar with cgroup v2 support. > > > > Azure AKS Kubernetes 1.25+ switches to use cgroup v2 [1] and AKS Kubernetes > > 1.24 goes End-of-life on July 30, 2023 [2]. This is why this is becoming > > urgent. > > > > GKE will continue to have a way to select between cgroup v1 & cgroup v2 > > [3]. However even GKE will default to cgroup v2 in new Kubernetes 1.26 > > clusters or node pools. > > AWS EKS v1.26 nodes will default to cgroup v2. > > > > I have backported https://github.com/apache/pulsar/pull/16832 to > > branch-2.10 in this PR: > > https://github.com/apache/pulsar/pull/20659 > > It requires including all refactorings for the Linux CPU, network and > > memory metrics. > > > > I'll proceed with cherry-picking > > https://github.com/apache/pulsar/pull/16832 to branch-3.0 and branch-2.11 . > > I don't think that supporting branch-2.9 is feasible since cgroup v2 > > support requires Java 11.0.16+. Thank you very much I agree that this is very important for k8s users Enrico > > > > -Lari > > > > [1] https://github.com/Azure/AKS/releases/tag/2023-03-05 > > [2] > > https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar > > [3] > > https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config#cgroup-mode-options > > > > On 2023/05/04 00:45:17 Frank Kelly wrote: > > > This sounds like a very important issue for those of us seeking to use > > > autoscaling - will the fix be back-ported to 2.11/2.10/2.9 etc? > > > Alternatively is there a work-around? > > > > > > -Frank > > > > > > On Thu, Apr 27, 2023 at 2:37 AM Lari Hotari wrote: > > > > > > > Thank you, Cong. That will be very helpful. > > > > > > > > -Lari > > > > > > > > On 2023/04/27 04:55:24 Cong Zhao wrote: > > > > > Hi Lari Hotar, > > > > > > > > > > I would like to pick up this work, I will update > > > > https://github.com/apache/pulsar/pull/16832 as soon. > > > > > > > > > > Thanks, > > > > > Cong Zhao > > > > > > > > > > On 2023/04/26 15:17:37 Lari Hotari wrote: > > > > > > Hi all, > > > > > > > > > > > > Pulsar doesn't support cgroup v2 which becomes default in Kubernetes > > > > v1.25+. > > > > > > Kubernetes announcement: > > > > > > https://kubernetes.io/blog/2022/08/31/cgroupv2-ga-1-25/ . > > > > > > Pulsar issue: https://github.com/apache/pulsar/issues/16601 > > > > > > > > > > > > The impact of this is that the Pulsar load balancer won't have > > > > > > correct > > > > > > CPU and memory information for making load balancing decisions. > > > > > > > > > > > > The cloud provider managed Kubernetes services have already switched > > > > > > to cgroup v2 as the default. This happened in AKS v1.25, GKE v1.26 > > > > > > and > > > > > > in EKS v1.26. > > > > > > For GKE, it's possible to keep using cgroup v1 also in GKE v1.26 > > > > > > ( > > > > https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config#cgroup-mode-options > > > > ). > > > > > > For AKS and EKS, it's unknown whether such a configuration option > > > > > > exists. > > > > > > > > > > > > There's a previous attempt in this PR to add cgroup v2 support to > > > > > > Pulsar: https://github.com/apache/pulsar/pull/16832 . Would it be > > > > > > possible to continue the work for supporting cgroup v2 in Pulsar > > > > > > either with the existing PR or a new one? > > > > > > > > > > > > Who would like to pick up this work? > > > > > > This is urgent since cgroup v2 is enabled by default for all latest > > > > > > managed Kubernetes services (AKS v1.25, GKE v1.26 and EKS v1.26). > > > > > > > > > > > > Regards, > > > > > > > > > > > > -Lari > > > > > > > > > > > > > > > > > > > >
Re: Pulsar doesn't support cgroup v2 which became the default in Kubernetes v1.25+ (AKS v1.25, GKE v1.26, EKS v1.26)
I take my words back about branch-2.9 . It's also using Java 11 at runtime, so it should be possible to apply https://github.com/apache/pulsar/pull/20659 to branch-2.9 too so that we get cgroup v2 support for Pulsar 2.9.x. -Lari On 2023/06/27 10:52:02 Lari Hotari wrote: > We don't have any released stable version of Pulsar with cgroup v2 support. > > Azure AKS Kubernetes 1.25+ switches to use cgroup v2 [1] and AKS Kubernetes > 1.24 goes End-of-life on July 30, 2023 [2]. This is why this is becoming > urgent. > > GKE will continue to have a way to select between cgroup v1 & cgroup v2 [3]. > However even GKE will default to cgroup v2 in new Kubernetes 1.26 clusters or > node pools. > AWS EKS v1.26 nodes will default to cgroup v2. > > I have backported https://github.com/apache/pulsar/pull/16832 to branch-2.10 > in this PR: > https://github.com/apache/pulsar/pull/20659 > It requires including all refactorings for the Linux CPU, network and memory > metrics. > > I'll proceed with cherry-picking https://github.com/apache/pulsar/pull/16832 > to branch-3.0 and branch-2.11 . I don't think that supporting branch-2.9 is > feasible since cgroup v2 support requires Java 11.0.16+. > > -Lari > > [1] https://github.com/Azure/AKS/releases/tag/2023-03-05 > [2] > https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar > [3] > https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config#cgroup-mode-options > > On 2023/05/04 00:45:17 Frank Kelly wrote: > > This sounds like a very important issue for those of us seeking to use > > autoscaling - will the fix be back-ported to 2.11/2.10/2.9 etc? > > Alternatively is there a work-around? > > > > -Frank > > > > On Thu, Apr 27, 2023 at 2:37 AM Lari Hotari wrote: > > > > > Thank you, Cong. That will be very helpful. > > > > > > -Lari > > > > > > On 2023/04/27 04:55:24 Cong Zhao wrote: > > > > Hi Lari Hotar, > > > > > > > > I would like to pick up this work, I will update > > > https://github.com/apache/pulsar/pull/16832 as soon. > > > > > > > > Thanks, > > > > Cong Zhao > > > > > > > > On 2023/04/26 15:17:37 Lari Hotari wrote: > > > > > Hi all, > > > > > > > > > > Pulsar doesn't support cgroup v2 which becomes default in Kubernetes > > > v1.25+. > > > > > Kubernetes announcement: > > > > > https://kubernetes.io/blog/2022/08/31/cgroupv2-ga-1-25/ . > > > > > Pulsar issue: https://github.com/apache/pulsar/issues/16601 > > > > > > > > > > The impact of this is that the Pulsar load balancer won't have correct > > > > > CPU and memory information for making load balancing decisions. > > > > > > > > > > The cloud provider managed Kubernetes services have already switched > > > > > to cgroup v2 as the default. This happened in AKS v1.25, GKE v1.26 and > > > > > in EKS v1.26. > > > > > For GKE, it's possible to keep using cgroup v1 also in GKE v1.26 > > > > > ( > > > https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config#cgroup-mode-options > > > ). > > > > > For AKS and EKS, it's unknown whether such a configuration option > > > > > exists. > > > > > > > > > > There's a previous attempt in this PR to add cgroup v2 support to > > > > > Pulsar: https://github.com/apache/pulsar/pull/16832 . Would it be > > > > > possible to continue the work for supporting cgroup v2 in Pulsar > > > > > either with the existing PR or a new one? > > > > > > > > > > Who would like to pick up this work? > > > > > This is urgent since cgroup v2 is enabled by default for all latest > > > > > managed Kubernetes services (AKS v1.25, GKE v1.26 and EKS v1.26). > > > > > > > > > > Regards, > > > > > > > > > > -Lari > > > > > > > > > > > > > > >
Re: Pulsar doesn't support cgroup v2 which became the default in Kubernetes v1.25+ (AKS v1.25, GKE v1.26, EKS v1.26)
We don't have any released stable version of Pulsar with cgroup v2 support. Azure AKS Kubernetes 1.25+ switches to use cgroup v2 [1] and AKS Kubernetes 1.24 goes End-of-life on July 30, 2023 [2]. This is why this is becoming urgent. GKE will continue to have a way to select between cgroup v1 & cgroup v2 [3]. However even GKE will default to cgroup v2 in new Kubernetes 1.26 clusters or node pools. AWS EKS v1.26 nodes will default to cgroup v2. I have backported https://github.com/apache/pulsar/pull/16832 to branch-2.10 in this PR: https://github.com/apache/pulsar/pull/20659 It requires including all refactorings for the Linux CPU, network and memory metrics. I'll proceed with cherry-picking https://github.com/apache/pulsar/pull/16832 to branch-3.0 and branch-2.11 . I don't think that supporting branch-2.9 is feasible since cgroup v2 support requires Java 11.0.16+. -Lari [1] https://github.com/Azure/AKS/releases/tag/2023-03-05 [2] https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar [3] https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config#cgroup-mode-options On 2023/05/04 00:45:17 Frank Kelly wrote: > This sounds like a very important issue for those of us seeking to use > autoscaling - will the fix be back-ported to 2.11/2.10/2.9 etc? > Alternatively is there a work-around? > > -Frank > > On Thu, Apr 27, 2023 at 2:37 AM Lari Hotari wrote: > > > Thank you, Cong. That will be very helpful. > > > > -Lari > > > > On 2023/04/27 04:55:24 Cong Zhao wrote: > > > Hi Lari Hotar, > > > > > > I would like to pick up this work, I will update > > https://github.com/apache/pulsar/pull/16832 as soon. > > > > > > Thanks, > > > Cong Zhao > > > > > > On 2023/04/26 15:17:37 Lari Hotari wrote: > > > > Hi all, > > > > > > > > Pulsar doesn't support cgroup v2 which becomes default in Kubernetes > > v1.25+. > > > > Kubernetes announcement: > > > > https://kubernetes.io/blog/2022/08/31/cgroupv2-ga-1-25/ . > > > > Pulsar issue: https://github.com/apache/pulsar/issues/16601 > > > > > > > > The impact of this is that the Pulsar load balancer won't have correct > > > > CPU and memory information for making load balancing decisions. > > > > > > > > The cloud provider managed Kubernetes services have already switched > > > > to cgroup v2 as the default. This happened in AKS v1.25, GKE v1.26 and > > > > in EKS v1.26. > > > > For GKE, it's possible to keep using cgroup v1 also in GKE v1.26 > > > > ( > > https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config#cgroup-mode-options > > ). > > > > For AKS and EKS, it's unknown whether such a configuration option > > > > exists. > > > > > > > > There's a previous attempt in this PR to add cgroup v2 support to > > > > Pulsar: https://github.com/apache/pulsar/pull/16832 . Would it be > > > > possible to continue the work for supporting cgroup v2 in Pulsar > > > > either with the existing PR or a new one? > > > > > > > > Who would like to pick up this work? > > > > This is urgent since cgroup v2 is enabled by default for all latest > > > > managed Kubernetes services (AKS v1.25, GKE v1.26 and EKS v1.26). > > > > > > > > Regards, > > > > > > > > -Lari > > > > > > > > > >
Re: Can we archive https://github.com/apache/pulsar-release?
Yes we can. It was a tentative goal of creating a new workflow to build releases, especially for the pulsar-all docker image, that depends both on pulsar main repo code and on Pulsar SQL, that we wanted to remove from the base repository. Enrico Il giorno mar 27 giu 2023 alle ore 08:27 Asaf Mesika ha scritto: > > Seems quite neglected
Re: [DISCUSS] PIP-277: Clusters list API return clusters with local flag
Thanks. addressed in the PIP Regards Jiwei Guo (Tboy) On Sun, Jun 25, 2023 at 6:05 PM PengHui Li wrote: > Hi Jiwei, > > Do you need to change the REST API? or will it just apply to the Admin CLI? > If it requires REST API changes, you should also reflect them in the > proposal. > > And yes, using the local and remote will be confusing because users can > have > multiple clusters without geo-replication. Maybe we should use the > --current option > to show which cluster are requesting. So that we will not introduce any > break changes. > If we don't have compatibility issue, I don't like the --current option :) > > With the --current option: > > ``` > bin/pulsar-admin clusters list --current > > us-west(*) > us-east > us-cent > ``` > > It looks like (*) is commonly used, e.g. kubectl config get-contexts ... > > Regards, > Penghui > > On Sun, Jun 25, 2023 at 3:53 PM Zixuan Liu wrote: > > > +1, this is a good improvement. > > > > In the proposal, I don't suggest you append the `local` to the cluster > > name: > > > > 1. It will be confusing when the original cluster name includes the > > `local` name > > 2. In some cases, it is not easy to distinguish between local and > > remote clusters > > > > so I suggest we should add the `local` and `remote` flags like: > > > > ``` > > # list remote clusters > > bin/pulsar-admin clusters list --remote > > us-east > > us-cent > > > > # list local clustes > > bin/pulsar-admin clusters list --local > > us-west > > > > # list all clusters > > bin/pulsar-admin clusters list > > us-east > > us-cent > > us-west > > ``` > > > > Thanks, > > Zixuan > > > > guo jiwei 于2023年6月20日周二 14:00写道： > > > > > > Hi community: > > > After configuring the geo-replication on Pulsar clusters, the > > `clusters > > > list` API will return multiple clusters, including the local Pulsar > > cluster > > > and remote clusters like > > > > > > ``` > > > bin/pulsar-admin clusters list > > > us-west > > > us-east > > > us-cent > > > ``` > > > But in this return, you can't distinguish the local and the remote > > cluster. > > > When you need to remove the geo-replication configuration, it will be > > hard > > > to decide which cluster should be removed on replicated tenants and > > > namespaces unless you record the cluster information. > > > > > > > > > ### Modification > > > Add `local` flag to distinguish clusters > > > ``` > > > bin/pulsar-admin clusters list > > > us-west(local) > > > us-east > > > us-cent > > > ``` > > > > > > PIP: https://github.com/apache/pulsar/pull/20614 > > > > > > > > > Regards > > > Jiwei Guo (Tboy) > > >
Can we archive https://github.com/apache/pulsar-release?
Seems quite neglected
