Re: Pulsar CI broken due to Eclipse Temurin repository broken

[Lari Hotari]

There's another problem related to the Eclipse Temurin repository.  
Sometimes the Pulsar docker image building fails with an error "Error reading 
from server. Remote end closed connection". I reported the issue to 
adoptium/installer project issues as a comment on an existing issue:
https://github.com/adoptium/installer/issues/754#issuecomment-1785810245

-Lari

On 2023/10/28 16:44:52 Lari Hotari wrote:
> The problem has been resolved in the Eclipse Temurin apt repository. Pulsar
> CI is unblocked.
> 
> -Lari
> 
> On Fri, 27 Oct 2023 at 09:00, Lari Hotari  wrote:
> 
> > Hi all,
> >
> > Pulsar CI is currently broken and fails in building docker images used in
> > end-to-end Integration tests. This fails due to Eclipse Temurin apt
> > repository being broken. Enrico shared the link to the Temurin issue [1] on
> > Pulsar slack [2].
> > Base on the latest issue comments, it seems that the repository issue is
> > fixed but the old state is still held in caches and the problem isn't yet
> > resolved. If that's the case, I would expect that the cache expiration is
> > less than 24h. Therefore we could wait until it is resolved.
> >
> > -Lari
> >
> > 1 - https://github.com/adoptium/installer/issues/766
> > 2 - https://apache-pulsar.slack.com/archives/C5ZSVEN4E/p1698413110189289
> >
> 

Call for Presentations now open: Community over Code EU 2024

[Ryan Skraba]

(Note: You are receiving this because you are subscribed to the dev@
list for one or more projects of the Apache Software Foundation.)

It's back *and* it's new!

We're excited to announce that the first edition of Community over
Code Europe (formerly known as ApacheCon EU) which will be held at the
Radisson Blu Carlton Hotel in Bratislava, Slovakia from June 03-05,
2024! This eagerly anticipated event will be our first live EU
conference since 2019.

The Call for Presentations (CFP) for Community Over Code EU 2024 is
now open at https://eu.communityovercode.org/blog/cfp-open/,
and will close 2024/01/12 23:59:59 GMT.

We welcome submissions on any topic related to the Apache Software
Foundation, Apache projects, or the communities around those projects.
We are specifically looking for presentations in the following
categories:

* API & Microservices
* Big Data Compute
* Big Data Storage
* Cassandra
* CloudStack
* Community
* Data Engineering
* Fintech
* Groovy
* Incubator
* IoT
* Performance Engineering
* Search
* Tomcat, Httpd and other servers

Additionally, we are thrilled to introduce a new feature this year: a
poster session. This addition will provide an excellent platform for
showcasing high-level projects and incubator initiatives in a visually
engaging manner. We believe this will foster lively discussions and
facilitate networking opportunities among participants.

All my best, and thanks so much for your participation,

Ryan Skraba (on behalf of the program committee)

[Countdown]: https://www.timeanddate.com/countdown/to?iso=20240112T2359=1440

Re: Reporting and tooling to detect thread leaks in Pulsar tests

[Asaf Mesika]

Ok, got it. While scrolling down on the main page, and cursor is on the
graph, nothing happens. When you move the mouse away from the graph, you
scroll and then you see those annotations.
Very nice additions and good to generally know about them.


On Mon, Oct 30, 2023 at 10:53 AM Lari Hotari  wrote:

> Thanks for the review. I merged the PR and triggered a manual build
> https://github.com/apache/pulsar/actions/runs/6690374946 to get the latest
> report of leaked threads.
>
> -Lari
>
> On Mon, 30 Oct 2023, 9.15 Enrico Olivelli,  wrote:
>
> > Il Lun 30 Ott 2023, 06:39 Lari Hotari  ha scritto:
> >
> > > Hi Asaf,
> > >
> > > Yes, the visibility aspect is already solved by using warnings in the
> > > summary view. Please check the example
> > > https://github.com/apache/pulsar/actions/runs/6680066364?pr=21450 .
> > >
> > > Job summaries could also be used, but they have less visibility in the
> > > summary view, as you can see from the example. Job summaries are on
> > placed
> > > on the summary page after errors/warnings and build artifacts and when
> > > there are more than a few summaries, each job summary will need to be
> > > explicitly expanded by clicking "Load Summary" to view the content.
> That
> > > makes their visibility lower than warnings.
> > >
> > > Since this is a change in the build and isn't really intrusive, I think
> > we
> > > could get it merged and revisit it based on the experiences we get from
> > the
> > > use of it. I have been iterating on the solution while fixing a lot of
> > the
> > > test resource leaks in the last few weeks. Without support for
> detecting
> > > the resource leaks, it's really hard to keep the test suite clean.
> > >
> >
> >
> >
> > > Looking forward to more reviews on
> > > https://github.com/apache/pulsar/pull/21450 . :)
> >
> >
> >
> >
> > Looks great
> >
> > Thanks
> > Enrico
> >
> > >
> > >
> > > -Lari
> > >
> > > On 2023/10/29 18:34:28 Asaf Mesika wrote:
> > > > Larry, I know there is a way to add like a Job summary, so we can
> write
> > > it
> > > > there - do you think this can increase visibility?
> > > >
> > > > On Sun, Oct 29, 2023 at 4:53 AM Lari Hotari 
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have submitted a PR (https://github.com/apache/pulsar/pull/21450
> )
> > > which
> > > > > includes changes to add reporting and tooling to detect thread
> leaks
> > in
> > > > > Pulsar tests.
> > > > >
> > > > > It should be ensured in each test that resources created by the
> test
> > > are
> > > > > properly cleaned up. Failing to do so can lead to memory leaks and,
> > in
> > > some
> > > > > instances, unnecessary CPU consumption. These issues can, in turn,
> > slow
> > > > > down test execution, increase Pulsar CI build durations, and cause
> > > > > flakiness.  A significant source of memory leaks in Pulsar tests
> > stems
> > > from
> > > > > thread leaks.
> > > > >
> > > > > After the PR is merged, it will be easy to detect thread leaks
> since
> > > the
> > > > > build will add warnings to the summary view for the GitHub Actions
> > > build
> > > > > run. An example can be seen in the PR build run:
> > > > > https://github.com/apache/pulsar/actions/runs/6680066364?pr=21450
> .
> > > > > There will be more detailed information in the "Report detected
> > thread
> > > > > leaks" build step, for example
> > > > >
> > >
> >
> https://github.com/apache/pulsar/actions/runs/6680066364/job/18153890519?pr=21450#step:16:23
> > > > > .
> > > > >
> > > > > Please review the PR https://github.com/apache/pulsar/pull/21450
> so
> > > that
> > > > > we can continue to get rid of the remaining thread leaks in the
> > future
> > > and
> > > > > keep the tests cleaner and less flaky.
> > > > >
> > > > > -Lari
> > > > >
> > > >
> > >
> >
>

RE: Re: [DISCUSS] Auto release Nuget artifacts for DotPulsar

[David Jensen]

 Hi Enrico Olivell,

When you need to release a package, it is important to follow the release
manual. Here is a summary of the steps you need to take:

1. Update the version and tag of a package.
2. Create a GitHub prerelease.
3. Download the source code.
4. Sign and stage the artifacts.
5. Run a vote.
6. Summary of vote.
7. Promote the release.
8. Create a GitHub release.
9. Announce the release.

Step 5 is the voting phase, where the release is approved or rejected.

You can find the full release manual by checking out the PR mentioned in
the first email.

Sincerely,
David Jensen

On 2023/10/30 07:08:20 Enrico Olivelli wrote:
> How do we handle the VOTE?
>
> Enrico
>
> Il Lun 30 Ott 2023, 08:06 Zike Yang  ha scritto:
>
> > Hi, David
> >
> > Thanks for initializing this discussion.
> > I'm +1 for making the auto-release process for the DotPulsar. I have
> > left some comments in the PR.
> >
> > Thanks,
> > Zike Yang
> >
> > On Thu, Oct 26, 2023 at 5:39 PM David Jensen  wrote:
> > >
> > > Hi everyone,
> > > I have worked with Tison to make the release process smoother for
> > DotPulsar
> > > . See GitHub Issue #184
> > Release
> > > process and GitHub Action
> > >  and the linked
> > issues.
> > >
> > > In summary, the document presents a suggested approach for creating
> > release
> > > notes and optimizing the release workflow of the Pulsar C# Client.
This
> > > approach involves working closely with team members, adhering to ASF
> > > guidelines, participating in Apache release voting, implementing
> > automation
> > > via GitHub Actions, and managing versioning through git tags and
GitHub
> > > releases.
> > >
> > > Sincerely
> > > David Jensen
> >
>

RE: Re: [DISCUSS] Auto release Nuget artifacts for DotPulsar

[David Jensen]

Thanks for the feedback. I have to respond to your question on GitHub. :)

On 2023/10/30 07:05:46 Zike Yang wrote:
> Hi, David
>
> Thanks for initializing this discussion.
> I'm +1 for making the auto-release process for the DotPulsar. I have
> left some comments in the PR.
>
> Thanks,
> Zike Yang
>
> On Thu, Oct 26, 2023 at 5:39 PM David Jensen  wrote:
> >
> > Hi everyone,
> > I have worked with Tison to make the release process smoother for
DotPulsar
> > . See GitHub Issue #184
Release
> > process and GitHub Action
> >  and the linked
issues.
> >
> > In summary, the document presents a suggested approach for creating
release
> > notes and optimizing the release workflow of the Pulsar C# Client. This
> > approach involves working closely with team members, adhering to ASF
> > guidelines, participating in Apache release voting, implementing
automation
> > via GitHub Actions, and managing versioning through git tags and GitHub
> > releases.
> >
> > Sincerely
> > David Jensen
>

Re: Reporting and tooling to detect thread leaks in Pulsar tests

[Lari Hotari]

Thanks for the review. I merged the PR and triggered a manual build
https://github.com/apache/pulsar/actions/runs/6690374946 to get the latest
report of leaked threads.

-Lari

On Mon, 30 Oct 2023, 9.15 Enrico Olivelli,  wrote:

> Il Lun 30 Ott 2023, 06:39 Lari Hotari  ha scritto:
>
> > Hi Asaf,
> >
> > Yes, the visibility aspect is already solved by using warnings in the
> > summary view. Please check the example
> > https://github.com/apache/pulsar/actions/runs/6680066364?pr=21450 .
> >
> > Job summaries could also be used, but they have less visibility in the
> > summary view, as you can see from the example. Job summaries are on
> placed
> > on the summary page after errors/warnings and build artifacts and when
> > there are more than a few summaries, each job summary will need to be
> > explicitly expanded by clicking "Load Summary" to view the content. That
> > makes their visibility lower than warnings.
> >
> > Since this is a change in the build and isn't really intrusive, I think
> we
> > could get it merged and revisit it based on the experiences we get from
> the
> > use of it. I have been iterating on the solution while fixing a lot of
> the
> > test resource leaks in the last few weeks. Without support for detecting
> > the resource leaks, it's really hard to keep the test suite clean.
> >
>
>
>
> > Looking forward to more reviews on
> > https://github.com/apache/pulsar/pull/21450 . :)
>
>
>
>
> Looks great
>
> Thanks
> Enrico
>
> >
> >
> > -Lari
> >
> > On 2023/10/29 18:34:28 Asaf Mesika wrote:
> > > Larry, I know there is a way to add like a Job summary, so we can write
> > it
> > > there - do you think this can increase visibility?
> > >
> > > On Sun, Oct 29, 2023 at 4:53 AM Lari Hotari 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have submitted a PR (https://github.com/apache/pulsar/pull/21450)
> > which
> > > > includes changes to add reporting and tooling to detect thread leaks
> in
> > > > Pulsar tests.
> > > >
> > > > It should be ensured in each test that resources created by the test
> > are
> > > > properly cleaned up. Failing to do so can lead to memory leaks and,
> in
> > some
> > > > instances, unnecessary CPU consumption. These issues can, in turn,
> slow
> > > > down test execution, increase Pulsar CI build durations, and cause
> > > > flakiness.  A significant source of memory leaks in Pulsar tests
> stems
> > from
> > > > thread leaks.
> > > >
> > > > After the PR is merged, it will be easy to detect thread leaks since
> > the
> > > > build will add warnings to the summary view for the GitHub Actions
> > build
> > > > run. An example can be seen in the PR build run:
> > > > https://github.com/apache/pulsar/actions/runs/6680066364?pr=21450 .
> > > > There will be more detailed information in the "Report detected
> thread
> > > > leaks" build step, for example
> > > >
> >
> https://github.com/apache/pulsar/actions/runs/6680066364/job/18153890519?pr=21450#step:16:23
> > > > .
> > > >
> > > > Please review the PR https://github.com/apache/pulsar/pull/21450 so
> > that
> > > > we can continue to get rid of the remaining thread leaks in the
> future
> > and
> > > > keep the tests cleaner and less flaky.
> > > >
> > > > -Lari
> > > >
> > >
> >
>

[CALL FOR CONTRIBUTE] The broker feature matrix page

[tison]

Hi,

As I proposed in my previous email[1], it's a heat path that our users want
to know what features Pulsar provides and at which version they are
experimental, beta, or stable.

* Experimental Since indicates a version; since then, the feature has been
experimental. Experimental means it is not meant to be used in production
yet.
* Beta Since indicates a version; since then, the feature has been beta.
Beta means you can use it in production, but it still needs battle-testing.
* Stable Since indicates a version; since then, the feature has been
stable. Stable means it is battle-tested. You can use it in production with
full confidence.

I made a pull request to initialize the feature matrix page with the
proposed structure and attach a preview in the PR thread[2].

Sorting the features we deliver is a good chance to review what we have
delivered in the past few years and how much they are away from a
product-ready, high-quality functionality. Also, only if we know what
features we have, can we start a discussion about the following vision and
make a more user-friendly explanation like what Flink does[3]

I'm here to call for contributions from all of the community members for
commenting the features you know, use, or contribute to the PR linked
above[2], with the missing information for when they're experimental, beta,
or stable, as well as any documents you ever read or referred to of them.

Best,
tison.

[1] https://lists.apache.org/thread/rrcn3ncqhl5vmm56wgygl921hybgmfgq
[2] https://github.com/apache/pulsar-site/pull/720
[3] https://flink.apache.org/roadmap/

Re: [VOTE] Pulsar Client Go Release 0.11.1 Candidate 1

[Max Xu]

The KEYS file link is dead, I think it should be
https://downloads.apache.org/pulsar/KEYS


On Mon, Oct 30, 2023 at 10:02 AM PengHui Li  wrote:

> +1 (binding)
>
> - Verified the signatures
> - Build from the source
> - Run the test on Pulsar 3.1.1 (make test PULSAR_VERSION=3.1.1)
> - Run the test on Pulsar 3.0.1 (make test PULSAR_VERSION=3.0.1)
>
> Regards,
> Penghui
>
> On Fri, Oct 20, 2023 at 4:46 PM Yunze Xu  wrote:
>
> > +1 (binding)
> >
> > - Verified checksum and signatures
> > - Build the perf tool from source
> > - Run the perf to produce and consumer for some time
> >
> > Thanks,
> > Yunze
> >
> > On Mon, Sep 11, 2023 at 6:08 PM Zike Yang  wrote:
> > >
> > > Hi everyone,
> > > Please review and vote on the release candidate #1 for the version
> > > 0.11.1, as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > This is the first release candidate for Apache Pulsar Go client,
> version
> > 0.11.1.
> > >
> > > It fixes the following issues:
> > >
> >
> https://github.com/apache/pulsar-client-go/compare/v0.11.0...v0.11.1-candidate-1
> > >
> > > Pulsar Client Go's KEYS file contains PGP keys we used to sign this
> > release:
> > > https://dist.apache.org/repos/dist/dev/pulsar/KEYS
> > >
> > > Please download these packages and review this release candidate:
> > > - Review release notes:
> > https://github.com/apache/pulsar-client-go/pull/1092
> > > - Download the source package (verify shasum, and asc) and follow the
> > > README.md to build and run the pulsar-client-go.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Source file:
> > >
> >
> https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-go-0.11.1-candidate-1/
> > >
> > > The tag to be voted upon:
> > > v0.11.1-candidate-1
> > > https://github.com/apache/pulsar-client-go/tree/v0.11.1-candidate-1
> > >
> > > SHA-512 checksums:
> > >
> >
> d2209c652918acee8d2c77d52a0a556af16ff7fc3e30ad96d05e01285b83a61d1a1f0d32bace184f830e2dd2e4dd20910e9ce5ae23aac4a40eb3d19885cb0182
> > >  apache-pulsar-client-go-0.11.1-src.tar.gz
> >
>

Re: Reporting and tooling to detect thread leaks in Pulsar tests

[Enrico Olivelli]

Il Lun 30 Ott 2023, 06:39 Lari Hotari  ha scritto:

> Hi Asaf,
>
> Yes, the visibility aspect is already solved by using warnings in the
> summary view. Please check the example
> https://github.com/apache/pulsar/actions/runs/6680066364?pr=21450 .
>
> Job summaries could also be used, but they have less visibility in the
> summary view, as you can see from the example. Job summaries are on placed
> on the summary page after errors/warnings and build artifacts and when
> there are more than a few summaries, each job summary will need to be
> explicitly expanded by clicking "Load Summary" to view the content. That
> makes their visibility lower than warnings.
>
> Since this is a change in the build and isn't really intrusive, I think we
> could get it merged and revisit it based on the experiences we get from the
> use of it. I have been iterating on the solution while fixing a lot of the
> test resource leaks in the last few weeks. Without support for detecting
> the resource leaks, it's really hard to keep the test suite clean.
>



> Looking forward to more reviews on
> https://github.com/apache/pulsar/pull/21450 . :)




Looks great

Thanks
Enrico

>
>
> -Lari
>
> On 2023/10/29 18:34:28 Asaf Mesika wrote:
> > Larry, I know there is a way to add like a Job summary, so we can write
> it
> > there - do you think this can increase visibility?
> >
> > On Sun, Oct 29, 2023 at 4:53 AM Lari Hotari  wrote:
> >
> > > Hi all,
> > >
> > > I have submitted a PR (https://github.com/apache/pulsar/pull/21450)
> which
> > > includes changes to add reporting and tooling to detect thread leaks in
> > > Pulsar tests.
> > >
> > > It should be ensured in each test that resources created by the test
> are
> > > properly cleaned up. Failing to do so can lead to memory leaks and, in
> some
> > > instances, unnecessary CPU consumption. These issues can, in turn, slow
> > > down test execution, increase Pulsar CI build durations, and cause
> > > flakiness.  A significant source of memory leaks in Pulsar tests stems
> from
> > > thread leaks.
> > >
> > > After the PR is merged, it will be easy to detect thread leaks since
> the
> > > build will add warnings to the summary view for the GitHub Actions
> build
> > > run. An example can be seen in the PR build run:
> > > https://github.com/apache/pulsar/actions/runs/6680066364?pr=21450 .
> > > There will be more detailed information in the "Report detected thread
> > > leaks" build step, for example
> > >
> https://github.com/apache/pulsar/actions/runs/6680066364/job/18153890519?pr=21450#step:16:23
> > > .
> > >
> > > Please review the PR https://github.com/apache/pulsar/pull/21450 so
> that
> > > we can continue to get rid of the remaining thread leaks in the future
> and
> > > keep the tests cleaner and less flaky.
> > >
> > > -Lari
> > >
> >
>

Re: [DISCUSS] Auto release Nuget artifacts for DotPulsar

[Enrico Olivelli]

How do we handle the VOTE?

Enrico

Il Lun 30 Ott 2023, 08:06 Zike Yang  ha scritto:

> Hi, David
>
> Thanks for initializing this discussion.
> I'm +1 for making the auto-release process for the DotPulsar. I have
> left some comments in the PR.
>
> Thanks,
> Zike Yang
>
> On Thu, Oct 26, 2023 at 5:39 PM David Jensen  wrote:
> >
> > Hi everyone,
> > I have worked with Tison to make the release process smoother for
> DotPulsar
> > . See GitHub Issue #184
> Release
> > process and GitHub Action
> >  and the linked
> issues.
> >
> > In summary, the document presents a suggested approach for creating
> release
> > notes and optimizing the release workflow of the Pulsar C# Client. This
> > approach involves working closely with team members, adhering to ASF
> > guidelines, participating in Apache release voting, implementing
> automation
> > via GitHub Actions, and managing versioning through git tags and GitHub
> > releases.
> >
> > Sincerely
> > David Jensen
>

Re: [DISCUSS] Auto release Nuget artifacts for DotPulsar

[Zike Yang]

Hi, David

Thanks for initializing this discussion.
I'm +1 for making the auto-release process for the DotPulsar. I have
left some comments in the PR.

Thanks,
Zike Yang

On Thu, Oct 26, 2023 at 5:39 PM David Jensen  wrote:
>
> Hi everyone,
> I have worked with Tison to make the release process smoother for DotPulsar
> . See GitHub Issue #184 Release
> process and GitHub Action
>  and the linked issues.
>
> In summary, the document presents a suggested approach for creating release
> notes and optimizing the release workflow of the Pulsar C# Client. This
> approach involves working closely with team members, adhering to ASF
> guidelines, participating in Apache release voting, implementing automation
> via GitHub Actions, and managing versioning through git tags and GitHub
> releases.
>
> Sincerely
> David Jensen