Re: [VOTE] Release 4.17.1, release candidate #0

[Enrico Olivelli]

+1 (binding)

- verified checksums and signatures
- built from sources (Ubuntu, JDK11), ran tests
- HerdDB tests are passing https://github.com/diennea/herddb/pull/830

Thanks for driving the release !

Enrico Olivelli

Il giorno dom 16 giu 2024 alle ore 11:40 zhiheng123
<903292...@qq.com.invalid> ha scritto:

> +1 (no binding)
>
> Hash matches
>
> Signature matches

Re: [RESULT] [VOTE] Release 4.17.0, release candidate #0

[Enrico Olivelli]

Yunze,
I missed your message.
Now it is all good

Enrico

Il giorno sab 11 mag 2024 alle ore 10:00 Yunze Xu
 ha scritto:

> Hi Enrico,
>
> The vote has passed for near a month and Pulsar has already upgraded the
> BK dependency
> To 4.17.0. However, I still cannot see the release here:
> https://github.com/apache/bookkeeper/releases
>
> The document of https://bookkeeper.apache.org/docs/overview/ is also
> wrong, whose title is
> "Apache BookKeeper 4.16.4-SNAPSHOT”.
>
> Could you continue the progress if it’s still not done?
>
> Thanks,
> Yunze
>
>
>
>
> > On Apr 17, 2024, at 19:45, Enrico Olivelli  wrote:
> >
> > The VOTE passes with 4 +1s (3 binding):
> > - Lari
> > - Hang Chen
> > - Enrico
> > - Andrey
> >
> > I will promote the artifacts and complete the release procedure
> >
> > Thanks
> > Enrico
> >
> > Il giorno gio 11 apr 2024 alle ore 15:29 Hang Chen 
> ha
> > scritto:
> >
> >> +1 (binding)
> >>
> >> - Verified signatures and checksums
> >> - Build from the source code with JDK 17, Maven 3.8.6 on M3 macbook
> >> - All the StreamNative tests are passed
> >> - Build Pulsar branch-3.2 with the staging packages
> >> - Run Pulsar cluster with one broker, one bookie and one zookeeper,
> >> verified:
> >>  - Produce messages to Pulsar topics
> >>  - Offload topic data to Lakehouse with Lakehouse tiered storage
> >>  - Pulsar consumer consume messages from Pulsar topics
> >>
> >>
> >> Thanks for driving the release
> >>
> >> Hang
> >>
> >> Andrey Yegorov  于2024年4月10日周三 01:19写道：
> >>>
> >>> +1 (binding)
> >>>
> >>> - built from tag
> >>> - built from archived sources
> >>> - ran small subset of unit tests
> >>> - ran standalone andlocalbookie from packaged bookkeeper-server
> >>> - spot-checked staged maven repo
> >>> - reviewed release notes
> >>>
> >>> Release notes need double-check, I found one PR listed as a "fix" while
> >> actually the PR was closed without the merge. The rest lgtm.
> >>>
> >>> Thank you.
> >>> --
> >>> Andrey
> >>>
> >>> On 2024/04/02 08:33:56 Enrico Olivelli wrote:
> >>>> Hi everyone,
> >>>> Please review and vote on the release candidate #0 for the version
> >>>> 4.17.0, as follows:
> >>>> [ ] +1, Approve the release
> >>>> [ ] -1, Do not approve the release (please provide specific comments)
> >>>>
> >>>> The complete staging area is available for your review, which
> includes:
> >>>> * Release notes [1]
> >>>> * The official Apache source and binary distributions to be deployed
> >>>> to dist.apache.org [2]
> >>>> * All artifacts to be deployed to the Maven Central Repository [3]
> >>>> * Source code tag "v4.17.0-rc0"  [4] with git sha
> >>>> cce4b6461691466c663f2cb4d00dd4d73dd9071e
> >>>>
> >>>> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> >>>> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >>>>
> >>>> Please download these packages and review this release candidate:
> >>>>
> >>>> - Review release notes
> >>>> - Download the source package (verify shasum, and asc) and follow the
> >>>> instructions to build and run the bookkeeper service.
> >>>> - Download the binary package (verify shasum, and asc) and follow the
> >>>> instructions to run the bookkeeper service.
> >>>> - Review maven repo, release tag, licenses, and any other things you
> >> think
> >>>> it is important to release.
> >>>>
> >>>> The vote will be open for at least 72 hours. It is adopted by majority
> >>>> approval, with at least 3 PMC affirmative votes.
> >>>>
> >>>> Thanks,
> >>>> Enrico Olivelli
> >>>>
> >>>> [1]
> >>
> https://github.com/apache/bookkeeper/pull/4260/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53
> >>>> [2]
> >>
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.17.0-rc0/
> >>>> [3]
> >>
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1095/
> >>>> [4] https://github.com/apache/bookkeeper/tree/v4.17.0-rc0
> >>>>
> >>
>
>

Re: [VOTE] Release 4.16.6, release candidate #0

[Enrico Olivelli]

+1 (binding)

- Verified checksums and signatures
- Ran unit test on Ubuntu, JDK11
- HerdDB tests are passing https://github.com/diennea/herddb/pull/831


Thanks for driving the release !

Enrico Olivelli


Il giorno dom 16 giu 2024 alle ore 11:41 zhiheng123
<903292...@qq.com.invalid> ha scritto:

> +1 (no binding)
>
> Hash matches
>
> Signature matches

[ACCOUNT] ZhangJian He added to the PMC of Apache BookKeeper

[Enrico Olivelli]

Hello everyone,
I am happy to announce that  ZhangJian He has accepted the invitation to
join the BookKeeper PMC.

More details about how the ASF works here:
https://apache.org/foundation/how-it-works/#roles

Congratulations ZhangJian !

Thank you for all your efforts to help the community and your contributions
to the project !

Enrico Olivelli

[ANNOUNCE] Lari Hotari as committer in BookKeeper

[Enrico Olivelli]

Hello everyone,
I am happy to announce that Lari Hotari has been invited by the PMC to
participate as a committer in our community and he accepted.

Congratulations Lari !

Note for new community members: you can find more about what it means to be
a "committer" here: https://apache.org/foundation/how-it-works/#roles

Best regards
Enrico Olivelli

Re: [DISCUSS] BookKeeper 4.16.6 release

[Enrico Olivelli]

Il Mar 28 Mag 2024, 09:18 ZhangJian He  ha scritto:

> Thank you for your trust. I think we are closer, there are PRs need to be
> handle before release
>
> https://github.com/apache/bookkeeper/pull/4398 BTW, Can we use one pr to
> include 3 cherry-pick PRs?  There are both vertx,netty dependency updates.
> Is this valid?
> https://github.com/apache/bookkeeper/pull/4400
> https://github.com/apache/bookkeeper/pull/4395


Yes you can but please not squash them

Enrico

>
>
> Thanks
> ZhangJian He
> Twitter: shoothzj
> Wechat: shoothzj
>
>
> On Mon, May 27, 2024 at 2:57 PM Enrico Olivelli 
> wrote:
>
> > +1 thanks
> >
> > Enrico
> >
> > Il Lun 27 Mag 2024, 08:52 Nicolò Boschi  ha
> scritto:
> >
> > > +1
> > > There are important bug fixes to release
> > >
> > > Thanks for taking care of the release
> > > Nicolò Boschi
> > >
> > > Il giorno sab 25 mag 2024 alle ore 04:37 ZhangJian He
> > >  ha scritto:
> > > >
> > > > Hi all,
> > > >
> > > > BookKeeper 4.16.5 was released one and half months ago, and I would
> > > > like to discuss the starting 4.16.6 release to include some critical
> > > > security and bug fixes.
> > > >
> > > > Here are the PRs for 4.16.6:
> > > > PR-4.16.6
> > > > <
> > >
> >
> https://github.com/apache/bookkeeper/pulls?q=is%3Apr+label%3Arelease%2F4.16.6+is%3Amerged
> > > >
> > > >
> > > > There are still some PRs not been cherry-picked due to conflict, I
> will
> > > > handle them these days. I removed open PRs that were targeting
> > > > 4.16.6 label.
> > > >
> > > > If you have other PRs that you want to be included in this release,
> > > please
> > > > tag the PR with the "release/4.16.6" and let me know in this thread.
> > > >
> > > > If there's any concern, I'd like to volunteer to guide this release.
> > > >
> > > > Thanks
> > > > ZhangJian He
> > > > Twitter: shoothzj
> > > > Wechat: shoothzj
> > >
> >
>

Re: [DISCUSS] BookKeeper 4.16.6 release

[Enrico Olivelli]

+1 thanks

Enrico

Il Lun 27 Mag 2024, 08:52 Nicolò Boschi  ha scritto:

> +1
> There are important bug fixes to release
>
> Thanks for taking care of the release
> Nicolò Boschi
>
> Il giorno sab 25 mag 2024 alle ore 04:37 ZhangJian He
>  ha scritto:
> >
> > Hi all,
> >
> > BookKeeper 4.16.5 was released one and half months ago, and I would
> > like to discuss the starting 4.16.6 release to include some critical
> > security and bug fixes.
> >
> > Here are the PRs for 4.16.6:
> > PR-4.16.6
> > <
> https://github.com/apache/bookkeeper/pulls?q=is%3Apr+label%3Arelease%2F4.16.6+is%3Amerged
> >
> >
> > There are still some PRs not been cherry-picked due to conflict, I will
> > handle them these days. I removed open PRs that were targeting
> > 4.16.6 label.
> >
> > If you have other PRs that you want to be included in this release,
> please
> > tag the PR with the "release/4.16.6" and let me know in this thread.
> >
> > If there's any concern, I'd like to volunteer to guide this release.
> >
> > Thanks
> > ZhangJian He
> > Twitter: shoothzj
> > Wechat: shoothzj
>

Re: [DISCUSS] Introducing Typo CI Check to Apache BookKeeper

[Enrico Olivelli]

+1

Enrico

Il giorno dom 26 mag 2024 alle ore 01:53 Hang Chen  ha
scritto:

> +1
>
> We need to keep an eye on the public methods and configurations.
>
> Best,
> Hang
>
> Andrey Yegorov  于2024年5月22日周三 02:15写道：
> >
> >
> > Awesome idea, thank you for tackling this.
> >
> > We need to keep an eye on changes in config parameter names in
> yaml/configs to avoid unexpected breaks.
> > So far I haven't noticed any, but it is an easy miss in a big PR.
> >
> > On 2024/05/19 07:20:30 ZhangJian He wrote:
> > > Hi, I want to introduce new lint tools: typos[1]. This tool is
> designed to
> > > automatically detect and correct typographical errors in our codebase,
> > > documentation, and comments. It’s a valuable addition to ensure
> > > high-quality contributions and maintain the professional standard of
> our
> > > project.
> > >
> > > It has also been used in the Apache OpenDAL and Kvrocks projects. I
> created
> > > a draft PR[2].
> > >
> > > I have been careful to avoid changes to public interfaces, but I may
> have
> > > made mistakes, and the changes are relatively huge. I think I would
> prefer
> > > to proceed as follows:
> > >
> > > First, commit typos in markdown, YAML, etc.
> > > Second, commit typos in test files.
> > > Lastly, fix other typos and commit the lint.
> > >
> > >
> > > 1. https://github.com/crate-ci/typos
> > > 2. https://github.com/apache/bookkeeper/pull/4375/files
> > >
> > > Thanks
> > > ZhangJian He
> > > Twitter: shoothzj
> > > Wechat: shoothzj
> > >
>

Re: Discussion on Default Implementation for isDriverMetadataServiceAvailable

[Enrico Olivelli]

ZhangJian,

Il giorno mer 8 mag 2024 alle ore 01:34 ZhangJian He 
ha scritto:

> Hi, bookKeepers
>
> I've reviewed the PR that introduces `isDriverMetadataServiceAvailable`. I
> have concerns about providing a default implementation that returns a
> constant value like true, it's not a default interface like this.
>
> https://github.com/apache/bookkeeper/pull/4342#discussion_r1591761669
>
> ```
> default E getFirstElement() {
> return getElements().get(0);
> }
>
> List getElements() {
> }
> ```
>
> - First we don't guarantee/make ABI compatible between minor releases
> - Second, People who implement metadata drivers should implement this
> correctly, it may lead to unintended behavior if it's not properly
> overridden by all implementations.
>
> I'd suggest making this method abstract instead to avoid potential
> misinterpretation.
>

It is fine to make it abstract and implement it in all of the
implementations that we maintain.


Enrico



>
> Thanks
> ZhangJian He
> Twitter: shoothzj
> Wechat: shoothzj
>

The 'news' section is not up-to-date

[Enrico Olivelli]

Hello,
I noticed that the News section is not up to date
https://bookkeeper.apache.org/releases

Any volunteers to fix it ?

Cheers
Enrico

Re: [VOTE] BP-66: support throttling for zookeeper read of rereplication

[Enrico Olivelli]

+1 (binding)

Thanks

Enrico Olivelli

Il Mar 30 Apr 2024, 05:17 ZhangJian He  ha scritto:

> +1(no-binding)
>
>
> Thanks
> ZhangJian He
>
>
> thetumbled 于2024年4月30日 周二11:06写道：
>
> > Hi, everyone.I would like to start voting thread for BP-66: support
> > throttling for zookeeper read of rereplication.Proposal PR:
> > https://github.com/apache/bookkeeper/pull/4258
> > Implementation PR:https://github.com/apache/bookkeeper/pull/4256
> >
> >
> >
> > Thanks,
> > thetumbled.
>

Re: [discuss] Proposal to Gradually Migrate from JUnit 4 to JUnit 5

[Enrico Olivelli]

ZhangJian

Il giorno mer 24 apr 2024 alle ore 18:58 ZhangJian He 
ha scritto:

> I am reaching out to propose a gradual migration of our testing framework
> from JUnit 4 to JUnit 5 across the BookKeeper. JUnit 5 offers significant
> improvements over JUnit 4, including more powerful and flexible testing
> capabilities like annotation-based coding(so we don't need to write
> assume), parallel test execution, and other features.
>


+1

Please ensure that on CI we always both the 2 families of tests until we
completed the migration

Thank you
Enrico


>
>
> I think we can write new test classes using junit5, and existing-test
> classes can be slowly migrated.
>
> Thanks
> ZhangJian He
>

[RESULT] [VOTE] Release 4.17.0, release candidate #0

[Enrico Olivelli]

The VOTE passes with 4 +1s (3 binding):
- Lari
- Hang Chen
- Enrico
- Andrey

I will promote the artifacts and complete the release procedure

Thanks
Enrico

Il giorno gio 11 apr 2024 alle ore 15:29 Hang Chen  ha
scritto:

> +1 (binding)
>
> - Verified signatures and checksums
> - Build from the source code with JDK 17, Maven 3.8.6 on M3 macbook
> - All the StreamNative tests are passed
> - Build Pulsar branch-3.2 with the staging packages
> - Run Pulsar cluster with one broker, one bookie and one zookeeper,
> verified:
>   - Produce messages to Pulsar topics
>   - Offload topic data to Lakehouse with Lakehouse tiered storage
>   - Pulsar consumer consume messages from Pulsar topics
>
>
> Thanks for driving the release
>
> Hang
>
> Andrey Yegorov  于2024年4月10日周三 01:19写道：
> >
> > +1 (binding)
> >
> > - built from tag
> > - built from archived sources
> > - ran small subset of unit tests
> > - ran standalone andlocalbookie from packaged bookkeeper-server
> > - spot-checked staged maven repo
> > - reviewed release notes
> >
> > Release notes need double-check, I found one PR listed as a "fix" while
> actually the PR was closed without the merge. The rest lgtm.
> >
> > Thank you.
> > --
> > Andrey
> >
> > On 2024/04/02 08:33:56 Enrico Olivelli wrote:
> > > Hi everyone,
> > > Please review and vote on the release candidate #0 for the version
> > > 4.17.0, as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * Release notes [1]
> > > * The official Apache source and binary distributions to be deployed
> > > to dist.apache.org [2]
> > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > * Source code tag "v4.17.0-rc0"  [4] with git sha
> > > cce4b6461691466c663f2cb4d00dd4d73dd9071e
> > >
> > > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > >
> > > Please download these packages and review this release candidate:
> > >
> > > - Review release notes
> > > - Download the source package (verify shasum, and asc) and follow the
> > > instructions to build and run the bookkeeper service.
> > > - Download the binary package (verify shasum, and asc) and follow the
> > > instructions to run the bookkeeper service.
> > > - Review maven repo, release tag, licenses, and any other things you
> think
> > > it is important to release.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Enrico Olivelli
> > >
> > > [1]
> https://github.com/apache/bookkeeper/pull/4260/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53
> > > [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.17.0-rc0/
> > > [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1095/
> > > [4] https://github.com/apache/bookkeeper/tree/v4.17.0-rc0
> > >
>

Re: [discuss] Change scheduleAtFixedRate to scheduleWithFixedDelay in GarbageCollectorThread

[Enrico Olivelli]

Il giorno mar 16 apr 2024 alle ore 11:06 ZhangJian He
 ha scritto:
>
> Hi teams. Current scheduling mechanism for garbage collection uses
> scheduleAtFixedRate. This approach schedules the next execution without
> considering whether the current task has finished, potentially leading to
> overlapping executions if a task takes longer than expected.
>
> In my test environment, After task accumulation in gc thread pool,
> sometimes there is no entrylog to extract and no entrylogger to compact.
> But every round of gc, still need to compare ledger meta between local and
> metadata store (zk), which will result in highly frequently access to
> metadata store and each access will bring considerable unnecessary data
> flow.
>
> I propose changing this to use scheduleWithFixedDelay instead, which
> ensures that there is a fixed delay period between the end of one execution
> and the start of the next.

makes sense

Enrico


> Thanks
> ZhangJian He

Re: Adding me to the dockerhub org

[Enrico Olivelli]

As you have committed karma I think it is fine to add you to the group
on dockerhub who is able to trigger builds and see the logs.

Please open an INFRA ticket, and ask to add them to add you to the
"bookkeeper" group

More context here:
https://issues.apache.org/jira/browse/INFRA-21985

Thanks

Enrico


Enrico

Il giorno mer 10 apr 2024 alle ore 14:29 Nicolò Boschi
 ha scritto:
>
> Hi
> I'm currently finalizing the release of 4.16.5 and the docker image have
> been through some issues (for example it's not present for 4.16.4)
>
> Could you please add me? (handle @nicoloboschi)
>
> Meanwhile, please ensure that the build for 4.16.5 in dockerhub is being
> pulled from the correct commit: 7018e3a1ac9f8d64e57d3f141e95dcdada48342a.
> (I had to delete and recreate the git tag due to an error in the release
> steps)
>
> Thanks,
> Nicolò Boschi

Re: [VOTE] Release 4.17.0, release candidate #0

[Enrico Olivelli]

+1 (binding)

- HerdDB tests  are passing: https://github.com/diennea/herddb/pull/829
- Built locally, ran some smoke tests

Enrico

Il giorno mer 3 apr 2024 alle ore 09:43 Lari Hotari
 ha scritto:
>
> +1 (non-binding)
>
> - signatures ok
> - checksums ok
> - Pulsar master branch runs fine with staged maven artifacts
>   - https://github.com/lhotari/pulsar/pull/177
> - ran simpletest with bookkeeper-all-4.17.0 binaries
>   ("bookkeeper localbookie 3" and "bookkeeper shell simpletest")
> - source package compilation succeeds
>
> -Lari
>
> On 2024/04/02 08:33:56 Enrico Olivelli wrote:
> > Hi everyone,
> > Please review and vote on the release candidate #0 for the version
> > 4.17.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed
> > to dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "v4.17.0-rc0"  [4] with git sha
> > cce4b6461691466c663f2cb4d00dd4d73dd9071e
> >
> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review maven repo, release tag, licenses, and any other things you think
> > it is important to release.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Enrico Olivelli
> >
> > [1] 
> > https://github.com/apache/bookkeeper/pull/4260/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53
> > [2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.17.0-rc0/
> > [3] 
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1095/
> > [4] https://github.com/apache/bookkeeper/tree/v4.17.0-rc0
> >

Re: [VOTE] Release 4.16.5, release candidate #1

[Enrico Olivelli]

+1 (binding)

- Veriified signatures and checksums
- HerdDB tests are passing https://github.com/diennea/herddb/pull/828
- built locally from sources and tests local binaries

Thanks for driving the release

Enrico

Il giorno ven 5 apr 2024 alle ore 15:03 Lari Hotari
 ha scritto:
>
> +1 (non-binding)
>
> - signatures ok
> - checksums ok
> - Pulsar master branch runs fine with staged maven artifacts
> - https://github.com/lhotari/pulsar/pull/178
> - ran simpletest with bookkeeper-all-4.16.5 binaries
> ("bookkeeper localbookie 3" and "bookkeeper shell simpletest")
> - source package compilation succeeds
>
> -Lari
>
> On 2024/04/03 12:43:45 Nicolò Boschi wrote:
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the version 4.16.5,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "v4.16.5-rc1" [4] with git sha 7018e3a
> >
> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review maven repo, release tag, licenses, and any other things you think
> > it is important for a release.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1] https://github.com/apache/bookkeeper/pull/4249
> > [2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.5-rc1/
> > [3]
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1096/
> > [4] https://github.com/apache/bookkeeper/tree/v4.16.5-rc1
> >

Re: Docker build is broken for 4.16.4

[Enrico Olivelli]

Il Mar 2 Apr 2024, 15:49 Lari Hotari  ha scritto:

> Does this work for 4.16.5 ?
>
> I fixed outdated Dockerfile issues in PR
> https://github.com/apache/bookkeeper/pull/4197 / (branch-4.16:
> https://github.com/apache/bookkeeper/pull/4206).
> Diff:
> https://github.com/apache/bookkeeper/pull/4206/files#diff-f34da55ca08f1a30591d8b0b3e885bcc678537b2a9a4aadea4f190806b374ddcL20
>
> ubuntu-22.10 is no longer supported and that's the source of the error
> message "The repository 'http://archive.ubuntu.com/ubuntu
> kinetic-updates Release' does not have a Release file."
> In the PR, I replaced it with ubuntu-22.04 (LTS version, support
> continues).
>


With your patch the docker images should work.
Maybe we should add an integration test that uses the docker images in CI

Enrico


> -Lari
>
>
> On Tue, 2 Apr 2024 at 16:06, Enrico Olivelli  wrote:
> >
> > Hello Bookkeepers,
> > it seems that docker hub is no more able to build our docker images
> >
> > You can find below the error
> >
> > Any ideas?
> >
> > Enrico
> >
> > Link to the build
> >
> https://hub.docker.com/repository/registry-1.docker.io/apache/bookkeeper/builds/114a5d41-f061-47d4-b9f4-744aafaaa665
> >
> > Full logs
> >
> > 2024-04-02T12:08:33Z Building in Docker Cloud's infrastructure...
> > 2024-04-02T12:08:34Z Cloning into '.'...
> > 2024-04-02T12:08:34Z Warning: Permanently added the RSA host key for
> > IP address '140.82.114.3' to the list of known hosts.
> > 2024-04-02T12:09:14Z Switched to a new branch 'release-4.16.4-docker'
> > 2024-04-02T12:09:14Z Executing build hook...
> > 2024-04-02T12:09:14Z + [[ 4.16.4 = \l\a\t\e\s\t ]]
> > 2024-04-02T12:09:14Z + docker build --build-arg BK_VERSION=4.16.4 -t
> > index.docker.io/apache/bookkeeper:4.16.4 .
> > 2024-04-02T12:09:20Z #1 [internal] load build definition from Dockerfile
> > 2024-04-02T12:09:20Z #1 transferring dockerfile:
> > 2024-04-02T12:09:20Z #1 transferring dockerfile: 2.71kB done
> > 2024-04-02T12:09:20Z #1 DONE 0.1s
> > 2024-04-02T12:09:20Z
> > 2024-04-02T12:09:20Z #2 [internal] load .dockerignore
> > 2024-04-02T12:09:20Z #2 transferring context: 2B done
> > 2024-04-02T12:09:20Z #2 DONE 0.0s
> > 2024-04-02T12:09:20Z
> > 2024-04-02T12:09:20Z #3 [internal] load metadata for
> > docker.io/library/ubuntu:22.10
> > 2024-04-02T12:09:21Z #3 ...
> > 2024-04-02T12:09:21Z
> > 2024-04-02T12:09:21Z #4 [auth] library/ubuntu:pull token for
> > registry-1.docker.io
> > 2024-04-02T12:09:21Z #4 DONE 0.0s
> > 2024-04-02T12:09:21Z
> > 2024-04-02T12:09:21Z #3 [internal] load metadata for
> > docker.io/library/ubuntu:22.10
> > 2024-04-02T12:09:21Z #3 DONE 0.6s
> > 2024-04-02T12:09:21Z
> > 2024-04-02T12:09:21Z #5 [1/5] FROM
> >
> docker.io/library/ubuntu:22.10@sha256:e322f4808315c387868a9135beeb11435b5b83130a8599fd7d0014452c34f489
> > 2024-04-02T12:09:21Z #5 resolve
> >
> docker.io/library/ubuntu:22.10@sha256:e322f4808315c387868a9135beeb11435b5b83130a8599fd7d0014452c34f489
> > 0.1s done
> > 2024-04-02T12:09:21Z #5 ...
> > 2024-04-02T12:09:21Z
> > 2024-04-02T12:09:21Z #6 [internal] load build context
> > 2024-04-02T12:09:21Z #6 transferring context: 18.21kB done
> > 2024-04-02T12:09:21Z #6 DONE 0.0s
> > 2024-04-02T12:09:21Z
> > 2024-04-02T12:09:21Z #5 [1/5] FROM
> >
> docker.io/library/ubuntu:22.10@sha256:e322f4808315c387868a9135beeb11435b5b83130a8599fd7d0014452c34f489
> > 2024-04-02T12:09:21Z #5
> > sha256:d36481db487c935436420cc4b705db8c81eec3872085ebd27963b775695fbe24
> > 424B / 424B done
> > 2024-04-02T12:09:21Z #5
> > sha256:692eb4a905c074054e0a35d647671f0e32ed150d15b23fd7bc745cfb2fdeddbd
> > 2.30kB / 2.30kB done
> > 2024-04-02T12:09:21Z #5
> > sha256:3ad6ea492c35ef48892b4587449510d372516d5969073eeaa6601a380e861c57
> > 0B / 26.70MB 0.1s
> > 2024-04-02T12:09:21Z #5
> > sha256:e322f4808315c387868a9135beeb11435b5b83130a8599fd7d0014452c34f489
> > 1.13kB / 1.13kB done
> > 2024-04-02T12:09:21Z #5
> > sha256:3ad6ea492c35ef48892b4587449510d372516d5969073eeaa6601a380e861c57
> > 22.02MB / 26.70MB 0.3s
> > 2024-04-02T12:09:21Z #5
> > sha256:3ad6ea492c35ef48892b4587449510d372516d5969073eeaa6601a380e861c57
> > 26.70MB / 26.70MB 0.4s
> > 2024-04-02T12:09:22Z #5
> > sha256:3ad6ea492c35ef48892b4587449510d372516d5969073eeaa6601a380e861c57
> > 26.70MB / 26.70MB 0.6s done
> > 2024-04-02T12:09:22Z #5 extracting
> > sha256:3ad6ea492c35ef48892b4587449510d372516d5969073eeaa6601a380e861c57
> > 2024-04-02T12:09:25Z #5 extracting
> > sha256:3ad6ea492c35ef48892b

Docker build is broken for 4.16.4

[Enrico Olivelli]

Hello Bookkeepers,
it seems that docker hub is no more able to build our docker images

You can find below the error

Any ideas?

Enrico

Link to the build
https://hub.docker.com/repository/registry-1.docker.io/apache/bookkeeper/builds/114a5d41-f061-47d4-b9f4-744aafaaa665

Full logs

2024-04-02T12:08:33Z Building in Docker Cloud's infrastructure...
2024-04-02T12:08:34Z Cloning into '.'...
2024-04-02T12:08:34Z Warning: Permanently added the RSA host key for
IP address '140.82.114.3' to the list of known hosts.
2024-04-02T12:09:14Z Switched to a new branch 'release-4.16.4-docker'
2024-04-02T12:09:14Z Executing build hook...
2024-04-02T12:09:14Z + [[ 4.16.4 = \l\a\t\e\s\t ]]
2024-04-02T12:09:14Z + docker build --build-arg BK_VERSION=4.16.4 -t
index.docker.io/apache/bookkeeper:4.16.4 .
2024-04-02T12:09:20Z #1 [internal] load build definition from Dockerfile
2024-04-02T12:09:20Z #1 transferring dockerfile:
2024-04-02T12:09:20Z #1 transferring dockerfile: 2.71kB done
2024-04-02T12:09:20Z #1 DONE 0.1s
2024-04-02T12:09:20Z
2024-04-02T12:09:20Z #2 [internal] load .dockerignore
2024-04-02T12:09:20Z #2 transferring context: 2B done
2024-04-02T12:09:20Z #2 DONE 0.0s
2024-04-02T12:09:20Z
2024-04-02T12:09:20Z #3 [internal] load metadata for
docker.io/library/ubuntu:22.10
2024-04-02T12:09:21Z #3 ...
2024-04-02T12:09:21Z
2024-04-02T12:09:21Z #4 [auth] library/ubuntu:pull token for
registry-1.docker.io
2024-04-02T12:09:21Z #4 DONE 0.0s
2024-04-02T12:09:21Z
2024-04-02T12:09:21Z #3 [internal] load metadata for
docker.io/library/ubuntu:22.10
2024-04-02T12:09:21Z #3 DONE 0.6s
2024-04-02T12:09:21Z
2024-04-02T12:09:21Z #5 [1/5] FROM
docker.io/library/ubuntu:22.10@sha256:e322f4808315c387868a9135beeb11435b5b83130a8599fd7d0014452c34f489
2024-04-02T12:09:21Z #5 resolve
docker.io/library/ubuntu:22.10@sha256:e322f4808315c387868a9135beeb11435b5b83130a8599fd7d0014452c34f489
0.1s done
2024-04-02T12:09:21Z #5 ...
2024-04-02T12:09:21Z
2024-04-02T12:09:21Z #6 [internal] load build context
2024-04-02T12:09:21Z #6 transferring context: 18.21kB done
2024-04-02T12:09:21Z #6 DONE 0.0s
2024-04-02T12:09:21Z
2024-04-02T12:09:21Z #5 [1/5] FROM
docker.io/library/ubuntu:22.10@sha256:e322f4808315c387868a9135beeb11435b5b83130a8599fd7d0014452c34f489
2024-04-02T12:09:21Z #5
sha256:d36481db487c935436420cc4b705db8c81eec3872085ebd27963b775695fbe24
424B / 424B done
2024-04-02T12:09:21Z #5
sha256:692eb4a905c074054e0a35d647671f0e32ed150d15b23fd7bc745cfb2fdeddbd
2.30kB / 2.30kB done
2024-04-02T12:09:21Z #5
sha256:3ad6ea492c35ef48892b4587449510d372516d5969073eeaa6601a380e861c57
0B / 26.70MB 0.1s
2024-04-02T12:09:21Z #5
sha256:e322f4808315c387868a9135beeb11435b5b83130a8599fd7d0014452c34f489
1.13kB / 1.13kB done
2024-04-02T12:09:21Z #5
sha256:3ad6ea492c35ef48892b4587449510d372516d5969073eeaa6601a380e861c57
22.02MB / 26.70MB 0.3s
2024-04-02T12:09:21Z #5
sha256:3ad6ea492c35ef48892b4587449510d372516d5969073eeaa6601a380e861c57
26.70MB / 26.70MB 0.4s
2024-04-02T12:09:22Z #5
sha256:3ad6ea492c35ef48892b4587449510d372516d5969073eeaa6601a380e861c57
26.70MB / 26.70MB 0.6s done
2024-04-02T12:09:22Z #5 extracting
sha256:3ad6ea492c35ef48892b4587449510d372516d5969073eeaa6601a380e861c57
2024-04-02T12:09:25Z #5 extracting
sha256:3ad6ea492c35ef48892b4587449510d372516d5969073eeaa6601a380e861c57
3.3s done
2024-04-02T12:09:26Z #5 DONE 4.5s
2024-04-02T12:09:26Z
2024-04-02T12:09:26Z #7 [2/5] RUN set -x && adduser "bookkeeper" &&
apt-get update && apt-get install -y --no-install-recommends
openjdk-17-jdk && apt-get install -y --no-install-recommends python3
pip && ln -s /usr/bin/python3 /usr/bin/python && apt-get install -y
--no-install-recommends gpg gpg-agent wget sudo && apt-get -y --purge
autoremove && apt-get autoclean && apt-get clean && rm -rf
/var/lib/apt/lists/* && mkdir -pv /opt && cd /opt && wget -q
"https://archive.apache.org/dist/bookkeeper/bookkeeper-4.16.4/bookkeeper-server-4.16.4-bin.tar.gz;
&& wget -q 
"https://archive.apache.org/dist/bookkeeper/bookkeeper-4.16.4/bookkeeper-server-4.16.4-bin.tar.gz.asc;
&& wget -q 
"https://archive.apache.org/dist/bookkeeper/bookkeeper-4.16.4/bookkeeper-server-4.16.4-bin.tar.gz.sha512;
&& sha512sum -c bookkeeper-server-4.16.4-bin.tar.gz.sha512 && wget
https://dist.apache.org/repos/dist/release/bookkeeper/KEYS && gpg
--import KEYS && gpg --batch --verify
"bookkeeper-server-4.16.4-bin.tar.gz.asc"
"bookkeeper-server-4.16.4-bin.tar.gz" && tar -xzf
"bookkeeper-server-4.16.4-bin.tar.gz" && mv bookkeeper-server-4.16.4/
/opt/bookkeeper/ && rm -rf "bookkeeper-server-4.16.4-bin.tar.gz"
"bookkeeper-server-4.16.4-bin.tar.gz.asc"
"bookkeeper-server-4.16.4-bin.tar.gz.sha512" && pip install zk-shell
2024-04-02T12:09:26Z #7 0.382 + adduser bookkeeper
2024-04-02T12:09:26Z #7 0.433 Adding user `bookkeeper' ...
2024-04-02T12:09:26Z #7 0.433 Adding new group `bookkeeper' (1000) ...
2024-04-02T12:09:26Z #7 0.480 Adding new user `bookkeeper' (1000) with
group `bookkeeper' ...
2024-04-02T12:09:26Z #7 0.577 Creating home directory `/home/bookkeeper' ...

[VOTE] Release 4.17.0, release candidate #0

[Enrico Olivelli]

Hi everyone,
Please review and vote on the release candidate #0 for the version
4.17.0, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* Release notes [1]
* The official Apache source and binary distributions to be deployed
to dist.apache.org [2]
* All artifacts to be deployed to the Maven Central Repository [3]
* Source code tag "v4.17.0-rc0"  [4] with git sha
cce4b6461691466c663f2cb4d00dd4d73dd9071e

BookKeeper's KEYS file contains PGP keys we used to sign this release:
https://dist.apache.org/repos/dist/release/bookkeeper/KEYS

Please download these packages and review this release candidate:

- Review release notes
- Download the source package (verify shasum, and asc) and follow the
instructions to build and run the bookkeeper service.
- Download the binary package (verify shasum, and asc) and follow the
instructions to run the bookkeeper service.
- Review maven repo, release tag, licenses, and any other things you think
it is important to release.

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Enrico Olivelli

[1] 
https://github.com/apache/bookkeeper/pull/4260/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53
[2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.17.0-rc0/
[3] https://repository.apache.org/content/repositories/orgapachebookkeeper-1095/
[4] https://github.com/apache/bookkeeper/tree/v4.17.0-rc0

Re: [DISCUSS] BookKeeper 4.16.5 release

[Enrico Olivelli]

Good idea

Thanks
Enrico

Il Gio 21 Mar 2024, 09:54 Nicolò Boschi  ha scritto:

> Hi all,
>
> BookKeeper 4.16.4 was released two months ago, and I would
> like to discuss the starting 4.16.5 release to include some critical
> security and bug fixes.
>
> Here are the PRs for 4.16.5:
>
> https://github.com/apache/bookkeeper/pulls?q=is%3Apr+label%3Arelease%2F4.16.5+is%3Aclosed
>
> I've just cherry-picked all of them. I moved open PRs that were targeting
> 4.16.5 to the 4.16.6 label.
>
> If you have other PRs that you want to be included in this release, please
> tag the PR with the "release/4.16.5" and let me know in this thread.
>
> If there's any concern, I'd like to volunteer to guide this release.
>
> Thanks,
> Nicolò
>

Re: Error while starting the release procedure

[Enrico Olivelli]

Hello
A quick update.
I have created  branch-4.17 and now master is 4.18

There is a problem in the release process due to some update to the pom
file, we are executing the source plugin twice.
I haven't dug yet

I will keep you updated

Enrico

Il Mer 6 Mar 2024, 16:13 Enrico Olivelli  ha scritto:

> Hello,
> it seems that I cannot perform the usual release procedure, we change
> the rules on the GH repo
>
> How did we do in the previous releases ?
> Below you can find the error
>
> Enrico
>
> ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-release-plugin:3.0.0-M5:branch
> (default-cli) on project bookkeeper: Unable to commit files
> [ERROR] Provider message:
> [ERROR] The git-push command failed.
> [ERROR] Command output:
> [ERROR] remote: error: GH006: Protected branch update failed for
> refs/heads/master.
> [ERROR] remote: error: Changes must be made through a pull request. 14
> of 14 required status checks are expected.
> [ERROR] To https://github.com/apache/bookkeeper.git
> [ERROR]  ! [remote rejected]   master -> master (protected branch
> hook declined)
> [ERROR] error: failed to push some refs to
> 'https://github.com/apache/bookkeeper.git'
>

Error while starting the release procedure

[Enrico Olivelli]

Hello,
it seems that I cannot perform the usual release procedure, we change
the rules on the GH repo

How did we do in the previous releases ?
Below you can find the error

Enrico

ERROR] Failed to execute goal
org.apache.maven.plugins:maven-release-plugin:3.0.0-M5:branch
(default-cli) on project bookkeeper: Unable to commit files
[ERROR] Provider message:
[ERROR] The git-push command failed.
[ERROR] Command output:
[ERROR] remote: error: GH006: Protected branch update failed for
refs/heads/master.
[ERROR] remote: error: Changes must be made through a pull request. 14
of 14 required status checks are expected.
[ERROR] To https://github.com/apache/bookkeeper.git
[ERROR]  ! [remote rejected]   master -> master (protected branch
hook declined)
[ERROR] error: failed to push some refs to
'https://github.com/apache/bookkeeper.git'

Re: [DISCUSS] Bookkeeper 4.17.0 release

[Enrico Olivelli]

Let's get started.

Let me prepare the RC, I have checked the list of pending PRs (with
Hang) and it seems that we are good to go.

It will take some time, it has been quite some time since I cut a BK
release and I have to prepare the environment.
Also, I would like to see all tests passing on my machine.


Enrico

Il giorno mar 5 mar 2024 alle ore 08:14 Hang Chen
 ha scritto:
>
> Hi Enrico,
> Thank you for volunteering to take on the role of release manager
> for this cycle. Your willingness to contribute in this capacity is
> greatly appreciated.
>
> Best,
> Hang
>
> Enrico Olivelli  于2024年2月27日周二 20:23写道：
> >
> > Hello Hang,
> > Thanks for bringing this up
> >
> > I would be happy to be the release manager for this cycle
> >
> > Enrico
> >
> > Il Mer 21 Feb 2024, 07:37 Hang Chen  ha scritto:
> >
> > > Hi Community,
> > >Bookkeeper 4.16.0 has been released for more than 9 months, and we
> > > have introduced batch read support [1] for entry reading and many
> > > important bug fixes[2] and improvements[3][4][5]. I would like to
> > > start to discuss releasing 4.17.0. All the PRs in 4.17.0 can be found
> > > at
> > > https://github.com/apache/bookkeeper/pulls?q=is%3Aopen+is%3Apr+milestone%3A4.17.0
> > >
> > >Do you have any concerns? Do you have any other issue/ticket to add
> > > for 4.17.0? You can add the target PRs into Milestone 4.17.0 if you
> > > want to include them into 4.17.0.
> > >
> > >
> > > [1] https://github.com/apache/bookkeeper/pull/4051
> > > [2] https://github.com/apache/bookkeeper/pull/4140
> > > [3] https://github.com/apache/bookkeeper/pull/4211
> > > [4] https://github.com/apache/bookkeeper/pull/4139
> > > [5] https://github.com/apache/bookkeeper/pull/4038
> > >
> > > Regards,
> > > Hang
> > >

Re: [DISCUSS] Bookkeeper 4.17.0 release

[Enrico Olivelli]

Hello Hang,
Thanks for bringing this up

I would be happy to be the release manager for this cycle

Enrico

Il Mer 21 Feb 2024, 07:37 Hang Chen  ha scritto:

> Hi Community,
>Bookkeeper 4.16.0 has been released for more than 9 months, and we
> have introduced batch read support [1] for entry reading and many
> important bug fixes[2] and improvements[3][4][5]. I would like to
> start to discuss releasing 4.17.0. All the PRs in 4.17.0 can be found
> at
> https://github.com/apache/bookkeeper/pulls?q=is%3Aopen+is%3Apr+milestone%3A4.17.0
>
>Do you have any concerns? Do you have any other issue/ticket to add
> for 4.17.0? You can add the target PRs into Milestone 4.17.0 if you
> want to include them into 4.17.0.
>
>
> [1] https://github.com/apache/bookkeeper/pull/4051
> [2] https://github.com/apache/bookkeeper/pull/4140
> [3] https://github.com/apache/bookkeeper/pull/4211
> [4] https://github.com/apache/bookkeeper/pull/4139
> [5] https://github.com/apache/bookkeeper/pull/4038
>
> Regards,
> Hang
>

Re: [VOTE] Release 4.16.4, release candidate #0

[Enrico Olivelli]

+1 (binding)

- Verified checksums and signatures
- Passed all the HerdDB tests (https://github.com/diennea/herddb/pull/819)
- Built locally on Mac, JDK11

Thanks
Enrico

Il giorno ven 19 gen 2024 alle ore 03:08 Hang Chen
 ha scritto:
>
> Hi everyone,
> Please review and vote on the release candidate #0 for version 4.16.4,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed
> to dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.16.4-rc0" [4] with git sha [5]
>
> BookKeeper's KEYS file contains the PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review the maven repo, release tag, licenses, and any other things
> you think it is important to release.
>
> The vote will be open for at least 72 hours. It is adopted by the
> majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/bookkeeper/pull/4182/files
> [2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.4-rc0/
> [3] 
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1091
> [4] https://github.com/apache/bookkeeper/tree/v4.16.4-rc0
> [5] 16de712d8c8f1f0296697331bb68c1ad0eb5787d

Re: [DISCUSS] BookKeeper 4.16.4 release

[Enrico Olivelli]

+1

If you want I can drive the release, it has been quite some time I haven't
prepared an RC

Enrico


Il Gio 18 Gen 2024, 06:22 Lari Hotari  ha scritto:

> +1
>
> It would be great to have 4.16.4 released to get the fix for the
> Java9IntHash checksum bug https://github.com/apache/bookkeeper/pull/4140
> . I guess this bug applies to all platforms that aren't Linux x86-64 since
> that's when Java9IntHash is usually active.
> I've come across the bug quite a few times when experimenting with Pulsar
> on Mac M3 hardware.
>
> Are we ready to start the 4.16.4 release?
>
> -Lari
>
> On 2023/12/25 03:43:16 Hang Chen wrote:
> > Hi everyone,
> >
> > BookKeeper 4.16.3 was released more than three months ago, and I would
> > like to discuss the starting 4.16.4 release to include some critical
> > bug fixes and improvements, such as [1] and [2]
> >
> > Here are the PRs for 4.16.4:
> >
> https://github.com/apache/bookkeeper/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F4.16.4
> >
> > If you have PRs that want to be included in this release, please
> > comment on that PR or tag the label `release/4.16.4`.
> >
> > After merging those PRs, I want to volunteer to release this patch.
> >
> > Thanks,
> > Hang
> >
> > [1] https://github.com/apache/bookkeeper/pull/4163
> > [2] https://github.com/apache/bookkeeper/pull/4128
> >
>

Docker image missing for 4.15.5

[Enrico Olivelli]

Hello,
a user reported that the docker image for 4.15.5 was missing.

https://github.com/apache/bookkeeper/issues/4170

I have fixed the problem by creating a git tag

git checkout release-4.15.5
git tag -s release-4.15.5-docker
git push origin release-4.15.5-docker

Is there any volunteer to update the release procedure ?

https://bookkeeper.apache.org/community/release-guide

Thanks
Enrico

Re: [VOTE] Release 4.15.5, release candidate #0

[Enrico Olivelli]

+1 (binding)

Built on Mac with JDK11
Run smoke tests on the compiled binaries
Verified checksums and signatures

Enrico


Il giorno gio 14 dic 2023 alle ore 05:41 Hang Chen
 ha scritto:
>
> +1 (binding)
>
> Thank Yong for driving this release. Overall it looks good.
>
> - Verified signatures and checksums
> - Build from source code (MacOS jdk 17, maven 3.8.6)
> - Checked all the CI in BookKeeper branch-4.15 [1]
> - Build Pulsar master code with this staging dependency and start
> Pulsar standalone cluster and 1 bookie, 1 broker and 1 zookeeper
> Pulsar cluster and run Pulsar perf produce and consume
> - Verified the Prometheus metrics output
>
> Thanks,
> Hang
>
>
> [1] https://github.com/apache/bookkeeper/commits/branch-4.15/
>
> Yong Zhang  于2023年12月14日周四 09:31写道：
> >
> > I have done it on my branch: https://github.com/zymap/pulsar/pull/53
> >
> > On Wed, 13 Dec 2023 at 22:50, Hang Chen  wrote:
> >
> > > Hi Yong,
> > >  Thank you for driving this release effort. I have submitted a
> > > pull request [1] to upgrade the Pulsar branch-2.11 BookKeeper
> > > dependency to version 4.15.5 for testing. Currently, I am awaiting the
> > > completion of CI checks.
> > >
> > > Thanks,
> > > Hang
> > >
> > >
> > > [1] https://github.com/apache/pulsar/pull/21720
> > >
> > > Yong Zhang  于2023年12月12日周二 18:38写道：
> > > >
> > > > Hi everyone,
> > > > Please review and vote on the release candidate #0 for the version
> > > 4.15.5,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The complete staging area is available for your review, which includes:
> > > > * Release notes [1]
> > > > * The official Apache source and binary distributions to be deployed to
> > > > dist.apache.org [2]
> > > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > > * Source code tag "release-4.15.5" [4] with git sha
> > > > 27de09989a6920b939e29dcf625720f437269b38
> > > >
> > > > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > >
> > > > Please download these packages and review this release candidate:
> > > >
> > > > - Review release notes
> > > > - Download the source package (verify shasum, and asc) and follow the
> > > > instructions to build and run the bookkeeper service.
> > > > - Download the binary package (verify shasum, and asc) and follow the
> > > > instructions to run the bookkeeper service.
> > > > - Review maven repo, release tag, licenses, and any other things you
> > > think
> > > > it is important to a release.
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Release Manager
> > > >
> > > > [1] https://github.com/apache/bookkeeper/pull/4151
> > > > [2]
> > > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.5-rc0/
> > > > [3]
> > > >
> > > https://repository.apache.org/content/repositories/orgapachebookkeeper-1090
> > > > [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.5-rc0
> > > > [5] 27de09989a6920b939e29dcf625720f437269b38
> > >

Re: [DISCUSS] BookKeeper 4.15.5 release

[Enrico Olivelli]

Il giorno gio 30 nov 2023 alle ore 11:20 Yong Zhang
 ha scritto:
>
> Hi everyone,
>
> BookKeeper 4.15.4 was released more than 6 months, and I would like to
> discuss the starting 4.15.5 release to fix the wrong native lib caused by
> the wrong
> platform published.
> The docker platform has been fixed:
> https://github.com/apache/bookkeeper/pull/4060
>
> Here are the PRs for 4.15.5.
> https://github.com/apache/bookkeeper/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F4.15.6
>
> It has the wrong label 4.15.6, I will fix it tomorrow. There are some PRs
> that
> have been open for a long time, and no one updated them. I will close them
> as well.

Good idea

>
> If you have PRs that want to be included in this release, please
> comment on that PR or tag the label `release/4.15.5`.
>
> And I would like to volunteer to release this patch.

thank you very much, please go ahead

Enrico

>
> Thanks,
> Yong

Re: [DISCUSS] BP-64: Improve the description of 47 commands for bin/bookkeeper

[Enrico Olivelli]

Steven,
This is a great idea.

I don't think this needs a BP, as you are not changing APIs or touching the
internals.

In my opinion you can check here if nobody objects, then you can start
sending out PRs

Thank you very much

Enrico



On Sun, Oct 8, 2023, 09:58 steven lu  wrote:

> we will improve the description of 47 commands for bin/bookkeeper with
> BP-64
> https://github.com/apache/bookkeeper/issues/4101
>
> # Motivation
>
> When executing the command `./bin/bookkeeper shell`, the usage description
> of each command is not precise enough,
> - some commands have no description;
> - if there are descriptions, the explanation of the parameters is not clear
>
> so it is difficult for us to understand and use each command. we need to
> combine the code to understand it, then it takes unnecessary time.
>
> Related screenshot information:
>
> https://github.com/apache/bookkeeper/assets/42990025/578d0d2f-b443-4f2d-a8f5-d7b3fd1c70c1
> https://github.com/apache/bookkeeper/assets/42990025/578d0d2f-b443-4f2d-a8f5-d7b3fd1c70c1
> ">
>
>
> # Proposal
>
> 1) I plan to improve this function. The first version only involves the
> optimization of the getUsage method of these 47 commands from
> `org.apache.bookkeeper.bookie.BookieShell`
> 2) Use this as the bookkeeper specification. You will also need to follow
> this specification when adding new command codes later.
>

Community Over Code - Halifax

[Enrico Olivelli]

Hello folks,
Is there anyone at Halifax from our community?

Enrico

Re: [VOTE] Release 4.16.3, release candidate #0

[Enrico Olivelli]

+1 (binding)

- Built from sources and run all the tests
- Tested on HerdDB, all good (this is the PR that verifies that all
the tests pass with the new version
https://github.com/diennea/herddb/pull/811)

Thanks
Enrico

Il giorno mar 5 set 2023 alle ore 08:03 Jia Zhai 
ha scritto:
>
> +1 (binding)
>
> - verified packages checksum and signatures.
>
> - the source package is built and runs successfully.
>
> - binary package 'bin/bookkeeper standalone' runs well.
>
>
> On Tue, Sep 5, 2023 at 9:46 AM Hang Chen  wrote:
>
> > All unit tests passed when the BookKeeper dependency of the Pulsar
> > master branch was upgraded to 4.16.3. [1]
> >
> >
> > [1] https://github.com/zymap/pulsar/pull/52
> >
> > Regards,
> > Hang
> >
> > Hang Chen  于2023年9月5日周二 09:34写道：
> > >
> > > +1 (binding)
> > >
> > > - Verified signatures and checksums
> > > - Build from source code (MacOS jdk 11, maven 3.8.6)
> > > - Build BookKeeper with Pulsar master (MacOS JDK 17, Maven 3.8.6)
> > > - Build Pulsar master code with this staging dependency and start
> > > Pulsar cluster with one Zookeeper, one Bookie and one Broker. Run
> > > Pulsar perf produce and consume with DirectIO disabled -> enabled ->
> > > disabled on Ubuntu 20.04
> > > - Build Pulsar master code with MacOS build BookKeeper dependency and
> > > start Pulsar cluster with one Zookeeper, one Bookie and one broker.
> > > Run Pulsar perf produce and consume with DirectIO disabled -> enabled
> > > -> disabled on MacOS
> > >
> > > Regards,
> > > Hang
> > >
> > > Yong Zhang  于2023年8月31日周四 19:14写道：
> > > >
> > > > Hi everyone,
> > > > Please review and vote on the release candidate #0 for the version
> > 4.16.3,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The complete staging area is available for your review, which includes:
> > > > * Release notes [1]
> > > > * The official Apache source and binary distributions to be deployed to
> > > > dist.apache.org [2]
> > > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > > * Source code tag "release-4.5.0" [4] with git sha
> > > > 5ca147b4a85a1044967002b3f389d17e468a264d
> > > >
> > > > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > >
> > > > Please download these packages and review this release candidate:
> > > >
> > > > - Review release notes
> > > > - Download the source package (verify shasum, and asc) and follow the
> > > > instructions to build and run the bookkeeper service.
> > > > - Download the binary package (verify shasum, and asc) and follow the
> > > > instructions to run the bookkeeper service.
> > > > - Review maven repo, release tag, licenses, and any other things you
> > think
> > > > it is important to a release.
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Release Manager
> > > >
> > > > [1]
> > > >
> > https://github.com/apache/bookkeeper/pull/4065/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53R4
> > > > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.3-rc0/
> > > > [3]
> > > >
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1089
> > > > [4] https://github.com/apache/bookkeeper/releases/tag/v4.16.3-rc0
> > > > [5] 5ca147b4a85a1044967002b3f389d17e468a264d
> >

Re: [DISCUSS] BookKeeper 4.16.3 release

[Enrico Olivelli]

+1

thanks

Enrico

Il giorno mar 29 ago 2023 alle ore 09:14 Hang Chen
 ha scritto:
>
> +1 for the new release.
>
> Please include the following two PRs to 4.16.3, thanks.
> https://github.com/apache/bookkeeper/pull/4041
> https://github.com/apache/bookkeeper/pull/4058
>
>
> Thanks,
> Hang
>
> Yong Zhang  于2023年8月29日周二 15:09写道：
> >
> > Hi everyone,
> >
> > Because the last release was published with arm, it causes the native lib
> > only can run on the arm platform which breaks the amd64 supports.
> > We would like to publish 4.16.3 to support the amd64 platform.
> >
> > See more context about the platform issue:
> > https://lists.apache.org/thread/wkn453lrx3n205n21zk5lhkq3q1bqsdb
> >
> > Here are the PRs for 4.16.3:
> > https://github.com/apache/bookkeeper/issues?q=label%3Arelease%2F4.16.3+is%3Aclosed+
> >
> > If you have PRs that want to be included in this release, please
> > comment on that PR or tag the label `release/4.16.3`.
> >
> > After those PRs merged, I would like to volunteer to release this patch.
> >
> > Thanks,
> > Yong

Re: [VOTE] Release 4.14.8, release candidate #0

[Enrico Olivelli]

+1 (binding)

- checked signatures
- run all the tests on JDK8, Mac M1

Thanks for running the release

Enrico

Il giorno lun 28 ago 2023 alle ore 04:21 Yong Zhang
 ha scritto:
>
> +1 (non-binding)
>
> - Signature and checksum look good
> - Built from archived sources (m1, jdk17, maven 3.8.6)
> - Run standalone and simpletest with the bookie-all package
>
> Thanks,
> Yong
>
> On Fri, 18 Aug 2023 at 17:28, Hang Chen  wrote:
>
> > Hi everyone,
> > Please review and vote on the release candidate #0 for version 4.14.8,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed
> > to dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "v4.14.8-rc0" [4] with git sha [5]
> >
> > BookKeeper's KEYS file contains the PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review the maven repo, release tag, licenses, and any other things
> > you think it is important to release.
> >
> > The vote will be open for at least 72 hours. It is adopted by the
> > majority approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1] https://github.com/apache/bookkeeper/pull/4053/files
> > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.8-rc0/
> > [3]
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1088
> > [4] https://github.com/apache/bookkeeper/tree/v4.14.8-rc0
> > [5] 965c3627328787e2750f41c2a4eda6c1d709d7c6
> >

Re: Force to use linux/amd64 to build release

[Enrico Olivelli]

+1

Thanks
We should cut a new release as soon as possible

Enrico

Il Lun 28 Ago 2023, 10:34 Yong Zhang  ha
scritto:

> Hi everyone,
>
> When releasing the 4.16.2, I did the release with an M1 macbook pro.
> That makes the native io lib compile and publish with the arm platform.
> So some users get the wrong lib when running the bookie with native io.
> https://github.com/apache/bookkeeper/issues/4043
>
> And the `NATIVE_IO.posix_fadvise(fd, offset, len, POSIX_FADV_DONTNEED);`
> also affected because the lib was built for arm.
>
> https://github.com/apache/bookkeeper/blob/ba3127447630315c97594822b1510ecd509f7671/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/PageCacheUtil.java#L97
>
> I made a change on the release Dockerfile to force the image to use the
> linux/amd64 platform, we can ensure the lib is compiled on the amd64.
> https://github.com/apache/bookkeeper/pull/4060
>
> Please help to review that PR. Once it merged, I would like to be a
> volunteer to
> release the 4.16.3 to fix the lib.
>
> Thanks,
> Yong
>

Re: Remove UnderReplicatedLedgersChangedCb in Auditor

[Enrico Olivelli]

+1

Enrico

Il Gio 17 Ago 2023, 10:06 horizonzy  ha scritto:

> In https://github.com/apache/bookkeeper/pull/2805, it introduces
> UnderReplicatedLedgersChangedCb to watch the zookeeper `underreplication`
> path. After a `underreplication` ledger mark is replicated, the
> UnderReplicatedLedgersChangedCb will call back.
>
> In the callback, it will list all the `unnderreplication` ledgers to record
> the metrics `underReplicatedLedgersGuageValue`. It's a heavy operation for
> zookeeper.
>
> And in the pulsar, it introduces a deadlock after implementing the new API
> LedgerUnderreplicationManager#notifyUnderReplicationLedgerChanged
> in PulsarLedgerUnderreplicationManager. see
> https://github.com/apache/pulsar/pull/21010.
>
> I would suggest you remove underReplicatedLedgersChangedCb in Auditor. If
> the user wants to record the metrics, a scheduled task will be better than
> a zk watch event to trigger it.
>

Re: BP62-New API for batched reads

[Enrico Olivelli]

Hello,
The proposal is well written, I have no questions.
We have been waiting for this feature for long time.

I am supporting it

Thanks
Enrico

Il Mar 15 Ago 2023, 04:28 horizonzy  ha scritto:

> Hi, everyone:
> There is a proposal about batched reading(
> https://github.com/apache/bookkeeper/pull/4051), to introduce it to
> improve
> read performance.
>
> The objective of this proposal is to enhance the performance of entry
> reading by introducing a batch entry reading protocol that takes into
> account the expected count and size of entries.
> 1. Optimize entry reading performance: By reading multiple entries in a
> single RPC request, the network communication and RPC call
> overhead can be reduced, thereby optimizing the reading performance.
> 2. Minimize CPU resource consumption: The aggregation of multiple entries
> into a single RPC request can help in reducing the number of requests and
> responses, which in turn can lower the CPU resource consumption.
> 3. Streamline client code: The ability to read entries based on the
> anticipated count or size, such as Apache Pulsar's approach of calculating
> the start and end entry IDs for each read request based on the average size
> of past entries, can add unnecessary complexity to the implementation and
> can't guarantee reliable behavioral outcomes.
>
> Here is the output of the BookKeeper perf tool with ensemble=1, write=1,
> and ack=1.
> Batch(100): Read 1000100 entries in 8904ms
> Batch(500): Read 1000500 entries in 12182ms
> Non-Batch: Read 1000130 entries in 199928ms
>
> If you have any questions, feel free to talk about it. Thanks!
>

Re: Data loss problem when DefaultEntryLogger switch to DirectEntryLogger, then switch back to DefaultEntryLogger.

[Enrico Olivelli]

Il giorno lun 31 lug 2023 alle ore 02:40 Yan Zhao
 ha scritto:
>
> I am concerned about a possible scenario where a user tries to use DirectIO, 
> runs it for a period of time, and experiences some problem on the bookie 
> server side, but is unsure of the cause and suspects it may be related to 
> DirectIO. As a result, the user disables DirectIO and restarts. In this case, 
> data may be lost.

You should never do this in production.

BTW do you have a possible solution?

Enrico

>
>
>
> On 2023/07/30 17:40:25 Enrico Olivelli wrote:
> > I think that you should not do this kind of things.
> > You should start a new bookie with the new configuration.
> >
> > It is not generally possible to guarantee compatibility between different
> > implementations.
> >
> > I tend to close the case as 'won't fix'
> >
> >
> > Enrico
> >
> >
> > Il Dom 30 Lug 2023, 13:00 horizonzy  ha scritto:
> >
> > > Hi, the followers.
> > > I would need to talk about a serious matter. Especially, if you use
> > > DirectEntryLogger in a production environment.
> > >
> > > Switching from DefaultEntryLogger to DirectEntryLogger and switching back
> > > from DirectEntryLogger to DefaultEntryLogger may result in data loss.
> > >
> > > Here is the detailed issue
> > > https://github.com/apache/bookkeeper/issues/4040
> > >
> >

Re: Improve DefaultEntryLogger read performance

[Enrico Olivelli]

Il Dom 30 Lug 2023, 09:19 horizonzy  ha scritto:

> When reading data from an EntryLog file, we create a BufferedReadChannel.
> In the size() method of BufferedReadChannel, FileChannelImpl.size() is
> called, which is a time-consuming operation.
>
> If the EntryLog file has been archived, it is unnecessary to call
> FileChannelImpl.size() every time we read from it. We only need to call it
> once, cache the result, and reuse it.
>
> This improvement can benefit two cases.
> 1. DefaultEntryLogger#scanEntryLog, it can help speed up compaction.
> 2. BufferedReadChannel#read, It can speed up reading entries from the
> EntryLog file.
>


I agree with this proposal.
I have left one little comment in the PR


Thanks

Enrico


>
> Performance Test.
> The entry log f file holds 1GB data.
> -rw-r--r--@  1 horizon  staff   1.0G  7 26 17:47 f.log
>
> 1. Test DefaultEntryLogger.scanEntryLog (10x performance improvement)
> ```
> public static void main(String[] args) throws IOException {
> ServerConfiguration serverConfiguration = new
> ServerConfiguration();
> serverConfiguration.setLedgerDirNames(new
> String[]{"/Users/horizon/Downloads/bk1/bk-data"});
> DefaultEntryLogger defaultEntryLogger = new
> DefaultEntryLogger(serverConfiguration);
> long l = System.currentTimeMillis();
> AtomicInteger entryCount = new AtomicInteger();
> defaultEntryLogger.scanEntryLog(15, new EntryLogScanner() {
> @Override
> public boolean accept(long ledgerId) {
> return true;
> }
>
> @Override
> public void process(long ledgerId, long offset, ByteBuf entry)
> throws IOException {
> entryCount.incrementAndGet();
> }
> });
> long spend = System.currentTimeMillis() - l;
> System.out.println("Scan entry end, entry count: " +
> entryCount.get());
> System.out.println("Spend: " + spend);
> }
> ```
>
> Before:
> Scan entry end, entry count: 17886769
> Spend: 37273
>
>
> After:
> Scan entry end, entry count: 17886769
> Spend: 3363
>
> 2. Test DefaultEntryLogger.readFromLogChannel (30x performance improvement)
> ```
> public static void main(String[] args) throws IOException {
> ServerConfiguration serverConfiguration = new
> ServerConfiguration();
> serverConfiguration.setLedgerDirNames(new
> String[]{"/Users/horizon/Downloads/bk1/bk-data"});
> DefaultEntryLogger defaultEntryLogger = new
> DefaultEntryLogger(serverConfiguration);
> long l = System.currentTimeMillis();
> BufferedReadChannel channel =
> defaultEntryLogger.getChannelForLogId(15);
> int pos = 0;
> //Read total 1gb data from the BufferedReadChannel.
> long gb = 1024 * 1024 * 1024;
> ByteBuf byteBuf = ByteBufAllocator.DEFAULT.heapBuffer(10);
> while (pos < gb) {
> defaultEntryLogger.readFromLogChannel(15, channel, byteBuf, 0);
> pos += byteBuf.readableBytes();
> byteBuf.clear();
> }
> long spend = System.currentTimeMillis() - l;
> System.out.println("Spend: " + spend);
> }
> ```
> Before:
> Spend: 69810
>
> After:
> Spend: 2338
>

Re: Data loss problem when DefaultEntryLogger switch to DirectEntryLogger, then switch back to DefaultEntryLogger.

[Enrico Olivelli]

I think that you should not do this kind of things.
You should start a new bookie with the new configuration.

It is not generally possible to guarantee compatibility between different
implementations.

I tend to close the case as 'won't fix'


Enrico


Il Dom 30 Lug 2023, 13:00 horizonzy  ha scritto:

> Hi, the followers.
> I would need to talk about a serious matter. Especially, if you use
> DirectEntryLogger in a production environment.
>
> Switching from DefaultEntryLogger to DirectEntryLogger and switching back
> from DirectEntryLogger to DefaultEntryLogger may result in data loss.
>
> Here is the detailed issue
> https://github.com/apache/bookkeeper/issues/4040
>

[ANNOUNCE] New committer Wenbing Shen

[Enrico Olivelli]

Hello Bookkeepers,

The Project Management Committee (PMC) for Apache BookKeeper
has invited Wenbing Shen to become a committer and we are pleased
to announce that they have accepted.

Being a committer enables easier contribution to the
project since there is no need to go via the patch
submission process. This should enable better productivity.

For more information about the ASF works you can follow this link
https://www.apache.org/foundation/how-it-works.html

Please join me and congratulate Wenbing Shen !

Enrico Olivelli

Re: [VOTE] Release 4.16.2, release candidate #1

[Enrico Olivelli]

+1 (binding)

- verified checksums and sigs
- run all the tests locally (JDK11 on Mac M1)


Enrico

Il giorno mer 28 giu 2023 alle ore 20:22 Andrey Yegorov
 ha scritto:
>
> +1 (binding)
>
> * localbookie and standalone bookie start ok
> * built from archived sources (mac intel, jdk 11)
> * built from tag
>
>
> On Mon, Jun 26, 2023 at 12:17 AM Yong Zhang 
> wrote:
>
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the version 4.16.2,,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "release-4.5.0" [4] with git sha
> > 0465f7e6ebc2289334cc97b06fbfb0a8f07e2e8b
> >
> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review maven repo, release tag, licenses, and any other things you think
> > it is important to a release.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1]
> >
> > https://github.com/apache/bookkeeper/pull/4002/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53R4
> > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.2-rc1/
> > [3]
> >
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1086/
> > [4] https://github.com/apache/bookkeeper/releases/tag/v4.16.2-rc1
> >
>
>
> --
> Andrey Yegorov

Re: [DISCUSS] Upgrade RocksDB version to 6.29.4.1 for branch-4.14

[Enrico Olivelli]

Hang,

Il giorno dom 25 giu 2023 alle ore 09:47 Hang Chen
 ha scritto:
>
> Hi guys,
>
> More than one year ago, we had a discussion[1] about reverting the
> RocksDB version for branch-4.14, and the main concern is that RocksDB
> 6.17.3 breaks runtime compatibility with older versions and it
> requires the code to be recompiled. Lari did a great investigation and
> you can find more details in this Apache Pulsar pull
> https://github.com/apache/pulsar/pull/14962. In the discussion, we
> have two solutions, and we choose solution 1 in the end.
> 1) Do not upgrade RocksDB in BookKeeper 4.14.5 and give the user which
> uses RocksDB an easy path to upgrade from BK 4.14.4 to 4.14.5
> 2) Upgrade RocksDB and requires to all the BK users to pay attention
> to the RocksDB version they have in the classpath. The risk is to have
> runtime errors which will require them to rebuild the application and
> redeploy their services.
>
> However, we found two issues about the current RocksDB version
> (6.16.4) on branch-4.14
> - In our production environment, we found both RocksDB 6.10.2 and
> 6.16.4 has native memory leak issue[2] and after upgraded the RocksDB
> version to 6.29.4.1, the memory leak issue was fixed.
> - We have upgraded the RocksDB version to 7.9.2 since 4.16.0, and the
> RocksDB can only rollback to 6.27+. It means BookKeeper 4.16.0+ can
> rollback to 4.15.x (RocksDB 6.29.4.1) but can't rollback to 4.14.x
> (RocksDB 6.16.4) if we don't upgrade the RocksDB version to 6.29.4.1
> for branch-4.14
>
> The main reason for rolling back RocksDB version for branch-4.14 is
> that RocksDB 6.17.3 breaks runtime compatibility with older versions
> and it requires the code to be recompiled.

If this is the only reason then I think that it is not a problem, because
BookKeeper users don't use RocksDB directly, the RocksDB APIs are not part of
BK APIs.

So one question is:
if I am on BK 4.14.7 and I upgrade to 4.14.8 with the new RocksDB
version 6.29.4.1,
can I rollback to 4.14.7 if I have some problems ?

if the answer is "yes" then I am supportive of this upgrade

Thanks for pushing forward this initiative

Enrico

 In my opinion, if users
> upgrade BookKeeper 4.14.7 to 4.14.8, which upgraded the RocksDB
> version from 6.16.4 to 6.29.4.1, they need to use the new BookKeeper
> package which has been compiled with new RocksDB version instead of
> replace the BookKeeper-server library directly. So it is safe to
> upgrade the RocksDB version. I have pushed one PR[4] to upgrade the
> RocksDB version to 6.29.4.1 for branch-4.14.
>
> Any concerns?
>
> Thanks,
> Hang
>
> [1] https://lists.apache.org/thread/dsdltmoy5ggqx4oj7hrt13hqok80d4jf
> [2] https://github.com/apache/bookkeeper/issues/3507
> [3] https://github.com/apache/bookkeeper/issues/3734#issuecomment-1407626941
> [4] https://github.com/apache/bookkeeper/pull/3947

Re: [DISCUSS] BookKeeper 4.16.2 release

[Enrico Olivelli]

+1

Enrico

Il Mer 14 Giu 2023, 09:35 Wenbing Shen  ha
scritto:

> +10
>
> I noticed https://github.com/apache/bookkeeper/pull/3979 that issue was
> introduced in bookkeeper-4.16.0 by
> https://github.com/apache/bookkeeper/pull/3847,
> and in pulsar-3.0. 0 version introduced bk-4.16.1.
>
> Our online business runs a pulsar-3.0.0 cluster, we and other pulsar-3.0.0
> users urgently need bk-4.16.2.
>
> Thanks,
> wenbingshen
>
> Hang Chen  于2023年6月13日周二 09:30写道：
>
> > Hi everyone,
> >
> > We have found one critical bug[1] in BookKeeper 4.16.1, and I would
> > like to discuss the starting 4.16.2 release to include some critical
> > bug fixes and improvements, such as [1] and [2]
> >
> > Here are the PRs for 4.16.2:
> >
> >
> https://github.com/apache/bookkeeper/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F4.16.2
> >
> > If you have PRs that want to be included in this release, please
> > comment on that PR or tag the label `release/4.16.2`.
> >
> > After those PRs merged, I would like to volunteer to release this patch.
> >
> > Thanks,
> > Hang
> >
> > [1] https://github.com/apache/bookkeeper/pull/3979
> > [2] https://github.com/apache/bookkeeper/pull/3983
> >
>

Re: [DISCUSS] BookKeeper 4.14.8 release

[Enrico Olivelli]

+1

thanks
Enrico

Il giorno mar 30 mag 2023 alle ore 11:09 Hang Chen
 ha scritto:
>
> Hi everyone,
>
> BookKeeper 4.14.7 was released more than 3 months, and I would like to
> discuss the starting 4.14.8 release to include some critical bug fixes
> and improvements, such as [1] and [2]
>
> Here are the PRs for 4.14.8:
> https://github.com/apache/bookkeeper/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F4.14.8
>
> If you have PRs that want to be included in this release, please
> comment on that PR or tag the label `release/4.14.8`.
>
> After those PRs merged, I would like to volunteer to release this patch.
>
> Thanks,
> Hang
>
> [1] https://github.com/apache/bookkeeper/pull/3959
> [2] https://github.com/apache/bookkeeper/pull/3917

Re: Introduce cold ledger storage layer.

[Enrico Olivelli]

Gavin,
This idea looks promising, as Dave mentioned, it could anticipate
adding support for moving cold data to cheaper cloud storage

Enrico

Il giorno ven 26 mag 2023 alle ore 06:17 Dave Fisher
 ha scritto:
>
>
>
> Sent from my iPhone
>
> > On May 25, 2023, at 7:37 PM, Gavin gao  wrote:
> >
> > In a typical bookkeeper deployment, SSD disks are used to store Journal log
> > data, while HDD disks are used to store Ledger data.
>
> What is used is a deployment choice. I know that when OMB is run locally 
> attached SSDs are used for both.
>
> I do agree that the choice of SSD and HDD disks can impact Bookkeeper 
> performance. Increasing IOPs and throughput will impact performance 
> significantly. For example in AWS a default gp3 attached disk will have large 
> latencies, but upping performance will give performance maybe 4x slower than 
> an SSD locally attached.
> > Data writes are
> > initially stored in memory and then asynchronously flushed to the HDD disk
> > in the background. However, due to memory limitations, the amount of data
> > that can be cached is restricted. Consequently, requests for historical
> > data ultimately rely on the HDD disk, which becomes a bottleneck for the
> > entire Bookkeeper cluster. Moreover, during data recovery processes
> > following node failures, a substantial amount of historical data needs to
> > be read from the HDD disk, leading to the disk's I/O utilization reaching
> > maximum capacity and resulting in significant read request delays or
> > failures.
> >
> > To address these challenges, a new architecture is proposed: the
> > introduction of a disk cache between the memory cache and the HDD disk,
> > utilizing an SSD disk as an intermediary medium to significantly extend
> > data caching duration. The data flow is as follows: journal -> write cache
> > -> SSD cache -> HDD disk. The SSD disk cache functions as a regular
> > LedgerStorage layer and is compatible with all existing LedgerStorage
> > implementations.
>
> A different way to look at this is to consider the cold layer as being 
> optional and within HDD or even S3. In S3 you could have advantages with 
> recovery into different AZs. You could also significantly improve replay.
> > The following outlines the process:
> >
> >   1. Data eviction from the disk cache to the Ledger data disk occurs on a
> >   per-log file basis.
> >   2. A new configuration parameter, diskCacheRetentionTime, is added to
> >   set the duration for which hot data is retained. Files with write
> >   timestamps older than the retention time will be evicted to the Ledger 
> > data
> >   disk.
>
> If you can adjust this to use a recent use approach then very long ledger can 
> be easily read with predictively moving ledgers from cold to hot.
>
> >   3. A new configuration parameter, diskCacheThreshold, is added. If the
> >   disk cache utilization exceeds the threshold, the eviction process is
> >   accelerated. Data is evicted to the Ledger data disk based on the order of
> >   file writes until the disk space recovers above the threshold.
> >   4. A new thread, ColdStorageArchiveThread, is introduced to periodically
> >   evict data from the disk cache to the Ledger data disk.
>
> Anotger thread is also needed - ColdStorageRetrievalThread.
>
> Just some thoughts.
>
> Best,
> Dave

Re: 回复：[Discuss] Bookie may lose data even though we turn on fsync for the journal

[Enrico Olivelli]

Il giorno lun 15 mag 2023 alle ore 10:50 Hang Chen
 ha scritto:
>
> > When flushing, only the lastMark value will be persisted to the file, but 
> > the lastMark value will not be updated.
> > The lastMark value is updated only when the ForceWriteRequest completes. So 
> > when the flush is triggered here, the position of lastMark is not 100MB's 
> > offset
> Yes, you are right. But the persisted lastMark value will be near to
> the latest lastMark value. Even though it is not 100MB's offset, it
> may be 100MB - 1 offset.
>
>
> The root cause of this bug is that EntryLogger1 triggers a checkpoint
> when its write cache is full, updating both EntryLogger1  and
> EntryLogger2's `lastMark` position. However, EntryLogger2's data may
> still be in WriteCache, which may lead to data loss when the bookie
> shutdown will `kill -9`
>
> There are two solutions for this bug.
>  Update `lastMark` position individually.
> - When EntryLogger1 triggers the checkpoint, we only update
> EntryLogger1's `lastMark` position instead of updating EntryLogger2's
> `lastMark` position at the same time.
> - When SyncThread triggers the checkpoint, we update all the
> EntryLoggers' `lastMark` positions.
> - When determining whether a journal file can be deleted, we should
> get the smallest `lastMark` position among all the writeable
> EntryLoggers, and delete the journal files which less than the
> smallest `lastMark` position.
> - When replying to the journal on bookie startups, we need to get the
> smallest `lastMark` position, and reply to the journal files with this
> position, otherwise, we will lose data.
>
> However, one case is hard to handle in replying to the journal stage.
> When one ledger disk transfers from ReadOnly to Writable mode, the
> `lastMark` position is an old value. Using the old position to reply
> to the journal files will lead to a target journal file not found
> exception.
>
>  Only update `lastMark` position in SyncThread
> Two places can trigger a checkpoint.
> - The scheduled tasks in SyncThread.doCheckpoint
> - The ledgerDir write-cache full and then flush
>
> The second way is the root cause of data loss if the ledger is
> configured with multiple directories.
> We can turn off the second way's update `lastMark` position operation
> and only make SyncThread update the `lastMark` position in a
> checkpoint when the ledger is configured with multiple directories.
>
> This is the simplest way to fix this bug, but it has two drawbacks.
> - The `lastMark` position updates depend on SyncThread doing
> checkpoint intervals. In Pulsar, the default interval is 60s. It means
> the journal file expires with at least 60s
> - The bookie startup replying to the journal depends on the `lastMark`
> position. It means the journal will reply to at least 60s journal data
> before the start-up is complete. It may lead to the bookie start-up
> speed slowing down.
>
> IMO, the above two drawbacks can be acceptable compared to data loss.

I agree

Thanks
Enrico

>
> Thanks,
> Hang
>
> Gavin Gao  于2022年6月10日周五 13:31写道：
> >
> > The problem is:
> > flush when writeCache is full is per ledger disks independent, but they 
> > share the same journal disk lastMark value.
> >
> > On 2022/06/07 04:16:58 lordcheng10 wrote:
> > >  In flushing
> > > the write cache, it will trigger a checkpoint to mark the journal’s
> > > lastMark position (100MB’s offset)
> > >
> > >
> > > When flushing, only the lastMark value will be persisted to the file, but 
> > > the lastMark value will not be updated.
> > > The lastMark value is updated only when the ForceWriteRequest completes. 
> > > So when the flush is triggered here, the position of lastMark is not 
> > > 100MB's offset
> > >
> > >
> > > I’m not sure whether I missed some logic.
> > >
> > > --原始邮件--
> > > 发件人: "Hang Chen" > > 发送时间: 2022年5月30日(星期一) 上午9:21
> > > 收件人: "dev" > > 主题: [Discuss] Bookie may lose data even though we turn on fsync for the 
> > > journal
> > >
> > >
> > >
> > > We found one place where the bookie may lose data even though we turn
> > > on fsync for the journal.
> > > Condition:
> > > - One journal disk, and turn on fsync for the journal
> > > - Configure two ledger disks, ledger1, and ledger2
> > >
> > > Assume we write 100MB data into one bookie, 70MB data written into
> > > ledger1's write cache, and 30 MB data written into ledger2's write
> > > cache. Ledger1's write cache is full and triggers flush. In flushing
> > > the write cache, it will trigger a checkpoint to mark the journal’s
> > > lastMark position (100MB’s offset) and write the lastMark position
> > > into both ledger1 and ledger2's lastMark file.
> > >
> > > At this time, this bookie shutdown without flush write cache, such as
> > > shutdown by `kill -9` command, and ledger2's write cache (30MB)
> > > doesn’t flush into ledger disk. But ledger2's lastMark position which
> > > persisted into lastMark file has been updated to 100MB’s offset.
> > >
> > > When the 

Re: [ANNOUNCE] Apache BookKeeper 4.16.0 released

[Enrico Olivelli]

Congratulations !

This is a great step forward !

Looking forward to using DirectIO and Embedded Bookie !

Enrico

Il giorno mar 2 mag 2023 alle ore 09:22 Hang Chen
 ha scritto:
>
> The Apache BookKeeper team is proud to announce Apache BookKeeper
> version 4.16.0.
>
> Apache BookKeeper is a scalable, fault-tolerant, and low-latency
> storage service optimized for real-time workloads. It has been used
> for a fundamental service to build reliable services. It is also the
> log segment store for Apache DistributedLog and the message store for
> Apache Pulsar.
>
> This is the 46rd release of the Apache BookKeeper.
>
> Release 4.16.0 includes multiple important features, improvements, bug
> fixes and some dependencies CVE fixes.
>
> Due to this version having one critical regression in the BookKeeper
> client, Apache BookKeeper users are encouraged to skip this version
> and upgrade to 4.16.1.
>
> The technical details of this release are summarized below.
>
> For BookKeeper release details and downloads, visit:
> https://bookkeeper.apache.org/releases
>
> BookKeeper 4.16.0 Release Notes are at:
> https://bookkeeper.apache.org/release-notes/#4160
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The BookKeeper Team

Re: [VOTE] Release 4.16.1, release candidate #0

[Enrico Olivelli]

+1 (binding)

Built locally
Verified that with the only commit added in respect to 4.16.0 the
regression discovered in Pulsar is fixed


Thanks for the quick release

Enrico

Il Sab 15 Apr 2023, 06:18 Hang Chen  ha scritto:

> Hi everyone,
> Please review and vote on the release candidate #0 for the version 4.16.1,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.16.1-rc0" [4] with git sha [5]
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/bookkeeper/pull/3922/files
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.1-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1084/
> [4] https://github.com/apache/bookkeeper/tree/v4.16.1-rc0
> [5] 8122a87c78a0159e422fc0baac43201eda44ae94
>

Re: [VOTE] Release 4.16.0, release candidate #2

[Enrico Olivelli]

+1 (binding)

- Built and run tests locally on jdk11 on M1
- Run smoke tests
- Verified checksum and signatures


Enrico

Il Sab 1 Apr 2023, 08:35 Andrey Yegorov  ha
scritto:

> > Have you ever run `mvn clean insteall -DskipTests` before
> running the tests
>
> of course.
>
> as listed above:
> $ git checkout v4.16.0-rc2
> $ mvn clean install -DskipTests
> $ mvn test
>
> On Fri, Mar 31, 2023 at 8:19 PM Hang Chen  wrote:
>
> > > I added the logs to the issue.
> >
> > It looks like the loopback address problem, and the Bookie server does
> > not allow loopback interfaces as the primary interface. You can set
> > allowLoopback to true and rerun the test.
> >
> >
> > > FWIW I ran the tests from the command line, no from IntelliJ.
> >
> > >$ mvn -v
> > >Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
> > >Maven home: /usr/local/Cellar/maven/3.8.6/libexec
> > >Java version: 11.0.11, vendor: AdoptOpenJDK, runtime:
> >
> >
> >/Users/andreyyegorov/Library/Java/JavaVirtualMachines/adopt-openjdk-11.0.11/Contents/Home
> > >Default locale: en_US, platform encoding: UTF-8
> > >OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac"
> >
> > I ran the test from the command line without any changes, and the tests
> > passed.
> > $ mvn test -Dtest=DbLedgerStorageWithDirectEntryLoggerTest
> > [INFO] Running
> >
> org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageWithDirectEntryLoggerTest
> > [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time
> > elapsed: 16.021 s - in
> >
> >
> org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageWithDirectEntryLoggerTest
> >
> > $ mvn -v
> > Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
> > Maven home: /Users/hangc/Cloud/apache-maven-3.8.6
> > Java version: 11.0.12, vendor: Oracle Corporation, runtime:
> > /Library/Java/JavaVirtualMachines/jdk-11.0.12.jdk/Contents/Home
> > Default locale: en_CN, platform encoding: UTF-8
> > OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac"
> >
> > @Andrey Have you ever run `mvn clean insteall -DskipTests` before
> > running the tests?
> >
> > Thanks,
> > Hang
> >
> > Matteo Merli  于2023年4月1日周六 01:32写道：
> > >
> > > +1 binding
> > >
> > > * Checked signatures
> > > * Check localbookie and local perf tool
> > >
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> > >
> > > On Wed, Mar 29, 2023 at 7:06 AM Hang Chen  wrote:
> > >
> > > > Hi everyone,
> > > > Please review and vote on the release candidate #2 for version
> 4.16.0,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * Release notes [1]
> > > > * The official Apache source and binary distributions to be deployed
> to
> > > > dist.apache.org [2]
> > > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > > * Source code tag "v4.16.0-rc2" [4] with git sha [5]
> > > >
> > > > BookKeeper's KEYS file contains the PGP keys we used to sign this
> > release:
> > > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > >
> > > > Please download these packages and review this release candidate:
> > > >
> > > > - Review release notes
> > > > - Download the source package (verify shasum, and asc) and follow the
> > > > instructions to build and run the bookkeeper service.
> > > > - Download the binary package (verify shasum, and asc) and follow the
> > > > instructions to run the bookkeeper service.
> > > > - Review the maven repo, release tag, licenses, and any other things
> > you
> > > > think
> > > > it is important to release.
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by the
> > majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Release Manager
> > > >
> > > > [1] https://github.com/apache/bookkeeper/pull/3882/files
> > > > [2]
> > > >
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.0-rc2/
> > > > [3]
> > > >
> >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1083/
> > > > [4] https://github.com/apache/bookkeeper/tree/v4.16.0-rc2
> > > > [5] 8890a61418e3961e108b3c2e8ea176691e5f
> > > >
> >
>
>
> --
> Andrey Yegorov
>

Re: [VOTE] Release 4.16.0, release candidate #1

[Enrico Olivelli]

+1 (binding)

- Built from sources and run all the tests (it is the first time I see
all tests passing the first time I launch them !!)
- Verified signatures and checksums
- Run some smoke tests on JDK11/Mac M1


Enrico

Il giorno lun 27 mar 2023 alle ore 03:41 Hang Chen
 ha scritto:
>
> I have created one PR[1] to upgrade the BookKeeper dependency to
> 4.16.0 for Apache Pulsar, and all the tests have been passed.
>
> Thanks,
> Hang
>
>
> [1] https://github.com/hangc0276/pulsar/pull/14
>
> Hang Chen  于2023年3月26日周日 13:26写道：
> >
> > Hi everyone,
> > Please review and vote on the release candidate #1 for version 4.16.0,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "v4.16.0-rc1" [4] with git sha [5]
> >
> > BookKeeper's KEYS file contains the PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review the maven repo, release tag, licenses, and any other things you 
> > think
> > it is important to release.
> >
> > The vote will be open for at least 72 hours. It is adopted by the
> > majority approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1] https://github.com/apache/bookkeeper/pull/3882/files
> > [2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.0-rc1/
> > [3] 
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1082/
> > [4] https://github.com/apache/bookkeeper/tree/v4.16.0-rc1
> > [5] 5474a84549de8e02470014b2c7462eb97e71034e

[ANNOUNCE] Hang Chen on the Apache BookKeeper PMC

[Enrico Olivelli]

The Project Management Committee (PMC) for Apache BookKeeper
has invited Hang Chen to become a PMC member and we are pleased
to announce that they have accepted.

Being a committer enables easier contribution to the
project since there is no need to go via the patch
submission process. This should enable better productivity.
A PMC member helps manage and guide the direction of the project.

More information about how the ASF works can be found here
https://www.apache.org/foundation/how-it-works.html

Please join me and celebrate !

Congratulations Hang !

Enrico Olivelli

Re: [VOTE] Release 4.15.4, release candidate #0

[Enrico Olivelli]

+1 (binding)

- Checked signatures/checksums
- Built locally from sources, run some tests

Enrico


There is one test that always fails on my machine:
[ERROR] Failures:
[ERROR] 
org.apache.bookkeeper.test.BookieZKExpireTest.testBookieServerZKSessionExpireBehaviour
[ERROR]   Run 1:
BookieZKExpireTest.testBookieServerZKSessionExpireBehaviour:182 Bookie
should shutdown on losing zk session
[ERROR]   Run 2:
BookieZKExpireTest.testBookieServerZKSessionExpireBehaviour:182 Bookie
should shutdown on losing zk session
[ERROR]   Run 3:
BookieZKExpireTest.testBookieServerZKSessionExpireBehaviour:182 Bookie
should shutdown on losing zk session
[INFO]

Il giorno gio 9 mar 2023 alle ore 01:30 Andrey Yegorov
 ha scritto:
>
> +1 (binding)
>
> * reviewed release notes
> * tested standalone & localbookie
> * built from sources (Mac, OpenJDK 17)
> * ran some tests for bookkeeper-server
>
> On Wed, Mar 1, 2023 at 11:30 PM Yong Zhang 
> wrote:
>
> > Hi everyone,
> > Please review and vote on the release candidate #1 for version 4.15.4,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "v4.15.4-rc0" [4] with git sha [5]
> >
> > BookKeeper's KEYS file contains the PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review the maven repo, release tag, licenses, and any other things you
> > think
> > it is important to release.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1] https://github.com/apache/bookkeeper/pull/3831/files
> > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.4-rc0/
> > [3]
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1080
> > [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.4-rc0
> > [5] 44f87efb8121a9f5428e7ddc70656aee85f563bf
> >
>
>
> --
> Andrey Yegorov

Re: [VOTE] BP-61: Revert BP-59 to release ByteBuf using ReferenceCountUtil.release() instead of ReferenceCountUtil.safeRelease()

[Enrico Olivelli]

+1 (binding)

Enrico

Il giorno mar 21 feb 2023 alle ore 05:38 Hang Chen
 ha scritto:
>
> Hi, All
>
> After a fascinating discussion, I would start the vote for BP-61.
>
> We will replace the `ReferenceCountUtil.safeRelease()` with
> `ReferenceCountUtil.release()`.
>
> Please leave +1/-1 in this thread to join the vote. And feel free to
> leave any concerns.
>
> Thanks to you guys.
>
> Best,
> Hang
>
> References:
> - BP: https://github.com/apache/bookkeeper/issues/3792
> - Discussion https://lists.apache.org/thread/9h7v7hs1pfp74mmopqtl7w4z0tq5rqhc

Re: [ANNOUNCE] Apache BookKeeper 4.14.7 released

[Enrico Olivelli]

Congrats !
Enrico

Il giorno lun 20 feb 2023 alle ore 09:53 Hang Chen
 ha scritto:
>
> The Apache BookKeeper team is proud to announce Apache BookKeeper
> version 4.14.7.
>
> Apache BookKeeper is a scalable, fault-tolerant, and low-latency
> storage service optimized for real-time workloads. It has been used
> for a fundamental service to build reliable services. It is also the
> log segment store for Apache DistributedLog and the message store for
> Apache Pulsar.
>
> This is the 44rd release of the Apache BookKeeper.
>
> Release 4.14.7 contains performance and stability improvements and
> various bug fixes.
>
> Apache BookKeeper users are encouraged to upgrade to 4.14.7. The
> technical details of this release are summarized below.
>
> For BookKeeper release details and downloads, visit:
> https://bookkeeper.apache.org/releases
>
> BookKeeper 4.14.7 Release Notes are at:
> https://bookkeeper.apache.org/release-notes/#4147
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The BookKeeper Team

Re: [DISCUSS] BookKeeper 4.15.4 release

[Enrico Olivelli]

Il giorno gio 16 feb 2023 alle ore 10:13 Yong Zhang
 ha scritto:
>
> There have two PRs #3673 <https://github.com/apache/bookkeeper/pull/3673>
> and #3674 <https://github.com/apache/bookkeeper/pull/3674> about
> the BP-59:Refactor ByteBuf
> release method has been cherry-picked into the branch-4.15.
> https://github.com/apache/bookkeeper/commits/branch-4.15
> Because that PIP is still in discussion, I will revert them from branch-4.15

Good idea !

Enrico

> first.
>
> Yong
>
> On Tue, 14 Feb 2023 at 15:21, Enrico Olivelli  wrote:
>
> > Good idea
> >
> > +1
> >
> > Enrico
> >
> > Il Mar 14 Feb 2023, 02:23 Hang Chen  ha scritto:
> >
> > > +1
> > >
> > > -  As we discussed in [2], For the PRs list in [1], I think we can
> > > skip cherry-pick first.
> > > - For PR [3], I removed label `release/4.15.4` due to [4]
> > >
> > > Thanks,
> > > Hang
> > >
> > > [1] https://github.com/apache/bookkeeper/issues/3686
> > > [2] https://lists.apache.org/thread/9h7v7hs1pfp74mmopqtl7w4z0tq5rqhc
> > > [3] https://github.com/apache/bookkeeper/pull/3653
> > > [4] https://github.com/apache/bookkeeper/issues/3734
> > >
> > > Yong Zhang  于2023年2月14日周二 08:40写道：
> > > >
> > > > Hi everyone,
> > > >
> > > > It has been a long time since we did the last release
> > > > from branch-4.15. We made a lot of improvements and
> > > > bug fixes. I'd propose doing a minor release for the
> > > > branch-4.15.
> > > >
> > > > Here is the PRs list for 4.15.4.
> > > > <
> > >
> > https://github.com/apache/bookkeeper/pulls?q=+is%3Apr+label%3Arelease%2F4.15.4+
> > > >
> > > >
> > > >
> > > > If you have PR want to include in this release, please
> > > > comment on that PR or tag the label `release/4.15.4`.
> > > >
> > > > Thanks,
> > > > Yong
> > >
> >

Re: [VOTE] Release 4.14.7, release candidate #1

[Enrico Olivelli]

+1 (binding)
- run tests locally
- built from sources
- verified signatures and checksums
- run Apache RAT for headers
- verified LICENSE using the script that we have in dev/

Thanks for driving the release

Enrico

Il giorno mar 14 feb 2023 alle ore 05:39 Hang Chen
 ha scritto:
>
> I opened one PR[1] to upgrade BookKeeper dependency to 4.14.7 for
> Pulsar branch-2.10 and all tests passed. Please help verify this
> candidate. Thanks a lot.
>
> Thanks,
> Hang
>
> [1] https://github.com/hangc0276/pulsar/pull/9
>
> Hang Chen  于2023年2月13日周一 21:17写道：
> >
> > Hi everyone,
> > Please review and vote on the release candidate #1 for version 4.14.7,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "v4.14.7-rc1" [4] with git sha [5]
> >
> > BookKeeper's KEYS file contains the PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review the maven repo, release tag, licenses, and any other things you 
> > think
> > it is important to release.
> >
> > The vote will be open for at least 72 hours. It is adopted by the majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1] https://github.com/apache/bookkeeper/pull/3770/files
> > [2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.7-rc1/
> > [3] 
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1079
> > [4] https://github.com/apache/bookkeeper/tree/v4.14.7-rc1
> > [5] c22136b03489db1643521f586e9cae2c4a511e10

Re: [DISCUSS] BookKeeper 4.15.4 release

[Enrico Olivelli]

Good idea

+1

Enrico

Il Mar 14 Feb 2023, 02:23 Hang Chen  ha scritto:

> +1
>
> -  As we discussed in [2], For the PRs list in [1], I think we can
> skip cherry-pick first.
> - For PR [3], I removed label `release/4.15.4` due to [4]
>
> Thanks,
> Hang
>
> [1] https://github.com/apache/bookkeeper/issues/3686
> [2] https://lists.apache.org/thread/9h7v7hs1pfp74mmopqtl7w4z0tq5rqhc
> [3] https://github.com/apache/bookkeeper/pull/3653
> [4] https://github.com/apache/bookkeeper/issues/3734
>
> Yong Zhang  于2023年2月14日周二 08:40写道：
> >
> > Hi everyone,
> >
> > It has been a long time since we did the last release
> > from branch-4.15. We made a lot of improvements and
> > bug fixes. I'd propose doing a minor release for the
> > branch-4.15.
> >
> > Here is the PRs list for 4.15.4.
> > <
> https://github.com/apache/bookkeeper/pulls?q=+is%3Apr+label%3Arelease%2F4.15.4+
> >
> >
> >
> > If you have PR want to include in this release, please
> > comment on that PR or tag the label `release/4.15.4`.
> >
> > Thanks,
> > Yong
>

Re: Prometheus metrics provider > upgrade dataskeches lib

[Enrico Olivelli]

Diego,

Il giorno lun 13 feb 2023 alle ore 11:37 Diego Salvi
 ha scritto:
>
> Hi,
> I was looking into prometheus-metrics-provider dependencies and I found a
> dependency to com.yahoo.datasketches:sketches-core:0.8.3.
> This is really a very old implementation pre apache incubating. Currently
> it has been ported from yahoo to apache with a current version 3.3.0.
> In the meantime it changed packages too.
> Wouldn't it be better to upgrade the dependency to a stable one?

I agree.
Could you send a patch ?
Maybe we are on time to deliver the change for 4.16.0

Enrico

>
> Diego Salvi

Re: [Discuss] BookKeeper upgrade the RocksDB version to 7.9.x

[Enrico Olivelli]

+1

Enrico

Il giorno mer 8 feb 2023 alle ore 17:44 Andrey Yegorov
 ha scritto:
>
> +1 to the plan.
> We can configure RocksDb to not use kXXH3 in all branches and provide a
> config option for the user to enable it when needed.
>
> On Sun, Jan 29, 2023 at 2:54 AM Hang Chen  wrote:
>
> > Hi guys,
> > When we introduced `deleteRange` for ledger index deletion, we
> > found a segment fault in ledger index deletion [1]. I suggest
> > upgrading the RocksDB version to the latest 7.9.x to see if the
> > RocksDB deleteRange segment fault bug has been fixed. I have tested
> > upgrading the RocksDB version from 6.10.2 to 7.9.2 and rollback to
> > 6.10.2. This is the testing result.
> >
> > - The upgrade process works fine in the following cases
> >   - Pulsar producer keeps producing messages to Pulsar topic
> >   - Pulsar consumer consumes messages from the Pulsar topic, and the
> > Pulsar broker fetches messages from the BookKeeper cluster
> >   - Trigger compaction to cleanup expired ledgers
> >
> > - Rollback RocksDB version from 7.9.2 to 6.10.2, and bookie failed to
> > startup with the following exception.
> >
> > ```java
> > 2023-01-29T17:09:27,794+0800 [main] ERROR
> > org.apache.bookkeeper.server.Main - Failed to build bookie server
> > java.io.IOException: Error open RocksDB database
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.(KeyValueStorageRocksDB.java:200)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.(KeyValueStorageRocksDB.java:89)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:63)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.(LedgerMetadataIndex.java:68)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.(SingleDirectoryDbLedgerStorage.java:170)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:150)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:129)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:819)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:120)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.server.service.BookieService.(BookieService.java:52)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at org.apache.bookkeeper.server.Main.doMain(Main.java:226)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at org.apache.bookkeeper.server.Main.main(Main.java:208)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > Caused by: org.rocksdb.RocksDBException: unknown checksum type 4 in
> > data/bookkeeper/ledgers/current/ledgers/25.sst offset 1078 size 33
> > at org.rocksdb.RocksDB.open(Native Method)
> > ~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
> > at org.rocksdb.RocksDB.open(RocksDB.java:239)
> > ~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.(KeyValueStorageRocksDB.java:197)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > ... 13 more
> > ```
> > The root cause of this exception is that RocksDB 7.9.2 uses `kXXH3`
> > by default, and `kXXH3` is only supported since RocksDB 6.27
> >
> > https://github.com/facebook/rocksdb/blob/79e57a39a33dbe17c8f51167e40e66d6c91f8eb4/include/rocksdb/table.h#L56
> >
> > For the BookKeeper master branch, we have upgraded the RocksDB to
> > `6.29.4.1`, which can support RocksDB upgrade to 7.9.2 and rollback to
> > 6.29.4.1.
> > For the RocksDB < 6.27, we can push a fix to ensure RocksDB 7.9.2 does
> > not use the latest checksum type `kXXH3`
> >
> > I suggest doing this upgradation. Do you have any concerns?
> >
> > [1] https://github.com/apache/bookkeeper/issues/3734
> >
> > Thanks,
> > Hang
> >
>
>
> --
> Andrey Yegorov

Re: [DISCUSS] BookKeeper 4.14.7 release

[Enrico Olivelli]

+1

thanks
Enrico

Il giorno mer 8 feb 2023 alle ore 08:16 Hang Chen
 ha scritto:
>
> Hi everyone,
>
> One core dump issue[1] related to RocksDB was introduced by [2] and
> released in BookKeeper 4.14.6. To fix the RocksDB core dump issue, I
> pushed one PR [3] to revert the [2] to make branch-4.14 stable.
>
> Due to BookKeeper 4.14.6 having the risk of stability, I would like to
> discuss starting the 4.14.7 release for those critical bug fixes.
>
> Here are the PRs for 4.14.7:
> https://github.com/apache/bookkeeper/pulls?q=is%3Apr+label%3Arelease%2F4.14.7+is%3Aclosed
>
> If you have PRs that want to be included in this release, please
> comment on that PR or tag the label `release/4.14.7`.
>
> After those PRs merged, I would like to volunteer to release this patch.
>
> Thanks,
> Hang
>
> [1] https://github.com/apache/bookkeeper/issues/3734
> [2] https://github.com/apache/bookkeeper/pull/3653
> [3] https://github.com/apache/bookkeeper/pull/3768

Re: [DISCUSS] BP-61: Revert BP-59 to release ByteBuf using ByteBuf.release() instead of ReferenceCountUtil.safeRelease()

[Enrico Olivelli]

Il giorno gio 2 feb 2023 alle ore 10:57 Yan Zhao
 ha scritto:
>
> > I think that we must do something, current master branch is not stable.
> >
> > My colleague Massimiliano opened this issue
> > https://github.com/apache/bookkeeper/issues/3751
> >
> > Basically in the current master there is some problem that leads to
> > Netty BytBuf corruption.
> >
> > The problem is solved by reverting this PR
> > https://github.com/apache/bookkeeper/pull/3528 Fix memory leak when
> > reading entry but the connection disconnected
> >
> > Enrico
>
> Hi, Enrico. We found that the pulsar timeout may not be introduced by 
> https://github.com/apache/bookkeeper/pull/3528. Only roll back the pulsar 
> version and still maintain https://github.com/apache/bookkeeper/pull/3528, 
> the problem is also solved.

I am sorry but https://github.com/apache/bookkeeper/issues/3751
doesn't report the full story ( I will ask Massimiliano do add more
logs)

The Bookie operation timed out because on the Bookie side the Bookie
sees a packet that is too big, and the cause is that something (the
Netty ByteBuf) is probably recycled



Enrico

>
> I guess the bk thread hangs for some reason in pulsar, the bk server handles 
> the request successfully and sends the response to the bk client, but the bk 
> thread hangs, it didn't handle the response, so timeout.
>
> Massimiliano is tracing it at pulsar side.

Re: [DISCUSS] BP-61: Revert BP-59 to release ByteBuf using ByteBuf.release() instead of ReferenceCountUtil.safeRelease()

[Enrico Olivelli]

Horizon

Il giorno lun 30 gen 2023 alle ore 11:14 Horizon
<1060026...@qq.com.invalid> ha scritto:
>
> Hi, Enrico. Just confirm, only revert 
> https://github.com/apache/bookkeeper/pull/3528, it work well?

yes
This is the only thing we reverted in our fork

see the history
https://github.com/datastax/bookkeeper/commits/ds-4.14

but I don't have all of the other commits in the history,
so maybe other changes may be bad

Unfortunately currently we (both BookKeeper community and in my
company) don't have
dedicated tests for BookKeeper but only for Pulsar,
that makes things harder to track

probably we should build some automated performance test:
a set of scripts that create a BK cluster  on k8s, run some
benchmarks/heavy load, verify that everything worked well
The workbench should be re-usable by anyone, I am not saying that we
should spend the ASF resources to run
this kind of tests, but that we should provide a reference load test.


Enrico

>
> > 2023年1月30日 16:04，Enrico Olivelli  写道：
> >
> > I think that we must do something, current master branch is not stable.
> >
> > My colleague Massimiliano opened this issue
> > https://github.com/apache/bookkeeper/issues/3751
> >
> > Basically in the current master there is some problem that leads to
> > Netty BytBuf corruption.
> >
> > The problem is solved by reverting this PR
> > https://github.com/apache/bookkeeper/pull/3528 Fix memory leak when
> > reading entry but the connection disconnected
> >
> > Enrico
> >
> > Il giorno lun 30 gen 2023 alle ore 08:04 steven lu
> >  ha scritto:
> >>
> >> There are three ways to write release in current pulsar and bookkeeper
> >> projects:
> >> 1.ByteBuf.release: Call release directly without handling any exceptions;
> >> 2.ReferenceCountUtil.release: return processing status;
> >> 3. ReferenceCountUtil.safeRelease: without return value, print exception
> >> information
> >>
> >> I don't think it is necessary to revert these PRs. We can discuss what is
> >> the correct way to release, and then modify these three ways into the
> >> correct way.
> >>
> >> Hang Chen  于2023年1月30日周一 12:28写道：
> >>
> >>> Hi guys,
> >>>   In BP-59, which was not discussed in the dev mail list and without
> >>> a vote, refactored the ByteBuf release method by
> >>> ReferenceCountUtil.safeRelease() instead of ByteBuf.release().
> >>>
> >>> In the `ReferenceCountUtil.safeRelease()`, it catches the release
> >>> exception. This change can make the ByteBuf release without any
> >>> exceptions and makes the code run well, but it will make bugs hide
> >>> deeper and more challenging to find out.
> >>>   ```java
> >>>   public static void safeRelease(Object msg) {
> >>>try {
> >>>release(msg);
> >>>} catch (Throwable t) {
> >>>logger.warn("Failed to release a message: {}", msg, t);
> >>>}
> >>>}
> >>>   ```
> >>> For example, if there is a bug to release the ByteBuf twice, whose
> >>> refcnt is 1, we can find out the bug quickly if we use
> >>> ByteBuf.release(), but the bug will be hard to find out if we use
> >>> `ReferenceCountUtil.safeRelease()`
> >>>
> >>> There are 12 PRs to refactor the ByteBuf release method, and I suggest
> >>> reverting those PRs. Do you have any ideas?
> >>>
> >>> https://github.com/apache/bookkeeper/pull/3673
> >>> https://github.com/apache/bookkeeper/pull/3674
> >>> https://github.com/apache/bookkeeper/pull/3687
> >>> https://github.com/apache/bookkeeper/pull/3688
> >>> https://github.com/apache/bookkeeper/pull/3689
> >>> https://github.com/apache/bookkeeper/pull/3691
> >>> https://github.com/apache/bookkeeper/pull/3693
> >>> https://github.com/apache/bookkeeper/pull/3694
> >>> https://github.com/apache/bookkeeper/pull/3695
> >>> https://github.com/apache/bookkeeper/pull/3698
> >>> https://github.com/apache/bookkeeper/pull/3700
> >>> https://github.com/apache/bookkeeper/pull/3703
> >>>
> >>>
> >>> Thanks,
> >>> Hang
> >>>
>

Re: [DISCUSS] BP-61: Revert BP-59 to release ByteBuf using ByteBuf.release() instead of ReferenceCountUtil.safeRelease()

[Enrico Olivelli]

I think that we must do something, current master branch is not stable.

My colleague Massimiliano opened this issue
https://github.com/apache/bookkeeper/issues/3751

Basically in the current master there is some problem that leads to
Netty BytBuf corruption.

The problem is solved by reverting this PR
https://github.com/apache/bookkeeper/pull/3528 Fix memory leak when
reading entry but the connection disconnected

Enrico

Il giorno lun 30 gen 2023 alle ore 08:04 steven lu
 ha scritto:
>
> There are three ways to write release in current pulsar and bookkeeper
> projects:
> 1.ByteBuf.release: Call release directly without handling any exceptions;
> 2.ReferenceCountUtil.release: return processing status;
> 3. ReferenceCountUtil.safeRelease: without return value, print exception
> information
>
> I don't think it is necessary to revert these PRs. We can discuss what is
> the correct way to release, and then modify these three ways into the
> correct way.
>
> Hang Chen  于2023年1月30日周一 12:28写道：
>
> > Hi guys,
> >In BP-59, which was not discussed in the dev mail list and without
> > a vote, refactored the ByteBuf release method by
> > ReferenceCountUtil.safeRelease() instead of ByteBuf.release().
> >
> > In the `ReferenceCountUtil.safeRelease()`, it catches the release
> > exception. This change can make the ByteBuf release without any
> > exceptions and makes the code run well, but it will make bugs hide
> > deeper and more challenging to find out.
> >```java
> >public static void safeRelease(Object msg) {
> > try {
> > release(msg);
> > } catch (Throwable t) {
> > logger.warn("Failed to release a message: {}", msg, t);
> > }
> > }
> >```
> > For example, if there is a bug to release the ByteBuf twice, whose
> > refcnt is 1, we can find out the bug quickly if we use
> > ByteBuf.release(), but the bug will be hard to find out if we use
> > `ReferenceCountUtil.safeRelease()`
> >
> > There are 12 PRs to refactor the ByteBuf release method, and I suggest
> > reverting those PRs. Do you have any ideas?
> >
> > https://github.com/apache/bookkeeper/pull/3673
> > https://github.com/apache/bookkeeper/pull/3674
> > https://github.com/apache/bookkeeper/pull/3687
> > https://github.com/apache/bookkeeper/pull/3688
> > https://github.com/apache/bookkeeper/pull/3689
> > https://github.com/apache/bookkeeper/pull/3691
> > https://github.com/apache/bookkeeper/pull/3693
> > https://github.com/apache/bookkeeper/pull/3694
> > https://github.com/apache/bookkeeper/pull/3695
> > https://github.com/apache/bookkeeper/pull/3698
> > https://github.com/apache/bookkeeper/pull/3700
> > https://github.com/apache/bookkeeper/pull/3703
> >
> >
> > Thanks,
> > Hang
> >

Re: [VOTE] Release 4.14.6, release candidate #0

[Enrico Olivelli]

Chen,
I saw on repository.apache.org that we still have a dangling staging
repository for this RC

I have dropped it

Enrico

Il giorno lun 19 dic 2022 alle ore 06:12 Hang Chen
 ha scritto:
>
> Hi guys,
>  One issue is the direct memory metric, incompatible with JDK8
> introduced by [1] and fixed by [2]. I will close this vote and create
> a new candidate soon.
>
> Thanks,
> Hang
>
> [1] https://github.com/apache/bookkeeper/pull/3252
> [2] https://github.com/apache/bookkeeper/pull/3677
>
> Enrico Olivelli  于2022年12月19日周一 05:20写道：
> >
> > +1 (binding)
> >
> > Enrico
> >
> > Il giorno gio 1 dic 2022 alle ore 07:30 Yong Zhang
> >  ha scritto:
> > >
> > > okay. Make sense.
> > >
> > > --
> > > Yong
> > >
> > > On Thu, 1 Dec 2022 at 07:52, Andrey Yegorov 
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > * ran bookkeeper standalone
> > > > * ran bookkeeper localbookie
> > > > * built from sources (mac, intel), ran some tests
> > > > * spot-checked maven repo
> > > >
> > > > @Yong - as I understand, build on M1 is not a regression for 4.14.x so 
> > > > it
> > > > should not be a blocker.
> > > >
> > > > On Mon, Nov 21, 2022 at 4:03 AM Hang Chen  wrote:
> > > >
> > > > > Hi everyone,
> > > > > Please review and vote on the release candidate #0 for version 4.14.6,
> > > > > as follows:
> > > > > [ ] +1, Approve the release
> > > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > > >
> > > > > The complete staging area is available for your review, which 
> > > > > includes:
> > > > > * Release notes [1]
> > > > > * The official Apache source and binary distributions to be deployed 
> > > > > to
> > > > > dist.apache.org [2]
> > > > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > > > * Source code tag "v4.14.6-rc0" [4] with git sha [5]
> > > > >
> > > > > BookKeeper's KEYS file contains the PGP keys we used to sign this
> > > > release:
> > > > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > > >
> > > > > Please download these packages and review this release candidate:
> > > > >
> > > > > - Review release notes
> > > > > - Download the source package (verify shasum, and asc) and follow the
> > > > > instructions to build and run the bookkeeper service.
> > > > > - Download the binary package (verify shasum, and asc) and follow the
> > > > > instructions to run the bookkeeper service.
> > > > > - Review the maven repo, release tag, licenses, and any other things 
> > > > > you
> > > > > think
> > > > > it is important to release.
> > > > >
> > > > > The vote will be open for at least 72 hours. It is adopted by the
> > > > majority
> > > > > approval, with at least 3 PMC affirmative votes.
> > > > >
> > > > > Thanks,
> > > > > Release Manager
> > > > >
> > > > > [1] https://github.com/apache/bookkeeper/pull/3659/files
> > > > > [2]
> > > > > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.6-rc0/
> > > > > [3]
> > > > >
> > > > https://repository.apache.org/content/repositories/orgapachebookkeeper-1076/
> > > > > [4] https://github.com/apache/bookkeeper/tree/v4.14.6-rc0
> > > > > [5] 9f453834c6039fac46dc19130288dbf0b2996f2e
> > > > >
> > > >
> > > >
> > > > --
> > > > Andrey Yegorov
> > > >

Re: BookKeeper CI heads up

[Enrico Olivelli]

Thank you Nicolò,
it is very helpful

Enrico

Il giorno mar 17 gen 2023 alle ore 10:44 Nicolò Boschi
 ha scritto:
>
> Hi folks,
>
> I just merged this pull request that refactors the current CI GIthub
> Actions.
> https://github.com/apache/bookkeeper/pull/3735
>
> We now have one main workflow that handles all the checks.
> The previously opened pull requests are not affected and they don't
> require to re-trigger the CI because the merge rules are not affected (I
> kept the same job names).
>
> Please let me know or open issues if you encounter any issues that might be
> related to the new workflow
>
> Cheers,
> Nicolò Boschi

Re: [VOTE] BP-60 Change PCBC limitStatsLogging default value to true

[Enrico Olivelli]

Hello,
the PR has already been merged.

I am not sure if it makes sense to run the VOTE, in any case this is my +1

+1 (binding)

Enrico

Il giorno mer 11 gen 2023 alle ore 09:23 Wenbing Shen
 ha scritto:
>
> Hi Bookkeeper Community,
> I would like to start a VOTE on " Change PCBC limitStatsLogging default
> value to true."(BP-60)
>
> Here is the design detail: https://github.com/apache/bookkeeper/issues/3494
>
> Discussion thread:
> https://lists.apache.org/thread/tjgwj9b4wg9q5wkd62223chk121gp753
>
> The vote will be open for at least 3 days.
>
> Thanks.
> Wenbing

Re: BP-60:Change PCBC limitStatsLogging default value to true

[Enrico Olivelli]

Hello,
someone merged the PR
https://github.com/apache/bookkeeper/pull/3719

this is a BP, so it needs a formal VOTE

Please hold off from committing PRs related to open BPs

Enrico

Il giorno mer 11 gen 2023 alle ore 08:43 steven lu
 ha scritto:
>
> +1
> Thanks for your great work!
>
> Wenbing Shen  于2022年12月26日周一 17:29写道：
>
> > Hi BookKeepers, I've changed the limitStatsLogging default value to true
> > from false:
> > BP-60 
> >
> > Motivation
> >
> > We have an efficient online bookie cluster with hundreds of bookie nodes
> > deployed on SSD disks.
> >
> > We separate the AutoRecovery cluster and the Bookie cluster for independent
> > deployment.
> >
> > I observed that our AutoRecovery cluster GC is very frequent. After
> > investigation, I found that the limitStatsLogging of the bookkeeper client
> > PCBC is disabled by default, and a large number of channel monitoring
> > indicators are generated. Due to the large number of bookie cluster nodes,
> > this metric data occupies a large amount of heap memory.
> >
> > A single StringWriter object occupies 16MB of memory, of which nearly 70
> > StringWriter objects are waiting for the next GC to be destroyed, occupying
> > 1GB+ heap memory.
> > Proposal
> >
> > In my use, I haven't found any usefulness of these PCBC monitoring metrics
> > data, at least so far, I haven't used it effectively.
> >
> > If our AutoRecovery and Bookie cluster are mixed in one process, these
> > large objects will affect the performance and stability of Bookie cluster.
> >
> > Since I can't find the meaning of these metrics by default, I suggest to
> > adjust the default value of limitStatsLogging to true.
> >
> > Everyone can choose to turn it on or off, but by default, it is difficult
> > for users to find out what effect this parameter will have, so that when
> > their cluster grows to hundreds or thousands, when they realize the problem
> > sometimes, it is necessary to restart hundreds to thousands of bookies in a
> > rolling manner.
> >
> > At the same time, I observed that in pulsar, various monitoring of the
> > bookkeeper client is turned off by default, because they really affect the
> > performance of the pulsar service, which is enough to show that we should
> > try to change it, especially some very redundant metrics created based on
> > channels.
> > Compatibility, Deprecation, and Migration PlanClients that rely on PCBC
> > metrics monitoring need to pay attention to this upgrade, but this will not
> > affect the actual functions of the client, only the metrics data, and users
> > can choose to open it again.
> >
> >
> > What do you think about it?
> >
> > Best.
> > Wenbing
> >

Re: BP-60:Change PCBC limitStatsLogging default value to true

[Enrico Olivelli]

I agree with the change.
I have never used those metrics

Enrico


Il giorno lun 26 dic 2022 alle ore 10:29 Wenbing Shen
 ha scritto:
>
> Hi BookKeepers, I've changed the limitStatsLogging default value to true
> from false:
> BP-60 
>
> Motivation
>
> We have an efficient online bookie cluster with hundreds of bookie nodes
> deployed on SSD disks.
>
> We separate the AutoRecovery cluster and the Bookie cluster for independent
> deployment.
>
> I observed that our AutoRecovery cluster GC is very frequent. After
> investigation, I found that the limitStatsLogging of the bookkeeper client
> PCBC is disabled by default, and a large number of channel monitoring
> indicators are generated. Due to the large number of bookie cluster nodes,
> this metric data occupies a large amount of heap memory.
>
> A single StringWriter object occupies 16MB of memory, of which nearly 70
> StringWriter objects are waiting for the next GC to be destroyed, occupying
> 1GB+ heap memory.
> Proposal
>
> In my use, I haven't found any usefulness of these PCBC monitoring metrics
> data, at least so far, I haven't used it effectively.
>
> If our AutoRecovery and Bookie cluster are mixed in one process, these
> large objects will affect the performance and stability of Bookie cluster.
>
> Since I can't find the meaning of these metrics by default, I suggest to
> adjust the default value of limitStatsLogging to true.
>
> Everyone can choose to turn it on or off, but by default, it is difficult
> for users to find out what effect this parameter will have, so that when
> their cluster grows to hundreds or thousands, when they realize the problem
> sometimes, it is necessary to restart hundreds to thousands of bookies in a
> rolling manner.
>
> At the same time, I observed that in pulsar, various monitoring of the
> bookkeeper client is turned off by default, because they really affect the
> performance of the pulsar service, which is enough to show that we should
> try to change it, especially some very redundant metrics created based on
> channels.
> Compatibility, Deprecation, and Migration PlanClients that rely on PCBC
> metrics monitoring need to pay attention to this upgrade, but this will not
> affect the actual functions of the client, only the metrics data, and users
> can choose to open it again.
>
>
> What do you think about it?
>
> Best.
> Wenbing

Re: [VOTE] Release 4.14.6, release candidate #1

[Enrico Olivelli]

+1 (binding)

- verified checksums, RAT, signatures
- run some smoke tests with the staged binaries

I couldn't build the sources on M1, but this is already known on
branch 4.14.x, so not a problem

Enrico

Il giorno ven 23 dic 2022 alle ore 14:15 Nicolò Boschi
 ha scritto:
>
> +1 non binding
> * Standalone and localbookie
> * Checksum and signature
> * Upgraded in Pulsar 2.10 and ran several integration tests in k8s, no
> regressions found
>
> Nicolò Boschi
>
>
> Il giorno gio 22 dic 2022 alle ore 21:56 Andrey Yegorov <
> andrey.yego...@datastax.com> ha scritto:
>
> > +1 (binding)
> >
> > * ran bookkeeper standalone
> > * ran bookkeeper localbookie
> > * built from sources (mac, intel, jdk 11), ran some tests
> > * spot-checked maven repo
> >
> >
> > On Mon, Dec 19, 2022 at 5:33 PM Hang Chen  wrote:
> >
> > > I opened one PR[1] to upgrade BookKeeper dependency to 4.14.6 for
> > > Pulsar branch-2.10, and all tests passed. Please help verify this
> > > candidate.
> > >
> > > Thanks,
> > > Hang
> > >
> > > [1] https://github.com/hangc0276/pulsar/pull/8
> > >
> > > Hang Chen  于2022年12月19日周一 13:14写道：
> > > >
> > > > Hi everyone,
> > > > Please review and vote on the release candidate #1 for version 4.14.6,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The complete staging area is available for your review, which includes:
> > > > * Release notes [1]
> > > > * The official Apache source and binary distributions to be deployed to
> > > > dist.apache.org [2]
> > > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > > * Source code tag "v4.14.6-rc1" [4] with git sha [5]
> > > >
> > > > BookKeeper's KEYS file contains the PGP keys we used to sign this
> > > release:
> > > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > >
> > > > Please download these packages and review this release candidate:
> > > >
> > > > - Review release notes
> > > > - Download the source package (verify shasum and asc) and follow the
> > > > instructions to build and run the bookkeeper service.
> > > > - Download the binary package (verify shasum and asc) and follow the
> > > > instructions to run the bookkeeper service.
> > > > - Review the maven repo, release tag, licenses, and any other things
> > you
> > > think
> > > > it is essential to release.
> > > >
> > > > The vote will be open for at least 72 hours. The majority adopts it
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Release Manager
> > > >
> > > > [1] https://github.com/apache/bookkeeper/pull/3659/files
> > > > [2]
> > > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.6-rc1/
> > > > [3]
> > >
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1077/
> > > > [4] https://github.com/apache/bookkeeper/tree/v4.14.6-rc1
> > > > [5] a96884ec279f5fbcd628c7ae3791d99a8e19ca69
> > >
> >
> >
> > --
> > Andrey Yegorov
> >

Re: [VOTE] Release 4.14.6, release candidate #0

[Enrico Olivelli]

+1 (binding)

Enrico

Il giorno gio 1 dic 2022 alle ore 07:30 Yong Zhang
 ha scritto:
>
> okay. Make sense.
>
> --
> Yong
>
> On Thu, 1 Dec 2022 at 07:52, Andrey Yegorov 
> wrote:
>
> > +1 (binding)
> >
> > * ran bookkeeper standalone
> > * ran bookkeeper localbookie
> > * built from sources (mac, intel), ran some tests
> > * spot-checked maven repo
> >
> > @Yong - as I understand, build on M1 is not a regression for 4.14.x so it
> > should not be a blocker.
> >
> > On Mon, Nov 21, 2022 at 4:03 AM Hang Chen  wrote:
> >
> > > Hi everyone,
> > > Please review and vote on the release candidate #0 for version 4.14.6,
> > > as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * Release notes [1]
> > > * The official Apache source and binary distributions to be deployed to
> > > dist.apache.org [2]
> > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > * Source code tag "v4.14.6-rc0" [4] with git sha [5]
> > >
> > > BookKeeper's KEYS file contains the PGP keys we used to sign this
> > release:
> > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > >
> > > Please download these packages and review this release candidate:
> > >
> > > - Review release notes
> > > - Download the source package (verify shasum, and asc) and follow the
> > > instructions to build and run the bookkeeper service.
> > > - Download the binary package (verify shasum, and asc) and follow the
> > > instructions to run the bookkeeper service.
> > > - Review the maven repo, release tag, licenses, and any other things you
> > > think
> > > it is important to release.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by the
> > majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Release Manager
> > >
> > > [1] https://github.com/apache/bookkeeper/pull/3659/files
> > > [2]
> > > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.6-rc0/
> > > [3]
> > >
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1076/
> > > [4] https://github.com/apache/bookkeeper/tree/v4.14.6-rc0
> > > [5] 9f453834c6039fac46dc19130288dbf0b2996f2e
> > >
> >
> >
> > --
> > Andrey Yegorov
> >

CVE-2022-32531: Apache BookKeeper: Java Client Uses Connection to Host that Failed Hostname Verification

[Enrico Olivelli]

Severity: Moderate

Description:

The Apache Bookkeeper Java Client (up to 4.14.5 and also 4.15.0) does
not close the connection to the
bookkeeper server when TLS hostname verification fails. This leaves
the bookkeeper client vulnerable to a man in the middle attack.

The problem affects BookKeeper client prior to versions 4.14.6 and 4.15.1.

Solution:

Upgrade to 4.14.6 or to 4.15.1

References:

https://bookkeeper.apache.org/
https://www.cve.org/CVERecord?id=CVE-2022-32531

[ANNONCE] Apache BookKeeper releases prior to 4.14 end-of-life

[Enrico Olivelli]

After discussing within the community we decided to declare EOL all the
release lines before 4.14.

This means that all the release lines up to 4.13 won't see any more
releases, even for security issues.

Current active branches are:
- 4.14.x
- 4.15.x
- 4.16.x (no release has been cut yet)
- master


Users of previous version are encouraged to upgrade to the newer releases.

Best regards
Enrico

Re: [VOTE] Release 4.15.3, release candidate #1

[Enrico Olivelli]

+1 (binding)

- RAT, checksums, signatures, all good
- built on Mac on JDK11
- tested localbookie from the results of the build

Thanks
Enrico

Il giorno ven 11 nov 2022 alle ore 06:11 Yong Zhang
 ha scritto:
>
> The maven repo link should be
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1075
>
> Yong
>
> On Fri, 11 Nov 2022 at 12:05, Yong Zhang  wrote:
>
> > Hi Andrey,
> >
> > I run the test multiple times, I found that depends on how the DNS
> > resolve the address.
> > When the DNS resolves to 127.0.0.1/localhost, the test will be failed.
> > But if it resolves another INet interface, like lo0, the hostname will be
> > 192.168.50.240(in my computer) and the test passed.
> >
> > Thanks,
> > Yong
> >
> > On Fri, 11 Nov 2022 at 09:46, Yong Zhang 
> > wrote:
> >
> >> I downloaded the src package and run the tests successfully.
> >> And build from branch-4.15 passed as well. But failed when
> >> I run it with IDE.
> >>
> >> It looks like sometimes it uses the loopback address and we
> >> are not allowed that.
> >> But it shouldn't be a problem for this release.
> >>
> >> Thanks,
> >> Yong
> >>
> >> On Fri, 11 Nov 2022 at 09:01, Andrey Yegorov 
> >> wrote:
> >>
> >>> I tested the following:
> >>>
> >>> * bookkeeper-all
> >>>   ** unpacks
> >>>   ** shell works, standalone starts, localbookie starts
> >>> * src
> >>>   ** unpacks, builds
> >>>   ** tests - there are some know flakes
> >>> and org.apache.bookkeeper.client.TestSpeculativeRead fails for me
> >>>
> >>> I don't recall any issues with TestSpeculativeRead in the previous
> >>> releases; does it repro for you (I built from unpacked sources for the
> >>> RC)?
> >>>
> >>> $ mvn test -f bookkeeper-server/pom.xml
> >>> -Dtest=org.apache.bookkeeper.client.TestSpeculativeRead
> >>>
> >>> [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> >>> 73.673 s <<< FAILURE! - in
> >>> org.apache.bookkeeper.client.TestSpeculativeRead
> >>> [ERROR]
> >>>
> >>> org.apache.bookkeeper.client.TestSpeculativeRead.testSequenceReadLocalEnsemble
> >>>  Time elapsed: 6.341 s  <<< ERROR!
> >>> java.lang.RuntimeException:
> >>> java.net.UnknownHostException: Trying to listen on loopback address,
> >>> 127.0.0.1:3181 but this is forbidden by default (see
> >>> ServerConfiguration#getAllowLoopback()).
> >>> If this happen, you can consider specifying the network interface to
> >>> listen
> >>> on (e.g. listeningInterface=eth0) or specifying the advertised address
> >>> (e.g. advertisedAddress=172.x.y.z)
> >>> at
> >>>
> >>> org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.initialize(LocalBookieEnsemblePlacementPolicy.java:68)
> >>> at
> >>>
> >>> org.apache.bookkeeper.client.BookKeeper.initializeEnsemblePlacementPolicy(BookKeeper.java:583)
> >>> at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:507)
> >>> at
> >>>
> >>> org.apache.bookkeeper.client.BookKeeperTestClient.(BookKeeperTestClient.java:51)
> >>> at
> >>>
> >>> org.apache.bookkeeper.client.TestSpeculativeRead.testSequenceReadLocalEnsemble(TestSpeculativeRead.java:395)
> >>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>> Method)
> >>> at
> >>>
> >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>> at
> >>>
> >>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>> at
> >>>
> >>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> >>> at
> >>>
> >>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> >>> at
> >>>
> >>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> >>> at
> >>>
> >>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> >>> at
> >>>
> >>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> >>> at
> >>>
> >>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> >>> at
> >>>
> >>> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> >>> at
> >>>
> >>> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> >>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>> at java.base/java.lang.Thread.run(Thread.java:829)
> >>> Caused by: java.net.UnknownHostException: Trying to listen on loopback
> >>> address, 127.0.0.1:3181 but this is forbidden by default (see
> >>> ServerConfiguration#getAllowLoopback()).
> >>> If this happen, you can consider specifying the network interface to
> >>> listen
> >>> on (e.g. listeningInterface=eth0) or specifying the advertised address
> >>> (e.g. advertisedAddress=172.x.y.z)
> >>> at
> >>>
> >>> org.apache.bookkeeper.bookie.BookieImpl.getBookieAddress(BookieImpl.java:292)
> >>> at
> >>> 

Re: Release 4.15.3, release candidate #0

[Enrico Olivelli]

+1 (binding)

- Built from sources (JDK11, MacOs)
- Run some smoke tests with localbookie
- Built and run smoke tests with Pulsar 2.11-SNAPSHOT
- Signatures/checksums look good

Thanks for driving the release

Enrico

Il giorno ven 4 nov 2022 alle ore 03:43 Yong Zhang
 ha scritto:
>
> Hi everyone,
> Please review and vote on the release candidate #0 for the version 4.15.3,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.15.3-rc0" [4] with git sha [5]
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1]
> https://github.com/apache/bookkeeper/pull/3602/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53R4
> 
> [2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.3-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1074
> [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.3-rc0
> [5] 029622128e82c5120d3c70f9fc0761e3687e05af

Re: [DISCUSS] BookKeeper 4.14.6 release

[Enrico Olivelli]

+1

Enrico

Il giorno lun 7 nov 2022 alle ore 05:24 Hang Chen
 ha scritto:
>
> Hi everyone,
>
> BookKeeper 4.14.5 has been released for 6 months and I would like to
> discuss the starting 4.14.6 release to include some bug fixes and
> improvements.
>
> Here are the PRs for 4.14.6:
> https://github.com/apache/bookkeeper/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F4.14.6
>
> If you have PRs that want to be included in this release, please
> comment on that PR or tag the label `release/4.14.6`.
>
> After those PRs merged, I would like to be the volunteer to release this 
> patch.
>
> Thanks,
> Hang

Re: [DISCUSS] BookKeeper 4.15.3 release

[Enrico Olivelli]

+1
thanks

FYI I am still working with Raul on this "Missing ledgers when
inducing network packet loss in Bookkeeper 4.15"
https://github.com/apache/bookkeeper/issues/3466
But it will take more time, too much to block a release.



Enrico

Il giorno gio 3 nov 2022 alle ore 11:06 wenbing shen
 ha scritto:
>
> +1
>
> Thanks for your great work!
>
> Thanks,
> Wenbing
>
> Hang Chen  于2022年11月3日周四 16:36写道：
>
> > +1
> >
> > Thanks for your great work!
> >
> > Thanks,
> > Hang
> >
> > Yong Zhang  于2022年11月3日周四 16:28写道：
> > >
> > > If no one objected, I will start to release 4.15.3 soon.
> > >
> > > On Wed, 26 Oct 2022 at 09:23, Yong Zhang 
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I want to discuss starting 4.15.3 release to including some
> > > > fixes/improvements
> > > > and compatibility fix(The RocksDB configuration values).
> > > >
> > > >
> > > > Here are the PRs for 4.15.3:
> > > >
> > > >
> > https://github.com/apache/bookkeeper/pulls?q=+is%3Apr+label%3Arelease%2F4.15.3+
> > > >
> > > > If you have PR want to include in this release, please comment on that
> > PR
> > > > or tag
> > > > the label `release/4.15.1`.
> > > >
> > > > After those PR merged, I would like to be the volunteer to release this
> > > > patch.
> > > >
> > > > Thanks,
> > > > Yong
> > > >
> >

Re: [DISCUSS] Add RegistrationSessionExpiredPolicy for session expired event

[Enrico Olivelli]

I have approved the PR

Enrico

Il Mer 19 Ott 2022, 10:51 wenbing shen  ha
scritto:

> Hi, does anyone have any thoughts on this?
>
> best!
> Wenbing
>
> wenbing shen  于2022年10月17日周一 15:48写道：
>
> > Hi, Bookkeeper Community
> >
> > At present, when the bookie registered session expired, the
> > RegistrationManager listener of BookieStateManager will re-register the
> > service state of bookie in the Registration service, including writable
> and
> > read-only, but this simple re-registration state may make the bookie
> > service in an inconsistent state, bookie It's hard to recover from this
> > (had to recreate the zk client + redo all initialization steps for all
> > subcomponents), like @dlg99's comment at
> > https://github.com/apache/bookkeeper/issues/3250#issuecomment-1190996890
> .
> >
> > I want to provide a session expired policy to choose whether it should
> > continue to reconnect or shutdown directlly.
> > Proposal PR - https://github.com/apache/bookkeeper/pull/3495
> >
> > Because Bookie is difficult to recover from Session Expired.
> >
> > So the reconnect policy is still a policy to be implemented and verified.
> > reconnect was introduced to keep the same logic as before. We can use
> > another PR to implement the complete policy of Bookie Reconnect Recovery
> > from Session Expired.
> >
> > Here I divide it into three steps:
> > 1. This PR introduces two policy options
> > 2. Change the default policy to shutdown to solve flaky-test fix
> > Flaky-test:  https://github.com/apache/bookkeeper/pull/3418
> > 3. Implement complete bookie recovery logic for the reconnect policy.
> >
> > What do you think about it?
> >
> > Thanks.
> > Wenbing
> >
> >
>

[ANNOUNCE] New Committer Steven Lu

[Enrico Olivelli]

Hello BookKeepers,
I am happy to announce that Steven Lu has been invited to join the
Apache BookKeeper project as committer and he accepted the invitation.


Congratulations Steven !

Best regards
Enrico

Re: Direct memory leak issue

[Enrico Olivelli]

Yan,
thanks for sharing your case

This is your PR
https://github.com/apache/bookkeeper/pull/3528

please anyone review the patch

we should cherry-pick to the 4.14 branch at least

Enrico

Il giorno gio 13 ott 2022 alle ore 15:05 horizonzy
 ha scritto:
>
> Hi all.
> Recently, in our customer product, we found that the bookie direct memory
> increases slowly over time. Finally, it throws netty allocate direct memory
> failed exception, then bookie restart.
>
> After our work, we found the root cause and pulled a request to fix it.
> It's essential for everyone.
> See the detail: https://github.com/apache/bookkeeper/issues/3527
> If you have some questions, just discuss them with me.

Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0

[Enrico Olivelli]

Il giorno gio 13 ott 2022 alle ore 04:35 Yong Zhang
 ha scritto:
>
> Hi,
>
> I have pushed a PR for fixing this issue.
> https://github.com/apache/bookkeeper/pull/3523
>
> As we discussed, we ignore the configuration file if they are not exists.
> Then using the configurations in the bk server configuration file.
>
> The rocksDB related configuration file already include in our release.
> I want to rename those files with `.default` suffix then every thing would
> keep the compatibility with previous versions. Then document it at website
> (https://bookkeeper.apache.org/docs/deployment/manual)
> to tell people how to use the independent configuration file to configure
> the rocksDB.
>
> But we already have two releases including this, renaming the file also may
> impact some people.
> Another way is let users to decide which way they want to use. We
> will document this in the release note 4.15.0, 4.15.1, and 4.15.2. Tell
> people
> how to avoid this change impactting their cluster.
>
> WDYT?

As far as I know the main user or the RocksDB based storage is Pulsar
I don't think that other users use that backend.
The RocksDB backend is usually needed by applications, like Pulsar, that perform
catch up reads frequently but most of the BK users (to my knowledge,
that is limited)
use BK as a Write-Ahead-Log.

I believe that we can do the way you suggest and add it to the release notes

We also missed to advertise this big breaking change in the 4.15.0 release notes
and AFAIK nobody moved to BK 4.15 due to a few problems we discovered
and that are still under evaluation

Enrico

>
> Yong
>
>
> On Mon, 10 Oct 2022 at 21:56, Hang Chen  wrote:
>
> > I agree with Yong, we should back-support the existing configuration,
> > otherwise, the pulsar helm chart also need to be changed when upgrade
> > BookKeeper 4.14.x to 4.15.x
> >
> > Thanks,
> > Hang
> >
> > Enrico Olivelli  于2022年10月10日周一 16:07写道：
> > >
> > > Il Lun 10 Ott 2022, 07:44 Yong Zhang  ha
> > > scritto:
> > >
> > > > We shouldn't remove any existing configuration or feature in a so short
> > > > time. I think at least we need to back-support the existing
> > configuration.
> > > > If we must do the break, we should make the existing things in a
> > > > deprecation
> > > > state for a while.
> > > >
> > > > Breaking existing configurations would make a huge impact.
> > > > Maybe we can add back the previous configurations in the next minor
> > > > release for 4.15. So user can upgrade their cluster to 4.15.3 without
> > > > changing any configuration.
> > > >
> > > > WDYT?
> > > >
> > >
> > > Great idea
> > >
> > > Enrico
> > >
> > >
> > > > Yong
> > > >
> > > >
> > > >
> > > >
> > > > On Sun, 9 Oct 2022 at 23:20, Enrico Olivelli 
> > wrote:
> > > >
> > > > > Il Dom 9 Ott 2022, 17:03 Hang Chen  ha scritto:
> > > > >
> > > > > > I found the PR https://github.com/apache/bookkeeper/pull/3056 has
> > > > > > changed the rocksDB default cache size from 10% of direct memory to
> > > > > > 206150041(196MB), which will lead to entry read performance
> > decrease
> > > > > > when there are huge number of entries stored in the ledger
> > directory.
> > > > > > It will have a huge impact on the read performance when the
> > BookKeeper
> > > > > > cluster upgrades from 4.14.x to 4.15.x.
> > > > > >
> > > > >
> > > > >
> > > > > I think that that patch shoud had passed from the BP process.
> > > > > It is a huge breaking change.
> > > > > Personally I missed it.
> > > > >
> > > > > BTW now it is there, we should have talked about it in the release
> > notes,
> > > > > and we now must add some guidance to users upgrading to this version
> > of
> > > > BK.
> > > > >
> > > > > It is a pity that we cannot keep the automatic value.
> > > > >  BTW in all the Pulsar cluster I know I think that they are
> > overriding
> > > > that
> > > > > value because the default value is not big enough.
> > > > >
> > > > > I think that at this point we only have to document on the website
> > the
> > > > > breaking change and add a page about BK and RocksDB
> > > > &

Re: [VOTE] BP-57 Add a public API for creating embeddable servers

[Enrico Olivelli]

Up

We need 1 more binding vote from a PMC member

Enrico

Il Mar 27 Set 2022, 11:36 Nicolò Boschi  ha scritto:

> +1 (non binding)
> Nicolò Boschi
>
>
> Il giorno mar 27 set 2022 alle ore 11:25 Enrico Olivelli <
> eolive...@gmail.com> ha scritto:
>
> > I think that we need some more binding VOTEs please
> >
> > Enrico
> >
> > Il giorno dom 25 set 2022 alle ore 16:15 ZhangJian He
> >  ha scritto:
> > >
> > > +1 , non binding
> > >
> > > Thanks
> > > ZhangJian He
> > >
> > > On Sun, 25 Sept 2022 at 21:56, Lan Liang 
> > wrote:
> > >
> > > > +1 , non binding.
> > > >
> > > > This is a useful feature!
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards,
> > > > Lan Liang
> > > >  Replied Message 
> > > > | From | steven lu |
> > > > | Date | 9/25/2022 08:25 |
> > > > | To |  |
> > > > | Subject | Re: [VOTE] BP-57 Add a public API for creating embeddable
> > > > servers |
> > > > +1 (non binding)
> > > > Great work
> > > >
> > > > Diego Salvi  于2022年9月24日周六 00:00写道：
> > > >
> > > > Hi Bookkeeper Community,
> > > > I would like to start a VOTE on " Add a public API for creating
> > embeddable
> > > > servers."(BP-57)
> > > >
> > > > It is a new public API to create bookie from configuration and/or
> > custom
> > > > implementations without having to deep dive into internal structure.
> > The
> > > > code will be the same used to build standard BK server through Main
> > class
> > > >
> > > > Here is the design detail:
> > > > https://github.com/apache/bookkeeper/issues/3494
> > > >
> > > > Diego Salvi
> > > >
> > > >
> >
>

Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0

[Enrico Olivelli]

Il Lun 10 Ott 2022, 07:44 Yong Zhang  ha
scritto:

> We shouldn't remove any existing configuration or feature in a so short
> time. I think at least we need to back-support the existing configuration.
> If we must do the break, we should make the existing things in a
> deprecation
> state for a while.
>
> Breaking existing configurations would make a huge impact.
> Maybe we can add back the previous configurations in the next minor
> release for 4.15. So user can upgrade their cluster to 4.15.3 without
> changing any configuration.
>
> WDYT?
>

Great idea

Enrico


> Yong
>
>
>
>
> On Sun, 9 Oct 2022 at 23:20, Enrico Olivelli  wrote:
>
> > Il Dom 9 Ott 2022, 17:03 Hang Chen  ha scritto:
> >
> > > I found the PR https://github.com/apache/bookkeeper/pull/3056 has
> > > changed the rocksDB default cache size from 10% of direct memory to
> > > 206150041(196MB), which will lead to entry read performance decrease
> > > when there are huge number of entries stored in the ledger directory.
> > > It will have a huge impact on the read performance when the BookKeeper
> > > cluster upgrades from 4.14.x to 4.15.x.
> > >
> >
> >
> > I think that that patch shoud had passed from the BP process.
> > It is a huge breaking change.
> > Personally I missed it.
> >
> > BTW now it is there, we should have talked about it in the release notes,
> > and we now must add some guidance to users upgrading to this version of
> BK.
> >
> > It is a pity that we cannot keep the automatic value.
> >  BTW in all the Pulsar cluster I know I think that they are overriding
> that
> > value because the default value is not big enough.
> >
> > I think that at this point we only have to document on the website the
> > breaking change and add a page about BK and RocksDB
> >
> > Enrico
> >
> >
> > > The motivation of PR 3056 is to change RocksDB configuration to a
> > > single configuration file, and it introduces the following drawbacks.
> > > 1. Most users, especially those unfamiliar with RocksDB, will be
> > > confused about tuning RocksDB performance due to lack of guidance on
> > > some important key parameters.
> > > 2. The RocksDB blockCacheSize configuration can only be set to a fixed
> > > value, not a percentage of direct memory.
> > >
> > > In order to simplify the RocksDB configuration and make it easy to
> > > tune the performance of the RocksDB, I prefer to separate the RocksDB
> > > configuration into two parts.
> > > 1. The most important configurations, which is usually changed to tune
> > > RocksDB performance, will be located in conf/bk_server.conf
> > > 2. Other advanced configuration will be in a separate RocksDB
> > > configuration file
> > >
> > > For the default RocksDB blockCacheSize change, Do you need to trigger
> > > a new release to change it back? Current Pulsar master branch uses
> > > BookKeeper 4.14.x, and we can change the default value back to 10% of
> > > direct memory to avoid the entry read performance degradation caused
> > > by upgrading Pulsar's BookKeeper dependency to 4.15.x.
> > >
> > > Do you guys have any suggestions?
> > >
> > > Thanks,
> > > Hang
> > >
> >
>

Re: [DISCUSS] BookKeeper RocksDB configuration changed default blockCacheSize since 4.15.0

[Enrico Olivelli]

Il Dom 9 Ott 2022, 17:03 Hang Chen  ha scritto:

> I found the PR https://github.com/apache/bookkeeper/pull/3056 has
> changed the rocksDB default cache size from 10% of direct memory to
> 206150041(196MB), which will lead to entry read performance decrease
> when there are huge number of entries stored in the ledger directory.
> It will have a huge impact on the read performance when the BookKeeper
> cluster upgrades from 4.14.x to 4.15.x.
>


I think that that patch shoud had passed from the BP process.
It is a huge breaking change.
Personally I missed it.

BTW now it is there, we should have talked about it in the release notes,
and we now must add some guidance to users upgrading to this version of BK.

It is a pity that we cannot keep the automatic value.
 BTW in all the Pulsar cluster I know I think that they are overriding that
value because the default value is not big enough.

I think that at this point we only have to document on the website the
breaking change and add a page about BK and RocksDB

Enrico


> The motivation of PR 3056 is to change RocksDB configuration to a
> single configuration file, and it introduces the following drawbacks.
> 1. Most users, especially those unfamiliar with RocksDB, will be
> confused about tuning RocksDB performance due to lack of guidance on
> some important key parameters.
> 2. The RocksDB blockCacheSize configuration can only be set to a fixed
> value, not a percentage of direct memory.
>
> In order to simplify the RocksDB configuration and make it easy to
> tune the performance of the RocksDB, I prefer to separate the RocksDB
> configuration into two parts.
> 1. The most important configurations, which is usually changed to tune
> RocksDB performance, will be located in conf/bk_server.conf
> 2. Other advanced configuration will be in a separate RocksDB
> configuration file
>
> For the default RocksDB blockCacheSize change, Do you need to trigger
> a new release to change it back? Current Pulsar master branch uses
> BookKeeper 4.14.x, and we can change the default value back to 10% of
> direct memory to avoid the entry read performance degradation caused
> by upgrading Pulsar's BookKeeper dependency to 4.15.x.
>
> Do you guys have any suggestions?
>
> Thanks,
> Hang
>

Re: [VOTE] Release 4.15.2, release candidate #0

[Enrico Olivelli]

Il Mer 5 Ott 2022, 03:25 steven lu  ha scritto:

> +1 (binding)
>

Sorry, but binding votes are only from PMC members

Enrico

deployed and verified in our own test environment, no problems were found
>
> Yong Zhang  于2022年9月23日周五 15:32写道：
>
> > Hi everyone,
> > Please review and vote on the release candidate #0 for the version
> 4.15.2,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "v4.15.2-rc0" [4] with git sha [5]
> >
> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review maven repo, release tag, licenses, and any other things you
> think
> > it is important to a release.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1]
> >
> >
> https://github.com/apache/bookkeeper/pull/3500/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53R4
> > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.2-rc0/
> > [3]
> >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1073
> > [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.2-rc0
> > [5] 3c9e4f66ab77a408fe208a3d757929527ff7bc0f
> >
>

Re: [DISCUSS] BP-51: BookKeeper client memory limits

[Enrico Olivelli]

Il giorno gio 29 set 2022 alle ore 15:06 Dave Fisher
 ha scritto:
>
>
> >
> > I think I need to change this proposal title to `BookKeeper client write
> > memory
> > limits` to make it clearly. What we observed is bookie will easily OOM when
> > WQ < AQ. So the main problem we want to use this proposal to resolve is
> > limit
> > the adds request memory usage.
>
> What is the use case for WQ < AQ?

it is a typo, WQ must be always >= QA

Enrico
>
> Best,
> Dave
> >
> > Thanks,
> > Yong
> >
> >
> >> On Thu, 29 Sept 2022 at 12:30, Michael Marshall 
> >> wrote:
> >>
> >> I support adding back pressure based on client memory limits to the
> >> bookkeeper client.
> >>
> >> My biggest concern is how the back pressure is propagated to the
> >> client application. If I am reading the draft implementation
> >> correctly, it is via a blocking operation on the calling thread for
> >> the `BookieClientImpl#addEntry` method.
> >>
> >> In my use case (the Pulsar broker), I think a blocking implementation
> >> will make this feature very hard to use. One quick thought is that
> >> maybe some kind of event or listener could meet the requirements
> >> without also blocking an application? The implementation could be
> >> something similar to Netty's channel writability events. Then, client
> >> applications could reactively get notified of the bookie client's
> >> state. Non blocking back pressure allows for client applications to
> >> continue processing other
> >>
> >> Additionally, I think client memory limits should affect the bookie
> >> client reading from the inbound connection. Otherwise, the bookie
> >> client could dispatch many read requests and then end up filling up
> >> memory when the requests arrive in the client's direct memory. When
> >> the bookie is starting to exceed its memory consumption, it'd be
> >> beneficial to stop reading from the connection and to let the TCP
> >> connection propagate back pressure to the server. In this case, we
> >> would need a reactive solution since it is an anti-pattern to block on
> >> a netty event loop.
> >>
> >> Thanks,
> >> Michael
> >>
> >>> On Wed, Sep 28, 2022 at 4:36 AM Enrico Olivelli 
> >>> wrote:
> >>>
> >>> Yong,
> >>>
> >>> Il giorno mer 28 set 2022 alle ore 10:23 Yong Zhang
> >>>  ha scritto:
> >>>>
> >>>> We have improved the memory issue with backpressure with PR
> >>>> https://github.com/apache/bookkeeper/pull/3324
> >>>>
> >>>> The backpressure way can prevent there have too many Add requests
> >>>> pending to the client and waiting for the response. It makes the add
> >>>> requests
> >>>> fail fast, so if the channel is not writable, it will fail the request
> >> as
> >>>> soon as
> >>>> possible and then release the memory.
> >>>>
> >>>> But that depends on the time. If your throughput is very smooth, and
> >> you
> >>>> have enough memory for the bookie client. With backpressure, it would
> >> work
> >>>> fine.
> >>>> If you have a huge adds to the bookie in a very short time, it
> >> acquires a
> >>>> lot of
> >>>> memory, then the bookie crashed with OOM.
> >>>> So we still need this proposal.
> >>>>
> >>>> I will continue to work on this. If there haven't objected, I will
> >> start a
> >>>> VOTE later
> >>>
> >>> Thanks
> >>>
> >>> Enrico
> >>>
> >>>> Thanks,
> >>>> Yong
> >>>>
> >>>> On Thu, 21 Apr 2022 at 19:17, r...@apache.org  >>>
> >>>> wrote:
> >>>>
> >>>>> Hello Yong:
> >>>>>
> >>>>> It seems to be a very useful feature. In the production environment,
> >> you
> >>>>> can often see similar phenomena happening.
> >>>>>
> >>>>> +1 (non-binding)
> >>>>>
> >>>>> --
> >>>>> Thanks
> >>>>> Xiaolong Ran
> >>>>>
> >>>>> Yong Zhang  于2022年4月21日周四 18:29写道：
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> The BP-51 BookKeeper client memory limits is ready for review.
> >>>>>> The proposal is here:
> >> https://github.com/apache/bookkeeper/issues/3231
> >>>>>> And the PR is here: https://github.com/apache/bookkeeper/pull/3139
> >>>>>>
> >>>>>> Please help to review this proposal.
> >>>>>>
> >>>>>> Thanks!
> >>>>>> Yong
> >>>>>>
> >>>>>
> >>

Re: [DISCUSS] BP-51: BookKeeper client memory limits

[Enrico Olivelli]

Yong,

Il giorno mer 28 set 2022 alle ore 10:23 Yong Zhang
 ha scritto:
>
> We have improved the memory issue with backpressure with PR
> https://github.com/apache/bookkeeper/pull/3324
>
> The backpressure way can prevent there have too many Add requests
> pending to the client and waiting for the response. It makes the add
> requests
> fail fast, so if the channel is not writable, it will fail the request as
> soon as
> possible and then release the memory.
>
> But that depends on the time. If your throughput is very smooth, and you
> have enough memory for the bookie client. With backpressure, it would work
> fine.
> If you have a huge adds to the bookie in a very short time, it acquires a
> lot of
> memory, then the bookie crashed with OOM.
> So we still need this proposal.
>
> I will continue to work on this. If there haven't objected, I will start a
> VOTE later

Thanks

Enrico

> Thanks,
> Yong
>
> On Thu, 21 Apr 2022 at 19:17, r...@apache.org 
> wrote:
>
> > Hello Yong:
> >
> > It seems to be a very useful feature. In the production environment, you
> > can often see similar phenomena happening.
> >
> > +1 (non-binding)
> >
> > --
> > Thanks
> > Xiaolong Ran
> >
> > Yong Zhang  于2022年4月21日周四 18:29写道：
> >
> > > Hi all,
> > >
> > > The BP-51 BookKeeper client memory limits is ready for review.
> > > The proposal is here: https://github.com/apache/bookkeeper/issues/3231
> > > And the PR is here: https://github.com/apache/bookkeeper/pull/3139
> > >
> > > Please help to review this proposal.
> > >
> > > Thanks!
> > > Yong
> > >
> >

Re: [VOTE] Release 4.15.2, release candidate #0

[Enrico Olivelli]

+1 (binding)
- verified checksums and signatures
- built from sources on Mac, JDK8, tests are passing (a few flakes)
- tested local bookie

Thank you!

Enrico

Il giorno ven 23 set 2022 alle ore 09:32 Yong Zhang
 ha scritto:
>
> Hi everyone,
> Please review and vote on the release candidate #0 for the version 4.15.2,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.15.2-rc0" [4] with git sha [5]
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1]
> https://github.com/apache/bookkeeper/pull/3500/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53R4
> [2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.2-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1073
> [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.2-rc0
> [5] 3c9e4f66ab77a408fe208a3d757929527ff7bc0f

Re: [VOTE] BP-57 Add a public API for creating embeddable servers

[Enrico Olivelli]

I think that we need some more binding VOTEs please

Enrico

Il giorno dom 25 set 2022 alle ore 16:15 ZhangJian He
 ha scritto:
>
> +1 , non binding
>
> Thanks
> ZhangJian He
>
> On Sun, 25 Sept 2022 at 21:56, Lan Liang  wrote:
>
> > +1 , non binding.
> >
> > This is a useful feature!
> >
> >
> >
> >
> >
> >
> >
> >
> > Best Regards,
> > Lan Liang
> >  Replied Message 
> > | From | steven lu |
> > | Date | 9/25/2022 08:25 |
> > | To |  |
> > | Subject | Re: [VOTE] BP-57 Add a public API for creating embeddable
> > servers |
> > +1 (non binding)
> > Great work
> >
> > Diego Salvi  于2022年9月24日周六 00:00写道：
> >
> > Hi Bookkeeper Community,
> > I would like to start a VOTE on " Add a public API for creating embeddable
> > servers."(BP-57)
> >
> > It is a new public API to create bookie from configuration and/or custom
> > implementations without having to deep dive into internal structure. The
> > code will be the same used to build standard BK server through Main class
> >
> > Here is the design detail:
> > https://github.com/apache/bookkeeper/issues/3494
> >
> > Diego Salvi
> >
> >

Re: [VOTE] BP-58 Change the API for org.apache.bookkeeper.stats.Counter

[Enrico Olivelli]

+1 (binding)

But I don't think we need this VOTE  as the patch has already been merged

Steven, it's up to you

Enrico

Il giorno mar 27 set 2022 alle ore 09:10 steven lu
 ha scritto:
>
> Hi Bookkeeper Community,
>  I would like to start a VOTE on " Change the API for
> org.apache.bookkeeper.stats.Counter."(BP-58) It changes the API for
> org.apache.bookkeeper.stats.Counter.
> we have updated its implementation in bookkeeper projects,
> when bookkeeper-4.16.0 is released, we can update its implementation in
> other projects
>
> Here is the design detail: https://github.com/apache/bookkeeper/issues/3502
>
> Steven Lu

Re: [DISCUSS] BP-58 Change the API for org.apache.bookkeeper.stats.Counter

[Enrico Olivelli]

Hello,
The PR has been merged without running a formal VOTE.

https://github.com/apache/bookkeeper/pull/3501

I think that given the impact of this change for this time we can move
forward and consider the BP "accepted"

A few people already commented by "supporting" the BP

Thoughts ?

Enrico

Il giorno mar 27 set 2022 alle ore 03:13 Yong Zhang
 ha scritto:
>
> LGTM.
>
> Best,
> Yong
>
> On Mon, 26 Sept 2022 at 15:56, wenbing shen 
> wrote:
>
> > +1 （non-binding）
> >
> > Thanks.
> > Wenbing Shen
> >
> > ZhangJian He  于2022年9月26日周一 11:15写道：
> >
> > > +1 （non-binding）
> > >
> > > Thanks
> > > ZhangJian He
> > >
> > > On Sun, 25 Sept 2022 at 20:45, Enrico Olivelli 
> > > wrote:
> > >
> > > > I agree
> > > >
> > > > Enrico
> > > >
> > > > Il Dom 25 Set 2022, 03:09 steven lu  ha
> > scritto:
> > > >
> > > > > Hi BookKeepers, I've changed the API for
> > > > > org.apache.bookkeeper.stats.Counter:
> > > > > BP-58 <https://github.com/apache/bookkeeper/issues/3502>
> > > > >
> > > > >
> > > > > "Motivation"
> > > > >
> > > > > The latency of the OpStatsLogger.registerSuccessfulEvent calculation
> > is
> > > > to
> > > > > convert the time to milliseconds
> > > > > but Counter.add nothing to do
> > > > > so when using Counter for latency statistics, the time unit and
> > > > > OpStatsLogger are not unified, which is easy to be misleading.
> > > > > then we unified latency metric unit
> > > > >
> > > > > "Proposal"
> > > > >
> > > > > We already have the Counter metric for counting , we just need to
> > > > refactor
> > > > > it and expose it as a public method.
> > > > > My idea is to change the API of org.apache.bookkeeper.stats.Counter
> > > > >
> > > > > when using Counter for count metric, use Counter.addCount,intead of
> > old
> > > > > API(Counter.add)
> > > > > when using Counter for latency metric, use Counter.addLatency
> > > > >
> > > > > "Release"
> > > > >
> > > > > because it change the API,so we don't cherry pick this into released
> > > > > branches
> > > > > so it should go into a major release, like Release-4.16, then we can
> > > > update
> > > > > its implementation in other projects, like pulsar
> > > > >
> > > > >
> > > > >
> > > > > What do you think about it?
> > > > >
> > > >
> > >
> >

[DISCUSS] End-Of-Life for some release lines

[Enrico Olivelli]

Hello,
I propose to declare end-of-life on all the release lines prior to 4.14.

We wanted to follow a time-based release plan, but actually we are not
following it.

This is the doc on our website
https://bookkeeper.apache.org/community/releases

If nobody objects, I will start a separate thread with a VOTE for this
resolution.

End-of-life generally means "no more releases" so:
- no more bug fixes
- no more security fixes
- no patches accepted to be cherry-picked to those branches

Please chime in with your thoughts

Enrico

Re: [DISCUSS] BP-58 Change the API for org.apache.bookkeeper.stats.Counter

[Enrico Olivelli]

I agree

Enrico

Il Dom 25 Set 2022, 03:09 steven lu  ha scritto:

> Hi BookKeepers, I've changed the API for
> org.apache.bookkeeper.stats.Counter:
> BP-58 
>
>
> "Motivation"
>
> The latency of the OpStatsLogger.registerSuccessfulEvent calculation is to
> convert the time to milliseconds
> but Counter.add nothing to do
> so when using Counter for latency statistics, the time unit and
> OpStatsLogger are not unified, which is easy to be misleading.
> then we unified latency metric unit
>
> "Proposal"
>
> We already have the Counter metric for counting , we just need to refactor
> it and expose it as a public method.
> My idea is to change the API of org.apache.bookkeeper.stats.Counter
>
> when using Counter for count metric, use Counter.addCount,intead of old
> API(Counter.add)
> when using Counter for latency metric, use Counter.addLatency
>
> "Release"
>
> because it change the API,so we don't cherry pick this into released
> branches
> so it should go into a major release, like Release-4.16, then we can update
> its implementation in other projects, like pulsar
>
>
>
> What do you think about it?
>

Re: [VOTE] BP-57 Add a public API for creating embeddable servers

[Enrico Olivelli]

+1 (binding)

thank you !
Enrico

Il giorno ven 23 set 2022 alle ore 18:00 Diego Salvi
 ha scritto:
>
> Hi Bookkeeper Community,
> I would like to start a VOTE on " Add a public API for creating embeddable
> servers."(BP-57)
>
> It is a new public API to create bookie from configuration and/or custom
> implementations without having to deep dive into internal structure. The
> code will be the same used to build standard BK server through Main class
>
> Here is the design detail: https://github.com/apache/bookkeeper/issues/3494
>
> Diego Salvi

Re: [DISCUSS] BookKeeper 4.15.2 release

[Enrico Olivelli]

Il giorno lun 19 set 2022 alle ore 07:36 Yong Zhang
 ha scritto:
>
> Hi everyone,
>
> I want to discuss starting 4.15.2 release to resolve the regression
> mentioned here:
> https://lists.apache.org/thread/d9gm50kysgjcpbq448s2sy822kho6z6k
>
> Although it is an unexpected setting caused, we never know how the users to
> configure
> it. And we have a PR for fixing it, let's start a new release to include it.
>
> I wanted to also wait for this issue
>  resolution before. But
> it looks won't be resolved
> recently.
>
> Any ideas?

I agree

I think that the PR has not been chery-picked yet to branch-4.15
https://github.com/apache/bookkeeper/commits/branch-4.15

Please also take a look if we need to upgrade some dependencies, like SnakeYAML
https://github.com/apache/bookkeeper/commit/457eaf8dbc0101deef2159e7ec1c312ab66af2db

also
https://github.com/apache/bookkeeper/pull/3470

Thanks

Enrico

>
> Yong

Re: [DISCUSS] BP-57 Add a public API for creating embeddable servers

[Enrico Olivelli]

+1
Really helpful!

Enrico

Il Dom 18 Set 2022, 15:02 Diego Salvi  ha
scritto:

> Hi BookKeepers,
> I've added a new BP to add a public API for creating embeddable servers:
> BP-57 
>
> *Motivation*
> BK is used and run in many OS project as an embedded resource.
> Every time a project needs to embed BK (think about Pulsar, Pravega or
> HerdDB as some examples) boilerplate code need to be written (and then
> mantained as per 4.15 changes).
> Depending from the contest new feature cannot be made directly available
> and more code need to be added.
> To run an embedded bookie you need to instantiate many components that are
> really in "public" api and can and will be subject to changes.
>
> *Proposal*
> We already have the whole code to start a BK server instance, it just needs
> to be refactored and exposed as a public API.
> My idea is to refactor all the code currently at
> org.apache.bookkeeper.server.Main#buildBookieServer and expose it with a
> builder pattern.
> The builder will accept an initial BookieConfiguration (and with no more
> interaction will produce the same server currently produced from
> org.apache.bookkeeper.server.Main#buildBookieServer) and will create from
> the configuration any component not directly provided to the builder.
> For example you can provide your instances for StatsProvider or
> MetadataBookieDriver instance or your custom ByteBufAllocator without the
> need to rewrite the same plain old code.
> Another big improvement: new features that whould need additional code to
> be added (just think when integrity checking has been added in BP-46) are
> already and directly usable.
>
> What do you think about it?
>
> Diego Salvi
>

Re: Public API for creating embeddable servers

[Enrico Olivelli]

Diego,
That's a great idea!

Looking forward for the BP


Enrico

Il Mer 14 Set 2022, 12:24 Lothruin Mirwen  ha
scritto:

> Hi bookkeepers!
>
> Using BK in many project I had to write each time boilerplate code to
> create a BK embedded instance. With changes published in 4.15 I had to
> change such code in each project to handle the new interfaces.
>
> I think a public API to create server instances from configuration would be
> really really useful.
>
> I published a PR https://github.com/apache/bookkeeper/pull/3489 to adda
> simple way to create an embeddable server. All the code was really already
> there in Main class, just needed to be refactored a little to insert a
> builder pattern. I've added many parameters on builder to be able to
> provide instances for many component instead to build them from the
> configuration (for example to provide your statsprovider instance).
>
> What do you think about it?
>
> Diego Salvi
>

Re: [VOTE] BP-56: Support data migration

[Enrico Olivelli]

+1 (binding)

Enrico

Il giorno ven 16 set 2022 alle ore 08:26 r...@apache.org
 ha scritto:
>
> LGTM +1
>
> On 2022/09/15 04:47:25 lordcheng10 wrote:
> > Hi Bookkeeper Community, I would like to start a VOTE on "Support data
> migration."(BP-56)
> >
> > Here is the design detail:
> https://github.com/apache/bookkeeper/issues/3456
> > and the discussion thread:
> https://lists.apache.org/thread/0zlsm7z5j34j23gtlhf7rf72lfq62jyk
> >
> > Thanks,
> > lordcheng10

Re: [Discuss] BP-56: Support non-stop bookie data migration and bookie offline

[Enrico Olivelli]

Yang,

Il Sab 27 Ago 2022, 11:05 Yang Yang  ha scritto:

> For the short term, you can see if the `recover` command is able to help
> you:
> https://bookkeeper.apache.org/docs/reference/cli#bookkeeper-shell-recover
>
> For the long term, I have proposed a solution to mark the bookie to be
> decommissioned in a `draining` state and let the autorecovery mechanism
> replicate the ledgers, please take a look and see if it could solve your
> use case: https://lists.apache.org/thread/1l9kzb1l0vok105gj2ody3g8nyv7s9l8


Would you be able to continue that proposal?

Enrico



>
> Best regards,
> Yang Yang
>
>
> On Thu, Aug 25, 2022 at 6:32 PM steven lu  wrote:
>
> > I think this feature is somewhat custom and not very generic; and there
> are
> > risks:
> > 1.  If you want to go offline on node A (which has already been
> extracted),
> > but wrong write B, this function will directly go offline on node B,
> which
> > is likely to cause online failures
> > 2.  If the node to be offline suddenly accesses traffic, how should it be
> > handled? It is easy to cause the loss of cluster replicas
> >
> > In response to these two problems, how to avoid, please help explain
> >
> > lordcheng10 <1572139...@qq.com.invalid> 于2022年8月25日周四 17:12写道：
> >
> > > Hi Bookkeeper Community,
> > >
> > >
> > > This is a BP discussion onSupport non-stop bookie data migration
> > and
> > > bookie offline
> > > The issue can be found:
> > > https://github.com/apache/bookkeeper/issues/3456;
> > >
> > >
> > > I copy the content here for convenience, any suggestions are welcome
> and
> > > appreciated.
> > >
> > >
> > >
> > >
> > > ### Motivation
> > > bookie offline steps:
> > > 1. Log on to the bookie node, check if there are underreplicated
> > > ledgers.If there are, the decommission command will force them to be
> > > replicated: bin/bookkeeper shell listunderreplicated
> > > 2. Stop the bookie : bin/bookkeeper-daemon.sh stop bookie
> > > 3. Run the decommission command. If you have logged onto the node you
> > wish
> > > to decommission, you don't need to provide -bookieid If you are running
> > the
> > > decommission command for target bookie node from another bookie node
> you
> > > should mention the target bookie id in the arguments for -bookieid
> > :
> > > bin/bookkeeper shell decommissionbookie or $ bin/bookkeeper shell
> > > decommissionbookie -bookieid  > > 4. Validate that there are no ledgers on decommissioned bookie $
> > > bin/bookkeeper shell listledgers -bookieid  > >
> > >
> > > For the current bookie offline solution, need to stop the bookie
> > > first，execute the decommission command and wait for the ledger
> migration
> > on
> > > the bookie to complete.
> > >
> > >
> > > it is very time-consuming to offline a bookie node. When we need to
> > > offline a lot of bookie nodes, the time-consuming of this solution will
> > not
> > > be acceptable.
> > >
> > >
> > > Therefore, we need a solution that can migrate data without stopping
> > > bookie, so that bookie nodes can be offlined in batches.
> > >
> > >
> > > ### Proposal
> > > In order to solve this solution, we propose a solution that can be
> > > replicated without stopping the bookie.
> > > The process is as follows:
> > > 1. Submit the bookie node to be offline;
> > > 5. Traverse each ledgers on the offline bookie, and persist these
> ledgers
> > > and the corresponding offline bookie nodes to the zookeeper directory:
> > > ledgers/offline_ledgers/ledgerId;
> > > 6. Get the ledger to be offline;
> > > 7. Traverse all fragments on a ledger, and filter out the fragments
> > > containing the offline bookie copy;
> > > 8. Copy data for each fragment;
> > > 9. When a ledger fragment is copied, delete the corresponding
> > > ledgers/offline_ledgers/ledgerId;
> > > 10. When all ledgerId directories under ledgers/offline_ledgers are
> > > deleted, it means that the data has been migrated, you can stop bookies
> > in
> > > batches and go offline;
> > >
> > >
> > > To achieve our goal, we need to achieve two things:
> > > 1. Implement a command to submit the bookie to be offline and the
> > > corresponding ledgers, for example:
> > > bin/bookkeeper shell decommissionbookie -offline_bookieids
> > > bookieId1,bookieId2,bookieId3,...bookieIdN
> > >  This command will write all ledgers on the offline bookie node
> to
> > > the zookeeper directory, for example: put
> > ledgers/offline_ledgers/ledgerId
> > > bookId1,bookId2,...bookIdn;
> > > 2. Design a ReassignLedgerWorker class to perform the actual ledger
> > > replication:
> > >  this class will obtain a ledger from the zookeeper
> directory
> > > ledgers/offline_ledgers for replication.
> > >  It will first filter out all the fragments containing the
> > > offline bookieId under the ledger,then copy these fragments;
> >
>