Re: [DISCUSS] Proposal for Updated JVM Support Strategy in Apache BookKeeper CI

[Andrey Yegorov]

I think we should start with refactoring the BK client into a separate
module built with JDK 8 and then move BK to JDK 17.
FWIW this is similar to the Pulsar's JDK support for client/server parts.

On Mon, Jun 24, 2024 at 4:22 AM Lari Hotari  wrote:

> For Pulsar 4.0 (scheduled for October), we need to migrate from Jetty 9 to
> Jetty 12 since Jetty 9 is not supported any more ([1]). This also impacts
> Bookkeeper since Pulsar bundles all dependencies in a single directory and
> this doesn't allow library conflicts between Pulsar and Bookkeeper. This is
> why we would first need to migrate to Jetty 12 in Bookkeeper, asap.
>
> The minimum supported Java version for Jetty 12 is Java 17. That's why I'm
> proposing that we resolve this problem by setting the minimum Java version
> to Java 17 in Bookkeeper master branch asap.
>
> -Lari
>
> 1 - https://github.com/apache/pulsar/issues/22939
>
> On 2024/05/27 01:17:14 ZhangJian He wrote:
> > Hi, BookKeepers, I want to propose and clarify a new CI strategy based on
> > our former practices.
> >
> > ## Current CI Jobs
> >
> > - **PR Validation Tests**: Currently, these jobs run on JDK 11 for every
> > pull request to ensure all changes meet our quality standards before
> > merging.
> > - **Platform Build Tests**: We run Windows and macOS builds on JDK 11.
> > - **Tests**: Major tests are currently only run on JDK 11 due to time
> > constraints(I think, please point out if I were wrong).
> > - **Compatibility Checks**: We perform basic compatibility checks across
> a
> > JDK matrix of 8, 11, 17, and 21, with JDK 21 added recently as per [this
> > pull request](https://github.com/apache/bookkeeper/pull/4350).
> > - **OWASP Check**: Run as part of our daily builds or pom file changes to
> > ensure security vulnerabilities are identified and addressed swiftly.
> > - **Daily Builds**: These are three types of job in daily builds: higher
> > jdk linux, owasp, windows jdk(current running in jdk21, limited to some
> > development jobs, there are still many tests failures to be solved), due
> to
> > its speed and the ability to expose issues.
> > - **GitHub CodeQL, TYPOS**: These jobs are not related to JVM.
> >
> > ## Proposed JVM Strategy
> >
> > Given that JDK 11 will reach its end of life in October 2024 [1], I
> suggest
> > transitioning our main CI operations to **JDK 17**. This would include
> the
> > `PR Validation Tests`, `Platform Build Tests`, and `Tests`.
> >
> > For `Compatibility Checks`, we should consider maintaining checks only
> for
> > OpenJDK LTS versions that are not EOL. For instance, we can remove JDK 11
> > from the compatibility matrix after October 2024.
> >
> > For `Daily Builds` on Linux, I propose adding OpenJDK LTS versions beyond
> > our main CI version.
> >
> > For `Daily Builds` on Windows and the `OWASP Check`, using the latest
> > OpenJDK version will leverage the most advanced and supported JDK
> features
> > and help expose more problems.
> >
> > Thank you for considering this strategic update. I look forward to your
> > feedback and further discussion.
> >
> > [1] [JDK 11 End of Life Update](
> https://access.redhat.com/articles/1299013)
> >
> > Thanks
> > ZhangJian He
> > Twitter: shoothzj
> > Wechat: shoothzj
> >
>

Re: [VOTE] Release 4.17.1, release candidate #0

[Andrey Yegorov]

+1 (binding)

* reviewed the release notes
* spot checked maven repo
* built from the tag
* ran the binaries (localbookie, standalone)


On Fri, Jun 14, 2024 at 7:05 AM ZhangJian He  wrote:

> Hi everyone,
> Please review and vote on the release candidate #0 for the version
> 4.17.1, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed
> to dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.17.1-rc0"  [4] with git sha
> f233320077991b4b50218598858f6d31a1914884
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> [1] https://github.com/apache/bookkeeper/pull/4389
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.17.1-rc0/
> [3]
>
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1098/
> [4] https://github.com/apache/bookkeeper/tree/v4.17.1-rc0
>
> Thanks
> ZhangJian He
> Twitter: shoothzj
> Wechat: shoothzj
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.16.6, release candidate #0

[Andrey Yegorov]

+1 (binding)

* reviewed the release notes
* spot checked maven repo
* built from the tag
* ran the binaries (localbookie, standalone)


On Fri, Jun 14, 2024 at 1:22 AM ZhangJian He  wrote:

> Hi everyone,
> Please review and vote on the release candidate #0 for the version
> 4.16.6, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed
> to dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.16.6-rc0"  [4] with git sha
> bfd871810f78a963029acd1a462d677de2865cf1
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> [1] https://github.com/apache/bookkeeper/pull/4388
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.6-rc0/
> [3]
>
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1097/
> [4] https://github.com/apache/bookkeeper/tree/v4.16.6-rc0
>
> Thanks
> ZhangJian He
> Twitter: shoothzj
> Wechat: shoothzj
>


-- 
Andrey Yegorov

Re: [ACCOUNT] ZhangJian He added to the PMC of Apache BookKeeper

[Andrey Yegorov]

Congrats!

On Fri, Jun 14, 2024 at 2:34 AM Enrico Olivelli 
wrote:

> Hello everyone,
> I am happy to announce that  ZhangJian He has accepted the invitation to
> join the BookKeeper PMC.
>
> More details about how the ASF works here:
> https://apache.org/foundation/how-it-works/#roles
>
> Congratulations ZhangJian !
>
> Thank you for all your efforts to help the community and your contributions
> to the project !
>
> Enrico Olivelli
>


-- 
Andrey Yegorov

Re: [DISCUSS] BookKeeper 4.16.6 release

[Andrey Yegorov]

+1

On Fri, May 24, 2024 at 7:38 PM ZhangJian He  wrote:

> Hi all,
>
> BookKeeper 4.16.5 was released one and half months ago, and I would
> like to discuss the starting 4.16.6 release to include some critical
> security and bug fixes.
>
> Here are the PRs for 4.16.6:
> PR-4.16.6
> <
> https://github.com/apache/bookkeeper/pulls?q=is%3Apr+label%3Arelease%2F4.16.6+is%3Amerged
> >
>
> There are still some PRs not been cherry-picked due to conflict, I will
> handle them these days. I removed open PRs that were targeting
> 4.16.6 label.
>
> If you have other PRs that you want to be included in this release, please
> tag the PR with the "release/4.16.6" and let me know in this thread.
>
> If there's any concern, I'd like to volunteer to guide this release.
>
> Thanks
> ZhangJian He
> Twitter: shoothzj
> Wechat: shoothzj
>

Re: [DISCUSS] BookKeeper 4.17.1 release

[Andrey Yegorov]

+1

WDYT about adding these recent PRs:
https://github.com/apache/bookkeeper/pull/4407
https://github.com/apache/bookkeeper/pull/4413
https://github.com/apache/bookkeeper/pull/4419


On Fri, May 24, 2024 at 7:48 PM ZhangJian He  wrote:

> Hi all,
>
> BookKeeper 4.17.0 was released one months ago, and I would
> like to discuss the starting 4.17.1 release to include some critical
> security and bug fixes.
>
> Here are the PRs for 4.17.1:
> PR-4.17.1
> <
> https://github.com/apache/bookkeeper/pulls?q=is%3Apr+label%3Arelease%2F4.17.1+is%3Amerged
> >
>
> I have cherry-picked it. I removed open PRs that were targeting 4.17.1
> label.
>
> If you have other PRs that you want to be included in this release, please
> tag the PR with the "release/4.17.1" and let me know in this thread.
>
> If there's any concern, I'd like to volunteer to guide this release.
>
> Thanks
> ZhangJian He
> Twitter: shoothzj
> Wechat: shoothzj
>

Re: [DISCUSS] Proposal for Updated JVM Support Strategy in Apache BookKeeper CI

[Andrey Yegorov]

In general, dropping older JDKs and updating the CI makes sense for the
BookKeeper server.
Here comes the problem that we want to maintain the BK client with lower
JDK compatibility, even if it is EOL.
Companies still use older JDKs, pay to 3rd party vendors to maintain the
JDKs/provide the security patches etc.

BK's client builds with the server which bring up this ancient issue:
https://github.com/apache/bookkeeper/issues/617

In short, IMO it would make sense to split the client into a separate
module first, build and run tests for the client using older JDKs and
maintain the server with more current JDKs.


On Sun, May 26, 2024 at 6:17 PM ZhangJian He  wrote:

> Hi, BookKeepers, I want to propose and clarify a new CI strategy based on
> our former practices.
>
> ## Current CI Jobs
>
> - **PR Validation Tests**: Currently, these jobs run on JDK 11 for every
> pull request to ensure all changes meet our quality standards before
> merging.
> - **Platform Build Tests**: We run Windows and macOS builds on JDK 11.
> - **Tests**: Major tests are currently only run on JDK 11 due to time
> constraints(I think, please point out if I were wrong).
> - **Compatibility Checks**: We perform basic compatibility checks across a
> JDK matrix of 8, 11, 17, and 21, with JDK 21 added recently as per [this
> pull request](https://github.com/apache/bookkeeper/pull/4350).
> - **OWASP Check**: Run as part of our daily builds or pom file changes to
> ensure security vulnerabilities are identified and addressed swiftly.
> - **Daily Builds**: These are three types of job in daily builds: higher
> jdk linux, owasp, windows jdk(current running in jdk21, limited to some
> development jobs, there are still many tests failures to be solved), due to
> its speed and the ability to expose issues.
> - **GitHub CodeQL, TYPOS**: These jobs are not related to JVM.
>
> ## Proposed JVM Strategy
>
> Given that JDK 11 will reach its end of life in October 2024 [1], I suggest
> transitioning our main CI operations to **JDK 17**. This would include the
> `PR Validation Tests`, `Platform Build Tests`, and `Tests`.
>
> For `Compatibility Checks`, we should consider maintaining checks only for
> OpenJDK LTS versions that are not EOL. For instance, we can remove JDK 11
> from the compatibility matrix after October 2024.
>
> For `Daily Builds` on Linux, I propose adding OpenJDK LTS versions beyond
> our main CI version.
>
> For `Daily Builds` on Windows and the `OWASP Check`, using the latest
> OpenJDK version will leverage the most advanced and supported JDK features
> and help expose more problems.
>
> Thank you for considering this strategic update. I look forward to your
> feedback and further discussion.
>
> [1] [JDK 11 End of Life Update](https://access.redhat.com/articles/1299013
> )
>
> Thanks
> ZhangJian He
> Twitter: shoothzj
> Wechat: shoothzj
>


-- 
Andrey Yegorov

Re: BP-63: Force Reschedule Auditor tasks

[Andrey Yegorov]

Does this new functionality do anything that is not covered by REST API?
https://bookkeeper.apache.org/docs/4.10.0/admin/http/#endpoint-apiv1autorecoverytrigger_audit

On 2023/07/10 06:34:35 Wenbing Shen wrote:
> Hi everyone,
> 
> I would like to initiate a discussion regarding the current bookie force
> reschedule auditor tasks. Below is the detailed BP content. If you have any
> questions or ideas, please feel free to reply to this email for further
> discussion. Thank you!
> 
> This is the master ticket for tracking BP-63 :
> Proposal PR - #3964 
> Motivation
> 
> Currently, the Bookie can reschedule Auditor check tasks in several ways,
> excluding the auditorBookieTask as it provides a separate mechanism to
> trigger task reexecution. This BP specifically discusses
> AuditorCheckAllLedgersTask/AuditorPlacementPolicyCheckTask/AuditorReplicasCheckTask:
> 
> 1: The Bookie provides three execution times based on ZooKeeper,
> checkallledgersctime/placementpolicycheckctime/replicascheckctime. By
> updating these execution times, we can dynamically adjust the execution
> frequency of auditor tasks, but it requires restarting the Auditor process
> or reopening the Auditor election to trigger task execution.
> 
> 2: By using the ForceAuditorChecksCmd tool, which is still based on the
> underlying logic of the first point, restarting the Auditor or performing
> an election is also necessary to trigger task execution.
> 
> 3: The Decommission and RecoveryBookie tools tend to focus on executing
> recovery logic and only check and recover a specific subset of Bookie
> services.
> 
> The above methods are complex and have poor stability when rescheduling the
> Auditor check tasks in a cluster.
> Proposal
> 
> Therefore, I propose further optimizing the rescheduling of Auditor tasks.
> 
> 1: The Auditor monitors the persistent znode path
> /ZK_LEDGERS_ROOT_PATH/underreplication/scheduleAuditor.
> 2: Users modify the task ctime using the ForceAuditorChecksCmd tool and
> forcefully create the above znode path using the force parameter.
> 3: The Auditor creates callbacks through scheduleAuditor to reschedule the
> aforementioned three tasks.
> 4: After the Auditor completes rescheduling the tasks, the scheduleAuditor
> node is deleted.
> 5: When the Auditor starts, it deletes the old scheduleAuditor node to
> avoid logical confusion.
> 
> This way, we can trigger the scheduling and execution of Auditor tasks
> through an online interface without relying on service restart or
> re-election.
> Compatibility, Deprecation, and Migration Plan
> 
> There are no compatibility issues. This BP introduces a new trigger flag
> that does not affect the original logic and does not involve any changes to
> other existing public APIs. There is no deprecation or migration plan.
> 
> 
> Best regards,
> 
> Wenbing Shen
> 

Re: [DISCUSS] BP-67: support skipping compaction at busy times

[Andrey Yegorov]

I am rather indifferent about the change.
There was an attempt in the to implement something similar:
https://github.com/apache/bookkeeper/pull/851

It was abandoned, as I remember, because the company where the author
worked had chosen to use REST API and cron to run enable/disable
compaction.

So you already can disable the compaction on schedule just not via bookie
configuration.

I'd support a mechanism that would dynamically pause/resume major/minor
compaction if the load on bookie is high (e.g. as tracked by pending
requests or request queue latency) if the disk space is enough.

On Wed, May 22, 2024 at 3:27 AM thetumbled  wrote:

> Hi all,
> I create a new BP to avoid executing compaction at busy times, Looking
> forward to your feedback.
> https://github.com/apache/bookkeeper/issues/4382
>
> Thanks,
> Wenzhi Feng(thetumbled).



-- 
Andrey Yegorov

Re: [DISCUSS] Introducing Typo CI Check to Apache BookKeeper

[Andrey Yegorov]

Awesome idea, thank you for tackling this.

We need to keep an eye on changes in config parameter names in yaml/configs to 
avoid unexpected breaks.
So far I haven't noticed any, but it is an easy miss in a big PR.

On 2024/05/19 07:20:30 ZhangJian He wrote:
> Hi, I want to introduce new lint tools: typos[1]. This tool is designed to
> automatically detect and correct typographical errors in our codebase,
> documentation, and comments. It’s a valuable addition to ensure
> high-quality contributions and maintain the professional standard of our
> project.
> 
> It has also been used in the Apache OpenDAL and Kvrocks projects. I created
> a draft PR[2].
> 
> I have been careful to avoid changes to public interfaces, but I may have
> made mistakes, and the changes are relatively huge. I think I would prefer
> to proceed as follows:
> 
> First, commit typos in markdown, YAML, etc.
> Second, commit typos in test files.
> Lastly, fix other typos and commit the lint.
> 
> 
> 1. https://github.com/crate-ci/typos
> 2. https://github.com/apache/bookkeeper/pull/4375/files
> 
> Thanks
> ZhangJian He
> Twitter: shoothzj
> Wechat: shoothzj
> 

Re: Discussion on Default Implementation for isDriverMetadataServiceAvailable

[Andrey Yegorov]

I am ok either way.

if this doesn't have default implementation it can't be included into the 
4.17.x releases and will have to wait for 4.18 (breaking change, e.g. pulsar 
implements MetadataClientDriver interface.)

OTOH if this is used for anything beyond tests I'd suggest default 
implementation to return false, but with this we can as well avoid default 
implementation and force people change code instead of failing at some random 
moment. 

On 2024/05/07 23:34:31 ZhangJian He wrote:
> Hi, bookKeepers
> 
> I've reviewed the PR that introduces `isDriverMetadataServiceAvailable`. I
> have concerns about providing a default implementation that returns a
> constant value like true, it's not a default interface like this.
> 
> https://github.com/apache/bookkeeper/pull/4342#discussion_r1591761669
> 
> ```
> default E getFirstElement() {
> return getElements().get(0);
> }
> 
> List getElements() {
> }
> ```
> 
> - First we don't guarantee/make ABI compatible between minor releases
> - Second, People who implement metadata drivers should implement this
> correctly, it may lead to unintended behavior if it's not properly
> overridden by all implementations.
> 
> I'd suggest making this method abstract instead to avoid potential
> misinterpretation.
> 
> Thanks
> ZhangJian He
> Twitter: shoothzj
> Wechat: shoothzj
> 

Re: [VOTE] BP-66: support throttling for zookeeper read of rereplication

[Andrey Yegorov]

+1 (binding)

--
Andrey

On 2024/04/30 03:05:56 thetumbled wrote:
> Hi, everyone.I would like to start voting thread for BP-66: support 
> throttling for zookeeper read of rereplication.Proposal 
> PR: https://github.com/apache/bookkeeper/pull/4258
> Implementation PR: https://github.com/apache/bookkeeper/pull/4256
> 
> 
> 
> Thanks,
> thetumbled.

Re: [DISCUSS] BP-66: support throttling for zookeeper read of rereplication

[Andrey Yegorov]

+1

I am ok with the change if we maintain default behavior the same as now (no
throttling).
I left comments on implementation PR.

On Mon, Apr 1, 2024 at 1:54 AM thetumbled  wrote:

> Hi all,
> I create a new BP to relieve the pressure of Zookeeper caused by auto
> recovery, Looking forward to your feedback.
> https://github.com/apache/bookkeeper/issues/4257
>
> Best Regards,
> thetumbled.



-- 
Andrey Yegorov

Re: [VOTE] Release 4.17.0, release candidate #0

[Andrey Yegorov]

+1 (binding)

- built from tag
- built from archived sources
- ran small subset of unit tests
- ran standalone andlocalbookie from packaged bookkeeper-server
- spot-checked staged maven repo
- reviewed release notes

Release notes need double-check, I found one PR listed as a "fix" while 
actually the PR was closed without the merge. The rest lgtm.

Thank you.
--
Andrey

On 2024/04/02 08:33:56 Enrico Olivelli wrote:
> Hi everyone,
> Please review and vote on the release candidate #0 for the version
> 4.17.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed
> to dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.17.0-rc0"  [4] with git sha
> cce4b6461691466c663f2cb4d00dd4d73dd9071e
> 
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> 
> Please download these packages and review this release candidate:
> 
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to release.
> 
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Enrico Olivelli
> 
> [1] 
> https://github.com/apache/bookkeeper/pull/4260/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53
> [2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.17.0-rc0/
> [3] 
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1095/
> [4] https://github.com/apache/bookkeeper/tree/v4.17.0-rc0
> 

Re: [VOTE] Release 4.16.5, release candidate #1

[Andrey Yegorov]

+1 (binding)

- built from archived sources
- built from tag
- ran small set of unittests
- ran BK (localbookie and standalone) from the binaries 
- reviewed release notes

Thank you!

--
Andrey

On 2024/04/03 12:43:45 Nicolò Boschi wrote:
> Hi everyone,
> Please review and vote on the release candidate #1 for the version 4.16.5,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.16.5-rc1" [4] with git sha 7018e3a
> 
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> 
> Please download these packages and review this release candidate:
> 
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important for a release.
> 
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Release Manager
> 
> [1] https://github.com/apache/bookkeeper/pull/4249
> [2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.5-rc1/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1096/
> [4] https://github.com/apache/bookkeeper/tree/v4.16.5-rc1
> 

Re: [VOTE] Release 4.16.4, release candidate #0

[Andrey Yegorov]

+1 (binding)

* reviewed release notes
* spot checked repository
* ran bk from packaged binaries
* built from packaged sources
  ** builds ok with jdk 17
  ** build error with jdk 21. This happens on master too, but I think this
is NOT the release stopper. I added
https://github.com/apache/bookkeeper/issues/4189 to address later


On Thu, Jan 18, 2024 at 6:08 PM Hang Chen  wrote:

> Hi everyone,
> Please review and vote on the release candidate #0 for version 4.16.4,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed
> to dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.16.4-rc0" [4] with git sha [5]
>
> BookKeeper's KEYS file contains the PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review the maven repo, release tag, licenses, and any other things
> you think it is important to release.
>
> The vote will be open for at least 72 hours. It is adopted by the
> majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/bookkeeper/pull/4182/files
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.4-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1091
> [4] https://github.com/apache/bookkeeper/tree/v4.16.4-rc0
> [5] 16de712d8c8f1f0296697331bb68c1ad0eb5787d
>


-- 
Andrey Yegorov

Re: [DISCUSS] BP-64: Improve the description of 47 commands for bin/bookkeeper

[Andrey Yegorov]

+1, I think it is a good idea.
Would be nice to make sure these improvements propagate into the docs on
the website.

On Sun, Oct 8, 2023 at 5:57 AM steven lu  wrote:

> we will improve the description of 47 commands for bin/bookkeeper with
> BP-64
> https://github.com/apache/bookkeeper/issues/4101
>
> # Motivation
>
> When executing the command `./bin/bookkeeper shell`, the usage description
> of each command is not precise enough,
> - some commands have no description;
> - if there are descriptions, the explanation of the parameters is not clear
>
> so it is difficult for us to understand and use each command. we need to
> combine the code to understand it, then it takes unnecessary time.
>
> Related screenshot information:
>
> https://github.com/apache/bookkeeper/assets/42990025/578d0d2f-b443-4f2d-a8f5-d7b3fd1c70c1
> https://github.com/apache/bookkeeper/assets/42990025/578d0d2f-b443-4f2d-a8f5-d7b3fd1c70c1
> ">
>
>
> # Proposal
>
> 1) I plan to improve this function. The first version only involves the
> optimization of the getUsage method of these 47 commands from
> `org.apache.bookkeeper.bookie.BookieShell`
> 2) Use this as the bookkeeper specification. You will also need to follow
> this specification when adding new command codes later.
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.14.8, release candidate #0

[Andrey Yegorov]

+1 (binding)

* reviewed release notes
* spot-checked maven repo
* downloaded binary packages and tried bookie standalone/localbookie

On Fri, Aug 18, 2023 at 2:28 AM Hang Chen  wrote:

> Hi everyone,
> Please review and vote on the release candidate #0 for version 4.14.8,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed
> to dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.14.8-rc0" [4] with git sha [5]
>
> BookKeeper's KEYS file contains the PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review the maven repo, release tag, licenses, and any other things
> you think it is important to release.
>
> The vote will be open for at least 72 hours. It is adopted by the
> majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/bookkeeper/pull/4053/files
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.8-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1088
> [4] https://github.com/apache/bookkeeper/tree/v4.14.8-rc0
> [5] 965c3627328787e2750f41c2a4eda6c1d709d7c6
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.16.3, release candidate #0

[Andrey Yegorov]

+1 (binding)

* reviewed release notes
* spot-checked maven repo
* downloaded binary packages and tried bookie standalone/localbookie


On Thu, Aug 31, 2023 at 4:14 AM Yong Zhang 
wrote:

> Hi everyone,
> Please review and vote on the release candidate #0 for the version 4.16.3,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "release-4.5.0" [4] with git sha
> 5ca147b4a85a1044967002b3f389d17e468a264d
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1]
>
> https://github.com/apache/bookkeeper/pull/4065/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53R4
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.3-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1089
> [4] https://github.com/apache/bookkeeper/releases/tag/v4.16.3-rc0
> [5] 5ca147b4a85a1044967002b3f389d17e468a264d
>


-- 
Andrey Yegorov

Re: integrationTests do not run on CI (and locally)

[Andrey Yegorov]

Steven,

the story is:
- I noticed we don't have integration test for later releases and added the
tests
- I noticed that integration tests do not run at all (not discovered)
- I fixed that and the tests attempted to run but failed

The problem appears to be around the Arquillian test runner and I am not
familiar with it / it seems to be obsolete (?).
If anyone here is familiar with Arquillian and can easily fix the tests it
would be amazing.

I didn't have time to investigate much deeper. Internally we talked about
it with the team and the consensus was that it makes sense to rewrite the
tests to testcontainers (as it is used in other integration tests,
including Pulsar's) but given the schedule/priorities we can't fix it right
now.


On Wed, Jul 5, 2023 at 2:28 AM steven lu  wrote:

> +1 (non-binding)
> but workflow run failed, this pr cannot be merged, please see below for
> details
> https://github.com/apache/bookkeeper/pull/3981#issuecomment-1617613606
>
> Andrey Yegorov  于2023年6月7日周三 06:38写道：
>
> > Hi,
> >
> > We have a few problems related to integration tests:
> >
> > 1. backwards compat tests did not include recent releases.
> > https://github.com/apache/bookkeeper/pull/3981 is supposed to fix that
> >
> > 2. Integration/backwards compat tests do not run on CI
> > E.g.:
> >
> >
> > mvn -B -nsu -DintegrationTests -pl :upgrade test
> > ..
> > [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> >
> >
> > and the same is with other tests in the integration test suites. CI does
> > not fail in this case.
> >
> > This means that we didn't have backwards compat tests executed during the
> > last releases.
> >
> > For the context, I started on this path as a follow up to a report of
> > problems during BK/pulsar upgrade
> >
> https://urldefense.com/v3/__https://apache-pulsar.slack.com/archives/C5ZSVEN4E/p1686019408280939__;!!PbtH5S7Ebw!cu0XwMngGAbPD4B-um2NYpkWL8ChQCqk79kBiCxkc7uNnEfokjLxK4V6H-UtyZvyJLE-zNpSHkOOtOrgSh8PqNYDA4c$
> >
> > Adding
> > ```
> > 
> >   org.junit.vintage
> >   junit-vintage-engine
> >   ${junit5.version}
> >   test
> > 
> > ```
> > to the backwards compat modules makes mvn find the test but there are
> > missing dependencies and/or some conflicts in the runtime that I haven't
> > figured out.
> >
> > I'd appreciate help from someone who understands what the heck Arquillian
> > test runner (used in groovy tests) needs.
> >
> > --
> > Andrey
> >
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.16.2, release candidate #1

[Andrey Yegorov]

+1 (binding)

* localbookie and standalone bookie start ok
* built from archived sources (mac intel, jdk 11)
* built from tag


On Mon, Jun 26, 2023 at 12:17 AM Yong Zhang 
wrote:

> Hi everyone,
> Please review and vote on the release candidate #1 for the version 4.16.2,,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "release-4.5.0" [4] with git sha
> 0465f7e6ebc2289334cc97b06fbfb0a8f07e2e8b
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1]
>
> https://github.com/apache/bookkeeper/pull/4002/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53R4
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.2-rc1/
> [3]
>
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1086/
> [4] https://github.com/apache/bookkeeper/releases/tag/v4.16.2-rc1
>


-- 
Andrey Yegorov

integrationTests do not run on CI (and locally)

[Andrey Yegorov]

Hi,

We have a few problems related to integration tests:

1. backwards compat tests did not include recent releases.
https://github.com/apache/bookkeeper/pull/3981 is supposed to fix that

2. Integration/backwards compat tests do not run on CI
E.g.:


mvn -B -nsu -DintegrationTests -pl :upgrade test
..
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0


and the same is with other tests in the integration test suites. CI does
not fail in this case.

This means that we didn't have backwards compat tests executed during the
last releases.

For the context, I started on this path as a follow up to a report of
problems during BK/pulsar upgrade
https://apache-pulsar.slack.com/archives/C5ZSVEN4E/p1686019408280939

Adding
```

  org.junit.vintage
  junit-vintage-engine
  ${junit5.version}
  test

```
to the backwards compat modules makes mvn find the test but there are
missing dependencies and/or some conflicts in the runtime that I haven't
figured out.

I'd appreciate help from someone who understands what the heck Arquillian
test runner (used in groovy tests) needs.

--
Andrey

Re: [DISCUSS] BookKeeper 4.14.8 release

[Andrey Yegorov]

+1

On Tue, May 30, 2023 at 2:09 AM Hang Chen  wrote:

> Hi everyone,
>
> BookKeeper 4.14.7 was released more than 3 months, and I would like to
> discuss the starting 4.14.8 release to include some critical bug fixes
> and improvements, such as [1] and [2]
>
> Here are the PRs for 4.14.8:
>
> https://github.com/apache/bookkeeper/pulls?q=is%3Aopen+is%3Apr+label%3Arelease%2F4.14.8
>
> If you have PRs that want to be included in this release, please
> comment on that PR or tag the label `release/4.14.8`.
>
> After those PRs merged, I would like to volunteer to release this patch.
>
> Thanks,
> Hang
>
> [1] https://github.com/apache/bookkeeper/pull/3959
> [2] https://github.com/apache/bookkeeper/pull/3917
>


-- 
Andrey Yegorov

Re: Introduce cold ledger storage layer.

[Andrey Yegorov]

gt; of
>file writes until the disk space recovers above the threshold.
>    4. A new thread, ColdStorageArchiveThread, is introduced to periodically
>evict data from the disk cache to the Ledger data disk.
>


-- 
Andrey Yegorov

Re: [DISCUSS] new idea: reverse reading a topic

[Andrey Yegorov]

Bookkeeper's ledger is a distributed WAL with random reads support.
ledger.readAsync(long firstEntry, long lastEntry) works if firstEntry ==
lastEntry
With that one can pipeline reads of as many entries in parallel as needed,
no BK API changes needed for such a niche case.

Reading backwards may backfire in performance, as there is a caching layer
in BK + OS pages data in memory in anticipation of sequential reads.
You can experiment with OS tuning and some BK parameters, with modern SSD
drives backwards reads should have minimal perf differences.

Majority of your changes are in pulsar, I haven't looked closely but my
best guess is that it will affect topic truncation that relies on
subscription moving forwards.

FWIW, consumer has seek() API
https://github.com/apache/pulsar/blob/e5a833a2dcb7ce13ada4ca94714cc045a02de276/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Consumer.java#L482
You can seek to N messages back from last offset, read messages in memory
forward sequentially, reverse, handle, repeat.

As for the PR and idea overall, I'd suggest calling into one of the Pulsar
community meetings to get faster feedback.

On Mon, Mar 6, 2023 at 7:37 AM Alexandre DUVAL  wrote:

> Hi,
>
> I'm wondering if it is possible to introduce a new feature on Pulsar
> which will enable users to read topic from a defined MessageId to
> previous messages until the begin of the topic.
>
> I tried to use Pulsar SQL but it requires so much RAM even for little
> queries (due to Presto design).
>
> Currently, every read in Pulsar are expected to be going forward. So it
> might be a bit tricky to prevent every weird behavior by introducing the
> feature.
>
> I'm currently tried to make an MVP/POC by introducting a readReverse
> field in the CommandSubscribe that is used by ReaderAPI and currently
> looking for to create a getFirstMessageId() on ManagedLedger
> (https://github.com/CleverCloud/pulsar/pull/3). I also removed
> startPosition < endPosition sanity checks in BookKeeper locally
> (https://github.com/CleverCloud/bookkeeper/pull/2).
>
> We definitely prefer a readPrevious(), hasPreviousMessageAvailable() in
> the ReaderAPI.
>
> I'm not familiar with these internals such as NonDurableCursor,
> RangeEntryCache, ManagedCursor so it's a bit tricky.
>
> So I wondering someone to help/guide me or even directly handle the
> subject (or the discuss).
>
> Regards,
>
> Kannar
>
>
>

-- 
Andrey Yegorov

Re: [VOTE] Release 4.16.1, release candidate #0

[Andrey Yegorov]

+1 (binding)

* reviewed release notes

* built from tag, Mac/Java 11

* checked signatures (ok) and checksums (ok)

* bin/bookkeeper standalone starts (from packaged binaries)

* bin/bookkeeper localbookie 3 starts (from packaged binaries)



On Fri, Apr 14, 2023 at 9:18 PM Hang Chen  wrote:

> Hi everyone,
> Please review and vote on the release candidate #0 for the version 4.16.1,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.16.1-rc0" [4] with git sha [5]
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/bookkeeper/pull/3922/files
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.1-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1084/
> [4] https://github.com/apache/bookkeeper/tree/v4.16.1-rc0
> [5] 8122a87c78a0159e422fc0baac43201eda44ae94
>

Re: [VOTE] Release 4.16.0, release candidate #2

[Andrey Yegorov]

> Have you ever run `mvn clean insteall -DskipTests` before
running the tests

of course.

as listed above:
$ git checkout v4.16.0-rc2
$ mvn clean install -DskipTests
$ mvn test

On Fri, Mar 31, 2023 at 8:19 PM Hang Chen  wrote:

> > I added the logs to the issue.
>
> It looks like the loopback address problem, and the Bookie server does
> not allow loopback interfaces as the primary interface. You can set
> allowLoopback to true and rerun the test.
>
>
> > FWIW I ran the tests from the command line, no from IntelliJ.
>
> >$ mvn -v
> >Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
> >Maven home: /usr/local/Cellar/maven/3.8.6/libexec
> >Java version: 11.0.11, vendor: AdoptOpenJDK, runtime:
>
> >/Users/andreyyegorov/Library/Java/JavaVirtualMachines/adopt-openjdk-11.0.11/Contents/Home
> >Default locale: en_US, platform encoding: UTF-8
> >OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac"
>
> I ran the test from the command line without any changes, and the tests
> passed.
> $ mvn test -Dtest=DbLedgerStorageWithDirectEntryLoggerTest
> [INFO] Running
> org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageWithDirectEntryLoggerTest
> [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 16.021 s - in
>
> org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageWithDirectEntryLoggerTest
>
> $ mvn -v
> Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
> Maven home: /Users/hangc/Cloud/apache-maven-3.8.6
> Java version: 11.0.12, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk-11.0.12.jdk/Contents/Home
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac"
>
> @Andrey Have you ever run `mvn clean insteall -DskipTests` before
> running the tests?
>
> Thanks,
> Hang
>
> Matteo Merli  于2023年4月1日周六 01:32写道：
> >
> > +1 binding
> >
> > * Checked signatures
> > * Check localbookie and local perf tool
> >
> >
> >
> > --
> > Matteo Merli
> > 
> >
> >
> > On Wed, Mar 29, 2023 at 7:06 AM Hang Chen  wrote:
> >
> > > Hi everyone,
> > > Please review and vote on the release candidate #2 for version 4.16.0,
> > > as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * Release notes [1]
> > > * The official Apache source and binary distributions to be deployed to
> > > dist.apache.org [2]
> > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > * Source code tag "v4.16.0-rc2" [4] with git sha [5]
> > >
> > > BookKeeper's KEYS file contains the PGP keys we used to sign this
> release:
> > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > >
> > > Please download these packages and review this release candidate:
> > >
> > > - Review release notes
> > > - Download the source package (verify shasum, and asc) and follow the
> > > instructions to build and run the bookkeeper service.
> > > - Download the binary package (verify shasum, and asc) and follow the
> > > instructions to run the bookkeeper service.
> > > - Review the maven repo, release tag, licenses, and any other things
> you
> > > think
> > > it is important to release.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by the
> majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Release Manager
> > >
> > > [1] https://github.com/apache/bookkeeper/pull/3882/files
> > > [2]
> > >
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.0-rc2/
> > > [3]
> > >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1083/
> > > [4] https://github.com/apache/bookkeeper/tree/v4.16.0-rc2
> > > [5] 8890a61418e3961e108b3c2e8ea176691e5f
> > >
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.16.0, release candidate #2

[Andrey Yegorov]

I added the logs to the issue.
FWIW I ran the tests from the command line, no from IntelliJ.

$ mvn -v
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /usr/local/Cellar/maven/3.8.6/libexec
Java version: 11.0.11, vendor: AdoptOpenJDK, runtime:
/Users/andreyyegorov/Library/Java/JavaVirtualMachines/adopt-openjdk-11.0.11/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac"

On Fri, Mar 31, 2023 at 9:42 AM Hang Chen  wrote:

> Hi Andrey,
> Thanks for your great job!
>
> For the failed tests,
> - org.apache.bookkeeper.test.BookieZKExpireTest is a test issue, and I
> will fix it later. It is not a blocker for this release.
> - org.apache.bookkeeper.client.TestSpeculativeRead, I have run the
> test on my Mac multiple times, and they are all passed. Would you
> please share the detailed logs with me? I think it also may not be a
> blocker for this release.
> - Direct entry logger tests fail on mac (probably by design, but tests
> should handle this / don't run on mac). I replied under this issue
> [1]. To make the tests run in the Intellij idea, we need to copy the
> `libnative-io.jnilib` file to a target lib directory. After copying
> the file, all the direct entry logger tests are passed on my Macbook.
>
> For the gpg warning, my gpg key has been signed by two guys [2]. I
> have no idea why it still reports not certified with a trusted
> signature.
>
> Thanks,
> Hang
>
> [1]
> https://github.com/apache/bookkeeper/issues/3897#issuecomment-1492138987
> [2]
> https://urldefense.com/v3/__https://pgp.mit.edu/pks/lookup?op=vindex&search=0xA641378C97944D34__;!!PbtH5S7Ebw!ekniQYxhGs8shdCYecyXm2-Rz7C3Og-OW-oWG3SLnQdQ1Jc93wyFh9jiohPMukFz45zKvC3sz0n1OWIqugtVYMlb$
>
> Andrey Yegorov  于2023年3月31日周五 04:58写道：
> >
> > +1 (binding)
> >
> > There are test failures on mac but I believe they fail only on mac
> (please
> > double check).
> > I briefly looked at these but had to stop/timed out.
> >
> > What I did:
> >
> > * reviewed release notes (left some comments)
> >
> > * built/test from tag, Mac/Java 11
> >
> > $ git checkout v4.16.0-rc2
> > $ mvn clean install -DskipTests
> > $ mvn test
> >
> > Failures:
> > 1. org.apache.bookkeeper.test.BookieZKExpireTest - added
> > https://github.com/apache/bookkeeper/issues/3898
> > 2. org.apache.bookkeeper.client.TestSpeculativeRead - added
> > https://github.com/apache/bookkeeper/issues/3899
> > 3. Direct entry logger tests fail on mac (probably by design but tests
> > should handle this / don't run on mac) - added
> > https://github.com/apache/bookkeeper/issues/3897
> >
> > * checked signatures (ok) and checksums (ok)
> >
> > $ cd ~/Downloads
> > $ wget --no-parent -r https://dist.apache.org/repos/dist/dev/bookkeeper/
> > bookkeeper-4.16.0-rc2/
> > $ cd dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.0-rc2
> > $ gpg --keyserver hkps://pgp.mit.edu/ --search-keys " > email here>"
> > $ for f in *.asc; do gpg --verify $f ${f%.*}; done
> >
> > All is good, there is a warning
> >gpg: WARNING: This key is not certified with a trusted signature!
> >gpg:  There is no indication that the signature belongs to the
> > owner.
> > I think you may need to "verify" your key, don't remember how.
> >
> > $ for f in *.tar.gz; do shasum -a 512 $f | diff $f.sha512 -; done
> >
> > * bin/bookkeeper standalone starts (from packaged binaries)
> >
> > * bin/bookkeeper localbookie 3 starts (from built binaries)
> >
> > * compared gzipped source with sources at tag (ok)
> >
> > * built from gzipped sources (didn't run the tests)
> >
> >
> > On Wed, Mar 29, 2023 at 7:07 AM Hang Chen  wrote:
> >
> > > Hi everyone,
> > > Please review and vote on the release candidate #2 for version 4.16.0,
> > > as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * Release notes [1]
> > > * The official Apache source and binary distributions to be deployed to
> > > dist.apache.org [2]
> > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > * Source code tag "v4.16.0-rc2" [4] with git sha [5]
> > >
> > > BookKeeper's KEYS file contains the PGP keys we used to sign this
> release:
> &g

Re: [VOTE] Release 4.16.0, release candidate #2

[Andrey Yegorov]

+1 (binding)

There are test failures on mac but I believe they fail only on mac (please
double check).
I briefly looked at these but had to stop/timed out.

What I did:

* reviewed release notes (left some comments)

* built/test from tag, Mac/Java 11

$ git checkout v4.16.0-rc2
$ mvn clean install -DskipTests
$ mvn test

Failures:
1. org.apache.bookkeeper.test.BookieZKExpireTest - added
https://github.com/apache/bookkeeper/issues/3898
2. org.apache.bookkeeper.client.TestSpeculativeRead - added
https://github.com/apache/bookkeeper/issues/3899
3. Direct entry logger tests fail on mac (probably by design but tests
should handle this / don't run on mac) - added
https://github.com/apache/bookkeeper/issues/3897

* checked signatures (ok) and checksums (ok)

$ cd ~/Downloads
$ wget --no-parent -r https://dist.apache.org/repos/dist/dev/bookkeeper/
bookkeeper-4.16.0-rc2/
$ cd dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.0-rc2
$ gpg --keyserver hkps://pgp.mit.edu/ --search-keys ""
$ for f in *.asc; do gpg --verify $f ${f%.*}; done

All is good, there is a warning
   gpg: WARNING: This key is not certified with a trusted signature!
   gpg:  There is no indication that the signature belongs to the
owner.
I think you may need to "verify" your key, don't remember how.

$ for f in *.tar.gz; do shasum -a 512 $f | diff $f.sha512 -; done

* bin/bookkeeper standalone starts (from packaged binaries)

* bin/bookkeeper localbookie 3 starts (from built binaries)

* compared gzipped source with sources at tag (ok)

* built from gzipped sources (didn't run the tests)


On Wed, Mar 29, 2023 at 7:07 AM Hang Chen  wrote:

> Hi everyone,
> Please review and vote on the release candidate #2 for version 4.16.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.16.0-rc2" [4] with git sha [5]
>
> BookKeeper's KEYS file contains the PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review the maven repo, release tag, licenses, and any other things you
> think
> it is important to release.
>
> The vote will be open for at least 72 hours. It is adopted by the majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/bookkeeper/pull/3882/files
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.0-rc2/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1083/
> [4] https://github.com/apache/bookkeeper/tree/v4.16.0-rc2
> [5] 8890a61418e3961e108b3c2e8ea176691e5f
>


-- 
Andrey Yegorov

Re: Bookkeeper decommission may be blocked by ledgers that cannot be replicated

[Andrey Yegorov]

Hi,

You can use "recover" command instead.

Switch bookie to read-only (via REST API)
bin/bookkeeper shell recover ..
recover command also has a flag to delete the cookie in ZK.
As an additional benefit, this way you can decomm bookie with ledgers
created with write quorum = 1.

HTH.

On Sun, Mar 26, 2023 at 9:27 PM Hang Chen  wrote:

> Hi guys, I found the BookKeeper decommission may be blocked by ledgers
> that cannot be replicated.
>
> Current bookie decommissions process.
>   - Step 1: Use the command `bin/bookkeeper shell listunderreplicated`
> to check whether there are some ledgers in the under-replicated state
>   - Step 2: After all the ledgers are replicated complete, stop the
> bookie and use the command `bin/bookkeeper shell decommissionbookie
> -bookieid ` to trigger decommission
>   - Step 3: Wait for all the ledgers to be replicated and the bookie
> decommission process will complete
>
> However, there is a bug in the decommissioning process.
>
> In Step 1, those under-replicated state ledgers are marked by the
> following steps:
>   - Auditor check lost bookie: it will be triggered by two cases: a)
> One bookie lost after `lostBookieRecoveryDelay`, b) Check every
> `auditorPeriodicBookieCheckInterval`.  The default is 24 hours.
>   - Auditor checks all ledgers: triggered every
> `auditorPeriodicCheckInterval`. The default is 7 days. It will check
> every ledger's fragments with the following steps:
> - For every fragment, calculate pending read entries according to
> `auditorLedgerVerificationPercentage`, default is `0`, which means
> only checking the first and last entries of this fragment.
> - Read those entries from all the bookies in the ensemble list for
> the pending read entries. If any entries read failed, mark the ledger
> into an under-replicated state.
>
>
> When we use the `bin/bookkeeper shell listunderreplicated` command to
> check whether some are under-replicated, it only represents those
> ledgers missing replicas before the last check. The lost bookie check
> was 24 hours ago, and the all ledgers check was seven days ago. The
> time range from the last check to the current timestamp won't mark any
> missing replicas ledgers. Suppose we set EnsembleSize=3,
> WriteQuorumSize=2, and AckQuorumSize=1, and decommission one bookie
> with the current decommission process. In that case, it may result in
> some ledgers that can't be replicated due to the only available
> replica on the decommissioned bookie.
>
> Moreover, the Auditor checks all ledgers and only checks the first and
> last entries of each fragment of those ledgers. If the bookie disabled
> writing journals and some entries are lost in one fragment, but the
> first and last entries still exist, the checker won't find it.
>
> ### Options
> There are two options to tune the decommissioning process.
>
> 1. Trigger-check all ledgers before Step 1. It has the following
> disadvantages.
>- It will cost a lot of resources
>- It only checks the first and last entries of each fragment of
> those ledgers by default. It can't cover all the entries that check
>
>  2. Turn the bookie into read-only mode instead of shutting it down
> before using the `bin/bookkeeper shell decommissionbookie -bookieid
> ` command to trigger commission. When replicating
> ledgers located on the decommission bookie, the ledgers can be
> replicated successfully if one replica is available.
>
> I suggest choosing the second option to tune the current bookie
> decommission process. Do you have any suggestions?
>
> Thanks,
> Hang
>


-- 
Andrey Yegorov

Re: [VOTE] BP-62: Streamline batch add requests

[Andrey Yegorov]

Overall, +1.
I do have some concerns related to handling various limiters, so I left a
comment on the PR.

On Mon, Mar 20, 2023 at 6:05 AM Hang Chen  wrote:

> This is the vote for BP-62.
>
> ### Motivation
> The bookie server process add-entry requests pipeline:
> - Get one request from the Netty socket channel
> - Choose one thread to process the written request
> - Write the entry into the target ledger entry logger's write
> cache(memory table)
> - Put the entry into the journal's pending queue
> - Journal thread takes the entry from the pending queue and writes it
> into PageCache/Journal Disk
> - Write the callback response to the Netty buffer and flush to the
> bookie client side.
>
> For every add entry request, the bookie server needs to go through the
> above steps one by one. It will introduce a lot of thread context
> switches.
>
> We can batch the add requests according to the Netty socket channel,
> and write a batch of entries into the ledger entry logger and journal
> disk.
>
> ### Modifications
> The PR will change the add requests processing pipeline into the
> following steps.
> - Get a batch of add-entry requests from the socket channel until the
> socket channel is empty or reached the max capacity (default is 1_000)
> - Choose one thread to process the batch of add-entry requests.
> - Write the entries into the target ledger entry logger's write cache one
> by one
> - Put the batch of entries into the journal's pending queue
> - Journal thread drains a batch of entries from the pending queue and
> writes them into PageCache/Journal disk
> - Write the callback response to the Netty buffer and flush to the
> bookie client side.
>
> With this change, we can save a lot of thread context switches.
>
>
> ### Performance
> I start one bookie on my laptop and use the Bookkeeper benchmark to
> test the performance
> ```shell
> bin/benchmark writes -ensemble 1 -quorum 1 -ackQuorum 1 -ledgers 50
> -throttle 2
> ```
>
> **Before this change**
>
> | times | ops/sec | p95 latency | p99 latency |
> | --- | --- | --- | --- |
> | 1 | 147507 | 114.93 | 122.42 |
> | 2 | 154571 |  111.46 | 115.86 |
> | 3 | 141459 | 117.23 | 124.18 |
> | 4 | 142037 | 121.75 | 128.54 |
> | 5 | 143682 | 121.05 | 127.97 |
>
>
> **After this change**
>
> | times | ops/sec | p95 latency | p99 latency |
> | --- | --- | --- | --- |
> | 1 | 157328 | 118.30 | 121.79 |
> | 2 | 165774 |  112.86 | 115.69 |
> | 3 | 144790 | 128.94 | 133.24 |
> | 4 | 151984 | 121.88 | 125.32 |
> | 5 | 154574 | 121.57 | 124.57 |
>
> The new change has a 2.2% improvement.
>
> Please leave +1/-1 in this thread to join the vote. And feel free to
> leave any concerns.
>
> Thanks,
> Hang
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.15.4, release candidate #0

[Andrey Yegorov]

+1 (binding)

* reviewed release notes
* tested standalone & localbookie
* built from sources (Mac, OpenJDK 17)
* ran some tests for bookkeeper-server

On Wed, Mar 1, 2023 at 11:30 PM Yong Zhang 
wrote:

> Hi everyone,
> Please review and vote on the release candidate #1 for version 4.15.4,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.15.4-rc0" [4] with git sha [5]
>
> BookKeeper's KEYS file contains the PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review the maven repo, release tag, licenses, and any other things you
> think
> it is important to release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/bookkeeper/pull/3831/files
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.4-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1080
> [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.4-rc0
> [5] 44f87efb8121a9f5428e7ddc70656aee85f563bf
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.14.7, release candidate #1

[Andrey Yegorov]

+1 (binding)

* tested localbookie
* tested standalone bookie
* built from sources (Mac/Intel)
 * ran some tests

Thank you for driving the release!

On Wed, Feb 15, 2023 at 12:08 AM Yong Zhang 
wrote:

> +1 (non-binding)
>
> - verified signatures and checksums
> - run simple test with localbookie
> - built from sources
>
> Yong
>
> On Tue, 14 Feb 2023 at 23:32, Enrico Olivelli  wrote:
>
> > +1 (binding)
> > - run tests locally
> > - built from sources
> > - verified signatures and checksums
> > - run Apache RAT for headers
> > - verified LICENSE using the script that we have in dev/
> >
> > Thanks for driving the release
> >
> > Enrico
> >
> > Il giorno mar 14 feb 2023 alle ore 05:39 Hang Chen
> >  ha scritto:
> > >
> > > I opened one PR[1] to upgrade BookKeeper dependency to 4.14.7 for
> > > Pulsar branch-2.10 and all tests passed. Please help verify this
> > > candidate. Thanks a lot.
> > >
> > > Thanks,
> > > Hang
> > >
> > > [1] https://github.com/hangc0276/pulsar/pull/9
> > >
> > > Hang Chen  于2023年2月13日周一 21:17写道：
> > > >
> > > > Hi everyone,
> > > > Please review and vote on the release candidate #1 for version
> 4.14.7,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * Release notes [1]
> > > > * The official Apache source and binary distributions to be deployed
> to
> > > > dist.apache.org [2]
> > > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > > * Source code tag "v4.14.7-rc1" [4] with git sha [5]
> > > >
> > > > BookKeeper's KEYS file contains the PGP keys we used to sign this
> > release:
> > > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > >
> > > > Please download these packages and review this release candidate:
> > > >
> > > > - Review release notes
> > > > - Download the source package (verify shasum, and asc) and follow the
> > > > instructions to build and run the bookkeeper service.
> > > > - Download the binary package (verify shasum, and asc) and follow the
> > > > instructions to run the bookkeeper service.
> > > > - Review the maven repo, release tag, licenses, and any other things
> > you think
> > > > it is important to release.
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by the
> > majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Release Manager
> > > >
> > > > [1] https://github.com/apache/bookkeeper/pull/3770/files
> > > > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.7-rc1/
> > > > [3]
> >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1079
> > > > [4] https://github.com/apache/bookkeeper/tree/v4.14.7-rc1
> > > > [5] c22136b03489db1643521f586e9cae2c4a511e10
> >
>


-- 
Andrey Yegorov

Re: [Discuss] BookKeeper upgrade the RocksDB version to 7.9.x

[Andrey Yegorov]

+1 to the plan.
We can configure RocksDb to not use kXXH3 in all branches and provide a
config option for the user to enable it when needed.

On Sun, Jan 29, 2023 at 2:54 AM Hang Chen  wrote:

> Hi guys,
> When we introduced `deleteRange` for ledger index deletion, we
> found a segment fault in ledger index deletion [1]. I suggest
> upgrading the RocksDB version to the latest 7.9.x to see if the
> RocksDB deleteRange segment fault bug has been fixed. I have tested
> upgrading the RocksDB version from 6.10.2 to 7.9.2 and rollback to
> 6.10.2. This is the testing result.
>
> - The upgrade process works fine in the following cases
>   - Pulsar producer keeps producing messages to Pulsar topic
>   - Pulsar consumer consumes messages from the Pulsar topic, and the
> Pulsar broker fetches messages from the BookKeeper cluster
>   - Trigger compaction to cleanup expired ledgers
>
> - Rollback RocksDB version from 7.9.2 to 6.10.2, and bookie failed to
> startup with the following exception.
>
> ```java
> 2023-01-29T17:09:27,794+0800 [main] ERROR
> org.apache.bookkeeper.server.Main - Failed to build bookie server
> java.io.IOException: Error open RocksDB database
> at
> org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.(KeyValueStorageRocksDB.java:200)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.(KeyValueStorageRocksDB.java:89)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:63)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.(LedgerMetadataIndex.java:68)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.(SingleDirectoryDbLedgerStorage.java:170)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:150)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:129)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at org.apache.bookkeeper.bookie.Bookie.(Bookie.java:819)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:120)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.server.service.BookieService.(BookieService.java:52)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at org.apache.bookkeeper.server.Main.doMain(Main.java:226)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at org.apache.bookkeeper.server.Main.main(Main.java:208)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> Caused by: org.rocksdb.RocksDBException: unknown checksum type 4 in
> data/bookkeeper/ledgers/current/ledgers/25.sst offset 1078 size 33
> at org.rocksdb.RocksDB.open(Native Method)
> ~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
> at org.rocksdb.RocksDB.open(RocksDB.java:239)
> ~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
> at
> org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.(KeyValueStorageRocksDB.java:197)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> ... 13 more
> ```
> The root cause of this exception is that RocksDB 7.9.2 uses `kXXH3`
> by default, and `kXXH3` is only supported since RocksDB 6.27
>
> https://github.com/facebook/rocksdb/blob/79e57a39a33dbe17c8f51167e40e66d6c91f8eb4/include/rocksdb/table.h#L56
>
> For the BookKeeper master branch, we have upgraded the RocksDB to
> `6.29.4.1`, which can support RocksDB upgrade to 7.9.2 and rollback to
> 6.29.4.1.
> For the RocksDB < 6.27, we can push a fix to ensure RocksDB 7.9.2 does
> not use the latest checksum type `kXXH3`
>
> I suggest doing this upgradation. Do you have any concerns?
>
> [1] https://github.com/apache/bookkeeper/issues/3734
>
> Thanks,
> Hang
>


-- 
Andrey Yegorov

Re: [DISCUSS] BP-61: Revert BP-59 to release ByteBuf using ByteBuf.release() instead of ReferenceCountUtil.safeRelease()

[Andrey Yegorov]

 necessary to revert these PRs. We can discuss
> what is
> > > >> the correct way to release, and then modify these three ways into
> the
> > > >> correct way.
> > > >>
> > > >> Hang Chen  于2023年1月30日周一 12:28写道：
> > > >>
> > > >>> Hi guys,
> > > >>>   In BP-59, which was not discussed in the dev mail list and
> without
> > > >>> a vote, refactored the ByteBuf release method by
> > > >>> ReferenceCountUtil.safeRelease() instead of ByteBuf.release().
> > > >>>
> > > >>> In the `ReferenceCountUtil.safeRelease()`, it catches the release
> > > >>> exception. This change can make the ByteBuf release without any
> > > >>> exceptions and makes the code run well, but it will make bugs hide
> > > >>> deeper and more challenging to find out.
> > > >>>   ```java
> > > >>>   public static void safeRelease(Object msg) {
> > > >>>try {
> > > >>>release(msg);
> > > >>>} catch (Throwable t) {
> > > >>>logger.warn("Failed to release a message: {}", msg, t);
> > > >>>}
> > > >>>}
> > > >>>   ```
> > > >>> For example, if there is a bug to release the ByteBuf twice, whose
> > > >>> refcnt is 1, we can find out the bug quickly if we use
> > > >>> ByteBuf.release(), but the bug will be hard to find out if we use
> > > >>> `ReferenceCountUtil.safeRelease()`
> > > >>>
> > > >>> There are 12 PRs to refactor the ByteBuf release method, and I
> suggest
> > > >>> reverting those PRs. Do you have any ideas?
> > > >>>
> > > >>> https://github.com/apache/bookkeeper/pull/3673
> > > >>> https://github.com/apache/bookkeeper/pull/3674
> > > >>> https://github.com/apache/bookkeeper/pull/3687
> > > >>> https://github.com/apache/bookkeeper/pull/3688
> > > >>> https://github.com/apache/bookkeeper/pull/3689
> > > >>> https://github.com/apache/bookkeeper/pull/3691
> > > >>> https://github.com/apache/bookkeeper/pull/3693
> > > >>> https://github.com/apache/bookkeeper/pull/3694
> > > >>> https://github.com/apache/bookkeeper/pull/3695
> > > >>> https://github.com/apache/bookkeeper/pull/3698
> > > >>> https://github.com/apache/bookkeeper/pull/3700
> > > >>> https://github.com/apache/bookkeeper/pull/3703
> > > >>>
> > > >>>
> > > >>> Thanks,
> > > >>> Hang
> > > >>>
> > >
>


-- 
Andrey Yegorov

Re: [DISCUSS] BP-61: Revert BP-59 to release ByteBuf using ByteBuf.release() instead of ReferenceCountUtil.safeRelease()

[Andrey Yegorov]

Hi,

I think it makes sense to replace ReferenceCountUtil.safeRelease
with ReferenceCountUtil.release.
FWIW ReferenceCountUtil.release's return value just says if the byte buf's
ref count reached zero or not, it is not an error handling mechanism.

safeRelease logs at warn level. Have you seen these messages in your prod
clusters so we can say this an actual problem and not a generic concern
(for prioritization)?

On Sun, Jan 29, 2023 at 8:28 PM Hang Chen  wrote:

> Hi guys,
>In BP-59, which was not discussed in the dev mail list and without
> a vote, refactored the ByteBuf release method by
> ReferenceCountUtil.safeRelease() instead of ByteBuf.release().
>
> In the `ReferenceCountUtil.safeRelease()`, it catches the release
> exception. This change can make the ByteBuf release without any
> exceptions and makes the code run well, but it will make bugs hide
> deeper and more challenging to find out.
>```java
>public static void safeRelease(Object msg) {
> try {
> release(msg);
> } catch (Throwable t) {
> logger.warn("Failed to release a message: {}", msg, t);
> }
> }
>```
> For example, if there is a bug to release the ByteBuf twice, whose
> refcnt is 1, we can find out the bug quickly if we use
> ByteBuf.release(), but the bug will be hard to find out if we use
> `ReferenceCountUtil.safeRelease()`
>
> There are 12 PRs to refactor the ByteBuf release method, and I suggest
> reverting those PRs. Do you have any ideas?
>
> https://github.com/apache/bookkeeper/pull/3673
> https://github.com/apache/bookkeeper/pull/3674
> https://github.com/apache/bookkeeper/pull/3687
> https://github.com/apache/bookkeeper/pull/3688
> https://github.com/apache/bookkeeper/pull/3689
> https://github.com/apache/bookkeeper/pull/3691
> https://github.com/apache/bookkeeper/pull/3693
> https://github.com/apache/bookkeeper/pull/3694
> https://github.com/apache/bookkeeper/pull/3695
> https://github.com/apache/bookkeeper/pull/3698
> https://github.com/apache/bookkeeper/pull/3700
> https://github.com/apache/bookkeeper/pull/3703
>
>
> Thanks,
> Hang
>


-- 
Andrey Yegorov

[DISCUSS] RocksDB: segfault in org.rocksdb.WriteBatch::delete / batch delete

[Andrey Yegorov]

Hi,

I've encountered an interesting segfault in prod, so far it is not a
widespread issue.
Details are at https://github.com/apache/bookkeeper/issues/3734

I'd like to check with the community if anyone encountered this issue in
their environments and discuss the best approach for fixing this that will
benefit the community in the long term.

My primary concern is that the problem starts happening frequently or/and
causes data corruption in the RocksDB.

Andrey.

Re: BP-60:Change PCBC limitStatsLogging default value to true

[Andrey Yegorov]

+1

On Mon, Jan 9, 2023 at 4:50 AM Enrico Olivelli  wrote:

> I agree with the change.
> I have never used those metrics
>
> Enrico
>
>
> Il giorno lun 26 dic 2022 alle ore 10:29 Wenbing Shen
>  ha scritto:
> >
> > Hi BookKeepers, I've changed the limitStatsLogging default value to true
> > from false:
> > BP-60 <https://github.com/apache/bookkeeper/issues/3718>
> >
> > Motivation
> >
> > We have an efficient online bookie cluster with hundreds of bookie nodes
> > deployed on SSD disks.
> >
> > We separate the AutoRecovery cluster and the Bookie cluster for
> independent
> > deployment.
> >
> > I observed that our AutoRecovery cluster GC is very frequent. After
> > investigation, I found that the limitStatsLogging of the bookkeeper
> client
> > PCBC is disabled by default, and a large number of channel monitoring
> > indicators are generated. Due to the large number of bookie cluster
> nodes,
> > this metric data occupies a large amount of heap memory.
> >
> > A single StringWriter object occupies 16MB of memory, of which nearly 70
> > StringWriter objects are waiting for the next GC to be destroyed,
> occupying
> > 1GB+ heap memory.
> > Proposal
> >
> > In my use, I haven't found any usefulness of these PCBC monitoring
> metrics
> > data, at least so far, I haven't used it effectively.
> >
> > If our AutoRecovery and Bookie cluster are mixed in one process, these
> > large objects will affect the performance and stability of Bookie
> cluster.
> >
> > Since I can't find the meaning of these metrics by default, I suggest to
> > adjust the default value of limitStatsLogging to true.
> >
> > Everyone can choose to turn it on or off, but by default, it is difficult
> > for users to find out what effect this parameter will have, so that when
> > their cluster grows to hundreds or thousands, when they realize the
> problem
> > sometimes, it is necessary to restart hundreds to thousands of bookies
> in a
> > rolling manner.
> >
> > At the same time, I observed that in pulsar, various monitoring of the
> > bookkeeper client is turned off by default, because they really affect
> the
> > performance of the pulsar service, which is enough to show that we should
> > try to change it, especially some very redundant metrics created based on
> > channels.
> > Compatibility, Deprecation, and Migration PlanClients that rely on PCBC
> > metrics monitoring need to pay attention to this upgrade, but this will
> not
> > affect the actual functions of the client, only the metrics data, and
> users
> > can choose to open it again.
> >
> >
> > What do you think about it?
> >
> > Best.
> > Wenbing
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.14.6, release candidate #1

[Andrey Yegorov]

+1 (binding)

* ran bookkeeper standalone
* ran bookkeeper localbookie
* built from sources (mac, intel, jdk 11), ran some tests
* spot-checked maven repo


On Mon, Dec 19, 2022 at 5:33 PM Hang Chen  wrote:

> I opened one PR[1] to upgrade BookKeeper dependency to 4.14.6 for
> Pulsar branch-2.10, and all tests passed. Please help verify this
> candidate.
>
> Thanks,
> Hang
>
> [1] https://github.com/hangc0276/pulsar/pull/8
>
> Hang Chen  于2022年12月19日周一 13:14写道：
> >
> > Hi everyone,
> > Please review and vote on the release candidate #1 for version 4.14.6,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "v4.14.6-rc1" [4] with git sha [5]
> >
> > BookKeeper's KEYS file contains the PGP keys we used to sign this
> release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review the maven repo, release tag, licenses, and any other things you
> think
> > it is essential to release.
> >
> > The vote will be open for at least 72 hours. The majority adopts it
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1] https://github.com/apache/bookkeeper/pull/3659/files
> > [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.6-rc1/
> > [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1077/
> > [4] https://github.com/apache/bookkeeper/tree/v4.14.6-rc1
> > [5] a96884ec279f5fbcd628c7ae3791d99a8e19ca69
>


-- 
Andrey Yegorov

Re: Website / herokuapp not available

[Andrey Yegorov]

Matteo fixed this: https://github.com/apache/bookkeeper/pull/3712

On Thu, Dec 22, 2022 at 7:11 AM Dave Fisher  wrote:

>
>
> Sent from my iPhone
>
> > On Dec 22, 2022, at 1:18 AM, Nicolò Boschi  wrote:
> >
> > We could use
> https://urldefense.com/v3/__https://communityinviter.com/__;!!PbtH5S7Ebw!enYnjSdxa6RUURqiUX-hMSIy2VkKSHNr4GZ9yN32OTc1MdGiYIDynWnsFNCzpLiqirMgy8gkBUFdZ4QjYNc$
> like in Pulsar
> > I think we just need a Slack admin that creates that app
> >
> > Are PMC members admin or do we have to go through ASF ?
>
> BK’s slack is not in the ASF slack -
> https://urldefense.com/v3/__https://apachebookkeeper.slack.com/__;!!PbtH5S7Ebw!enYnjSdxa6RUURqiUX-hMSIy2VkKSHNr4GZ9yN32OTc1MdGiYIDynWnsFNCzpLiqirMgy8gkBUFdFPrc5F8$
>
> someone in the community must be an admin.
>
> Best,
> Dave
>
> >
> > Nicolò Boschi
> >
> >
> >> Il giorno gio 22 dic 2022 alle ore 04:54 tison 
> ha
> >> scritto:
> >>
> >> This is like https://github.com/apache/pulsar-site/pull/335. Briefly,
> >> Heroku does not provide free tier anymore.
> >>
> >> For BookKeeper's case, I suggest we remove the self-register link now
> until
> >> some contributor builds an alternative.
> >>
> >> Best,
> >> tison.
> >>
> >>
> >> Andrey Yegorov  于2022年12月22日周四 11:50写道：
> >>
> >>> Hi,
> >>>
> >>> Build's dead link checker caught that a link to
> >>>
> https://urldefense.com/v3/__https://apachebookkeeper.herokuapp.com/__;!!PbtH5S7Ebw!enYnjSdxa6RUURqiUX-hMSIy2VkKSHNr4GZ9yN32OTc1MdGiYIDynWnsFNCzpLiqirMgy8gkBUFdKReQvw4$
> is dead.
> >>> Herokuapp is linked at https://bookkeeper.apache.org/community/slack
> >> ("You
> >>> can self-register at ...") and currently returns error 404.
> >>>
> >>> Who maintains/maintained it and knows how to fix this?
> >>>
> >>> --
> >>> Andrey
> >>>
> >>
>


-- 
Andrey Yegorov

Website / herokuapp not available

[Andrey Yegorov]

Hi,

Build's dead link checker caught that a link to
https://apachebookkeeper.herokuapp.com/ is dead.
Herokuapp is linked at https://bookkeeper.apache.org/community/slack ("You
can self-register at ...") and currently returns error 404.

Who maintains/maintained it and knows how to fix this?

--
Andrey

Re: [VOTE] Release 4.14.6, release candidate #0

[Andrey Yegorov]

+1 (binding)

* ran bookkeeper standalone
* ran bookkeeper localbookie
* built from sources (mac, intel), ran some tests
* spot-checked maven repo

@Yong - as I understand, build on M1 is not a regression for 4.14.x so it
should not be a blocker.

On Mon, Nov 21, 2022 at 4:03 AM Hang Chen  wrote:

> Hi everyone,
> Please review and vote on the release candidate #0 for version 4.14.6,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.14.6-rc0" [4] with git sha [5]
>
> BookKeeper's KEYS file contains the PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review the maven repo, release tag, licenses, and any other things you
> think
> it is important to release.
>
> The vote will be open for at least 72 hours. It is adopted by the majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/bookkeeper/pull/3659/files
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.6-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1076/
> [4] https://github.com/apache/bookkeeper/tree/v4.14.6-rc0
> [5] 9f453834c6039fac46dc19130288dbf0b2996f2e
>


-- 
Andrey Yegorov

Re: [ANNOUNCE] Apache BookKeeper 4.15.3 released

[Andrey Yegorov]

Thank you for driving this release!

On Wed, Nov 16, 2022 at 7:01 PM Yong Zhang  wrote:

> The Apache BookKeeper team is proud to announce Apache BookKeeper
> version 4.15.3.
>
> Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage
> service optimized for real-time workloads. It has been used for a
> fundamental
> service to build reliable services. It is also the log segment store for
> Apache
> DistributedLog and the message store for Apache Pulsar.
>
> This is the 35rd release of the Apache BookKeeper.
>
> Release 4.15.3 contains performance and stability improvements, new
> metrics,
> and various bug fixes.
>
> Apache BookKeeper users are encouraged to upgrade to 4.15.3. The technical
> details of this release are summarized below.
> For BookKeeper release details and downloads, visit:
>
> https://bookkeeper.apache.org/releases
>
> BookKeeper 4.15.3 Release Notes are at:
>
> https://bookkeeper.apache.org/release-notes/#4153
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The BookKeeper Team
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.15.3, release candidate #1

[Andrey Yegorov]

Thank you for driving the release!

On Sun, Nov 13, 2022 at 6:29 PM Yong Zhang 
wrote:

> The vote is now closed and the release was approved with
> 4 +1s(3 bindings and 1 non-binding);
>
> Binding votes:
> * Enrico Olivelli
> * Andrey Yegorov
> * Jia Zhai
>
> Non-Binding votes:
> * Hang Chen
>
> Thank you all guys for voting!
>
>
>
> On Sat, 12 Nov 2022 at 16:14, Jia Zhai  wrote:
>
> > +1 (binding)
> >
> > - verified packages checksum and signatures, they look good.
> >
> > - the source package build and test all run successfully.
> >
> > - 'bin/bookkeeper standalone' and 'bin/bookkeeper shell bookiesanity' run
> > well.
> >
> >
> > On Sat, Nov 12, 2022 at 8:54 AM Hang Chen  wrote:
> >
> > > +1 (non-binding)
> > >
> > > - Build from source code (JDK8, maven 3.8.6 on MacOs)
> > > - Verified checksums and signatures
> > > - Verified the licenses
> > > - Run bkperf to test journal write
> > > - Integrate with Pulsar master code and Pulsar unit tests passed
> > > （https://github.com/hangc0276/pulsar/pull/7）
> > >
> > >
> > > Thanks Yong for driving the release
> > >
> > > Thanks,
> > > Hang
> > >
> > > Andrey Yegorov  于2022年11月12日周六 00:08写道：
> > > >
> > > > +1 (binding)
> > > >
> > > > On Thu, Nov 10, 2022 at 8:06 PM Yong Zhang <
> zhangyong1025...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Andrey,
> > > > >
> > > > > I run the test multiple times, I found that depends on how the DNS
> > > > > resolve the address.
> > > > > When the DNS resolves to 127.0.0.1/localhost, the test will be
> > failed.
> > > > > But if it resolves another INet interface, like lo0, the hostname
> > will
> > > be
> > > > > 192.168.50.240(in my computer) and the test passed.
> > > > >
> > > > > Thanks,
> > > > > Yong
> > > > >
> > > > > On Fri, 11 Nov 2022 at 09:46, Yong Zhang <
> zhangyong1025...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > I downloaded the src package and run the tests successfully.
> > > > > > And build from branch-4.15 passed as well. But failed when
> > > > > > I run it with IDE.
> > > > > >
> > > > > > It looks like sometimes it uses the loopback address and we
> > > > > > are not allowed that.
> > > > > > But it shouldn't be a problem for this release.
> > > > > >
> > > > > > Thanks,
> > > > > > Yong
> > > > > >
> > > > > > On Fri, 11 Nov 2022 at 09:01, Andrey Yegorov <
> > > > > andrey.yego...@datastax.com>
> > > > > > wrote:
> > > > > >
> > > > > >> I tested the following:
> > > > > >>
> > > > > >> * bookkeeper-all
> > > > > >>   ** unpacks
> > > > > >>   ** shell works, standalone starts, localbookie starts
> > > > > >> * src
> > > > > >>   ** unpacks, builds
> > > > > >>   ** tests - there are some know flakes
> > > > > >> and org.apache.bookkeeper.client.TestSpeculativeRead fails for
> me
> > > > > >>
> > > > > >> I don't recall any issues with TestSpeculativeRead in the
> previous
> > > > > >> releases; does it repro for you (I built from unpacked sources
> for
> > > the
> > > > > >> RC)?
> > > > > >>
> > > > > >> $ mvn test -f bookkeeper-server/pom.xml
> > > > > >> -Dtest=org.apache.bookkeeper.client.TestSpeculativeRead
> > > > > >>
> > > > > >> [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time
> > > elapsed:
> > > > > >> 73.673 s <<< FAILURE! - in
> > > > > >> org.apache.bookkeeper.client.TestSpeculativeRead
> > > > > >> [ERROR]
> > > > > >>
> > > > > >>
> > > > >
> > >
> >
> org.apache.bookkeeper.client.TestSpeculativeRead.testSequenceReadLocalEnsemble
> > > > > >>  Time elapsed: 6.341 s  <<< ERROR!

Re: [VOTE] Release 4.15.3, release candidate #1

[Andrey Yegorov]

+1 (binding)

On Thu, Nov 10, 2022 at 8:06 PM Yong Zhang 
wrote:

> Hi Andrey,
>
> I run the test multiple times, I found that depends on how the DNS
> resolve the address.
> When the DNS resolves to 127.0.0.1/localhost, the test will be failed.
> But if it resolves another INet interface, like lo0, the hostname will be
> 192.168.50.240(in my computer) and the test passed.
>
> Thanks,
> Yong
>
> On Fri, 11 Nov 2022 at 09:46, Yong Zhang 
> wrote:
>
> > I downloaded the src package and run the tests successfully.
> > And build from branch-4.15 passed as well. But failed when
> > I run it with IDE.
> >
> > It looks like sometimes it uses the loopback address and we
> > are not allowed that.
> > But it shouldn't be a problem for this release.
> >
> > Thanks,
> > Yong
> >
> > On Fri, 11 Nov 2022 at 09:01, Andrey Yegorov <
> andrey.yego...@datastax.com>
> > wrote:
> >
> >> I tested the following:
> >>
> >> * bookkeeper-all
> >>   ** unpacks
> >>   ** shell works, standalone starts, localbookie starts
> >> * src
> >>   ** unpacks, builds
> >>   ** tests - there are some know flakes
> >> and org.apache.bookkeeper.client.TestSpeculativeRead fails for me
> >>
> >> I don't recall any issues with TestSpeculativeRead in the previous
> >> releases; does it repro for you (I built from unpacked sources for the
> >> RC)?
> >>
> >> $ mvn test -f bookkeeper-server/pom.xml
> >> -Dtest=org.apache.bookkeeper.client.TestSpeculativeRead
> >>
> >> [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> >> 73.673 s <<< FAILURE! - in
> >> org.apache.bookkeeper.client.TestSpeculativeRead
> >> [ERROR]
> >>
> >>
> org.apache.bookkeeper.client.TestSpeculativeRead.testSequenceReadLocalEnsemble
> >>  Time elapsed: 6.341 s  <<< ERROR!
> >> java.lang.RuntimeException:
> >> java.net.UnknownHostException: Trying to listen on loopback address,
> >> 127.0.0.1:3181 but this is forbidden by default (see
> >> ServerConfiguration#getAllowLoopback()).
> >> If this happen, you can consider specifying the network interface to
> >> listen
> >> on (e.g. listeningInterface=eth0) or specifying the advertised address
> >> (e.g. advertisedAddress=172.x.y.z)
> >> at
> >>
> >>
> org.apache.bookkeeper.bookie.LocalBookieEnsemblePlacementPolicy.initialize(LocalBookieEnsemblePlacementPolicy.java:68)
> >> at
> >>
> >>
> org.apache.bookkeeper.client.BookKeeper.initializeEnsemblePlacementPolicy(BookKeeper.java:583)
> >> at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:507)
> >> at
> >>
> >>
> org.apache.bookkeeper.client.BookKeeperTestClient.(BookKeeperTestClient.java:51)
> >> at
> >>
> >>
> org.apache.bookkeeper.client.TestSpeculativeRead.testSequenceReadLocalEnsemble(TestSpeculativeRead.java:395)
> >> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> Method)
> >> at
> >>
> >>
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> at
> >>
> >>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >> at
> >>
> >>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> >> at
> >>
> >>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> >> at
> >>
> >>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> >> at
> >>
> >>
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> >> at
> >>
> >>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> >> at
> >>
> >>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> >> at
> >>
> >>
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> >> at
> >>
> >>
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> >> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >> at java.base/j

Re: [VOTE] Release 4.15.3, release candidate #1

[Andrey Yegorov]

e70a1d2f80f92d369a59c866e1976ff41ec435f53R4
> <https://github.com/apache/bookkeeper/pull/3602/files>
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.3-rc1/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1074
> [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.3-rc1
> [5] 50f6820f31528fbd08aeecae89e6106ff67e7c8f
>


-- 
Andrey Yegorov

Re: [VOTE] BP-57 Add a public API for creating embeddable servers

[Andrey Yegorov]

+1 (binding)

On Tue, Oct 11, 2022 at 3:31 PM Matteo Merli  wrote:

> +1 (binding)
>
> Very good addition!
>
>
> --
> Matteo Merli
> 
>
> On Tue, Oct 11, 2022 at 4:38 AM Enrico Olivelli 
> wrote:
> >
> > Up
> >
> > We need 1 more binding vote from a PMC member
> >
> > Enrico
> >
> > Il Mar 27 Set 2022, 11:36 Nicolò Boschi  ha
> scritto:
> >
> > > +1 (non binding)
> > > Nicolò Boschi
> > >
> > >
> > > Il giorno mar 27 set 2022 alle ore 11:25 Enrico Olivelli <
> > > eolive...@gmail.com> ha scritto:
> > >
> > > > I think that we need some more binding VOTEs please
> > > >
> > > > Enrico
> > > >
> > > > Il giorno dom 25 set 2022 alle ore 16:15 ZhangJian He
> > > >  ha scritto:
> > > > >
> > > > > +1 , non binding
> > > > >
> > > > > Thanks
> > > > > ZhangJian He
> > > > >
> > > > > On Sun, 25 Sept 2022 at 21:56, Lan Liang 
> > > > wrote:
> > > > >
> > > > > > +1 , non binding.
> > > > > >
> > > > > > This is a useful feature!
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best Regards,
> > > > > > Lan Liang
> > > > > >  Replied Message 
> > > > > > | From | steven lu |
> > > > > > | Date | 9/25/2022 08:25 |
> > > > > > | To |  |
> > > > > > | Subject | Re: [VOTE] BP-57 Add a public API for creating
> embeddable
> > > > > > servers |
> > > > > > +1 (non binding)
> > > > > > Great work
> > > > > >
> > > > > > Diego Salvi  于2022年9月24日周六 00:00写道：
> > > > > >
> > > > > > Hi Bookkeeper Community,
> > > > > > I would like to start a VOTE on " Add a public API for creating
> > > > embeddable
> > > > > > servers."(BP-57)
> > > > > >
> > > > > > It is a new public API to create bookie from configuration and/or
> > > > custom
> > > > > > implementations without having to deep dive into internal
> structure.
> > > > The
> > > > > > code will be the same used to build standard BK server through
> Main
> > > > class
> > > > > >
> > > > > > Here is the design detail:
> > > > > > https://github.com/apache/bookkeeper/issues/3494
> > > > > >
> > > > > > Diego Salvi
> > > > > >
> > > > > >
> > > >
> > >
>


-- 
Andrey Yegorov

Re: [DISCUSS] BP-51: BookKeeper client memory limits

[Andrey Yegorov]

onally, I think client memory limits should affect the
> bookie
> > > > >> client reading from the inbound connection. Otherwise, the bookie
> > > > >> client could dispatch many read requests and then end up filling
> up
> > > > >> memory when the requests arrive in the client's direct memory.
> When
> > > > >> the bookie is starting to exceed its memory consumption, it'd be
> > > > >> beneficial to stop reading from the connection and to let the TCP
> > > > >> connection propagate back pressure to the server. In this case, we
> > > > >> would need a reactive solution since it is an anti-pattern to
> block on
> > > > >> a netty event loop.
> > > > >>
> > > > >> Thanks,
> > > > >> Michael
> > > > >>
> > > > >>> On Wed, Sep 28, 2022 at 4:36 AM Enrico Olivelli <
> eolive...@gmail.com
> > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>> Yong,
> > > > >>>
> > > > >>> Il giorno mer 28 set 2022 alle ore 10:23 Yong Zhang
> > > > >>>  ha scritto:
> > > > >>>>
> > > > >>>> We have improved the memory issue with backpressure with PR
> > > > >>>> https://github.com/apache/bookkeeper/pull/3324
> > > > >>>>
> > > > >>>> The backpressure way can prevent there have too many Add
> requests
> > > > >>>> pending to the client and waiting for the response. It makes
> the add
> > > > >>>> requests
> > > > >>>> fail fast, so if the channel is not writable, it will fail the
> > > request
> > > > >> as
> > > > >>>> soon as
> > > > >>>> possible and then release the memory.
> > > > >>>>
> > > > >>>> But that depends on the time. If your throughput is very
> smooth, and
> > > > >> you
> > > > >>>> have enough memory for the bookie client. With backpressure, it
> > > would
> > > > >> work
> > > > >>>> fine.
> > > > >>>> If you have a huge adds to the bookie in a very short time, it
> > > > >> acquires a
> > > > >>>> lot of
> > > > >>>> memory, then the bookie crashed with OOM.
> > > > >>>> So we still need this proposal.
> > > > >>>>
> > > > >>>> I will continue to work on this. If there haven't objected, I
> will
> > > > >> start a
> > > > >>>> VOTE later
> > > > >>>
> > > > >>> Thanks
> > > > >>>
> > > > >>> Enrico
> > > > >>>
> > > > >>>> Thanks,
> > > > >>>> Yong
> > > > >>>>
> > > > >>>> On Thu, 21 Apr 2022 at 19:17, r...@apache.org <
> > > ranxiaolong...@gmail.com
> > > > >>>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Hello Yong:
> > > > >>>>>
> > > > >>>>> It seems to be a very useful feature. In the production
> > > environment,
> > > > >> you
> > > > >>>>> can often see similar phenomena happening.
> > > > >>>>>
> > > > >>>>> +1 (non-binding)
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>> Thanks
> > > > >>>>> Xiaolong Ran
> > > > >>>>>
> > > > >>>>> Yong Zhang  于2022年4月21日周四 18:29写道：
> > > > >>>>>
> > > > >>>>>> Hi all,
> > > > >>>>>>
> > > > >>>>>> The BP-51 BookKeeper client memory limits is ready for review.
> > > > >>>>>> The proposal is here:
> > > > >> https://github.com/apache/bookkeeper/issues/3231
> > > > >>>>>> And the PR is here:
> > > https://github.com/apache/bookkeeper/pull/3139
> > > > >>>>>>
> > > > >>>>>> Please help to review this proposal.
> > > > >>>>>>
> > > > >>>>>> Thanks!
> > > > >>>>>> Yong
> > > > >>>>>>
> > > > >>>>>
> > > > >>
> > >
>


-- 
Andrey Yegorov

Re: [DISCUSS] End-Of-Life for some release lines

[Andrey Yegorov]

I agree.

Upgrades from 4.11+ up to 4.14 are straightforward (e.g. DataStax's fork of
Pulsar 2.7 uses BK 4.14).
BK 4.15 has breaking API changes so we'll need to keep 4.14 alive for a
while.

On Mon, Sep 26, 2022 at 11:43 PM Enrico Olivelli 
wrote:

> Hello,
> I propose to declare end-of-life on all the release lines prior to 4.14.
>
> We wanted to follow a time-based release plan, but actually we are not
> following it.
>
> This is the doc on our website
> https://bookkeeper.apache.org/community/releases
>
> If nobody objects, I will start a separate thread with a VOTE for this
> resolution.
>
> End-of-life generally means "no more releases" so:
> - no more bug fixes
> - no more security fixes
> - no patches accepted to be cherry-picked to those branches
>
> Please chime in with your thoughts
>
> Enrico
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.15.2, release candidate #0

[Andrey Yegorov]

+1 (binding)

- Built from sources (mac, JDK 11), ran tests (some known flakes like
org.apache.bookkeeper.test.BookieZKExpireTest, otherwise ok).
- tested localbookie
- tested bookkeeper standalone
- ran dlog local

Thank you!

On Fri, Sep 23, 2022 at 12:32 AM Yong Zhang 
wrote:

> Hi everyone,
> Please review and vote on the release candidate #0 for the version 4.15.2,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.15.2-rc0" [4] with git sha [5]
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1]
>
> https://github.com/apache/bookkeeper/pull/3500/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53R4
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.2-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1073
> [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.2-rc0
> [5] 3c9e4f66ab77a408fe208a3d757929527ff7bc0f
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.15.1, release candidate #3

[Andrey Yegorov]

+1 (binding)
Smoke test with localbookie passed.
Build from packaged sources succeeds, some test failed but they look like
known flakes on CI

I left a couple of comments on the release notes, the rest looks good.
Thank you!


On Mon, Aug 29, 2022 at 7:33 AM Enrico Olivelli  wrote:

> +1 (binding)
> All tests passed
> Smoke tests with localbookie passed
> GPG signatures are good
>
> thank you !
>
> Enrico
>
> Il giorno sab 27 ago 2022 alle ore 07:31 steven lu
>  ha scritto:
> >
> > Great work!
> > I’m verifying this
> >
> > Yong Zhang  于2022年8月24日周三 10:41写道：
> >
> > > Hi everyone,
> > > Please review and vote on the release candidate #1 for the version
> 4.15.1,
> > > as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * Release notes [1]
> > > * The official Apache source and binary distributions to be deployed to
> > > dist.apache.org [2]
> > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > * Source code tag "v4.15.1-rc3" [4] with git sha [5]
> > >
> > > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > >
> > > Please download these packages and review this release candidate:
> > >
> > > - Review release notes
> > > - Download the source package (verify shasum, and asc) and follow the
> > > instructions to build and run the bookkeeper service.
> > > - Download the binary package (verify shasum, and asc) and follow the
> > > instructions to run the bookkeeper service.
> > > - Review maven repo, release tag, licenses, and any other things you
> think
> > > it is important to a release.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Release Manager
> > >
> > > [1]
> > >
> > >
> https://github.com/apache/bookkeeper/pull/3440/files#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53
> > > [2]
> > >
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.1-rc3/
> > > [3]
> > >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1072
> > > [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.1-rc3
> > > [5] 1f930273aa329c9b04a19f27a11417624bc02954
> > >
>


-- 
Andrey Yegorov

Re: [DISCUSS] Bookkeeper 4.16.0 release

[Andrey Yegorov]

+1 for the 4.16 release.
Hang, did you just volunteer to do the release? :)

We need to do a pass on open PRs to review/merge what we can, there are 100
open PRs right now.
I'd love to see at least these in the release:
https://github.com/apache/bookkeeper/pull/3361
https://github.com/apache/bookkeeper/pull/3383
https://github.com/apache/bookkeeper/pull/3384
https://github.com/apache/bookkeeper/pull/3390
https://github.com/apache/bookkeeper/pull/3325
https://github.com/apache/bookkeeper/pull/3329
https://github.com/apache/bookkeeper/pull/3330

As for the minor releases, 4.15.1 is ok.
I think we also need 4.14.6; 4.15 was a major change and requires a bit of
work to upgrade to (https://github.com/apache/pulsar/pull/15142), e.g.
older Pulsar versions (2.10 and below) could benefit from 4.14 release.

Btw, why https://bookkeeper.apache.org/docs/4.14.5/overview/ says "Apache
BookKeeper 4.14.5, which is no longer actively maintained"?
I thought 4.14 was still maintained.

On Fri, Jul 15, 2022 at 12:05 AM Enrico Olivelli 
wrote:

> Hang,
>
> Il giorno ven 15 lug 2022 alle ore 08:25 ZhangJian He
>  ha scritto:
> >
> > Hi, Hang Chen. I hope to get this change into version 4.16.0.
> > https://github.com/apache/bookkeeper/pull/3406
> >
> > Thanks
> > ZhangJian He
> >
> > Hang Chen  于2022年7月15日周五 11:32写道：
> >
> > > Hi Community,
> > >Bookkeeper 4.15.0 has been released for 2 months, and we have
> > > introduced Direct IO support[1] for ledger and many important bug
> > > fixes, such as fixing memory leaks [2][3], applying back pressure for
> > > v2 protocol[4], and so on.
>
> for bug fixes we should cut a 4.15.1 release as well
>
> I am +1 to cut a release for 4.16.0
>
> I would also add this patch (that only needs a second reviewer)
> https://github.com/apache/bookkeeper/pull/3390
>
> Enrico
>
>
> I would like to start to discuss releasing
> > > 4.16.0.
> > >
> > >Do you have any concerns? Do you have any other issue/ticket to add
> > > for 4.16.0?
> > >
> > > [1]
> > >
> https://github.com/apache/bookkeeper/pulls?q=is%3Apr+bp-47+is%3Aclosed+author%3Ahangc0276
> > > [2] https://github.com/apache/bookkeeper/pull/3321
> > > [3] https://github.com/apache/bookkeeper/pull/3323
> > > [4] https://github.com/apache/bookkeeper/pull/3324
> > >
>


-- 
Andrey Yegorov

Re: [RESULT] [VOTE] Release 4.15.0, release candidate #2

[Andrey Yegorov]

The release is completed.

Announcement:
https://lists.apache.org/thread/1w4f7q4hlfr347tc3b9j7gx67bs4xjj0

On Mon, May 9, 2022 at 10:37 AM Andrey Yegorov 
wrote:

>
> I'm happy to announce that we have unanimously approved this release.
>
> There are 4 approving votes, 4 of which are binding:
> * Matteo Merli (binding)
> * Enrico Olivelli (binding)
> * Henry Saputra (binding)
> * Andrey Yegorov (binding)
>
> There are no disapproving votes.
>
> Thanks everyone!
>
> --
> Andrey Yegorov
>


-- 
Andrey Yegorov

[ANNOUNCE] Apache BookKeeper 4.15.0 released

[Andrey Yegorov]

The Apache BookKeeper team is proud to announce Apache BookKeeper version
4.15.0.

Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage
service optimized for
real-time workloads. It has been used for a fundamental service to build
reliable services.
It is also the log segment store for Apache DistributedLog and the message
store for Apache Pulsar.

This is the 31st release of the Apache BookKeeper.

Release 4.15 includes many upgrades to third party libraries marked with
CVEs, adds more configuration options, extends REST API, adds an option to
run without journal, improves memory utilization and stability, and more!

For BookKeeper release details and downloads, visit:

https://bookkeeper.apache.org/releases

BookKeeper 4.15.0 Release Notes are at:

https://bookkeeper.apache.org/release-notes#4150

We would like to thank the contributors that made the release possible.

Regards,
The BookKeeper Team

[RESULT] [VOTE] Release 4.15.0, release candidate #2

[Andrey Yegorov]

I'm happy to announce that we have unanimously approved this release.

There are 4 approving votes, 4 of which are binding:
* Matteo Merli (binding)
* Enrico Olivelli (binding)
* Henry Saputra (binding)
* Andrey Yegorov (binding)

There are no disapproving votes.

Thanks everyone!

-- 
Andrey Yegorov

Re: [VOTE] Release 4.15.0, release candidate #2

[Andrey Yegorov]

Hi everyone,

https://github.com/apache/bookkeeper/issues/3258 is closed as a test
environment issue so I believe we don't have -1 from Raul anymore.

We have 4 binding +1s and I'll close the vote and proceed with the release.

On Fri, May 6, 2022 at 10:34 AM Henry Saputra 
wrote:

> +1 (binding)
>
> Source artifact compiled and pass tests
> Checksum files look good
> Signature file looks good
> Source artifacts does not contain executable
> LICENSE file exists
>
> Thanks,
>
> Henry
>
>
> On Fri, Apr 29, 2022 at 2:09 PM Andrey Yegorov <
> andrey.yego...@datastax.com>
> wrote:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version
> 4.15.0,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "release-4.15.0-rc2" [4] with git
> > sha f771b7664f7f83ae4ae3e8eaa516c879c4bc7897
> >
> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review maven repo, release tag, licenses, and any other things you
> think
> > it is important to a release.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1]
> >
> >
> https://github.com/apache/bookkeeper/pull/3155/files#diff-eff52edf7022fd953f60c68061de09ec5d148eb37893de6af85a7964a2e57855
> >
> > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.0-rc2/
> >
> > [3]
> >
> >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1066/
> > [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.0-rc2/
> >
> >
> > --
> > Andrey Yegorov
> >
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.15.0, release candidate #2

[Andrey Yegorov]

+1 (binding)

I built Pulsar with BK 4.15 RC,
CI ran the tests with RC1 (RC 2 in progress but it is not functionally
different): https://github.com/apache/pulsar/pull/15142

Performance regression is not confirmed:
https://github.com/apache/bookkeeper/issues/3258#issuecomment-1118033982 in
other environments.
I agree with Enrico we can do .1 release if there is a regression that
shows up in specific cases.


On Thu, May 5, 2022 at 8:04 AM Enrico Olivelli  wrote:

> +1 (binding)
>  verified checksums, signatures, RAT
> run all tests on Ubuntu JDK11
>
> Regarding the performance issues reported by Raul while it looks scary
> I don't find any performance regressions on the other projects that I
> follow other than Pravega (Pulsar, HerdDB...), so I am not sure we
> should hold the release.
>
> if we find the problem I pretty sure it won't be something about a API
> change or anything we can't fix in a 4.15.1 release
>
>
> Enrico
>
> Il giorno mer 4 mag 2022 alle ore 20:27 Gracia, Raul
>  ha scritto:
> >
> > Hi all,
> >
> > -1 (not binding)
> >
> > Reason: significant performance regression found in Bookkeeper 4.15, see
> https://github.com/apache/bookkeeper/issues/3258
> > Also helping to find out where the regression could be.
> >
> > Thanks,
> > Raúl.
> >
> >
> > Internal Use - Confidential
> >
> > -Original Message-
> > From: Matteo Merli 
> > Sent: miércoles, 4 de mayo de 2022 18:58
> > To: Bookkeeper-Dev
> > Subject: Re: [VOTE] Release 4.15.0, release candidate #2
> >
> >
> > [EXTERNAL EMAIL]
> >
> > +1
> >
> > Verified:
> >  * Checksums
> >  * Building
> >  * Starting standalone bookie
> >
> > Thanks,
> > Matteo
> >
> >
> > --
> > Matteo Merli
> > 
> >
> > On Wed, May 4, 2022 at 8:53 AM Andrey Yegorov <
> andrey.yego...@datastax.com> wrote:
> > >
> > > Hi everyone,
> > >
> > > The vote has been out for 5 days - any feedback?
> > >
> > > On Fri, Apr 29, 2022 at 2:08 PM Andrey Yegorov
> > > 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #1 for the version
> > > > 4.15.0, as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific
> > > > comments)
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * Release notes [1]
> > > > * The official Apache source and binary distributions to be deployed
> > > > to dist.apache.org [2]
> > > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > > * Source code tag "release-4.15.0-rc2" [4] with git sha
> > > > f771b7664f7f83ae4ae3e8eaa516c879c4bc7897
> > > >
> > > > BookKeeper's KEYS file contains PGP keys we used to sign this
> release:
> > > > https://urldefense.com/v3/__https://dist.apache.org/repos/dist/relea
> > > > se/bookkeeper/KEYS__;!!LpKI!kuT1AoZ1L3lHLUu2Ui1-pfpJG3EYMaFXsAW3MjSO
> > > > HuEm0lPsWdzZOR9n6S1dx3z59pZOujAVMx7AqIn3ZK5PmQ$
> > > > [dist[.]apache[.]org]
> > > >
> > > > Please download these packages and review this release candidate:
> > > >
> > > > - Review release notes
> > > > - Download the source package (verify shasum, and asc) and follow
> > > > the instructions to build and run the bookkeeper service.
> > > > - Download the binary package (verify shasum, and asc) and follow
> > > > the instructions to run the bookkeeper service.
> > > > - Review maven repo, release tag, licenses, and any other things you
> > > > think it is important to a release.
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> > > > majority approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Release Manager
> > > >
> > > > [1]
> > > > https://urldefense.com/v3/__https://github.com/apache/bookkeeper/pul
> > > > l/3155/files*diff-eff52edf7022fd953f60c68061de09ec5d148eb37893de6af8
> > > > 5a7964a2e57855__;Iw!!LpKI!kuT1AoZ1L3lHLUu2Ui1-pfpJG3EYMaFXsAW3MjSOHu
> > > > Em0lPsWdzZOR9n6S1dx3z59pZOujAVMx7AqImAYILkbQ$ [github[.]com]
> > > >
> > > > [2]
> > > > https://urldefense.com/v3/_

Re: [VOTE] Release 4.15.0, release candidate #2

[Andrey Yegorov]

Hi everyone,

The vote has been out for 5 days - any feedback?

On Fri, Apr 29, 2022 at 2:08 PM Andrey Yegorov 
wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 4.15.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "release-4.15.0-rc2" [4] with git
> sha f771b7664f7f83ae4ae3e8eaa516c879c4bc7897
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1]
> https://github.com/apache/bookkeeper/pull/3155/files#diff-eff52edf7022fd953f60c68061de09ec5d148eb37893de6af85a7964a2e57855
>
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.0-rc2/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1066/
> [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.0-rc2/
>
>
> --
> Andrey Yegorov
>


-- 
Andrey Yegorov

Re: [ANNOUNCE] New committer ZhangJian He (shoothzj)

[Andrey Yegorov]

Congrats!

On Wed, May 4, 2022 at 5:36 AM Enrico Olivelli  wrote:

> The Project Management Committee (PMC) for Apache BookKeeper
> has invited  ZhangJian He to become a committer and we are pleased
> to announce that he has accepted.
>
> Being a committer enables easier contribution to the
> project since there is no need to go via the patch
> submission process. This should enable better productivity.
> A PMC member helps manage and guide the direction of the project.
>
> In order to  know more about how the ASF works you can check this link:
> https://www.apache.org/foundation/how-it-works.html
>
>
> Congratulations ZhangJian !
> Enrico
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.14.5, release candidate #0

[Andrey Yegorov]

+1 (binding)

- built from sources on Mac, JDK11
- ran bookkeeper localbookie and standalone, from server and all binary
packages
- built and tested with Pulsar https://github.com/apache/pulsar/pull/15396
(some tests are still running at the moment, ignore unrelated OWASP, so far
everything looks good)

On Wed, Apr 27, 2022 at 3:17 AM Enrico Olivelli  wrote:

> +1 (binding)
>
> - run unit tests on Ubuntu, JDK8
> - run smoke tests on the binaries compiled from the source code
> - run all tests of HerdDB (https://github.com/diennea/herddb/pull/786)
> - verified RAT, checksums and signatures
>
> Thank you Nicolò for driving the release !
>
>
> Enrico
>
> Il giorno gio 21 apr 2022 alle ore 03:39 Yong Zhang
>  ha scritto:
> >
> > +1
> >
> > - verified the binary package and source package signature and shasum
> > - run localbookie service and do simpletest using binary package
> > - build code from source package
> >
> > On Wed, 13 Apr 2022 at 17:28, Nicolò Boschi 
> wrote:
> >
> > > Hi everyone,
> > > Please review and vote on the release candidate #0 for the version
> 4.14.5,
> > > as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * Release notes [1]
> > > * The official Apache source and binary distributions to be deployed to
> > > dist.apache.org [2]
> > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > * Source code tag "release-4.14.5" [4] with git sha
> > > c88608f9e9d9fb3952be5219b91bb484361efe28
> > >
> > > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > >
> > > Please download these packages and review this release candidate:
> > >
> > > - Review release notes
> > > - Download the source package (verify shasum, and asc) and follow the
> > > instructions to build and run the bookkeeper service.
> > > - Download the binary package (verify shasum, and asc) and follow the
> > > instructions to run the bookkeeper service.
> > > - Review maven repo, release tag, licenses, and any other things you
> think
> > > it is important to a release.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Release Manager
> > >
> > > [1]
> > >
> > >
> https://github.com/apache/bookkeeper/pull/3199/files/8691a431ae66a94009eb8607a7918c339d6d7ff9#diff-d361215d5d5abaa96ac3362e70a1d2f80f92d369a59c866e1976ff41ec435f53
> > > [2]
> > >
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.5-rc0/
> > > [3]
> > >
> > >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1064/
> > > [4] https://github.com/apache/bookkeeper/releases/tag/v4.14.5-rc0
> > >
> > >
> > > Nicolò Boschi
> > >
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.15.0, release candidate #1

[Andrey Yegorov]

I started vote for RC2

On Thu, Apr 28, 2022 at 12:55 PM Andrey Yegorov 
wrote:

> Hi everyone,
>
> Fix for the failed tests:
>
> https://github.com/apache/bookkeeper/pull/3251
>
> This is a test issue (not the scripts/code issue); tests didn't not run on
> CI so were skipped in respective changes.
>
> Do we have to build another RC for this?
>
>
> On Wed, Apr 27, 2022 at 9:45 AM Enrico Olivelli 
> wrote:
>
>> Il Mer 27 Apr 2022, 18:39 Andrey Yegorov  ha
>> scritto:
>>
>> > Hi,
>> >
>> > I can repro this on branch-4.15 (branch-4.14 is ok).
>> >
>> > The problem is these tests don't run on "mvn test" and I suspect they
>> don't
>> > run on CI too.
>> >
>> > To repro locally:
>> > cd tests/scripts
>> >
>> > mvn clean install -DskipTests
>> > mvn test
>> >
>> > -> success
>> >
>> >  mvn clean install
>> >
>> > -> failed tests
>> >
>>
>> Exactly the same failure
>>
>>
>> I think we have to fix this.
>>
>>
>>
>> -1
>>
>> Enrico
>>
>>
>>
>>
>> >
>> >
>> #--
>> > # System data
>> > #
>> > # test run info
>> > shells="/bin/sh /bin/bash"
>> > tests="bk_test_bin_common.sh"
>> >
>> > # system info
>> > $ date
>> > Wed Apr 27 09:35:25 PDT 2022
>> > $ uname -mprsv
>> > Darwin 20.6.0 Darwin Kernel Version 20.6.0: Wed Nov 10 22:23:07 PST
>> 2021;
>> > root:xnu-7195.141.14~1/RELEASE_X86_64 x86_64 i386
>> >
>> >
>> >
>> #--
>> > # Running the test suite with /bin/sh
>> > #
>> > shell name: sh
>> > shell version: 3.2.57(1)-release
>> >
>> > --- Executing the 'bin_common' test suite ---
>> > testDefaultVariables
>> > ASSERT:NETTY_RECYCLER_MAXCAPACITY is not set correctly expected:<1000>
>> but
>> > was:<>
>> > ASSERT:NETTY_RECYCLER_LINKCAPACITY is not set correctly expected:<1024>
>> but
>> > was:<>
>> > testFindModuleJarAt
>> > testFindModuleJar
>> > testLoadEnvfiles
>> > testSetModuleClasspath
>> > testBuildBookieJVMOpts
>> > testBuildCLIJVMOpts
>> > testBuildNettyOpts
>> > testBuildBookieOpts
>> > testBuildLoggingOpts
>> > ASSERT:Logging OPTS is not set correctly
>> > expected:<-Dlog4j.configurationFile=test.conf
>> > -Dbookkeeper.log.root.level=
>> > -Dbookkeeper.log.root.appender=test_log_dir
>> > -Dbookkeeper.log.dir=test_log_dir
>>  -Dbookkeeper.log.file=test_log_file>
>> > but was:<-Dlog4j.configurationFile=test.conf
>> > -Dbookkeeper.log.root.level=INFO -Dbookkeeper.log.root.appender=TEST
>> >   -Dbookkeeper.log.dir=test_log_dir
>> >  -Dbookkeeper.log.file=test_log_file>
>> > testBuildCLILoggingOpts
>> > ASSERT:Logging OPTS is not set correctly
>> > expected:<-Dlog4j.configurationFile=test.conf
>> > -Dbookkeeper.cli.root.logger=INFO,TEST
>> > -Dbookkeeper.cli.log.dir=test_log_dir
>> > -Dbookkeeper.cli.log.file=test_log_file> but
>> > was:<-Dlog4j.configurationFile=test.conf
>> > -Dbookkeeper.cli.log.root.level=test_log_dir
>> > -Dbookkeeper.cli.log.root.appender=test_log_file
>> > -Dbookkeeper.cli.log.dir=INFO,TEST -Dbookkeeper.cli.log.file=>
>> >
>> > Ran 11 tests.
>> >
>> > FAILED (failures=4)
>> >
>> >
>> >
>> #--
>> > # Running the test suite with /bin/bash
>> > #
>> > shell name: bash
>> > shell version: 3.2.57(1)-release
>> >
>> > --- Executing the 'bin_common' test suite ---
>> > testDefaultVariables
>> > ASSERT:NETTY_RECYCLER_MAXCAPACITY is not set correctly expected:<1000>
>> but
>> > was:<>
>> > ASSERT:NETTY_RECYCLER_LINKCAPACITY is not set correctly expected:<1024>
>> but
>> > was:<>
>> > testFindModuleJarAt
>> > testFindModuleJar
>> > testLoadEnvfiles
>> > testSetModuleClasspath
>> > testBuildBookieJVMOpts
>> > testBuildCLIJVMOpts
>> > tes

[VOTE] Release 4.15.0, release candidate #2

[Andrey Yegorov]

Hi everyone,

Please review and vote on the release candidate #1 for the version 4.15.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* Release notes [1]
* The official Apache source and binary distributions to be deployed to
dist.apache.org [2]
* All artifacts to be deployed to the Maven Central Repository [3]
* Source code tag "release-4.15.0-rc2" [4] with git
sha f771b7664f7f83ae4ae3e8eaa516c879c4bc7897

BookKeeper's KEYS file contains PGP keys we used to sign this release:
https://dist.apache.org/repos/dist/release/bookkeeper/KEYS

Please download these packages and review this release candidate:

- Review release notes
- Download the source package (verify shasum, and asc) and follow the
instructions to build and run the bookkeeper service.
- Download the binary package (verify shasum, and asc) and follow the
instructions to run the bookkeeper service.
- Review maven repo, release tag, licenses, and any other things you think
it is important to a release.

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1]
https://github.com/apache/bookkeeper/pull/3155/files#diff-eff52edf7022fd953f60c68061de09ec5d148eb37893de6af85a7964a2e57855

[2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.0-rc2/

[3]
https://repository.apache.org/content/repositories/orgapachebookkeeper-1066/
[4] https://github.com/apache/bookkeeper/releases/tag/v4.15.0-rc2/


-- 
Andrey Yegorov

Re: [discuss] Buildsystems, keep gradle or maven in future

[Andrey Yegorov]

+1 from me.
Gradle might have its benefits but not in its current state and I don't
know it well enough to commit to fixing it.

IIRC Lari and Matteo were proponents of gradle during one of the recent
Pulsar community meetings.
Search for "gradle" in Meeting Notes
<https://www.google.com/url?q=https://docs.google.com/document/d/19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE/edit&sa=D&source=calendar&ust=1651599874669558&usg=AOvVaw1tTF4s87Vbfo1ETHmJXjmw>
.

I think we should give them an opportunity to talk on this issue before
moving forward with removal.


On Thu, Apr 28, 2022 at 6:15 AM Nicolò Boschi  wrote:

> +1 for removing Gradle
>
> We still have not figured out how to replace the release process using
> Gradle and we have other issues and no one actually volunteered to fix
> them.
>
> Nicolò Boschi
>
>
> Il giorno gio 28 apr 2022 alle ore 08:31 Enrico Olivelli <
> eolive...@gmail.com> ha scritto:
>
> > Il giorno gio 28 apr 2022 alle ore 04:26 ZhangJian He
> >  ha scritto:
> > >
> > > Hello, everyone. After talking with enrico and nicolo. I want to start
> a
> > > discussion about building systems.
> > > We are now maintaining two build systems, I don't think that we want to
> > > maintain both `maven` and `gradle`.
> > >
> > > I think that we need to decide to keep maven or gradle in the future.
> > >
> > > Personally, I tend to use maven, the main reason is that we don't have
> > > active gradle experts now and gradle still has problems to solve.
> >
> > +1
> > I think we can keep Maven and remove Gradle.
> > I see no strong reason to keep Gradle,
> > it looks like the initial energy put into the migration faded away and
> > we have no one who has time to work on it
> >
> > Enrico
> >
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.15.0, release candidate #1

[Andrey Yegorov]

Hi everyone,

Fix for the failed tests:

https://github.com/apache/bookkeeper/pull/3251

This is a test issue (not the scripts/code issue); tests didn't not run on
CI so were skipped in respective changes.

Do we have to build another RC for this?


On Wed, Apr 27, 2022 at 9:45 AM Enrico Olivelli  wrote:

> Il Mer 27 Apr 2022, 18:39 Andrey Yegorov  ha
> scritto:
>
> > Hi,
> >
> > I can repro this on branch-4.15 (branch-4.14 is ok).
> >
> > The problem is these tests don't run on "mvn test" and I suspect they
> don't
> > run on CI too.
> >
> > To repro locally:
> > cd tests/scripts
> >
> > mvn clean install -DskipTests
> > mvn test
> >
> > -> success
> >
> >  mvn clean install
> >
> > -> failed tests
> >
>
> Exactly the same failure
>
>
> I think we have to fix this.
>
>
>
> -1
>
> Enrico
>
>
>
>
> >
> >
> #--
> > # System data
> > #
> > # test run info
> > shells="/bin/sh /bin/bash"
> > tests="bk_test_bin_common.sh"
> >
> > # system info
> > $ date
> > Wed Apr 27 09:35:25 PDT 2022
> > $ uname -mprsv
> > Darwin 20.6.0 Darwin Kernel Version 20.6.0: Wed Nov 10 22:23:07 PST 2021;
> > root:xnu-7195.141.14~1/RELEASE_X86_64 x86_64 i386
> >
> >
> >
> #--
> > # Running the test suite with /bin/sh
> > #
> > shell name: sh
> > shell version: 3.2.57(1)-release
> >
> > --- Executing the 'bin_common' test suite ---
> > testDefaultVariables
> > ASSERT:NETTY_RECYCLER_MAXCAPACITY is not set correctly expected:<1000>
> but
> > was:<>
> > ASSERT:NETTY_RECYCLER_LINKCAPACITY is not set correctly expected:<1024>
> but
> > was:<>
> > testFindModuleJarAt
> > testFindModuleJar
> > testLoadEnvfiles
> > testSetModuleClasspath
> > testBuildBookieJVMOpts
> > testBuildCLIJVMOpts
> > testBuildNettyOpts
> > testBuildBookieOpts
> > testBuildLoggingOpts
> > ASSERT:Logging OPTS is not set correctly
> > expected:<-Dlog4j.configurationFile=test.conf
> > -Dbookkeeper.log.root.level=
> > -Dbookkeeper.log.root.appender=test_log_dir
> > -Dbookkeeper.log.dir=test_log_dir
>  -Dbookkeeper.log.file=test_log_file>
> > but was:<-Dlog4j.configurationFile=test.conf
> > -Dbookkeeper.log.root.level=INFO -Dbookkeeper.log.root.appender=TEST
> >   -Dbookkeeper.log.dir=test_log_dir
> >  -Dbookkeeper.log.file=test_log_file>
> > testBuildCLILoggingOpts
> > ASSERT:Logging OPTS is not set correctly
> > expected:<-Dlog4j.configurationFile=test.conf
> > -Dbookkeeper.cli.root.logger=INFO,TEST
> > -Dbookkeeper.cli.log.dir=test_log_dir
> > -Dbookkeeper.cli.log.file=test_log_file> but
> > was:<-Dlog4j.configurationFile=test.conf
> > -Dbookkeeper.cli.log.root.level=test_log_dir
> > -Dbookkeeper.cli.log.root.appender=test_log_file
> > -Dbookkeeper.cli.log.dir=INFO,TEST -Dbookkeeper.cli.log.file=>
> >
> > Ran 11 tests.
> >
> > FAILED (failures=4)
> >
> >
> >
> #--
> > # Running the test suite with /bin/bash
> > #
> > shell name: bash
> > shell version: 3.2.57(1)-release
> >
> > --- Executing the 'bin_common' test suite ---
> > testDefaultVariables
> > ASSERT:NETTY_RECYCLER_MAXCAPACITY is not set correctly expected:<1000>
> but
> > was:<>
> > ASSERT:NETTY_RECYCLER_LINKCAPACITY is not set correctly expected:<1024>
> but
> > was:<>
> > testFindModuleJarAt
> > testFindModuleJar
> > testLoadEnvfiles
> > testSetModuleClasspath
> > testBuildBookieJVMOpts
> > testBuildCLIJVMOpts
> > testBuildNettyOpts
> > testBuildBookieOpts
> > testBuildLoggingOpts
> > ASSERT:Logging OPTS is not set correctly
> > expected:<-Dlog4j.configurationFile=test.conf
> > -Dbookkeeper.log.root.level=
> > -Dbookkeeper.log.root.appender=test_log_dir
> > -Dbookkeeper.log.dir=test_log_dir
>  -Dbookkeeper.log.file=test_log_file>
> > but was:<-Dlog4j.configurationFile=test.conf
> > -Dbookkeeper.log.root.level=INFO -Dbookkeeper.log.root.appender=TEST
> >   -Dbookkeeper.log.dir=test_log_dir
> >  -Dbookkeeper.log.file=test_log_file>
> > testBuildCLILoggingOpts
> > ASSERT:Log

Re: [VOTE] Release 4.15.0, release candidate #1

[Andrey Yegorov]

Hi,

I can repro this on branch-4.15 (branch-4.14 is ok).

The problem is these tests don't run on "mvn test" and I suspect they don't
run on CI too.

To repro locally:
cd tests/scripts

mvn clean install -DskipTests
mvn test

-> success

 mvn clean install

-> failed tests

#--
# System data
#
# test run info
shells="/bin/sh /bin/bash"
tests="bk_test_bin_common.sh"

# system info
$ date
Wed Apr 27 09:35:25 PDT 2022
$ uname -mprsv
Darwin 20.6.0 Darwin Kernel Version 20.6.0: Wed Nov 10 22:23:07 PST 2021;
root:xnu-7195.141.14~1/RELEASE_X86_64 x86_64 i386

#--
# Running the test suite with /bin/sh
#
shell name: sh
shell version: 3.2.57(1)-release

--- Executing the 'bin_common' test suite ---
testDefaultVariables
ASSERT:NETTY_RECYCLER_MAXCAPACITY is not set correctly expected:<1000> but
was:<>
ASSERT:NETTY_RECYCLER_LINKCAPACITY is not set correctly expected:<1024> but
was:<>
testFindModuleJarAt
testFindModuleJar
testLoadEnvfiles
testSetModuleClasspath
testBuildBookieJVMOpts
testBuildCLIJVMOpts
testBuildNettyOpts
testBuildBookieOpts
testBuildLoggingOpts
ASSERT:Logging OPTS is not set correctly
expected:<-Dlog4j.configurationFile=test.conf
-Dbookkeeper.log.root.level=
-Dbookkeeper.log.root.appender=test_log_dir
-Dbookkeeper.log.dir=test_log_dir -Dbookkeeper.log.file=test_log_file>
but was:<-Dlog4j.configurationFile=test.conf
-Dbookkeeper.log.root.level=INFO -Dbookkeeper.log.root.appender=TEST
  -Dbookkeeper.log.dir=test_log_dir -Dbookkeeper.log.file=test_log_file>
testBuildCLILoggingOpts
ASSERT:Logging OPTS is not set correctly
expected:<-Dlog4j.configurationFile=test.conf
-Dbookkeeper.cli.root.logger=INFO,TEST
-Dbookkeeper.cli.log.dir=test_log_dir
-Dbookkeeper.cli.log.file=test_log_file> but
was:<-Dlog4j.configurationFile=test.conf
-Dbookkeeper.cli.log.root.level=test_log_dir
-Dbookkeeper.cli.log.root.appender=test_log_file
-Dbookkeeper.cli.log.dir=INFO,TEST -Dbookkeeper.cli.log.file=>

Ran 11 tests.

FAILED (failures=4)

#--
# Running the test suite with /bin/bash
#
shell name: bash
shell version: 3.2.57(1)-release

--- Executing the 'bin_common' test suite ---
testDefaultVariables
ASSERT:NETTY_RECYCLER_MAXCAPACITY is not set correctly expected:<1000> but
was:<>
ASSERT:NETTY_RECYCLER_LINKCAPACITY is not set correctly expected:<1024> but
was:<>
testFindModuleJarAt
testFindModuleJar
testLoadEnvfiles
testSetModuleClasspath
testBuildBookieJVMOpts
testBuildCLIJVMOpts
testBuildNettyOpts
testBuildBookieOpts
testBuildLoggingOpts
ASSERT:Logging OPTS is not set correctly
expected:<-Dlog4j.configurationFile=test.conf
-Dbookkeeper.log.root.level=
-Dbookkeeper.log.root.appender=test_log_dir
-Dbookkeeper.log.dir=test_log_dir -Dbookkeeper.log.file=test_log_file>
but was:<-Dlog4j.configurationFile=test.conf
-Dbookkeeper.log.root.level=INFO -Dbookkeeper.log.root.appender=TEST
  -Dbookkeeper.log.dir=test_log_dir -Dbookkeeper.log.file=test_log_file>
testBuildCLILoggingOpts
ASSERT:Logging OPTS is not set correctly
expected:<-Dlog4j.configurationFile=test.conf
-Dbookkeeper.cli.root.logger=INFO,TEST
-Dbookkeeper.cli.log.dir=test_log_dir
-Dbookkeeper.cli.log.file=test_log_file> but
was:<-Dlog4j.configurationFile=test.conf
-Dbookkeeper.cli.log.root.level=test_log_dir
-Dbookkeeper.cli.log.root.appender=test_log_file
-Dbookkeeper.cli.log.dir=INFO,TEST -Dbookkeeper.cli.log.file=>

Ran 11 tests.

FAILED (failures=4)

On Wed, Apr 27, 2022 at 9:04 AM Enrico Olivelli  wrote:

> Andrey,
> thanks for driving the release.
>
> I see this error while running the tests on the sources unpacked from
> the tarball
>
> it is the "scripts" project, that runs the tests against bash scripts:
>
> testBuildLoggingOpts
>
> ASSERT:Logging OPTS is not set correctly
> expected:<-Dlog4j.configurationFile=test.conf
> -Dbookkeeper.log.root.level=
> -Dbookkeeper.log.root.appender=test_log_dir
> -Dbookkeeper.log.dir=test_log_dir
> -Dbookkeeper.log.file=test_log_file> but
> was:<-Dlog4j.configurationFile=test.conf
> -Dbookkeeper.log.root.level=INFO
> -Dbookkeeper.log.root.appender=TEST
> -Dbookkeeper.log.dir=test_log_dir
> -Dbookkeeper.log.file=test_log_file>
>
> testBuildCLILoggingOpts
>
> ASSERT:Logging OPTS is not set correctly
> expected:<-Dlog4j.configurationFile=test.conf
> -Dbookkeeper.cli.root.logger=INFO,TEST
> -Dbookkeeper.cli.log.dir=test_log_dir
> -Dbookkeeper.cli.log.file=test_log_file> but
> was:<-Dlog4j.configurationFile=test.conf
> -Dbookkeeper.cli.log.root.level=test_log_dir
> -Dbookkeeper.cli.log.root.ap

[VOTE] Release 4.15.0, release candidate #1

[Andrey Yegorov]

Hi everyone,

Please review and vote on the release candidate #1 for the version 4.15.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* Release notes [1]
* The official Apache source and binary distributions to be deployed to
dist.apache.org [2]
* All artifacts to be deployed to the Maven Central Repository [3]
* Source code tag "release-4.15.0-rc1" [4] with git
sha451c54b72651d7cecb23c96b69a5de7f7dc55bb1

BookKeeper's KEYS file contains PGP keys we used to sign this release:
https://dist.apache.org/repos/dist/release/bookkeeper/KEYS

Please download these packages and review this release candidate:

- Review release notes
- Download the source package (verify shasum, and asc) and follow the
instructions to build and run the bookkeeper service.
- Download the binary package (verify shasum, and asc) and follow the
instructions to run the bookkeeper service.
- Review maven repo, release tag, licenses, and any other things you think
it is important to a release.

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1]
https://github.com/apache/bookkeeper/pull/3155/files#diff-eff52edf7022fd953f60c68061de09ec5d148eb37893de6af85a7964a2e57855
-
I will update the release notes to include changes between rc0 and rc1
[2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.0-rc1/

[3]
https://repository.apache.org/content/repositories/orgapachebookkeeper-1065/
[4] https://github.com/apache/bookkeeper/releases/tag/v4.15.0-rc1/

-- 
Andrey Yegorov

Re: [VOTE] Release 4.15.0, release candidate #0

[Andrey Yegorov]

Hi,

I don't think multi-core build with maven is a blocker, I don't think it
ever worked.

But I agree with Enrico that the log4j v2 is a blocker, I looked at te
problem a bit more, my comment on this:
https://github.com/apache/bookkeeper/pull/3148#issuecomment-1089196546


On Fri, Apr 1, 2022 at 9:08 PM Hang Chen  wrote:

> Hi Andrey,
> I download the source code and build it by multi cpu cores, it
> failed occasionally.  I opened an issue to track it.
> https://github.com/apache/bookkeeper/issues/3174
> It maybe an old issue, and it is not the blocker for this release.
>
> Thanks,
> Hang
>
> Andrey Yegorov  于2022年4月1日周五 23:02写道：
> >
> > ok, please cherry-pick or label PRs to cherry-pick into the release,
> > we'll have to go through RC1.
> > I tried upgrading Pulsar and enountered breaking changes listed here:
> >
> https://github.com/apache/bookkeeper/pull/3155/files#diff-eff52edf7022fd953f60c68061de09ec5d148eb37893de6af85a7964a2e57855
> > I'll look at the ways to simplify upgrade but making it simply a
> dependency
> > change might not be possible.
> >
> > On Fri, Apr 1, 2022 at 5:40 AM Enrico Olivelli 
> wrote:
> >
> > > -1
> > > I have run the tests and a few smoke tests and everything is passing.
> > >
> > > But unfortunately the Bookie is not logging anymore due to the switch
> to
> > > log4j2
> > >
> > > I have tracked the problem here
> > > https://github.com/apache/bookkeeper/issues/3165
> > >
> > >
> > > Not blocker for a release:
> > > I should also note that upgrading downstream applications that run the
> > > Bookie from Java code is quite hard
> > > I tried to upgrade these projects:
> > > - HerdDB: https://github.com/diennea/herddb/pull/784
> > > - Pravega.io: https://github.com/pravega/pravega/pull/6669
> > > Same problem: the BookieServer constructor changed, and the structure
> > > of the Bookie changed a lot (when we switched to Bookie to be a
> > > interface and created BookieImpl)
> > > I didn't try to upgrade Apache Pulsar, but I guess the problem will be
> the
> > > same
> > >
> > > I am also aware of other non-OSS projects that start the Bookie using
> > > Java in order to run unit tests
> > >
> > > It may make sense to create a BookieService utility class to bootstrap
> > > the Bookie from Java as many projects do.
> > >
> > >
> > > Enrico
> > >
> > >
> > > Il giorno mer 30 mar 2022 alle ore 01:33 Andrey Yegorov
> > >  ha scritto:
> > > >
> > > > Hi everyone,
> > > > Please review and vote on the release candidate #0 for the version
> > > 4.15.0,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * Release notes [1]
> > > > * The official Apache source and binary distributions to be deployed
> to
> > > > dist.apache.org [2]
> > > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > > * Source code tag "release-4.15.0" [4] with git
> > > > sha 9e2db102fe0dc9340f405a3afeacd2ef94359564
> > > >
> > > > BookKeeper's KEYS file contains PGP keys we used to sign this
> release:
> > > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > >
> > > > Please download these packages and review this release candidate:
> > > >
> > > > - Review release notes
> > > > - Download the source package (verify shasum, and asc) and follow the
> > > > instructions to build and run the bookkeeper service.
> > > > - Download the binary package (verify shasum, and asc) and follow the
> > > > instructions to run the bookkeeper service.
> > > > - Review maven repo, release tag, licenses, and any other things you
> > > think
> > > > it is important to a release.
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Release Manager
> > > >
> > > > [1]
> > > >
> > >
> https://github.com/apache/bookkeeper/pull/3155/files#diff-eff52edf7022fd953f60c68061de09ec5d148eb37893de6af85a7964a2e57855
> > > > [2]
> > >
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.0-rc0/
> > > > [3]
> > > >
> > >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1063/
> > > > [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.0-rc0
> > > >
> > > > --
> > > > Andrey Yegorov
> > >
> >
> >
> > --
> > Andrey Yegorov
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.15.0, release candidate #0

[Andrey Yegorov]

ok, please cherry-pick or label PRs to cherry-pick into the release,
we'll have to go through RC1.
I tried upgrading Pulsar and enountered breaking changes listed here:
https://github.com/apache/bookkeeper/pull/3155/files#diff-eff52edf7022fd953f60c68061de09ec5d148eb37893de6af85a7964a2e57855
I'll look at the ways to simplify upgrade but making it simply a dependency
change might not be possible.

On Fri, Apr 1, 2022 at 5:40 AM Enrico Olivelli  wrote:

> -1
> I have run the tests and a few smoke tests and everything is passing.
>
> But unfortunately the Bookie is not logging anymore due to the switch to
> log4j2
>
> I have tracked the problem here
> https://github.com/apache/bookkeeper/issues/3165
>
>
> Not blocker for a release:
> I should also note that upgrading downstream applications that run the
> Bookie from Java code is quite hard
> I tried to upgrade these projects:
> - HerdDB: https://github.com/diennea/herddb/pull/784
> - Pravega.io: https://github.com/pravega/pravega/pull/6669
> Same problem: the BookieServer constructor changed, and the structure
> of the Bookie changed a lot (when we switched to Bookie to be a
> interface and created BookieImpl)
> I didn't try to upgrade Apache Pulsar, but I guess the problem will be the
> same
>
> I am also aware of other non-OSS projects that start the Bookie using
> Java in order to run unit tests
>
> It may make sense to create a BookieService utility class to bootstrap
> the Bookie from Java as many projects do.
>
>
> Enrico
>
>
> Il giorno mer 30 mar 2022 alle ore 01:33 Andrey Yegorov
>  ha scritto:
> >
> > Hi everyone,
> > Please review and vote on the release candidate #0 for the version
> 4.15.0,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "release-4.15.0" [4] with git
> > sha 9e2db102fe0dc9340f405a3afeacd2ef94359564
> >
> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review maven repo, release tag, licenses, and any other things you
> think
> > it is important to a release.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1]
> >
> https://github.com/apache/bookkeeper/pull/3155/files#diff-eff52edf7022fd953f60c68061de09ec5d148eb37893de6af85a7964a2e57855
> > [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.0-rc0/
> > [3]
> >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1063/
> > [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.0-rc0
> >
> > --
> > Andrey Yegorov
>


-- 
Andrey Yegorov

Re: [ANNOUNCE] new BookKeeper Committer Nicolò Boschi

[Andrey Yegorov]

Congrats!

On Wed, Mar 30, 2022 at 7:17 AM ZhangJian He  wrote:

> Congratulations Nicolò !
>
> Thanks
> ZhangJian He
>
> Matteo Minardi  于2022年3月30日周三 17:36写道：
>
> > Congratulations Nico, well deserved!!
> >
> > Matteo
> > On 30 Mar 2022, 09:16 +0200, Enrico Olivelli ,
> wrote:
> > > The Project Management Committee (PMC) for Apache BookKeeper
> > > has invited Nicolò Boschi to become a committer and we are pleased
> > > to announce that he has accepted.
> > >
> > > Being a committer enables easier contribution to the
> > > project since there is no need to go via the patch
> > > submission process. This should enable better productivity.
> > > A PMC member helps manage and guide the direction of the project.
> > >
> > > More information about how ASF works here:
> > > https://www.apache.org/foundation/how-it-works.html
> > >
> > >
> > > Congratulations Nicolò !
> > >
> > > Best regards
> > > Enrico Olivelli
> >
>


-- 
Andrey Yegorov

[VOTE] Release 4.15.0, release candidate #0

[Andrey Yegorov]

Hi everyone,
Please review and vote on the release candidate #0 for the version 4.15.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* Release notes [1]
* The official Apache source and binary distributions to be deployed to
dist.apache.org [2]
* All artifacts to be deployed to the Maven Central Repository [3]
* Source code tag "release-4.15.0" [4] with git
sha 9e2db102fe0dc9340f405a3afeacd2ef94359564

BookKeeper's KEYS file contains PGP keys we used to sign this release:
https://dist.apache.org/repos/dist/release/bookkeeper/KEYS

Please download these packages and review this release candidate:

- Review release notes
- Download the source package (verify shasum, and asc) and follow the
instructions to build and run the bookkeeper service.
- Download the binary package (verify shasum, and asc) and follow the
instructions to run the bookkeeper service.
- Review maven repo, release tag, licenses, and any other things you think
it is important to a release.

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1]
https://github.com/apache/bookkeeper/pull/3155/files#diff-eff52edf7022fd953f60c68061de09ec5d148eb37893de6af85a7964a2e57855
[2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.0-rc0/
[3]
https://repository.apache.org/content/repositories/orgapachebookkeeper-1063/
[4] https://github.com/apache/bookkeeper/releases/tag/v4.15.0-rc0

--
Andrey Yegorov

Re: Blocker problem for 4.15 release ? Gradle build problems - Revert to Maven ?

[Andrey Yegorov]

Please review: https://github.com/apache/bookkeeper/pull/3130
It is easier to review change-by-change than all files.

It builds locally but I haven't run the tests.
I probably missed something related to CI.


On Mon, Mar 21, 2022 at 5:42 PM Yong Zhang 
wrote:

> +1.
>
> Next time when we migrate to the Gradle, I think we may need to keep both
> Maven
> and Gradle for a while, until the Gradle build is stable, then we remove
> the Maven build.
>
> On Tue, 22 Mar 2022 at 00:41, Andrey Yegorov 
> wrote:
>
> > +1.
> >
> > My attempts to dry run the release with gradle didn't succeed so far.
> > We still have one PR pending for the release 4.15
> > https://github.com/apache/bookkeeper/pull/3110
> > but I think we'll have high chance of success if we go with maven for
> 4.15.
> >
> > I can try restoring the deleted maven build + fix what was added without
> > maven.
> > After that we can switch CI to maven.
> >
> >
> > On Mon, Mar 21, 2022 at 6:31 AM Enrico Olivelli 
> > wrote:
> >
> > > I see that the source tarball has a wrong name in current master
> > >
> > > I filed this issue, I believe it is a problem for a release
> > > https://github.com/apache/bookkeeper/issues/3129
> > >
> > > I am also facing other problems with the Gradle build here
> > > https://github.com/apache/bookkeeper/pull/3084
> > > I am not able to exclude LogBack and some Netty modules imported by
> > > ZooKeeper,
> > > I see that we are importing dependencies using "dependency.create" and
> > > I don't find much documentation about this.
> > >
> > > When I build locally BK I see many warnings from Gradle, like:
> > > "Expiring Daemon because JVM heap space is exhausted" or
> > > or
> > >
> > > Gradle detected a problem with the following location:
> > >
> > >
> >
> '/Users/enrico.olivelli/dev/bookkeeper/bookkeeper-dist/build/distributions/bookkeeper-4.15.0-SNAPSHOT-src-bin.tar.gz'.
> > > Reason: Task ':bookkeeper-dist-src:distTarSha512' uses this output of
> > > task ':bookkeeper-dist:distTar' without declaring an explicit or
> > > implicit dependency. This can lead to incorrect results being
> > > produced, depending on what order the tasks are executed
> > >
> > >
> > > This Gradle migration is basically blocking the release and we are very
> > > late.
> > >
> > > IIUC there was some part of the community who pushed hard to do this
> > > migration, but all the enthusiasm seems to have gone away and now we
> > > have a broken repository.
> > >
> > > I wonder if it would be better to roll back to Maven, we didn't
> > > refactor the repository and restoring the pom.xml files may not be so
> > > hard as fixing all the Gradle problems.
> > >
> > >
> > >
> > >
> > > Enrico
> > >
> >
> >
> > --
> > Andrey Yegorov
> >
>


-- 
Andrey Yegorov

Re: Blocker problem for 4.15 release ? Gradle build problems - Revert to Maven ?

[Andrey Yegorov]

+1.

My attempts to dry run the release with gradle didn't succeed so far.
We still have one PR pending for the release 4.15
https://github.com/apache/bookkeeper/pull/3110
but I think we'll have high chance of success if we go with maven for 4.15.

I can try restoring the deleted maven build + fix what was added without
maven.
After that we can switch CI to maven.


On Mon, Mar 21, 2022 at 6:31 AM Enrico Olivelli  wrote:

> I see that the source tarball has a wrong name in current master
>
> I filed this issue, I believe it is a problem for a release
> https://github.com/apache/bookkeeper/issues/3129
>
> I am also facing other problems with the Gradle build here
> https://github.com/apache/bookkeeper/pull/3084
> I am not able to exclude LogBack and some Netty modules imported by
> ZooKeeper,
> I see that we are importing dependencies using "dependency.create" and
> I don't find much documentation about this.
>
> When I build locally BK I see many warnings from Gradle, like:
> "Expiring Daemon because JVM heap space is exhausted" or
> or
>
> Gradle detected a problem with the following location:
>
> '/Users/enrico.olivelli/dev/bookkeeper/bookkeeper-dist/build/distributions/bookkeeper-4.15.0-SNAPSHOT-src-bin.tar.gz'.
> Reason: Task ':bookkeeper-dist-src:distTarSha512' uses this output of
> task ':bookkeeper-dist:distTar' without declaring an explicit or
> implicit dependency. This can lead to incorrect results being
> produced, depending on what order the tasks are executed
>
>
> This Gradle migration is basically blocking the release and we are very
> late.
>
> IIUC there was some part of the community who pushed hard to do this
> migration, but all the enthusiasm seems to have gone away and now we
> have a broken repository.
>
> I wonder if it would be better to roll back to Maven, we didn't
> refactor the repository and restoring the pom.xml files may not be so
> hard as fixing all the Gradle problems.
>
>
>
>
> Enrico
>


-- 
Andrey Yegorov

Re: [ANNOUNCE] Andrey Yegorov joins the Apache BookKeeper PMC!

[Andrey Yegorov]

Thank you!

On Fri, Mar 18, 2022 at 3:44 PM Henry Saputra 
wrote:

> Woot! Congrats, Andrey!
>
> On Fri, Mar 18, 2022 at 12:28 PM Enrico Olivelli 
> wrote:
>
> > I am happy to announce that Andrey Yegorov joined today the Apache
> > BookKeeper PMC !
> >
> > If you are curious and you want to know more about how Apache works
> > and what is a PMC you can read this page
> > https://www.apache.org/dev/pmc.html
> >
> > Congratulations!
> >
> > Enrico
> >
>


-- 
Andrey Yegorov

Re: Website updated with the latest versions

[Andrey Yegorov]

I spot-checked the new website.
Overall it looks good; couple of problems that I noticed:

* Missing images, e.g. see
https://bookkeeper.staged.apache.org/docs/api/distributedlog-api
https://bookkeeper.staged.apache.org/docs/deployment/dcos
There are placeholders like "![DC/OS services]({{ img/dcos/services.png)"
where the images should be.
The old site has the images.

* I don't see links to the Javadoc even for the older versions but given
your p.2 it doesn matter much.

I briefly looked at javadoc.io and it looks like it only shows docs for the
latest versions,
do we have to publish snapshot artifacts to get "latest" javadoc?
OTOH we can be ok with the docs for the released version only.


On Fri, Mar 18, 2022 at 6:06 AM Nicolò Boschi  wrote:

> Dear community,
> The website (https://bookkeeper.apache.org/) is now updated with the
> latest
> releases (the latest website update was on May '21).
>
> I'd like to discuss a couple of points:
> 1) The new Docusaurus website is live at
> https://bookkeeper.staged.apache.org/. My plan would be to replace the
> current as soon as possible. I haven't received feedback yet. I'd like to
> set a deadline (two weeks?) and then deploy the new one. (thanks to GH
> history we can easily rollback in case of issues)
>
> 2) Currently, Javadoc files are missing for the latest versions. In order
> to improve maintainability, I propose to remove the javadoc from the
> website files and use an external service such as javadoc.io. IIUC we just
> need to push the JAR with the javadoc to Maven central. Then we have to
> update links from the website. We could recover the missing javadoc
> manually.
>
> Let me know what you think
>
> BR,
> Nicolò Boschi
>


-- 
Andrey Yegorov

Re: New Docusaurus website - ready for review

[Andrey Yegorov]

I merged it to let Nicolo continue on this.
I cannot review all 64000 lines there ;) but I cursory looked at the
changes + they are isolated
in their own directory and we can always rebuild the old site.

On Wed, Mar 16, 2022 at 8:18 AM Nicolò Boschi  wrote:

> Since I didn't get any feedback, I think we can merge the work
>
> I will follow up with a PR to deploy it to staging
> Thanks Enrico
>
> Nicolò Boschi
>
>
> Il giorno mer 9 mar 2022 alle ore 22:06 Nicolò Boschi <
> boschi1...@gmail.com>
> ha scritto:
>
> > Thanks Enrico
> >
> > In general it is possible that we cherry pick new little features and
> that
> >> we have to have versioned docs.
> >
> >
> > I would prefer to specify this change in the specific doc, saying
> > something like "this flag has been introduced in 4.15.y"
> >
> >
> >
> > Il giorno mer 9 mar 2022 alle ore 21:40 Enrico Olivelli <
> > eolive...@gmail.com> ha scritto:
> >
> >> I totally support this initiative!
> >> Especially if we can simply working on the website, adding docs and
> >> writing
> >> release notes.
> >>
> >> The point I have about your PR is that we are keeping only one version
> per
> >> each major release.
> >> In general it is possible that we cherry pick new little features and
> that
> >> we have to have versioned docs.
> >> That said, I have been following this projects for much time and I guess
> >> it
> >> won't happen often.
> >>
> >> I am +1 to committing your PR as it is in the current form as soon as
> >> possible.
> >>
> >> Thank you very much!
> >>
> >> Enrico
> >>
> >>
> >> Side note: we should exclude running tests for changes in the 'site3'
> >> directory
> >>
> >> Il Mer 9 Mar 2022, 21:31 Nicolò Boschi  ha
> scritto:
> >>
> >> > Dear community,
> >> >
> >> > I resumed the initial work intended to migrate the website to the
> >> > Docusaurus engine.
> >> > I recreated the whole website under the 'site3' directory. This is the
> >> pull
> >> > https://github.com/apache/bookkeeper/pull/3088
> >> >
> >> > There are a couple of changes I have had to do and they are all
> >> mentioned
> >> > in the pull request.
> >> > I pushed the website here:
> >> >
> https://urldefense.com/v3/__https://nicoloboschi.github.io/bookkeeper__;!!PbtH5S7Ebw!aBgi8sqV9ztSRdKefg9Eb2BRM7I1CBVx0hT1AMUErTfeif1CTLkb6-_RzgvlDTCkfZ_cwM9JEqVGb3k5P6d4PzmxMA$
> >> >
> >> > My plan would be the following:
> >> > - Have people try it out, find bugs, problems, typos and fix them all
> >> > - Create the apache staging deployment (as explained here [0]).
> >> Basically
> >> > we'll need to create a github action to push to the asf-staging-site
> >> branch
> >> > and enable deployment with .asf.yaml file
> >> > - After a while, make the official switch
> >> >
> >> > WDYT ?
> >> >
> >> >
> >> > [0]
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Stagingawebsitepreviewdomain
> >> >
> >> >
> >> >
> >> > BR,
> >> > Nicolò Boschi
> >> >
> >>
> >
> >
> > Nicolò Boschi
> >
>


-- 
Andrey Yegorov

Re: Manual call RocksDB compactRange operation causes CPU to spike to 100%

[Andrey Yegorov]

I am on the same page as Enrico: I don't have much experience with RocksDB;
as you can see from discussion on the PR
https://github.com/apache/bookkeeper/pull/2686#discussion_r613468033
the perf impact was a concern, OTOH the PR is fixing a perf issue a seek
time.

I searched and found this guide:
https://github.com/EighteenZi/rocksdb_wiki/blob/master/RocksDB-Tuning-Guide.md
Is it possible that tuning max_background_compactions and some other
parameters can help?
This PR https://github.com/apache/bookkeeper/pull/3056 made the RocksDb
tuning easier.

I'll help with review of a PR (hopefully supplemented with perf tests
results) but I cannot commit to fixing it.
I hope you and Maurice (author of the original PR) can find a workable
compromise.

On Tue, Mar 15, 2022 at 7:14 AM Enrico Olivelli  wrote:

> Hang,
>
> Il giorno mar 15 mar 2022 alle ore 02:47 Hang Chen
>  ha scritto:
> >
> > Hi BookKeeper Community,
> >
> >   For BookKeeper 4.14.0+, I have noticed that index deletion is
> > sometimes taking around 60 seconds which cause the CPU to spike to
> > 100%
> > ```
> > [2022-02-28T07:25:42.531Z] INFO db-storage-cleanup-10-1
> > EntryLocationIndex:191 Deleting indexes for ledgers: [3385184,
> > 3385239, 3385159, 3385142, 3385124, 3385193, 3384879, 3385165,
> > 3385916]
> > [2022-02-28T07:26:34.089Z] INFO db-storage-cleanup-10-1
> > EntryLocationIndex:266 Deleted indexes for 201065 entries from 9
> > ledgers in 51.557 seconds
> > [2022-02-28T07:40:42.534Z] INFO db-storage-cleanup-10-1
> > EntryLocationIndex:191 Deleting indexes for ledgers: [3385379,
> > 3385367, 3385718, 3385365, 3385412, 3385167, 3385357, 3386141]
> > [2022-02-28T07:41:47.867Z] INFO db-storage-cleanup-10-1
> > EntryLocationIndex:266 Deleted indexes for 134590 entries from 8
> > ledgers in 65.332 seconds
> > ```
> >
> > RocksDB compaction is a heavy operation and the checkpoint will be
> > triggered in high frequency, which causes db-storage-cleanup thread
> > always into high load, and makes the cpu keep 100%.
> >
> > This change was introduced by
> > https://github.com/apache/bookkeeper/pull/2686, The motivation of this
> > Pr is:
> >
> > > After deleting many ledgers, seeking to the end of the RocksDB
> metadata can take a long time and trigger timeouts upstream. Address this
> by improving the seek logic as well as compacting out tombstones in
> situations where we've just deleted many entries. This affects the entry
> location index and the ledger metadata index.
> >
> > For RocksDB, the CompactRange operation is a high overload operation.
> > we'd better avoid manual calls. Since RocksDB 7.0, the `compactRange`
> > API has been removed.
> > https://github.com/facebook/rocksdb/pull/9444
> >
> > IMO, we'd better remove the manual call compactRange in this PR, and
> > increase the `max_background_jobs` to accelerate auto compaction.
> >
> > Would you please give me more ideas?
> I don't have much experience with RocksDB.
>
> Did you make a prototype ?
>
> Sharing some results in a prototype would help a lot.
>
> I am not sure, but maybe we can add a option to enable/disable manual
> compaction and to tune max_background_jobs
> this way we can rollback in case of problems with your proposal
>
> Enrico
>
> >
> > Thanks,
> > Hang
>


-- 
Andrey Yegorov

gradle and BK release

[Andrey Yegorov]

Hi,

I am looking for help with the BK release procedure with gradle.
We dropped maven support in this version but all release scripts and docs
use maven.

I found a gradle plugin that supposedly implements functionality similar to
maven's release plugin: https://github.com/researchgate/gradle-release
One catch is the project does not look active/no commits in 2+ years.

Are there better alternatives?

Current instructions:
https://bookkeeper.apache.org/community/release_guide/

Scripts that use maven:
---
$ pwd
  ~/src/bookkeeper/dev

$ grep -rI "mvn " .

docker/run.sh:echo '  mvn clean install'
release/000-run-docker.sh:VERSION=`cd $BOOKKEEPER_ROOT && mvn
org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate
-Dexpression=project.version | grep -Ev '(^\[|Download\w+:)' | sed
's/^\(.*\)-SNAPSHOT/\1/'`
release/003-release-perform.sh:mvn release:perform \
release/001-release-branch.sh:mvn release:branch \
release/002-release-prepare.sh:mvn release:prepare \

$ pwd
~/src/bookkeeper/site/scripts

$ grep -r mvn .
./javadoc-gen.sh:mvn clean install javadoc:aggregate -DskipTests
-

-- 
Andrey Yegorov

Re: CI on JDK11

[Andrey Yegorov]

LGTM but I'd appreciate a second opinion.

On Fri, Feb 25, 2022 at 1:32 AM Nicolò Boschi  wrote:

> Dear community,
>
> I created this pull [1] to move the CI to JDK11.
> BookKeeper has been running with JDK11 for a long time. The official docker
> image is using JDK11 so it's much better to verify the codebase using the
> same jdk version used in production by most of the users.
>
> [1] https://github.com/apache/bookkeeper/pull/3027
>
> BR,
> Nicolò Boschi
>


-- 
Andrey Yegorov

Re: BookKeeper 4.15 release ?

[Andrey Yegorov]

I think it is time to resume this conversation.
It looks like the PRs we were waiting for are merged.
I added a couple of PRs that fix flaky tests etc.
https://github.com/apache/bookkeeper/pull/3032
https://github.com/apache/bookkeeper/pull/3043
are the ones left at the moment.

Also there is a fix for the situation with uneven thread usage on
OrderedExecutor due to ledger ids being (mostly?) even.
https://github.com/apache/bookkeeper/pull/3023
Please let me know if you feel comfortable with this change.

I'll start going through the PRs/merging what's approved next week.
Please help with reviews/rebasing if needed.

Assuming we don't have blocking issues it's time for the release.
I'll volunteer as a release manager unless there are objections or other
volunteers.
I may need some help with the release, we moved to gradle and didn't update
the release guide from maven.
We'll see how it goes.
https://bookkeeper.apache.org/community/release_guide/

Action items:

1. Please review open PRs (we have 88 at the moment) and approve or request
changes.
Committers are welcome to merge approved PRs.

2. Reply yey/nay for the release. Let the blockers be known.

3. Reply if you want to be a release manager or ok with me doing the
release.

Thank you!
Andrey.

On Mon, Dec 13, 2021 at 11:57 AM Nicolò Boschi  wrote:

> I think it's worth to wait for
> https://github.com/apache/bookkeeper/pull/2937, at least for 4.15
>
> BTW we should focus on releasing other active branches cherry-picking the
> patch for SLF4J vulnerability
>
> Nicolò Boschi
>
> Il Lun 13 Dic 2021, 20:32 Andrey Yegorov  ha
> scritto:
>
> > https://github.com/apache/bookkeeper/pull/1949 has merge conflicts and
> > multiple comments not addressed in almost two years.
> > I don't know if we can expect Rajan to commit to getting that into a
> > mergeable state fast enough.
> >
> > On Mon, Dec 13, 2021 at 3:03 AM Hang Chen  wrote:
> >
> > > We'd better include this PR into 4.15
> > > https://github.com/apache/bookkeeper/pull/1949
> > >
> > > Thanks,
> > > Hang
> > >
> > > Jack Vanlightly  于2021年12月13日周一 18:59写道：
> > > >
> > > > The original BP-44 was running without the journal but by mistake I
> > also
> > > > created the observability changes as BP-44 as well so I changed the
> > > running
> > > > without the journal to BP-46.
> > > >
> > > > BP-44 is all in. I have submitted the PR for BP-46 but there are test
> > > > failures I need to fix, so very close there.
> > > >
> > > > I can get BP-46 ready for review hopefully by end of day tomorrow
> > > (14th). I
> > > > don't have a strong opinion either way, on waiting for that vs
> cutting
> > a
> > > > release now.
> > > >
> > > > Jack
> > > >
> > > > On Mon, Dec 13, 2021 at 11:38 AM Enrico Olivelli <
> eolive...@gmail.com>
> > > > wrote:
> > > >
> > > > > I can't see any pending PR for BP-44, is it completed ?
> > > > >
> > > > >
> > > > > Enrico
> > > > >
> > > > > Il giorno lun 13 dic 2021 alle ore 11:05 Prashant Kumar <
> > > > > prashant.kumar.si...@gmail.com> ha scritto:
> > > > >
> > > > > > We are good to go. I think we are waiting for BP-44.
> > > > > > If we get a green light there. We are good to go as far as I
> could
> > > tell.
> > > > > > Thanks
> > > > > > -Prashant
> > > > > >
> > > > > > On Mon, Dec 13, 2021 at 2:02 AM Enrico Olivelli <
> > eolive...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello folks,
> > > > > > > I believe it is time for cutting the 4.15 release.
> > > > > > >
> > > > > > > The Gradle migration is in good shape (please anyone correct me
> > if
> > > I am
> > > > > > > wrong).
> > > > > > >
> > > > > > > The "Skip journal" BP is still in progress but we are too late
> on
> > > the
> > > > > > > Schedule.
> > > > > > >
> > > > > > > Thoughts ?
> > > > > > >
> > > > > > > Enrico
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> >
> > --
> > Andrey Yegorov
> >
>


-- 
Andrey Yegorov

Re: Pull request to remove Maven from the project

[Andrey Yegorov]

+1.

Maven build does not work at the moment anyway (see
https://github.com/apache/bookkeeper/pull/2994 ) and maintaining two build
systems is an extra pain.

But it is not the end of the PIP-43 yet, we still need to update
https://bookkeeper.apache.org/community/release_guide/ and
https://bookkeeper.apache.org/community/contributing/

On Wed, Jan 26, 2022 at 9:35 AM Nicolò Boschi  wrote:

> Dear community,
>
> I've created a pull request to remove POM files from the project.
> https://github.com/apache/bookkeeper/pull/3009
>
> After we merge that, the PIP-43 will have finally reached the end of his -
> long - journey.
>
> BR,
> Nicolò Boschi
>


-- 
Andrey Yegorov

Re: [VOTE] Release 4.14.4, release candidate #0

[Andrey Yegorov]

+1, ship it

* checked localbookie starts/works
* spot-checked artifacts on maven central
* checked release tag/sha


On Tue, Jan 4, 2022 at 7:23 AM Dave Fisher  wrote:

> It looks like this VOTE needs one more binding +1 ...
>
> > On Dec 23, 2021, at 11:14 PM, Enrico Olivelli 
> wrote:
> >
> > +1 (binding)
> > Built from source on JDK8
> > Verified signatures/md5
> > Run LocalBookie, some quick tests
> > Checked git log
> >
> > Please don't forget about the release notes!
> >
> > Enrico
> >
> > Il giorno gio 23 dic 2021 alle ore 11:26 Enrico Olivelli <
> > eolive...@gmail.com> ha scritto:
> >
> >> Matteo,
> >> Are you preparing the release notes ?
> >> Which are the contents of the release ?
> >>
> >> it looks like we are cutting this release with some dependency updates,
> >> Netty and GRPC, and with some fixes about the Metadata Driver
> >>
> >> This is the git log from GitHub
> >> https://github.com/apache/bookkeeper/commits/branch-4.14
> >>
> >> Enrico
> >>
> >> Il giorno gio 23 dic 2021 alle ore 09:48 Nicolò Boschi <
> >> boschi1...@gmail.com> ha scritto:
> >>
> >>> +1 (non binding)
> >>>
> >>> Verified the followings:
> >>> - checksum and signatures
> >>> - build from sources w jdk11
> >>> - ledgers write/read with "localbookie" command
> >>> - checked netty and grpc jars in the "lib" directory
> >>>
> >>> Il giorno mer 22 dic 2021 alle ore 18:39 Matteo Merli <
> mme...@apache.org>
> >>> ha scritto:
> >>>
> >>>> Hi everyone,
> >>>>
> >>>> Please review and vote on the release candidate #0 for the version
> >>>> 4.14.4, as follows:
> >>>> [ ] +1, Approve the release
> >>>> [ ] -1, Do not approve the release (please provide specific comments)
> >>>>
> >>>> The complete staging area is available for your review, which
> includes:
> >>>> * The official Apache source and binary distributions to be deployed
> >>>> to
> >>>>
> >>>
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.4-rc0/
> >>>> * All artifacts to be deployed to the Maven Central Repository:
> >>>>
> >>>
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1061
> >>>> * Source code tag "v4.14.4-rc0" with git sha
> >>>> 2071149af5c7fb0c905efaa84c7dcd911fc5dca0
> >>>>
> >>>> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> >>>> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >>>>
> >>>> Please download these packages and review this release candidate:
> >>>>
> >>>> - Download the source package (verify shasum, and asc) and follow the
> >>>> instructions to build and run the bookkeeper service.
> >>>> - Download the binary package (verify shasum, and asc) and follow the
> >>>> instructions to run the bookkeeper service.
> >>>> - Review maven repo, release tag, licenses, and any other things you
> >>>> think are important to a release.
> >>>>
> >>>> The vote will be open for at least 72 hours. It is adopted by majority
> >>>> approval, with at least 3 PMC affirmative votes.
> >>>>
> >>>> Thanks,
> >>>> Matteo
> >>>>
> >>>> --
> >>>> Matteo Merli
> >>>> 
> >>>>
> >>>
> >>>
> >>> --
> >>> Nicolò Boschi
> >>>
> >>
>
>

-- 
Andrey Yegorov

Re: BP-44: direct io entry logging

[Andrey Yegorov]

Hi Maurice,

This is an interesting addition to the Bookkeeper!
Thank you for contributing it back to the community.
Can you share some performance data/charts to highlight benefits of this
feature?

Also, is your workload write-heavy with reads mostly from the cold data?
I imagine this option will increase latency of tailing reads (short delay,
after the data made it to the disk).


On Wed, Dec 15, 2021 at 9:36 AM Maurice Barnum 
wrote:

> In order to improve IO utilization, Splunk developed support for logging
> entries bypassing the kernel's buffer cache via O_DIRECT.  The code has
> been in production for several months, running on Linux, with an
> implementation for MacOS also included.
>
> BP-44 proposes to merge the code into the mainstream code base.
>
> https://github.com/apache/bookkeeper/pull/2932 is mostly complete merge of
> the code base to the Bookkeeper master branch.  Some unit tests fail, most
> likely due to merge errors that need to be resolved.
>
> Comments appreciated.
>


-- 
Andrey Yegorov

Re: BookKeeper 4.15 release ?

[Andrey Yegorov]

https://github.com/apache/bookkeeper/pull/1949 has merge conflicts and
multiple comments not addressed in almost two years.
I don't know if we can expect Rajan to commit to getting that into a
mergeable state fast enough.

On Mon, Dec 13, 2021 at 3:03 AM Hang Chen  wrote:

> We'd better include this PR into 4.15
> https://github.com/apache/bookkeeper/pull/1949
>
> Thanks,
> Hang
>
> Jack Vanlightly  于2021年12月13日周一 18:59写道：
> >
> > The original BP-44 was running without the journal but by mistake I also
> > created the observability changes as BP-44 as well so I changed the
> running
> > without the journal to BP-46.
> >
> > BP-44 is all in. I have submitted the PR for BP-46 but there are test
> > failures I need to fix, so very close there.
> >
> > I can get BP-46 ready for review hopefully by end of day tomorrow
> (14th). I
> > don't have a strong opinion either way, on waiting for that vs cutting a
> > release now.
> >
> > Jack
> >
> > On Mon, Dec 13, 2021 at 11:38 AM Enrico Olivelli 
> > wrote:
> >
> > > I can't see any pending PR for BP-44, is it completed ?
> > >
> > >
> > > Enrico
> > >
> > > Il giorno lun 13 dic 2021 alle ore 11:05 Prashant Kumar <
> > > prashant.kumar.si...@gmail.com> ha scritto:
> > >
> > > > We are good to go. I think we are waiting for BP-44.
> > > > If we get a green light there. We are good to go as far as I could
> tell.
> > > > Thanks
> > > > -Prashant
> > > >
> > > > On Mon, Dec 13, 2021 at 2:02 AM Enrico Olivelli  >
> > > > wrote:
> > > >
> > > > > Hello folks,
> > > > > I believe it is time for cutting the 4.15 release.
> > > > >
> > > > > The Gradle migration is in good shape (please anyone correct me if
> I am
> > > > > wrong).
> > > > >
> > > > > The "Skip journal" BP is still in progress but we are too late on
> the
> > > > > Schedule.
> > > > >
> > > > > Thoughts ?
> > > > >
> > > > > Enrico
> > > > >
> > > >
> > >
>


-- 
Andrey Yegorov

Re: Using GitHub merge button

[Andrey Yegorov]

+1 from me too.

On Wed, Oct 20, 2021 at 10:27 AM Henry Saputra 
wrote:

> Other than formatting of the subject and some additional info to the PR
> commit, and automatically set tags to the PR, I think the script do not do
> anything special, right?
>
> I think this is kind of "legacy" approach when we commit to the Apache
> managed Git repo, following approach that Apache Spark was doing.
>
> I am +1 to just use Github merge tool.
>
> - Henry
>
> On Wed, Oct 20, 2021 at 10:05 AM Enrico Olivelli 
> wrote:
>
> > Do we need some VOTE?
> >
> > I think we do not need it
> >
> > But we should update the committer guide
> >
> > Enrico
> >
> > Il Mer 20 Ott 2021, 18:50 Henry Saputra  ha
> > scritto:
> >
> > > Ack for this one. I have encountered this issue for some PRs.
> > >
> > > - Henry
> > >
> > > On Tue, Oct 5, 2021 at 2:44 AM Enrico Olivelli 
> > > wrote:
> > >
> > > > Hello,
> > > > The bk-merge-script has new problems, now it fails for me
> consistently,
> > > it
> > > > looks like a problem with the GH Merge API, see the error message
> > below.
> > > >
> > > > In Pulsar we are using the GH merge button
> > > > the main limitations of using the Merge Button are:
> > > > - you have to manually set labels and milestone
> > > > - it does not handle cherry-picks
> > > >
> > > > I propose to switch to using the GH merge button, or at least to
> allow
> > > > Committers to use it
> > > > We can write instructions for committers
> > > >
> > > > Thoughts ?
> > > > Enrico
> > > >
> > > >
> > > > From https://github.com/apache/bookkeeper
> > > >
> > > >  * [new ref] refs/pull/2808/head -> PR_TOOL_MERGE_PR_2808
> > > >
> > > > Enter reviewers [Prashant Kumar , Enrico Olivelli <
> > > > eolive...@gmail.com>,
> > > > Andrey Yegorov ]:
> > > >
> > > > ['git', 'log', 'HEAD..PR_TOOL_MERGE_PR_2808', '--pretty=format:%h
> [%an]
> > > > %s']
> > > >
> > > > List pull request commits in squashed commit message? (y/n): n
> > > >
> > > > git rev-parse PR_TOOL_MERGE_PR_2808
> > > >
> > > >
> > > > Merge Pull Request (2808) with following details:
> > > >
> > > > {
> > > >
> > > > "commit_title": "set response body for bookie is_ready api",
> > > >
> > > > "commit_message": "API `/api/v1/bookie/is_ready` should set some
> > > > response body, when code 200 returned.\r\nOr, this will confuse
> people,
> > > > they don't know if this api  return
> successfully.\r\n\r\n\r\n![image](
> > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://user-images.githubusercontent.com/9278488/134855390-f8c18ba2-3968-4352-82fd-f59883dcd2cb.png__;!!PbtH5S7Ebw!IQLVsLV2a1X63kW-oAhpojfVT_4VyXM2S3GYG4W4Dd_m-G50MHIm4-qOAV6n6yz9CbtPqA$
> > > > )\n\nReviewers:
> > > > Prashant Kumar , Enrico Olivelli , Andrey
> > > > Yegorov \n\nThis closes #2808 from
> > > > gaozhangmin/api_bookie_is_ready_message_when_return_code_200",
> > > >
> > > > "sha": "89f16843232848972bbd551b96ecc13428f3f36d",
> > > >
> > > > "merge_method": "squash"
> > > >
> > > > } (y/n): y
> > > >
> > > > Unable to fetch URL, exiting:
> > > > https://api.github.com/repos/apache/bookkeeper/pulls/2808/merge -
> HTTP
> > > > Error 404: Not Found
> > > >
> > > > HTTP Error 404: Not Found
> > > >
> > >
> >
>


-- 
Andrey Yegorov

Re: Release Bookkeeper 4.15

[Andrey Yegorov]

Prashant,

+1 from me.
Thank you for pushing the gradle migration.

We may need to check the timeline of BP-44 with Jack.
BP-44 will require major release and if it is a week away it might be worth
holding off.

On Wed, Oct 20, 2021 at 10:20 AM Prashant Kumar <
prashant.kumar.si...@gmail.com> wrote:

> Dear Apache Bookkeeper community,
> We have successfully migrated build and release to gradle.
> As the next step we would like to do a release bookkeeper version 4.15 and
> document the release process,
> before shutting down maven which essentially means removing pom.xml files
> and updating documentation.
>
> In addition to that, that time also almost coincides with the normal
> release cycle for Bookkeeper 4.15.
>
> Kindly, do let me know your thoughts.
> Thanks and Best Regards
>
> -Prahant Kumar
>


-- 
Andrey Yegorov

Re: [DISCUSS] Release 4.14.3

[Andrey Yegorov]

I am ok with the 4.14.3 release.

Do we want to include other PRs into the release, i.e.
https://github.com/apache/bookkeeper/pull/2796
https://github.com/apache/bookkeeper/pull/2794
https://github.com/apache/bookkeeper/pull/2813
https://github.com/apache/bookkeeper/pull/2793
https://github.com/apache/bookkeeper/pull/2816


On Mon, Oct 11, 2021 at 6:25 PM Yong Zhang  wrote:

> Hey,
>
> Here has a critical issue about the auto-recovery service which will
> cause the service will continue to restart. The PR is
> https://github.com/apache/bookkeeper/pull/2802.
>
> We want to create a new release 4.14.3 to address the problem. Any thought?
> If there is no objection, I can release it in the next few days.
>
> Regards,
> Yong
>


-- 
Andrey Yegorov

[DISCUSS] BP-45: a pluggable way to modify payload sent to the ledger

[Andrey Yegorov]

Hi,

I created a new BP and I'd love to hear some feedback.

The proposed change targets addition of a pluggable way to modify payload
sent to the ledger. Specific use cases include GDPR compliance, encryption,
and compression. While these can be implemented at the application level,
having this as a BK client level extension unlocks such functionality to
any application built on top of Bookkeeper.

Master Issue: https://github.com/apache/bookkeeper/issues/2731
PR: https://github.com/apache/bookkeeper/pull/2730
Link to the BP's text:
https://github.com/apache/bookkeeper/blob/efe94fa37c5fe061de668a6d11c87a1e82f8a25b/site/bps/BP-45-LedgerPayloadInterceptor.md

-- 
Andrey Yegorov

Re: [DISCUSS] Release 4.14.1

[Andrey Yegorov]

I am ok releasing 4.14.1 now.
I have finalized all the steps for the 4.14: updated the site etc.
I will NOT send an email about the 4.14.0 release and we'll use 4.14.1
to notify the community to limit the distribution of the build with
the blocker.

Does it sound ok to you?

On Tue, May 25, 2021 at 8:39 AM Enrico Olivelli  wrote:
>
> Il giorno mar 25 mag 2021 alle ore 17:15 Matteo Merli
>  ha scritto:
> >
> > On Tue, May 25, 2021 at 6:00 AM Enrico Olivelli  wrote:
> > > Before rushing to a release, isn't it better to wait for a couple of
> > > weeks in order to see
> > > if other people in the community find other problems on 4.14.0
> >
> > As it is, the 4.14.0 release cannot be used by Pulsar. At the same
> > time, there are blocker issues for the upcoming Pulsar 2.8.0 that are
> > only fixed in the BK 4.14 branch.
> >
> > I don't think that waiting 2 weeks will help at all here. We already
> > have a blocker issue, which is already fixed and that could unblock
> > 4.14 and Pulsar 2.8.
> >
> > > AFAIK BK 4.14.0 has not been announced yet publicly
> >
> > Since we cannot change 4.14.0 at this point (the vote was cast, the
> > jars are in maven central), I don't think that it would be a problem
> > to just announce 4.14.1 directly, since the 4.14.0 is not usable.
>
> I agree.
>
> Enrico
>
> >
> > Matteo

Re: [VOTE] Release 4.14.0, release candidate #0

[Andrey Yegorov]

Docker is published:
https://hub.docker.com/r/apache/bookkeeper/tags?page=1&ordering=last_updated
Python client: https://pypi.org/project/apache-bookkeeper-client/4.14.0/

I noticed that backwards-compat tests haven't been touched since release
4.9.2; updated https://github.com/apache/bookkeeper/pull/2719 - I haven't
spent much time figuring out what's going on there, I'll appreciate the
review.

Docs and release notes changes approved by Henry:
https://github.com/apache/bookkeeper/pull/2712
Unless I see requests to change I'll merge these to get the website updated.


On Mon, May 17, 2021 at 5:07 PM Andrey Yegorov 
wrote:

> Thank you!
>
> We have  quorum of 3 binding votes:
> +1 from Amit Chavan
> +1 from Matteo Merli (binding)
> +1 from Sijie Guo (binding)
> +1 from Enrico Olivelli (binding)
>
> I'll proceed with the final steps of the release tonight/tomorrow morning
> as time permits.
>
>
> On Mon, May 17, 2021 at 12:43 PM Enrico Olivelli 
> wrote:
>
>> +1 (binding)
>>
>> Built from sources
>> Run test from sources
>> Basic verifications
>>
>> Thanks for driving the release
>>
>> Enrico
>>
>> Il Lun 17 Mag 2021, 20:53 Sijie Guo  ha scritto:
>>
>> > +1 (binding)
>> >
>> > All things below are good.
>> >
>> >   * Signatures
>> >   * Source package structure, and compile
>> >   * Binary package
>> >   * Running standalone bookie and client
>> >
>> > On Mon, May 17, 2021 at 9:43 AM Matteo Merli  wrote:
>> >
>> > > +1 (binding)
>> > >
>> > > Checked:
>> > >   * Signatures
>> > >   * Source package structure, and compile
>> > >   * Binary package
>> > >   * Running standalone bookie and client
>> > >
>> > >
>> > > --
>> > > Matteo Merli
>> > > 
>> > >
>> > > On Fri, May 14, 2021 at 2:23 PM Andrey Yegorov
>> > >  wrote:
>> > > >
>> > > > I do not hear voices against it so I assume everyone is ok with
>> this.
>> > > > (I started building RC1 and docker hanged mid-process; happy to not
>> > deal
>> > > > with retry).
>> > > >
>> > > > The vote has been out for about 5 days already.
>> > > > I need votes from 3 PMCs to proceed.
>> > > >
>> > > >
>> > > > On Fri, May 14, 2021 at 12:10 PM Henry Saputra <
>> > henry.sapu...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > I am +1 to keep Gradle out from this release since the files are
>> ok
>> > > not to
>> > > > > be included.
>> > > > >
>> > > > > I am ok to keep RC0 for 4.14.0 for VOTE as release candidate.
>> > > > >
>> > > > > - Henry
>> > > > >
>> > > > > On Fri, May 14, 2021 at 12:14 AM Enrico Olivelli <
>> > eolive...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Andrey,
>> > > > > > I have merged the patch.
>> > > > > >
>> > > > > > In my opinion the source tarball should reflect the git repo
>> > without
>> > > > > > the '.git' database.
>> > > > > > That said, the most important thing is that the source release
>> > > > > > contains the files to build the project and produce working
>> > binaries.
>> > > > > >
>> > > > > > This is in the spirit of Open Source (of the ASF at least), you
>> can
>> > > > > > pick the sources, modify them, build and run the new binaries.
>> > > > > >
>> > > > > > AFAIK The Gradle build is not fully implemented so it is not so
>> > > > > > important to see those files at the moment
>> > > > > > I agree that we can strip away the website, it is useless to the
>> > > > > > purpose of building the project.
>> > > > > >
>> > > > > > Enrico
>> > > > > >
>> > > > > >
>> > > > > > Il giorno gio 13 mag 2021 alle ore 23:43 Andrey Yegorov
>> > > > > >  ha scritto:
>> > > > > > >
>> > > > > > > I looked at the situation with src release artifact, and:
>

Re: [VOTE] Release 4.14.0, release candidate #0

[Andrey Yegorov]

Thank you!

We have  quorum of 3 binding votes:
+1 from Amit Chavan
+1 from Matteo Merli (binding)
+1 from Sijie Guo (binding)
+1 from Enrico Olivelli (binding)

I'll proceed with the final steps of the release tonight/tomorrow morning
as time permits.


On Mon, May 17, 2021 at 12:43 PM Enrico Olivelli 
wrote:

> +1 (binding)
>
> Built from sources
> Run test from sources
> Basic verifications
>
> Thanks for driving the release
>
> Enrico
>
> Il Lun 17 Mag 2021, 20:53 Sijie Guo  ha scritto:
>
> > +1 (binding)
> >
> > All things below are good.
> >
> >   * Signatures
> >   * Source package structure, and compile
> >   * Binary package
> >   * Running standalone bookie and client
> >
> > On Mon, May 17, 2021 at 9:43 AM Matteo Merli  wrote:
> >
> > > +1 (binding)
> > >
> > > Checked:
> > >   * Signatures
> > >   * Source package structure, and compile
> > >   * Binary package
> > >   * Running standalone bookie and client
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> > > On Fri, May 14, 2021 at 2:23 PM Andrey Yegorov
> > >  wrote:
> > > >
> > > > I do not hear voices against it so I assume everyone is ok with this.
> > > > (I started building RC1 and docker hanged mid-process; happy to not
> > deal
> > > > with retry).
> > > >
> > > > The vote has been out for about 5 days already.
> > > > I need votes from 3 PMCs to proceed.
> > > >
> > > >
> > > > On Fri, May 14, 2021 at 12:10 PM Henry Saputra <
> > henry.sapu...@gmail.com>
> > > > wrote:
> > > >
> > > > > I am +1 to keep Gradle out from this release since the files are ok
> > > not to
> > > > > be included.
> > > > >
> > > > > I am ok to keep RC0 for 4.14.0 for VOTE as release candidate.
> > > > >
> > > > > - Henry
> > > > >
> > > > > On Fri, May 14, 2021 at 12:14 AM Enrico Olivelli <
> > eolive...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Andrey,
> > > > > > I have merged the patch.
> > > > > >
> > > > > > In my opinion the source tarball should reflect the git repo
> > without
> > > > > > the '.git' database.
> > > > > > That said, the most important thing is that the source release
> > > > > > contains the files to build the project and produce working
> > binaries.
> > > > > >
> > > > > > This is in the spirit of Open Source (of the ASF at least), you
> can
> > > > > > pick the sources, modify them, build and run the new binaries.
> > > > > >
> > > > > > AFAIK The Gradle build is not fully implemented so it is not so
> > > > > > important to see those files at the moment
> > > > > > I agree that we can strip away the website, it is useless to the
> > > > > > purpose of building the project.
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > > >
> > > > > > Il giorno gio 13 mag 2021 alle ore 23:43 Andrey Yegorov
> > > > > >  ha scritto:
> > > > > > >
> > > > > > > I looked at the situation with src release artifact, and:
> > > > > > > - it was definitely built from the right changelist (I spot
> > checked
> > > > > some
> > > > > > > files affected by the last commit)
> > > > > > > - site/ is explicitly excluded
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://github.com/apache/bookkeeper/blob/732b6cf2a6576c844a4b43176125e82d500a6ea8/bookkeeper-dist/src/assemble/src.xml#L61
> > > > > > > - site2 is not excluded (should be fixed)
> > > > > > > - gradle files are not explicitly included, pom are
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://github.com/apache/bookkeeper/blob/732b6cf2a6576c844a4b43176125e82d500a6ea8/bookkeeper-dist/src/assemble/src.xml#L33
> > > > > > >
> > > > > > > that src.xml is used to package the release artifacts
> > &

Re: [VOTE] Release 4.14.0, release candidate #0

[Andrey Yegorov]

I do not hear voices against it so I assume everyone is ok with this.
(I started building RC1 and docker hanged mid-process; happy to not deal
with retry).

The vote has been out for about 5 days already.
I need votes from 3 PMCs to proceed.


On Fri, May 14, 2021 at 12:10 PM Henry Saputra 
wrote:

> I am +1 to keep Gradle out from this release since the files are ok not to
> be included.
>
> I am ok to keep RC0 for 4.14.0 for VOTE as release candidate.
>
> - Henry
>
> On Fri, May 14, 2021 at 12:14 AM Enrico Olivelli 
> wrote:
>
> > Andrey,
> > I have merged the patch.
> >
> > In my opinion the source tarball should reflect the git repo without
> > the '.git' database.
> > That said, the most important thing is that the source release
> > contains the files to build the project and produce working binaries.
> >
> > This is in the spirit of Open Source (of the ASF at least), you can
> > pick the sources, modify them, build and run the new binaries.
> >
> > AFAIK The Gradle build is not fully implemented so it is not so
> > important to see those files at the moment
> > I agree that we can strip away the website, it is useless to the
> > purpose of building the project.
> >
> > Enrico
> >
> >
> > Il giorno gio 13 mag 2021 alle ore 23:43 Andrey Yegorov
> >  ha scritto:
> > >
> > > I looked at the situation with src release artifact, and:
> > > - it was definitely built from the right changelist (I spot checked
> some
> > > files affected by the last commit)
> > > - site/ is explicitly excluded
> > >
> >
> https://github.com/apache/bookkeeper/blob/732b6cf2a6576c844a4b43176125e82d500a6ea8/bookkeeper-dist/src/assemble/src.xml#L61
> > > - site2 is not excluded (should be fixed)
> > > - gradle files are not explicitly included, pom are
> > >
> >
> https://github.com/apache/bookkeeper/blob/732b6cf2a6576c844a4b43176125e82d500a6ea8/bookkeeper-dist/src/assemble/src.xml#L33
> > >
> > > that src.xml is used to package the release artifacts
> > >
> >
> https://github.com/apache/bookkeeper/blob/master/bookkeeper-dist/pom.xml#L55
> > >
> > > It looks like the plugin resolves the situation with the file not
> > matching
> > > includes and excludes by excluding it.
> > >
> > > PR that adds gradle into the includes patterns and excludes site2:
> > > https://github.com/apache/bookkeeper/pull/2714
> > >
> > > I can either rebuild the RC or we can decide to proceed.
> > > Gradle was not used to build/release 4.14.0
> > >
> > >
> > > On Thu, May 13, 2021 at 8:34 AM Andrey Yegorov <
> > andrey.yego...@datastax.com>
> > > wrote:
> > >
> > > > I ran the release script, as described in the release procedure.
> > > > It all ran from the docker container.
> > > >
> > > > On Wed, May 12, 2021 at 11:30 PM Henry Saputra <
> > henry.sapu...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi Andrey,
> > > >>
> > > >> How did you generate the source release artifact? Looks like it is
> > missing
> > > >> some files like the site directory and the gradle build files.
> > > >>
> > > >> - Henry
> > > >>
> > > >>
> > > >>
> > > >> On Sun, May 9, 2021 at 11:16 PM Andrey Yegorov <
> > > >> andrey.yego...@datastax.com>
> > > >> wrote:
> > > >>
> > > >> > Hi everyone,
> > > >> > Please review and vote on the release candidate #0 for the version
> > > >> 4.14.0,
> > > >> > as follows:
> > > >> > [ ] +1, Approve the release
> > > >> > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > >> >
> > > >> > The complete staging area is available for your review, which
> > includes:
> > > >> > * Release notes [1]
> > > >> > * The official Apache source and binary distributions to be
> > deployed to
> > > >> > dist.apache.org [2]
> > > >> > * All artifacts to be deployed to the Maven Central Repository [3]
> > > >> > * Source code tag "release-4.14.0" [4] with git sha
> > > >> > 4729682f00f05f23a1821211cbce064e653edb83
> > > >> >
> > > >> > BookKeeper's KEYS file contains PGP keys we used to sig

Re: [VOTE] Release 4.14.0, release candidate #0

[Andrey Yegorov]

I looked at the situation with src release artifact, and:
- it was definitely built from the right changelist (I spot checked some
files affected by the last commit)
- site/ is explicitly excluded
https://github.com/apache/bookkeeper/blob/732b6cf2a6576c844a4b43176125e82d500a6ea8/bookkeeper-dist/src/assemble/src.xml#L61
- site2 is not excluded (should be fixed)
- gradle files are not explicitly included, pom are
https://github.com/apache/bookkeeper/blob/732b6cf2a6576c844a4b43176125e82d500a6ea8/bookkeeper-dist/src/assemble/src.xml#L33

that src.xml is used to package the release artifacts
https://github.com/apache/bookkeeper/blob/master/bookkeeper-dist/pom.xml#L55

It looks like the plugin resolves the situation with the file not matching
includes and excludes by excluding it.

PR that adds gradle into the includes patterns and excludes site2:
https://github.com/apache/bookkeeper/pull/2714

I can either rebuild the RC or we can decide to proceed.
Gradle was not used to build/release 4.14.0


On Thu, May 13, 2021 at 8:34 AM Andrey Yegorov 
wrote:

> I ran the release script, as described in the release procedure.
> It all ran from the docker container.
>
> On Wed, May 12, 2021 at 11:30 PM Henry Saputra 
> wrote:
>
>> Hi Andrey,
>>
>> How did you generate the source release artifact? Looks like it is missing
>> some files like the site directory and the gradle build files.
>>
>> - Henry
>>
>>
>>
>> On Sun, May 9, 2021 at 11:16 PM Andrey Yegorov <
>> andrey.yego...@datastax.com>
>> wrote:
>>
>> > Hi everyone,
>> > Please review and vote on the release candidate #0 for the version
>> 4.14.0,
>> > as follows:
>> > [ ] +1, Approve the release
>> > [ ] -1, Do not approve the release (please provide specific comments)
>> >
>> > The complete staging area is available for your review, which includes:
>> > * Release notes [1]
>> > * The official Apache source and binary distributions to be deployed to
>> > dist.apache.org [2]
>> > * All artifacts to be deployed to the Maven Central Repository [3]
>> > * Source code tag "release-4.14.0" [4] with git sha
>> > 4729682f00f05f23a1821211cbce064e653edb83
>> >
>> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
>> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>> >
>> > Please download these packages and review this release candidate:
>> >
>> > - Review release notes
>> > - Download the source package (verify shasum, and asc) and follow the
>> > instructions to build and run the bookkeeper service.
>> > - Download the binary package (verify shasum, and asc) and follow the
>> > instructions to run the bookkeeper service.
>> > - Review maven repo, release tag, licenses, and any other things you
>> think
>> > it is important to a release.
>> >
>> > The vote will be open for at least 72 hours. It is adopted by majority
>> > approval, with at least 3 PMC affirmative votes.
>> >
>> > Thanks,
>> > Release Manager
>> >
>> > [1] https://github.com/apache/bookkeeper/pull/2712
>> >
>> >
>> https://github.com/apache/bookkeeper/blob/4f88ae1bf271d025945437e8499955118000ce29/site/docs/4.14.0/overview/releaseNotes.md
>> >
>> > [2]
>> >
>> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.0-rc0/
>> >
>> > [3]
>> >
>> >
>> https://repository.apache.org/content/repositories/orgapachebookkeeper-1052/
>> >
>> > [4] https://github.com/apache/bookkeeper/commits/branch-4.14 @ sha
>> 4729682
>> >
>> > --
>> > Andrey Yegorov
>> >
>>
>
>
> --
>
> --
> Andrey Yegorov
>


-- 

--
Andrey Yegorov

Re: [VOTE] Release 4.14.0, release candidate #0

[Andrey Yegorov]

I ran the release script, as described in the release procedure.
It all ran from the docker container.

On Wed, May 12, 2021 at 11:30 PM Henry Saputra 
wrote:

> Hi Andrey,
>
> How did you generate the source release artifact? Looks like it is missing
> some files like the site directory and the gradle build files.
>
> - Henry
>
>
>
> On Sun, May 9, 2021 at 11:16 PM Andrey Yegorov <
> andrey.yego...@datastax.com>
> wrote:
>
> > Hi everyone,
> > Please review and vote on the release candidate #0 for the version
> 4.14.0,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "release-4.14.0" [4] with git sha
> > 4729682f00f05f23a1821211cbce064e653edb83
> >
> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review maven repo, release tag, licenses, and any other things you
> think
> > it is important to a release.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1] https://github.com/apache/bookkeeper/pull/2712
> >
> >
> https://github.com/apache/bookkeeper/blob/4f88ae1bf271d025945437e8499955118000ce29/site/docs/4.14.0/overview/releaseNotes.md
> >
> > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.0-rc0/
> >
> > [3]
> >
> >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1052/
> >
> > [4] https://github.com/apache/bookkeeper/commits/branch-4.14 @ sha
> 4729682
> >
> > --
> > Andrey Yegorov
> >
>


-- 

--
Andrey Yegorov

Re: Cutting BookKeeper 4.13.1 or 4.14.0 release?

[Andrey Yegorov]

I noticed that PRs are in so I built the RC0 while I had time.
Please vote on the corresponding thread.

On Fri, May 7, 2021 at 3:15 PM Andrey Yegorov 
wrote:

> Matteo,
>
> https://github.com/apache/bookkeeper/pull/2649 also a good candidate for
> this release (fits the compaction improvements theme), also has changes
> requested
>
> On Fri, May 7, 2021 at 3:01 PM Andrey Yegorov 
> wrote:
>
>> ok, I'll hold on on the release until these two are ready to merge
>>
>> Both PRs have changes requested by Enrico.
>> I'll postpone my attempt to build the release until Monday.
>>
>> If there are other PRs that *have* to be in 4.14 please move them back to
>> the 4.14 milestone and update this thread.
>>
>>
>> On Fri, May 7, 2021 at 1:06 PM Matteo Merli  wrote:
>>
>>> Andrey, there are several PRs that would be good to get into 4.14. We
>>> shouldn't just push everything out to 4.15.
>>>
>>> Just a couple of examples:
>>>  * https://github.com/apache/bookkeeper/pull/2710
>>>  * https://github.com/apache/bookkeeper/pull/2698
>>>
>>>
>>>
>>> --
>>> Matteo Merli
>>> 
>>>
>>> On Fri, May 7, 2021 at 12:55 PM Andrey Yegorov
>>>  wrote:
>>> >
>>> > I created https://github.com/apache/bookkeeper/pull/2712 with docs and
>>> > release notes update for the v 4.14.0
>>> > The most interesting part is the release notes:
>>> >
>>> https://github.com/apache/bookkeeper/blob/e3c5994c05c970e6343fa9b43d1e63bac6142e60/site/docs/4.14.0/overview/releaseNotes.md
>>> >
>>> > Some PRs missed milestones and/or release labels, probably merged
>>> manually.
>>> > I tracked changelists from git history and updated the
>>> labels/milestones.
>>> >
>>> > I'll start working on the release.
>>> >
>>> > On Fri, May 7, 2021 at 8:52 AM Andrey Yegorov <
>>> andrey.yego...@datastax.com>
>>> > wrote:
>>> >
>>> > > I added https://github.com/apache/bookkeeper/issues/2711 for the
>>> TLS 1.3
>>> > > support
>>> > > Unless someone objects in the next 30min, I'll merge Lari's PR.
>>> > > After that I'll start working on the BK 4.14.0 release.
>>> > >
>>> > > On Thu, May 6, 2021 at 1:25 AM Lari Hotari  wrote:
>>> > >
>>> > >> I can confirm that the PR checks pass after excluding TLSv1.3 from
>>> enabled
>>> > >> protocols:
>>> > >>
>>> > >>
>>> https://github.com/apache/bookkeeper/pull/2696/commits/6003a374d5aec30d7059a21e473ac91417b5cdc3
>>> > >>
>>> > >> There should be tests for both TLSv1.2 and TLSv1.3 because of the
>>> > >> differences in TLS handshake described in
>>> > >>
>>> > >>
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_a_62465859&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=0B1UvYMwy7dr9qtqFwQCfxUyrozUgZzbOshynTIaYUY&m=76JE79AuinlMNecD5DDFGgg-jXzCGZEh3PANpQOJUoE&s=iZz_eExfeElZI--ooxMmyMABWjailhDc7rKIAZNg59s&e=
>>> > >> .
>>> > >>
>>> > >> This also impacts some production code in Bookkeeper. The PR already
>>> > >> includes a change to catch SSLException instead of
>>> SSLHandshakeException (
>>> > >>
>>> > >>
>>> https://github.com/apache/bookkeeper/pull/2696/commits/fcbd707a633ed1b8cf8290cb5d70a3070e010196
>>> > >> ).
>>> > >> TLSv1.3 doesn't throw SSLHandshakeException for certificate issues
>>> because
>>> > >> of the differences in the protocols. This change should work for
>>> both
>>> > >> protocols, but we should have test coverage to ensure that.
>>> > >>
>>> > >> TLSv1.3 has been enabled by default since Netty 4.1.52.Final (when
>>> the JDK
>>> > >> contains TLSv1.3). TLSv1.3 support has been available in Java 8
>>> since
>>> > >> 8u262
>>> > >> .
>>> > >>
>>> > >> One of the remaining problems with TLSv1.3 support in BK is the
>>> state
>>> > >> machine and TLS counters in PerChannelBookieClient . It doesn't
>>> properly
>>> > >> model the way TLS 1.3 behaves. Currently there's a counter
>>> &g

[VOTE] Release 4.14.0, release candidate #0

[Andrey Yegorov]

Hi everyone,
Please review and vote on the release candidate #0 for the version 4.14.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* Release notes [1]
* The official Apache source and binary distributions to be deployed to
dist.apache.org [2]
* All artifacts to be deployed to the Maven Central Repository [3]
* Source code tag "release-4.14.0" [4] with git sha
4729682f00f05f23a1821211cbce064e653edb83

BookKeeper's KEYS file contains PGP keys we used to sign this release:
https://dist.apache.org/repos/dist/release/bookkeeper/KEYS

Please download these packages and review this release candidate:

- Review release notes
- Download the source package (verify shasum, and asc) and follow the
instructions to build and run the bookkeeper service.
- Download the binary package (verify shasum, and asc) and follow the
instructions to run the bookkeeper service.
- Review maven repo, release tag, licenses, and any other things you think
it is important to a release.

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1] https://github.com/apache/bookkeeper/pull/2712
https://github.com/apache/bookkeeper/blob/4f88ae1bf271d025945437e8499955118000ce29/site/docs/4.14.0/overview/releaseNotes.md

[2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.0-rc0/

[3]
https://repository.apache.org/content/repositories/orgapachebookkeeper-1052/

[4] https://github.com/apache/bookkeeper/commits/branch-4.14 @ sha 4729682

--
Andrey Yegorov

Re: Cutting BookKeeper 4.13.1 or 4.14.0 release?

[Andrey Yegorov]

Matteo,

https://github.com/apache/bookkeeper/pull/2649 also a good candidate for
this release (fits the compaction improvements theme), also has changes
requested

On Fri, May 7, 2021 at 3:01 PM Andrey Yegorov 
wrote:

> ok, I'll hold on on the release until these two are ready to merge
>
> Both PRs have changes requested by Enrico.
> I'll postpone my attempt to build the release until Monday.
>
> If there are other PRs that *have* to be in 4.14 please move them back to
> the 4.14 milestone and update this thread.
>
>
> On Fri, May 7, 2021 at 1:06 PM Matteo Merli  wrote:
>
>> Andrey, there are several PRs that would be good to get into 4.14. We
>> shouldn't just push everything out to 4.15.
>>
>> Just a couple of examples:
>>  * https://github.com/apache/bookkeeper/pull/2710
>>  * https://github.com/apache/bookkeeper/pull/2698
>>
>>
>>
>> --
>> Matteo Merli
>> 
>>
>> On Fri, May 7, 2021 at 12:55 PM Andrey Yegorov
>>  wrote:
>> >
>> > I created https://github.com/apache/bookkeeper/pull/2712 with docs and
>> > release notes update for the v 4.14.0
>> > The most interesting part is the release notes:
>> >
>> https://github.com/apache/bookkeeper/blob/e3c5994c05c970e6343fa9b43d1e63bac6142e60/site/docs/4.14.0/overview/releaseNotes.md
>> >
>> > Some PRs missed milestones and/or release labels, probably merged
>> manually.
>> > I tracked changelists from git history and updated the
>> labels/milestones.
>> >
>> > I'll start working on the release.
>> >
>> > On Fri, May 7, 2021 at 8:52 AM Andrey Yegorov <
>> andrey.yego...@datastax.com>
>> > wrote:
>> >
>> > > I added https://github.com/apache/bookkeeper/issues/2711 for the TLS
>> 1.3
>> > > support
>> > > Unless someone objects in the next 30min, I'll merge Lari's PR.
>> > > After that I'll start working on the BK 4.14.0 release.
>> > >
>> > > On Thu, May 6, 2021 at 1:25 AM Lari Hotari  wrote:
>> > >
>> > >> I can confirm that the PR checks pass after excluding TLSv1.3 from
>> enabled
>> > >> protocols:
>> > >>
>> > >>
>> https://github.com/apache/bookkeeper/pull/2696/commits/6003a374d5aec30d7059a21e473ac91417b5cdc3
>> > >>
>> > >> There should be tests for both TLSv1.2 and TLSv1.3 because of the
>> > >> differences in TLS handshake described in
>> > >>
>> > >>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_a_62465859&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=0B1UvYMwy7dr9qtqFwQCfxUyrozUgZzbOshynTIaYUY&m=76JE79AuinlMNecD5DDFGgg-jXzCGZEh3PANpQOJUoE&s=iZz_eExfeElZI--ooxMmyMABWjailhDc7rKIAZNg59s&e=
>> > >> .
>> > >>
>> > >> This also impacts some production code in Bookkeeper. The PR already
>> > >> includes a change to catch SSLException instead of
>> SSLHandshakeException (
>> > >>
>> > >>
>> https://github.com/apache/bookkeeper/pull/2696/commits/fcbd707a633ed1b8cf8290cb5d70a3070e010196
>> > >> ).
>> > >> TLSv1.3 doesn't throw SSLHandshakeException for certificate issues
>> because
>> > >> of the differences in the protocols. This change should work for both
>> > >> protocols, but we should have test coverage to ensure that.
>> > >>
>> > >> TLSv1.3 has been enabled by default since Netty 4.1.52.Final (when
>> the JDK
>> > >> contains TLSv1.3). TLSv1.3 support has been available in Java 8 since
>> > >> 8u262
>> > >> .
>> > >>
>> > >> One of the remaining problems with TLSv1.3 support in BK is the state
>> > >> machine and TLS counters in PerChannelBookieClient . It doesn't
>> properly
>> > >> model the way TLS 1.3 behaves. Currently there's a counter
>> > >> FAILED_TLS_HANDSHAKE_COUNTER  which is expected to count also the
>> > >> certificate issues (code:
>> > >>
>> > >>
>> https://github.com/apache/bookkeeper/blob/fcbd707a633ed1b8cf8290cb5d70a3070e010196/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java#L1535-L1543
>> > >> ). Since TLSv1.3 doesn't detect certificate issues (mutual TLS)
>> during
>> > >> handshake, this counter doesn't count certificate issues. Certificate
>> > >> i

Re: Cutting BookKeeper 4.13.1 or 4.14.0 release?

[Andrey Yegorov]

ok, I'll hold on on the release until these two are ready to merge

Both PRs have changes requested by Enrico.
I'll postpone my attempt to build the release until Monday.

If there are other PRs that *have* to be in 4.14 please move them back to
the 4.14 milestone and update this thread.


On Fri, May 7, 2021 at 1:06 PM Matteo Merli  wrote:

> Andrey, there are several PRs that would be good to get into 4.14. We
> shouldn't just push everything out to 4.15.
>
> Just a couple of examples:
>  * https://github.com/apache/bookkeeper/pull/2710
>  * https://github.com/apache/bookkeeper/pull/2698
>
>
>
> --
> Matteo Merli
> 
>
> On Fri, May 7, 2021 at 12:55 PM Andrey Yegorov
>  wrote:
> >
> > I created https://github.com/apache/bookkeeper/pull/2712 with docs and
> > release notes update for the v 4.14.0
> > The most interesting part is the release notes:
> >
> https://github.com/apache/bookkeeper/blob/e3c5994c05c970e6343fa9b43d1e63bac6142e60/site/docs/4.14.0/overview/releaseNotes.md
> >
> > Some PRs missed milestones and/or release labels, probably merged
> manually.
> > I tracked changelists from git history and updated the labels/milestones.
> >
> > I'll start working on the release.
> >
> > On Fri, May 7, 2021 at 8:52 AM Andrey Yegorov <
> andrey.yego...@datastax.com>
> > wrote:
> >
> > > I added https://github.com/apache/bookkeeper/issues/2711 for the TLS
> 1.3
> > > support
> > > Unless someone objects in the next 30min, I'll merge Lari's PR.
> > > After that I'll start working on the BK 4.14.0 release.
> > >
> > > On Thu, May 6, 2021 at 1:25 AM Lari Hotari  wrote:
> > >
> > >> I can confirm that the PR checks pass after excluding TLSv1.3 from
> enabled
> > >> protocols:
> > >>
> > >>
> https://github.com/apache/bookkeeper/pull/2696/commits/6003a374d5aec30d7059a21e473ac91417b5cdc3
> > >>
> > >> There should be tests for both TLSv1.2 and TLSv1.3 because of the
> > >> differences in TLS handshake described in
> > >>
> > >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_a_62465859&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=0B1UvYMwy7dr9qtqFwQCfxUyrozUgZzbOshynTIaYUY&m=76JE79AuinlMNecD5DDFGgg-jXzCGZEh3PANpQOJUoE&s=iZz_eExfeElZI--ooxMmyMABWjailhDc7rKIAZNg59s&e=
> > >> .
> > >>
> > >> This also impacts some production code in Bookkeeper. The PR already
> > >> includes a change to catch SSLException instead of
> SSLHandshakeException (
> > >>
> > >>
> https://github.com/apache/bookkeeper/pull/2696/commits/fcbd707a633ed1b8cf8290cb5d70a3070e010196
> > >> ).
> > >> TLSv1.3 doesn't throw SSLHandshakeException for certificate issues
> because
> > >> of the differences in the protocols. This change should work for both
> > >> protocols, but we should have test coverage to ensure that.
> > >>
> > >> TLSv1.3 has been enabled by default since Netty 4.1.52.Final (when
> the JDK
> > >> contains TLSv1.3). TLSv1.3 support has been available in Java 8 since
> > >> 8u262
> > >> .
> > >>
> > >> One of the remaining problems with TLSv1.3 support in BK is the state
> > >> machine and TLS counters in PerChannelBookieClient . It doesn't
> properly
> > >> model the way TLS 1.3 behaves. Currently there's a counter
> > >> FAILED_TLS_HANDSHAKE_COUNTER  which is expected to count also the
> > >> certificate issues (code:
> > >>
> > >>
> https://github.com/apache/bookkeeper/blob/fcbd707a633ed1b8cf8290cb5d70a3070e010196/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java#L1535-L1543
> > >> ). Since TLSv1.3 doesn't detect certificate issues (mutual TLS) during
> > >> handshake, this counter doesn't count certificate issues. Certificate
> > >> issues will show up as successfully established connections.
> > >> The original issue for adding TLS counters was
> > >> https://github.com/apache/bookkeeper/issues/1103 and PR commit was
> > >>
> > >>
> https://github.com/apache/bookkeeper/commit/fa10b7dcd89c40222ba5f30bb60f785bd21669b2
> > >> .
> > >>
> > >> How do we revisit the TLS counter solution for TLSv1.3 compatibility?
> Do
> > >> we
> > >> make changes to the code or do we simply skip the test on TLSv1.3
> which
> > >> e

Re: Cutting BookKeeper 4.13.1 or 4.14.0 release?

[Andrey Yegorov]

I created https://github.com/apache/bookkeeper/pull/2712 with docs and
release notes update for the v 4.14.0
The most interesting part is the release notes:
https://github.com/apache/bookkeeper/blob/e3c5994c05c970e6343fa9b43d1e63bac6142e60/site/docs/4.14.0/overview/releaseNotes.md

Some PRs missed milestones and/or release labels, probably merged manually.
I tracked changelists from git history and updated the labels/milestones.

I'll start working on the release.

On Fri, May 7, 2021 at 8:52 AM Andrey Yegorov 
wrote:

> I added https://github.com/apache/bookkeeper/issues/2711 for the TLS 1.3
> support
> Unless someone objects in the next 30min, I'll merge Lari's PR.
> After that I'll start working on the BK 4.14.0 release.
>
> On Thu, May 6, 2021 at 1:25 AM Lari Hotari  wrote:
>
>> I can confirm that the PR checks pass after excluding TLSv1.3 from enabled
>> protocols:
>>
>> https://github.com/apache/bookkeeper/pull/2696/commits/6003a374d5aec30d7059a21e473ac91417b5cdc3
>>
>> There should be tests for both TLSv1.2 and TLSv1.3 because of the
>> differences in TLS handshake described in
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_a_62465859&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=0B1UvYMwy7dr9qtqFwQCfxUyrozUgZzbOshynTIaYUY&m=76JE79AuinlMNecD5DDFGgg-jXzCGZEh3PANpQOJUoE&s=iZz_eExfeElZI--ooxMmyMABWjailhDc7rKIAZNg59s&e=
>> .
>>
>> This also impacts some production code in Bookkeeper. The PR already
>> includes a change to catch SSLException instead of SSLHandshakeException (
>>
>> https://github.com/apache/bookkeeper/pull/2696/commits/fcbd707a633ed1b8cf8290cb5d70a3070e010196
>> ).
>> TLSv1.3 doesn't throw SSLHandshakeException for certificate issues because
>> of the differences in the protocols. This change should work for both
>> protocols, but we should have test coverage to ensure that.
>>
>> TLSv1.3 has been enabled by default since Netty 4.1.52.Final (when the JDK
>> contains TLSv1.3). TLSv1.3 support has been available in Java 8 since
>> 8u262
>> .
>>
>> One of the remaining problems with TLSv1.3 support in BK is the state
>> machine and TLS counters in PerChannelBookieClient . It doesn't properly
>> model the way TLS 1.3 behaves. Currently there's a counter
>> FAILED_TLS_HANDSHAKE_COUNTER  which is expected to count also the
>> certificate issues (code:
>>
>> https://github.com/apache/bookkeeper/blob/fcbd707a633ed1b8cf8290cb5d70a3070e010196/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java#L1535-L1543
>> ). Since TLSv1.3 doesn't detect certificate issues (mutual TLS) during
>> handshake, this counter doesn't count certificate issues. Certificate
>> issues will show up as successfully established connections.
>> The original issue for adding TLS counters was
>> https://github.com/apache/bookkeeper/issues/1103 and PR commit was
>>
>> https://github.com/apache/bookkeeper/commit/fa10b7dcd89c40222ba5f30bb60f785bd21669b2
>> .
>>
>> How do we revisit the TLS counter solution for TLSv1.3 compatibility? Do
>> we
>> make changes to the code or do we simply skip the test on TLSv1.3 which
>> ensures that a certificate issue is counted in
>> FAILED_TLS_HANDSHAKE_COUNTER
>> ? Skipping the test for TLSv1.3 would be one option. WDYT?
>>
>> -Lari
>>
>> On Wed, May 5, 2021 at 10:32 PM Andrey Yegorov <
>> andrey.yego...@datastax.com>
>> wrote:
>>
>> > Lari and I have looked at the Netty upgrade.
>> > There are some test breaks, and so far everything is related to behavior
>> > changes related to TLS 1.3, see
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_a_62465859&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=0B1UvYMwy7dr9qtqFwQCfxUyrozUgZzbOshynTIaYUY&m=76JE79AuinlMNecD5DDFGgg-jXzCGZEh3PANpQOJUoE&s=iZz_eExfeElZI--ooxMmyMABWjailhDc7rKIAZNg59s&e=
>> > We managed to fix some of the issues
>> > https://github.com/apache/bookkeeper/pull/2696 but "the client won't
>> know
>> > whether the server has accepted the certificate or not until it next
>> reads
>> > data from the server" complicates things.
>> >
>> > Currently we are considering simply setting "java
>> > -Djdk.tls.client.protocols=TLSv1.2" to unbreak the tests and handling
>> > tls1.3 as a separate work item.
>> >
>> > Lari is planning on spending a little bit more time on this tomorrow
>> (his
>> > tomorrow) to see if there is a better way to address this quickly; we'll

Re: Cutting BookKeeper 4.13.1 or 4.14.0 release?

[Andrey Yegorov]

I added https://github.com/apache/bookkeeper/issues/2711 for the TLS 1.3
support
Unless someone objects in the next 30min, I'll merge Lari's PR.
After that I'll start working on the BK 4.14.0 release.

On Thu, May 6, 2021 at 1:25 AM Lari Hotari  wrote:

> I can confirm that the PR checks pass after excluding TLSv1.3 from enabled
> protocols:
>
> https://github.com/apache/bookkeeper/pull/2696/commits/6003a374d5aec30d7059a21e473ac91417b5cdc3
>
> There should be tests for both TLSv1.2 and TLSv1.3 because of the
> differences in TLS handshake described in
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_a_62465859&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=0B1UvYMwy7dr9qtqFwQCfxUyrozUgZzbOshynTIaYUY&m=76JE79AuinlMNecD5DDFGgg-jXzCGZEh3PANpQOJUoE&s=iZz_eExfeElZI--ooxMmyMABWjailhDc7rKIAZNg59s&e=
> .
>
> This also impacts some production code in Bookkeeper. The PR already
> includes a change to catch SSLException instead of SSLHandshakeException (
>
> https://github.com/apache/bookkeeper/pull/2696/commits/fcbd707a633ed1b8cf8290cb5d70a3070e010196
> ).
> TLSv1.3 doesn't throw SSLHandshakeException for certificate issues because
> of the differences in the protocols. This change should work for both
> protocols, but we should have test coverage to ensure that.
>
> TLSv1.3 has been enabled by default since Netty 4.1.52.Final (when the JDK
> contains TLSv1.3). TLSv1.3 support has been available in Java 8 since 8u262
> .
>
> One of the remaining problems with TLSv1.3 support in BK is the state
> machine and TLS counters in PerChannelBookieClient . It doesn't properly
> model the way TLS 1.3 behaves. Currently there's a counter
> FAILED_TLS_HANDSHAKE_COUNTER  which is expected to count also the
> certificate issues (code:
>
> https://github.com/apache/bookkeeper/blob/fcbd707a633ed1b8cf8290cb5d70a3070e010196/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java#L1535-L1543
> ). Since TLSv1.3 doesn't detect certificate issues (mutual TLS) during
> handshake, this counter doesn't count certificate issues. Certificate
> issues will show up as successfully established connections.
> The original issue for adding TLS counters was
> https://github.com/apache/bookkeeper/issues/1103 and PR commit was
>
> https://github.com/apache/bookkeeper/commit/fa10b7dcd89c40222ba5f30bb60f785bd21669b2
> .
>
> How do we revisit the TLS counter solution for TLSv1.3 compatibility? Do we
> make changes to the code or do we simply skip the test on TLSv1.3 which
> ensures that a certificate issue is counted in FAILED_TLS_HANDSHAKE_COUNTER
> ? Skipping the test for TLSv1.3 would be one option. WDYT?
>
> -Lari
>
> On Wed, May 5, 2021 at 10:32 PM Andrey Yegorov <
> andrey.yego...@datastax.com>
> wrote:
>
> > Lari and I have looked at the Netty upgrade.
> > There are some test breaks, and so far everything is related to behavior
> > changes related to TLS 1.3, see
> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_a_62465859&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=0B1UvYMwy7dr9qtqFwQCfxUyrozUgZzbOshynTIaYUY&m=76JE79AuinlMNecD5DDFGgg-jXzCGZEh3PANpQOJUoE&s=iZz_eExfeElZI--ooxMmyMABWjailhDc7rKIAZNg59s&e=
> > We managed to fix some of the issues
> > https://github.com/apache/bookkeeper/pull/2696 but "the client won't
> know
> > whether the server has accepted the certificate or not until it next
> reads
> > data from the server" complicates things.
> >
> > Currently we are considering simply setting "java
> > -Djdk.tls.client.protocols=TLSv1.2" to unbreak the tests and handling
> > tls1.3 as a separate work item.
> >
> > Lari is planning on spending a little bit more time on this tomorrow (his
> > tomorrow) to see if there is a better way to address this quickly; we'll
> > hear more then.
> >
> >
> > On Wed, May 5, 2021 at 9:23 AM Henry Saputra 
> > wrote:
> >
> > > I am +1 for having next release as 4.14.0
> > >
> > > - Henry
> > >
> > > On Tue, May 4, 2021 at 2:51 PM Andrey Yegorov <
> > andrey.yego...@datastax.com
> > > >
> > > wrote:
> > >
> > > > Overall +1 for 4.14.0 - the milestone is due May 16th anyway.
> > > > There is nothing that breaks compatibility with 4.13 so we can skip
> > > 4.13.1.
> > > >
> > > > One thing I'd love to see in 4.14 is
> > > > https://github.com/apache/bookkeeper/pull/2696 , to fix
> > > > https://github.com/netty/netty/issues/10986
> > > > It looks like there are issues with

Re: Cutting BookKeeper 4.13.1 or 4.14.0 release?

[Andrey Yegorov]

Lari and I have looked at the Netty upgrade.
There are some test breaks, and so far everything is related to behavior
changes related to TLS 1.3, see https://stackoverflow.com/a/62465859
We managed to fix some of the issues
https://github.com/apache/bookkeeper/pull/2696 but "the client won't know
whether the server has accepted the certificate or not until it next reads
data from the server" complicates things.

Currently we are considering simply setting "java
-Djdk.tls.client.protocols=TLSv1.2" to unbreak the tests and handling
tls1.3 as a separate work item.

Lari is planning on spending a little bit more time on this tomorrow (his
tomorrow) to see if there is a better way to address this quickly; we'll
hear more then.


On Wed, May 5, 2021 at 9:23 AM Henry Saputra 
wrote:

> I am +1 for having next release as 4.14.0
>
> - Henry
>
> On Tue, May 4, 2021 at 2:51 PM Andrey Yegorov  >
> wrote:
>
> > Overall +1 for 4.14.0 - the milestone is due May 16th anyway.
> > There is nothing that breaks compatibility with 4.13 so we can skip
> 4.13.1.
> >
> > One thing I'd love to see in 4.14 is
> > https://github.com/apache/bookkeeper/pull/2696 , to fix
> > https://github.com/netty/netty/issues/10986
> > It looks like there are issues with vertx
> > https://github.com/apache/bookkeeper/pull/2693#issuecomment-823774769
> > I hope we can upgrade to latest vertx 3.9.7 and netty 4.1.60
> >
> > Also https://github.com/apache/bookkeeper/pull/2695 upgrades libthrift
> to
> > address security issues.
> >
> > All these PRs are from Lari, I'll follow up with him.
> >
> > I can be a RM if there are no other volunteers.
> >
> >
> > On Tue, May 4, 2021 at 1:57 PM Matteo Merli  wrote:
> >
> > > +1 We should do 4.14, carrying all the fixes that are there in master
> as
> > > well.
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> > > On Tue, May 4, 2021 at 11:57 AM Sijie Guo  wrote:
> > > >
> > > > +1
> > > >
> > > > On Tue, May 4, 2021 at 2:22 AM Yunze Xu  >
> > > > wrote:
> > > >
> > > > > Hello,
> > > > > About 10 days ago I found a heap memory copy problem in Apache
> > Pulsar,
> > > see
> > > > > [1].
> > > > > It’s a problem of BK side because when `LedgerHandle#asyncAddEntry`
> > > > > accepts a `CompositeByteBuf` or a wrapper, it will finally call
> > > > > `ByteBuf#nioBuffer()`, which would make a heap copy from direct
> > memory.
> > > > > [2] fixed this problem and has been merged for a week.
> > > > >
> > > > > Since it has a significant impact on Pulsar, Pulsar side needs a
> new
> > BK
> > > > > release with [2] merged to fix it. Is there any plan to cut a
> 4.13.1
> > > > > release or 4.14.0 release so that we can upgrade the dependency in
> > > Pulsar?
> > > > >
> > > > > Thanks,
> > > > > Yunze
> > > > >
> > > > > [1] https://github.com/apache/pulsar/pull/10330 <
> > > > > https://github.com/apache/pulsar/pull/10330>
> > > > >
> > > > > [2] https://github.com/apache/bookkeeper/pull/2701 <
> > > > > https://github.com/apache/bookkeeper/pull/2701>
> > > > >
> > > > >
> > >
> >
> >
> > --
> >
> > --
> > Andrey Yegorov
> >
>


-- 

--
Andrey Yegorov