Re: [DISCUSS] Proposal to Backport `dbStorage_readAheadCacheBatchBytesSize` to BookKeeper 4.16.x

[Matteo Merli]

Sounds good to me. This won't add any new risk.
--
Matteo Merli



On Thu, Jun 27, 2024 at 4:30 PM Hang Chen  wrote:

> Hi team,
>
> I would like to discuss the potential backporting of a recently
> introduced feature from the BookKeeper 4.17.0 release to the 4.16.x
> branch, which is used in Pulsar 3.0 LTS. The feature in question is
> the `dbStorage_readAheadCacheBatchBytesSize` configuration option,
> added through PR [1]. This configuration limits the prefetch size of
> the server read cache from the ledger disk.
>
> Upon testing, I found that this new configuration significantly
> improves catch-up read latency. Given that Pulsar 3.0 is a Long-Term
> Support (LTS) version and relies on BookKeeper 4.16.x, incorporating
> this feature could provide substantial benefits to users of this
> version.
>
> Importantly, this configuration is disabled by default, ensuring that
> its introduction to the 4.16.x branch would not lead to breaking
> changes. This makes it a safe candidate for cherry-picking to
> branch-4.16.
>
> I believe this enhancement will provide a tangible improvement to the
> performance of Pulsar 3.0 LTS. I would appreciate any feedback or
> concerns regarding this proposal.
>
> Thank you for considering this improvement.
>
> Best regards,
> Hang
>
>
>
> [1] https://github.com/apache/bookkeeper/pull/3895
>

Re: [VOTE] Release 4.16.0, release candidate #2

[Matteo Merli]

+1 binding

* Checked signatures
* Check localbookie and local perf tool



--
Matteo Merli



On Wed, Mar 29, 2023 at 7:06 AM Hang Chen  wrote:

> Hi everyone,
> Please review and vote on the release candidate #2 for version 4.16.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.16.0-rc2" [4] with git sha [5]
>
> BookKeeper's KEYS file contains the PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review the maven repo, release tag, licenses, and any other things you
> think
> it is important to release.
>
> The vote will be open for at least 72 hours. It is adopted by the majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/bookkeeper/pull/3882/files
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.16.0-rc2/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1083/
> [4] https://github.com/apache/bookkeeper/tree/v4.16.0-rc2
> [5] 8890a61418e3961e108b3c2e8ea176691e5f
>

Re: [VOTE] BP-62: Streamline batch add requests

[Matteo Merli]

+1
--
Matteo Merli



On Mon, Mar 20, 2023 at 6:05 AM Hang Chen  wrote:

> This is the vote for BP-62.
>
> ### Motivation
> The bookie server process add-entry requests pipeline:
> - Get one request from the Netty socket channel
> - Choose one thread to process the written request
> - Write the entry into the target ledger entry logger's write
> cache(memory table)
> - Put the entry into the journal's pending queue
> - Journal thread takes the entry from the pending queue and writes it
> into PageCache/Journal Disk
> - Write the callback response to the Netty buffer and flush to the
> bookie client side.
>
> For every add entry request, the bookie server needs to go through the
> above steps one by one. It will introduce a lot of thread context
> switches.
>
> We can batch the add requests according to the Netty socket channel,
> and write a batch of entries into the ledger entry logger and journal
> disk.
>
> ### Modifications
> The PR will change the add requests processing pipeline into the
> following steps.
> - Get a batch of add-entry requests from the socket channel until the
> socket channel is empty or reached the max capacity (default is 1_000)
> - Choose one thread to process the batch of add-entry requests.
> - Write the entries into the target ledger entry logger's write cache one
> by one
> - Put the batch of entries into the journal's pending queue
> - Journal thread drains a batch of entries from the pending queue and
> writes them into PageCache/Journal disk
> - Write the callback response to the Netty buffer and flush to the
> bookie client side.
>
> With this change, we can save a lot of thread context switches.
>
>
> ### Performance
> I start one bookie on my laptop and use the Bookkeeper benchmark to
> test the performance
> ```shell
> bin/benchmark writes -ensemble 1 -quorum 1 -ackQuorum 1 -ledgers 50
> -throttle 2
> ```
>
> **Before this change**
>
> | times | ops/sec | p95 latency | p99 latency |
> | --- | --- | --- | --- |
> | 1 | 147507 | 114.93 | 122.42 |
> | 2 | 154571 |  111.46 | 115.86 |
> | 3 | 141459 | 117.23 | 124.18 |
> | 4 | 142037 | 121.75 | 128.54 |
> | 5 | 143682 | 121.05 | 127.97 |
>
>
> **After this change**
>
> | times | ops/sec | p95 latency | p99 latency |
> | --- | --- | --- | --- |
> | 1 | 157328 | 118.30 | 121.79 |
> | 2 | 165774 |  112.86 | 115.69 |
> | 3 | 144790 | 128.94 | 133.24 |
> | 4 | 151984 | 121.88 | 125.32 |
> | 5 | 154574 | 121.57 | 124.57 |
>
> The new change has a 2.2% improvement.
>
> Please leave +1/-1 in this thread to join the vote. And feel free to
> leave any concerns.
>
> Thanks,
> Hang
>

Re: [VOTE] BP-57 Add a public API for creating embeddable servers

[Matteo Merli]

+1 (binding)

Very good addition!


--
Matteo Merli


On Tue, Oct 11, 2022 at 4:38 AM Enrico Olivelli  wrote:
>
> Up
>
> We need 1 more binding vote from a PMC member
>
> Enrico
>
> Il Mar 27 Set 2022, 11:36 Nicolò Boschi  ha scritto:
>
> > +1 (non binding)
> > Nicolò Boschi
> >
> >
> > Il giorno mar 27 set 2022 alle ore 11:25 Enrico Olivelli <
> > eolive...@gmail.com> ha scritto:
> >
> > > I think that we need some more binding VOTEs please
> > >
> > > Enrico
> > >
> > > Il giorno dom 25 set 2022 alle ore 16:15 ZhangJian He
> > >  ha scritto:
> > > >
> > > > +1 , non binding
> > > >
> > > > Thanks
> > > > ZhangJian He
> > > >
> > > > On Sun, 25 Sept 2022 at 21:56, Lan Liang 
> > > wrote:
> > > >
> > > > > +1 , non binding.
> > > > >
> > > > > This is a useful feature!
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Best Regards,
> > > > > Lan Liang
> > > > >  Replied Message 
> > > > > | From | steven lu |
> > > > > | Date | 9/25/2022 08:25 |
> > > > > | To |  |
> > > > > | Subject | Re: [VOTE] BP-57 Add a public API for creating embeddable
> > > > > servers |
> > > > > +1 (non binding)
> > > > > Great work
> > > > >
> > > > > Diego Salvi  于2022年9月24日周六 00:00写道：
> > > > >
> > > > > Hi Bookkeeper Community,
> > > > > I would like to start a VOTE on " Add a public API for creating
> > > embeddable
> > > > > servers."(BP-57)
> > > > >
> > > > > It is a new public API to create bookie from configuration and/or
> > > custom
> > > > > implementations without having to deep dive into internal structure.
> > > The
> > > > > code will be the same used to build standard BK server through Main
> > > class
> > > > >
> > > > > Here is the design detail:
> > > > > https://github.com/apache/bookkeeper/issues/3494
> > > > >
> > > > > Diego Salvi
> > > > >
> > > > >
> > >
> >

Re: [Discuss] Optimize write cache: ①Two write caches become multi-cache blocks; ②All data directories share write cache resources;

[Matteo Merli]

The reason for having 2 block is on purpose to keep the entries in the
cache for the longest possible time. That will maximize the number of
entries for the same ledger that will be stored sequentially on the disk
and it will increase the read performance.

On Wed, Jun 8, 2022 at 12:05 AM lordcheng10 <1572139...@qq.com.invalid>
wrote:

> The current write cache has only 2 blocks, which has the following
> deficiencies:
>1.Insufficient use of cache: When half of the write
> cache is flushed, it must wait for all the data in the write cache to be
> flushed before it can be released. With the same write cache size, if we
> divide the write cache into multiple blocks, the cache resources can be
> released faster, preventing the write request from waiting too long for the
> cache resources to be released;
>
>2.The current write cache is allocated
> according to the configured data directory, so that the write cache
> resources between directories cannot be shared. When the write cache
> resources corresponding to one directory continue to be full, it is
> possible that the cache resources of other directories are still relatively
> low. Idle, so that the full utilization of the write cache is not used;
>
> So I made the following changes:
> 1.Two write caches become multi-cache blocks;
> 2.All data directories share write cache resources;
>
> For BP and PR :
> https://github.com/apache/bookkeeper/issues/3322
> https://github.com/apache/bookkeeper/pull/3260

-- 
--
Matteo Merli

Re: [VOTE] Release 4.15.0, release candidate #2

[Matteo Merli]

+1

Verified:
 * Checksums
 * Building
 * Starting standalone bookie

Thanks,
Matteo


--
Matteo Merli


On Wed, May 4, 2022 at 8:53 AM Andrey Yegorov
 wrote:
>
> Hi everyone,
>
> The vote has been out for 5 days - any feedback?
>
> On Fri, Apr 29, 2022 at 2:08 PM Andrey Yegorov 
> wrote:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version 4.15.0,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "release-4.15.0-rc2" [4] with git
> > sha f771b7664f7f83ae4ae3e8eaa516c879c4bc7897
> >
> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review maven repo, release tag, licenses, and any other things you think
> > it is important to a release.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1]
> > https://github.com/apache/bookkeeper/pull/3155/files#diff-eff52edf7022fd953f60c68061de09ec5d148eb37893de6af85a7964a2e57855
> >
> > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.15.0-rc2/
> > [3]
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1066/
> > [4] https://github.com/apache/bookkeeper/releases/tag/v4.15.0-rc2/
> >
> >
> > --
> > Andrey Yegorov
> >
>
>
> --
> Andrey Yegorov

[RESULT] [VOTE] Release 4.14.4, release candidate #0

[Matteo Merli]

Closing this vote with 3 binding and 2 non-binding +1s :

Binding +1s:
 * Matteo
 * Enrico
 * Jia

Non-Binding +1s:
 * Nicolò
 * Andrey


Thanks everyone,
Matteo


--
Matteo Merli


On Wed, Jan 5, 2022 at 5:16 AM Jia Zhai  wrote:
>
> +1 (binding)
>
> - verified packages checksum and signatures, they look good.
>
> - the source package build and test all run successfully.
>
> - in both binary packages(server & all), 'bin/bookkeeper standalone' and
> 'bin/bookkeeper shell bookiesanity' runs well.
>
> On Wed, Jan 5, 2022 at 6:58 AM Andrey Yegorov 
> wrote:
>
> > +1, ship it
> >
> > * checked localbookie starts/works
> > * spot-checked artifacts on maven central
> > * checked release tag/sha
> >
> >
> > On Tue, Jan 4, 2022 at 7:23 AM Dave Fisher  wrote:
> >
> > > It looks like this VOTE needs one more binding +1 ...
> > >
> > > > On Dec 23, 2021, at 11:14 PM, Enrico Olivelli 
> > > wrote:
> > > >
> > > > +1 (binding)
> > > > Built from source on JDK8
> > > > Verified signatures/md5
> > > > Run LocalBookie, some quick tests
> > > > Checked git log
> > > >
> > > > Please don't forget about the release notes!
> > > >
> > > > Enrico
> > > >
> > > > Il giorno gio 23 dic 2021 alle ore 11:26 Enrico Olivelli <
> > > > eolive...@gmail.com> ha scritto:
> > > >
> > > >> Matteo,
> > > >> Are you preparing the release notes ?
> > > >> Which are the contents of the release ?
> > > >>
> > > >> it looks like we are cutting this release with some dependency
> > updates,
> > > >> Netty and GRPC, and with some fixes about the Metadata Driver
> > > >>
> > > >> This is the git log from GitHub
> > > >> https://github.com/apache/bookkeeper/commits/branch-4.14
> > > >>
> > > >> Enrico
> > > >>
> > > >> Il giorno gio 23 dic 2021 alle ore 09:48 Nicolò Boschi <
> > > >> boschi1...@gmail.com> ha scritto:
> > > >>
> > > >>> +1 (non binding)
> > > >>>
> > > >>> Verified the followings:
> > > >>> - checksum and signatures
> > > >>> - build from sources w jdk11
> > > >>> - ledgers write/read with "localbookie" command
> > > >>> - checked netty and grpc jars in the "lib" directory
> > > >>>
> > > >>> Il giorno mer 22 dic 2021 alle ore 18:39 Matteo Merli <
> > > mme...@apache.org>
> > > >>> ha scritto:
> > > >>>
> > > >>>> Hi everyone,
> > > >>>>
> > > >>>> Please review and vote on the release candidate #0 for the version
> > > >>>> 4.14.4, as follows:
> > > >>>> [ ] +1, Approve the release
> > > >>>> [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > >>>>
> > > >>>> The complete staging area is available for your review, which
> > > includes:
> > > >>>> * The official Apache source and binary distributions to be deployed
> > > >>>> to
> > > >>>>
> > > >>>
> > > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.4-rc0/
> > > >>>> * All artifacts to be deployed to the Maven Central Repository:
> > > >>>>
> > > >>>
> > >
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1061
> > > >>>> * Source code tag "v4.14.4-rc0" with git sha
> > > >>>> 2071149af5c7fb0c905efaa84c7dcd911fc5dca0
> > > >>>>
> > > >>>> BookKeeper's KEYS file contains PGP keys we used to sign this
> > release:
> > > >>>> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > >>>>
> > > >>>> Please download these packages and review this release candidate:
> > > >>>>
> > > >>>> - Download the source package (verify shasum, and asc) and follow
> > the
> > > >>>> instructions to build and run the bookkeeper service.
> > > >>>> - Download the binary package (verify shasum, and asc) and follow
> > the
> > > >>>> instructions to run the bookkeeper service.
> > > >>>> - Review maven repo, release tag, licenses, and any other things you
> > > >>>> think are important to a release.
> > > >>>>
> > > >>>> The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > >>>> approval, with at least 3 PMC affirmative votes.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Matteo
> > > >>>>
> > > >>>> --
> > > >>>> Matteo Merli
> > > >>>> 
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Nicolò Boschi
> > > >>>
> > > >>
> > >
> > >
> >
> > --
> > Andrey Yegorov
> >

Re: [VOTE] Release 4.14.4, release candidate #0

[Matteo Merli]

On Thu, Dec 23, 2021 at 2:27 AM Enrico Olivelli  wrote:
>
> Matteo,
> Are you preparing the release notes ?
> Which are the contents of the release ?

Added PR for release notes:
https://github.com/apache/bookkeeper/pull/2986

[VOTE] Release 4.14.4, release candidate #0

[Matteo Merli]

Hi everyone,

Please review and vote on the release candidate #0 for the version
4.14.4, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* The official Apache source and binary distributions to be deployed
to https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.4-rc0/
* All artifacts to be deployed to the Maven Central Repository:
https://repository.apache.org/content/repositories/orgapachebookkeeper-1061
* Source code tag "v4.14.4-rc0" with git sha
2071149af5c7fb0c905efaa84c7dcd911fc5dca0

BookKeeper's KEYS file contains PGP keys we used to sign this release:
https://dist.apache.org/repos/dist/release/bookkeeper/KEYS

Please download these packages and review this release candidate:

- Download the source package (verify shasum, and asc) and follow the
instructions to build and run the bookkeeper service.
- Download the binary package (verify shasum, and asc) and follow the
instructions to run the bookkeeper service.
- Review maven repo, release tag, licenses, and any other things you
think are important to a release.

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Matteo

--
Matteo Merli

Re: [DISCUSS] Bookkeeper 4.14.3 release

[Matteo Merli]

On Thu, Dec 16, 2021 at 10:18 PM Enrico Olivelli  wrote:
>
> +1
>
> Especially if we have to deliver the upgrade of log4j2

BK 4.14 is including log4j 1.2.x, which has other vulnerabilities,
though not the log4shell one.

[DISCUSS] Bookkeeper 4.14.3 release

[Matteo Merli]

There are few fixes in the 4.14 that would be good to release soon.

Given that the timeline for 4.15 can be quite long, I'd propose to
release one more patch release right now.

Let me know if there are other fixes to include.

Matteo


--
Matteo Merli

Re: [VOTE] Release 4.14.3 release candidate #2

[Matteo Merli]

+1

Verified:
 * Checksums
 * Building
 * Starting standalone bookie


--
Matteo Merli


On Fri, Nov 5, 2021 at 4:26 AM Enrico Olivelli  wrote:
>
> +1 (binding)
>
> - verified signatures and digests
> - built from source, on JDK8 + Ubuntu
> - verified Apache RAT check
> - run single machine cluster smoke tests
> - run unit tests
>
>
> Some notes:
> - tests do no pass on JDK11, we are already working on this
> - I found one flaky test, this is a fix
> https://github.com/apache/bookkeeper/pull/2881
>
> Thank you very much for being the Release Manager Yong!
>
> Enrico Olivelli
>
>
> Il giorno mar 2 nov 2021 alle ore 11:14 Yong Zhang  ha
> scritto:
>
> > Hi everyone,
> > Please review and vote on the release candidate #0 for the version 4.14.3,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "v4.14.3-rc2" [4] with git sha [5]
> >
> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review maven repo, release tag, licenses, and any other things you think
> > it is important to a release.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Release Manager
> >
> > [1]
> >
> > https://github.com/apache/bookkeeper/pull/2864/files#diff-5405f1e88785133938fb03ee37380c8df913b3cbaec94a2d9bfcda1f8bd57cd5
> > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.3-rc2/
> > [3]
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1060
> > [4] https://github.com/apache/bookkeeper/releases/tag/v4.14.3-rc2
> > [5] 32f9cd7d5ce73cc0f29c7e66ce5adc0e6640d222
> >

Re: [VOTE] Release 4.14.3 release candidate #1

[Matteo Merli]

+1

Verified:
 * Checksums
 * Building
 * Starting standalone bookie


Matteo
--
Matteo Merli


On Mon, Nov 1, 2021 at 8:42 AM Matteo Merli  wrote:
>
> I think if the issue with Java 11 is in that test, we shouldn't block
> the release as this was probably already there before.
>
> We should instead make sure the BK CI is using Java 11 to run the
> tests, to avoid this kind of issue.
>
> Matteo
>
> --
> Matteo Merli
> 
>
> On Mon, Nov 1, 2021 at 7:08 AM Yong Zhang  wrote:
> >
> > Hi Enrico, I reproduce this issue locally when using jdk11.
> >
> > Then I can add this options ` --add-opens
> > java.base/jdk.internal.loader=ALL-UNNAMED`
> > in the pom and the BookieJournalTest can pass.
> >
> >
> >
> > On Mon, 1 Nov 2021 at 15:46, Enrico Olivelli  wrote:
> >
> > > Il Lun 1 Nov 2021, 05:04 Yong Zhang  ha
> > > scritto:
> > >
> > > > It looks all tests are passed on the CI with JDK1.8
> > > > https://github.com/apache/bookkeeper/tree/v4.14.3-rc1
> > >
> > >
> > > I am talking about JDK11
> > > In Pulsar we run by default on jdk11
> > > Having those tests failing on JDK11
> > > isn't very good.
> > > I will rerun my tests on jdk8
> > >
> > > Can you please double check on jdk11?
> > >
> > > Enrico
> > >
> > > >
> > > >
> > > > So it shouldn't be a block for releasing 4.14.3, right?
> > > >
> > > > On Mon, 1 Nov 2021 at 11:23, Yong Zhang 
> > > > wrote:
> > > >
> > > > > Thanks Enrico. I will take a look and test it locally.
> > > > >
> > > > > On Sat, 30 Oct 2021 at 00:20, Enrico Olivelli 
> > > > wrote:
> > > > >
> > > > >> Yong
> > > > >> I am not able so see some tests passing on JDK11
> > > > >>
> > > > >> I created this issue
> > > > >> https://github.com/apache/bookkeeper/issues/2872
> > > > >>
> > > > >> probably there is some library missing in the classpath ?
> > > > >>
> > > > >> Enrico
> > > > >>
> > > > >>
> > > > >> Il giorno gio 28 ott 2021 alle ore 14:52 Nicolò Boschi <
> > > > >> boschi1...@gmail.com>
> > > > >> ha scritto:
> > > > >>
> > > > >> > +1 (non binding)
> > > > >> >
> > > > >> > Verified the followings:
> > > > >> > - checksum and signatures
> > > > >> > - build from sources
> > > > >> > - ledgers write/read with "localbookie 3" and "standalone"
> > > > >> >
> > > > >> > Il giorno gio 28 ott 2021 alle ore 09:06 Yong Zhang <
> > > y...@apache.org>
> > > > >> ha
> > > > >> > scritto:
> > > > >> >
> > > > >> > > Hi everyone,
> > > > >> > > Please review and vote on the release candidate #0 for the 
> > > > >> > > version
> > > > >> > 4.14.3,
> > > > >> > > as follows:
> > > > >> > > [ ] +1, Approve the release
> > > > >> > > [ ] -1, Do not approve the release (please provide specific
> > > > comments)
> > > > >> > >
> > > > >> > > The complete staging area is available for your review, which
> > > > >> includes:
> > > > >> > > * Release notes [1]
> > > > >> > > * The official Apache source and binary distributions to be
> > > deployed
> > > > >> to
> > > > >> > > dist.apache.org [2]
> > > > >> > > * All artifacts to be deployed to the Maven Central Repository 
> > > > >> > > [3]
> > > > >> > > * Source code tag "v4.14.3-rc1" [4] with git sha [5]
> > > > >> > >
> > > > >> > > BookKeeper's KEYS file contains PGP keys we used to sign this
> > > > release:
> > > > >> > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > > >> > >
> > > > >> > > Please download these packages and review this release candidate:
> > > > >> > >
> > > > >> > > - Review release notes
> > > > >> > > - Download the source package (verify shasum, and asc) and follow
> > > > the
> > > > >> > > instructions to build and run the bookkeeper service.
> > > > >> > > - Download the binary package (verify shasum, and asc) and follow
> > > > the
> > > > >> > > instructions to run the bookkeeper service.
> > > > >> > > - Review maven repo, release tag, licenses, and any other things
> > > you
> > > > >> > think
> > > > >> > > it is important to a release.
> > > > >> > >
> > > > >> > > The vote will be open for at least 72 hours. It is adopted by
> > > > majority
> > > > >> > > approval, with at least 3 PMC affirmative votes.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Release Manager
> > > > >> > >
> > > > >> > > [1]
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > > https://github.com/apache/bookkeeper/pull/2864/files#diff-5405f1e88785133938fb03ee37380c8df913b3cbaec94a2d9bfcda1f8bd57cd5
> > > > >> > > [2]
> > > > >> > >
> > > > >>
> > > > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.3-rc1/
> > > > >> > > [3]
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > > https://repository.apache.org/content/repositories/orgapachebookkeeper-1059
> > > > >> > > [4] https://github.com/apache/bookkeeper/releases/tag/v4.14.3-rc1
> > > > >> > > [5] 11913dd7e261524427b4600a0fca970e58cc7f3b
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > Nicolò Boschi
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >

Re: [VOTE] Release 4.14.3 release candidate #1

[Matteo Merli]

I think if the issue with Java 11 is in that test, we shouldn't block
the release as this was probably already there before.

We should instead make sure the BK CI is using Java 11 to run the
tests, to avoid this kind of issue.

Matteo

--
Matteo Merli


On Mon, Nov 1, 2021 at 7:08 AM Yong Zhang  wrote:
>
> Hi Enrico, I reproduce this issue locally when using jdk11.
>
> Then I can add this options ` --add-opens
> java.base/jdk.internal.loader=ALL-UNNAMED`
> in the pom and the BookieJournalTest can pass.
>
>
>
> On Mon, 1 Nov 2021 at 15:46, Enrico Olivelli  wrote:
>
> > Il Lun 1 Nov 2021, 05:04 Yong Zhang  ha
> > scritto:
> >
> > > It looks all tests are passed on the CI with JDK1.8
> > > https://github.com/apache/bookkeeper/tree/v4.14.3-rc1
> >
> >
> > I am talking about JDK11
> > In Pulsar we run by default on jdk11
> > Having those tests failing on JDK11
> > isn't very good.
> > I will rerun my tests on jdk8
> >
> > Can you please double check on jdk11?
> >
> > Enrico
> >
> > >
> > >
> > > So it shouldn't be a block for releasing 4.14.3, right?
> > >
> > > On Mon, 1 Nov 2021 at 11:23, Yong Zhang 
> > > wrote:
> > >
> > > > Thanks Enrico. I will take a look and test it locally.
> > > >
> > > > On Sat, 30 Oct 2021 at 00:20, Enrico Olivelli 
> > > wrote:
> > > >
> > > >> Yong
> > > >> I am not able so see some tests passing on JDK11
> > > >>
> > > >> I created this issue
> > > >> https://github.com/apache/bookkeeper/issues/2872
> > > >>
> > > >> probably there is some library missing in the classpath ?
> > > >>
> > > >> Enrico
> > > >>
> > > >>
> > > >> Il giorno gio 28 ott 2021 alle ore 14:52 Nicolò Boschi <
> > > >> boschi1...@gmail.com>
> > > >> ha scritto:
> > > >>
> > > >> > +1 (non binding)
> > > >> >
> > > >> > Verified the followings:
> > > >> > - checksum and signatures
> > > >> > - build from sources
> > > >> > - ledgers write/read with "localbookie 3" and "standalone"
> > > >> >
> > > >> > Il giorno gio 28 ott 2021 alle ore 09:06 Yong Zhang <
> > y...@apache.org>
> > > >> ha
> > > >> > scritto:
> > > >> >
> > > >> > > Hi everyone,
> > > >> > > Please review and vote on the release candidate #0 for the version
> > > >> > 4.14.3,
> > > >> > > as follows:
> > > >> > > [ ] +1, Approve the release
> > > >> > > [ ] -1, Do not approve the release (please provide specific
> > > comments)
> > > >> > >
> > > >> > > The complete staging area is available for your review, which
> > > >> includes:
> > > >> > > * Release notes [1]
> > > >> > > * The official Apache source and binary distributions to be
> > deployed
> > > >> to
> > > >> > > dist.apache.org [2]
> > > >> > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > >> > > * Source code tag "v4.14.3-rc1" [4] with git sha [5]
> > > >> > >
> > > >> > > BookKeeper's KEYS file contains PGP keys we used to sign this
> > > release:
> > > >> > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > >> > >
> > > >> > > Please download these packages and review this release candidate:
> > > >> > >
> > > >> > > - Review release notes
> > > >> > > - Download the source package (verify shasum, and asc) and follow
> > > the
> > > >> > > instructions to build and run the bookkeeper service.
> > > >> > > - Download the binary package (verify shasum, and asc) and follow
> > > the
> > > >> > > instructions to run the bookkeeper service.
> > > >> > > - Review maven repo, release tag, licenses, and any other things
> > you
> > > >> > think
> > > >> > > it is important to a release.
> > > >> > >
> > > >> > > The vote will be open for at least 72 hours. It is adopted by
> > > majority
> > > >> > > approval, with at least 3 PMC affirmative votes.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Release Manager
> > > >> > >
> > > >> > > [1]
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> > https://github.com/apache/bookkeeper/pull/2864/files#diff-5405f1e88785133938fb03ee37380c8df913b3cbaec94a2d9bfcda1f8bd57cd5
> > > >> > > [2]
> > > >> > >
> > > >>
> > > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.3-rc1/
> > > >> > > [3]
> > > >> > >
> > > >> >
> > > >>
> > >
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1059
> > > >> > > [4] https://github.com/apache/bookkeeper/releases/tag/v4.14.3-rc1
> > > >> > > [5] 11913dd7e261524427b4600a0fca970e58cc7f3b
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Nicolò Boschi
> > > >> >
> > > >>
> > > >
> > >
> >

Re: Skip replicating ledger after bookkeeper server starts up again

[Matteo Merli]

This is already happening, to a certain extent.

When the auto-recovery workers are starting to replicate a ledger that
got marked as under-replicated, they will first check every fragment
of the ledger and will perform a test to decide whether it really
needs to get replicated.

There's a configurable threshold of a percentage of entries to try to
read from each bookie on the ensemble. If all the replicas can be
read, then the ledger is skipped.

By default, we only read the first and last entry on each fragment. If
these are good on all the bookies, then we skip.

The gotcha here is that this check is done before starting replicating
a ledger. If the copy process gets started, it will then continue
until done. A potential improvement here would be to even break the
copy of 1 ledger.

Matteo
--
Matteo Merli


On Sat, Aug 28, 2021 at 6:47 AM zhangao  wrote:
>
> Maybe the idea remove ledger from underreplicated when bookie 
> starts up has some problem, What important is, replicate legder puts so much 
> heavy loads on cluster. I want to reduce the affect.
>
>
>
>
> suppose there are 5 copies and 4 bookies are lost, when one lost bookie is 
> online again, I don't simply mark the ledger 
> replicated.When the replicateWork gets the failed bookies, it 
> will ignore the one which online again. So, this will reduce 
> replicating one copy of data.
>
>
>
>
>
>
> --原始邮件--
> 发件人:  
>   "dev"   
>  
>  发送时间:2021年8月28日(星期六) 晚上6:07
> 收件人:"Bookkeeper-Dev"
> 主题:Re: Skip replicating ledger after bookkeeper server starts up again
>
>
>
> Candy Rain,
> Thanks for sharing your proposal
>
> Il Ven 27 Ago 2021, 04:30 Candy Rain  scritto:
>
>  Describe
> 
>  Whennbsp; bookies in the cluster are down, the auto-recovery gets
>  triggered, but as these come back online, the rereplication worker should
>  Ideally skip the rereplication of the ledgers that are marked as
>  underreplicated.
>
>
> The replicator should be able to listen on zookeeper for changes I'm the
> availability of a bookie.
> Probably there is some space for improvements here.
>
> I don't think it is a good idea to eagerly set the ledger as no more
> underreplicated as soon as one of the bookie in the ensemble comes back
> only.
> The fact that a ledger is underreplicated is to be verified against the
> requested number of replicas, as it is not enough that one bookie is running
>
> Regards
> Enrico
>
>
> But the ledgers are rereplicated instead
> 
> 
> 
>  Expected behavior
> 
>  Ideally as and when the bookies come up the ledgers marked as
>  underreplicated would be read by the rereplicaton worker and from the
>  metadata the worked should skip these as the bookies are available.

Re: [VOTE] BookKeeper release 4.14.1, release candidate #0

[Matteo Merli]

The vote is now closed and the release was approved with 6 +1s (4
binding and 2 non-binding);

Binding votes:
 * Matteo
 * Jia
 * Sijie
 * Enrico

Non-Binding votes:
 * Yong
 * Hang

Thank you.
Matteo


--
Matteo Merli


On Mon, May 31, 2021 at 5:47 AM hangc  wrote:
>
> +1（non-binding）
>
> * git clone source code and checkout tag release-4.14.1-rc0
> * mvn install to localhost
> * change pulsar’s bookkeeper dependency to 4.14.1 in pom.xml
> * mvn package pulsar source code
> * deploy pulsar lib on BIGO pulsar test cluster
> * produce to and consume from pulsar topic
> * start prometheus server and check BookKeeper and Pulsar prometheus 
> parsed metrics
> * All metrics looks good
> * BookKeeper and Pulsar works fine.
>
> Thank you Matteo for this hot fix release.
>
> Thanks,
> Hang
>
>
> On 2021/05/31 00:28:37, PengHui Li  wrote:
> > +1 (non-binding)>
> >
> >   * Download binary package>
> >   * Change the stats provider to>
> > org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider>
> >   * Running standalone bookie>
> >   * Check localhost:8080/metrics to check the metrics>
> >   * All metrics looks good.>
> >
> > Thanks,>
> > Penghui>
> >
> >
> > On 2021/05/29 05:34:19, Matteo Merli  wrote:>
> > > Hi everyone,>>
> > >>
> > > Please review and vote on the release candidate #0 for the version>>
> > > 4.14.1, as follows:>>
> > > [ ] +1, Approve the release>>
> > > [ ] -1, Do not approve the release (please provide specific comments)>>
> > >>
> > > The complete staging area is available for your review, which includes:>>
> > > * Release notes [1]>>
> > > * The official Apache source and binary distributions to be deployed>>
> > > to dist.apache.org [2]>>
> > > * All artifacts to be deployed to the Maven Central Repository [3]>>
> > > * Source code tag "release-4.14.1-rc0" [4] with git sha [5]>>
> > >>
> > > BookKeeper's KEYS file contains PGP keys we used to sign this release:>>
> > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS>>
> > >>
> > > Please download these packages and review this release candidate:>>
> > >>
> > > - Review release notes>>
> > > - Download the source package (verify shasum, and asc) and follow the>>
> > > instructions to build and run the bookkeeper service.>>
> > > - Download the binary package (verify shasum, and asc) and follow the>>
> > > instructions to run the bookkeeper service.>>
> > > - Review maven repo, release tag, licenses, and any other things you>
> > think>>
> > > it is important to release.>>
> > >>
> > > The vote will be open for at least 72 hours. It is adopted by majority>>
> > > approval, with at least 3 PMC affirmative votes.>>
> > >>
> > > Thanks,>>
> > > Release Manager>>
> > >>
> > > [1] https://github.com/apache/bookkeeper/pull/2724>>
> > > [2]>
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.1-rc0/>>
> > > [3]>
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1053/>>
> >
> > > [4] https://github.com/apache/bookkeeper/releases/tag/release-4.14.1-rc0>>
> > > [5] f7a94422e44a7da6a5a8667853ad468bb0194642>>
> > >>
> > >>
> > > Thank you,>>
> > > Matteo>>
> > >>
> > > -->>
> > > Matteo Merli>>
> > > >>
> > >>
> >

[VOTE] BookKeeper release 4.14.1, release candidate #0

[Matteo Merli]

Hi everyone,

Please review and vote on the release candidate #0 for the version
4.14.1, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* Release notes [1]
* The official Apache source and binary distributions to be deployed
to dist.apache.org [2]
* All artifacts to be deployed to the Maven Central Repository [3]
* Source code tag "release-4.14.1-rc0" [4] with git sha [5]

BookKeeper's KEYS file contains PGP keys we used to sign this release:
https://dist.apache.org/repos/dist/release/bookkeeper/KEYS

Please download these packages and review this release candidate:

- Review release notes
- Download the source package (verify shasum, and asc) and follow the
instructions to build and run the bookkeeper service.
- Download the binary package (verify shasum, and asc) and follow the
instructions to run the bookkeeper service.
- Review maven repo, release tag, licenses, and any other things you think
it is important to release.

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1] https://github.com/apache/bookkeeper/pull/2724
[2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.1-rc0/
[3] https://repository.apache.org/content/repositories/orgapachebookkeeper-1053/
[4] https://github.com/apache/bookkeeper/releases/tag/release-4.14.1-rc0
[5] f7a94422e44a7da6a5a8667853ad468bb0194642


Thank you,
Matteo

--
Matteo Merli

Re: [DISCUSS] Release 4.14.1

[Matteo Merli]

On Tue, May 25, 2021 at 6:00 AM Enrico Olivelli  wrote:
> Before rushing to a release, isn't it better to wait for a couple of
> weeks in order to see
> if other people in the community find other problems on 4.14.0

As it is, the 4.14.0 release cannot be used by Pulsar. At the same
time, there are blocker issues for the upcoming Pulsar 2.8.0 that are
only fixed in the BK 4.14 branch.

I don't think that waiting 2 weeks will help at all here. We already
have a blocker issue, which is already fixed and that could unblock
4.14 and Pulsar 2.8.

> AFAIK BK 4.14.0 has not been announced yet publicly

Since we cannot change 4.14.0 at this point (the vote was cast, the
jars are in maven central), I don't think that it would be a problem
to just announce 4.14.1 directly, since the 4.14.0 is not usable.

Matteo

[DISCUSS] Release 4.14.1

[Matteo Merli]

While 4.14.0 is being released (tag was voted and pushed, jars
available in Maven central although not yet announced), an issue was
found that prevents Prometheus metrics to be collected for Bookies.
Details: https://github.com/apache/bookkeeper/pull/2718

Since we need 4.14.0 changes to be included in the imminent Pulsar
2.8.0 release, I think we should release a patch ASAP in order to
unblock that release.

If there are no objections, I'll move forward with the patch release.

Thanks,
Matteo

--
Matteo Merli

Re: [VOTE] Release 4.14.0, release candidate #0

[Matteo Merli]

+1 (binding)

Checked:
  * Signatures
  * Source package structure, and compile
  * Binary package
  * Running standalone bookie and client


--
Matteo Merli


On Fri, May 14, 2021 at 2:23 PM Andrey Yegorov
 wrote:
>
> I do not hear voices against it so I assume everyone is ok with this.
> (I started building RC1 and docker hanged mid-process; happy to not deal
> with retry).
>
> The vote has been out for about 5 days already.
> I need votes from 3 PMCs to proceed.
>
>
> On Fri, May 14, 2021 at 12:10 PM Henry Saputra 
> wrote:
>
> > I am +1 to keep Gradle out from this release since the files are ok not to
> > be included.
> >
> > I am ok to keep RC0 for 4.14.0 for VOTE as release candidate.
> >
> > - Henry
> >
> > On Fri, May 14, 2021 at 12:14 AM Enrico Olivelli 
> > wrote:
> >
> > > Andrey,
> > > I have merged the patch.
> > >
> > > In my opinion the source tarball should reflect the git repo without
> > > the '.git' database.
> > > That said, the most important thing is that the source release
> > > contains the files to build the project and produce working binaries.
> > >
> > > This is in the spirit of Open Source (of the ASF at least), you can
> > > pick the sources, modify them, build and run the new binaries.
> > >
> > > AFAIK The Gradle build is not fully implemented so it is not so
> > > important to see those files at the moment
> > > I agree that we can strip away the website, it is useless to the
> > > purpose of building the project.
> > >
> > > Enrico
> > >
> > >
> > > Il giorno gio 13 mag 2021 alle ore 23:43 Andrey Yegorov
> > >  ha scritto:
> > > >
> > > > I looked at the situation with src release artifact, and:
> > > > - it was definitely built from the right changelist (I spot checked
> > some
> > > > files affected by the last commit)
> > > > - site/ is explicitly excluded
> > > >
> > >
> > https://github.com/apache/bookkeeper/blob/732b6cf2a6576c844a4b43176125e82d500a6ea8/bookkeeper-dist/src/assemble/src.xml#L61
> > > > - site2 is not excluded (should be fixed)
> > > > - gradle files are not explicitly included, pom are
> > > >
> > >
> > https://github.com/apache/bookkeeper/blob/732b6cf2a6576c844a4b43176125e82d500a6ea8/bookkeeper-dist/src/assemble/src.xml#L33
> > > >
> > > > that src.xml is used to package the release artifacts
> > > >
> > >
> > https://github.com/apache/bookkeeper/blob/master/bookkeeper-dist/pom.xml#L55
> > > >
> > > > It looks like the plugin resolves the situation with the file not
> > > matching
> > > > includes and excludes by excluding it.
> > > >
> > > > PR that adds gradle into the includes patterns and excludes site2:
> > > > https://github.com/apache/bookkeeper/pull/2714
> > > >
> > > > I can either rebuild the RC or we can decide to proceed.
> > > > Gradle was not used to build/release 4.14.0
> > > >
> > > >
> > > > On Thu, May 13, 2021 at 8:34 AM Andrey Yegorov <
> > > andrey.yego...@datastax.com>
> > > > wrote:
> > > >
> > > > > I ran the release script, as described in the release procedure.
> > > > > It all ran from the docker container.
> > > > >
> > > > > On Wed, May 12, 2021 at 11:30 PM Henry Saputra <
> > > henry.sapu...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi Andrey,
> > > > >>
> > > > >> How did you generate the source release artifact? Looks like it is
> > > missing
> > > > >> some files like the site directory and the gradle build files.
> > > > >>
> > > > >> - Henry
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Sun, May 9, 2021 at 11:16 PM Andrey Yegorov <
> > > > >> andrey.yego...@datastax.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi everyone,
> > > > >> > Please review and vote on the release candidate #0 for the version
> > > > >> 4.14.0,
> > > > >> > as follows:
> > > > >> > [ ] +1, Approve the release
> > > > >> > [ ] -1, Do not approve the release (please provide specific
> > > comment

Re: Cutting BookKeeper 4.13.1 or 4.14.0 release?

[Matteo Merli]

Andrey, there are several PRs that would be good to get into 4.14. We
shouldn't just push everything out to 4.15.

Just a couple of examples:
 * https://github.com/apache/bookkeeper/pull/2710
 * https://github.com/apache/bookkeeper/pull/2698



--
Matteo Merli


On Fri, May 7, 2021 at 12:55 PM Andrey Yegorov
 wrote:
>
> I created https://github.com/apache/bookkeeper/pull/2712 with docs and
> release notes update for the v 4.14.0
> The most interesting part is the release notes:
> https://github.com/apache/bookkeeper/blob/e3c5994c05c970e6343fa9b43d1e63bac6142e60/site/docs/4.14.0/overview/releaseNotes.md
>
> Some PRs missed milestones and/or release labels, probably merged manually.
> I tracked changelists from git history and updated the labels/milestones.
>
> I'll start working on the release.
>
> On Fri, May 7, 2021 at 8:52 AM Andrey Yegorov 
> wrote:
>
> > I added https://github.com/apache/bookkeeper/issues/2711 for the TLS 1.3
> > support
> > Unless someone objects in the next 30min, I'll merge Lari's PR.
> > After that I'll start working on the BK 4.14.0 release.
> >
> > On Thu, May 6, 2021 at 1:25 AM Lari Hotari  wrote:
> >
> >> I can confirm that the PR checks pass after excluding TLSv1.3 from enabled
> >> protocols:
> >>
> >> https://github.com/apache/bookkeeper/pull/2696/commits/6003a374d5aec30d7059a21e473ac91417b5cdc3
> >>
> >> There should be tests for both TLSv1.2 and TLSv1.3 because of the
> >> differences in TLS handshake described in
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_a_62465859=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=0B1UvYMwy7dr9qtqFwQCfxUyrozUgZzbOshynTIaYUY=76JE79AuinlMNecD5DDFGgg-jXzCGZEh3PANpQOJUoE=iZz_eExfeElZI--ooxMmyMABWjailhDc7rKIAZNg59s=
> >> .
> >>
> >> This also impacts some production code in Bookkeeper. The PR already
> >> includes a change to catch SSLException instead of SSLHandshakeException (
> >>
> >> https://github.com/apache/bookkeeper/pull/2696/commits/fcbd707a633ed1b8cf8290cb5d70a3070e010196
> >> ).
> >> TLSv1.3 doesn't throw SSLHandshakeException for certificate issues because
> >> of the differences in the protocols. This change should work for both
> >> protocols, but we should have test coverage to ensure that.
> >>
> >> TLSv1.3 has been enabled by default since Netty 4.1.52.Final (when the JDK
> >> contains TLSv1.3). TLSv1.3 support has been available in Java 8 since
> >> 8u262
> >> .
> >>
> >> One of the remaining problems with TLSv1.3 support in BK is the state
> >> machine and TLS counters in PerChannelBookieClient . It doesn't properly
> >> model the way TLS 1.3 behaves. Currently there's a counter
> >> FAILED_TLS_HANDSHAKE_COUNTER  which is expected to count also the
> >> certificate issues (code:
> >>
> >> https://github.com/apache/bookkeeper/blob/fcbd707a633ed1b8cf8290cb5d70a3070e010196/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java#L1535-L1543
> >> ). Since TLSv1.3 doesn't detect certificate issues (mutual TLS) during
> >> handshake, this counter doesn't count certificate issues. Certificate
> >> issues will show up as successfully established connections.
> >> The original issue for adding TLS counters was
> >> https://github.com/apache/bookkeeper/issues/1103 and PR commit was
> >>
> >> https://github.com/apache/bookkeeper/commit/fa10b7dcd89c40222ba5f30bb60f785bd21669b2
> >> .
> >>
> >> How do we revisit the TLS counter solution for TLSv1.3 compatibility? Do
> >> we
> >> make changes to the code or do we simply skip the test on TLSv1.3 which
> >> ensures that a certificate issue is counted in
> >> FAILED_TLS_HANDSHAKE_COUNTER
> >> ? Skipping the test for TLSv1.3 would be one option. WDYT?
> >>
> >> -Lari
> >>
> >> On Wed, May 5, 2021 at 10:32 PM Andrey Yegorov <
> >> andrey.yego...@datastax.com>
> >> wrote:
> >>
> >> > Lari and I have looked at the Netty upgrade.
> >> > There are some test breaks, and so far everything is related to behavior
> >> > changes related to TLS 1.3, see
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_a_62465859=DwIFaQ=adz96Xi0w1RHqtPMowiL2g=0B1UvYMwy7dr9qtqFwQCfxUyrozUgZzbOshynTIaYUY=76JE79AuinlMNecD5DDFGgg-jXzCGZEh3PANpQOJUoE=iZz_eExfeElZI--ooxMmyMABWjailhDc7rKIAZNg59s=
> >> > We managed to fix some of the issues
> >> > https://github.com/apache/bookkeeper/pull/2696 but "the clien

Re: Cutting BookKeeper 4.13.1 or 4.14.0 release?

[Matteo Merli]

+1 We should do 4.14, carrying all the fixes that are there in master as well.


--
Matteo Merli


On Tue, May 4, 2021 at 11:57 AM Sijie Guo  wrote:
>
> +1
>
> On Tue, May 4, 2021 at 2:22 AM Yunze Xu 
> wrote:
>
> > Hello,
> > About 10 days ago I found a heap memory copy problem in Apache Pulsar, see
> > [1].
> > It’s a problem of BK side because when `LedgerHandle#asyncAddEntry`
> > accepts a `CompositeByteBuf` or a wrapper, it will finally call
> > `ByteBuf#nioBuffer()`, which would make a heap copy from direct memory.
> > [2] fixed this problem and has been merged for a week.
> >
> > Since it has a significant impact on Pulsar, Pulsar side needs a new BK
> > release with [2] merged to fix it. Is there any plan to cut a 4.13.1
> > release or 4.14.0 release so that we can upgrade the dependency in Pulsar?
> >
> > Thanks,
> > Yunze
> >
> > [1] https://github.com/apache/pulsar/pull/10330 <
> > https://github.com/apache/pulsar/pull/10330>
> >
> > [2] https://github.com/apache/bookkeeper/pull/2701 <
> > https://github.com/apache/bookkeeper/pull/2701>
> >
> >

Re: Unbounded memory usage for WQ > AQ ?

[Matteo Merli]

On Fri, Jan 8, 2021 at 2:15 PM Venkateswara Rao Jujjuri
 wrote:
>
> > otherwise the write will timeout internally and it will get replayed to a
> new bookie.
> If Qa is met and the writes of Qw-Qa fail after we send the success to the
> client, why would the write replayed on a new bookie?

I think the original intention was to avoid having 1 bookie with a
"hole" in the entries sequence. If you then lose one of the 2 bookies,
it would be difficult to know which entries need to be recovered.

Re: Unbounded memory usage for WQ > AQ ?

[Matteo Merli]

On Fri, Jan 8, 2021 at 8:27 AM Enrico Olivelli  wrote:
>
> Hi Matteo,
> in this comment you are talking about an issue you saw when WQ is greater 
> that AQ
> https://github.com/apache/bookkeeper/issues/2497#issuecomment-734423246
>
> IIUC you are saying that if one bookie is slow the client continues to 
> accumulate references to the entries that still have not received the 
> confirmation from it.
> I think that this is correct.
>
> Have you seen problems in production related to this scenario ?
> Can you tell more about them ?

Yes, for simplicity, assume e=3, w=3, a=2.

If one bookie is slow (not down, just slow), the BK client will the
acks to the user that the entries are written after the first 2 acks.
In the meantime, it will keep waiting for the 3rd bookie to respond.
If the bookie responds within the timeout, the entries can now be
dropped from memory, otherwise the write will timeout internally and
it will get replayed to a new bookie.

In both cases, the amount of memory used in the client will max at
"throughput" * "timeout". This can be a large amount of memory and
easily cause OOM errors.

Part of the problem is that it cannot be solved from outside the BK
client, since there's no visibility on what entries have 2 or 3 acks
and therefore it's not possible to apply backpressure. Instead,
there should be a backpressure mechanism in the BK client itself to
prevent this kind of issue.
One possibility there could be to use the same approach as described
in https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits,
giving a max memory limit per BK client instance and throttling
everything after the quota is reached.


Matteo

Re: Contributing Splunk changes back to OSS

[Matteo Merli]

+1 from me on Gradle.

The advantages in having very fast (and correct) incremental builds
are very nice.

I wouldn't expect major problems in setting up for Apache releases
since several other ASF projects are already using it.

Based on feedback on this, I'd also like to later start a similar
Gradle proposal for Pulsar builds too.

Matteo

On Tue, Nov 10, 2020 at 3:08 AM Enrico Olivelli  wrote:
>
> Il giorno mar 10 nov 2020 alle ore 11:09 Ivan Kelly  ha
> scritto:
>
> > Hi folks,
> >
> > It's been about a year since Streamlio joined Splunk and since then
> > we've had a bit of forking with our BK branch.
> > It has gotten to a stage where it's starting to be a problem for us,
> > so we'd like to start to get things back in sync.
> >
> > There are a couple of big chunks of work to come back.
> > We've added a data integrity checker that replaces a lot of the
> > functionality of autorecovery and allows us to run without a journal.
> > We refactored the bookie to allow dependency injection.
> > We've rewritten the entry logger to use direct I/O (allowing 2GBps
> > writes per bookie).
> >
>
> Cool, eager to see those changes in ASF BK
>
>
> >
> > One other thing we've done is to change the build system to use gradle.
> > The major driver for this was that maven is just slow, even before you
> > start running tests.
> > "mvn clean package -DskipTests" takes 4m30 on my laptop. "./gradlew
> > clean jar" takes 40s.
> > Subsequent builds on gradle are much much faster, as it does
> > incremental building.
> > Incremental building exists in maven, but it doesn't work.
> > Gradle also handle multimodule projects better. If I make a change in
> > bookkeeper-common,
> > "./gradlew :bookkeeper-server:test" will pick up the change without
> > having to explicitly
> > "mvn install" the bookkeeper-common. In my opinion it's just a much
> > nicer build system
> > to work with. Even the poms it generates are better as they avoid
> > dependency pollution.
> >
>
> I am not a big fan of Gradle, but I don't want to start a battle. There are
> pros and cons.
> To me it is a matter of taste, both of the two worlds are pretty widespread.
> Personally I have experienced the move of some projects from Maven to
> Gradle with a little bit of pain,
> but as said I am not against a change.
>
> Usually changing the build system is problematic for:
> - existing contributors/committers
> - forked repositories
>
> If you have time and resources to drive the change and to help the
> community to understand how to work with Gradle I am happy to accept it.
> I will be also a good change to reduce some tech debt, when you rewrite the
> build system/configuration you can decide to chop useless stuff that you
> aren't dropping because it is better to not fix things that aren't broken
> So +1, a BP is a good starting point please
>
>
> > What are peoples opinions on moving BookKeeper to gradle (assuming
> > I/splunk do the legwork)?
> > If people are open to it, I'll submit a BP.
> >
> > Another thing that BK (and the whole ecosystem) is missing is
> > structured logging.
> > We also plan to add structured logging to BK in soon. This is a major
> > motivator for converging the branches,
> > as it touches a lot of places.
> >
>
> +1
>
>
> > Anyhow, any feedback appreciated.
> >
>
> I am happy that the community can start to work together again as a whole
>
> Enrico
>
>
> >
> > -Ivan
> >

[RESULT] [VOTE] Release 4.9.2, release candidate #0

[Matteo Merli]

The vote is now closed with 3 +1 votes:

+1s :
 * Matteo
 * Enrico
 * Jia

No -1s.

Thanks

On Wed, May 15, 2019 at 6:25 PM Jia Zhai  wrote:
>
> +1 (binding)
>
> Environment: macOS 10.14.2
>
> - verified 4 packages checksum ( asc and sha all good)
>
> - the source package build and test all run successfully.
>
> - in both binary package(server & all), 'bin/bookkeeper standalone' and
> 'bin/bookkeeper shell bookiesanity' runs well.
>
>
> On Thu, May 9, 2019 at 7:37 PM Enrico Olivelli  wrote:
>
> > +1 (binding)
> > Checked signatures and checksums
> > Checked RAT/Spotbugs on JDK8
> > Built on JDK8 + Linux, run tests
> > Run "standalone" mode and localbookie (with JDK8)
> > Compared tarballs with 4.9.1 onces, no significant changes (I have used
> > 'meld')
> >
> > Thank you Matteo
> >
> > Enrico
> >
> > Il giorno mer 8 mag 2019 alle ore 02:24 Matteo Merli 
> > ha
> > scritto:
> >
> > > Hi everyone,
> > >
> > > Please review and vote on the release candidate #0 for the version
> > > 4.9.2, as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * Release notes [1]
> > > * The official Apache source and binary distributions to be deployed
> > > to dist.apache.org [2]
> > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > * Source code tag "v4.9.2-rc0" [4] with git sha
> > > 2cda8e7192b3899105e9bf05d38e825b94e36901
> > >
> > > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > >
> > > Please download these packages and review this release candidate:
> > >
> > > - Review release notes
> > > - Download the source package (verify shasum, and asc) and follow the
> > > instructions to build and run the bookkeeper service.
> > > - Download the binary package (verify shasum, and asc) and follow the
> > > instructions to run the bookkeeper service.
> > > - Review maven repo, release tag, licenses, and any other things you
> > think
> > > it is important to a release.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Matteo
> > >
> > >
> > > [1] https://github.com/apache/bookkeeper/pull/2088
> > > [2]
> > > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.9.2-rc0/
> > > [3]
> > >
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1042
> > > [4] https://github.com/apache/bookkeeper/tree/v4.9.2-rc0
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> >

Re: [DRAFT] BookKeeper Board Report of May

[Matteo Merli]

+1


--
Matteo Merli


On Wed, May 8, 2019 at 6:47 PM Sijie Guo  wrote:
>
> Hi all,
>
> Here is a draft for BookKeeper board report of May. Please take a look.
>
> If there is no objections, I will submit it today.
>
> ---
>
>
> ## Description:
>
> BookKeeper is a scalable, fault-tolerant, and low-latency storage
> service optimized for append-only workloads. It has been used as
> a fundamental service to build high available and replicated services
> in companies like Twitter, Yahoo and Salesforce. It is also the log
> segment store for Apache DistributedLog and message store for Apache Pulsar.
>
> Apache DistributedLog is a high-level API and service layer for
> Apache BookKeeper, providing easier access to the BookKeeper
> primitives. It is a subproject of Apache BookKeeper.
>
> ## Issues:
> There are no issues requiring board attention at this time.
>
> ## Activity:
> - 4.9.0 was released on January 31 2019
> - 4.8.2 was released on March 19 2019
> - 4.9.1 was released on April 7 2019
> - 4.9.2 is under release voting
> - The growth of Apache Pulsar community also help grow the adoption
> of BookKeeper.
>  This helps building the ecosystem around BookKeeper.
>
> ## Health report:
> - Development continues at a steady pace. We are merging multiple PRs per
> week on average.
> - Mailing list and slack discussions are brisk, in particularly around the
> active projects.
>
> ## PMC changes:
>  - Currently 15 PMC members.
>  - No new PMC members added in the last 3 months
>  - Last PMC addition was Enrico Olivelli on Fri Feb 23 2018
>
> ## Committer base changes:
>  - Currently 21 committers.
>  - No new committers added in the last 3 months
>  - Last committer addition was Andrey Yegorov at Sat Feb 10 2018
>
> ## Releases:
> - 4.9.0 was released on January 31 2019
> - 4.8.2 was released on March 19 2019
> - 4.9.1 was released on April 7 2019
>
> ## Mailing list activity:
>  - dev@bookkeeper.apache.org:
> - 104 subscribers (down -1 in the last 3 months):
> - 67 emails sent to list (112 in previous quarter)
>
>  - distributedlog-comm...@bookkeeper.apache.org:
> - 12 subscribers (up 0 in the last 3 months):
> - 0 emails sent to list (0 in previous quarter)
>
>  - distributedlog-...@bookkeeper.apache.org:
> - 41 subscribers (up 0 in the last 3 months):
> - 0 emails sent to list (0 in previous quarter)
>
>  - distributedlog-iss...@bookkeeper.apache.org:
> - 9 subscribers (up 0 in the last 3 months):
> - 6 emails sent to list (2 in previous quarter)
>
>  - distributedlog-u...@bookkeeper.apache.org:
> - 26 subscribers (up 0 in the last 3 months):
> - 0 emails sent to list (0 in previous quarter)
>
>  - iss...@bookkeeper.apache.org:
> - 8 subscribers (up 0 in the last 3 months):
> - 1825 emails sent to list (1867 in previous quarter)
>
>  - u...@bookkeeper.apache.org:
> - 117 subscribers (up 0 in the last 3 months):
> - 8 emails sent to list (11 in previous quarter)
>
>
> 
>
> - Sijie

[VOTE] Release 4.9.2, release candidate #0

[Matteo Merli]

Hi everyone,

Please review and vote on the release candidate #0 for the version
4.9.2, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* Release notes [1]
* The official Apache source and binary distributions to be deployed
to dist.apache.org [2]
* All artifacts to be deployed to the Maven Central Repository [3]
* Source code tag "v4.9.2-rc0" [4] with git sha
2cda8e7192b3899105e9bf05d38e825b94e36901

BookKeeper's KEYS file contains PGP keys we used to sign this release:
https://dist.apache.org/repos/dist/release/bookkeeper/KEYS

Please download these packages and review this release candidate:

- Review release notes
- Download the source package (verify shasum, and asc) and follow the
instructions to build and run the bookkeeper service.
- Download the binary package (verify shasum, and asc) and follow the
instructions to run the bookkeeper service.
- Review maven repo, release tag, licenses, and any other things you think
it is important to a release.

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Matteo


[1] https://github.com/apache/bookkeeper/pull/2088
[2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.9.2-rc0/
[3] https://repository.apache.org/content/repositories/orgapachebookkeeper-1042
[4] https://github.com/apache/bookkeeper/tree/v4.9.2-rc0

--
Matteo Merli

Re: [DISCUSSION] Preparing for BK 4.9.2

[Matteo Merli]

If that's ok, then I'll start preparing for 4.9.2.


--
Matteo Merli


On Fri, May 3, 2019 at 11:51 AM Enrico Olivelli  wrote:
>
> Il ven 3 mag 2019, 19:01 Matteo Merli  ha scritto:
>
> > Hi all,
> >
> > I wanted to check for opinions on starting a BK 4.9.2 release. There
> > are several fixes that I'd like to get included in next Pulsar 2.3.2
> > release since they're affecting the end users.
> > In particular I'm referring to:
> >  * mmh3 python dependency
> >  * Table service RocksDB closing
> > https://github.com/apache/bookkeeper/pull/2022
> >
> > Complete list of changes currently marked for 4.9.2:
> >
> > https://github.com/apache/bookkeeper/issues?q=label%3Arelease%2F4.9.2+is%3Aclosed
> >
> > Thoughts? Anything else that should go in?
> >
>
>
> Nothing from my side
>
> Go ahead
>
>
> Enrico
>
> >
> >
> > --
> > Matteo Merli
> > 
> >

[DISCUSSION] Preparing for BK 4.9.2

[Matteo Merli]

Hi all,

I wanted to check for opinions on starting a BK 4.9.2 release. There
are several fixes that I'd like to get included in next Pulsar 2.3.2
release since they're affecting the end users.
In particular I'm referring to:
 * mmh3 python dependency
 * Table service RocksDB closing https://github.com/apache/bookkeeper/pull/2022

Complete list of changes currently marked for 4.9.2:
https://github.com/apache/bookkeeper/issues?q=label%3Arelease%2F4.9.2+is%3Aclosed

Thoughts? Anything else that should go in?


--
Matteo Merli

Re: Cutting 4.9.1

[Matteo Merli]

Hi Enrico,

looked into it but haven't found root cause yet.
Please proceed with release.


--
Matteo Merli


On Tue, Mar 19, 2019 at 5:43 AM Enrico Olivelli  wrote:
>
> News?
>
> Enrico
>
> Il lun 18 mar 2019, 17:18 Matteo Merli  ha scritto:
>>
>> Thanks Enrico for calling this out.
>>
>> Let me try to get a fix today for this. Otherwise let's go ahead with 4.9.1.
>>
>> Thanks,
>> Matteo
>>
>> --
>> Matteo Merli
>> 
>>
>> On Mon, Mar 18, 2019 at 4:01 AM Enrico Olivelli  wrote:
>> >
>> > Matteo and Ivan,
>> > I see this issue marked for 4.9.1
>> > https://github.com/apache/bookkeeper/issues/1970
>> >
>> > Do you have time to provide a fix ?
>> > Is this a blocker for you ?
>> >
>> > Otherwise I am going to cut the release as soon as possible, I am
>> > triaging issues and creating release notes
>> >
>> > Enrico
>> >
>> >
>> > Il giorno gio 14 mar 2019 alle ore 07:41 Enrico Olivelli
>> >  ha scritto:
>> > >
>> > > Good.
>> > > I will start today
>> > >
>> > > Thanks
>> > > Enrico
>> > >
>> > > Il gio 14 mar 2019, 04:53 Venkateswara Rao Jujjuri  
>> > > ha scritto:
>> > >>
>> > >> +1
>> > >>
>> > >> On Wed, Mar 13, 2019 at 6:36 PM Sijie Guo  wrote:
>> > >>
>> > >> > +1
>> > >> >
>> > >> > On Wed, Mar 13, 2019 at 5:16 PM Enrico Olivelli 
>> > >> > wrote:
>> > >> >
>> > >> > > Hi guys,
>> > >> > > I need to cut 4.9.1 because we are seeing very often this issue
>> > >> > >
>> > >> > >
>> > >> > https://github.com/apache/bookkeeper/commit/25c7506c0513351c533db643cb10c953d1e6d0b7
>> > >> > >
>> > >> > > Please tag any issue you want to merge into 4.9 branch.
>> > >> > >
>> > >> > > I will start the release process hopefully within the end of the 
>> > >> > > week
>> > >> > >
>> > >> > > Please remember that Ivan started a VOTE thread for the 4.8 branch 
>> > >> > > as
>> > >> > well
>> > >> > >
>> > >> > > Cheers
>> > >> > > Enrico
>> > >> > >
>> > >> >
>> > >>
>> > >>
>> > >> --
>> > >> Jvrao
>> > >> ---
>> > >> First they ignore you, then they laugh at you, then they fight you, then
>> > >> you win. - Mahatma Gandhi

Re: Cutting 4.9.1

[Matteo Merli]

Thanks Enrico for calling this out.

Let me try to get a fix today for this. Otherwise let's go ahead with 4.9.1.

Thanks,
Matteo

--
Matteo Merli


On Mon, Mar 18, 2019 at 4:01 AM Enrico Olivelli  wrote:
>
> Matteo and Ivan,
> I see this issue marked for 4.9.1
> https://github.com/apache/bookkeeper/issues/1970
>
> Do you have time to provide a fix ?
> Is this a blocker for you ?
>
> Otherwise I am going to cut the release as soon as possible, I am
> triaging issues and creating release notes
>
> Enrico
>
>
> Il giorno gio 14 mar 2019 alle ore 07:41 Enrico Olivelli
>  ha scritto:
> >
> > Good.
> > I will start today
> >
> > Thanks
> > Enrico
> >
> > Il gio 14 mar 2019, 04:53 Venkateswara Rao Jujjuri  ha 
> > scritto:
> >>
> >> +1
> >>
> >> On Wed, Mar 13, 2019 at 6:36 PM Sijie Guo  wrote:
> >>
> >> > +1
> >> >
> >> > On Wed, Mar 13, 2019 at 5:16 PM Enrico Olivelli 
> >> > wrote:
> >> >
> >> > > Hi guys,
> >> > > I need to cut 4.9.1 because we are seeing very often this issue
> >> > >
> >> > >
> >> > https://github.com/apache/bookkeeper/commit/25c7506c0513351c533db643cb10c953d1e6d0b7
> >> > >
> >> > > Please tag any issue you want to merge into 4.9 branch.
> >> > >
> >> > > I will start the release process hopefully within the end of the week
> >> > >
> >> > > Please remember that Ivan started a VOTE thread for the 4.8 branch as
> >> > well
> >> > >
> >> > > Cheers
> >> > > Enrico
> >> > >
> >> >
> >>
> >>
> >> --
> >> Jvrao
> >> ---
> >> First they ignore you, then they laugh at you, then they fight you, then
> >> you win. - Mahatma Gandhi

Re: [VOTE] Release 4.9.0, release candidate #1

[Matteo Merli]

+1 (binding)

(MacOS 10.14)
Verified:
  * Signatures
  * Source package build
  * Binary with localbookie
  * Client library jars in Maven


--
Matteo Merli

On Mon, Jan 28, 2019 at 10:21 PM Jia Zhai  wrote:
>
> +1 (binding)
>
> Environment: macOS 10.14.2
>
>
> - verified 4 packages checksum ( asc and sha all good)
>
> - the source package build and test all run successfully.
>
> - in both binary package(server & all), 'bin/bookkeeper standalone' and
> 'bin/bookkeeper shell bookiesanity' runs well.
>
>
>
> On Tue, Jan 29, 2019 at 12:26 AM Enrico Olivelli 
> wrote:
>
> > +1 (binding)
> >
> > * Run tests on jdk8 + linux, all passed.
> > * Checked signatures, all okay
> > * Run localbookie and "standalone mode" (without testing storage service)
> > * Used with a downstream project and all tests are okay, but the
> > migration from 8.1 was a "little" painful but doable:
> > - old LedgerMetadata was repleaced with o.a.b.api.LedgerMetadata
> > - changes in EnsemblePlacementPolicy (expected, we are braking that
> > interface at every major version)
> > - TestStatsProvider moved to bookkeeper-common:tests artifact, I had
> > to fix a bunch of poms
> >
> > I have created a bunch of issues about running the bookie on JDK11,
> > not blocker for a release.
> > I have left a comment on the release notes PR, not blocker for a release.
> >
> > Thank you Sijie for putting all together !
> >
> > Enrico
> >
> > Il giorno dom 27 gen 2019 alle ore 05:38 Sijie Guo 
> > ha scritto:
> > >
> > > Hi everyone,
> > >
> > > Please review and vote on the release candidate #1 for the version 4.9.0,
> > > as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * Release notes [1]
> > > * The official Apache source and binary distributions to be deployed to
> > > dist.apache.org [2]
> > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > * Source code tag "v4.9.0-rc1" [4] with git sha
> > > b0e3adfea04e7e65512cca54e8b63d197abf910c
> > >
> > > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > >
> > > Please download these packages and review this release candidate:
> > >
> > > - Review release notes
> > > - Download the source package (verify shasum, and asc) and follow the
> > > instructions to build and run the bookkeeper service.
> > > - Download the binary package (verify shasum, and asc) and follow the
> > > instructions to run the bookkeeper service.
> > > - Review maven repo, release tag, licenses, and any other things you
> > think
> > > it is important to a release.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Release Manager
> > >
> > > [1] https://github.com/apache/bookkeeper/pull/1910
> > > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.9.0-rc1/
> > > [3]
> > >
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-1038/
> > > [4] https://github.com/apache/bookkeeper/tree/v4.9.0-rc1
> >

Re: [VOTE] Release 4.7.3, release candidate #0

[Matteo Merli]

+1

 * Check binary, checksums, and ran a localbookie
 * Source, checked rat and compiled with tests

--
Matteo Merli



On Fri, Nov 30, 2018 at 2:53 AM Sijie Guo  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #0 for the version 4.7.3,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.7.3-rc0" [4] with git sha
> 7a25ac301910271e80986e287db4595ae5f7785e
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Sijie
>
> [1] https://github.com/apache/bookkeeper/pull/1850
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.7.3-rc0/
> [3]
>
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1036/
> [4] https://github.com/apache/bookkeeper/tree/v4.7.3-rc0
>

Re: [DISCUSSION] PIP 24: Simplify memory settings

[Matteo Merli]

On Wed, Oct 10, 2018 at 4:03 PM Sijie Guo  wrote:

> the proposal looks good to me.
>
> just out of curiosity, why PoolingPolicy only has `UnpooledHeap` and
> `PooledDirect`?
>

I think these are the 2 that make sense (from my perspective):

 1. Pooling heap memory still suffers from interactions with GC and will
still involve copying the buffers multiple times when reading/writing on
network.
 Sure, it would reduce the allocation rate, but if one wants to pool
memory it's preferable to use direct memory.

 2. Using unpooled direct memory for each buffer would be very slow so not
to be useful in practice


-- 
Matteo Merli

[DISCUSSION] PIP 24: Simplify memory settings

[Matteo Merli]

.
 */
FallbackToHeap,

/**
 * If it's not possible to allocate memory, kill the JVM process so
that it can be restarted immediately.
 *
 */
KillProcess,
}

/**
 * Define the policy for the Netty leak detector
 */
public enum LeakDetectionPolicy {

/**
 * No leak detection and no overhead
 */
Disabled,

/**
 * Instruments 1% of the allocated buffer to track for leaks
 */
Simple,

/**
 * Instruments 1% of the allocated buffer to track for leaks, reporting
stack traces of places where the buffer was
 * used
 */
Advanced,

/**
 * Instruments 100% of the allocated buffer to track for leaks,
reporting stack traces of places where the buffer
 * was used. Introduce very significant overhead.
 */
Paranoid,
}
```

It will be possible to create an allocator through the builder and then
pass it through
to Netty client/server or just directly allocate buffers.

```java
ByteBufAllocator allocator = ByteBufAllocatorBuilder.create()
.poolingPolicy(PoolingPolicy.PooledDirect)
.oomPolicy(OomPolicy.FallbackToHeap)
.leakDetectionPolicy(LeakDetectionPolicy.Disabled)
.build();
```

### Component changes

In addition to used the policies based allocator wrapper, each component
will have
additional changes.

 Pulsar broker

Add configuration options in `broker.conf` to allow configuration of the
allocator. Eg.:

```properties
allocatorPoolingPolicy=PooledDirect
allocatorPoolingConcurrency=4
allocatorOomPolicy=FallbackToHeap
allocatorLeakDetectionPolicy=Disabled
```

# Managed ledger cache

Currently, in Pulsar broker, the only memory pooled from the direct memory
region, in
addition to regular IO buffer is the ManagedLedgerCache. This cache is used
to dispatch
directly to consumers (once a message is persisted), avoiding reads from
bookies for
consumers that are caught up with producers.

By default, the managed ledger cache size will be set to 1/3rd of the total
available
direct memory (or heap if pooling is disabled).

The setting will be left empty to indicate the default dynamic behavior:

```
managedLedgerCacheSizeMb=
```

 BookKeeper Client

Add options to configure the allocator in `ClientConfiguration` object.

 Bookie

Add options to configure the allocator in `ServerConfiguration` object and
`bookkeeper.conf`.

By default, in Pulsar we configure BookKeeper to use DbLedgerStorage. This
storage
implementation has 2 main sources of memory allocations, the read and write
caches.

By default, the configured direct memory region will be divided into 3
portions:
 * IO buffers - (50% and max to 4GB)
 * Write cache - 25 %
 * Read cache - 25 %

If there is a lot of direct memory available, max 4GB will be assigned to
IO buffers and
the rest will be split between read and write caches.

This will still not take into account the memory used by RocksDB block
cache, since this
will be allocated from within the JNI library and not accounted for in JVM
heap or
direct memory regions.

The rule of thumb here would be to default to a size pegged to the direct
memory size,
say 1/5th of it.

 Pulsar Client

Add options to configure allocator policies in `PulsarClientBuilder`.

Additionally, for `PulsarClient` we should be able to define a max amount
of memory
that a single instance is allowed to use.

This memory will be used when accumulating messages in the producers
pending messages
queue or consumer receiving queues.

When the assigned client memory is filled up, some actions will be taken:

 * For producer it would be the same as the producer queue full condition,
with either
   immediate send error or blocking behavior, depending on existing
configuration.
 * For consumers, the flow control mechanism will be slowed down, by not
asking the
   brokers for more messages, once the memory is full.

A reasonable default might be to use 64 MB per client instance, which will
be shared
across all producers consumers created by that instance.

-- 
Matteo Merli

Re: [VOTE] Apache BookKeeper Release 4.7.2, release candidate #0

[Matteo Merli]

s:
> > > >> > * Release notes [1]
> > > >> > * The official Apache source and binary distributions to be
> deployed
> > > to
> > > >> > dist.apache.org [2]
> > > >> > * All artifacts to be deployed to the Maven Central Repository [3]
> > > >> > * Source code tag "v4.7.2-rc0" [4] with git
> > > >> > sha ea072c8f8b9d20651b513036c09f264ca5c7ec6e
> > > >> >
> > > >> > BookKeeper's KEYS file contains PGP keys we used to sign this
> > release:
> > > >> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > >> >
> > > >> > Please download these packages and review this release candidate:
> > > >> >
> > > >> > - Review release notes
> > > >> > - Download the source package (verify shasum, and asc) and follow
> > the
> > > >> > instructions to build and run the bookkeeper service.
> > > >> > - Download the binary package (verify shasum, and asc) and follow
> > the
> > > >> > instructions to run the bookkeeper service.
> > > >> > - Review maven repo, release tag, licenses, and any other things
> you
> > > >> think
> > > >> > it is important to a release.
> > > >> >
> > > >> > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > >> > approval, with at least 3 PMC affirmative votes.
> > > >> >
> > > >> > Thanks,
> > > >> > Sijie Guo
> > > >> >
> > > >> > [1] https://github.com/apache/bookkeeper/pull/1628
> > > >> > [2]
> > > >> >
> > >
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.7.2-rc0/
> > > >> > [3]
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1031/
> > > >> > [4] https://github.com/apache/bookkeeper/tree/v4.7.2-rc0
> > > >> >
> > > >>
> > > >
> > >
> >
>
-- 
Matteo Merli

Re: OrderedScheduler & OrderedExecutor in bookkeeper client

[Matteo Merli]

The reason for the performance difference between the OrderedExecutor and
the OrderedScheduler is
the underlying queue that is used to pass tasks to the executor thread.

In the case of OrderedExecutor, it needs a BlockingQueue and the current
default is to use JDK LinkedBlockingQueue which relies on CAS for
enqueue/dequeue. Additional room for improvement here is to use a more
specialized MP-SC queue with different wait strategies.

For OrderedScheduler, it needs a priority queue, since tasks can be either
immediate or scheduler for future execution. With priority queue, a mutex
is typically necessary and will introduce contention when there are
multiple threads passing tasks to the scheduler.

Overall, I think we should still keep separated the critical data path from
the rest of operations that are not necessarily time critical.

> What I want to achieve here is that, after a operation is submitted to
a ledger, it only ever operates on a single thread.

I think a viable option would be to keep the scheduler separate and then
always jump in the ledger thread. eg:

scheduler.schedule(() -> {
executor.executeOrdered(ledgerId, SafeRunnable.safeRun(() -> {
// do something
});
}, 100, TimeUnit.MILLISECONDS);

If the scheduler is not always used, the thread could also be lazily
created.

Matteo

On Thu, Aug 23, 2018 at 8:48 AM Ivan Kelly  wrote:

> > I don't think it is accidently. OrderedExecutor has performance
> advantages
> > than OrderedScheduler.
> >
> > A bit background on this:
> >
> > - OrderedScheduler was introduced by me. And I changed existing
> > OrderedSafeExecutor to be extending from OrderedScheduler.
> >   Trying to standarize to one `OrderedScheduler`.
> > - I think Matteo noticed performance difference between `executor` and
> > `scheduler`, so he made the change
> >
> >
> https://github.com/apache/bookkeeper/commit/46171e67e526702487641438144f28b7eb1aa07b
> > .
> > So the `executor` is used as the main callback executor, since it just
> > requires ordering but doesn't need scheduling capability.
> > the `scheduler` is used for scheduling tasks but doesn't require
> ordering.
>
> The scheduler does need ordering in one case for explicit Lac.
>
> I think we could modify the scheduler, so that we could use the same
> scheduler object for the executor and for scheduling.
>
> Instead of having multiple executors in the scheduler, just create one
> scheduled executor, which then submits to the executor service after
> the delay.
>
> What I want to achieve here is that, after a operation is submitted to
> a ledger, it only ever operates on a single thread.
> If you look at LedgerHandle now, you have to jump around 4 files to
> deduce which thread methods like handleBookieFailure or
> sendAddSuccessCallbacks are called on, and even then you can't even be
> sure, so we wrap everything in synchronized when we don't really need
> to.
>
> -Ivan
>
-- 
Matteo Merli

Re: [VOTE] Apache BookKeeper Release 4.7.1, release candidate #0

[Matteo Merli]

+1 (binding)

Checked:
 * Signatures
 * Src and bin package - localbookie and sanity
 * Maven repo


Matteo

On Sat, Jun 16, 2018 at 1:34 AM Enrico Olivelli  wrote:

> +1 (binding)
>
> verified sha + signatures
> built from source, run tests and rat, all run as expected
> run basic tests with bookkeeper all binary package (localbookie, standalone
> bookie, bookie sanity) all is ok
> run tests of downstream projects without errors
>
> I left a comment on release notes about a change in the new version of
> Netty which impacts clients and bookies on jdk9/10. Will follow up the
> discussion on PR, nothing scary or blocker.
>
> Thank you Sijie and to all the contributors of this release !
>
> Enrico
>
>
> Il giorno gio 14 giu 2018 alle ore 22:45 Sijie Guo  ha
> scritto:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #0 for the version 4.7.1,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "v4.7.1-rc0" [4] with git
> > sha 5ef0920a7d413b5d07512f7c9925af351f98a9b2
> >
> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review maven repo, release tag, licenses, and any other things you
> think
> > it is important to a release.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Sijie Guo
> >
> > [1] https://github.com/apache/bookkeeper/pull/1518
> > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.7.1-rc0/
> > [3]
> >
> >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1030/
> > [4] https://github.com/apache/bookkeeper/tree/v4.7.1-rc0
> >
>
-- 
Matteo Merli

Re: [VOTE] Apache BookKeeper Release 4.7.0, release candidate #1

[Matteo Merli]

+1

Checked:
 * Src and bin distribution
 * Checked signatures, checksums
 * License and Notice
 * Compiled and ran test in Pulsar with this RC version

Matteo

On Mon, Apr 16, 2018 at 2:42 PM Sijie Guo <guosi...@gmail.com> wrote:

> +1 from me as well.
>
> - packages (both source & binary) are good. shasum, asc are good
> - source package can compile and  run localbookie
> - binary package can run localbookie and bookie shell
> - deploy the package into gcp and aws and run a load test over the weekend.
> all is good
>
> - Sijie
>
> On Fri, Apr 13, 2018 at 12:55 PM, Sijie Guo <guosi...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #0 for the version 4.7.0,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "release-4.7.0" [4] with git sha
> > 03deee6c94383db6080031a510c9c28d0b083fc0
> >
> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review maven repo, release tag, licenses, and any other things you
> think
> > it is important to a release.
> >
> > NOTE: 4.7.0 release introduced `circe-checksum` module, which currently
> > only ships a precompiled linux JNI lib along with the artifact.
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Sijie Guo
> >
> > [1] https://github.com/apache/bookkeeper/pull/1330
> > [2] https://dist.apache.org/repos/dist/dev/bookkeeper/
> > bookkeeper-4.7.0-rc1/
> > [3] https://repository.apache.org/content/repositories/
> > orgapachebookkeeper-1029/
> > [4] https://github.com/apache/bookkeeper/tree/release-4.7.0
> >
> >
>
-- 
Matteo Merli
<mme...@apache.org>

Re: OrderedScheduler and OrderedSafeExecutor

[Matteo Merli]

One other thing that is not clear to me is why we need delayed execution
AND key ordering.

Shouldn't be fine to retry later in a random thread? If the requirement is
to do something from a specific thread, we can also jump into that thread
when the delay task is finally executed.


On Sat, Mar 24, 2018 at 11:16 AM Sijie Guo <guosi...@gmail.com> wrote:

> On Sat, Mar 24, 2018 at 8:38 AM, Matteo Merli <mme...@apache.org> wrote:
>
> > While profiling the allocations of a BookKeeper client (from current
> > master) writing entries, I've noticed that there are multiple allocations
> > per entry related to the OrderedScheduler.
> >
> > I think OrderedScheduler was introduced in 4.6 and now
> OrderedSafeExecutor
> > is just an extension of OrderedScheduler as well.
> >
>
> OrderedSafeExecutor extends OrderedScheduler was for BC. We made common
> util classes in bookkeeper-common.
>
>
> >
> > The main problem is that since OrderedScheduler supports both immediate
> and
> > delayed execution, it uses a ScheduledThreadPoolExecutor per each bucket,
> > plus some decorating wrappers.
> >
>
> Sound like this is the solution:
>
> - Keep the OrderedScheduler
> - Copy the OrderedScheduler to OrderedExecutor and change it to the
> original executor implementation
> - Let OrderedSafeScheduler extends OrderedExecutor (bc purpose)
>
>
> >
> > The ScheduledThreadPoolExecutor, needs a Priority queue and also always
> > return future objects.
> >
> > From my memory snapshot I have counted 12 objects allocated (per addEntry
> > operation) for a total of 472 bytes per each entry, all due the scheduled
> > executor.
> >
> > I think the delayed execution is only used in unit tests. In critical
> paths
> > we're only using the immediate task submission.
> >
> > I think it might be worth to refactor OrderedScheduler or
> > OrderedSafeExecutor to avoid that overhead.
> >
> > Thoughts?
> >
> > Matteo
> > --
> > Matteo Merli
> > <mme...@apache.org>
> >
>
-- 
Matteo Merli
<mme...@apache.org>

OrderedScheduler and OrderedSafeExecutor

[Matteo Merli]

While profiling the allocations of a BookKeeper client (from current
master) writing entries, I've noticed that there are multiple allocations
per entry related to the OrderedScheduler.

I think OrderedScheduler was introduced in 4.6 and now OrderedSafeExecutor
is just an extension of OrderedScheduler as well.

The main problem is that since OrderedScheduler supports both immediate and
delayed execution, it uses a ScheduledThreadPoolExecutor per each bucket,
plus some decorating wrappers.

The ScheduledThreadPoolExecutor, needs a Priority queue and also always
return future objects.

>From my memory snapshot I have counted 12 objects allocated (per addEntry
operation) for a total of 472 bytes per each entry, all due the scheduled
executor.

I think the delayed execution is only used in unit tests. In critical paths
we're only using the immediate task submission.

I think it might be worth to refactor OrderedScheduler or
OrderedSafeExecutor to avoid that overhead.

Thoughts?

Matteo
-- 
Matteo Merli
<mme...@apache.org>

Re: Pulsar now using standard Apache BookKeeper

[Matteo Merli]

On Fri, Mar 2, 2018 at 12:57 PM Dave Fisher <dave2w...@comcast.net> wrote:

> Hi -
>
> This is great. Now that this is in the two projects (which share
> committers) will need to co-ordinate releases.
>
> Keep in mind the next Pulsar release needs to be based on a particular
> BookKeeper release. An Apache project cannot release against another
> project's SNAPSHOT since that has not been validated by their release
> process.
>

Absolutely, we'll switch the the 4.7.0 final release as soon as it's
available. The only reason to switch earlier was to have more time to find
out potential issues before getting closer to Pulsar release timeline.

Matteo


-- 
Matteo Merli
<mme...@apache.org>

Pulsar now using standard Apache BookKeeper

[Matteo Merli]

In Pulsar master branch, we have switched the BookKeeper dependency from
the Yahoo fork to the 4.7.0-SNAPSHOT version of BookKeeper from main master
branch.

All the the changes that were in the Yahoo fork, which was based on 4.3.1
release, have been already merged upstream.

This was a big effort that took ~ 1 year to get through. There were 246
commits to merge into a codebase that slightly changed in a 4 years
timespan. For the curious, this is the spreadsheet we used to track the
merging.
https://docs.google.com/spreadsheets/d/1jAy3EfjViqNEKpCKpWiRv-PCZGzdjwm_PclL7Obog4Q/

I would like to call out Ivan Kelly & Jia Zhai for giving a big push of
porting many of the changes into BookKeeper and the BookKeeper community
for being very receptive and helpful in getting this load of changes back
into mainline.


Matteo
-- 
Matteo Merli
<mme...@apache.org>

Re: Journal Corruption when disk is full

[Matteo Merli]

> One thing that should circumvent this is that the bookie should go
> into readonly mode when it hits 95% full disk.

I think this is only applying to the ledgers disk, but not for the journal.

And, to answer to Bobby, the switch to read-only mode feature was already
present in 4.3 (again, just for storage device).

Matteo

On Thu, Feb 15, 2018 at 2:56 PM Ivan Kelly <iv...@apache.org> wrote:

> On Thu, Feb 15, 2018 at 9:49 PM, Bobby Evans <ev...@oath.com.invalid>
> wrote:
> > I don't have the read only mode on disk full feature yet.  I will look at
> > pulling it back to our fork, but I will also look at fixing the
> journaling
> > in general.  Having spoken with the HDFS team here, they have seen a lot
> of
> > scary things that appear similar to this situation when a disk starts to
> go
> > bad. It would probably be in our best interest to guard against some of
> > those things on the bookies too.
> What scary things are the HDFS team doing? One thing we are doing in
> the journal, is that we preallocate the disk before we write to it. I
> remember, back in the day, this was mostly to get smoother latency, as
> the filesystem would get less involved, but this should also avoid the
> situation you described in your original email (unless the filesystem
> is overcommitting, or theirs some strange CoW stuff going on). Also, I
> recall some changes that came in from twitter that would pad each
> write to the journal out to the expected block size (i don't think we
> queried the actual size), which would ensure that you didn't try to
> rewrite a block, which could corrupt data if you failed in the middle
> of a rewrite. Of course, there's no guarantee that these things are
> bug free, but they should have handled the situation you described.
>
> -Ivan
>
-- 
Matteo Merli
<mme...@apache.org>

Re: [ANNOUNCE] Apache BookKeeper 4.5.1 released

[Matteo Merli]

Great work Enrico!

On Wed, Nov 22, 2017 at 12:53 PM Enrico Olivelli <eolive...@apache.org>
wrote:

> The Apache BookKeeper team is proud to announce Apache BookKeeper version
> 4.5.1.
>
> Apache BookKeeper is a scalable, fault-tolerant, and low-latency
> storage service optimized for
> real-time workloads. It has been used for a fundamental service to
> build reliable services.
> It is also the log segment store for Apache DistributedLog and the
> message store for Apache Pulsar.
>
> This is the 6th release of the Apache BookKeeper.
>
> This is a bugfix release, it fixes bugs around parallel recovery,
> Prometheus stats provider and placement policies.
>
> For BookKeeper release details and downloads, visit:
>  http://www.apache.org/dyn/closer.cgi/bookkeeper
>
> BookKeeper 4.5.1 Release Notes are at:
> https://bookkeeper.apache.org/docs/4.5.1/overview/releaseNotes/
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The BookKeeper Team
>
-- 
Matteo Merli
<mme...@apache.org>

Re: [VOTE] Move completely to github

[Matteo Merli]

+1

On Mon, Oct 9, 2017 at 10:08 AM Sijie Guo <guosi...@gmail.com> wrote:

> +1
>
> On Mon, Oct 9, 2017 at 2:16 AM, Ivan Kelly <iv...@apache.org> wrote:
>
> > Hi folks,
> >
> > We discussed this in
> >
> http://mail-archives.apache.org/mod_mbox/bookkeeper-dev/201710.mbox/ajax/%
> > 3CCAO2yDyarL1%2Bf7AWB89NfUH3Ji35RZVaR-CmLYV16i-1RW%2BA8Lg%40mail.
> > gmail.com%3E
> >
> > But we never formally called a vote, so here's the vote.
> >
> > The bylaws don't actually cover votes like this, so lets use lazy
> > majority, active committers.
> >
> > -Ivan
> >
>
-- 
Matteo Merli
<mme...@apache.org>

Re: HerdDB a distributed JVM-embeddable database built on Apache BookKeeper

[Matteo Merli]

Very nice!

On Fri, Sep 15, 2017 at 6:02 AM Enrico Olivelli <eolive...@gmail.com> wrote:

> Hi guys,
> I am proud to share with you the first GA release of HerdDB (
> http://www.herddb.org).
>
> HerdDB is a distributed database, it uses Apache BookKeeper as distributed
> write-ahead-log.
>
> We are using it to replace MySQL, so it has JDBC bindings and it can be
> accessed using usual SQL language and it supports multi-table, multi-row
> transactions and indexes.
> But it adds the ability to spread data on a cluster of machines, without
> any shared media (thanks to BookKeeper and ZooKeeper).
>
> HerdDB Server can work in standalone mode but we designed it to be embedded
> in the JVM of clients and create clusters of peers which share a common
> database.
>
> You can find more details in this post or on the projects wiki on GitHub
> http://eolivelli.blogspot.it/2017/09/herddb-distributed-jvm-embeddable.html
>
> The code is open source and you can find it as usual on GitHub
> https://github.com/diennea/herddb
>
> Any contribution will be welcome !
>
> Cheers
>
> Enrico Olivelli
>
-- 
Matteo Merli
<mme...@apache.org>

Re: Time Based Release for onwards releases.

[Matteo Merli]

I would suggest to do quarterly releases (eg: every 3 months), since that
seems to me a good tradeoff ratio between planning and agility of shipping
new features/improvements.

Matteo

On Fri, Aug 11, 2017 at 4:36 AM Enrico Olivelli <eolive...@gmail.com> wrote:

> Il ven 11 ago 2017, 13:10 Sijie Guo <guosi...@gmail.com> ha scritto:
>
> > Folks, please take a look at this. We need to decide the schedule for
> > 4.6.0.
> >
>
> Maybe we can take 4.3 clients as compatible baseline for 4.6.
>
> What is the next step?
> Shiuld you call a vote ?
>
> Enrico
>
>
> > - Sijie
> >
> > On Mon, Aug 7, 2017 at 6:59 PM, Sijie Guo <guosi...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > First thanks everyone who contributed to 4.5.0 in the past year, and
> > > especially thanks JV for spending time doing the release. The first
> > release
> > > candidate of 4.5.0 is finally out of review now. We are almost there.
> > >
> > > We eventually merge the major features from 3 main folked branches
> > > (Salesforce, Twitter and Yahoo), so that we can converge to one main
> open
> > > source branch across different organizations. We added a lot of
> features,
> > > bug fixes and improvements. We moved to github to make contribution
> > easier
> > > and friendly and we have new website with more documentation. There are
> > > tons of works we did very well in 4.5.0.
> > >
> > > However, I think the release has taken too long to complete. It causes
> a
> > > lot of inconsistencies between code, configuration and documentation.
> > This
> > > causes most of the contributions were spent on improving documentation
> at
> > > the end of the release. And also people can't really follow what's
> > > happening in a long-cycle release and they eventually left.
> > >
> > > I am thinking of changing the release plan/schedule to a more
> time-based
> > > mechanism what other projects (like Kafka, Flink) are doing:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
> > > Some of the benefits are documented in their wikis (also copied them in
> > > the email for easy to read).
> > >
> > > Any thoughts? Shall we try to adopt this method?
> > >
> > >
> > >1.
> > >
> > >A quicker feedback cycle and users can benefit from features shipped
> > >quicker
> > >2.
> > >
> > >Predictability for contributors and users:
> > >1.
> > >
> > >   Developers and reviewers can decide in advance what release they
> > >   are aiming for with specific features.
> > >   2.
> > >
> > >   If a feature misses a release we have a good idea of when it will
> > >   show up.
> > >   3.
> > >
> > >   Users know when to expect their features
> > >   3.
> > >
> > >Transparency - There will be a published cut-off date (AKA feature
> > >freeze) for the release and people will know about it in advance.
> > Hopefully
> > >this will remove the contention around which features make it.
> > >4.
> > >
> > >Quality - we've seen issues pop up in release candidates due to
> > >last-minute features that didn't have proper time to bake in. More
> > time
> > >between feature freeze and release will let us test more, document
> > more and
> > >resolve more issues.
> > >
> > >
> > > - Sijie
> > >
> >
> --
>
>
> -- Enrico Olivelli
>
-- 
Matteo Merli
<mme...@apache.org>

Re: [VOTE] Release 4.5.0, release candidate #0

[Matteo Merli]

+1

Checked src and bin package
 * Signatures ok
 * Build
 * Rat
 * Run local bookie
   (I had to set allowLoopback=true in conf/bk_server.conf for that. I
agree we can
document it and improve it later.)

Matteo

On Mon, Aug 7, 2017 at 7:13 AM Venkateswara Rao Jujjuri <jujj...@gmail.com>
wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #0 for version 4.5.0, as
> follows:
>
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
>
> * Release Notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "release-4.5.0" [4]
>
> BookKeeper's KEY file contains PGP keys we use to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please review this release candidate.
>
> - Review release notes
> - Download the source package (verify md5, shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify md5, shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> [1] https://github.com/apache/bookkeeper/pull/402
> [2] https://dist.apache.org/repos/dist/dev/bookkeeper/4.5.0-rc0/
> [3] https://repository.apache.org/content/repositories/
> orgapachebookkeeper-1012/
> [4] https://github.com/apache/bookkeeper/tree/release-4.5.0
>
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi
>
-- 
Matteo Merli
<mme...@apache.org>

Re: [VOTE] Release 4.5.0, release candidate #0

[Matteo Merli]

I have downloaded the src package, compiled and tried to run "localbookie"
but I get this error:

2017-08-07 10:21:50,711 - ERROR - [main:LocalBookKeeper@304] - Failed to
run 3 bookies : zk ensemble = '127.0.0.1:2181'
java.net.UnknownHostException: Trying to listen on loopback address,
127.0.0.1:5000 but this is forbidden by default (see
ServerConfiguration#getAllowLoopback())
at org.apache.bookkeeper.bookie.Bookie.getBookieAddress(Bookie.java:601)
at
org.apache.bookkeeper.proto.BookieNettyServer.(BookieNettyServer.java:139)
at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:98)
at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:90)
at
org.apache.bookkeeper.util.LocalBookKeeper.runBookies(LocalBookKeeper.java:229)
at
org.apache.bookkeeper.util.LocalBookKeeper.runBookies(LocalBookKeeper.java:138)
at
org.apache.bookkeeper.util.LocalBookKeeper.startLocalBookiesInternal(LocalBookKeeper.java:287)
at org.apache.bookkeeper.util.LocalBookKeeper.main(LocalBookKeeper.java:337)


Should we always enable the "allowLoopback" setting in the localbookie mode?

On Mon, Aug 7, 2017 at 7:13 AM Venkateswara Rao Jujjuri <jujj...@gmail.com>
wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #0 for version 4.5.0, as
> follows:
>
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
>
> * Release Notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "release-4.5.0" [4]
>
> BookKeeper's KEY file contains PGP keys we use to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please review this release candidate.
>
> - Review release notes
> - Download the source package (verify md5, shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify md5, shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> [1] https://github.com/apache/bookkeeper/pull/402
> [2] https://dist.apache.org/repos/dist/dev/bookkeeper/4.5.0-rc0/
> [3] https://repository.apache.org/content/repositories/
> orgapachebookkeeper-1012/
> [4] https://github.com/apache/bookkeeper/tree/release-4.5.0
>
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi
>
-- 
Matteo Merli
<mme...@apache.org>

Re: [VOTE] Slack Channel for BookKeeper

[Matteo Merli]

+1

--
Matteo Merli
<matteo.me...@gmail.com>

On Thu, Jul 27, 2017 at 11:06 AM, Enrico Olivelli <eolive...@gmail.com>
wrote:

> +1 non binding
>
> Enrico
>
> Il gio 27 lug 2017, 20:02 Sijie Guo <guosi...@gmail.com> ha scritto:
>
> > Start a vote thread for transferring the DL slack channel to BK.
> >
> > The propose is:
> >
> > - transfer the dl slack channel apachedistributedlog.slack.com to
> > apachebookkeeper.slack.com
> > - the owner will be BookKeeper PMC (priv...@bookkeeper.apache.org)
> >
> > One note:
> >
> > - the slack channel is for informal/immediate discussion. no decisions
> are
> > made in slack channel. decision related discussions should be recorded in
> > ASF (either mailing list, JIR or wiki)
> >
> > Please vote +1, 0, -1. The vote will be open for 72 hours.
> >
> > - Sijie
> >
> --
>
>
> -- Enrico Olivelli
>

Re: [website] branch asf-site

[Matteo Merli]

Sure, make sure to place it under the "content/" folder which is where
gitpubsub is expecting it to be.

--
Matteo Merli
<matteo.me...@gmail.com>

On Thu, Jul 27, 2017 at 10:27 AM, Sijie Guo <guosi...@gmail.com> wrote:

> I created a blank branch 'asf-site' for enabling gitpubsub for the new
> website. I am also going to put a index.hml as a place holder there and ask
> INFRA team to enable gitpubsub under bookkeeper.apache.org/test. So we can
> explore the solution for allowing the co-exists of new website and old
> documentation from old releases.
>

Re: [DISCUSS] Slack Channel for BookKeeper

[Matteo Merli]

On Tue, Jul 25, 2017 at 2:57 PM, Ivan Kelly  wrote:

> Is there an apache "team" in slack? it would save having to have
> multiple teams open if a person is involved in multiple projects. This
> would be especially useful for people who use slack in the browser. It
> would also reduce the signup barrier.


Yes, there's one: https://theasf.slack.com

Though it's only open to ASF committers because it requires a @apache.org
account.

Re: BookKeeper website menus not working

[Matteo Merli]

That's true, though I would swear that it was working just a couple of
hours ago. :)

--
Matteo Merli
<matteo.me...@gmail.com>

On Tue, Jul 18, 2017 at 4:23 PM, Edward Ribeiro <edward.ribe...@gmail.com>
wrote:

> FYI, the 'Documentation', 'Get Involved' and 'Project Info' drop down menus
> on https://bookkeeper.apache.org site are not working.
>
> Edward
>

[jira] [Commented] (BOOKKEEPER-1091) Remove Hedwig from BookKeeper website page

[Matteo Merli (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035662#comment-16035662
 ] 

Matteo Merli commented on BOOKKEEPER-1091:
--

+1

> Remove Hedwig from BookKeeper website page
> --
>
> Key: BOOKKEEPER-1091
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1091
> Project: Bookkeeper
>  Issue Type: Task
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
> Attachments: BOOKKEEPER-1091.diff
>
>
> We removed Hedwig before, but it still exists in the website and links to a 
> deleted wiki page. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] Issue Tracking - Jira or Github Issues

[Matteo Merli]

To add more context, this topic was discussed recently at ApacheCon.

See this talk https://www.youtube.com/watch?v=yWurOHvm5WM
Relevant portion start at 18:00.

In short, they are saying that the
ASF is trying to leverage Github more and more in the future.

Matteo

On Thu, Jun 1, 2017 at 4:48 PM Sijie Guo  wrote:

> I think the biggest advantage is JIRA has very good workflow management.
> Github issues doesn't have such feature. But projects like docker, k8s are
> using labels for such management. I think more and more large projects (non
> ASF) are using Github for both issues, pull requests and code. They might
> already have good practices, which we probably can learn and iterate from
> there to get rid of it.
>
> For example, the practice from moby (docker) -
> https://github.com/moby/moby/blob/master/project/ISSUE-TRIAGE.md
>
> - Sijie
>
> On Thu, Jun 1, 2017 at 3:43 PM, Venkateswara Rao Jujjuri <
> jujj...@gmail.com>
> wrote:
>
> > I agree, having one place is more convenient. But what is the feature
> > parity between Jira and Github?
> > Here is what I found by googling.
> > https://confluence.sakaiproject.org/display/PMC/
> > Github+Issues+vs+Jira+pros+and+cons
> >
> > On Thu, Jun 1, 2017 at 2:59 PM, Sijie Guo  wrote:
> >
> > > Actually thinking a bit more, I think there is a bit inconvenient in
> > > current approach (JIRA for issue tracking and PR for patches). Each
> time
> > I
> > > went to github for reviewing pull requests. I have to go back to JIRA
> (by
> > > copying the JIRA id and typing the URL) to check the descriptions and
> > > discussions. Moving to Github will make the life much easier.
> > >
> > > - Sijie
> > >
> > > On Fri, May 26, 2017 at 9:54 AM, Sijie Guo  wrote:
> > >
> > > > I don't any documents at INFRA to point. I do observe more incubator
> > > > projects are using Github Issues directly when they transfer to the
> > ASF.
> > > I
> > > > knew some of the projects are switching to use Github issues. For
> > > example,
> > > > Traffic Server switches to Github Issues and makes their JIRA
> readonly
> > at
> > > > the beginning of this year.
> > > >
> > > > The preference is up to projects, I believe.
> > > >
> > > > - Sijie
> > > >
> > > > On Fri, May 26, 2017 at 11:29 PM, Flavio Junqueira 
> > > wrote:
> > > >
> > > >> I have seen some large projects relying on Github Issues, Docker
> being
> > > >> one of them. I have recently been using it in the Pravega project
> and
> > I
> > > do
> > > >> find that it doesn't offer right up front some of the features that
> > jira
> > > >> offers. For example, it doesn't give you the ability of creating a
> > > >> workflow, although what we have done and have seen others doing it
> to
> > > >> create labels to represent steps of a workflow. We ended up
> > overloading
> > > the
> > > >> use of labels, but it looks decent with the colors and such.
> > > >>
> > > >> I also find a bit confusing the relationship between issues and pull
> > > >> requests at times. We have been trying to enforce that each pull
> > request
> > > >> requires at least one issue, but sometimes it feels unnatural
> because
> > > you
> > > >> also have space for a description and the ability to comment in a
> pull
> > > >> request.
> > > >>
> > > >> I'm not sure what the story is for github issues and apache infra,
> > > >> though. The information I have is the same as Bobby's. Does anyone
> > have
> > > a
> > > >> pointer?
> > > >>
> > > >> -Flavio
> > > >>
> > > >> > On 26 May 2017, at 17:06, Bobby Evans  >
> > > >> wrote:
> > > >> >
> > > >> > Apache does have a requirement that community discussions and
> > > >> especially votes are stored on apache servers.  This is often done
> by
> > > >> linking different systems together (like pull requests to JIRA) or
> by
> > > >> having a fire-hose of changes from the external system sent to some
> > > apache
> > > >> mailing list that it can archive.
> > > >> > I have not used github issues much but from what I have done it
> does
> > > >> not look even close to being as full featured as JIRA. So my vote
> > would
> > > be
> > > >> to ask people to use JIRA, but not ignore the github issues.
> > > >> >
> > > >> > - Bobby
> > > >> >
> > > >> > On Friday, May 26, 2017, 9:57:43 AM CDT, Sijie Guo <
> > > guosi...@gmail.com>
> > > >> wrote:Hi all,
> > > >> >
> > > >> > Currently we are using Jira for issue tracking and using Github
> for
> > > >> > managing pull requests. For a new developer, he has to create two
> > > >> accounts
> > > >> > in order to engage with BookKeeper community. I am thinking -
> shall
> > we
> > > >> also
> > > >> > move the issue tracking to use Github Issues (which I believe
> Apache
> > > >> Infra
> > > >> > supports that now)? So most of the development activities will
> > happen
> > > in
> > > >> > Github.
> > > >> >
> > > >> > Another reason I asked this - I saw a Github issue was 

Re: [VOTE] Use Github Issues for Issue Tracking

[Matteo Merli]

+1

Having issues and PR in one tool is a great plus.

If this proposal is accepted, we should also ask INFRA to allow to assign
issues/PRs to committers/contributors and to create labels that fit our
workflow.


On Thu, Jun 1, 2017 at 3:03 PM Sijie Guo  wrote:

> My vote on this is +1.
>
> Simply because I'd like to have one central place for tracking issues and
> reviewing pull requests.
>
> - Sijie
>
> On Thu, Jun 1, 2017 at 12:59 PM, Sijie Guo  wrote:
>
> > Per the community meeting
> > <
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/2017-06-01+Meeting+notes
> >
> > this morning, JV proposed to start use Github issues for issue tracking
> for
> > a few months and see how does it work out. I am starting this email
> thread
> > to vote for this.
> >
> > The vote will be:
> >
> > - Start using Github issues/pull requests for issue tracking for 3
> months.
> > - During this 3 months, we will continue using both JIRA and Github
> issues.
> > - After 3 months, if the community decides whether we should continue
> > using Github issues or moving from JIRA to Github issues. (The final
> > decision will be a separate vote in 3 months)
> >
> > Please vote +1 if in favor of trying out Github issues and -1 if not.
> >
> > See below thread and community meeting notes
> > <
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/2017-06-01+Meeting+notes
> >
> > for reference:
> >
> > http://mail-archives.apache.org/mod_mbox/bookkeeper-dev/201705.mbox/%
> > 3CCAO2yDyYKmUiSfGfkGCKtfP8mmQtcJubGoMO-KsWsjM9_3pOd0Q%40mail.gmail.com
> %3E
> >
> > - Sijie
> >
>

[jira] [Created] (BOOKKEEPER-1090) Use LOG.isDebugEnabled() to avoid unexpected allocations

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1090:


 Summary: Use LOG.isDebugEnabled() to avoid unexpected allocations
 Key: BOOKKEEPER-1090
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1090
 Project: Bookkeeper
  Issue Type: Improvement
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Minor
 Fix For: 4.5.0


Using {{LOG.debug(...)}} can lead to multiple unexpected memory allocation, 
even when the logger it's turned off.

For example, {{int}} and {{long}} parameter are boxed into {{Integer}} and 
{{Long}} objects and the var-arg parameters are using an {{Object[]}} to hold
them.

We should guard all usages of {{LOG.debug()}} with the {{if 
(LOG.isDebugEnabled()}} guard.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BOOKKEEPER-1071) BookieRecoveryTest is failing due to a Netty4 IllegalReferenceCountException

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli reassigned BOOKKEEPER-1071:


Assignee: Matteo Merli

> BookieRecoveryTest is failing due to a Netty4 IllegalReferenceCountException
> 
>
> Key: BOOKKEEPER-1071
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1071
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Affects Versions: 4.5.0
>Reporter: Enrico Olivelli
>    Assignee: Matteo Merli
>Priority: Blocker
> Fix For: 4.5.0
>
>
> this is the killer commit:
> e44c7388399e5589cf44e38c58bb84c74da544af BOOKKEEPER-1069: If client uses V2 
> proto, set the connection to always decode V2 messages
> this commit was working:
> 0f81461d2d1dc5cf9db4de9a46599d7d64e3dac6 BOOKKEEPER-1048: Use ByteBuf in 
> LedgerStorage interface
> {code}
> 2017-05-18 04:50:39,691 - WARN  - [bookie-io-4:Slf4JLogger@151] - An 
> exceptionCaught() event was fired, and it reached at the tail of the 
> pipeline. It usually means the last handler in the pipeline did not handl
> e the exception.
> io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
> at 
> io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:101)
> at 
> io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:89)
> at 
> io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:84)
> at 
> io.netty.channel.DefaultChannelPipeline.onUnhandledInboundMessage(DefaultChannelPipeline.java:1169)
> at 
> io.netty.channel.DefaultChannelPipeline$TailContext.channelRead(DefaultChannelPipeline.java:1221)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at 
> org.apache.bookkeeper.proto.BookieRequestHandler.channelRead(BookieRequestHandler.java:77)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at 
> org.apache.bookkeeper.proto.AuthHandler$ServerSideHandler.channelRead(AuthHandler.java:90)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerC

[jira] [Commented] (BOOKKEEPER-1071) BookieRecoveryTest is failing due to a Netty4 IllegalReferenceCountException

[Matteo Merli (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015905#comment-16015905
 ] 

Matteo Merli commented on BOOKKEEPER-1071:
--

Sure, looking at it.

> BookieRecoveryTest is failing due to a Netty4 IllegalReferenceCountException
> 
>
> Key: BOOKKEEPER-1071
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1071
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Affects Versions: 4.5.0
>Reporter: Enrico Olivelli
>    Assignee: Matteo Merli
>Priority: Blocker
> Fix For: 4.5.0
>
>
> this is the killer commit:
> e44c7388399e5589cf44e38c58bb84c74da544af BOOKKEEPER-1069: If client uses V2 
> proto, set the connection to always decode V2 messages
> this commit was working:
> 0f81461d2d1dc5cf9db4de9a46599d7d64e3dac6 BOOKKEEPER-1048: Use ByteBuf in 
> LedgerStorage interface
> {code}
> 2017-05-18 04:50:39,691 - WARN  - [bookie-io-4:Slf4JLogger@151] - An 
> exceptionCaught() event was fired, and it reached at the tail of the 
> pipeline. It usually means the last handler in the pipeline did not handl
> e the exception.
> io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
> at 
> io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:101)
> at 
> io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:89)
> at 
> io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:84)
> at 
> io.netty.channel.DefaultChannelPipeline.onUnhandledInboundMessage(DefaultChannelPipeline.java:1169)
> at 
> io.netty.channel.DefaultChannelPipeline$TailContext.channelRead(DefaultChannelPipeline.java:1221)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at 
> org.apache.bookkeeper.proto.BookieRequestHandler.channelRead(BookieRequestHandler.java:77)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at 
> org.apache.bookkeeper.proto.AuthHandler$ServerSideHandler.channelRead(AuthHandler.java:90)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
> at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invo

[jira] [Created] (BOOKKEEPER-1069) If client uses V2 proto, set the connection to always decode V2 messages

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1069:


 Summary: If client uses V2 proto, set the connection to always 
decode V2 messages
 Key: BOOKKEEPER-1069
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1069
 Project: Bookkeeper
  Issue Type: Improvement
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Minor
 Fix For: 4.5.0


Avoid handling parsing exception for each request and instead adapt to what the 
client is sending.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1068) Expose ByteBuf in LedgerEntry to avoid data copy

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1068:


 Summary: Expose ByteBuf in LedgerEntry to avoid data copy
 Key: BOOKKEEPER-1068
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1068
 Project: Bookkeeper
  Issue Type: Improvement
Reporter: Matteo Merli
Assignee: Matteo Merli
 Fix For: 4.5.0


To avoid copying the entries payloads when writing/reading on a ledger and 
having to allocate a lot of byte[] on the JVM heap, we need to accept Netty 
ByteBuf buffer.

By passing a ByteBuf, an application can use a pooled buffer, pointing to 
direct memory, to the {{LedgerHandle.addEntry()}} and have the same buffer 
forwarded on the connection sockets to the bookies.

The same thing on the read side, {{LedgerEntry}} exposes an additional 
{{getEntryBuffer()}} method that can be used to get the underlying buffer and 
possibly forward that to some other connection, with zero-copy behavior 
(excluding getting data in-out of the kernel).





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BOOKKEEPER-1048) Use ByteBuf in LedgerStorageInterface

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli resolved BOOKKEEPER-1048.
--
Resolution: Fixed

Issue resolved by merging pull request 139
[https://github.com/apache/bookkeeper/pull/139]

{noformat}
commit 0f81461d2d1dc5cf9db4de9a46599d7d64e3dac6
Author: Matteo Merli <mme...@apache.org>
AuthorDate: Mon May 15 15:31:56 2017 -0700
Commit: Matteo Merli <mme...@apache.org>
CommitDate: Mon May 15 15:31:56 2017 -0700

BOOKKEEPER-1048: Use ByteBuf in LedgerStorage interface

To pass ref-counted buffer from Netty directly to the storage and the 
Journal, we need to have LedgerStorage to accept ByteBuf instead of ByteBuffer

 Note

This commit is on top of BOOKKEEPER-1048 / #138. Once that gets merged, I 
will rebase. Posting now to get Jenkins run. Please review last commit 
f53f772f79d0a334edc0f05e66edb7cc645b1ffa in this PR for now.

Author: Matteo Merli <mme...@apache.org>

Reviewers: Jia Zhai , Sijie Guo 

Closes #139 from merlimat/bytebuf-in-ledger-storage

{noformat}


> Use ByteBuf in LedgerStorageInterface
> -
>
> Key: BOOKKEEPER-1048
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1048
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Matteo Merli
>Assignee: Matteo Merli
> Fix For: 4.5.0
>
>
> To pass ref-counted buffer from Netty directly to the storage and the 
> Journal, we need to have LedgerStorage to accept ByteBuf instead of ByteBuffer



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BOOKKEEPER-887) Allow to use multiple bookie journals

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli resolved BOOKKEEPER-887.
-
Resolution: Duplicate

> Allow to use multiple bookie journals
> -
>
> Key: BOOKKEEPER-887
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-887
> Project: Bookkeeper
>  Issue Type: Improvement
>    Reporter: Matteo Merli
>    Assignee: Matteo Merli
> Fix For: 4.5.0
>
>
> Using multiple independent journals in the bookie can improve the bookie 
> throughput and write latencies. 
> When using HDDs for the journal, the writes and fsyncs can be spread across 
> multiple discs (or RAID groups) on different filesystem mount points. 
> If the journal is writing to an SSD, having multiple journal threads writing 
> to the same disc will let the bookie to better utilize the disk write 
> bandwidth.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BOOKKEEPER-1047) Add missing error code in ZK setData return path

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli resolved BOOKKEEPER-1047.
--
Resolution: Fixed

> Add missing error code in ZK setData return path
> 
>
> Key: BOOKKEEPER-1047
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1047
> Project: Bookkeeper
>  Issue Type: Bug
>    Reporter: Matteo Merli
>    Assignee: Matteo Merli
>Priority: Trivial
> Fix For: 4.5.0
>
>
> The log warning is not printing the error code returned by ZooKeeper



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BOOKKEEPER-1063) Use executure.execute() instead of submit() to avoid creation of unused FutureTask

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli resolved BOOKKEEPER-1063.
--
Resolution: Fixed

> Use executure.execute() instead of submit() to avoid creation of unused 
> FutureTask
> --
>
> Key: BOOKKEEPER-1063
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1063
> Project: Bookkeeper
>  Issue Type: Improvement
>    Reporter: Matteo Merli
>Assignee: Matteo Merli
>Priority: Minor
> Fix For: 4.5.0
>
>
> When submitting tasks to an executor, if the {{FutureTask}} object is not 
> being used we should use {{execute()}} instead of {{submit()}} in order to 
> avoid the task object allocation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BOOKKEEPER-1065) OrderedSafeExecutor should only have 1 thread per bucket

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli resolved BOOKKEEPER-1065.
--
Resolution: Invalid

As commented on the github PR, the current behavior is correct. No changes 
needed here.

> OrderedSafeExecutor should only have 1 thread per bucket
> 
>
> Key: BOOKKEEPER-1065
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1065
> Project: Bookkeeper
>  Issue Type: Bug
>    Reporter: Matteo Merli
>    Assignee: Matteo Merli
> Fix For: 4.5.0
>
>
> In a earlier commit, "BOOKKEEPER-874: Explict LAC from Writer to Bookie", 
> there was this change in the OrderedSafeExecutor implementation: 
> {noformat}
>  for (int i = 0; i < numThreads; i++) {
> -queues[i] = new LinkedBlockingQueue();
> -threads[i] =  new ThreadPoolExecutor(1, 1,
> -0L, TimeUnit.MILLISECONDS, queues[i],
> +threads[i] =  new ScheduledThreadPoolExecutor(1,
>  new ThreadFactoryBuilder()
>  .setNameFormat(name + "-orderedsafeexecutor-" + i + 
> "-%d")
>  .setThreadFactory(threadFactory)
>  .build());
> +threads[i].setMaximumPoolSize(1);
> {noformat}
> Then, as part of "BOOKKEEPER-1013: Fix findbugs errors on latest master", the 
> max pool size line has been removed.
> {noformat}
> @@ -183,7 +183,6 @@ public class OrderedSafeExecutor {
>  .setNameFormat(name + "-orderedsafeexecutor-" + i + 
> "-%d")
>  .setThreadFactory(threadFactory)
>  .build());
> -threads[i].setMaximumPoolSize(1);
>  // Save thread ids
>  final int idx = i;
> {noformat}
> Without that the thread pool would create multiple threads for the same 
> bucket, breaking the ordering guarantee of the executor.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1066) Introduce GrowableArrayBlockingQueue

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1066:


 Summary: Introduce GrowableArrayBlockingQueue
 Key: BOOKKEEPER-1066
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1066
 Project: Bookkeeper
  Issue Type: Improvement
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Minor
 Fix For: 4.5.0


In multiple places, (eg: journal, ordered executor, etc..), we are using 
{{LinkedBlockingQueue}} instances to pass objects between threads.

The {{LinkedBlockingQueue}} differs from the {{ArrayBlockingQueue}} in that it 
doesn't require to define a max queue size, though, being implemented with a 
linked list, it requires to allocates list nodes each time an item is added.

We can use a {{GrowableArrayBlockingQueue}} that behaves in the same way as the 
{{LinkedBlockingQueue}}, but it's implemented with an array that can be resized 
when the queue reaches the capacity.





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Next community meeting and notes

[Matteo Merli]

I think there has been some confusion. Some of us "met" last Thursday,
since it was 2 weeks from the previous call.
On Thu, May 11, 2017 at 3:46 AM Enrico Olivelli  wrote:

> I have just created the wiki page for next meeting
>
>
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/2017-5-11+Meeting+notes
>
> I have drafted an agenda with the current pending issues to discuss
> from my point of view, please everyone add your issues
>
> I'm sorry I am not sure I will be able to attend the meeting,
>
> -- Enrico
>
>
> 2017-04-06 18:01 GMT+02:00 Enrico Olivelli :
> > Here are the meeting notes, fell free to edit/update/correct
> >
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/2017-4-6+Meeting+notes
> >
> > Thank you very much to all of the attendees !
> >
> > 2017-04-06 17:06 GMT+02:00 Yiming Zang :
> >>
> >> Hi Bobby, here's the link to the meeting
> >> https://goo.gl/iyRA6G
> >>
> >> On Thu, Apr 6, 2017 at 8:04 AM, Bobby Evans  >
> >> wrote:
> >>
> >> > What is the URL.  I can never find it
> >> >
> >> >
> >> > - Bobby
> >> >
> >> > On Thursday, April 6, 2017, 5:02:53 AM CDT, Enrico Olivelli <
> >> > eolive...@gmail.com> wrote:Anyone on the Hangout ?
> >> >
> >> > 2017-04-05 14:19 GMT+02:00 Enrico Olivelli :
> >> >
> >> > > Hi all,
> >> > > I have just created the page for the upcoming meeting
> >> > > https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> >> > > 2017-4-6+Meeting+notes
> >> > >
> >> > > I have just created the page (please update with missing parts)
> >> > > https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> >> > > 2017-3-24+Meeting+notes
> >> > >
> >> > > I'm sorry I think I told I would have done the notes for the last
> >> > > meeting
> >> > > but I forgot
> >> > >
> >> > > I have created a proposed agenda, I think the most important topics
> >> > > are:
> >> > > - Netty 4 patch
> >> > > - V2 protocol support
> >> > > - Current PRs https://github.com/apache/bookkeeper/pulls
> >> > >
> >> > > @JV and @Kishore
> >> > > do you have any news about Netty 4 patch ?
> >> > >
> >> > > Cheers
> >> > >
> >> > > -- Enrico
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >
> >
>

Re: Issue in GarbageCollectorThread compaction logic

[Matteo Merli]

I think there might be a misunderstanding regarding FileChannel and
BufferedFileChannel.

BufferedFileChannel is a BK class and has a  flush(boolean) method.

 * flush(false) --> just writes the in-memory byte array to the FileChannel
(no fsync/fdatasync involved)
 * flush(true) --> write to FileChannel and issue forceWrite(false) on the
channel (fdatasync)

.. I know.. it's a bit confusing

On Wed, May 10, 2017 at 5:25 PM Venkateswara Rao Jujjuri <jujj...@gmail.com>
wrote:

> Yes; My question is what is the impact we are taking by doing force(true)
> all the time.
> Just to avoid any concern caused by Charan's statement about "java API
> for force method -
> https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#force(boolean),
> it says the implementation is unspecified and system-dependent. "
>
> On Wed, May 10, 2017 at 5:09 PM, Sijie Guo <guosi...@gmail.com> wrote:
>
>>
>>
>> On Thu, May 11, 2017 at 8:02 AM, Venkateswara Rao Jujjuri <
>> jujj...@gmail.com> wrote:
>>
>>> That is correct. But we are talking about active entrylog file(s) and
>>> journal file(s).
>>> Do you see lot of active journal files that won't get updated but read
>>> for long time?
>>>
>>
>> I am a bit confused about this question here. When talking about
>> durability, entry log files and journal files should be same, right?
>>
>>
>>
>>> Anyway just trying broaden the discussion and if we have any doubt if
>>> java does fdatasync all the time.
>>>
>>> On Wed, May 10, 2017 at 4:57 PM, Matteo Merli <matteo.me...@gmail.com>
>>> wrote:
>>>
>>>> My understanding is that fsync() forces the IO on the metadata even if
>>>> the file size has not changed, for example to update the access timestamp.
>>>>
>>>> On Wed, May 10, 2017 at 4:47 PM Venkateswara Rao Jujjuri <
>>>> jujj...@gmail.com> wrote:
>>>>
>>>>> Let me try to consolidate the discussion.
>>>>>
>>>>> 1. We need to flush the metadata whenever we cause metadata changes
>>>>> due to data changes. Mostly Append/Filesize change.
>>>>> 2. I don't know any operations where we perform "metadata operation"
>>>>> only and we needed it to be "persisted"
>>>>> Even if we can think/come up with some obscure cases of #2, majority
>>>>> of of our use-case falls into #1.
>>>>> So why not put force(true) all the time? What cases we may incur
>>>>> penalty? If we pass true, then there is no debate on 
>>>>> implementation/source.
>>>>>
>>>>> What do you guys say?
>>>>> JV
>>>>>
>>>>> On Wed, May 10, 2017 at 4:42 PM, Sijie Guo <guosi...@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, May 11, 2017 at 2:28 AM, Charan Reddy G <
>>>>>> reddychara...@gmail.com> wrote:
>>>>>>
>>>>>>>  @JV and @Sijie I think Android java source is not the appropriate
>>>>>>> one to look into. For FileChannelImpl we need to look into jdk code
>>>>>>>
>>>>>>>
>>>>>>> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/sun/nio/ch/FileChannelImpl.java#FileChannelImpl.force%28boolean%29
>>>>>>>
>>>>>>>
>>>>>>> public void force(boolean metaData) throws IOException {
>>>>>>> ensureOpen();
>>>>>>> int rv = -1;
>>>>>>> int ti = -1;
>>>>>>> try {
>>>>>>> begin();
>>>>>>> ti = threads.add();
>>>>>>> if (!isOpen())
>>>>>>> return;
>>>>>>> do {
>>>>>>> rv = nd.force(fd, metaData);
>>>>>>> } while ((rv == IOStatus.INTERRUPTED) && isOpen());
>>>>>>> } finally {
>>>>>>> threads.remove(ti);
>>>>>>> end(rv > -1);
>>>>>>> assert IOStatus.check(rv);
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> here 'nd' is of type FileDispatcherImpl. I tried to look into its
>>>>

Re: Issue in GarbageCollectorThread compaction logic

[Matteo Merli]

My understanding is that fsync() forces the IO on the metadata even if the
file size has not changed, for example to update the access timestamp.

On Wed, May 10, 2017 at 4:47 PM Venkateswara Rao Jujjuri 
wrote:

> Let me try to consolidate the discussion.
>
> 1. We need to flush the metadata whenever we cause metadata changes due to
> data changes. Mostly Append/Filesize change.
> 2. I don't know any operations where we perform "metadata operation" only
> and we needed it to be "persisted"
> Even if we can think/come up with some obscure cases of #2, majority of of
> our use-case falls into #1.
> So why not put force(true) all the time? What cases we may incur penalty?
> If we pass true, then there is no debate on implementation/source.
>
> What do you guys say?
> JV
>
> On Wed, May 10, 2017 at 4:42 PM, Sijie Guo  wrote:
>
>>
>>
>> On Thu, May 11, 2017 at 2:28 AM, Charan Reddy G 
>> wrote:
>>
>>>  @JV and @Sijie I think Android java source is not the appropriate one
>>> to look into. For FileChannelImpl we need to look into jdk code
>>>
>>>
>>> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/sun/nio/ch/FileChannelImpl.java#FileChannelImpl.force%28boolean%29
>>>
>>>
>>> public void force(boolean metaData) throws IOException {
>>> ensureOpen();
>>> int rv = -1;
>>> int ti = -1;
>>> try {
>>> begin();
>>> ti = threads.add();
>>> if (!isOpen())
>>> return;
>>> do {
>>> rv = nd.force(fd, metaData);
>>> } while ((rv == IOStatus.INTERRUPTED) && isOpen());
>>> } finally {
>>> threads.remove(ti);
>>> end(rv > -1);
>>> assert IOStatus.check(rv);
>>> }
>>> }
>>>
>>> here 'nd' is of type FileDispatcherImpl. I tried to look into its code
>>> but since it is native code, we wont be able to read that
>>>
>>>
>>> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/sun/nio/ch/FileDispatcherImpl.java#FileDispatcherImpl
>>>
>>> So it is not clear if it is just difference of fdatasync() and fsync().
>>> Even if we assume that it is the difference of fdatasync() and fsync(),
>>> man pages of fdatasync() says
>>>
>>> "fdatasync() is similar to fsync(), but does not flush modified metadata
>>> unless that metadata is needed in order to allow a subsequent data
>>> retrieval to be correctly handled.   For  example, changes  to
>>>  st_atime or st_mtime (respectively, time of last access and time of last
>>> modification; see stat(2)) do not require flushing because they are not
>>> necessary for a subsequent data read to be handled correctly.  *On the
>>> other hand, a change to the file size (st_size, as made by say
>>> ftruncate(2)), would require a metadata flush.**"*
>>>
>>>
>> fdatasync - "does not flush modified metadata unless that metadata is
>> needed in order to allow a subsequent data retrieval to be correctly
>> handled"
>>
>> In practice, this means that unless the file size changed, the file
>> metadata will not be synced when calling fdatasync.
>>
>>
>>> Also, when we look into java API for force method -
>>> https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#force(boolean),
>>> it says the implementation is unspecified and system-dependent. So I don't
>>> think it is safe to make any assumptions here.
>>>
>>> "The metaData parameter can be used to limit the number of I/O
>>> operations that this method is required to perform. Passing false for
>>> this parameter indicates that only updates to the file's content need be
>>> written to storage; passing true indicates that updates to both the
>>> file's content and metadata must be written, which generally requires at
>>> least one more I/O operation. Whether this parameter actually has any
>>> effect is dependent upon the underlying operating system and is therefore
>>> unspecified. Invoking this method may cause an I/O operation to occur
>>> even if the channel was only opened for reading. Some operating systems,
>>> for example, maintain a last-access time as part of a file's metadata, and
>>> this time is updated whenever the file is read. Whether or not this is
>>> actually done is system-dependent and is therefore unspecified."
>>>
>>> Thanks,
>>> Charan
>>>
>>> On Wed, May 10, 2017 at 4:07 AM, Sijie Guo  wrote:
>>>


 On May 10, 2017 12:42 PM, "Venkateswara Rao Jujjuri" 
 wrote:

 Looked at the source and it is fdatasync()

 https://android.googlesource.com/platform/libcore/+/2496a68/luni/src/main/java/java/nio/FileChannelImpl.java
 if (metadata) {
 Libcore.os.fsync(fd);
 } else {
  Libcore.os.fdatasync(fd);
 }

 So, agreed. fdatasync() fluesh metadata when the file size etc changes,
 and avoids metadata flush if no data 

[jira] [Created] (BOOKKEEPER-1063) Use executure.execute() instead of submit() to avoid creation of unused FutureTask

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1063:


 Summary: Use executure.execute() instead of submit() to avoid 
creation of unused FutureTask
 Key: BOOKKEEPER-1063
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1063
 Project: Bookkeeper
  Issue Type: Improvement
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Minor
 Fix For: 4.5.0


When submitting tasks to an executor, if the {{FutureTask}} object is not being 
used we should use {{execute()}} instead of {{submit()}} in order to avoid the 
task object allocation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1062) AuditorLedgerCheckerTest intermittent failures

[Matteo Merli (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005502#comment-16005502
 ] 

Matteo Merli commented on BOOKKEEPER-1062:
--

Example from 
https://builds.apache.org/job/bookkeeper-master-git-pullrequest/364/

> AuditorLedgerCheckerTest intermittent failures
> --
>
> Key: BOOKKEEPER-1062
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1062
> Project: Bookkeeper
>  Issue Type: Test
>    Reporter: Matteo Merli
> Fix For: 4.5.0
>
>
> A couple of tests in {{AuditorLedgerCheckerTest}} keep failing from time to 
> time on Jenkins builds, irrespective of the PR changes.
> Example:
> {noformat}
> org.apache.bookkeeper.replication.AuditorLedgerCheckerTest.testReadOnlyBookieExclusionFromURLedgersCheck[4]
> org.apache.bookkeeper.replication.AuditorLedgerCheckerTest.testReadOnlyBookieShutdown[4]
> {noformat}
> {noformat}
> java.lang.AssertionError: latch should not have completed
>   at 
> org.apache.bookkeeper.replication.AuditorLedgerCheckerTest.testReadOnlyBookieExclusionFromURLedgersCheck(AuditorLedgerCheckerTest.java:283)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BOOKKEEPER-1054) Add gitignore file

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli resolved BOOKKEEPER-1054.
--
Resolution: Fixed

Issue resolved by merging pull request 144
[https://github.com/apache/bookkeeper/pull/144]

{noformat}
commit eeaddeda66fc687d20c2309a8cbc109c5dfeed89
Author: Matteo Merli <mme...@apache.org>
AuthorDate: Wed May 10 14:17:43 2017 -0700
Commit: Matteo Merli <mme...@apache.org>
CommitDate: Wed May 10 14:17:43 2017 -0700

BOOKKEEPER-1054: Add gitignore file

We should have a .gitignore file to hide all build generated files.

Author: Matteo Merli <mme...@apache.org>

Reviewers: Jia Zhai 

Closes #144 from merlimat/git-ignore

{noformat}


> Add gitignore file
> --
>
> Key: BOOKKEEPER-1054
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1054
> Project: Bookkeeper
>  Issue Type: Task
>Reporter: Matteo Merli
>Assignee: Matteo Merli
>Priority: Trivial
> Fix For: 4.5.0
>
>
> We should have a .gitignore file to hide all build generated files.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1061) BookieWatcher should not do ZK blocking operations from ZK async callback thread

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1061:


 Summary: BookieWatcher should not do ZK blocking operations from 
ZK async callback thread
 Key: BOOKKEEPER-1061
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1061
 Project: Bookkeeper
  Issue Type: Bug
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Minor
 Fix For: 4.5.0


In some cases, the BookieWatcher can get the ZK event thread stuck. This 
happens when a ZK blocking request is issued from a ZK callback thread. 

We should decouple the blocking requests in a separate executor to avoid 
deadlocking ZK client.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BOOKKEEPER-1060) Add utility to use SafeRunnable from Java8 Lambda

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli resolved BOOKKEEPER-1060.
--
Resolution: Fixed

Issue resolved by merging pull request 148
[https://github.com/apache/bookkeeper/pull/148]

{noformat}
commit af397e2247d453af25db6a1f33426fb9c69e2cdf
Author: Matteo Merli <mme...@apache.org>
AuthorDate: Wed May 10 13:47:02 2017 -0700
Commit: Matteo Merli <mme...@apache.org>
CommitDate: Wed May 10 13:47:02 2017 -0700

BOOKKEEPER-1060: Add utility to use SafeRunnable from Java8 Lambda

Since BK-4.5.0 is already switched to Java8, we should have a simple and 
concise way to pass lambdas where a `SafeRunnable` is required.

Author: Matteo Merli <mme...@apache.org>

Reviewers: Enrico Olivelli 

Closes #148 from merlimat/safe-runnable

{noformat}


> Add utility to use SafeRunnable from Java8 Lambda
> -
>
> Key: BOOKKEEPER-1060
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1060
> Project: Bookkeeper
>  Issue Type: Task
>Reporter: Matteo Merli
>Assignee: Matteo Merli
>Priority: Trivial
> Fix For: 4.5.0
>
>
> Since BK-4.5.0 is already switched to Java8, we should have a simple and 
> concise way to pass lambdas where a {{SafeRunnable}} is required.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Issue in GarbageCollectorThread compaction logic

[Matteo Merli]

> But the scenario in question is for EntryLogger.logChannel and
in GarbageCollectorThread.doCompactEntryLogs codepath. Here we just keep
adding entries to the EntryLogger.logChannel and flush it before deleting
the compacted entrylog.

But in that case, the entry log is just issuing the fdatasync() when the
flush happens, not for each entry appended


On Wed, May 10, 2017 at 11:47 AM Charan Reddy G <reddychara...@gmail.com>
wrote:

> Hey Matteo,
>
> I agree with what you said for JournalChannel
>
> JournalChannel.preAllocIfNeeded
>
> void preAllocIfNeeded(long size) throws IOException {
> if (bc.position() + size > nextPrealloc) {
> nextPrealloc += preAllocSize;
> zeros.clear();
> fc.write(zeros, nextPrealloc - journalAlignSize);
> }
> }
>
> But the scenario in question is for EntryLogger.logChannel and
> in GarbageCollectorThread.doCompactEntryLogs codepath. Here we just keep
> adding entries to the EntryLogger.logChannel and flush it before deleting
> the compacted entrylog.
>
> Thanks,
> Charan
>
> On Wed, May 10, 2017 at 11:35 AM, Matteo Merli <matteo.me...@gmail.com>
> wrote:
>
>> > On the other hand, a change to the file size (st_size, as made by say
>> ftruncate(2)), would require a metadata flush."
>>
>> This is why the Journal is pre-allocating the blocks instead of just
>> appending entries at the end of the file.
>>
>> That way the metadata is only synced each time you expand a block (eg:
>> every 16 MB)
>>
>>
>>
>> On Wed, May 10, 2017 at 11:28 AM Charan Reddy G <reddychara...@gmail.com>
>> wrote:
>>
>>>  @JV and @Sijie I think Android java source is not the appropriate one
>>> to look into. For FileChannelImpl we need to look into jdk code
>>>
>>>
>>> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/sun/nio/ch/FileChannelImpl.java#FileChannelImpl.force%28boolean%29
>>>
>>>
>>> public void force(boolean metaData) throws IOException {
>>> ensureOpen();
>>> int rv = -1;
>>> int ti = -1;
>>> try {
>>> begin();
>>> ti = threads.add();
>>> if (!isOpen())
>>> return;
>>> do {
>>> rv = nd.force(fd, metaData);
>>> } while ((rv == IOStatus.INTERRUPTED) && isOpen());
>>> } finally {
>>> threads.remove(ti);
>>> end(rv > -1);
>>> assert IOStatus.check(rv);
>>> }
>>> }
>>>
>>> here 'nd' is of type FileDispatcherImpl. I tried to look into its code
>>> but since it is native code, we wont be able to read that
>>>
>>>
>>> http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/sun/nio/ch/FileDispatcherImpl.java#FileDispatcherImpl
>>>
>>> So it is not clear if it is just difference of fdatasync() and fsync().
>>> Even if we assume that it is the difference of fdatasync() and fsync(),
>>> man pages of fdatasync() says
>>>
>>> "fdatasync() is similar to fsync(), but does not flush modified metadata
>>> unless that metadata is needed in order to allow a subsequent data
>>> retrieval to be correctly handled.   For  example, changes  to
>>>  st_atime or st_mtime (respectively, time of last access and time of last
>>> modification; see stat(2)) do not require flushing because they are not
>>> necessary for a subsequent data read to be handled correctly.  *On the
>>> other hand, a change to the file size (st_size, as made by say
>>> ftruncate(2)), would require a metadata flush.**"*
>>>
>>> Also, when we look into java API for force method -
>>> https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#force(boolean),
>>> it says the implementation is unspecified and system-dependent. So I don't
>>> think it is safe to make any assumptions here.
>>>
>>> "The metaData parameter can be used to limit the number of I/O
>>> operations that this method is required to perform. Passing false for
>>> this parameter indicates that only updates to the file's content need be
>>> written to storage; passing true indicates that updates to both the
>>> file's content and metadata must be written, which generally requires at
>>> least one more I/O operation. Whether this parameter actually has any
>>&

[jira] [Created] (BOOKKEEPER-1058) Ignore already deleted ledger on replication audit

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1058:


 Summary: Ignore already deleted ledger on replication audit
 Key: BOOKKEEPER-1058
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1058
 Project: Bookkeeper
  Issue Type: Bug
Reporter: Matteo Merli
Assignee: Matteo Merli
 Fix For: 4.5.0


Replication auditor should skip ledgers that were deleted since the auditing 
was started.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1057) Do not log error message after read failure in PendingReadOp

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1057:


 Summary: Do not log error message after read failure in 
PendingReadOp
 Key: BOOKKEEPER-1057
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1057
 Project: Bookkeeper
  Issue Type: Task
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Trivial
 Fix For: 4.5.0


In {{PendingReadOp}}, there is an error message that is printed each time a 
read on a specific bookie is failing: 

{noformat}
LOG.error("Read of ledger entry failed: L{} E{}-E{}, Heard from {}. First 
unread entry is {}",
new Object[] { lh.getId(), startEntryId, endEntryId, heardFromHosts, 
firstUnread });
{noformat}

This message is getting printed each time a ledger is recovered and there is no 
error, since the ledger recovery logic is to keep reading and incrementing the 
entryId until a NoEntry error is received.

This log message should be set at debug level.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1056) Removed PacketHeader serialization/deserialization allocation

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1056:


 Summary: Removed PacketHeader serialization/deserialization 
allocation
 Key: BOOKKEEPER-1056
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1056
 Project: Bookkeeper
  Issue Type: Improvement
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Minor
 Fix For: 4.5.0


When parsing the request packet header, use static methods to avoid creating a 
{{PacketHeader}} instance.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1055) Optimize handling of masterKey in case it is empty

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1055:


 Summary: Optimize handling of masterKey in case it is empty
 Key: BOOKKEEPER-1055
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1055
 Project: Bookkeeper
  Issue Type: Improvement
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Minor
 Fix For: 4.5.0


On each request client and bookies are exchanging the ledger masterKey, which 
is a 20 bytes MAC digest of the ledger password.

For each request there is a considerable overhead in allocating byte arrays 
when parsing the add/read requests. 

If the client is a passing an empty password, we should optimize the data path 
to skip all allocations (related to the masterKey) and instead rely on a static 
byte array.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BOOKKEEPER-1053) Upgrade RAT maven version to 0.12 and ignore Eclipse project files

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli resolved BOOKKEEPER-1053.
--
Resolution: Fixed

Issue resolved by merging pull request 143
[https://github.com/apache/bookkeeper/pull/143]

{noformat}
commit d8a7ac310947049f65687e33d92b847c44c91a39
Author: Matteo Merli <mme...@apache.org>
AuthorDate: Tue May 9 18:17:19 2017 -0700
Commit: Matteo Merli <mme...@apache.org>
CommitDate: Tue May 9 18:17:19 2017 -0700

BOOKKEEPER-1053: Upgrade RAT maven version to 0.12 and ignore Eclipse 
project files

Author: Matteo Merli <mme...@apache.org>

Reviewers: Jia Zhai 

Closes #143 from merlimat/rat

{noformat}


> Upgrade RAT maven version to 0.12 and ignore Eclipse project files
> --
>
> Key: BOOKKEEPER-1053
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1053
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Matteo Merli
>Assignee: Matteo Merli
>Priority: Trivial
> Fix For: 4.5.0
>
>
> RAT maven plugin should ignore existing Eclipse project files.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1054) Add gitignore file

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1054:


 Summary: Add gitignore file
 Key: BOOKKEEPER-1054
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1054
 Project: Bookkeeper
  Issue Type: Task
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Trivial
 Fix For: 4.5.0


We should have a .gitignore file to hide all build generated files.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1053) Upgrade RAT maven version to 0.12 and ignore Eclipse project files

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1053:


 Summary: Upgrade RAT maven version to 0.12 and ignore Eclipse 
project files
 Key: BOOKKEEPER-1053
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1053
 Project: Bookkeeper
  Issue Type: Improvement
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Trivial
 Fix For: 4.5.0


RAT maven plugin should ignore existing Eclipse project files.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BOOKKEEPER-1052) Print autorecovery enabled or not in bookie shell

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli resolved BOOKKEEPER-1052.
--
Resolution: Fixed

Issue resolved by merging pull request 142
[https://github.com/apache/bookkeeper/pull/142]

{noformat}
commit 5531014ab1cf282d35d4d53db975192b853bf7a6
Author: Siddharth Boobna <sboo...@yahoo-inc.com>
AuthorDate: Tue May 9 14:07:45 2017 -0700
Commit: Matteo Merli <mme...@apache.org>
CommitDate: Tue May 9 14:07:45 2017 -0700

BOOKKEEPER-1052: Print autorecovery enabled or not in bookie shell

Print the current status of auto-recovery, whether it's enabled or disabled.

Author: Siddharth Boobna <sboo...@yahoo-inc.com>

Reviewers: Enrico Olivelli 

Closes #142 from merlimat/autorecovery-enabled

{noformat}


> Print autorecovery enabled or not in bookie shell
> -
>
> Key: BOOKKEEPER-1052
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1052
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Matteo Merli
>Assignee: Matteo Merli
>Priority: Minor
> Fix For: 4.5.0
>
>
> Print the current status of auto-recovery, whether it's enabled or disabled.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BOOKKEEPER-1051) Fast shutdown for GarbageCollectorThread

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli resolved BOOKKEEPER-1051.
--
Resolution: Fixed

Issue resolved by merging pull request 141
[https://github.com/apache/bookkeeper/pull/141]

{noformat}
commit 1a2e017b65812e6e50975213a365aa37eba9fa96
Author: Matteo Merli <mme...@apache.org>
AuthorDate: Tue May 9 14:06:02 2017 -0700
Commit: Matteo Merli <mme...@apache.org>
CommitDate: Tue May 9 14:06:02 2017 -0700

BOOKKEEPER-1051: Fast shutdown for GarbageCollectorThread

Several unit tests are taking very long time to complete (eg: 
`BookieLedgerIndexTest` taking ~10 minutes).
The reason is that these tests are playing with the ZK quorum shutting it 
down and after the test succeeds, the shutdown phase is taking long time, since 
we try to do graceful shutdown with 1min wait time.
I think is better to interrupt immediately the garbage collector thread 
when shutting down the bookie.

Author: Matteo Merli <mme...@apache.org>

Reviewers: Enrico Olivelli 

Closes #141 from merlimat/fast-gc-thread-shutdown

{noformat}


> Fast shutdown for GarbageCollectorThread
> 
>
> Key: BOOKKEEPER-1051
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1051
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Matteo Merli
>Assignee: Matteo Merli
>Priority: Minor
> Fix For: 4.5.0
>
>
> Several unit tests are taking very long time to complete (eg: 
> {{BookieLedgerIndexTest}} taking ~10 minutes). 
> The reason is that these tests are playing with the ZK quorum shutting it 
> down and after the test succeeds, the shutdown phase is taking long time, 
> since we try to do graceful shutdown with 1min wait time. 
> I think is better to interrupt immediately the garbage collector thread when 
> shutting down the bookie.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1050) Cache journalFormatVersionToWrite when starting Journal

[Matteo Merli (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16003478#comment-16003478
 ] 

Matteo Merli commented on BOOKKEEPER-1050:
--

{noformat}
commit 61a5446b8923e5ce0b2a46891a385fd480275bac
Author: Matteo Merli <mme...@apache.org>
Date:   Tue May 9 13:24:53 2017 -0700

BOOKKEEPER-1050: Cache journalFormatVersionToWrite when starting Journal

Reading the journal version format from `ServiceConfiguration` each time is 
inefficient.
`ServiceConfiguration` is based on Java properties which is based on a 
String to object hashtable. Each read implies acquiring a mutex and converting 
from object to int.

Author: Matteo Merli <mme...@apache.org>

Reviewers: Enrico Olivelli 

Closes #140 from merlimat/cache-journal-conf
{noformat}

> Cache journalFormatVersionToWrite when starting Journal
> ---
>
> Key: BOOKKEEPER-1050
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1050
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Matteo Merli
>Assignee: Matteo Merli
>Priority: Minor
> Fix For: 4.5.0
>
>
> Reading the journal version format from {{ServiceConfiguration}} each time is 
> inefficient. 
> {{ServiceConfiguration}} is based on Java properties which is based on a 
> String to object hashtable. Each read implies acquiring a mutex and 
> converting from object to int.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1051) Fast shutdown for GarbageCollectorThread

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1051:


 Summary: Fast shutdown for GarbageCollectorThread
 Key: BOOKKEEPER-1051
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1051
 Project: Bookkeeper
  Issue Type: Improvement
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Minor
 Fix For: 4.5.0


Several unit tests are taking very long time to complete (eg: 
{{BookieLedgerIndexTest}} taking ~10 minutes). 
The reason is that these tests are playing with the ZK quorum shutting it down 
and after the test succeeds, the shutdown phase is taking long time, since we 
try to do graceful shutdown with 1min wait time. 

I think is better to interrupt immediately the garbage collector thread when 
shutting down the bookie.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1050) Cache journalFormatVersionToWrite when starting Journal

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1050:


 Summary: Cache journalFormatVersionToWrite when starting Journal
 Key: BOOKKEEPER-1050
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1050
 Project: Bookkeeper
  Issue Type: Improvement
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Minor
 Fix For: 4.5.0


Reading the journal version format from {{ServiceConfiguration}} each time is 
inefficient. 

{{ServiceConfiguration}} is based on Java properties which is based on a String 
to object hashtable. Each read implies acquiring a mutex and converting from 
object to int.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1048) Use ByteBuf in LedgerStorageInterface

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1048:


 Summary: Use ByteBuf in LedgerStorageInterface
 Key: BOOKKEEPER-1048
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1048
 Project: Bookkeeper
  Issue Type: Improvement
Reporter: Matteo Merli
Assignee: Matteo Merli
 Fix For: 4.5.0


To pass ref-counted buffer from Netty directly to the storage and the Journal, 
we need to have LedgerStorage to accept ByteBuf instead of ByteBuffer



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BOOKKEEPER-1046) Avoid long to Long conversion in OrderedSafeExecutor task submit

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli resolved BOOKKEEPER-1046.
--
Resolution: Fixed

> Avoid long to Long conversion in OrderedSafeExecutor task submit
> 
>
> Key: BOOKKEEPER-1046
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1046
> Project: Bookkeeper
>  Issue Type: Improvement
>    Reporter: Matteo Merli
>    Assignee: Matteo Merli
>Priority: Trivial
> Fix For: 4.5.0
>
>
> When submitting tasks to an OrderedSafeExecutor, most of the time a ledger id 
> is being passed. Given that the method accepts and Object, the primitive long 
> is boxed into a Long allocated on the heap.
> Added specific method overload to directly accept longs as the key in the 
> OrderedSafeExecutor.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1047) Add missing error code in ZK setData return path

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1047:


 Summary: Add missing error code in ZK setData return path
 Key: BOOKKEEPER-1047
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1047
 Project: Bookkeeper
  Issue Type: Bug
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Trivial
 Fix For: 4.5.0


The log warning is not printing the error code returned by ZooKeeper



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1046) Avoid long to Long conversion in OrderedSafeExecutor task submit

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1046:


 Summary: Avoid long to Long conversion in OrderedSafeExecutor task 
submit
 Key: BOOKKEEPER-1046
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1046
 Project: Bookkeeper
  Issue Type: Improvement
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Trivial
 Fix For: 4.5.0


When submitting tasks to an OrderedSafeExecutor, most of the time a ledger id 
is being passed. Given that the method accepts and Object, the primitive long 
is boxed into a Long allocated on the heap.

Added specific method overload to directly accept longs as the key in the 
OrderedSafeExecutor.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BOOKKEEPER-1045) Execute tests in different JVM processes

[Matteo Merli (JIRA)]

Matteo Merli created BOOKKEEPER-1045:


 Summary: Execute tests in different JVM processes
 Key: BOOKKEEPER-1045
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1045
 Project: Bookkeeper
  Issue Type: Test
Reporter: Matteo Merli
Assignee: Matteo Merli
Priority: Minor
 Fix For: 4.5.0


The current Maven Surefire configuration is using:

{code:xml}
always
{code}

This is a deprecated config and apparently it's not creating new processes for 
each test as intended. 

Currently the tests are leaking a big number of files and threads due to 
several reasons: 

 * Tests that instantiate bookies and call shutdown() without calling start() 
before are creating and initializing the ledger storage but not closing it, 
leaking threads and several fds
 * ZooKeeperClient sometimes doesn't shutdown the zk handle if the test 
completes too quickly, leaking sockets.
 * Several tests are passing bad config, so the bookie/client start gets 
exception (on purpose) and then doesn't clean up some partial objects.
 * ...

That make running the test suite to be dependent on ulimit of the machine. 

Until we can fix (almost) all the test to do proper cleanup, we should make 
maven to run tests in separated processes.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BOOKKEEPER-552) 64 Bits Ledger ID Generation

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli resolved BOOKKEEPER-552.
-
   Resolution: Fixed
Fix Version/s: 4.5.0



{noformat}
commit 057af8dbce6c08794eb8b46ca52ca13f222d9bbb
Author: Kyle Nusbaum <knusb...@yahoo-inc.com>
Date:   Tue May 2 14:04:57 2017 -0700

BOOKKEEPER-552: 64 Bits Ledger ID Generation

This PR supersedes #112

Instead of moving LongHierarchicalLedgerManager to 
HierarchicalLedgerManager, LongHierarchicalLedgerManager is still a stand-alone 
manager. HierarchicalLedgerManager has been moved to 
LegacyHierarchicalLedgerManager, and a new HierarchicalLedgerManager class has 
been put in its place, which is backwards-compatible with the original (now 
legacy) HierarchicalLedgerManager.

This new HierarchicalLedgerManager leverages the new 
LongZkLedgerIdGenerator to generate Ids, and uses the 
LongHierarchicalLedgerManager to manage metadata for ledger IDs > 31 bits long. 
For shorter ledger ids, the LegacyHierarchicalLedgerManager is used, to keep 
backwards-compatibility.

Author: Kyle Nusbaum <knusb...@yahoo-inc.com>

Reviewers: Enrico Olivelli <eolive...@gmail.com>, Matteo Merli 
<mme...@apache.org>, Venkateswararao Jujjuri (JV) 

Closes #114 from knusbaum/BOOKKEEPER-552

{noformat}

> 64 Bits Ledger ID Generation
> 
>
> Key: BOOKKEEPER-552
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-552
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client, bookkeeper-server, hedwig-server
>Reporter: Jiannan Wang
>Assignee: Kyle Nusbaum
> Fix For: 4.5.0
>
> Attachments: BOOKKEEPER-552.patch, BOOKKEEPER-552.patch, 
> BOOKKEEPER-552.patch
>
>
> This task aims to find and implement 64 bits global unique ledger id 
> generation mechanisms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [GitHub] bookkeeper issue #116: BOOKKEEPER-1008 Move to Netty4.1

[Matteo Merli]

> Kishore is seeing epoll compatibility issues with nio on netty 4.1.

What do you guys mean by incompatibility?
Netty can have the event loop to work either with the regular JDK nio
select or by using JNI to call epoll directly, but it should be either
everything goes with NIO or everything goes with epoll.


On Sun, Apr 30, 2017 at 7:57 AM Venkateswara Rao Jujjuri 
wrote:

> Kishore is seeing epoll compatibility issues with nio on netty 4.1.
> Kishore did that go away when you moved to 4.1.9 as suggested by
> @eolivelli?
>
>
> JV
>
> On Sat, Apr 29, 2017 at 6:38 AM, eolivelli  wrote:
>
> > Github user eolivelli commented on the issue:
> >
> > https://github.com/apache/bookkeeper/pull/116
> >
> > I got these other errors which can be due to some resource leak
> > ```
> >
>  
> recoverWithoutPasswordInConf[7](org.apache.bookkeeper.client.BookieRecoveryTest)
> > Time elapsed: 0.299 sec  <<< ERROR!
> > java.lang.IllegalStateException: failed to create a child event loop
> > at org.apache.bookkeeper.client.BookieRecoveryTest.
> > recoverWithoutPasswordInConf(BookieRecoveryTest.java:846)
> > Caused by: io.netty.channel.ChannelException: failed to open a new
> > selector
> > at org.apache.bookkeeper.client.BookieRecoveryTest.
> > recoverWithoutPasswordInConf(BookieRecoveryTest.java:846)
> > Caused by: java.io.IOException: Too many open files
> > at org.apache.bookkeeper.client.BookieRecoveryTest.
> > recoverWithoutPasswordInConf(BookieRecoveryTest.java:846)
> >
> > testSyncBookieRecoveryToSpecificBookie[7](org.apache.
> > bookkeeper.client.BookieRecoveryTest)  Time elapsed: 2.27 sec  <<< ERROR!
> > java.lang.IllegalStateException: failed to create a child event loop
> > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp(
> > BookieRecoveryTest.java:112)
> > Caused by: io.netty.channel.ChannelException: failed to open a new
> > selector
> > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp(
> > BookieRecoveryTest.java:112)
> > Caused by: java.io.IOException: Too many open files
> > at org.apache.bookkeeper.client.BookieRecoveryTest.setUp(
> > BookieRecoveryTest.java:112)
> >
> > ```
> >
> >
> > ---
> > If your project is set up for it, you can reply to this email and have
> your
> > reply appear on GitHub as well. If your project does not have this
> feature
> > enabled and wishes so, or if the feature is enabled but not working,
> please
> > contact infrastructure at infrastruct...@apache.org or file a JIRA
> ticket
> > with INFRA.
> > ---
> >
>
>
>
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi
>

Re: Publish SNAPSHOT artifacts on Maven Central

[Matteo Merli]

+1
On Fri, Apr 21, 2017 at 7:06 AM Enrico Olivelli  wrote:

> I wonder if we can start publishing the 4.5.0-SNAPSHOT binaries on
> Maven Central.
>
> This will help projects which want to try the upcoming version without
> dealing with custom local Maven Repositories.
>
> I see recently Apache Tomcat started doing so
> http://marc.info/?l=tomcat-dev=149269803006113=2
>
>
> I you all agree I would like to work on this.
>
>
> -- Enrico
>

[jira] [Updated] (BOOKKEEPER-1031) ReplicationWorker.rereplicate fails to call close() on ReadOnlyLedgerHandle

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli updated BOOKKEEPER-1031:
-
Fix Version/s: 4.5.0

> ReplicationWorker.rereplicate fails to call close() on ReadOnlyLedgerHandle
> ---
>
> Key: BOOKKEEPER-1031
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1031
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-auto-recovery
>Affects Versions: 4.4.0
>Reporter: Samuel Just
>Assignee: Samuel Just
> Fix For: 4.5.0
>
>
> This has the effect of permanently adding 1 listener per call into 
> AbstractZkLedgerManager.listenerSet



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BOOKKEEPER-1031) ReplicationWorker.rereplicate fails to call close() on ReadOnlyLedgerHandle

[Matteo Merli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Merli reassigned BOOKKEEPER-1031:


Assignee: Samuel Just

> ReplicationWorker.rereplicate fails to call close() on ReadOnlyLedgerHandle
> ---
>
> Key: BOOKKEEPER-1031
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1031
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-auto-recovery
>Affects Versions: 4.6.0
>Reporter: Samuel Just
>Assignee: Samuel Just
>
> This has the effect of permanently adding 1 listener per call into 
> AbstractZkLedgerManager.listenerSet



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)