[ACCOUNT] ZhangJian He added to the PMC of Apache BookKeeper

[Enrico Olivelli]

Hello everyone,
I am happy to announce that  ZhangJian He has accepted the invitation to
join the BookKeeper PMC.

More details about how the ASF works here:
https://apache.org/foundation/how-it-works/#roles

Congratulations ZhangJian !

Thank you for all your efforts to help the community and your contributions
to the project !

Enrico Olivelli

[ANNOUNCE] Lari Hotari as committer in BookKeeper

[Enrico Olivelli]

Hello everyone,
I am happy to announce that Lari Hotari has been invited by the PMC to
participate as a committer in our community and he accepted.

Congratulations Lari !

Note for new community members: you can find more about what it means to be
a "committer" here: https://apache.org/foundation/how-it-works/#roles

Best regards
Enrico Olivelli

Re: [ANNOUNCE] Apache BookKeeper 4.16.0 released

[Enrico Olivelli]

Congratulations !

This is a great step forward !

Looking forward to using DirectIO and Embedded Bookie !

Enrico

Il giorno mar 2 mag 2023 alle ore 09:22 Hang Chen
 ha scritto:
>
> The Apache BookKeeper team is proud to announce Apache BookKeeper
> version 4.16.0.
>
> Apache BookKeeper is a scalable, fault-tolerant, and low-latency
> storage service optimized for real-time workloads. It has been used
> for a fundamental service to build reliable services. It is also the
> log segment store for Apache DistributedLog and the message store for
> Apache Pulsar.
>
> This is the 46rd release of the Apache BookKeeper.
>
> Release 4.16.0 includes multiple important features, improvements, bug
> fixes and some dependencies CVE fixes.
>
> Due to this version having one critical regression in the BookKeeper
> client, Apache BookKeeper users are encouraged to skip this version
> and upgrade to 4.16.1.
>
> The technical details of this release are summarized below.
>
> For BookKeeper release details and downloads, visit:
> https://bookkeeper.apache.org/releases
>
> BookKeeper 4.16.0 Release Notes are at:
> https://bookkeeper.apache.org/release-notes/#4160
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The BookKeeper Team

Re: [ANNOUNCE] Apache BookKeeper 4.14.7 released

[Enrico Olivelli]

Congrats !
Enrico

Il giorno lun 20 feb 2023 alle ore 09:53 Hang Chen
 ha scritto:
>
> The Apache BookKeeper team is proud to announce Apache BookKeeper
> version 4.14.7.
>
> Apache BookKeeper is a scalable, fault-tolerant, and low-latency
> storage service optimized for real-time workloads. It has been used
> for a fundamental service to build reliable services. It is also the
> log segment store for Apache DistributedLog and the message store for
> Apache Pulsar.
>
> This is the 44rd release of the Apache BookKeeper.
>
> Release 4.14.7 contains performance and stability improvements and
> various bug fixes.
>
> Apache BookKeeper users are encouraged to upgrade to 4.14.7. The
> technical details of this release are summarized below.
>
> For BookKeeper release details and downloads, visit:
> https://bookkeeper.apache.org/releases
>
> BookKeeper 4.14.7 Release Notes are at:
> https://bookkeeper.apache.org/release-notes/#4147
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The BookKeeper Team

[ANNONCE] Apache BookKeeper releases prior to 4.14 end-of-life

[Enrico Olivelli]

After discussing within the community we decided to declare EOL all the
release lines before 4.14.

This means that all the release lines up to 4.13 won't see any more
releases, even for security issues.

Current active branches are:
- 4.14.x
- 4.15.x
- 4.16.x (no release has been cut yet)
- master


Users of previous version are encouraged to upgrade to the newer releases.

Best regards
Enrico

Re: something want to konw about Journal

[Enrico Olivelli]

Il Dom 21 Nov 2021, 16:15 735367737 <735367...@qq.com> ha scritto:

> hi：
>I'm a newer to bookkeeper. i read the Durability with BookKeeper
> 
> before i read bookkeeper's code. i got a question about journal.I paper it
> says journal is  write-ahead log and stores synchronously and sequentially
> all updates the bookie executes . but when i read code .i found the entry
> write into ledgerStorage before write into
> journal(Bookie.addEntryInternal).is there something i miss ?
>

It is actually not a problem because:
- entries are immutabile
- the reader is prevented from reading data not acknowledged by the writer
by the Last Add Confirmed protocol


Enrico

 thanks.
>

Re: [ANNOUNCE] New Committer: Yong Zhang

[Enrico Olivelli]

Congratulations!

Welcome aboard


Enrico

Il Gio 8 Lug 2021, 23:57 Sijie Guo  ha scritto:

> The Project Management Committee (PMC) for Apache BookKeeper has
> invited Yong Zhang
> to become a committer and we are pleased to announce that he has accepted!
>
> Congratulations and welcome onboard Yong Zhang! Looking forward to
> seeing more contributions from Yong!
>
> Please join us to welcome Yong Zhang!
>
> Sijie Guo
> on behalf of Apache BookKeeper PMC
>

Re: [ANNOUNCE] Apache BookKeeper 4.13.0 released

[Enrico Olivelli]

Thank you Andrey for driving the release.

We did lots of improvements on the BookKeeper Stream Storage service,
that's a KV distributed database bundled with BK.

I hope more people will start to use it soon

Enrico

Il giorno ven 26 feb 2021 alle ore 01:54 Wesley Peng
 ha scritto:
>
> great to hear that. thank you.
>
>
> On 26.02.2021 04:00, Andrey Yegorov wrote:
> > The Apache BookKeeper team is proud to announce Apache BookKeeper
> > version
> > 4.13.0.
> >
> > Apache BookKeeper is a scalable, fault-tolerant, and low-latency
> > storage service optimized for
> > real-time workloads. It has been used as a fundamental service to
> > build reliable services.
> > It is also the log segment store for Apache DistributedLog and the
> > message store for Apache Pulsar.
> >
> > This is the 25th release of the Apache BookKeeper.
> >
> > Release 4.13.0 improves the reliability of the Stream Storage, brings
> > additional configuration options for the Stream Storage and Prometheus
> > HTTP Server, fixes multiple bugs, and brings critical dependencies
> > up-to-date.
> >
> > For BookKeeper release details and downloads, visit:
> >
> > https://bookkeeper.apache.org/releases/
> >
> > BookKeeper 4.13.0 Release Notes are at:
> >
> > https://bookkeeper.apache.org/docs/4.13.0/overview/releaseNotes/
> >
> > We would like to thank the contributors that made the release
> > possible.
> >
> > Regards,
> > The BookKeeper Team

Re: [ANNOUNCE] Apache BookKeeper 4.12.0 released

[Enrico Olivelli]

Congrats !
Great to see these big steps for BK

Thank you Jia for driving the release

Enrico

Il giorno mar 17 nov 2020 alle ore 09:59 Jia Zhai  ha
scritto:

> The Apache BookKeeper team is proud to announce the Apache BookKeeper
> version
> 4.12.0.
>
> Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage
> service optimized for
> real-time workloads. It has been used for a fundamental service to build
> reliable services.
> It is also the log segment store for Apache DistributedLog and the message
> store for Apache Pulsar.
>
> This is the 23rd release of the Apache BookKeeper.
>
> Release highlights
> -  Enable ExplicitLAC but default on the reader side and in the New
> org.apache.bookkeeper.client.api.ReadHandle API
> - BP-41 Bookie Network Address Change Tracking + BookieId
> - BP-42 List and Access LedgerMetadata on the new API
> - Support Java 11 and switch to Java 11 default Docker images
> - BP-40 clean up output for tools
> - Certificate role based authorization
>
> For BookKeeper release details and downloads, visit:
> https://bookkeeper.apache.org/releases/
>
> BookKeeper 4.12.0 Release Notes are at:
>
> https://bookkeeper.apache.org/docs/4.12.0/overview/releaseNotes/
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The BookKeeper Team
>

Re: Moving BooKeeper to JDK11

[Enrico Olivelli]

FYI
Latest BK image is now running on OpenJDK11 by default
docker run apache/bookkeeper:latest

The same will be for BK 4.12, but in BK 4.12 we are able to also support
JRE distribution and not full JDK.

Enrico

Il giorno ven 23 ott 2020 alle ore 22:06 Enrico Olivelli <
eolive...@gmail.com> ha scritto:

>
>
> Il Ven 23 Ott 2020, 18:57 Anup Ghatage  ha scritto:
>
>> So I've been trying something similar.
>>
>> I noticed that most of the Bookkeeper codebase can build with JDK8 and
>> JDK11.
>> So I hacked around a bit and found that if I compile BK using JDK8, reset
>> JAVA_HOME to JDK11 and use 'mvn surefire:test', it just runs the tests
>> without compilation.
>> In this way, all of our unit tests run on JDK11 while being compiled on
>> JDK8. We already have 78% + coverage from unit tests which means that 78%+
>> code base will have no problems.
>>
>> When it comes to runtime, most obvious problems running on JDK11 are most
>> of the JDK8 GC options which have been retired so we can have scripts
>> which
>> deploy differently for different underlying JDK versions.
>>
>> Since we can run JDK8 compiled bits on JDK11, we could propose a phased
>> approach.
>> For the next two releases, ship both JDK8 and JDK11. Post that, we can
>> fully move to JDK11 for compilation and runtime.
>>
>
> We have been running Bookies and BK clients on JDK9-15 without issues.
> Also BK scripts support jdk11 and from 4.12 we will be also supporting
> JRE11+.
>
>
> Personally I would prefer to stay on bytecode compiled for JDK8, until we
> hit a time in which supporting it would be too costly.
>
> In HerdDB we are using multi release har in order to leverage modern jdk
> features (like new array comparisons, hotspot intrinsics).
> We could have this approach for CRC32C as well.
>
> IIRC we are also running a CI pull validation action with JDK11
>
> Enrico
>
>
>
>> Regards,
>> Anup
>>
>>
>>
>>
>> On Wed, Oct 21, 2020 at 11:49 PM Alessandro Luccaroni - Diennea
>>  wrote:
>>
>> > Hi,
>> > for the sake of the ZK project I hope they don’t go down that route
>> (I’ve
>> > already exposed my concern on the ZK mailing list) since there are many
>> > application in the wild that will be stuck on an older version of ZK.
>> >
>> > From my own company point of view we already support Java11 in all our
>> > applications so we are not directly impacted (and we have upgrade path
>> for
>> > older versions to provide to our customers).
>> > Since BK is a much younger project I don’t expect that the move to
>> Java11
>> > will be a problem. I think that Pulsar still supports Java8.
>> >
>> > Even in the case that the ZK team will decide to not go down that route
>> > (and still maintain Java8 support) I think that we should plan that move
>> > for BK to prevent a similar situation in the future (naturally, as soon
>> as
>> > Pulsar will drop support for Java8).
>> >
>> > Alessandro Luccaroni
>> > Platform Manager @ Diennea - MagNews
>> > Tel.: (+39) 0546 066100 Int. 924 - Mob.: (+39) 393 7273519
>> > Viale G.Marconi 30/14 - 48018 Faenza (RA) - Italy
>> >
>> > Da: Enrico Olivelli 
>> > Inviato: mercoledì 21 ottobre 2020 11:24
>> > A: Bookkeeper-Dev ; user <
>> > user@bookkeeper.apache.org>
>> > Oggetto: Moving BooKeeper to JDK11
>> >
>> > Hello guys,
>> > (crossposting to user and dev)
>> > In the ZooKeeper project there is an open discussion about moving
>> > ZooKeeper 3.7 to JDK11 and basically dropping support for JDK8 in the
>> > client.
>> >
>> > If they go that route we will probably need to move in the same
>> direction.
>> >
>> > How do you feel about this?
>> > Is JDK8 still required for your applications ?
>> >
>> > Best regards
>> > Enrico
>> >
>> >
>> > 
>> >
>> > CONFIDENTIALITY & PRIVACY NOTICE
>> > This e-mail (including any attachments) is strictly confidential and may
>> > also contain privileged information. If you are not the intended
>> recipient
>> > you are not authorised to read, print, save, process or disclose this
>> > message. If you have received this message by mistake, please inform the
>> > sender immediately and destroy this e-mail, its attachments and any
>> copies.
>> > Any use, distribution, reproduction or disclosure by any person other
>> than
>> > the intended recipient is strictly prohibited and the person responsible
>> > may incur in penalties.
>> > The use of this e-mail is only for professional purposes; there is no
>> > guarantee that the correspondence towards this e-mail will be read only
>> by
>> > the recipient, because, under certain circumstances, there may be a
>> need to
>> > access this email by third subjects belonging to the Company.
>> >
>>
>>
>> --
>> Anup Ghatage
>> www.ghatage.com
>>
>

BP-41 Separate BookieId from Bookie Network Address is ready

[Enrico Olivelli]

Hi BookKeepers,
(cross-posting to user@ and dev@)

I am happy to announce that most of the work has been committed regarding
BP-41
http://bookkeeper.apache.org/bps/BP-41-bookieid/

Basically the idea is that from BK 4.12.0 we are no longer referring to a
bookie using a network endpoint hostname:port but with a generic string
"BookieID".

The new behaviour is optional and totally compatible with older BK clients
and servers, you have to explicitly enable it in your Bookie configuration
(bk_server.conf).

Currently (up to 4.11.0) BookKeeper stores the "location" of each ledger in
the LedgerMetadata on ZooKeeper (or ETCD if you prefer).

This location was the network address of the bookie, and this was a big
problem if you want to use dynamic addresses for the Bookie or if you need
to move the Bookie to another network.

Now if you set "bookieId" configuration entry in bk_server.conf then the
clients will store such "id" (think about a UUID like xxx-xxx-xxx- or
some meaningful bookie name) in the metadata.
This way you can "move" the Bookie to a new address.

This change also opens the door to enabling multiple network endpoints for
a Bookie and multiple protocols (like pure TLS).

If you have time please check the current master and build it and try to
set a "bookieId" to your bookie and play with the tools and the clients.
Just changing the bookie port was enough to mess up BK 4.11.

- git clone https://github.com/apache/bookkeeper
- mvn clean install -DskipTests
- look inside bookkeeper-dist/server/target for the binaries

Enjoy BookKeeeper

Enrico

Re: Moving BooKeeper to JDK11

[Enrico Olivelli]

Il Ven 23 Ott 2020, 18:57 Anup Ghatage  ha scritto:

> So I've been trying something similar.
>
> I noticed that most of the Bookkeeper codebase can build with JDK8 and
> JDK11.
> So I hacked around a bit and found that if I compile BK using JDK8, reset
> JAVA_HOME to JDK11 and use 'mvn surefire:test', it just runs the tests
> without compilation.
> In this way, all of our unit tests run on JDK11 while being compiled on
> JDK8. We already have 78% + coverage from unit tests which means that 78%+
> code base will have no problems.
>
> When it comes to runtime, most obvious problems running on JDK11 are most
> of the JDK8 GC options which have been retired so we can have scripts which
> deploy differently for different underlying JDK versions.
>
> Since we can run JDK8 compiled bits on JDK11, we could propose a phased
> approach.
> For the next two releases, ship both JDK8 and JDK11. Post that, we can
> fully move to JDK11 for compilation and runtime.
>

We have been running Bookies and BK clients on JDK9-15 without issues.
Also BK scripts support jdk11 and from 4.12 we will be also supporting
JRE11+.


Personally I would prefer to stay on bytecode compiled for JDK8, until we
hit a time in which supporting it would be too costly.

In HerdDB we are using multi release har in order to leverage modern jdk
features (like new array comparisons, hotspot intrinsics).
We could have this approach for CRC32C as well.

IIRC we are also running a CI pull validation action with JDK11

Enrico



> Regards,
> Anup
>
>
>
>
> On Wed, Oct 21, 2020 at 11:49 PM Alessandro Luccaroni - Diennea
>  wrote:
>
> > Hi,
> > for the sake of the ZK project I hope they don’t go down that route (I’ve
> > already exposed my concern on the ZK mailing list) since there are many
> > application in the wild that will be stuck on an older version of ZK.
> >
> > From my own company point of view we already support Java11 in all our
> > applications so we are not directly impacted (and we have upgrade path
> for
> > older versions to provide to our customers).
> > Since BK is a much younger project I don’t expect that the move to Java11
> > will be a problem. I think that Pulsar still supports Java8.
> >
> > Even in the case that the ZK team will decide to not go down that route
> > (and still maintain Java8 support) I think that we should plan that move
> > for BK to prevent a similar situation in the future (naturally, as soon
> as
> > Pulsar will drop support for Java8).
> >
> > Alessandro Luccaroni
> > Platform Manager @ Diennea - MagNews
> > Tel.: (+39) 0546 066100 Int. 924 - Mob.: (+39) 393 7273519
> > Viale G.Marconi 30/14 - 48018 Faenza (RA) - Italy
> >
> > Da: Enrico Olivelli 
> > Inviato: mercoledì 21 ottobre 2020 11:24
> > A: Bookkeeper-Dev ; user <
> > user@bookkeeper.apache.org>
> > Oggetto: Moving BooKeeper to JDK11
> >
> > Hello guys,
> > (crossposting to user and dev)
> > In the ZooKeeper project there is an open discussion about moving
> > ZooKeeper 3.7 to JDK11 and basically dropping support for JDK8 in the
> > client.
> >
> > If they go that route we will probably need to move in the same
> direction.
> >
> > How do you feel about this?
> > Is JDK8 still required for your applications ?
> >
> > Best regards
> > Enrico
> >
> >
> > 
> >
> > CONFIDENTIALITY & PRIVACY NOTICE
> > This e-mail (including any attachments) is strictly confidential and may
> > also contain privileged information. If you are not the intended
> recipient
> > you are not authorised to read, print, save, process or disclose this
> > message. If you have received this message by mistake, please inform the
> > sender immediately and destroy this e-mail, its attachments and any
> copies.
> > Any use, distribution, reproduction or disclosure by any person other
> than
> > the intended recipient is strictly prohibited and the person responsible
> > may incur in penalties.
> > The use of this e-mail is only for professional purposes; there is no
> > guarantee that the correspondence towards this e-mail will be read only
> by
> > the recipient, because, under certain circumstances, there may be a need
> to
> > access this email by third subjects belonging to the Company.
> >
>
>
> --
> Anup Ghatage
> www.ghatage.com
>

[ANNOUNCE] Apache BookKeeper 4.11.1 released

[Enrico Olivelli]

The Apache BookKeeper team is proud to announce Apache BookKeeper version
4.11.1.

Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage
service optimized for
real-time workloads. It has been used for a fundamental service to build
reliable services.
It is also the log segment store for Apache DistributedLog and the message
store for Apache Pulsar.

This is the 22h release of the Apache BookKeeper.

Release highlights
- Upgrade Netty,Vertx and RocksDB
- Better error reporting in case of ZooKeeper related errors
- Fix error that prevents Garbage Collections in case of corrupted
EntryLogger file
- Support for Apache ZooKeeper 3.6.x

For BookKeeper release details and downloads, visit:

https://bookkeeper.apache.org/releases/

BookKeeper 4.11.1 Release Notes are at:

https://bookkeeper.apache.org/docs/4.11.1/overview/releaseNotes/

We would like to thank the contributors that made the release possible.

Regards,

The BookKeeper Team

Moving BooKeeper to JDK11

[Enrico Olivelli]

Hello guys,
(crossposting to user and dev)
In the ZooKeeper project there is an open discussion about moving ZooKeeper
3.7 to JDK11 and basically dropping support for JDK8 in the client.

If they go that route we will probably need to move in the same direction.

How do you feel about this?
Is JDK8 still required for your applications ?

Best regards
Enrico

ApacheCon is going to start

[Enrico Olivelli]

Hi BookKeepers,

Probably you already know that ApacheCon@Home is going to take place this
week.

Even if it will be a "virtual" conference this time, it will also be a
great chance to meet and see each other.

This is the list of Tracks
https://apachecon.com/acah2020/tracks/

As you can there are going to be talks about cool projects from people in
our community.

This is the link to the Slack workspace of the event
http://s.apache.org/apachecon-slack

Don't miss this chance of being part of this great ASF community event !

Attending to the event is free
https://hopin.to/events/apachecon-home#schedule

See you at ApacheCon@Home !

Enrico

Re: [ANNOUNCE] Apache BookKeeper 4.11.0 released

[Enrico Olivelli]

Thank you Rajan!

That's another big step forward for BK!


Enrico

Il Ven 10 Lug 2020, 23:48 Rajan Dhabalia  ha scritto:

> The Apache BookKeeper team is proud to announce Apache BookKeeper version
> 4.11.0.
>
> Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage
> service optimized for
> real-time workloads. It has been used for a fundamental service to build
> reliable services.
> It is also the log segment store for Apache DistributedLog and the message
> store for Apache Pulsar.
>
> This is the 21st release of the Apache BookKeeper.
>
> *News and noteworthy:*
> - Upgraded ZooKeeper version to 3.5.7
> - BookKeeper-server depends on org.apache.httpcomponents-httpcore-4.4.9
> - Publish Bookie Service info including all advertised addresses on
> Metadata Service
> - Add REST API to manage auto-recovery
> - Support metadata decoding for list-ledger api
>
>
> For BookKeeper release details and downloads, visit:
>
> https://bookkeeper.apache.org/releases/
>
> BookKeeper 4.11.0 Release Notes are at:
>
> https://bookkeeper.apache.org/docs/4.11.0/overview/releaseNotes/
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The BookKeeper Team
>

Re: Rollback of Apache Bookeeper

[Enrico Olivelli]

Subash
I will discuss about this topic on d...@bookkeeper.apache.org list
best regards
Enrico

Il giorno ven 17 apr 2020 alle ore 12:47 Subash K 
ha scritto:

> If rollback is technically supported by Bookkeeper, may be updating the
> website will help us better for now.
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Friday, April 17, 2020 3:44 PM
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
>
>
> Il Ven 17 Apr 2020, 11:59 Subash K  ha scritto:
>
> Sure. But as I’m just evaluating Apache Pulsar along with Bookkeeper, we
> are trying to understand various factors like Deployment, Life Cycle
> Management, etc.. Because these factors will be critical as it would create
> an impact on our existing product strategy.
>
>
>
> For now, we will take in that Rollback is not supported out of box. I feel
> that, rollback support should be brought in at the earliest to support
> organizations like us.
>
> In my opinion current releases of Bookkeeper support rollback.
>
> We do not have automated testing but we have this principle of being
> backward compatible with the previous version.
>
>
>
> Maybe we just lack an official statement on the website about this rule.
>
>
>
> Enrico
>
>
>
>
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Friday, April 17, 2020 3:16 PM
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
>
>
> Il Ven 17 Apr 2020, 10:59 Subash K  ha scritto:
>
> Thanks for your answers Enrico.
>
>
>
> On a high level, I understand that rollback is not supported and there is
> no plan to test and support it. But technically it is possible to do a
> rollback if new feature is not enabled after upgrade which is likely to
> change the file format.
>
>
>
> From the document, I see that there is option to rollback the file format
> after upgrade. Will this help us if we have to perform the rollback of data
> format as well?
>
>
>
> bookkeeper-server/bin/bookkeeper upgrade --rollback
>
> Actually that command was needed for very old versions of BK.
>
> I have been using it since 4.4 and never needed that command.
>
> In my opinion it is because since 4.4 BK payed more attention in making
> changes more safely
>
>
>
> A good option to you may be to check the version you are using and the one
> you want to upgrade to and if you have questions ask on this ML.
>
>
>
> We will be happy to support
>
>
>
> Enrico
>
>
>
>
>
>
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* 16 April 2020 19:47
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
>
>
>
>
> Il giorno gio 16 apr 2020 alle ore 15:01 Subash K 
> ha scritto:
>
> Hi Enrico,
>
>
>
> Thanks for your response.
>
>
>
> I’ve following queries on top of your response:
>
>- In general is community planning to test and support issues in
>rollback of Bookkeeper down the lane?
>
> Unfortunately no.
>
> We are only testing regular upgrades, not rollback.
>
> It should be doable with current docker based integration testing tools.
>
>
>
>
>
>
>-
>- We might not frequently upgrade Bookkeeper unless it is demanded by
>Pulsar. In that case, we might skip few intermediate versions and upgrade
>directly to an higher version. In that case, are we foreseeing any
>compatibility change that will cause hinderance during rollback?
>
> Usually this is not a problem. As far as you continue to upgrade from a
> supported version to a supported version. If I remember correctly currently
> we are officially supporting only BK 4.8 to 4.10.
>
>
>
>
>-
>- I’ve not fully gone through the internal working of Bookkeeper. But
>I would like to understand, can old version of bookie(after rollback)
>access the new data stored by upgraded bookie?
>
> Usually yes, as far as you do not activate new features that need a new
> format of data on disk
>
>
>
> Enrico
>
>
>
>
>
>
>-
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* 16 April 2020 16:05
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
> Hi Subash,
>
> usually by default there is no problem in rolling back to the previous
> version.
>
> For instance if you are running 4.9 and you upgrade to 4.10 your bookies
> you can downgrade to 4.9 just by using the old co

Re: Rollback of Apache Bookeeper

[Enrico Olivelli]

Il Ven 17 Apr 2020, 11:59 Subash K  ha scritto:

> Sure. But as I’m just evaluating Apache Pulsar along with Bookkeeper, we
> are trying to understand various factors like Deployment, Life Cycle
> Management, etc.. Because these factors will be critical as it would create
> an impact on our existing product strategy.
>
>
>
> For now, we will take in that Rollback is not supported out of box. I feel
> that, rollback support should be brought in at the earliest to support
> organizations like us.
>
In my opinion current releases of Bookkeeper support rollback.
We do not have automated testing but we have this principle of being
backward compatible with the previous version.

Maybe we just lack an official statement on the website about this rule.

Enrico



>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Friday, April 17, 2020 3:16 PM
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
>
>
> Il Ven 17 Apr 2020, 10:59 Subash K  ha scritto:
>
> Thanks for your answers Enrico.
>
>
>
> On a high level, I understand that rollback is not supported and there is
> no plan to test and support it. But technically it is possible to do a
> rollback if new feature is not enabled after upgrade which is likely to
> change the file format.
>
>
>
> From the document, I see that there is option to rollback the file format
> after upgrade. Will this help us if we have to perform the rollback of data
> format as well?
>
>
>
> bookkeeper-server/bin/bookkeeper upgrade --rollback
>
> Actually that command was needed for very old versions of BK.
>
> I have been using it since 4.4 and never needed that command.
>
> In my opinion it is because since 4.4 BK payed more attention in making
> changes more safely
>
>
>
> A good option to you may be to check the version you are using and the one
> you want to upgrade to and if you have questions ask on this ML.
>
>
>
> We will be happy to support
>
>
>
> Enrico
>
>
>
>
>
>
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* 16 April 2020 19:47
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
>
>
>
>
> Il giorno gio 16 apr 2020 alle ore 15:01 Subash K 
> ha scritto:
>
> Hi Enrico,
>
>
>
> Thanks for your response.
>
>
>
> I’ve following queries on top of your response:
>
>- In general is community planning to test and support issues in
>rollback of Bookkeeper down the lane?
>
> Unfortunately no.
>
> We are only testing regular upgrades, not rollback.
>
> It should be doable with current docker based integration testing tools.
>
>
>
>
>
>
>-
>- We might not frequently upgrade Bookkeeper unless it is demanded by
>Pulsar. In that case, we might skip few intermediate versions and upgrade
>directly to an higher version. In that case, are we foreseeing any
>compatibility change that will cause hinderance during rollback?
>
> Usually this is not a problem. As far as you continue to upgrade from a
> supported version to a supported version. If I remember correctly currently
> we are officially supporting only BK 4.8 to 4.10.
>
>
>
>
>    -
>    - I’ve not fully gone through the internal working of Bookkeeper. But
>I would like to understand, can old version of bookie(after rollback)
>access the new data stored by upgraded bookie?
>
> Usually yes, as far as you do not activate new features that need a new
> format of data on disk
>
>
>
> Enrico
>
>
>
>
>
>
>-
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* 16 April 2020 16:05
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
> Hi Subash,
>
> usually by default there is no problem in rolling back to the previous
> version.
>
> For instance if you are running 4.9 and you upgrade to 4.10 your bookies
> you can downgrade to 4.9 just by using the old code.
>
> This is because we introduce new features but they are disabled by default
> at least for one release.
>
>
>
> But I suggest you to perform a test about the rollback before going to
> production
>
>
>
> Enrico
>
>
>
>
>
> Il giorno gio 16 apr 2020 alle ore 12:23 Subash K 
> ha scritto:
>
> Hi,
>
>
>
> We are planning to introduce Apache Pulsar in our product. In general we
> always support upgrade and rollback of our product.
>
>
> So I'm trying to understand is there any defined steps provided to support
> rollback of Apache Bookeeper to its earlier version after completing full
> cluster upgrade? I was not able to find those information in
> https://bookkeeper.apache.org/docs/latest/admin/upgrade/
> <https://slack-redir.net/link?url=https%3A%2F%2Fbookkeeper.apache.org%2Fdocs%2Flatest%2Fadmin%2Fupgrade%2F=3>
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
>

Re: Rollback of Apache Bookeeper

[Enrico Olivelli]

Il Ven 17 Apr 2020, 10:59 Subash K  ha scritto:

> Thanks for your answers Enrico.
>
>
>
> On a high level, I understand that rollback is not supported and there is
> no plan to test and support it. But technically it is possible to do a
> rollback if new feature is not enabled after upgrade which is likely to
> change the file format.
>
>
>
> From the document, I see that there is option to rollback the file format
> after upgrade. Will this help us if we have to perform the rollback of data
> format as well?
>
>
>
> bookkeeper-server/bin/bookkeeper upgrade --rollback
>
Actually that command was needed for very old versions of BK.
I have been using it since 4.4 and never needed that command.
In my opinion it is because since 4.4 BK payed more attention in making
changes more safely

A good option to you may be to check the version you are using and the one
you want to upgrade to and if you have questions ask on this ML.

We will be happy to support

Enrico




>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* 16 April 2020 19:47
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
>
>
>
>
> Il giorno gio 16 apr 2020 alle ore 15:01 Subash K 
> ha scritto:
>
> Hi Enrico,
>
>
>
> Thanks for your response.
>
>
>
> I’ve following queries on top of your response:
>
>- In general is community planning to test and support issues in
>rollback of Bookkeeper down the lane?
>
> Unfortunately no.
>
> We are only testing regular upgrades, not rollback.
>
> It should be doable with current docker based integration testing tools.
>
>
>
>
>
>
>-
>- We might not frequently upgrade Bookkeeper unless it is demanded by
>Pulsar. In that case, we might skip few intermediate versions and upgrade
>directly to an higher version. In that case, are we foreseeing any
>compatibility change that will cause hinderance during rollback?
>
> Usually this is not a problem. As far as you continue to upgrade from a
> supported version to a supported version. If I remember correctly currently
> we are officially supporting only BK 4.8 to 4.10.
>
>
>
>
>-
>- I’ve not fully gone through the internal working of Bookkeeper. But
>I would like to understand, can old version of bookie(after rollback)
>access the new data stored by upgraded bookie?
>
> Usually yes, as far as you do not activate new features that need a new
> format of data on disk
>
>
>
> Enrico
>
>
>
>
>
>
>-
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* 16 April 2020 16:05
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
> Hi Subash,
>
> usually by default there is no problem in rolling back to the previous
> version.
>
> For instance if you are running 4.9 and you upgrade to 4.10 your bookies
> you can downgrade to 4.9 just by using the old code.
>
> This is because we introduce new features but they are disabled by default
> at least for one release.
>
>
>
> But I suggest you to perform a test about the rollback before going to
> production
>
>
>
> Enrico
>
>
>
>
>
> Il giorno gio 16 apr 2020 alle ore 12:23 Subash K 
> ha scritto:
>
> Hi,
>
>
>
> We are planning to introduce Apache Pulsar in our product. In general we
> always support upgrade and rollback of our product.
>
>
> So I'm trying to understand is there any defined steps provided to support
> rollback of Apache Bookeeper to its earlier version after completing full
> cluster upgrade? I was not able to find those information in
> https://bookkeeper.apache.org/docs/latest/admin/upgrade/
> <https://slack-redir.net/link?url=https%3A%2F%2Fbookkeeper.apache.org%2Fdocs%2Flatest%2Fadmin%2Fupgrade%2F=3>
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
>

Re: Rollback of Apache Bookeeper

[Enrico Olivelli]

Il giorno gio 16 apr 2020 alle ore 15:01 Subash K 
ha scritto:

> Hi Enrico,
>
>
>
> Thanks for your response.
>
>
>
> I’ve following queries on top of your response:
>
>- In general is community planning to test and support issues in
>rollback of Bookkeeper down the lane?
>
> Unfortunately no.
We are only testing regular upgrades, not rollback.
It should be doable with current docker based integration testing tools.



>
>-
>- We might not frequently upgrade Bookkeeper unless it is demanded by
>Pulsar. In that case, we might skip few intermediate versions and upgrade
>directly to an higher version. In that case, are we foreseeing any
>compatibility change that will cause hinderance during rollback?
>
> Usually this is not a problem. As far as you continue to upgrade from a
supported version to a supported version. If I remember correctly currently
we are officially supporting only BK 4.8 to 4.10.


>
>-
>- I’ve not fully gone through the internal working of Bookkeeper. But
>I would like to understand, can old version of bookie(after rollback)
>access the new data stored by upgraded bookie?
>
> Usually yes, as far as you do not activate new features that need a new
format of data on disk

Enrico



>
>-
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* 16 April 2020 16:05
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
> Hi Subash,
>
> usually by default there is no problem in rolling back to the previous
> version.
>
> For instance if you are running 4.9 and you upgrade to 4.10 your bookies
> you can downgrade to 4.9 just by using the old code.
>
> This is because we introduce new features but they are disabled by default
> at least for one release.
>
>
>
> But I suggest you to perform a test about the rollback before going to
> production
>
>
>
> Enrico
>
>
>
>
>
> Il giorno gio 16 apr 2020 alle ore 12:23 Subash K 
> ha scritto:
>
> Hi,
>
>
>
> We are planning to introduce Apache Pulsar in our product. In general we
> always support upgrade and rollback of our product.
>
>
> So I'm trying to understand is there any defined steps provided to support
> rollback of Apache Bookeeper to its earlier version after completing full
> cluster upgrade? I was not able to find those information in
> https://bookkeeper.apache.org/docs/latest/admin/upgrade/
> <https://slack-redir.net/link?url=https%3A%2F%2Fbookkeeper.apache.org%2Fdocs%2Flatest%2Fadmin%2Fupgrade%2F=3>
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
>

Re: Rollback of Apache Bookeeper

[Enrico Olivelli]

Hi Subash,
usually by default there is no problem in rolling back to the previous
version.
For instance if you are running 4.9 and you upgrade to 4.10 your bookies
you can downgrade to 4.9 just by using the old code.
This is because we introduce new features but they are disabled by default
at least for one release.

But I suggest you to perform a test about the rollback before going to
production

Enrico


Il giorno gio 16 apr 2020 alle ore 12:23 Subash K 
ha scritto:

> Hi,
>
>
>
> We are planning to introduce Apache Pulsar in our product. In general we
> always support upgrade and rollback of our product.
>
>
> So I'm trying to understand is there any defined steps provided to support
> rollback of Apache Bookeeper to its earlier version after completing full
> cluster upgrade? I was not able to find those information in
> https://bookkeeper.apache.org/docs/latest/admin/upgrade/
> 
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>

[ANNOUNCE] New BookKeeper Committer: Rajan Dhabalia

[Enrico Olivelli]

The Apache BookKeeper PMC recently extended committer karma to Rajan
Dhabalia and he has accepted.

Rajan Dhabalia has been making great contributions to bookkeeper
community and he also a committer in Apache Pulsar project.

It is great to have him onboard as bookkeeper committer.

We are looking forward to more contributions from him.

Congratulations and welcome onboard, Rajan !

Enrico on behave of the BookKeeper PMC

Re: How do stop current segment and create a new segment for the current ledger?

[Enrico Olivelli]

Il Sab 7 Mar 2020, 11:46 Wei Liu  ha scritto:

> I want to dynamic change ensemble size when added new bookie.
>

This is usually not needed.
Otherwise you can simply start a new ledger.
Btw it is very common in my experience.
Usually you build a stream made of ledgers and the application deals with
ledgers not with the internals of the ledger

Enrico


> Enrico Olivelli  于2020年3月7日周六 下午6:08写道：
>
>> Hi Liu,
>> This is done automatically when a bookie is down, this is called
>> 'ensemble change' in code.
>> You should not care about ledger segments, it is all automatic and BK
>> takes care of dealing it
>>
>> Can you please tell more about your case?
>>
>> Enrico
>>
>> Il Sab 7 Mar 2020, 10:42 Wei Liu  ha scritto:
>>
>>> How do stop current segment and create a new segment for the current
>>> ledger?
>>>
>>> Thanks~
>>>
>>> --
>>> 一个人只拥有今生今世是不够的，
>>> 他还应该拥有诗意的世界。
>>>
>>>liuwe...@gmail.com
>>>
>>
>
> --
> 一个人只拥有今生今世是不够的，
> 他还应该拥有诗意的世界。
>
>liuwe...@gmail.com
>

Re: How do stop current segment and create a new segment for the current ledger?

[Enrico Olivelli]

Hi Liu,
This is done automatically when a bookie is down, this is called 'ensemble
change' in code.
You should not care about ledger segments, it is all automatic and BK takes
care of dealing it

Can you please tell more about your case?

Enrico

Il Sab 7 Mar 2020, 10:42 Wei Liu  ha scritto:

> How do stop current segment and create a new segment for the current
> ledger?
>
> Thanks~
>
> --
> 一个人只拥有今生今世是不够的，
> 他还应该拥有诗意的世界。
>
>liuwe...@gmail.com
>

Re: How do I get an ledger id by Client API ？

[Enrico Olivelli]

Right!

Hope that helps

Enrico

Il Dom 23 Feb 2020, 03:40 Wei Liu  ha scritto:

> Thanks~
>
> I found "CreateAdvBuilder makeAdv()"
>
>
> Enrico Olivelli  于2020年2月23日周日 上午1:15写道：
>
>> Wei Liu,
>> If you want you can also set manually a ledger Id at creation time, but
>> you will have to use the 'WriteAdvHandle' interface
>>
>>
>> Enrico
>>
>> Il Sab 22 Feb 2020, 15:34 Wei Liu  ha scritto:
>>
>>> Thanks.
>>> I found it.
>>>
>>> Enrico Olivelli  于2020年2月22日周六 下午5:41写道：
>>>
>>>> Can you explain better? WriteHandle and ReadHandle has a getId() method
>>>>
>>>> Enrico
>>>>
>>>> Il Sab 22 Feb 2020, 10:26 Wei Liu  ha scritto:
>>>>
>>>>> How do I get an ledger id by  Client API ？
>>>>> Thanks~
>>>>>
>>>>> --
>>>>> 一个人只拥有今生今世是不够的，
>>>>> 他还应该拥有诗意的世界。
>>>>>
>>>>>liuwe...@gmail.com
>>>>>
>>>>
>>>
>>> --
>>> 一个人只拥有今生今世是不够的，
>>> 他还应该拥有诗意的世界。
>>>
>>>liuwe...@gmail.com
>>>
>>
>
> --
> 一个人只拥有今生今世是不够的，
> 他还应该拥有诗意的世界。
>
>liuwe...@gmail.com
>

Re: How do I get an ledger id by Client API ？

[Enrico Olivelli]

Wei Liu,
If you want you can also set manually a ledger Id at creation time, but you
will have to use the 'WriteAdvHandle' interface


Enrico

Il Sab 22 Feb 2020, 15:34 Wei Liu  ha scritto:

> Thanks.
> I found it.
>
> Enrico Olivelli  于2020年2月22日周六 下午5:41写道：
>
>> Can you explain better? WriteHandle and ReadHandle has a getId() method
>>
>> Enrico
>>
>> Il Sab 22 Feb 2020, 10:26 Wei Liu  ha scritto:
>>
>>> How do I get an ledger id by  Client API ？
>>> Thanks~
>>>
>>> --
>>> 一个人只拥有今生今世是不够的，
>>> 他还应该拥有诗意的世界。
>>>
>>>liuwe...@gmail.com
>>>
>>
>
> --
> 一个人只拥有今生今世是不够的，
> 他还应该拥有诗意的世界。
>
>liuwe...@gmail.com
>

Re: How do I get an ledger id by Client API ？

[Enrico Olivelli]

Can you explain better? WriteHandle and ReadHandle has a getId() method

Enrico

Il Sab 22 Feb 2020, 10:26 Wei Liu  ha scritto:

> How do I get an ledger id by  Client API ？
> Thanks~
>
> --
> 一个人只拥有今生今世是不够的，
> 他还应该拥有诗意的世界。
>
>liuwe...@gmail.com
>

Re: Mock BookKeeper for tests or downstream applications

[Enrico Olivelli]

Ivan,
Thank you for adding this experience

I knew about MockBookKeeper
I think that the major limit it that you have to hard code intoyour
application the selection this particular implementation.

In my approach you will be using the real BK client, so no change in
the application (as soon as you can configure both ClientConfiguration
and ServerConfiguration) but only selecting simpler implementations

Enrico

Il giorno ven 21 feb 2020 alle ore 15:46 Ivan Kelly 
ha scritto:
>
> > Does anyone have experience ?
> > Does anyone would like these features ?
> There's already something in the tree.
> https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/MockBookKeeper.java
>
> The problem with this is that sometimes the mock in incomplete or you
> want to trigger certain behaviour. If you rely on the mock in the
> bookkeeper release, you need to wait for a new release to get what you
> want. In most cases you're better off copying the mock into your own
> code and modifying it accordingly. Pulsar does this. In fact, this
> mock was originally part of pulsar, was copied into BK then copied
> back into pulsar when we realized some behaviour was broken (I think
> it was close behaviour at the time).
>
> -Ivan

Mock BookKeeper for tests or downstream applications

[Enrico Olivelli]

Hi,
I would like to ease the development of BookKeeper based applications.

Currently if you want to perform unit tests of your application you
have to start a ZooKeeper server and at least one Bookie.
This is more like an integration test ! and it is really heavy weight.

In order to get rid of ZooKeeper dependency in tests we could set up a
mock Metadata Driver that holds all Metadata in memory.
In order to not use network we can use the Netty Local Transport
option (already present in BK since 4.5)
We could have a mock LedgerStorageManager that holds data in memory
Then we only have to implement a mock Journal and all is done.


Does anyone have experience ?
Does anyone would like these features ?

I would move this conversation to dev@bookkeeper if there is any interest.


Cheers
Enrico

Re: mvn package is failing

[Enrico Olivelli]

:105)
>
> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
>
> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
>
> at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:62)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke (Method.java:498)
>
> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
> (Launcher.java:289)
>
> at org.codehaus.plexus.classworlds.launcher.Launcher.launch
> (Launcher.java:229)
>
> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
> (Launcher.java:415)
>
> at org.codehaus.plexus.classworlds.launcher.Launcher.main
> (Launcher.java:356)
>
> Caused by: org.apache.tools.ant.BuildException: Could not launch g++:
> java.io.IOException: Cannot run program "g++" (in directory
> "/home/prabhu/final_git_bk/bookkeeper/circe-checksum/src/main"): error=2,
> No such file or directory
>
> at com.github.maven_nar.cpptasks.CUtil.runCommand (CUtil.java:422)
>
> at
> com.github.maven_nar.cpptasks.compiler.CommandLineCompiler.runCommand
> (CommandLineCompiler.java:568)
>
> at com.github.maven_nar.cpptasks.compiler.CommandLineCompiler.compile
> (CommandLineCompiler.java:240)
>
> at
> com.github.maven_nar.cpptasks.compiler.CommandLineCompilerConfiguration.compile
> (CommandLineCompilerConfiguration.java:164)
>
> at com.github.maven_nar.cpptasks.CCTask$Core.run (CCTask.java:107)
>
> [ERROR]
>
> [ERROR]
>
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
>
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
>
> [ERROR]
>
> [ERROR] After correcting the problems, you can resume the build with the
> command
>
> [ERROR]   mvn  -rf :circe-checksum
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Thursday, December 19, 2019 1:14 PM
> *To:* user
> *Subject:* Re: mvn package is failing
>
>
>
> [EXTERNAL EMAIL]
>
> If you run
>
> gcc --version
>
> or
>
> /bin/sh -c 'gcc' '--version'
>
>
>
> is is working ?
>
>
>
> maybe you can try to use -X Maven flag to see more debug about the Maven
> Nar Plugin
>
> mvn clean package -DskipTests -X
>
>
>
> Enrico
>
>
>
>
>
> Il giorno gio 19 dic 2019 alle ore 08:40  ha
> scritto:
>
> Thanks Enrico for the quick reply, I am running this on a ubuntu and I do
> have gcc as well still I am getting this error.
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Thursday, December 19, 2019 1:05 PM
> *To:* user
> *Subject:* Re: mvn package is failing
>
>
>
> [EXTERNAL EMAIL]
>
> You need Linux or Mac
>
> You also have to install gcc
>
>
>
>
>
> Enrico
>
>
>
> Il gio 19 dic 2019, 07:27  ha scritto:
>
> Hi team,
>
>
>
> I am working for a project which is using apache bookkeeper, I have cloned
> the bookkeeper code and am following the steps to build that are given in
> the link
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Developer+Setup#DeveloperSetup-Build
>  but
> when I am trying to build it by using *mvn clean package -DskipTests, *I
> am getting the following *error [ERROR] Failed to execute goal
> com.github.maven-nar:nar-maven-plugin:3.5.2:nar-validate
> (default-nar-validate) on project circe-checksum: Could not launch /bin/sh
> -c 'gcc' '--version': Error while executing process. Cannot run program
> "gcc": error=2, No such file or directory -> [Help 1] *. What should I do
> to solve this?
>
>
>
>
>
> Thanks,
>
> Prabhaker Saxena
>
>
>
>

Re: Can't create ledger

[Enrico Olivelli - Diennea]

It depends on your log library, BookKeeper uses slf4j that is a wrapper over 
your own logging library, I see logs at level INFO, ERROR and WARN
I am sorry I don’t know
Which library are you using ? log4j, logback ?

Enrico

Il giorno 04/12/19, 11:34 "Wei Liu" 
mailto:liuwe...@gmail.com>> ha scritto:

How to enable debug log level?

Enrico Olivelli - Diennea 
mailto:enrico.olive...@diennea.com>> 于2019年12月4日周三 
下午6:18写道：
From the logs we see that the client is able to perform discovery of bookies 
and in order to perform this operation it must read from ZooKeeper.

Do you see errors on ZooKeeper server side ?

Can you enable DEBUG log level on your client application ?
Enrico

Il giorno 04/12/19, 10:50 "Wei Liu" 
mailto:liuwe...@gmail.com>> ha scritto:

I use the zkCli.sh client to see that there is no new id under this path 
"/ledgers/00/".

Flavio Junqueira mailto:f...@apache.org>> 于2019年12月4日周三 
下午5:48写道：
Do you know if the ledger znode has been created in zk? Maybe use the zkCli to 
check it.

-Flavio

On 4 Dec 2019, at 10:35, Wei Liu 
mailto:liuwe...@gmail.com>> wrote:

Dear Enrico:

   1) I don't see the new ledger id on zk

   2)It looks like the server is not return?
   at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at 
org.apache.bookkeeper.client.SyncCallbackUtils.waitForResult(SyncCallbackUtils.java:55)
at org.apache.bookkeeper.client.BookKeeper.createLedger(BookKeeper.java:918)
at org.apache.bookkeeper.client.BookKeeper.createLedger(BookKeeper.java:870)
at org.apache.bookkeeper.client.BookKeeper.createLedger(BookKeeper.java:851)
at com.mytest.App.CreateLedger(App.java:38)
at com.mytest.App.main(App.java:259)
   3) Below is jvm stack:

  Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed 
mode):

"Attach Listener" #23 daemon prio=9 os_prio=0 tid=0x7fbc18001000 nid=0x5636 
waiting on condition [0x]
   java.lang.Thread.State: RUNNABLE
"main-EventThread" #21 daemon prio=5 os_prio=0 tid=0x7fbc60645800 
nid=0x55ff waiting on condition [0x7fbc38c6d000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00076ddd7868> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:504)
"main-SendThread(10.209.242.147:2181<http://10.209.242.147:2181/>)" #20 daemon 
prio=5 os_prio=0 tid=0x7fbc60649000 nid=0x55fe runnable [0x7fbc38d6e000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00076dd86f50> (a sun.nio.ch.Util$3)
- locked <0x00076dd86ec8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00076dd85b30> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
"BookKeeperClientWorker-OrderedExecutor-7-0" #19 prio=5 os_prio=0 
tid=0x7fbc605ce800 nid=0x55fd waiting on condition [0x7fbc39298000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00076dac6570> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
"BookKeeperClientWorker-OrderedExecutor-6-0&q

Re: Can't create ledger

[Enrico Olivelli - Diennea]

afe.park(Native Method)
- parking to wait for  <0x00076f04d0b8> (a 
java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at 
org.apache.bookkeeper.client.SyncCallbackUtils.waitForResult(SyncCallbackUtils.java:55)
at org.apache.bookkeeper.client.BookKeeper.createLedger(BookKeeper.java:918)
at org.apache.bookkeeper.client.BookKeeper.createLedger(BookKeeper.java:870)
at org.apache.bookkeeper.client.BookKeeper.createLedger(BookKeeper.java:851)
at com.mytest.App.CreateLedger(App.java:38)
at com.mytest.App.main(App.java:259)
"VM Thread" os_prio=0 tid=0x7fbc601c8800 nid=0x55eb runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x7fbc60021000 nid=0x55e3 
runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x7fbc60023000 nid=0x55e4 
runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x7fbc60024800 nid=0x55e5 
runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x7fbc60026800 nid=0x55e6 
runnable

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x7fbc60028000 nid=0x55e7 
runnable

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x7fbc6002a000 nid=0x55e8 
runnable

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x7fbc6002b800 nid=0x55e9 
runnable

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x7fbc6002d800 nid=0x55ea 
runnable

"VM Periodic Task Thread" os_prio=0 tid=0x7fbc6021e000 nid=0x55f4 waiting 
on condition

JNI global references: 392

Enrico Olivelli - Diennea 
mailto:enrico.olive...@diennea.com>> 于2019年12月4日周三 
下午3:48写道：
Liu,

From your logs I see that the client is able to reach ZooKeeper, so I think 
that createLedger should complete.
Are you sure that your program is not stuck at “addEntry” ?
Is your client machine able to connect to the bookies ?
You can dump the stacktrace of the JVM with jstack

I suggest you to use the org.apache.bookkeeper.client.api package, the code is 
like this (I don’t have an IDE now, I am not sure it compiles, but it gives an 
idea):

org.apache.bookkeeper.client.api.Bookkeeper bookkeeper = 
org.apache.bookkeeper.client.api.Bookkeeper.forConfig(config).build();
org.apache.bookkeeper.client.api.WriteHandle handle = 
bookkeeper.newCreateLedgerOp()
  
.withEnsembleSize(1)
  
.withWriteQuorumSize(1)

.withAckQuorumSize(1)

.withDigestType(DigestType.CRC32C)
 .execute()
 .get();
  long entryId = handle.append("Some entry data".getBytes());
  long entryId1 = handle. append ("111Some entry data".getBytes());
  long entryId2 = handle. append ("222Some entry data".getBytes());

   // always close a WriteHandle !!!
  handle.close();


Enrico



Il giorno 04/12/19, 03:43 "Wei Liu" 
mailto:liuwe...@gmail.com>> ha scritto:

Dear All:

I can't create ledger using bookkeeper 4.10.0 version.

Below is my test code:

 static String connectionString = "xxx.xxx.xxx.xxx:2181";
 bkClient = new BookKeeper(connectionString);

  byte[] password = "some-password".getBytes();
  handle = bkClient.createLedger(BookKeeper.DigestType.MAC, password);

  long entryId = handle.addEntry("Some entry data".getBytes());
  long entryId1 = handle.addEntry("111Some entry data".getBytes());
  long entryId2 = handle.addEntry("222Some entry data".getBytes());

Stuck in 'createLedger' when the program is running.

 Below is log:
 2019-12-04 10:13:39,236 - INFO  - [main:MetadataDrivers@107] - BookKeeper 
metadata driver manager initialized
2019-12-04 10:13:39,239 - INFO  - [main:MetadataDrivers@107] - BookKeeper 
metadata driver manager initialized
2019-12-04 10:13:39,240 - INFO  - [main:MetadataDrivers@107] - BookKeeper 
metadata driver manager initialized
2019-12-04 10:13:39,247 - INFO  - [main:ZKMetadataDriverBase@192] - Initialize 
zookeeper metadata driver at metadata service uri 
zk+null://xxx..xxx.xxx:2181/ledgers : zkServers = xxx..xxx.xxx, 
ledgersRootPath = /ledgers.
2019-12-04 10:13:39,288 - INFO  - [main-EventThread:ZooKeeperWatcherBase@130] - 
ZooKeeper client is connected now.
2019-12-04 10:13:39,411 - 

Re: Can't create ledger

[Enrico Olivelli - Diennea]

Liu,

From your logs I see that the client is able to reach ZooKeeper, so I think 
that createLedger should complete.
Are you sure that your program is not stuck at “addEntry” ?
Is your client machine able to connect to the bookies ?
You can dump the stacktrace of the JVM with jstack

I suggest you to use the org.apache.bookkeeper.client.api package, the code is 
like this (I don’t have an IDE now, I am not sure it compiles, but it gives an 
idea):

org.apache.bookkeeper.client.api.Bookkeeper bookkeeper = 
org.apache.bookkeeper.client.api.Bookkeeper.forConfig(config).build();
org.apache.bookkeeper.client.api.WriteHandle handle = 
bookkeeper.newCreateLedgerOp()
  
.withEnsembleSize(1)
  
.withWriteQuorumSize(1)

.withAckQuorumSize(1)

.withDigestType(DigestType.CRC32C)
 .execute()
 .get();
  long entryId = handle.append("Some entry data".getBytes());
  long entryId1 = handle. append ("111Some entry data".getBytes());
  long entryId2 = handle. append ("222Some entry data".getBytes());

   // always close a WriteHandle !!!
  handle.close();


Enrico



Il giorno 04/12/19, 03:43 "Wei Liu" 
mailto:liuwe...@gmail.com>> ha scritto:

Dear All:

I can't create ledger using bookkeeper 4.10.0 version.

Below is my test code:

 static String connectionString = "xxx.xxx.xxx.xxx:2181";
 bkClient = new BookKeeper(connectionString);

  byte[] password = "some-password".getBytes();
  handle = bkClient.createLedger(BookKeeper.DigestType.MAC, password);

  long entryId = handle.addEntry("Some entry data".getBytes());
  long entryId1 = handle.addEntry("111Some entry data".getBytes());
  long entryId2 = handle.addEntry("222Some entry data".getBytes());

Stuck in 'createLedger' when the program is running.

 Below is log:
 2019-12-04 10:13:39,236 - INFO  - [main:MetadataDrivers@107] - BookKeeper 
metadata driver manager initialized
2019-12-04 10:13:39,239 - INFO  - [main:MetadataDrivers@107] - BookKeeper 
metadata driver manager initialized
2019-12-04 10:13:39,240 - INFO  - [main:MetadataDrivers@107] - BookKeeper 
metadata driver manager initialized
2019-12-04 10:13:39,247 - INFO  - [main:ZKMetadataDriverBase@192] - Initialize 
zookeeper metadata driver at metadata service uri 
zk+null://xxx..xxx.xxx:2181/ledgers : zkServers = xxx..xxx.xxx, 
ledgersRootPath = /ledgers.
2019-12-04 10:13:39,288 - INFO  - [main-EventThread:ZooKeeperWatcherBase@130] - 
ZooKeeper client is connected now.
2019-12-04 10:13:39,411 - ERROR - 
[main:RackawareEnsemblePlacementPolicyImpl@267] - Failed to initialize DNS 
Resolver org.apache.bookkeeper.net.ScriptBasedMapping, used default subnet 
resolver : java.lang.RuntimeException: No network topology script is found when 
using script based DNS resolver.
2019-12-04 10:13:39,430 - INFO  - 
[main:RackawareEnsemblePlacementPolicyImpl@214] - Initialize rackaware ensemble 
placement policy @ http://127.0.1.1:0>> @ /default-rack : 
org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$DefaultResolver.
2019-12-04 10:13:39,430 - INFO  - 
[main:RackawareEnsemblePlacementPolicyImpl@224] - Not weighted
2019-12-04 10:13:39,433 - INFO  - [main:BookKeeper@509] - Weighted ledger 
placement is not enabled
2019-12-04 10:13:39,458 - INFO  - 
[BookKeeperClientScheduler-OrderedScheduler-0-0:NetworkTopologyImpl@429] - 
Adding a new node: /default-rack/xxx..xxx.xxx:3181
2019-12-04 10:13:39,789 - INFO  - 
[BookKeeperClientScheduler-OrderedScheduler-0-0:NetworkTopologyImpl@429] - 
Adding a new node: /default-rack/xxx..xxx.xxx:3181
2019-12-04 10:13:39,791 - INFO  - 
[BookKeeperClientScheduler-OrderedScheduler-0-0:NetworkTopologyImpl@429] - 
Adding a new node: /default-rack/xxx..xxx.xxx:4181
2019-12-04 10:13:40,123 - INFO  - 
[BookKeeperClientScheduler-OrderedScheduler-0-0:NetworkTopologyImpl@429] - 
Adding a new node: /default-rack/xxx..xxx.xxx:3181
1
2019-12-04 10:13:40,153 - WARN  - [main:BookieWatcherImpl@240] - New ensemble: 
[xxx..xxx.xxx:4181, xxx..xxx.xxx:3181, xxx..xxx.xxx:3181] is not 
adhering to Placement Policy. quarantinedBookies: []
--
一个人只拥有今生今世是不够的，
他还应该拥有诗意的世界。

   liuwe...@gmail.com



CONFIDENTIALITY & PRIVACY NOTICE
This e-mail (including any attachments) is strictly confidential and may also 
contain privileged information. If you are not the intended recipient you are 
not authorised to read, print, save, process or disclose this message. If you 
have received this message by mistake, please inform the sender immediately and 
destroy this e-mail, its attachments and 

Re: Bookie graceful shutdown and restart

[Enrico Olivelli]

Il giorno mar 3 dic 2019 alle ore 11:41  ha
scritto:

> Hi Bookkeeper Team,
>
>
>
> We’re currently running into an issue where on bookie restart it is not
> able to find certain entries in ledgers and hence reads for these entries
> keep failing.
>
> In most cases, the bookie is able to recover from these errors after a
> period and things work as usual.
>
> However, in a case where all  bookies are deleted simultaneously, the
> recovery never happens and we’re stuck in a state were we see data loss.
>
>
>
> I needed to understand what is the guidance to ensure graceful termination
> of a bookie?
>
> What is the right way to handle bookie restarts?
>

The Bookie handles both graceful stops and machine crashes without need of
a manual operation.


> Is there an option to move a bookie to a “readonly” mode prior to shutdown
> so we can sort of switch off writes?
>

This is done automatically in recent BookKeeper versions (IIRC from 4.8+)


>
>
> Is auto recovery supposed to correct the missing data in a bookie post
> restart?
>

I suppose you run into some kind of bug,
but autorecovery will write again the missing entries on the Bookie, so it
should fix the problem. a copy of the entry must be available on another
Bookie (up and running)



> In this case the solve could be to run manual recovery every time a bookie
> restarts..
>

Enrico


>
>
> -Thanks,
>
> Prajakta
>
>
>
>
>

Re: Bookeeper exception on pods restart

[Enrico Olivelli]

sh-4.2# ./bookkeeper shell ledger -m 56
ERROR: invalid ledger id 56
ledger: Dump ledger index entries into readable format.
usage: ledger   [-m] 
 -m,--meta   Print meta information

Does it work for other ledgers ?

Enrico



Il giorno lun 2 dic 2019 alle ore 10:06 Sharda, Ravi 
ha scritto:

> Hello Sijie,
>
> Any luck with this? Please let us know what could be going wrong.
>
> Thanks & best regards,
> Ravi
> --
> *From:* Sharda, Ravi 
> *Sent:* Friday, November 29, 2019 3:28 PM
> *To:* Sijie Guo 
> *Cc:* user ; Flavio Junqueira 
> *Subject:* Re: Bookeeper exception on pods restart
>
> Thanks. Here's the output of the command:
>
> sh-4.2# ./bookkeeper shell ledger -m 56
> ERROR: invalid ledger id 56
> ledger: Dump ledger index entries into readable format.
> usage: ledger   [-m] 
>  -m,--meta   Print meta information
> --
> *From:* Sijie Guo 
> *Sent:* Friday, November 29, 2019 3:15 PM
> *To:* Sharda, Ravi 
> *Cc:* user ; Flavio Junqueira 
> *Subject:* Re: Bookeeper exception on pods restart
>
>
> [EXTERNAL EMAIL]
> Sorry, my bad. The command for reading ledger index should be "bookkeeper
> shell ledger".
>
> From the `ls` output, I didn't find entry 1.log under ledgers directory.
> So I guess the log file doesn't exist. If you can provide the output of
> `bookkeeper shell ledger`, we can take a look at the index file to
> understand more.
>
> On Fri, Nov 29, 2019 at 1:20 AM Sharda, Ravi  wrote:
>
> For the following error,
>
> OrderedExecutor-0-0:ReadEntryProcessorV3@235] - IOException while reading
> entry: 25 from ledger 56
>
> java.io.FileNotFoundException: No file for log 1 for 56 with location
> 4744138143
>
> at org.apache.bookkeeper.bookie.EntryLogger.findFile(EntryLogger.java:1165)
>
> at
> org.apache.bookkeeper.bookie.EntryLogger.getChannelForLogId(EntryLogger.java:1100)
>
> at
> org.apache.bookkeeper.bookie.EntryLogger.internalReadEntry(EntryLogger.java:1002)
>
> at
> org.apache.bookkeeper.bookie.EntryLogger.readEntry(EntryLogger.java:1051)
>
> at
> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:305)
>
> at
> org.apache.bookkeeper.bookie.SortedLedgerStorage.getEntry(SortedLedgerStorage.java:153)
>
> at org.apache.bookkeeper.bookie.L
> ---
> sh-4.2# ./bookkeeper shell readledger --ledgerid 56
>
> ERROR: invalid value for option ledgerid : 56
> Must specify a ledger id
>
> ---
> I didn't know how to check that the log file exists. Attaching the output
> of "ls -R -L", instead.
>
> --
> *From:* Sijie Guo 
> *Sent:* Friday, November 29, 2019 2:31 PM
> *To:* Sharda, Ravi 
> *Cc:* user ; Flavio Junqueira 
> *Subject:* Re: Bookeeper exception on pods restart
>
>
> [EXTERNAL EMAIL]
> If it is a permanent error,
>
> - check if the log file (indicated in the error message) exists or not.
> - use `bookkeeper shell readledger` to dump the index of the given ledger.
> - see if the index points to the right entry log file or not
>
> - Sijie
>
> On Fri, Nov 29, 2019 at 12:46 AM Sharda, Ravi 
> wrote:
>
> The latest instance we have seen is a permanent error. The bookies haven't
> recovered in the environment (last 2 days). In some previous instances,
> developers had reported that the bookies had recovered, but it is also
> possible that the error was slightly different from what we are seeing now.
>
> Thanks & best regards,
> Ravi
> --
> *From:* Sijie Guo 
> *Sent:* Friday, November 29, 2019 2:10 PM
> *To:* user 
> *Cc:* Flavio Junqueira ; Sharda, Ravi <
> ravi.sha...@dell.com>
> *Subject:* Re: Bookeeper exception on pods restart
>
>
> [EXTERNAL EMAIL]
> Sorry for jumping into the discussion. But the error message indicates
> that the entry log file 1 is not found.
> It seems to me that entry log file was removed but the entry index still
> points to the old location. Is this error transient error or a permanent
> error?
>
> - Sijie
>
>
>
> On Fri, Nov 29, 2019 at 12:11 AM  wrote:
>
> + Ravi, who will be looking into this ….
>
>
>
> *From:* Enrico Olivelli - Diennea 
> *Sent:* Thursday, November 28, 2019 7:00 PM
> *To:* user@bookkeeper.apache.org
> *Cc:* f...@apache.org
> *Subject:* Re: Bookeeper exception on pods restart
>
>
>
> [EXTERNAL EMAIL]
>
> From the error it looks like one client is trying to read an entry from
> the Bookie but the entry is not there.
>
> I see two reasons:
>
> 1) The write never reached the bookie
>
> 2) The bookie is missing some fi

Re: Bookeeper exception on pods restart

[Enrico Olivelli - Diennea]

From the error it looks like one client is trying to read an entry from the 
Bookie but the entry is not there.
I see two reasons:
1) The write never reached the bookie
2) The bookie is missing some file

For 1)
Do you have logs on the writer ? something that could tell us that a write did 
not succeed ?
How old is supposed to be the entry ?
Do you have logs on the reader that is trying to read the entry ?


For 2)
Do you have other errors in the logs about failed writes or whatever ?
If you were on 4.9 we could use the ‘localconsistency checker’ and check for 
inconsistency on the bookie, it scans the bookie looking for every entry that 
should reside on the bookie itsself.
If you were writing your ledgers with writequorum >= 2 maybe you can recover 
your data.


In order to debug the problem we should compare the logs of:

  *   The bookie
  *   The writer
  *   The reader


Enrico


Il giorno 28/11/19, 13:59 
"prajakta.belgu...@dell.com<mailto:prajakta.belgu...@dell.com>" 
mailto:prajakta.belgu...@dell.com>> ha scritto:

I understand that auto recovery would replicate data for under replicated 
ledgers.
But it is scheduled to run only once in a while and may not have run before a 
reader tries to read this data from a certain bookie.

Generally what does below exception indicate about the state of BK?
Does it indicate that the entry is missing on the specific bookie and so we 
don’t find it?
Or that something in the ledger metadata or ledgers could have been corrupted??

Found the same issue with another product where you seem to have provided a 
custom fix:
https://github.com/diennea/herddb/issues/194

All in all want to understand if this can be the result of BK misconfiguration 
or is just a temporary unavailability problem that will resolve itself when 
auto-replication runs??

-Thanks,
Prajakta

From: Enrico Olivelli - Diennea 
Sent: Thursday, November 28, 2019 6:11 PM
To: user@bookkeeper.apache.org
Cc: f...@apache.org
Subject: Re: Bookeeper exception on pods restart


[EXTERNAL EMAIL]
I don’t think there is a good value.
You can use WriteQuorumSize = AckQuorumSize, this way you will see an error on 
the writing client in case of write failure to any of the bookies

Usually you are enabling the Autorecovery feature to fill in the gaps of 
underreplicated ledgers:
http://bookkeeper.apache.org/docs/4.10.0/admin/autorecovery/


Hope that helps
Enrico

Il giorno 28/11/19, 13:30 
"prajakta.belgu...@dell.com<mailto:prajakta.belgu...@dell.com>" 
mailto:prajakta.belgu...@dell.com>> ha scritto:

What EnsembleSize, WriteQuorumSize and AckQuorumSize would you recommend, so we 
never see this?
What other ledger creation parameters do you need information about?

-Thanks,
Prajakta
From: Enrico Olivelli - Diennea 
mailto:enrico.olive...@diennea.com>>
Sent: Thursday, November 28, 2019 5:19 PM
To: user@bookkeeper.apache.org<mailto:user@bookkeeper.apache.org>
Cc: f...@apache.org<mailto:f...@apache.org>
Subject: Re: Bookeeper exception on pods restart


[EXTERNAL EMAIL]
Hi Prajakta,
What ledger creation parameters are you using ?  Ensamble size, Write quorum 
size, Ack quorum size ?
If ackQuorumSize < WriteQuorumSize it is possible that a write to the bookie 
failed and even if the entry is supposed to be on the bookie it never reached 
it but the overall single write succeeded because a writequorum of bookies 
acknowledged the write.

Enrico

Il giorno 28/11/19, 12:44 
"prajakta.belgu...@dell.com<mailto:prajakta.belgu...@dell.com>" 
mailto:prajakta.belgu...@dell.com>> ha scritto:

Hello Team,

We have a question about an issue we are running into with Bookeeper.
We use bookkeeper version 4.7.3.

This issue occurs occasionally when Bookkeeper servers are restarted.
We see the following error in the logs for some time, which blocks Pravega's 
operations for the same duration. Not knowing the internals of Bookeeper, but 
just based on the exception alone, it seems like Bookeeper might not be locate 
the files temporarily. What could be causing this?


2019-11-28 03:52:26,491 - ERROR - 
[BookieReadThreadPool-OrderedExecutor-0-0:ReadEntryProcessorV3@235] - 
IOException while reading entry: 25 from ledger 56
java.io<https://slack-redir.net/link?url=http%3A%2F%2Fjava.io>.FileNotFoundException:
 No file for log 1 for 56 with location 4744138143
at 
org.apache.bookkeeper.bookie.EntryLogger.findFile(EntryLogger.java:1165)
at 
org.apache.bookkeeper.bookie.EntryLogger.getChannelForLogId(EntryLogger.java:1100)
at 
org.apache.bookkeeper.bookie.EntryLogger.internalReadEntry(EntryLogger.java:1002)
at 
org.apache.bookkeeper.bookie.EntryLogger.readEntry(EntryLogger.java:1051)
at 
org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:305)
at 
org.apache.bookkeeper.bookie.SortedLedgerStorage.getEntry(SortedLedgerStorage.java:153)
 

Re: Bookeeper exception on pods restart

[Enrico Olivelli - Diennea]

I don’t think there is a good value.
You can use WriteQuorumSize = AckQuorumSize, this way you will see an error on 
the writing client in case of write failure to any of the bookies

Usually you are enabling the Autorecovery feature to fill in the gaps of 
underreplicated ledgers:
http://bookkeeper.apache.org/docs/4.10.0/admin/autorecovery/


Hope that helps
Enrico

Il giorno 28/11/19, 13:30 
"prajakta.belgu...@dell.com<mailto:prajakta.belgu...@dell.com>" 
mailto:prajakta.belgu...@dell.com>> ha scritto:

What EnsembleSize, WriteQuorumSize and AckQuorumSize would you recommend, so we 
never see this?
What other ledger creation parameters do you need information about?

-Thanks,
Prajakta
From: Enrico Olivelli - Diennea 
Sent: Thursday, November 28, 2019 5:19 PM
To: user@bookkeeper.apache.org
Cc: f...@apache.org
Subject: Re: Bookeeper exception on pods restart


[EXTERNAL EMAIL]
Hi Prajakta,
What ledger creation parameters are you using ?  Ensamble size, Write quorum 
size, Ack quorum size ?
If ackQuorumSize < WriteQuorumSize it is possible that a write to the bookie 
failed and even if the entry is supposed to be on the bookie it never reached 
it but the overall single write succeeded because a writequorum of bookies 
acknowledged the write.

Enrico

Il giorno 28/11/19, 12:44 
"prajakta.belgu...@dell.com<mailto:prajakta.belgu...@dell.com>" 
mailto:prajakta.belgu...@dell.com>> ha scritto:

Hello Team,

We have a question about an issue we are running into with Bookeeper.
We use bookkeeper version 4.7.3.

This issue occurs occasionally when Bookkeeper servers are restarted.
We see the following error in the logs for some time, which blocks Pravega's 
operations for the same duration. Not knowing the internals of Bookeeper, but 
just based on the exception alone, it seems like Bookeeper might not be locate 
the files temporarily. What could be causing this?


2019-11-28 03:52:26,491 - ERROR - 
[BookieReadThreadPool-OrderedExecutor-0-0:ReadEntryProcessorV3@235] - 
IOException while reading entry: 25 from ledger 56
java.io<https://slack-redir.net/link?url=http%3A%2F%2Fjava.io>.FileNotFoundException:
 No file for log 1 for 56 with location 4744138143
at 
org.apache.bookkeeper.bookie.EntryLogger.findFile(EntryLogger.java:1165)
at 
org.apache.bookkeeper.bookie.EntryLogger.getChannelForLogId(EntryLogger.java:1100)
at 
org.apache.bookkeeper.bookie.EntryLogger.internalReadEntry(EntryLogger.java:1002)
at 
org.apache.bookkeeper.bookie.EntryLogger.readEntry(EntryLogger.java:1051)
at 
org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:305)
at 
org.apache.bookkeeper.bookie.SortedLedgerStorage.getEntry(SortedLedgerStorage.java:153)
at 
org.apache.bookkeeper.bookie.LedgerDescriptorImpl.readEntry(LedgerDescriptorImpl.java:153)
at org.apache.bookkeeper.bookie.Bookie.readEntry(Bookie.java:1305)
at 
org.apache.bookkeeper.proto.ReadEntryProcessorV3.readEntry(ReadEntryProcessorV3.java:175)
at 
org.apache.bookkeeper.proto.ReadEntryProcessorV3.readEntry(ReadEntryProcessorV3.java:155)
at 
org.apache.bookkeeper.proto.ReadEntryProcessorV3.getReadResponse(ReadEntryProcessorV3.java:218)
at 
org.apache.bookkeeper.proto.ReadEntryProcessorV3.executeOp(ReadEntryProcessorV3.java:264)
at 
org.apache.bookkeeper.proto.ReadEntryProcessorV3.safeRun(ReadEntryProcessorV3.java:260)
at 
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

-Thanks,
Prajakta




CONFIDENTIALITY & PRIVACY NOTICE
This e-mail (including any attachments) is strictly confidential and may also 
contain privileged information. If you are not the intended recipient you are 
not authorised to read, print, save, process or disclose this message. If you 
have received this message by mistake, please inform the sender immediately and 
destroy this e-mail, its attachments and any copies. Any use, distribution, 
reproduction or disclosure by any person other than the intended recipient is 
strictly prohibited and the person responsible may incur in penalties.
The use of this e-mail is only for professional purposes; there is no guarantee 
that the correspondence towards this e-mail will be read only by the recipient, 
because, under certain circumstances, there may be a need to access this email 
by third subjects belonging to the Company.



CONFIDENTIALITY & PRIVACY NOTICE
This e-mail (including any attachments) is strictly confidential and may also 
contain privileged information. If you are not the intended recipient you are 
not authorised to read, print, save, process or disclose this message.

Re: Bookeeper exception on pods restart

[Enrico Olivelli - Diennea]

Hi Prajakta,
What ledger creation parameters are you using ?  Ensamble size, Write quorum 
size, Ack quorum size ?
If ackQuorumSize < WriteQuorumSize it is possible that a write to the bookie 
failed and even if the entry is supposed to be on the bookie it never reached 
it but the overall single write succeeded because a writequorum of bookies 
acknowledged the write.

Enrico

Il giorno 28/11/19, 12:44 
"prajakta.belgu...@dell.com" 
mailto:prajakta.belgu...@dell.com>> ha scritto:

Hello Team,

We have a question about an issue we are running into with Bookeeper.
We use bookkeeper version 4.7.3.

This issue occurs occasionally when Bookkeeper servers are restarted.
We see the following error in the logs for some time, which blocks Pravega's 
operations for the same duration. Not knowing the internals of Bookeeper, but 
just based on the exception alone, it seems like Bookeeper might not be locate 
the files temporarily. What could be causing this?


2019-11-28 03:52:26,491 - ERROR - 
[BookieReadThreadPool-OrderedExecutor-0-0:ReadEntryProcessorV3@235] - 
IOException while reading entry: 25 from ledger 56
java.io.FileNotFoundException:
 No file for log 1 for 56 with location 4744138143
at 
org.apache.bookkeeper.bookie.EntryLogger.findFile(EntryLogger.java:1165)
at 
org.apache.bookkeeper.bookie.EntryLogger.getChannelForLogId(EntryLogger.java:1100)
at 
org.apache.bookkeeper.bookie.EntryLogger.internalReadEntry(EntryLogger.java:1002)
at 
org.apache.bookkeeper.bookie.EntryLogger.readEntry(EntryLogger.java:1051)
at 
org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:305)
at 
org.apache.bookkeeper.bookie.SortedLedgerStorage.getEntry(SortedLedgerStorage.java:153)
at 
org.apache.bookkeeper.bookie.LedgerDescriptorImpl.readEntry(LedgerDescriptorImpl.java:153)
at org.apache.bookkeeper.bookie.Bookie.readEntry(Bookie.java:1305)
at 
org.apache.bookkeeper.proto.ReadEntryProcessorV3.readEntry(ReadEntryProcessorV3.java:175)
at 
org.apache.bookkeeper.proto.ReadEntryProcessorV3.readEntry(ReadEntryProcessorV3.java:155)
at 
org.apache.bookkeeper.proto.ReadEntryProcessorV3.getReadResponse(ReadEntryProcessorV3.java:218)
at 
org.apache.bookkeeper.proto.ReadEntryProcessorV3.executeOp(ReadEntryProcessorV3.java:264)
at 
org.apache.bookkeeper.proto.ReadEntryProcessorV3.safeRun(ReadEntryProcessorV3.java:260)
at 
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

-Thanks,
Prajakta




CONFIDENTIALITY & PRIVACY NOTICE
This e-mail (including any attachments) is strictly confidential and may also 
contain privileged information. If you are not the intended recipient you are 
not authorised to read, print, save, process or disclose this message. If you 
have received this message by mistake, please inform the sender immediately and 
destroy this e-mail, its attachments and any copies. Any use, distribution, 
reproduction or disclosure by any person other than the intended recipient is 
strictly prohibited and the person responsible may incur in penalties.
The use of this e-mail is only for professional purposes; there is no guarantee 
that the correspondence towards this e-mail will be read only by the recipient, 
because, under certain circumstances, there may be a need to access this email 
by third subjects belonging to the Company.

[ANNOUNCE] Apache BookKeeper 4.10.0 released

[Enrico Olivelli]

The Apache BookKeeper team is proud to announce Apache BookKeeper version
4.10.0.

Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage
service optimized for
real-time workloads. It has been used for a fundamental service to build
reliable services.
It is also the log segment store for Apache DistributedLog and the message
store for Apache Pulsar.

This is the 20th release of the Apache BookKeeper.

News and noteworthy:
- Add new bkctl shell tool
- Cluster Metadata Checker
- Journal should respect to flushWhenQueueEmpty setting
- Allow to override default SASL service name ‘bookkeeper’
- Make default Bookie scripts work on JDK11+

For BookKeeper release details and downloads, visit:

http://bookkeeper.apache.org/releases/

BookKeeper 4.10.0 Release Notes are at:

http://bookkeeper.apache.org/docs/4.10.0/overview/releaseNotes/

We would like to thank the contributors that made the release possible.

Regards,

The BookKeeper Team

Re: Invitation to Apache Pulsar Meetup | Shanghai on 11/16

[Enrico Olivelli]

Thank you Jennifer

Are you going to record the event and share it?
I am from Europe and so I am not able to attend.

Cheers
Enrico

Il mer 30 ott 2019, 14:14 Jinfeng Huang  ha scritto:

> Dear BookKeeper community,
>
> We’re hosting Apache Pulsar Meetup | Shanghai on 11/16. The meetup
> provides an opportunity for Pulsar and BookKeeper users, contributors,
> committers and PMC members to get together and share Pulsar updates. The
> upcoming meetup mainly covers:
>
>
>-
>
>Pulsar adoption stories at Orange Financial, Zhaopin, Tuya, etc
>-
>
>Pulsar 2.5.0 preview
>-
>
>Pulsar manager and pulsarctl
>
> If you’re interested in the meetup, come and join us to learn more about
> Pulsar community.
>
>
>-
>
>*Time: *2019/11/16 13:30~18:00 (Saturday)
>-
>
>*Venue:* Hongkou Technology and Finance Building, 463 Tanggu Road,
>Hongkou District, Shanghai, China (中国上海市虹口区塘沽路463号虹口科技金融大厦一层)
>
> The following register platforms are available:
>
>
>-
>
>*Eventbrite*(global):
>
> https://www.eventbrite.com/e/apache-pulsar-meetup-shanghai-tickets-79293658467
>-
>
>*Huodongxing *(in China):
>https://www.huodongxing.com/event/5515876233300
>
> For more information on Pulsar meetup, visit StreamNative website:
> https://streamnative.io/developer/meetup/
>
> Best Regards,
> Jennifer
>

Re: Error starting bookie using Apache Bookkeeper 4.9.1

[Enrico Olivelli]

I feel that a "metaformat" has always been needed in BookKeeper (at least
since 4.5) , here the problem is about using a non first level path

Anyway, from 4.7 to 4.9 we did a lot of refactors, and we finished the
abstraction of the Metadata Service, giving the ability to support
pluggable Metadata Services, namely we are supporting now ZooKeeper and
Etcd (patially).

Maybe there was not test or active user that shown up for this kind of
"regression".

Beside note:
We are going to release 4.10 soon, maybe it is worth to check compatibility
with the current master
and/or setup periodic compatibility tests against current "master", we are
deploying -SNAPSHOT version to Apache Maven Repositories for this purpose.
This will help the community in preventing regressions

For instance in 4.10 we invested my time in the new CLI, named 'bkctl' and
if major OSS projects (like Pravega) migrate to the new CLI we will be more
confident in deprecating the old CLI




Il giorno gio 10 ott 2019 alle ore 11:01  ha
scritto:

> Thanks Enrico.
>
>
>
> This was not needed with Bookkeeper 4.7.3 and without any init/metaformat
> we were able to bring up the bookie.
>
> What has changed in 4.9.1 because of which this is needed now?
>
>
>
> -Thanks,
>
> Prajakta
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Thursday, October 10, 2019 2:06 PM
> *To:* user
> *Cc:* Flavio Junqueira
> *Subject:* Re: Error starting bookie using Apache Bookkeeper 4.9.1
>
>
>
> [EXTERNAL EMAIL]
>
> Looking at the code I think that the short answer is "yes"
>
>
> https://github.com/apache/bookkeeper/blob/2f996dcf0159f945f7ec97ce7402e5d293009444/bookkeeper-server/src/main/java/org/apache/bookkeeper/discover/ZKRegistrationManager.java#L403
>
>
>
> we are using ZooKeeper#create, that does not recursively create all the
> hierarchy.
>
> I feel this is due to the fact that usually a BooKeeper cluster stores
> metadata in /ledgers, as a direct child of '/'
>
>
>
> IIRC in some product of my company we had the same problem and we simply
> pre-created the znode structure before '/ledgers'
>
>
>
> Maybe you can give it a try.
>
> I also think it is worth to submit a simply patch in order to change the
> behaviour and create the full hierarchy (the change is really simple)
>
>
>
> Hope that helprs
>
> Enrico
>
>
>
>
>
>
>
> Il giorno gio 10 ott 2019 alle ore 10:28  ha
> scritto:
>
> Does that need to be created manually?
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Thursday, October 10, 2019 12:25 PM
> *To:* user
> *Cc:* Flavio Junqueira
> *Subject:* Re: Error starting bookie using Apache Bookkeeper 4.9.1
>
>
>
> [EXTERNAL EMAIL]
>
> I see your base path is /pravega/pravega/bookkeeper/ledgers
>
>
>
> Maybe the root path does not yet exist, I mean /pravega/pravega/bookkeeper
>
>
>
> Enrico
>
>
>
> Il giorno gio 10 ott 2019 alle ore 08:20  ha
> scritto:
>
> Thanks for your response, Enrico.
>
>
>
> I tried using both `metaformat` and `initnewcluster` shell commands to
> initialize zookeeper before starting the bookie.
>
> But both failed with similar exceptions….
>
>
>
> Using `bin/bookkeeper shell metaformat -nonInteractive -force` I see …
>
> “
>
> Exception in thread "main" java.util.concurrent.ExecutionException:
> KeeperErrorCode = NoNode for /pravega/pravega/bookkeeper/ledgers
> at
> org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:378)
> at
> org.apache.bookkeeper.client.BookKeeperAdmin.format(BookKeeperAdmin.java:1150)
> at
> org.apache.bookkeeper.bookie.BookieShell$MetaFormatCmd.runCmd(BookieShell.java:328)
> at
> org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:277)
> at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:3081)
> at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:3172)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /pravega/pravega/bookkeeper/ledgers
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:114)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$1901(ZooKeeperClient.java:70)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient$9.call(ZooKeeperClient.java:711)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient$9.call(ZooKeeperClient.java:705)
> at
> org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRet

Re: Error starting bookie using Apache Bookkeeper 4.9.1

[Enrico Olivelli]

Looking at the code I think that the short answer is "yes"
https://github.com/apache/bookkeeper/blob/2f996dcf0159f945f7ec97ce7402e5d293009444/bookkeeper-server/src/main/java/org/apache/bookkeeper/discover/ZKRegistrationManager.java#L403

we are using ZooKeeper#create, that does not recursively create all the
hierarchy.
I feel this is due to the fact that usually a BooKeeper cluster stores
metadata in /ledgers, as a direct child of '/'

IIRC in some product of my company we had the same problem and we simply
pre-created the znode structure before '/ledgers'

Maybe you can give it a try.
I also think it is worth to submit a simply patch in order to change the
behaviour and create the full hierarchy (the change is really simple)

Hope that helprs
Enrico



Il giorno gio 10 ott 2019 alle ore 10:28  ha
scritto:

> Does that need to be created manually?
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Thursday, October 10, 2019 12:25 PM
> *To:* user
> *Cc:* Flavio Junqueira
> *Subject:* Re: Error starting bookie using Apache Bookkeeper 4.9.1
>
>
>
> [EXTERNAL EMAIL]
>
> I see your base path is /pravega/pravega/bookkeeper/ledgers
>
>
>
> Maybe the root path does not yet exist, I mean /pravega/pravega/bookkeeper
>
>
>
> Enrico
>
>
>
> Il giorno gio 10 ott 2019 alle ore 08:20  ha
> scritto:
>
> Thanks for your response, Enrico.
>
>
>
> I tried using both `metaformat` and `initnewcluster` shell commands to
> initialize zookeeper before starting the bookie.
>
> But both failed with similar exceptions….
>
>
>
> Using `bin/bookkeeper shell metaformat -nonInteractive -force` I see …
>
> “
>
> Exception in thread "main" java.util.concurrent.ExecutionException:
> KeeperErrorCode = NoNode for /pravega/pravega/bookkeeper/ledgers
> at
> org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:378)
> at
> org.apache.bookkeeper.client.BookKeeperAdmin.format(BookKeeperAdmin.java:1150)
> at
> org.apache.bookkeeper.bookie.BookieShell$MetaFormatCmd.runCmd(BookieShell.java:328)
> at
> org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:277)
> at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:3081)
> at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:3172)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /pravega/pravega/bookkeeper/ledgers
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:114)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$1901(ZooKeeperClient.java:70)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient$9.call(ZooKeeperClient.java:711)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient$9.call(ZooKeeperClient.java:705)
> at
> org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient.create(ZooKeeperClient.java:705)
> at
> org.apache.bookkeeper.discover.ZKRegistrationManager.prepareFormat(ZKRegistrationManager.java:403)
> at
> org.apache.bookkeeper.client.BookKeeperAdmin.lambda$format$2(BookKeeperAdmin.java:1152)
> at
> org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:373)
>
>
>
> “
>
> Using ` bin/bookkeeper shell initnewcluster` I see:
>
> “
>
> Exception in thread "main" java.util.concurrent.ExecutionException:
> KeeperErrorCode = NoNode
> at
> org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:378)
> at
> org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithRegistrationManager(MetadataDrivers.java:398)
> at
> org.apache.bookkeeper.client.BookKeeperAdmin.initNewCluster(BookKeeperAdmin.java:1197)
> at
> org.apache.bookkeeper.bookie.BookieShell$InitNewCluster.runCmd(BookieShell.java:368)
> at
> org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:277)
> at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:3081)
> at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:3172)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:114)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1015)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:919)
> at
> or

Re: Error starting bookie using Apache Bookkeeper 4.9.1

[Enrico Olivelli]

I see your base path is /pravega/pravega/bookkeeper/ledgers

Maybe the root path does not yet exist, I mean /pravega/pravega/bookkeeper

Enrico

Il giorno gio 10 ott 2019 alle ore 08:20  ha
scritto:

> Thanks for your response, Enrico.
>
>
>
> I tried using both `metaformat` and `initnewcluster` shell commands to
> initialize zookeeper before starting the bookie.
>
> But both failed with similar exceptions….
>
>
>
> Using `bin/bookkeeper shell metaformat -nonInteractive -force` I see …
>
> “
>
> Exception in thread "main" java.util.concurrent.ExecutionException:
> KeeperErrorCode = NoNode for /pravega/pravega/bookkeeper/ledgers
> at
> org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:378)
> at
> org.apache.bookkeeper.client.BookKeeperAdmin.format(BookKeeperAdmin.java:1150)
> at
> org.apache.bookkeeper.bookie.BookieShell$MetaFormatCmd.runCmd(BookieShell.java:328)
> at
> org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:277)
> at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:3081)
> at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:3172)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /pravega/pravega/bookkeeper/ledgers
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:114)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$1901(ZooKeeperClient.java:70)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient$9.call(ZooKeeperClient.java:711)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient$9.call(ZooKeeperClient.java:705)
> at
> org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient.create(ZooKeeperClient.java:705)
> at
> org.apache.bookkeeper.discover.ZKRegistrationManager.prepareFormat(ZKRegistrationManager.java:403)
> at
> org.apache.bookkeeper.client.BookKeeperAdmin.lambda$format$2(BookKeeperAdmin.java:1152)
> at
> org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:373)
>
>
>
> “
>
> Using ` bin/bookkeeper shell initnewcluster` I see:
>
> “
>
> Exception in thread "main" java.util.concurrent.ExecutionException:
> KeeperErrorCode = NoNode
> at
> org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:378)
> at
> org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithRegistrationManager(MetadataDrivers.java:398)
> at
> org.apache.bookkeeper.client.BookKeeperAdmin.initNewCluster(BookKeeperAdmin.java:1197)
> at
> org.apache.bookkeeper.bookie.BookieShell$InitNewCluster.runCmd(BookieShell.java:368)
> at
> org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:277)
> at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:3081)
> at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:3172)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:114)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1015)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:919)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$801(ZooKeeperClient.java:70)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient$2.call(ZooKeeperClient.java:466)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient$2.call(ZooKeeperClient.java:455)
> at
> org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140)
> at
> org.apache.bookkeeper.zookeeper.ZooKeeperClient.multi(ZooKeeperClient.java:455)
> at
> org.apache.bookkeeper.discover.ZKRegistrationManager.initNewCluster(ZKRegistrationManager.java:453)
> at
> org.apache.bookkeeper.client.BookKeeperAdmin.lambda$initNewCluster$3(BookKeeperAdmin.java:1199)
> at
> org.apache.bookkeeper.meta.MetadataDrivers.lambda$runFunctionWithRegistrationManager$1(MetadataDrivers.java:398)
>     at
> org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:373)
>
> “
>
> What initialization step am I missing that needs to be done before
> invoking: ` /opt/bookkeeper/scripts/entrypoint.sh bookie` ??
>
>
>
> -Thanks,
>
> Prajakta
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Friday, Octobe

Re: Help needed on the usage of bkctl

[Enrico Olivelli]

It seems that we are using this syntax in the init_bookie script

/bkctl --service-uri ${BK_metadataServiceUri} cluster init

see
https://github.com/apache/bookkeeper/blob/master/docker/scripts/init_bookie.sh#L54




Enrico

Il giorno mer 11 set 2019 alle ore 22:22 Dennis Mercuriali <
mercuriali...@gmail.com> ha scritto:

> Hi everyone,
>
> I'm writing to ask you for some help regarding the usage of the "bkctl
> cluster init" command. I've been trying to update BookKeeper's Zookeeper
> dependency from 3.4.13 to 3.5.5 (this is the PR if you are interested
> https://github.com/apache/bookkeeper/pull/2112) but a single test keeps
> failing with the following error:
>
> java.lang.IllegalArgumentException: No service URI is providedat 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)at 
> org.apache.bookkeeper.stream.cli.commands.cluster.InitClusterCommand.apply(InitClusterCommand.java:118)at
>  
> org.apache.bookkeeper.stream.cli.commands.cluster.InitClusterCommand.apply(InitClusterCommand.java:53)at
>  org.apache.bookkeeper.tools.common.BKCommand.apply(BKCommand.java:97)at 
> org.apache.bookkeeper.tools.common.BKCommand.lambda$apply$0(BKCommand.java:50)at
>  org.apache.bookkeeper.tools.framework.Cli.run(Cli.java:226)at 
> org.apache.bookkeeper.tools.framework.Cli.runCli(Cli.java:248)at 
> org.apache.bookkeeper.tools.common.BKCommand.apply(BKCommand.java:52)at 
> org.apache.bookkeeper.tools.common.BKCommand.apply(BKCommand.java:40)at 
> org.apache.bookkeeper.tools.framework.Cli.run(Cli.java:236)at 
> org.apache.bookkeeper.tools.framework.Cli.runCli(Cli.java:248)at 
> org.apache.bookkeeper.tools.framework.CliCommandGroup.apply(CliCommandGroup.java:68)at
>  org.apache.bookkeeper.tools.framework.Cli.run(Cli.java:236)at 
> org.apache.bookkeeper.tools.framework.Cli.runCli(Cli.java:248)at 
> org.apache.bookkeeper.tools.cli.BKCtl.main(BKCtl.java:59)
>
> The test tries to execute this command:
>
> bkctl --service-uri zk://metadata-store:2181/ledgers cluster init
>
> This error doesn't show up in other PRs but I couldn't find a reason why
> my changes should have broken the parameter-passing of this command. I did
> some local manual tests and the same error occurs when I try to run that
> command against my branch, the current master and BookKeeper 4.9.2
>
> I also tried to run
>
>-
>
>"bkctl cluster init" getting the same error and a "Usage:  bkctl
>cluster [command] [command options]"
>-
>
>"bkctl cluster init --service-uri zk://localhost:2181/ledgers" this is
>more interesting since it prints "Usage:  bkctl cluster init [flags]
>"
>-
>
>finally tried "bkctl cluster init zk://localhost:2181/ledgers" getting
>no console output
>
> I'm new to the project and I am probably missing something, can anybody
> tell me to the correct usage of the command?
>
> Thank you all in advance,
> Dennis
>

Re: I want to know how to read the last message without close the current ledger?

[Enrico Olivelli]

Il mar 30 lug 2019, 04:17 Wei Liu  ha scritto:

> Thank you very much for your explanation.
>
> I think the current design is therefore a single writer, who writes the
> same message to multiple bookies  in parallel(no leader) and the server
> updates lac only rely on client submit.
> It may result in delay in receiving the last message or in user usage
> complexity.
>

If you have a continuous stream of data from the writer you get a great
throughput and all of the guarantees of BK.

If you have very few writes you have to do what I said before .

If you are experiencing low throughput consider using the asynchronous API
and tune you journal settings on the bookie

Enrico



> Enrico Olivelli  于2019年7月29日周一 下午7:07写道：
>
>> Hello Wei Liu,
>> This is about the core feature of BookKeeper, that is the Last
>> AddConfirmed Protocol (LAC for friends).
>>
>> BookKeeper guarantees to the readers that once you read an entry you will
>> always be able to read it.
>> The reader is able to "see"  the entries only after it has the guarantee
>> that the writer has received the acknowledgement of a successful write with
>> an quorum of bookies.
>>
>> In order to make it possible the writer "piggy backs" the id of the
>> LastAddConfirmed entry to the bookies and the reader then is able to use
>> this value (by reading it with readLastAddConfirmed) in order to have this
>> reference id.
>>
>> You have these ways to get your work done:
>> 1) write no-operation entries to the ledger, this will make the LAC
>> advance even in absence of "application" entries
>> 2) use the "ExplicitLAC" feature, that is a background activity that
>> sends the current LastAddConfirmed to the bookies even in absence of writes
>> after a configurable delay (this is like option 1 but is it automatic, but
>> it is still not available in the new API)
>> 3) use readUnconfirmedEntries: this bypasses the LAC protocol at all, so
>> you can read every entry, but you will need to use an out-of-band channel
>> to pass the id of the entries to read
>>
>> I hope that helps
>>
>> Enrico
>>
>>
>>
>>
>>
>>
>>
>> Il giorno lun 29 lug 2019 alle ore 12:49 Wei Liu  ha
>> scritto:
>>
>>>
>>> Dear All :
>>>
>>>  1) First create ledger, add msg 1, 2, 3 into the ledger
>>>  2) Do not close the current ledger;
>>>  3) Received the msg 1 and 2;
>>>  4) Received the msg 3 after closed the ledger;
>>>
>>>  I want to know how to read the last message without close the
>>> current ledger?
>>>
>>> Thanks~
>>> --
>>> 一个人只拥有今生今世是不够的，
>>> 他还应该拥有诗意的世界。
>>>
>>>liuwe...@gmail.com
>>>
>>
>
> --
> 一个人只拥有今生今世是不够的，
> 他还应该拥有诗意的世界。
>
>liuwe...@gmail.com
>

[ANNOUNCE] Apache BookKeeper 4.9.1 released

[Enrico Olivelli]

The Apache BookKeeper team is proud to announce Apache BookKeeper version
4.9.1.

Apache BookKeeper is a scalable, fault-tolerant, and low-latency
storage service optimized for
real-time workloads. It has been used for a fundamental service to
build reliable services.
It is also the log segment store for Apache DistributedLog and the
message store for Apache Pulsar.

This is the 18th release of the Apache BookKeeper.

The 4.9.1 release incorporates a few critical bug fixes, since
previous major release, 4.9.0.

For BookKeeper release details and downloads, visit:

https://bookkeeper.apache.org/releases/

BookKeeper 4.9.1 Release Notes are at:

https://bookkeeper.apache.org/docs/4.9.1/overview/releaseNotes/

We would like to thank the contributors that made the release possible.

Regards,

The BookKeeper Team

Introducing BlobIt - a Distribute Binary Large Object Storage

[Enrico Olivelli]

Hi BookKeepers,
I would like to share with you this brand new project based on Apache
BookKeeper.
It is BlobIt, and it is a distributed storage system that uses
BookKeeper for mid term storage of rawdata.

This is a simple introduction post from my personal blog:
https://eolivelli.blogspot.com/2019/04/blobit-scalable-binary-large-object.html

You can find the homepage on GitHub at:
https://github.com/diennea/blobit

If you are interested feel free to reach me directly or join the
(brand new) community on github.

Best regards
#bookkeeperocks

Enrico

Re: [ANNOUNCE] Apache BookKeeper 4.9.0 released

[Enrico Olivelli]

Congrats

Thank you Sijie for putting all together.

Thanks to Ivan that did the huge reworking around metadata.

Enrico

Il giorno sab 2 feb 2019, 01:49 Sijie Guo  ha scritto:

> The Apache BookKeeper team is proud to announce Apache BookKeeper version
> 4.9.0.
>
> Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage
> service optimized for
> real-time workloads. It has been used for a fundamental service to build
> reliable services.
> It is also the log segment store for Apache DistributedLog and the message
> store for Apache Pulsar.
>
> This is the 16th release of the Apache BookKeeper.
>
> The 4.9.0 release incorporates hundreds of bug fixes, improvements,
> and features since previous major release, 4.8.0, which was released four
> months ago.
> It is a new milestone in Apache BookKeeper community.
>
> There are a few big changes around metadata management, such as refactoring
> ledger metatadata
> in ledger handle to make it immutable, storing ledger metadata in binary
> format and a new metadata
> driver implementation based on Etcd. Additionally, there are huge
> improvements around memory
> management, tooling, and documentation.
>
> For BookKeeper release details and downloads, visit:
>
> http://bookkeeper.apache.org/releases/
>
> BookKeeper 4.9.0 Release Notes are at:
>
> http://bookkeeper.apache.org/docs/4.9.0/overview/releaseNotes/
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The BookKeeper Team
>

DCOS/OS Package users

[Enrico Olivelli]

Hi BookKeepers !

During the release process we are submitting packages to Mesosphere
Universe for running BookKeeper in DCOS environment, like this PR:
https://github.com/mesosphere/universe/pull/1998

Is there any active user of that package ?

If there is no interest we can stop delivering such artifacts.

We will still continue to provide the docker image
https://hub.docker.com/r/apache/bookkeeper/


Best regards
Enrico

[ANNOUNCE] Apache BookKeeper 4.8.1 released

[Enrico Olivelli]

The Apache BookKeeper team is proud to announce Apache BookKeeper version
4.8.1.

Apache BookKeeper is a scalable, fault-tolerant, and low-latency
storage service optimized for
real-time workloads. It has been used for a fundamental service to
build reliable services.
It is also the log segment store for Apache DistributedLog and the
message store for Apache Pulsar.

This is the 14th release of the Apache BookKeeper.

Highlights

Use default metrics registry in Prometheus exporter

Don’t cache Bookie hostname DNS resolution forever

Reduce stack traces in logs for common cases

Ledger deletion racing with flush can cause a ledger index to be resurrected

EntryMemTable.newEntry retains reference to passed ByteBuffer
array, can cause corruption on journal replay


For BookKeeper release details and downloads, visit:

https://bookkeeper.apache.org/releases/

BookKeeper 4.8.1 Release Notes are at:

https://bookkeeper.apache.org/docs/4.8.1/overview/releaseNotes/

We would like to thank the contributors that made the release possible.

Regards,

The BookKeeper Team

Re: Community meeting tomorrow

[Enrico Olivelli]

Il giorno mer 14 nov 2018 alle ore 19:35 Sijie Guo
 ha scritto:
>
>
>
> On Wed, Nov 14, 2018 at 10:26 AM Enrico Olivelli  wrote:
>>
>> Hi BookKeepers !
>> tomorrow we will 'meet' on hangout for the usual Community Meeting.
>> It is the place to discuss current projects around BookKeeper.
>>
>> https://bookkeeper.apache.org/community/meeting/
>>
>> We are going to release 4.8.1.
>> I am working on JDK12 support
>
>
> Did our CI (jenkins or travis) cover JDK11?

only travis, I never sent the PR for upgrading the other CI jobs, I
never did because there are a bunch of things to change (precommit,
nightly builds...)

I am performing pre-production tests on JDK11 for BookKeeper, I hope
to see some Bookie and clients in production within some week.

It has been long time ago that I have started running regular tests of
BookKeeper based projects on JDK11, and never found issues, but I have
only bookies and client on JDK10 in production.
Now JDK10 and JDK8 are EOL so the switch to JDK11 is mandatory


Enrico


>
>>
>> but the road will be very long
>> (PowerMock is broken)
>>
>> I will be there,
>> See you
>>
>> Enrico

Re: Migrating from zkServers to metadataServiceURI

[Enrico Olivelli]

Il mar 9 ott 2018, 22:45 Sijie Guo  ha scritto:

>
>
> On Mon, Oct 8, 2018 at 1:51 AM Enrico Olivelli 
> wrote:
>
>> Il giorno lun 8 ott 2018 alle ore 09:59 Sijie Guo 
>> ha scritto:
>> >
>> >
>> >
>> > On Sun, Oct 7, 2018 at 11:51 PM Enrico Olivelli 
>> wrote:
>> >>
>> >> Hi,
>> >> I am going to migrate a bunch of applications from the deprecated
>> >> zkServers to metadataServiceURI.
>> >> I think that the change is not really straightforward.
>> >
>> >
>> > why?
>>
>> uhm just forgot to mention FlatLedgerManagerFactory, I have several
>> "old" BookKeeper clusters with it.
>>
>
> if you want to be more explicit, you can use:
>
> zk+flat://
> and
> zk+hierarchical://
>
> otherwise, you can omit the "+" part, using "zk://" will lookup the layout
> and use the factory class stored in layout.
>

Oh great! This is what I was suspecting and it is great, so the migration
can be automatic and straightforward, just drop ledgerManagerFactory option
and concat zkServers with ledgersPathand addd a zk:// prefix.

Thank you very much Sijie !

It would be good to write this down somewhere (or I am missing such page on
the website)

Enrico


>
>>
>> I would like to have a "recipe" for the conversion.
>>
>> Having this discussion on the ML can be useful to share this issue,
>> that all of the users must address before we drop zkAddress
>> configuration entry
>>
>> Enrico
>>
>> >
>> > the metadataServiceURI should be "zk://".
>> >
>> > For example, if your zkServers is "127.0.0.1:2181" and your ledgers
>> root path is "/ledgers".
>> > The metadata service uri is "zk://127.0.0.1:2181/ledgers"
>> >
>> >>
>> >>
>> >> Do you have any best practice for doing this change in configurations ?
>> >>
>> >>
>> >>
>> >> Enrico
>>
> --


-- Enrico Olivelli

Re: Migrating from zkServers to metadataServiceURI

[Enrico Olivelli]

Il lun 8 ott 2018, 10:50 Enrico Olivelli  ha scritto:

> Il giorno lun 8 ott 2018 alle ore 09:59 Sijie Guo 
> ha scritto:
> >
> >
> >
> > On Sun, Oct 7, 2018 at 11:51 PM Enrico Olivelli 
> wrote:
> >>
> >> Hi,
> >> I am going to migrate a bunch of applications from the deprecated
> >> zkServers to metadataServiceURI.
> >> I think that the change is not really straightforward.
> >
> >
> > why?
>
> uhm just forgot to mention FlatLedgerManagerFactory, I have several
> "old" BookKeeper clusters with it.
>
> I would like to have a "recipe" for the conversion.
>

What about zk+null?.
Was is something  like 'auto' ?

Enrico


> Having this discussion on the ML can be useful to share this issue,
> that all of the users must address before we drop zkAddress
> configuration entry
>
> Enrico
>
> >
> > the metadataServiceURI should be "zk://".
> >
> > For example, if your zkServers is "127.0.0.1:2181" and your ledgers
> root path is "/ledgers".
> > The metadata service uri is "zk://127.0.0.1:2181/ledgers"
> >
> >>
> >>
> >> Do you have any best practice for doing this change in configurations ?
> >>
> >>
> >>
> >> Enrico
>
-- 


-- Enrico Olivelli

Re: Migrating from zkServers to metadataServiceURI

[Enrico Olivelli]

Il giorno lun 8 ott 2018 alle ore 09:59 Sijie Guo 
ha scritto:
>
>
>
> On Sun, Oct 7, 2018 at 11:51 PM Enrico Olivelli  wrote:
>>
>> Hi,
>> I am going to migrate a bunch of applications from the deprecated
>> zkServers to metadataServiceURI.
>> I think that the change is not really straightforward.
>
>
> why?

uhm just forgot to mention FlatLedgerManagerFactory, I have several
"old" BookKeeper clusters with it.

I would like to have a "recipe" for the conversion.

Having this discussion on the ML can be useful to share this issue,
that all of the users must address before we drop zkAddress
configuration entry

Enrico

>
> the metadataServiceURI should be "zk://".
>
> For example, if your zkServers is "127.0.0.1:2181" and your ledgers root path 
> is "/ledgers".
> The metadata service uri is "zk://127.0.0.1:2181/ledgers"
>
>>
>>
>> Do you have any best practice for doing this change in configurations ?
>>
>>
>>
>> Enrico

Migrating from zkServers to metadataServiceURI

[Enrico Olivelli]

Hi,
I am going to migrate a bunch of applications from the deprecated
zkServers to metadataServiceURI.
I think that the change is not really straightforward.

Do you have any best practice for doing this change in configurations ?



Enrico

Re: Missing Apache BookKeeper 4.8.0 artifacts from Maven Central

[Enrico Olivelli]

Now it is okay
https://search.maven.org/search?q=g:org.apache.bookkeeper

Cheers
Enrico
Il giorno gio 4 ott 2018 alle ore 09:55 Enrico Olivelli
 ha scritto:
>
> Hi,
> During the release procedure of 4.8.0 something went wrong.
> Apache Nexus Repositories did not publish artifacts to Maven Central.
>
> Artifacts will be available soon (I have simply "pushed the button"
> another time on Nexus UI)
>
> Enrico

Missing Apache BookKeeper 4.8.0 artifacts from Maven Central

[Enrico Olivelli]

Hi,
During the release procedure of 4.8.0 something went wrong.
Apache Nexus Repositories did not publish artifacts to Maven Central.

Artifacts will be available soon (I have simply "pushed the button"
another time on Nexus UI)

Enrico

[ANNOUNCE] Apache BookKeeper 4.8.0 released

[Enrico Olivelli]

The Apache BookKeeper team is proud to announce Apache BookKeeper version 4.8.0.

Apache BookKeeper is a scalable, fault-tolerant, and low-latency
storage service optimized for
real-time workloads. It has been used for a fundamental service to
build reliable services.
It is also the log segment store for Apache DistributedLog and the
message store for Apache Pulsar.

This is the 13th release of the Apache BookKeeper.

The main features in 4.8.0 are around following areas:

- Relaxed Durability: new DEFERRED_SYNC WriteFlag to defer waiting for
sync on Journals

- ExplicitLAC feature: Now ExplicitLAC is no more best-effort but is
can be persisted durably on Bookies

- New Table Storage Service: a new scalable distributed Key-Value
store embedded in Bookies

For BookKeeper release details and downloads, visit:

https://bookkeeper.apache.org/releases/

BookKeeper 4.8.0 Release Notes are at:

https://bookkeeper.apache.org/docs/4.8.0/overview/releaseNotes/

We would like to thank the contributors that made the release possible.

Regards,

The BookKeeper Team

Re: Upcoming Comunity Meeting

[Enrico Olivelli]

Here are the minutes

https://cwiki.apache.org/confluence/display/BOOKKEEPER/2018-09-06+Meeting+Notes

JV found a memory leak, I will hold with 4.8 release, in order to include
his fix

Thank you very much for joining !!

Enrico



Il giorno gio 6 set 2018 alle ore 08:01 Enrico Olivelli 
ha scritto:

> Hi all,
> We are going to met on Hangout at 8 AM PST
>
> this is the agenda
>
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/2018-09-06+Meeting+Notes
>
> you can find the link on the website
> https://bookkeeper.apache.org/community/meeting/
>
> Personally I do not have items for the agenda, Sijie recently launched a
> BP about 128bit ledger id and there were some other discussions on Slack.
> I am preparing 4.8 release.
>
> Feel free to add items to the agenda or to propose your topics during the
> meeting.
>
> Enrico
>

Upcoming Comunity Meeting

[Enrico Olivelli]

Hi all,
We are going to met on Hangout at 8 AM PST

this is the agenda
https://cwiki.apache.org/confluence/display/BOOKKEEPER/2018-09-06+Meeting+Notes

you can find the link on the website
https://bookkeeper.apache.org/community/meeting/

Personally I do not have items for the agenda, Sijie recently launched a BP
about 128bit ledger id and there were some other discussions on Slack.
I am preparing 4.8 release.

Feel free to add items to the agenda or to propose your topics during the
meeting.

Enrico

Re: [ANNOUNCE] Apache BookKeeper 4.7.2 released

[Enrico Olivelli - Diennea]

Congrats to every one !
And thunks to Sijie for being the release manager

Enrico


Il giorno mar, 04/09/2018 alle 14.12 -0700, Sijie Guo ha scritto:

The Apache BookKeeper team is proud to announce Apache BookKeeper version
4.7.2.

Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage
service optimized for
real-time workloads. It has been used for a fundamental service to build
reliable services.
It is also the log segment store for Apache DistributedLog and the message
store for Apache Pulsar.

This is the 12th release of Apache BookKeeper.

This is a bugfix release, which fixes a bunch of issues reported from users
of 4.7.1.
These fixes include bug fixes around DbLedgerStorage, failure handling
around ensemble changes, bookie shutdown and such.

For BookKeeper release details and downloads, visit:

https://bookkeeper.apache.org/releases/

BookKeeper 4.7.2 Release Notes are at:

https://bookkeeper.apache.org/docs/4.7.2/overview/releaseNotes/

We would like to thank the contributors that made the release possible.

Regards,

The BookKeeper Team


--

Enrico Olivelli
Software Development Manager @ Diennea - MagNews
Tel.: (+39) 0546 066100 - Int. 125
Viale G.Marconi 30/14 - 48018 Faenza (RA)



Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email 
marketing! http://www.magnews.it/newsletter/

The information in this email is confidential and may be legally privileged. If 
you are not the intended recipient please notify the sender immediately and 
destroy this email. Any unauthorized, direct or indirect, disclosure, copying, 
storage, distribution or other use is strictly forbidden.

Re: Agenda for Next Community meeting - 19th July (today)

[Enrico Olivelli]

Here are the minutes
https://cwiki.apache.org/confluence/display/BOOKKEEPER/2018-07-19+Meeting+Notes

Thank you to all the attendees !

Cheers
Enrico

Il giorno gio 26 lug 2018 alle ore 12:21 Enrico Olivelli <
eolive...@gmail.com> ha scritto:

> Hi,
> We are going to meet on Hangout for the usual bi-weekly community meeting
>
> I have draft an agenda, just only writing down all the topics under
> discussion recently
>
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/2018-07-19+Meeting+Notes
>
> Feel free to join !
>
>
> Enrico
>

Agenda for Next Community meeting - 19th July (today)

[Enrico Olivelli]

Hi,
We are going to meet on Hangout for the usual bi-weekly community meeting

I have draft an agenda, just only writing down all the topics under
discussion recently
https://cwiki.apache.org/confluence/display/BOOKKEEPER/2018-07-19+Meeting+Notes

Feel free to join !


Enrico

Re: latency of bookkeeper

[Enrico Olivelli]

Hi (your name?),
first step
how does your client look like ?

WriteHandle writer = bookkeeper.createLedgerOp().execute().get();

BAD WAY
writer.append() ---> SYNC, will block until fsync on bookies !!!
writer.append() ---> SYNC
writer.append() ---> SYNC
writer.append() ---> SYNC
writer.append() ---> SYNC

GOOD WAY
writer.appendAsync() ---> ASYNC
writer.appendAsync() ---> ASYNC
writer.appendAsync() ---> ASYNC
writer.appendAsync() ---> ASYNC
writer.appendAsync() ---> ASYNC
writer.appendAsync() ---> ASYNC
writer.appendAsync().get()BLOCK ONLY ON THE last entry (or after a
batch of X entries)

Cheers
Enrico





Il giorno ven 6 lug 2018 alle ore 08:46 li.peng...@zhaopin.com.cn <
li.peng...@zhaopin.com.cn> ha scritto:

>
> Hi
>
> I try to use bookkeeper.  i care latency of write. so start a
> test in single thread.  get 400 ops/s  in double SSD.
>
> how to improve performance to get the low-latency.
>
> Thanks.
>
> --
> li.peng...@zhaopin.com.cn
>

Re: [ANNOUNCE] Apache BookKeeper 4.7.1 released

[Enrico Olivelli]

Great! Thank you Sijie

Enrico

Il mer 20 giu 2018, 17:31 Sijie Guo  ha scritto:

> The Apache BookKeeper team is proud to announce Apache BookKeeper version
> 4.7.1.
>
> Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage
> service optimized for
> real-time workloads. It has been used for a fundamental service to build
> reliable services.
> It is also the log segment store for Apache DistributedLog and the message
> store for Apache Pulsar.
>
> This is the 11th release of Apache BookKeeper.
>
> This is a bugfix release, which fixes a bunch of issues reported from
> users of 4.7.0.
> These changes include bug fixes around ledger cache and object pooling,
> performance
> enhancement avoid memory copies and various bug fixes and improvements
> around
> bookkeeper table service.
>
> For BookKeeper release details and downloads, visit:
>
> https://bookkeeper.apache.org/releases/
>
> BookKeeper 4.7.1 Release Notes are at:
>
> https://bookkeeper.apache.org/docs/4.7.1/overview/releaseNotes/
>
> We would like to thank the contributors that made the release possible.
>
> Regards,
>
> The BookKeeper Team
>
-- 


-- Enrico Olivelli

Re: Community Meeting

[Enrico Olivelli]

Il giorno gio 14 giu 2018 alle ore 15:40 Venkateswara Rao Jujjuri <
jujj...@gmail.com> ha scritto:

> Sorry I can’t make it today. But I will get to the BP 14 review today.
> Please publish minutes. Thanks for taking care of this
>
> JV
>


Here are  the minutes
https://cwiki.apache.org/confluence/display/BOOKKEEPER/2018-06-14+Meeting+Notes

Cheers
Enrico



>
> On Thu, Jun 14, 2018 at 4:55 AM Enrico Olivelli 
> wrote:
>
>> Hi all,
>> we are going to "meet" at 8 AM (PST) / 5 PM (CET) today !
>>
>> Here is the lint to the hangout room
>> https://bookkeeper.apache.org/community/meeting/
>>
>> I apologize for not send this email in advance.
>> I suggest to every one to add the invitation to your personal Google
>> Calendar :-)
>>
>> Agenda (very skinny today)
>>
>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/2018-06-14+Meeting+Notes
>>
>>
>> See you there
>> Enrico
>>
>> --
> Sent from iPhone
>

Community Meeting

[Enrico Olivelli]

Hi all,
we are going to "meet" at 8 AM (PST) / 5 PM (CET) today !

Here is the lint to the hangout room
https://bookkeeper.apache.org/community/meeting/

I apologize for not send this email in advance.
I suggest to every one to add the invitation to your personal Google
Calendar :-)

Agenda (very skinny today)
https://cwiki.apache.org/confluence/display/BOOKKEEPER/2018-06-14+Meeting+Notes


See you there
Enrico

Re: [ANNOUNCE] Apache BookKeeper 4.7.0 released

[Enrico Olivelli]

Great news !

Thank you Sijie

Enrico

Il ven 20 apr 2018, 12:17 Jia Zhai <zhaiji...@gmail.com> ha scritto:

> con~
>
> On Fri, Apr 20, 2018 at 4:34 PM, Sijie Guo <si...@apache.org> wrote:
>
>> The Apache BookKeeper team is proud to announce Apache BookKeeper version
>> 4.7.0.
>>
>> Apache BookKeeper is a scalable, fault-tolerant, and low-latency
>> storage service
>> optimized for real-time workloads. It has been used for a fundamental
>> service to build
>> reliable services. It is also the log segment store for Apache
>> DistributedLog and the message
>> store for Apache Pulsar.
>>
>> This is the 10th release of Apache BookKeeper.
>>
>> This is a feature release. It is a great milestone for Apache BookKeeper.
>> We finally merge changes from both Salesforce and Yahoo branches to
>> upstream. Apache Pulsar also switched to use official 4.7.0 release for
>> its
>> upcoming 2.0 release.  It is also the first release of Apache
>> DistributedLog after it is merged as sub modules of Apache BookKeeper.
>>
>> This release adopts 7 BPs and fixes 446 issues. It introduces new metadata
>> service API, rocksdb based ledger storage, have extensive performance
>> enhancements around reducing jvm garbage collection, reducing lock
>> contentions, and various improvements around operability.
>>
>> For BookKeeper release details and downloads, visit:
>>
>> https://bookkeeper.apache.org/releases/
>>
>> BookKeeper 4.7.0 Release Notes are at:
>>
>> http://bookkeeper.apache.org/docs/4.7.0/overview/releaseNotes/
>>
>> We would like to thank all the contributors that made the release
>> possible.
>>
>> Regards,
>>
>> The BookKeeper Team
>>
>
> --


-- Enrico Olivelli

Re: Help with bad errors on 4.6.1

[Enrico Olivelli]

End of this story
With this patch the problem does not occur anymore

https://github.com/apache/bookkeeper/pull/1293

The patch does not address directly the problem, the root source is still
unknown, this is very bad. But with that change no error is reported
anymore, so actually it is enough to go to production with java9 (10) and
my application.

I will start release process for 4.6.2 soon.

Thanks to everyone who helped

Enrico

Il ven 16 mar 2018, 12:30 Enrico Olivelli <eolive...@gmail.com> ha scritto:

> resending (GMAIL webmail messed up prev email)
>
>
>
> 2018-03-16 10:34 GMT+01:00 Sijie Guo <guosi...@gmail.com>:
>
>> On Fri, Mar 16, 2018 at 2:26 AM, Enrico Olivelli <eolive...@gmail.com>
>> wrote:
>>
>> > Thank you Sijie,
>> > I have already applied a similar patch to my local code based on 4.6.1
>> but
>> > the problem remains.
>> >
>>
>> What do you mean "the problem" here?
>>
>
> missing buf.release(), that you fixed on 4.7 and will cherry pick to 4.6
> branch
>
>
>
>> Do you mean the corruption problem or the leaking problem? The change I
>> pointed out only address the leaking problem. I don't think it is the
>> problem of corruption.
>>
>>
> Actually we are talking about a wire protocol level corruption.
> The only good "fix" that I have at the moment is to use use
> UnpooledByteBufAllocator.DEFAULT in encoder methods for v3
>
>
> Enrico
>
>
>
>
>
>>
>> >
>> > I am looking into Netty allocateUnitializedArray, which is used for
>> Pooled
>> > Heap Buffers.
>> > You all are more aware of BK code than me, is there any point in which
>> we
>> > assume that the buffer is full of zeroes ?
>> >
>> >
>> > Enrico
>> >
>> >
>> >
>> >
>> > 2018-03-16 10:22 GMT+01:00 Ivan Kelly <iv...@apache.org>:
>> >
>> > > > With "paranoid" log in Netty I found this that is very interesting,
>> but
>> > > it
>> > > > happens even on Java 8.
>> > > I don't think leaks are the problem here though. This seems to be more
>> > > like a doublefree issue.
>> > >
>> > > -Ivan
>> > >
>> >
>>
> --


-- Enrico Olivelli

Re: Help with bad errors on 4.6.1

[Enrico Olivelli]

2018-03-09 8:59 GMT+01:00 Sijie Guo <guosi...@gmail.com>:

> Sent out a PR for the issues that I observed:
>
> https://github.com/apache/bookkeeper/pull/1240
>


Other findings:
- my problem is not related to jdk9, it happens with jdk8 too
- the "tailing reader" is able to make progress and follow the WAL, so not
all the reads fail
- the "writer" is able to make progress and write to the WAL, so not all
the write fail

I have run BK 4.6.1 tests with Netty 4.1.21Final but there is no issue
(quite the OS as the machines with the error, I am on Fedora, machines are
on CentOS)

Enrico


>
>
> On Thu, Mar 8, 2018 at 10:47 PM, Sijie Guo <guosi...@gmail.com> wrote:
>
>> So the problem here is:
>>
>> - a corrupted request failed the V3 request decoder, so bookie switched
>> to use v2 request decoder. Once the switch happen, the bookie will always
>> use v2 request decoder decoding v3 request. then all your v3 requests will
>> be failing with unknown op and trigger the bytebuf double releasing issue.
>>
>> - a corrupted response failed the V3 response decoder, so client switched
>> to use v3 response decoder. Once the switch happen, the client will always
>> use v2 request decoder decoding v3 response. so all the responses will be
>> failing with " Received unknown response : op code"
>>
>> Although I don't know how the first request/response happened (it can be
>> any issue, even network corruption), the problem is once this happen,
>> either client/bookie will be forcing to use v2 request/response decoder and
>> never change. so the problem will remain until you restarted. And this is
>> the behavior that Enrico is seeing.
>>
>> There are a couple of issues to address here:
>>
>> 1) we need to add a flag to disable falling back to use v2
>> request/response coder and make sure it always use v3 protocol. In this
>> way, we will guarantee the problem not persist in the channel level.
>> 2) we need to throw exception on unknown op code at request decode :
>> https://github.com/apache/bookkeeper/blob/master/bookkeepe
>> r-server/src/main/java/org/apache/bookkeeper/proto/
>> BookieProtoEncoding.java#L195 . As what we did at response decoder :
>> https://github.com/apache/bookkeeper/blob/master/bookkeepe
>> r-server/src/main/java/org/apache/bookkeeper/proto/
>> BookieProtoEncoding.java#L304 in https://github.com/apache/b
>> ookkeeper/issues/198
>>
>>
>> Details are listed as below:
>>
>> --
>>
>> Details:
>>
>> - The client side stacktrace clearly showed that it is using v2 decoder
>> on decoding responses. That means client failed to parse v3 response and it
>> falls back to use v2 decoder on decoding responses. Because it is a
>> "corrupted" v3 response, so v2 decoder can't
>> find a right op code.  Then it throws illegal state exception.
>>
>>
>> *Caused by: java.lang.IllegalStateException: Received unknown response :
>> op code = 6at
>> org.apache.bookkeeper.proto.BookieProtoEncoding$ResponseEnDeCoderPreV3.decode(BookieProtoEncoding.java:294)*
>>
>> - For the stacktrace at bookie side:
>>
>> 1. It is clear that BookieRequestHandler#L77 is called. That means the
>> message is neither v2 request nor v3 request. It is a ByteBuf.
>>
>> https://github.com/apache/bookkeeper/blob/master/bookkeeper-
>> server/src/main/java/org/apache/bookkeeper/proto/
>> BookieRequestHandler.java#L77
>>
>> 2. V3 request decoder is using protobuf to decode bytes array. If it is
>> not a valid v3 request, it will always throw exceptions. so the code
>> mentioned above will never be triggered
>>
>> https://github.com/apache/bookkeeper/blob/master/bookkeeper-
>> server/src/main/java/org/apache/bookkeeper/proto/
>> BookieProtoEncoding.java#L344
>>
>> 3. The only case that BookieRequestHandler#L77 can happen is v2 request
>> decoder doesn't parse the buffer. This leads to the bug in
>>
>> https://github.com/apache/bookkeeper/blob/master/bookkeeper-
>> server/src/main/java/org/apache/bookkeeper/proto/
>> BookieProtoEncoding.java#L194
>>
>>
>> - How this can happen?
>>
>> If the client is using v3 protocol, but both client and bookie side
>> stacktraces show it is using v2 protocol, that means there was a corruption
>> causing client and bookie switching from v3 protocol to v2 protocol.
>>
>> https://github.com/apache/bookkeeper/blob/master/bookkeeper-
>> server/src/main/java/org/apache/bookkeeper/proto/
>> BookieProtoEncoding.java#L502
>> https://github.com/ap

Re: Help with bad errors on 4.6.1

[Enrico Olivelli]

Il gio 8 mar 2018, 22:22 Enrico Olivelli <eolive...@gmail.com> ha scritto:

>
>
> Il gio 8 mar 2018, 22:08 Sijie Guo <guosi...@gmail.com> ha scritto:
>
>> This is even interesting with v3 protocol. Currently the v3 protocol
>> copies the byte string and it doesn’t even use pooled buffer. So I am not
>> sure if the issue comes from bookie or an issue at the netty layer. It
>> would be good to get a repo.
>>
>
> Interesting idea! I have upgraded to latest netty version both bookie and
> clients in this new version of my application!
>
> In production I have an older version of Netty,
>
4.1.21, not 4.1.22



I cannot check which version now, I will report back the version within
> some hour, when I will have access to my work laptop
>

> Enrico
>
>
>> Sijie
>>
>> On Thu, Mar 8, 2018 at 12:29 PM Enrico Olivelli <eolive...@gmail.com>
>> wrote:
>>
>>>
>>>
>>> Il gio 8 mar 2018, 20:52 Sijie Guo <guosi...@gmail.com> ha scritto:
>>>
>>>> On Thu, Mar 8, 2018 at 7:42 AM, Enrico Olivelli <eolive...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> 2018-03-08 14:50 GMT+01:00 Ivan Kelly <iv...@apache.org>:
>>>>>
>>>>>> It just occurred to me that this could be a problem with the recycler.
>>>>>> If we recycle a buffer too early, but then keep using it, another user
>>>>>> could pick it up, and between them they could corrupt the data that
>>>>>> would cause it to be unhandled.
>>>>>>
>>>>>
>>>>>
>>>>> Maybe current work from Sijie, Andrey and JV which Sijie is
>>>>> backporting to branch-4.6 could resolve the issue
>>>>>
>>>>
>>>>
>>>> I think that is unrelated. The change Andrey, JV and me dealt with is
>>>> the PendingAddOp. It is on client side not bookie side.
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> -Ivan
>>>>>>
>>>>>> On Thu, Mar 8, 2018 at 1:49 PM, Enrico Olivelli <eolive...@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> >
>>>>>> > 2018-03-08 11:57 GMT+01:00 Ivan Kelly <iv...@apache.org>:
>>>>>> >>
>>>>>> >> Hmm, yes, looks messed up. Do you have a reliable repro of this?
>>>>>> >>
>>>>>> >> "at
>>>>>> io.netty.channel.DefaultChannelPipeline.onUnhandledInboundMessage"
>>>>>> >> <- it seems the message itself wasn't handled. Do you have any more
>>>>>> >> bookie side logs leading up to this event?
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > The only suspicious logs are for logger named
>>>>>> > "org.apache.bookkeeper.bookie.EntryLogger"
>>>>>> >
>>>>>> >
>>>>>> > 18-03-08-09-22-40   Created new entry log file
>>>>>> > /data/qasql/bookie/./bookie_data/current/4b0.log for logId 1200.
>>>>>> > 18-03-08-09-32-41   Failed to get ledgers map index from:
>>>>>> 53.log : No
>>>>>> > ledgers map index found on entryLogId53
>>>>>> > 18-03-08-09-33-10   Failed to get ledgers map index from:
>>>>>> 54.log : No
>>>>>> > ledgers map index found on entryLogId54
>>>>>> > 18-03-08-09-33-10   Failed to get ledgers map index from:
>>>>>> 375.log : No
>>>>>> > ledgers map index found on entryLogId375
>>>>>> > 18-03-08-09-33-13   Failed to get ledgers map index from:
>>>>>> 878.log : No
>>>>>> > ledgers map index found on entryLogId878
>>>>>> > 18-03-08-09-33-22   Failed to get ledgers map index from:
>>>>>> 1197.log : No
>>>>>> > ledgers map index found on entryLogId1197
>>>>>> > 18-03-08-09-33-57   Failed to get ledgers map index from:
>>>>>> 1198.log : No
>>>>>> > ledgers map index found on entryLogId1198
>>>>>> > 18-03-08-09-34-54   Failed to get ledgers map index from:
>>>>>> 1199.log : No
>>>>&g

Re: Help with bad errors on 4.6.1

[Enrico Olivelli]

Il gio 8 mar 2018, 21:33 Andrey Yegorov <andrey.yego...@gmail.com> ha
scritto:

> I am looking more at the PendigAddOp and it looks like, in addition to the
> case that Sijie has fixed, there is another scenario where recycler can get
> triggered.
>

I think that this is another issue, but your idea makes sense to me.

This problem is on bookie side.

>
> I.e.:
> first sendWriteRequest() in PendingAddOp.safeRun() fails in PCBC's
> writeAndFlush (channel == null or channel.writeAndFlush throws)
> hasRun is set to true at that point. pendingWriteRequests is still zero
> (not incremented yet)
> PCBC's errorOut triggers callback that is sure that since
> pendingWriteRequests==0 and hasRun it can release PendingAddOp back to
> recycler
>
> now there are still sendWriteRequest's to follow and all bets are off.
> i.e. by the time they are executed the addOp can get reused so they end up
> sending another request to the wrong bookie.
>
> So far my idea of solution for that is to run PCBC's errorOut
> asynchronously / submit it back to orderedSafeExecutor so addOp can
> increment counts properly.
>
>
>
> --
> Andrey Yegorov
>
> On Thu, Mar 8, 2018 at 11:52 AM, Sijie Guo <guosi...@gmail.com> wrote:
>
>>
>>
>> On Thu, Mar 8, 2018 at 7:42 AM, Enrico Olivelli <eolive...@gmail.com>
>> wrote:
>>
>>>
>>>
>>> 2018-03-08 14:50 GMT+01:00 Ivan Kelly <iv...@apache.org>:
>>>
>>>> It just occurred to me that this could be a problem with the recycler.
>>>> If we recycle a buffer too early, but then keep using it, another user
>>>> could pick it up, and between them they could corrupt the data that
>>>> would cause it to be unhandled.
>>>>
>>>
>>>
>>> Maybe current work from Sijie, Andrey and JV which Sijie is backporting
>>> to branch-4.6 could resolve the issue
>>>
>>
>>
>> I think that is unrelated. The change Andrey, JV and me dealt with is the
>> PendingAddOp. It is on client side not bookie side.
>>
>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>> -Ivan
>>>>
>>>> On Thu, Mar 8, 2018 at 1:49 PM, Enrico Olivelli <eolive...@gmail.com>
>>>> wrote:
>>>> >
>>>> >
>>>> > 2018-03-08 11:57 GMT+01:00 Ivan Kelly <iv...@apache.org>:
>>>> >>
>>>> >> Hmm, yes, looks messed up. Do you have a reliable repro of this?
>>>> >>
>>>> >> "at
>>>> io.netty.channel.DefaultChannelPipeline.onUnhandledInboundMessage"
>>>> >> <- it seems the message itself wasn't handled. Do you have any more
>>>> >> bookie side logs leading up to this event?
>>>> >
>>>> >
>>>> >
>>>> > The only suspicious logs are for logger named
>>>> > "org.apache.bookkeeper.bookie.EntryLogger"
>>>> >
>>>> >
>>>> > 18-03-08-09-22-40   Created new entry log file
>>>> > /data/qasql/bookie/./bookie_data/current/4b0.log for logId 1200.
>>>> > 18-03-08-09-32-41   Failed to get ledgers map index from: 53.log
>>>> : No
>>>> > ledgers map index found on entryLogId53
>>>> > 18-03-08-09-33-10   Failed to get ledgers map index from: 54.log
>>>> : No
>>>> > ledgers map index found on entryLogId54
>>>> > 18-03-08-09-33-10   Failed to get ledgers map index from: 375.log
>>>> : No
>>>> > ledgers map index found on entryLogId375
>>>> > 18-03-08-09-33-13   Failed to get ledgers map index from: 878.log
>>>> : No
>>>> > ledgers map index found on entryLogId878
>>>> > 18-03-08-09-33-22   Failed to get ledgers map index from:
>>>> 1197.log : No
>>>> > ledgers map index found on entryLogId1197
>>>> > 18-03-08-09-33-57   Failed to get ledgers map index from:
>>>> 1198.log : No
>>>> > ledgers map index found on entryLogId1198
>>>> > 18-03-08-09-34-54   Failed to get ledgers map index from:
>>>> 1199.log : No
>>>> > ledgers map index found on entryLogId1199
>>>>
>>>
>>
>> This is also unrelated. It is a fine logging, since it means it attempts
>> to read ledgers map index, but didn't find one.
>>
>> the ledgers map index is only written when it is enabled at the bookies.
>>
>>
>

Re: Help with bad errors on 4.6.1

[Enrico Olivelli]

Il gio 8 mar 2018, 20:52 Sijie Guo <guosi...@gmail.com> ha scritto:

> On Thu, Mar 8, 2018 at 7:42 AM, Enrico Olivelli <eolive...@gmail.com>
> wrote:
>
>>
>>
>> 2018-03-08 14:50 GMT+01:00 Ivan Kelly <iv...@apache.org>:
>>
>>> It just occurred to me that this could be a problem with the recycler.
>>> If we recycle a buffer too early, but then keep using it, another user
>>> could pick it up, and between them they could corrupt the data that
>>> would cause it to be unhandled.
>>>
>>
>>
>> Maybe current work from Sijie, Andrey and JV which Sijie is backporting
>> to branch-4.6 could resolve the issue
>>
>
>
> I think that is unrelated. The change Andrey, JV and me dealt with is the
> PendingAddOp. It is on client side not bookie side.
>
>
>>
>>
>>
>>
>>
>>>
>>> -Ivan
>>>
>>> On Thu, Mar 8, 2018 at 1:49 PM, Enrico Olivelli <eolive...@gmail.com>
>>> wrote:
>>> >
>>> >
>>> > 2018-03-08 11:57 GMT+01:00 Ivan Kelly <iv...@apache.org>:
>>> >>
>>> >> Hmm, yes, looks messed up. Do you have a reliable repro of this?
>>> >>
>>> >> "at io.netty.channel.DefaultChannelPipeline.onUnhandledInboundMessage"
>>> >> <- it seems the message itself wasn't handled. Do you have any more
>>> >> bookie side logs leading up to this event?
>>> >
>>> >
>>> >
>>> > The only suspicious logs are for logger named
>>> > "org.apache.bookkeeper.bookie.EntryLogger"
>>> >
>>> >
>>> > 18-03-08-09-22-40   Created new entry log file
>>> > /data/qasql/bookie/./bookie_data/current/4b0.log for logId 1200.
>>> > 18-03-08-09-32-41   Failed to get ledgers map index from: 53.log :
>>> No
>>> > ledgers map index found on entryLogId53
>>> > 18-03-08-09-33-10   Failed to get ledgers map index from: 54.log :
>>> No
>>> > ledgers map index found on entryLogId54
>>> > 18-03-08-09-33-10   Failed to get ledgers map index from: 375.log
>>> : No
>>> > ledgers map index found on entryLogId375
>>> > 18-03-08-09-33-13   Failed to get ledgers map index from: 878.log
>>> : No
>>> > ledgers map index found on entryLogId878
>>> > 18-03-08-09-33-22   Failed to get ledgers map index from: 1197.log
>>> : No
>>> > ledgers map index found on entryLogId1197
>>> > 18-03-08-09-33-57   Failed to get ledgers map index from: 1198.log
>>> : No
>>> > ledgers map index found on entryLogId1198
>>> > 18-03-08-09-34-54   Failed to get ledgers map index from: 1199.log
>>> : No
>>> > ledgers map index found on entryLogId1199
>>>
>>
>
> This is also unrelated. It is a fine logging, since it means it attempts
> to read ledgers map index, but didn't find one.
>
> the ledgers map index is only written when it is enabled at the bookies.
>
>
>> >
>>> > I will try to run Bookie with level debug, but it produces lots of spam
>>> >
>>> > Thanks
>>> > Enrico
>>> >
>>> >>
>>> >>
>>> >> -Ivan
>>> >>
>>> >> On Thu, Mar 8, 2018 at 9:18 AM, Enrico Olivelli <eolive...@gmail.com>
>>> >> wrote:
>>> >> > Hi all,
>>> >> > I am seeing this bad errors in some test environments with 4.6.1
>>> >> > The errors appear during rolling restart of the application, the
>>> test
>>> >> > env is
>>> >> > made of 6 machines:
>>> >> > - 3 bookies
>>> >> > - 3 client machines (with several BK clients, of different
>>> sub-systems)
>>> >> > - running with 4.6.1 both client and servers
>>> >> >
>>> >> > *I do not have* reports of this errors from production, already
>>> running
>>> >> > 4.6.1 for the last month
>>> >> >
>>> >> > But the problem is quite scary
>>> >> >
>>> >> > This is a sample of relevant errors on clients (in this case
>>> Majordodo
>>> >> > brokers, with log level = INFO)
>>> >> >
>>> >> > logs/org.apache.bookkeeper.client.PendingAddOp:
>>> >> > 18-03-07-09-51-43 

Re: Help with bad errors on 4.6.1

[Enrico Olivelli]

2018-03-08 14:50 GMT+01:00 Ivan Kelly <iv...@apache.org>:

> It just occurred to me that this could be a problem with the recycler.
> If we recycle a buffer too early, but then keep using it, another user
> could pick it up, and between them they could corrupt the data that
> would cause it to be unhandled.
>


Maybe current work from Sijie, Andrey and JV which Sijie is backporting to
branch-4.6 could resolve the issue





>
> -Ivan
>
> On Thu, Mar 8, 2018 at 1:49 PM, Enrico Olivelli <eolive...@gmail.com>
> wrote:
> >
> >
> > 2018-03-08 11:57 GMT+01:00 Ivan Kelly <iv...@apache.org>:
> >>
> >> Hmm, yes, looks messed up. Do you have a reliable repro of this?
> >>
> >> "at io.netty.channel.DefaultChannelPipeline.onUnhandledInboundMessage"
> >> <- it seems the message itself wasn't handled. Do you have any more
> >> bookie side logs leading up to this event?
> >
> >
> >
> > The only suspicious logs are for logger named
> > "org.apache.bookkeeper.bookie.EntryLogger"
> >
> >
> > 18-03-08-09-22-40   Created new entry log file
> > /data/qasql/bookie/./bookie_data/current/4b0.log for logId 1200.
> > 18-03-08-09-32-41   Failed to get ledgers map index from: 53.log : No
> > ledgers map index found on entryLogId53
> > 18-03-08-09-33-10   Failed to get ledgers map index from: 54.log : No
> > ledgers map index found on entryLogId54
> > 18-03-08-09-33-10   Failed to get ledgers map index from: 375.log :
> No
> > ledgers map index found on entryLogId375
> > 18-03-08-09-33-13   Failed to get ledgers map index from: 878.log :
> No
> > ledgers map index found on entryLogId878
> > 18-03-08-09-33-22   Failed to get ledgers map index from: 1197.log :
> No
> > ledgers map index found on entryLogId1197
> > 18-03-08-09-33-57   Failed to get ledgers map index from: 1198.log :
> No
> > ledgers map index found on entryLogId1198
> > 18-03-08-09-34-54   Failed to get ledgers map index from: 1199.log :
> No
> > ledgers map index found on entryLogId1199
> >
> > I will try to run Bookie with level debug, but it produces lots of spam
> >
> > Thanks
> > Enrico
> >
> >>
> >>
> >> -Ivan
> >>
> >> On Thu, Mar 8, 2018 at 9:18 AM, Enrico Olivelli <eolive...@gmail.com>
> >> wrote:
> >> > Hi all,
> >> > I am seeing this bad errors in some test environments with 4.6.1
> >> > The errors appear during rolling restart of the application, the test
> >> > env is
> >> > made of 6 machines:
> >> > - 3 bookies
> >> > - 3 client machines (with several BK clients, of different
> sub-systems)
> >> > - running with 4.6.1 both client and servers
> >> >
> >> > *I do not have* reports of this errors from production, already
> running
> >> > 4.6.1 for the last month
> >> >
> >> > But the problem is quite scary
> >> >
> >> > This is a sample of relevant errors on clients (in this case Majordodo
> >> > brokers, with log level = INFO)
> >> >
> >> > logs/org.apache.bookkeeper.client.PendingAddOp:
> >> > 18-03-07-09-51-43   Write of ledger entry to quorum failed:
> L366634
> >> > E2557
> >> > 18-03-07-09-51-43   Write of ledger entry to quorum failed:
> L366634
> >> > E2558
> >> > 18-03-07-09-51-43   Write of ledger entry to quorum failed:
> L366634
> >> > E2559
> >> > 18-03-07-15-59-55   Failed to write entry (366680, 1865): Bookie
> >> > operation timeout
> >> > 18-03-07-15-59-55   Failed to write entry (366680, 1865): Bookie
> >> > operation timeout
> >> > 18-03-07-16-00-00   Failed to write entry (366680, 1865): Bookie
> >> > operation timeout
> >> > 18-03-07-16-00-00   Failed to write entry (366680, 1865): Bookie
> >> > operation timeout
> >> > 18-03-07-16-00-05   Failed to write entry (366680, 1865): Bookie
> >> > operation timeout
> >> >
> >> >
> >> > org.apache.bookkeeper.proto.PerChannelBookieClient:
> >> >
> >> > 18-03-07-10-06-44   Unexpected exception caught by bookie client
> >> > channel
> >> > handler
> >> > 18-03-07-10-06-44   io.netty.handler.codec.DecoderException:
> >> > java.lang.IllegalStateException: Received unknown response : op code
> = 6
>

Re: [ANNOUNCE] New BookKeeper Committer: Andrey Yegorov

[Enrico Olivelli]

Il mar 20 feb 2018, 20:51 Andrey Yegorov <andrey.yego...@gmail.com> ha
scritto:

> Thank you, guys!
>

Feel free to send  PR to add yourself to the list on the website

Enrico

>
> --
> Andrey Yegorov
>
> On Tue, Feb 20, 2018 at 4:05 AM, Ivan Kelly <iv...@apache.org> wrote:
>
>> Welcome to the team Andrey and Sam! Great to see the team growing :D
>>
>> On Tue, Feb 20, 2018 at 12:36 PM, Enrico Olivelli <eolive...@gmail.com>
>> wrote:
>> > Congrats Andrey!
>> >
>> > Enrico
>> >
>> >
>> > Il mar 20 feb 2018, 11:16 Sijie Guo <guosi...@gmail.com> ha scritto:
>> >>
>> >> The Apache BookKeeper PMC recently extended committer karma to Andrey
>> >> Yegorov and he has accepted.
>> >>
>> >> Andrey Yegorov has been making contributions to bookkeeper community
>> since
>> >> a few years ago. He is active in sending pull requests, attending the
>> >> discussions in the community meetings. He made a couple of great
>> >> contributions to bookkeeper, such as bookie compaction, performance
>> tuning
>> >> and so on. It is great to have him onboard as bookkeeper committer. We
>> are
>> >> looking forward to more contributions from him.
>> >>
>> >> Congratulations and welcome onboard, Andrey!
>> >>
>> >> Sijie on behave of the BookKeeper PMC
>> >>
>> >>
>> > --
>> >
>> >
>> > -- Enrico Olivelli
>>
>
> --


-- Enrico Olivelli

Problem with BookKeeper 4.6.1 on Java 9

[Enrico Olivelli]

Hi guys,
did you ever see this error ?
It is very strange, I am investigating.
I am using 4.6.1 bookkeeper-server-shaded, I get this only in some unit
tests, not in "real" environments on Java 9


java.lang.RuntimeException: java.lang.IncompatibleClassChangeError:
Inconsistent constant
pool data in classfile for class org/apache/bookkeeper/versioning/Version.
Method
lambda$static$0(Lorg/apache/bookkeeper/versioning/Version;)Lorg/apache/bookkeeper/versioning/Version$Occurred;
at index 51 is CONSTANT_MethodRef and should be CONSTANT_InterfaceMethodRef
at x.tasklog.TaskLoggerTest.test(TaskLoggerTest.java:75)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)
Caused by: java.lang.IncompatibleClassChangeError: Inconsistent constant
pool data in classfile for class org/apache/bookkeeper/versioning/Version.
Method
lambda$static$0(Lorg/apache/bookkeeper/versioning/Version;)Lorg/apache/bookkeeper/versioning/Version$Occurred;
at index 51 is CONSTANT_MethodRef and should be CONSTANT_InterfaceMethodRef
at org.apache.bookkeeper.versioning.Version.(Version.java:47)
at
org.apache.bookkeeper.discover.ZKRegistrationClient$WatchTask.(ZKRegistrationClient.java:79)
at
org.apache.bookkeeper.discover.ZKRegistrationClient.watchWritableBookies(ZKRegistrationClient.java:287)
at
org.apache.bookkeeper.client.BookieWatcher.initialBlockingBookieRead(BookieWatcher.java:152)
at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:506)
at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:365)
at
org.apache.bookkeeper.client.BookKeeperAdmin.format(BookKeeperAdmin.java:1181)
at ..

Enrico

Re: [ANNOUNCE] New BookKeeper Committer: Yiming Zang

[Enrico Olivelli]

Congrats Yiming!
Enrico

Il lun 18 dic 2017, 19:56 Sijie Guo <si...@apache.org> ha scritto:

> The Apache BookKeeper PMC recently extended committer karma to Yiming Zang
> and he has accepted.
>
> Yiming Zang has been contributing bookkeeper & distributedlog changes to
> the community for a while. He is active in code contributions, discussion
> and code reviews. We are looking forward to more contributions from him.
>
> Congratulations and welcome onboard, Yiming!
>
> Sijie on behave of the BookKeeper PMC
>
-- 


-- Enrico Olivelli

[ANNOUNCE] Apache BookKeeper 4.5.1 released

[Enrico Olivelli]

The Apache BookKeeper team is proud to announce Apache BookKeeper version
4.5.1.

Apache BookKeeper is a scalable, fault-tolerant, and low-latency
storage service optimized for
real-time workloads. It has been used for a fundamental service to
build reliable services.
It is also the log segment store for Apache DistributedLog and the
message store for Apache Pulsar.

This is the 6th release of the Apache BookKeeper.

This is a bugfix release, it fixes bugs around parallel recovery,
Prometheus stats provider and placement policies.

For BookKeeper release details and downloads, visit:
 http://www.apache.org/dyn/closer.cgi/bookkeeper

BookKeeper 4.5.1 Release Notes are at:
https://bookkeeper.apache.org/docs/4.5.1/overview/releaseNotes/

We would like to thank the contributors that made the release possible.

Regards,

The BookKeeper Team

[RESULT] [VOTE] Release 4.5.1, release candidate #1

[Enrico Olivelli]

I'm happy to announce that we have unanimously approved this release.

There are 4 approving votes, 3 of which are binding:
* Sijie Guo
* Ivan Kelly
* Dave Rusek
* Jia Zhai (non binding)

There are no disapproving votes.

Thanks everyone!

Enrico Olivelli

Re: log compaction of entries

[Enrico Olivelli]

Il mer 8 nov 2017, 16:27 Ivan Kelly <iv...@apache.org> ha scritto:

> > I believe I'm missing the implication of this. Does that mean we need
> > to logically name ledgers in a way that can keep track, because each
> > has only one uninterrupted session of write operations, otherwise it
> > is read only?
> It's not possible to specify a name on ledger creation. When you
> create a ledger that ledger is assigned an ID. If you want to keep
> track of different ledgers, you need to store a mapping from a logical
> name to ledger is somewhere.
>

Nowadays you can specify manually a ledger id, but it still has to be a
positive long.
Enrico

>
> -Ivan
>
-- 


-- Enrico Olivelli

Re: HTTP / REST API?

[Enrico Olivelli]

2017-11-08 16:24 GMT+01:00 Ivan Kelly :

> >> Writing is difficult, because the java client writer is really the core
> of
> >> BookKeeper, it deals with fencing, ensemble changes
> >> It would be simple to write a client which pushes data to bookies but
> all
> >> the LAC+fencing protocols (+ metadata management) are hard to
> re-implement.
> The http client would have a record of LAC, since any entry acked by
> the http endpoint, would also by definition have to be acked by the
> bookies.
>
> > I don't think we need to reimplement all this details in http api. what
> > people probably need here, is streaming the entries to a ledger, all the
> > fencing are delegated to http endpoint.
> This means that the user should stick to the http endpoint for all
> adds on a single ledger, due single writer constraints. This is
> sensible, but it does mean a layer of coordination between endpoints
> to know where which ledger handles are opened.
>

Or simply we should support a WebSocket endpoint for writers, once the
websocket connection is closed the ledger gets closed and maybe for
"tailing" readers


>
> I think though, that it would make more sense to have the DL api
> behind an http, rather than the raw ledger API. In general I think we
> should be discouraging the use of the raw ledger handle apis, in
> favour of DL. In fact DL already has a stateless frontend with a
> thrift interface for writing, which is probably better than HTTP for
> this kind of workload anyhow.
>
> -Ivan
>

Re: HTTP / REST API?

[Enrico Olivelli]

2017-11-07 16:16 GMT+01:00 Ivan Kelly :

> > We have an issue to document this feature.  At the same time, you can
> > checkout this BP for details -
> > https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> BP-17%3A+Define+BookKeeper+public+http+endpoints
> BP-17 only defines endpoints for metadata operations though, nothing
> for adding entries ledger, or even creating a ledger.
>
> Perhaps we could add these to allow simple non-java clients to be written.
>

I think that reading data would be simple to be implemented, both in the
HTTP API and even for a non java language (because we are using protobuf).

Writing is difficult, because the java client writer is really the core of
BookKeeper, it deals with fencing, ensemble changes
It would be simple to write a client which pushes data to bookies but all
the LAC+fencing protocols (+ metadata management) are hard to re-implement.

Maybe the best option would be to add an HTTP API or some kind of proxy

I will be happy to see such API

Enrico


>
> -Ivan
>

Re: Cannot compact bookie due to NegativeArraySizeException on BK 4.5

[Enrico Olivelli]

Thank you Jia,
The error is clear.
The bookie is in fact able to work but it cannot reclaim space
I am not sure if a good patch would to simply drop the broken file...

Enrico

On mer 20 set 2017, 10:45 Jia Zhai <zhaiji...@gmail.com> wrote:

> Seems the failing place was scan the old EntryLog, not doing write, maybe
> it not broken the EntryLog file.
>
> The error indicates that we read a negative entrySize from entryLog file.
>
> /**
>  * Scan entry log.
>  *
>  * @param entryLogId Entry Log Id
>  * @param scanner Entry Log Scanner
>  * @throws IOException
>  */
> protected void scanEntryLog(long entryLogId, EntryLogScanner scanner) throws 
> IOException {
> ByteBuffer sizeBuff = ByteBuffer.allocate(4);
> ByteBuffer lidBuff = ByteBuffer.allocate(8);
> BufferedReadChannel bc;
> // Get the BufferedChannel for the current entry log file
> try {
> bc = getChannelForLogId(entryLogId);
> } catch (IOException e) {
> LOG.warn("Failed to get channel to scan entry log: " + entryLogId + 
> ".log");
> throw e;
> }
> // Start the read position in the current entry log file to be after
> // the header where all of the ledger entries are.
> long pos = LOGFILE_HEADER_SIZE;
>
> // Read through the entry log file and extract the ledger ID's.
> while (true) {
> // Check if we've finished reading the entry log file.
> if (pos >= bc.size()) {
> break;
> }
> if (readFromLogChannel(entryLogId, bc, sizeBuff, pos) != 
> sizeBuff.capacity()) {
> LOG.warn("Short read for entry size from entrylog {}", 
> entryLogId);
> return;
> }
> long offset = pos;
> pos += 4;
> sizeBuff.flip();
> int entrySize = sizeBuff.getInt();   < === 2, here we get entrySize 
> from buffer.
>
> sizeBuff.clear();
> // try to read ledger id first
> if (readFromLogChannel(entryLogId, bc, lidBuff, pos) != 
> lidBuff.capacity()) {
> LOG.warn("Short read for ledger id from entrylog {}", entryLogId);
> return;
> }
> lidBuff.flip();
> long lid = lidBuff.getLong();
> lidBuff.clear();
> if (lid == INVALID_LID || !scanner.accept(lid)) {
> // skip this entry
> pos += entrySize;
> continue;
> }
> // read the entry
> byte data[] = new byte[entrySize];   < === 1, here is line 1006, 
> seems it get a negative entrySize.
> ByteBuffer buff = ByteBuffer.wrap(data);
> int rc = readFromLogChannel(entryLogId, bc, buff, pos);
> if (rc != data.length) {
> LOG.warn("Short read for ledger entry from entryLog {}@{} ({} != 
> {})", new Object[] { entryLogId, pos,
> rc, data.length });
> return;
> }
> buff.flip();
> // process the entry
> scanner.process(lid, offset, buff);
> // Advance position to the next entry
> pos += entrySize;
> }
> }
>
>
>
>
>
> On Tue, Sep 19, 2017 at 8:42 PM, Enrico Olivelli <eolive...@gmail.com>
> wrote:
>
>> Hi
>> in one of my staging sites I got this error and the bookie cannot reclaim
>> disk space
>>
>> I am running a pure 4.5 BookKeeper bookie
>> below is the error
>>
>> Surely it is a broken EntryLog file.
>> Do you think the bookie could be recoverable ?
>> I did a backup, it does not contain sensitive data, I could share it
>>
>> -- Enrico
>>
>> 17-09-19-14-37-22Unexpected throwable caught
>> 17-09-19-14-37-22java.lang.NegativeArraySizeException
>> java.lang.NegativeArraySizeException
>> at
>> org.apache.bookkeeper.bookie.EntryLogger.scanEntryLog(EntryLogger.java:1006)
>> at
>> org.apache.bookkeeper.bookie.EntryLogger.extractEntryLogMetadataByScanning(EntryLogger.java:1118)
>> at
>> org.apache.bookkeeper.bookie.EntryLogger.getEntryLogMetadata(EntryLogger.java:1031)
>> at
>> org.apache.bookkeeper.bookie.GarbageCollectorThread.extractMetaFromEntryLogs(GarbageCollectorThread.java:569)
>> at
>> org.apache.bookkeeper.bookie.GarbageCollectorThread.safeRun(GarbageCollectorThread.java:342)
>> at org.apache.bookkeeper.util.SafeRunnable.run(SafeRunnable.java:33)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>
> --


-- Enrico Olivelli

Cannot compact bookie due to NegativeArraySizeException on BK 4.5

[Enrico Olivelli]

Hi
in one of my staging sites I got this error and the bookie cannot reclaim
disk space

I am running a pure 4.5 BookKeeper bookie
below is the error

Surely it is a broken EntryLog file.
Do you think the bookie could be recoverable ?
I did a backup, it does not contain sensitive data, I could share it

-- Enrico

17-09-19-14-37-22Unexpected throwable caught
17-09-19-14-37-22java.lang.NegativeArraySizeException
java.lang.NegativeArraySizeException
at
org.apache.bookkeeper.bookie.EntryLogger.scanEntryLog(EntryLogger.java:1006)
at
org.apache.bookkeeper.bookie.EntryLogger.extractEntryLogMetadataByScanning(EntryLogger.java:1118)
at
org.apache.bookkeeper.bookie.EntryLogger.getEntryLogMetadata(EntryLogger.java:1031)
at
org.apache.bookkeeper.bookie.GarbageCollectorThread.extractMetaFromEntryLogs(GarbageCollectorThread.java:569)
at
org.apache.bookkeeper.bookie.GarbageCollectorThread.safeRun(GarbageCollectorThread.java:342)
at org.apache.bookkeeper.util.SafeRunnable.run(SafeRunnable.java:33)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

HerdDB a distributed JVM-embeddable database built on Apache BookKeeper

[Enrico Olivelli]

Hi guys,
I am proud to share with you the first GA release of HerdDB (
http://www.herddb.org).

HerdDB is a distributed database, it uses Apache BookKeeper as distributed
write-ahead-log.

We are using it to replace MySQL, so it has JDBC bindings and it can be
accessed using usual SQL language and it supports multi-table, multi-row
transactions and indexes.
But it adds the ability to spread data on a cluster of machines, without
any shared media (thanks to BookKeeper and ZooKeeper).

HerdDB Server can work in standalone mode but we designed it to be embedded
in the JVM of clients and create clusters of peers which share a common
database.

You can find more details in this post or on the projects wiki on GitHub
http://eolivelli.blogspot.it/2017/09/herddb-distributed-jvm-embeddable.html

The code is open source and you can find it as usual on GitHub
https://github.com/diennea/herddb

Any contribution will be welcome !

Cheers

Enrico Olivelli

Re: Publish SNAPSHOT jar

[Enrico Olivelli]

On mer 23 ago 2017, 03:56 Jia Zhai <zhaiji...@gmail.com> wrote:

> 
>
> On Wed, Aug 23, 2017 at 9:02 AM, Sijie Guo <guosi...@gmail.com> wrote:
>
> > Hi all,
> >
> > I cloned the bookkeeper-release-nightly-snapshot CI job to deploy the
> > SNAPSHOT of the latest master to artifactory. So the developers can use
> > following artifact to access latest snapshot version and leverage the new
> > features coming from new release ASAP.
> >
> > https://builds.apache.org/job/distributedlog-release-nightly-snapshot/
> >
> > You can use
> >
> >
> >   org.apache.distributedlog
> > distributedlog-core
> > 0.5.0-SNAPSHOT
> >
>
Great!
An additional note

Remember to configure the repository section in your pom.xml to use
repository.apache.org for downloading snapshots

Snapshots are not published to Maven central

This ia the actual position
https://repository.apache.org/content/groups/snapshots/org/apache/distributedlog/distributedlog-core/

Enrico


>
> > -  Sijie
> >
>
-- 


-- Enrico Olivelli

Re: [ANNOUNCE] Apache BookKeeper 4.5.0 released

[Enrico Olivelli]

Congrats to everyone! Now we have a common baseline and a great future
together!

Enrico

Il ven 11 ago 2017, 20:03 Venkateswara Rao Jujjuri <jujj...@gmail.com> ha
scritto:

> Thanks a lot Sijie for patiently walking me through the process.
> Now I have real appreciation for the release mangers. ;)
>
> On Fri, Aug 11, 2017 at 10:59 AM, Sijie Guo <guosi...@gmail.com> wrote:
>
>> Awesome! Well done! This is a great milestone to the bookkeeper community!
>>
>> Thanks JV for doing the release! Thanks everyone for making this happen.
>>
>> Sijie
>>
>> On Aug 11, 2017 9:56 AM, "JV Jujjuri" <jujj...@apache.org> wrote:
>>
>> > The Apache BookKeeper team is proud to announce Apache BookKeeper
>> version
>> > 4.5.0.
>> >
>> > This is the fifth release of BookKeeper as an Apache Top Level Project!
>> >
>> > The 4.5.0 release incorporates hundreds of new fixes, improvements, and
>> > features since previous major release, 4.4.0, which was released over a
>> > year ago.
>> >
>> > It is a big milestone in Apache BookKeeper community, converging from
>> three
>> > main branches (Salesforce, Twitter and Yahoo).
>> >
>> > For BookKeeper release details and downloads, visit:
>> >
>> > http://bookkeeper.apache.org/releases/
>> >
>> > BookKeeper 4.5.0 Release Notes are at:
>> >
>> > http://bookkeeper.apache.org/docs/4.5.0/overview/releaseNotes/
>> >
>> > We would like to thank all the contributors that made the release
>> possible.
>> >
>> > Regards,
>> >
>> > The Apache BookKeeper Team
>> >
>>
>
>
>
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi
>
>
> --


-- Enrico Olivelli

Re: BookKeeper docker image is avaliable under apache now

[Enrico Olivelli]

Great!
Thank you all, expecially Jia and Francesco !

Enrico

Il ven 11 ago 2017, 10:48 Jia Zhai <zhaiji...@gmail.com> ha scritto:

> FYI. BookKeeper docker image is avaliable under apache now:
> https://hub.docker.com/r/apache/bookkeeper/
>
-- 


-- Enrico Olivelli

Re: BookKeeper#openLedgerNoRecovery hangs

[Enrico Olivelli]

Looking at this PR from Sijie I noticed that there is a rate limiter for
our internal subclass of ZooKeeper client.
https://github.com/apache/bookkeeper/pull/264

The rate limiter is not enabled and cannot be enabled.
I wonder if I hit a bug in our getData or ZkRetryRunnable or it is enough
to enable the rate limiter.

@Sijie
I left a comment on the PR, for me it is OK but it seems that it lacks
support for client-side BookKeeper, it enables it only on the Bookie

-- Enrico



2017-07-19 11:27 GMT+02:00 Enrico Olivelli <eolive...@gmail.com>:

>
>
> Il mer 19 lug 2017, 11:11 Sijie Guo <guosi...@gmail.com> ha scritto:
>
>> On Wed, Jul 19, 2017 at 4:04 PM, Enrico Olivelli <eolive...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> in some internal benchmarks we are experiencing openLedgerNoRecovery
>>> calls which remain hung.
>>> I see that basically that function calls ZookKeeper#getData.
>>>
>>
>>> Does anyone have an idea of how it can happen ?
>>>
>>
>> What version are you testing? Is it related your recent change on bumping
>> zookeeper version? If that's the case, we should consider rolling back the
>> zookeeper version.
>>
>
> 3.5.1 and 3.5.3
>
>>
>>
>>>
>>> Is there any implicit timeout on ZK.getData() ? I did not find any way
>>> and personally I never got into this problem.
>>>
>>
>> As far as I know, there is no timeout on zookeeper requests. It would be
>> a good question to zookeeper community.
>>
>
> I will do
>
>>
>>
>>>
>>> Maybe there is space for an improvement to add a timeout on
>>> openLedgerXXX operations, but anyway it is strange that the callback is
>>> never called.
>>>
>>> Unfortunately the problem happens only in integration tests, mabye I can
>>> work to reproduce it on a BK only test case.
>>>
>>> The case is simple: start ZK + 1 Bookie + 1 BookKeeper, create
>>> concurrencly many ledgers, write and concurrently open them with
>>> openLedgerNoRecovery from other threads.
>>> The fact is that no error is on ZK logs and BK logs
>>>
>>
>> Can you turn on debugging log for the bookkeeper client and also
>> zookeeper? There might be logs for checking.
>>
>
> Yes I am koggong at info, I will try at debug
>
>>
>> Another solution is to do a TCP dump for tracing the zookeeper calls to
>> see if the getData request and response is received at both sides.
>>
>>
>>>
>>> Any suggestion ?
>>>
>>
>
> Thank you again
> Enrico
>
>>
>>> Thanks
>>>
>>> -- Enrico
>>>
>>>
>>> --
>
>
> -- Enrico Olivelli
>

Re: InterleavedLedgerStorage vs SortedLedgerStorage

[Enrico Olivelli]

2017-07-19 11:58 GMT+02:00 Sijie Guo <guosi...@gmail.com>:

>
>
> On Wed, Jul 19, 2017 at 5:23 PM, Enrico Olivelli <eolive...@gmail.com>
> wrote:
>
>> Can you please  tell me  major benefits of Interleaved over Sorted? The
>> default is sorted
>> I can dig in code anyway
>>
>
> Sorted is in general better than Interleaved - not relying on filesystem
> page cache for caching, better I/O isolation.
>
> But if you only have a few ledgers or you rarely read from the ledgers,
> the memtables in Sorted might bring in overhead for you.
>
>

Thank you
I will do benchmarks with my workload, but I think that Sorted will be my
option

Enrico


> - Sijie
>
>
>> Enrico
>>
>> Il mer 19 lug 2017, 11:12 Sijie Guo <guosi...@gmail.com> ha scritto:
>>
>>> Currently we don't have such documentation about these two ledger
>>> storages. We are working on improving the documentation as part of BP-11.
>>>
>>> - Sijie
>>>
>>> On Wed, Jul 19, 2017 at 3:34 PM, Enrico Olivelli <eolive...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> is there any doc about the differences and usecases for
>>>> InterleavedLedgerStorage vs SortedLedgerStorage ?
>>>>
>>>>
>>>> --- Enrico
>>>>
>>>
>>> --
>>
>>
>> -- Enrico Olivelli
>>
>
>

Re: BookKeeper#openLedgerNoRecovery hangs

[Enrico Olivelli]

Il mer 19 lug 2017, 11:11 Sijie Guo <guosi...@gmail.com> ha scritto:

> On Wed, Jul 19, 2017 at 4:04 PM, Enrico Olivelli <eolive...@gmail.com>
> wrote:
>
>> Hi,
>> in some internal benchmarks we are experiencing openLedgerNoRecovery
>> calls which remain hung.
>> I see that basically that function calls ZookKeeper#getData.
>>
>
>> Does anyone have an idea of how it can happen ?
>>
>
> What version are you testing? Is it related your recent change on bumping
> zookeeper version? If that's the case, we should consider rolling back the
> zookeeper version.
>

3.5.1 and 3.5.3

>
>
>>
>> Is there any implicit timeout on ZK.getData() ? I did not find any way
>> and personally I never got into this problem.
>>
>
> As far as I know, there is no timeout on zookeeper requests. It would be a
> good question to zookeeper community.
>

I will do

>
>
>>
>> Maybe there is space for an improvement to add a timeout on openLedgerXXX
>> operations, but anyway it is strange that the callback is never called.
>>
>> Unfortunately the problem happens only in integration tests, mabye I can
>> work to reproduce it on a BK only test case.
>>
>> The case is simple: start ZK + 1 Bookie + 1 BookKeeper, create
>> concurrencly many ledgers, write and concurrently open them with
>> openLedgerNoRecovery from other threads.
>> The fact is that no error is on ZK logs and BK logs
>>
>
> Can you turn on debugging log for the bookkeeper client and also
> zookeeper? There might be logs for checking.
>

Yes I am koggong at info, I will try at debug

>
> Another solution is to do a TCP dump for tracing the zookeeper calls to
> see if the getData request and response is received at both sides.
>
>
>>
>> Any suggestion ?
>>
>

Thank you again
Enrico

>
>> Thanks
>>
>> -- Enrico
>>
>>
>> --


-- Enrico Olivelli

Re: InterleavedLedgerStorage vs SortedLedgerStorage

[Enrico Olivelli]

Can you please  tell me  major benefits of Interleaved over Sorted? The
default is sorted
I can dig in code anyway
Enrico

Il mer 19 lug 2017, 11:12 Sijie Guo <guosi...@gmail.com> ha scritto:

> Currently we don't have such documentation about these two ledger
> storages. We are working on improving the documentation as part of BP-11.
>
> - Sijie
>
> On Wed, Jul 19, 2017 at 3:34 PM, Enrico Olivelli <eolive...@gmail.com>
> wrote:
>
>> Hi,
>> is there any doc about the differences and usecases for
>> InterleavedLedgerStorage vs SortedLedgerStorage ?
>>
>>
>> --- Enrico
>>
>
> --


-- Enrico Olivelli

BookKeeper#openLedgerNoRecovery hangs

[Enrico Olivelli]

Hi,
in some internal benchmarks we are experiencing openLedgerNoRecovery calls
which remain hung.
I see that basically that function calls ZookKeeper#getData.

Does anyone have an idea of how it can happen ?

Is there any implicit timeout on ZK.getData() ? I did not find any way and
personally I never got into this problem.

Maybe there is space for an improvement to add a timeout on openLedgerXXX
operations, but anyway it is strange that the callback is never called.

Unfortunately the problem happens only in integration tests, mabye I can
work to reproduce it on a BK only test case.

The case is simple: start ZK + 1 Bookie + 1 BookKeeper, create concurrencly
many ledgers, write and concurrently open them with openLedgerNoRecovery
from other threads.
The fact is that no error is on ZK logs and BK logs

Any suggestion ?

Thanks

-- Enrico

Re: [ANNOUNCE] JV Jujjuri joins the Apache BookKeeper PMC

[Enrico Olivelli]

Congrats JV !

Il gio 6 lug 2017, 18:24 Sijie Guo <guosi...@gmail.com> ha scritto:

> On behalf of the Apache BookKeeper PMC I am pleased to announce that JV
> Jujjuri has accepted our invitation to become a PMC member on the Apache
> BookKeeper project.
>
> JV has been an active contributor in many areas, including driving
> community meetings, reviewing patches, helping with triaging bugs and
> organizing the bookkeeper meetups. He has the ability and passion to move
> the development of bookkeeper forward and has been speaking of Apache
> BookKeeper in different conferences and meetups.
>
> Please join me in thanking JV for his contributions to date and
> anticipation of many more contributions.
>
> Welcome to the PMC, JV!
>
> - Sijie
>
-- 


-- Enrico Olivelli

Re: BookKeeper and CheckStyle

[Enrico Olivelli]

David,,
I see you have created an issue for the first part of bookkeeper-server.
Maybe you can also create issues for every package, this way anyone can
start working and we can easily coordinate such great effort.

We are currently trying to switch to github issue tracker, this will a good
opportunity to start using this new tool.

Thank you John, feel free to pick up the issues, all help is really welcome
!

Enrico

Il mar 4 lug 2017, 21:47 John Lonergan <john.loner...@gmail.com> ha scritto:

> Re"It contains almost 5000 issues."
>
> Have you put the gist somewhere for information.
> Folk like myself are probably happy to contribute some time. It's an easy
> way to contribute something to the community.
>
> JL
>
> On 4 Jul 2017 5:17 pm, "Enrico Olivelli" <eolive...@gmail.com> wrote:
>
>>
>>
>> Il mar 4 lug 2017, 18:06 Dávid Szigecsán <sige...@gmail.com> ha scritto:
>>
>>> Hi,
>>>
>>> Yes, I created a PR for the first part of the change. (All modules except
>>> bookkeeper-server)
>>> I started to do the second part (bookkeeper-server), but It is a huge
>>> change. It contains almost 5000 issues.
>>> I'm thinking about how to slice it up to small steps.
>>>
>>
>> Thank you David,
>> Yep, the idea is to create a single patch per package if possible
>>
>> Enrico
>>
>>
>>> 2017-07-04 17:06 GMT+02:00 Sijie Guo <guosi...@gmail.com>:
>>>
>>> > Those modules are fine, they are rarely touched any way.
>>> >
>>> > On Jul 4, 2017 8:57 AM, "Enrico Olivelli" <eolive...@gmail.com> wrote:
>>> >
>>> > > 2017-07-04 16:50 GMT+02:00 Sijie Guo <guosi...@gmail.com>:
>>> > > > It is fine to me if we do modules by modules and packages by
>>> packages
>>> > in
>>> > > > bookkeeper-server. We can keep the changes smaller for reviews and
>>> > easier
>>> > > > to merge.
>>> > >
>>> > > I see in the issue and PR
>>> > > https://github.com/apache/bookkeeper/pull/231 that he is adding CS
>>> to
>>> > > every maven module except from bookkeeper-server
>>> > > maybe it is a good starting point.
>>> > > I have written a comment in order to invite him to join the list
>>> > >
>>> > > I am also OK with applying such changes to bookkeeper-server one
>>> > > package at a time
>>> > >
>>> > > -- Enrico
>>> > >
>>> > > >
>>> > > > Also, it might be good to also discuss on the issue to keep David
>>> > updated
>>> > > > if he is not in the dev@ list.
>>> > > >
>>> > > > Sijie
>>> > > >
>>> > > > On Jul 4, 2017 6:43 AM, "Enrico Olivelli" <eolive...@gmail.com>
>>> wrote:
>>> > > >
>>> > > > Hi all,
>>> > > > as you can see from github emails there is an ongoing proposal to
>>> add
>>> > > > "checkstyle" plugin to BookKeeper build.
>>> > > > I am really in favour of this change. It is already used in
>>> > > > DistributedLog and it will ease the review, preventing us from
>>> writing
>>> > > > comments for minor typos.
>>> > > >
>>> > > > https://github.com/apache/bookkeeper/issues/230
>>> > > > https://github.com/apache/bookkeeper/issues/230
>>> > > >
>>> > > > Thanks to David (I hope he is subscribed to this list) we will be
>>> able
>>> > > > to add this kind of support soon.
>>> > > >
>>> > > > My concern is that this change will make us change all big pull
>>> > requests.
>>> > > >
>>> > > > We should decide when to get checkstyle in:
>>> > > > 1) as soon as possible (after review of the patch)
>>> > > > 2) before 4.5 release, as last step
>>> > > > 3) after merging biggest changes (Twitter changes and Salesforce
>>> > > > changes) which are waiting for review/merge
>>> > > > 4) defer to the start of 4.6
>>> > > >
>>> > > > My proposal is to defer to the start of 4.6, the only problem is
>>> that
>>> > > > David will be doing a big effort to keep the patch in synch with
>>> the
>>> > > > actual master
>>> > > >
>>> > > > -- Enrico
>>> > >
>>> >
>>>
>> --
>>
>>
>> -- Enrico Olivelli
>>
> --


-- Enrico Olivelli

Re: BookKeeper and CheckStyle

[Enrico Olivelli]

Il mar 4 lug 2017, 18:06 Dávid Szigecsán <sige...@gmail.com> ha scritto:

> Hi,
>
> Yes, I created a PR for the first part of the change. (All modules except
> bookkeeper-server)
> I started to do the second part (bookkeeper-server), but It is a huge
> change. It contains almost 5000 issues.
> I'm thinking about how to slice it up to small steps.
>

Thank you David,
Yep, the idea is to create a single patch per package if possible

Enrico


> 2017-07-04 17:06 GMT+02:00 Sijie Guo <guosi...@gmail.com>:
>
> > Those modules are fine, they are rarely touched any way.
> >
> > On Jul 4, 2017 8:57 AM, "Enrico Olivelli" <eolive...@gmail.com> wrote:
> >
> > > 2017-07-04 16:50 GMT+02:00 Sijie Guo <guosi...@gmail.com>:
> > > > It is fine to me if we do modules by modules and packages by packages
> > in
> > > > bookkeeper-server. We can keep the changes smaller for reviews and
> > easier
> > > > to merge.
> > >
> > > I see in the issue and PR
> > > https://github.com/apache/bookkeeper/pull/231 that he is adding CS to
> > > every maven module except from bookkeeper-server
> > > maybe it is a good starting point.
> > > I have written a comment in order to invite him to join the list
> > >
> > > I am also OK with applying such changes to bookkeeper-server one
> > > package at a time
> > >
> > > -- Enrico
> > >
> > > >
> > > > Also, it might be good to also discuss on the issue to keep David
> > updated
> > > > if he is not in the dev@ list.
> > > >
> > > > Sijie
> > > >
> > > > On Jul 4, 2017 6:43 AM, "Enrico Olivelli" <eolive...@gmail.com>
> wrote:
> > > >
> > > > Hi all,
> > > > as you can see from github emails there is an ongoing proposal to add
> > > > "checkstyle" plugin to BookKeeper build.
> > > > I am really in favour of this change. It is already used in
> > > > DistributedLog and it will ease the review, preventing us from
> writing
> > > > comments for minor typos.
> > > >
> > > > https://github.com/apache/bookkeeper/issues/230
> > > > https://github.com/apache/bookkeeper/issues/230
> > > >
> > > > Thanks to David (I hope he is subscribed to this list) we will be
> able
> > > > to add this kind of support soon.
> > > >
> > > > My concern is that this change will make us change all big pull
> > requests.
> > > >
> > > > We should decide when to get checkstyle in:
> > > > 1) as soon as possible (after review of the patch)
> > > > 2) before 4.5 release, as last step
> > > > 3) after merging biggest changes (Twitter changes and Salesforce
> > > > changes) which are waiting for review/merge
> > > > 4) defer to the start of 4.6
> > > >
> > > > My proposal is to defer to the start of 4.6, the only problem is that
> > > > David will be doing a big effort to keep the patch in synch with the
> > > > actual master
> > > >
> > > > -- Enrico
> > >
> >
>
-- 


-- Enrico Olivelli

Visual BookKeeper Admin tool

[Enrico Olivelli]

Hi,
I am looking for some kind of user-friendly admin tool to have a high
level overview of a bookkeeper cluster.

Some features:
- inspect list of bookies in cluster, avaiables, status
- inspect ledgers, placement, replication status

I am thinking abount some kind of web user interface

Does anyone have such tool ?

Cheers

-- Enrico

BOOKKEEPER-867 New Client API to allow applications pass-in EntryId.

[Enrico Olivelli]

Hi,
I am trying to understand this API
https://issues.apache.org/jira/browse/BOOKKEEPER-867

In particular I would like to know which are the constraints on the
sequence of entry ids.

I am going to use one LedgerHandle from multiple threads and handle
locally the sequence of ids in order to "book" blocks of entry ids.
I am using a ledger in order to store blobs (binary large object) and
I need to split a blob into multiple entries and I would like all the
entries of a single blob to spawn a contiguous range or entry ids (so
that I can store only the first and the last id and not all the chain)

Example:
Thread-1 starts writing entries from 5 to 10
While Thread-1 already sent entries 5 and 6 Thread-2 starts to add
entries 20 to 30.


Thanks

-- Enrico

Re: Write guarantees and the LAC Protocol

[Enrico Olivelli]

Starting from top as I found a good explanation for my "failing" flaky test
case.
I'm asking to the user list some confirm of my explanation

Usecase:
1) 1 single writer, 1 bookie
2) The writer creates a new ledger and writes 1 entry and waits for the
acknowledge (addEntry, blocking call), let's call it "entry A"
3) The writer issues some other asynch writes (say B,C,D) and asserts that
at least one ends with BKNotEnoughBookiesException)
4) Stop the bookie (this will cause BKNotEnoughBookiesException to some of
the writes)
5) Start the bookie (boots OK, no errors)
6) Recover the ledger (open a new client with openLedger, WITH recovery),
use readLastAddConfirmed to be sure to read from all the bookies the max
LastAddConfirmed
7) The entry A some times is present (case a), sometimes is not present
(case b)

case A: success for entry A
at least ONE of the writes completed with success before the stop of the
Bookie, so the LastAddConfirmed has been written to the bookie and so it
can be recovered

case B: entry A disappeared
NONE of the writes completed with success before the stop of the Bookie, so
the LastAddConfirmed has NEVER been written to the bookie and so the ledger
appears EMPTY

Is this a good explanation ?


Thank you
Enrico





2017-04-04 10:05 GMT+02:00 Enrico Olivelli <eolive...@gmail.com>:

>
>
> 2017-04-04 9:47 GMT+02:00 Enrico Olivelli <eolive...@gmail.com>:
>
>> I have completed by debug session and I have an answer.
>>
>> The problematic flow is the following:
>> - LedgerHandler#asyncAddEntry
>> - in case of BKNotEnoughBookiesException I'm triggering an async callback
>> (with a CompletableFuture#handleAsync) which in turn calls
>> LedgerHandler#close
>> - the LedgerHandler#close method truncates the ledger and some entry is
>> lost (at the time of the close call I have local LastAddConfirmed ==
>> LastAddPushed == 0 on the writer)
>>
>> Is this a possible explanation ?
>> I wonder which is the best way to react to an error (in this case
>> BKNotEnoughBookiesException) in asyncAddEntry, usually I'm calling
>> LedgerHandler#close in order to the necessary clean up, but in this case it
>> seems very dangerous
>> (I am running BK 4.4.0)
>>
>> This is the issue on my project, where I have put logs
>> https://github.com/diennea/herddb/issues/100
>>
>> Thanks
>> -- Enrico
>>
>>
>>
>
> I'm sorry, during other test cases skipping the close() did not fix the
> issue consistently, I will debug more
>
>
>>
>>
>>
>>
>>
>>
>> 2017-03-17 16:14 GMT+01:00 Enrico Olivelli <eolive...@gmail.com>:
>>
>>>
>>>
>>> 2017-03-17 7:48 GMT+01:00 Sijie Guo <guosi...@gmail.com>:
>>>
>>>>
>>>>
>>>> On Wed, Mar 15, 2017 at 9:17 AM, Enrico Olivelli <eolive...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi BookKeepers,
>>>>> I'm trying to understand some basic aspect of BookKeeper
>>>>>
>>>>> I have a strange case, and I would like to have some opinion from any
>>>>> "expert".
>>>>>
>>>>> I am using BookKeeper as a transaction log for a database, this is an
>>>>> example scenario:
>>>>>
>>>>> 1) the user inserts a record in a table
>>>>> the "fact" is logged to the Write-Ahead-log (Bookkeeper ledger), and
>>>>> the systems blocks until the asyncAddEntry completes (the callback is
>>>>> called with rc=0)
>>>>>
>>>>> now I (the 'writer') assume that the entry is written to durable
>>>>> storage and has been synched to disk
>>>>> then the DB applies the change to the local memory as every
>>>>> application with a WAL does
>>>>>
>>>>> 2) The writer 2  other entries for which the asyncAddEntry completes
>>>>> (the callback is called with rc=0). I block until the callback as been
>>>>> called
>>>>>
>>>>> 2) for tests I'm using an ensemble of 1 bookie, I "stop" the only
>>>>> bookie
>>>>> the bookie stops "gracefully" (no strange errors)
>>>>>
>>>>> 3) The writer issues other asyncAddEntry, which fail due to
>>>>> "notEnoughBookiesException" and then "closes" the LedgerHandle, inside the
>>>>> "callback" cade
>>>>>
>>>>> 4) I restart the bookie (no strange errors on logs)
>>>>>
>>>>> 5) I restart

Re: Write guarantees and the LAC Protocol

[Enrico Olivelli]

2017-04-04 9:47 GMT+02:00 Enrico Olivelli <eolive...@gmail.com>:

> I have completed by debug session and I have an answer.
>
> The problematic flow is the following:
> - LedgerHandler#asyncAddEntry
> - in case of BKNotEnoughBookiesException I'm triggering an async callback
> (with a CompletableFuture#handleAsync) which in turn calls
> LedgerHandler#close
> - the LedgerHandler#close method truncates the ledger and some entry is
> lost (at the time of the close call I have local LastAddConfirmed ==
> LastAddPushed == 0 on the writer)
>
> Is this a possible explanation ?
> I wonder which is the best way to react to an error (in this case
> BKNotEnoughBookiesException) in asyncAddEntry, usually I'm calling
> LedgerHandler#close in order to the necessary clean up, but in this case it
> seems very dangerous
> (I am running BK 4.4.0)
>
> This is the issue on my project, where I have put logs
> https://github.com/diennea/herddb/issues/100
>
> Thanks
> -- Enrico
>
>
>

I'm sorry, during other test cases skipping the close() did not fix the
issue consistently, I will debug more


>
>
>
>
>
>
> 2017-03-17 16:14 GMT+01:00 Enrico Olivelli <eolive...@gmail.com>:
>
>>
>>
>> 2017-03-17 7:48 GMT+01:00 Sijie Guo <guosi...@gmail.com>:
>>
>>>
>>>
>>> On Wed, Mar 15, 2017 at 9:17 AM, Enrico Olivelli <eolive...@gmail.com>
>>> wrote:
>>>
>>>> Hi BookKeepers,
>>>> I'm trying to understand some basic aspect of BookKeeper
>>>>
>>>> I have a strange case, and I would like to have some opinion from any
>>>> "expert".
>>>>
>>>> I am using BookKeeper as a transaction log for a database, this is an
>>>> example scenario:
>>>>
>>>> 1) the user inserts a record in a table
>>>> the "fact" is logged to the Write-Ahead-log (Bookkeeper ledger), and
>>>> the systems blocks until the asyncAddEntry completes (the callback is
>>>> called with rc=0)
>>>>
>>>> now I (the 'writer') assume that the entry is written to durable
>>>> storage and has been synched to disk
>>>> then the DB applies the change to the local memory as every application
>>>> with a WAL does
>>>>
>>>> 2) The writer 2  other entries for which the asyncAddEntry completes
>>>> (the callback is called with rc=0). I block until the callback as been
>>>> called
>>>>
>>>> 2) for tests I'm using an ensemble of 1 bookie, I "stop" the only bookie
>>>> the bookie stops "gracefully" (no strange errors)
>>>>
>>>> 3) The writer issues other asyncAddEntry, which fail due to
>>>> "notEnoughBookiesException" and then "closes" the LedgerHandle, inside the
>>>> "callback" cade
>>>>
>>>> 4) I restart the bookie (no strange errors on logs)
>>>>
>>>> 5) I restart my "writer", opening the ledger with "openLedger" (with
>>>> recovery and fencing...) in order to perform recovery from the WAL
>>>>
>>>> 6) now the LastAddConfirmed is 0 ! so my first entry is lost !
>>>>
>>>> Am I missing something ?
>>>> The LAC has not been advanced even with 3 successful writes to the
>>>> ledger
>>>>
>>>
>>> Are you using readLastAddConfirmed at step 5)? 0 is expected when you
>>> using #readLastAddConfirmed, since the call is retrieving LAC from bookies.
>>>
>>>
>> I have added a call to readLastAddConfirmed but it does not help, the LAC
>> is still 0
>>
>>
>> I have double checked my code and the "close" happens in another thread,
>> maybe I am hitting a race condition in my code, in fact the problem is not
>> reproducible at 100%
>>
>>
>> I will continue debugging
>>
>> Thanks
>>
>>
>>
>>
>>> At step 3), it already closed the ledger and update the metadata's last
>>> entry id with 2. so for tailing read, you might need to call
>>> #readLastAddConfirmed and then get the last add confirmed using
>>> #getLastAddConfirmed.
>>>
>>>
>>>
>>>>
>>>> From the "writer" point-of-view the fact that the callback as been
>>>> called is not a guarantee that the entry as been written to durable storage
>>>> ?
>>>>
>>>> I understand that of a "tailing" reader (a ledger opened with
>>>> openLedgerNoRecovery) it is very important not to be able to "read" data
>>>> before the LAC, but for the "writer" it is important that it can count on
>>>> the fact the entry will be recovered with success
>>>>
>>>>
>>>> Thanks
>>>> -- Enrico
>>>>
>>>>
>>>
>>
>