Re: [VOTE] Release ZooKeeper 3.3.3 (candidate 1)

[Ivan Kelly]

+1

On 25 Feb 2011, at 09:25, Patrick Hunt wrote:

> +1, looks good to me. Signing is correct, versions seem right, and the
> tests all pass on my box.
> 
> Patrick
> 
> On Wed, Feb 23, 2011 at 7:49 PM, Benjamin Reed  wrote:
>> i've created yet another candidate build for ZooKeeper 3.3.3. this is a bug
>> fix
>> release addressing 14 issues (two of them extremely critical) -- see the
>> release notes for details.
>> 
>> *** Please download, test and VOTE before the
>> *** vote closes 11pm pacific time, Saturday, February 26.***
>> 
>> http://people.apache.org/~breed/zookeeper-3.3.3-candidate-1/
>> 
>> one thing that has not been fixed in this release is that the docs still
>> reference hadoop. this will be fixed in a future release.
>> 
>> should we release this?
>> 
>> ben
>> 

Re: [DISCUSS] move hedwig/bookkeeper to subproject

[Ivan Kelly]

I agree about the separation of bookkeeper and hedwig. They solve very 
different problems, so lumping them together feels clunky. Perhaps bookkeeper 
could be moved out of zookeeper first, leaving hedwig in until there's more 
community interest in it.

-Ivan

On 15 Mar 2011, at 23:58, Dhruba Borthakur wrote:

> I am interested in contributing to the bookkeeper code. It would be nice to
> have a community around it. An incubator proposal sounds good, but the
> zk-subproject should also work well. It woud be nice to separate out hedwig
> and bookkeeper since they have quite different functionality.
> 
> -dhruba
> 
> On Mon, Mar 14, 2011 at 8:56 PM, Mahadev Konar  wrote:
> 
>> I like the idea of BookKeeper/Hedwig being subprojects.
>> 
>> thanks
>> mahadev
>> 
>> On Thu, Mar 10, 2011 at 1:34 AM, Patrick Hunt  wrote:
>> 
>>> 
>>> 
>>> On Tue, Mar 8, 2011 at 1:01 PM, Flavio Junqueira >> wrote:
>>> 
 I'm sorry for not replying before. I didn't feel that the message was
>> for
 me, since it should be pretty obvious that I'm interested in those
 projects. Here are some thoughts, though:
 
 - It would be really nice to have committers for bookkeeper/hedwig;
 - It would be really nice to have independent releases for
 bookkeeper/hedwig;
 - It sounds like bookkeeper and hedwig don't always go together, and
>> hdfs
 is an instance in which it happens. But, hedwig builds on top of
>> bookkeeper
 (and other components), so using hedwig implies using bookkeeper.
 Consequently, if we choose only one to be a main project, perhaps
>> bookkeeper
 would be a better choice;
 
>>> 
>>> Perhaps one could argue that bk/hedwig fall under "distributed system
>>> coordination" and therefore should be part of ZK? Or is that too much of
>> a
>>> stretch? ;-)
>>> 
>>> RESOLVED, that the Apache ZooKeeper Project be and hereby is responsible
>>> for the creation and maintenance of software related to distributed
>> system
>>> coordination; and be it further
>>> 
>>> 
 - I don't think we have anyone who could be a project lead for these
 projects right now, so it could be a problem to split up at this point.
>> For
 this reason, a zookeeper subproject sounds like a better option compared
>> to
 incubator, unless we are able to find a project lead.
 
 -Flavio
 
 On Mar 2, 2011, at 6:55 PM, Benjamin Reed wrote:
 
 i wanted to start a discussion about making hedwig and bookkeeper a
 subproject. (actually pat started the discussion last month in general
 about all of the contrib projects.) there are three questions, in my
 mind, that we need to answer to move forward:
 
 1) should it be a hedwig/bookkeeper subproject, or should there be two
 separate projects? we need to build a developer community and i'm
 wondering if we should try to build a single dev community or two. the
 relationship is a bit asymmetrical: hedwig depends on bookkeeper, but
 not visa-versa. i'm inclined to say we do a hedwig subproject and
 include bookkeeper with it, but i don't feel strongly.
 
 2) should we propose a subproject to zookeeper or to incubator? i'm a
 bit more inclined to propose a zookeeper subproject simply because it
 fits well with the zookeeper community, but it does introduce a bit
 more overhead to the zookeeper PMC.
 
 3) do we have the developer interest to make it happen in the first
 place? i know we can get at least 3 initial committers from yahoo!,
 but projects should be represented by multiple companies. (the goal is
 at least 3.) so, is there interest in working on the project from
 others?
 
 please comment. these are all open issues, so opinions are what i'm
 looking for. if there isn't much discussion, i think that will
 implicitly answer 3 :)
 
 thanx
 ben
 
 
  *flavio*
 *junqueira*
 
 research scientist
 
 f...@yahoo-inc.com
 direct +34 93-183-8828
 
 avinguda diagonal 177, 8th floor, barcelona, 08018, es
 phone (408) 349 3300fax (408) 349 3301
 
 
 
>>> 
>> 
> 
> 
> 
> -- 
> Connect to me at http://www.facebook.com/dhruba

Re: [VOTE] BookKeeper (including Hedwig) subproject proposal

[Ivan Kelly]

+1

On 18 Mar 2011, at 22:40, Utkarsh Srivastava wrote:

> +1
> 
> Utkarsh
> 
> On Fri, Mar 18, 2011 at 2:29 PM, Mahadev Konar  wrote:
>> +1.
>> 
>> thanks
>> mahadev
>> On Fri, Mar 18, 2011 at 2:26 PM, Patrick Hunt  wrote:
>> 
>>> -0. I'm all for bk/hedwig moving out from contrib, but as I stated earlier
>>> I think it should move to incubator and not subproject. At the same time
>>> it's important that the project can develop on it's own, so I won't stand in
>>> the way.
>>> 
>>> Patrick
>>> 
>>> 
>>> On Fri, Mar 18, 2011 at 2:18 PM, Flavio Junqueira wrote:
>>> 
>>>> +1.
>>>> 
>>>> -Flavio
>>>> 
>>>> On Mar 18, 2011, at 10:11 PM, Benjamin Reed wrote:
>>>> 
>>>> +1 i'm all for it of course :)
>>>> 
>>>> On Fri, Mar 18, 2011 at 2:11 PM, Benjamin Reed  wrote:
>>>> 
>>>> Proposal
>>>> 
>>>> 
>>>> BookKeeper is a distributed write ahead logging (WAL) service. It is
>>>> 
>>>> built on top of ZooKeeper and is used for distributed recovery and
>>>> 
>>>> reliability. Much like ZooKeeper itself, BookKeeper is a distributed
>>>> 
>>>> tool used for reliability, but unlike ZooKeeper it is used to store
>>>> 
>>>> large amounts of application data in the form of byte streams, which
>>>> 
>>>> we call ledgers. It is made up of Bookies, which store data, and a
>>>> 
>>>> client library. All other meta-data is stored in ZooKeeper.
>>>> 
>>>> 
>>>> The BookKeeper subproject also includes Hedwig, which is a pub/sub
>>>> 
>>>> system built on both BookKeeper and ZooKeeper. It's coupling with
>>>> 
>>>> BookKeeper is tight and many of the performance features of BookKeeper
>>>> 
>>>> were added in response to Hedwig's requirements. Hedwig is made up of
>>>> 
>>>> a rather thin client library and stateless Brokers that cache and
>>>> 
>>>> distribute messages.
>>>> 
>>>> 
>>>> Background
>>>> 
>>>> 
>>>> BookKeeper was developed as a WAL for the Hadoop NameNode and was also
>>>> 
>>>> used to build the Hedwig pub/sub system. Both are currently contribs
>>>> 
>>>> to ZooKeeper. The work to get the hooks necessary to integrate
>>>> 
>>>> BookKeeper with the NameNode is almost complete (HDFS-1580).
>>>> 
>>>> 
>>>> Rational
>>>> 
>>>> 
>>>> We have contributors that we would like to make committers to
>>>> 
>>>> BookKeeper and Hedwig. It would be nice to allow a development
>>>> 
>>>> community to grow around BookKeeper.
>>>> 
>>>> 
>>>> Also, hudson does not run against contrib. Making BookKeeper its own
>>>> 
>>>> subproject would allow us to better qa our changes.
>>>> 
>>>> 
>>>> We also would like to decouple BookKeeper releases from ZooKeeper
>>>> 
>>>> releases. ZooKeeper is quite mature and has relatively long release
>>>> 
>>>> cycles. We would like shorter release cycles for BookKeeper.
>>>> 
>>>> 
>>>> In theory we could make two projects BookKeeper and Hedwig, but doing
>>>> 
>>>> so would double the project management and release overhead. The
>>>> 
>>>> development community between BookKeeper and Hedwig overlaps heavily,
>>>> 
>>>> so we would be increasing the burden on the same group of
>>>> 
>>>> contributors.
>>>> 
>>>> 
>>>> Because of the developer community overlap with ZooKeeper and the fact
>>>> 
>>>> that BookKeeper is inline with the general mission of ZooKeeper, we
>>>> 
>>>> think BookKeeper should be a subproject of ZooKeeper.
>>>> 
>>>> 
>>>> Call for vote
>>>> 
>>>> 
>>>> I propose that BookKeeper become a ZooKeeper subproject subject to
>>>> 
>>>> ZooKeeper PMC and Bylaws. I, Benjamin Reed, will champion the
>>>> 
>>>> proposal. BookKeeper will have the following initial committers:
>>>> 
>>>> 
>>>> Dhruba Borthakur (Facebook)
>>>> 
>>>> Flavio Junqueira (Yahoo)
>>>> 
>>>> Ivan Kelly (Yahoo)
>>>> 
>>>> Benjamin Reed (Yahoo)
>>>> 
>>>> Utkarsh Srivastava (Twitter)
>>>> 
>>>> 
>>>> 
>>>>   *flavio*
>>>> *junqueira*
>>>> 
>>>> research scientist
>>>> 
>>>> f...@yahoo-inc.com
>>>> direct +34 93-183-8828
>>>> 
>>>> avinguda diagonal 177, 8th floor, barcelona, 08018, es
>>>> phone (408) 349 3300fax (408) 349 3301
>>>> 
>>>> 
>>>> 
>>> 
>> 

Issue with addEntry api

[Ivan Kelly]

I'm having an issue with the LedgerHandle#addEntry api.

[1] best illustrates it. I'm buffering namenode transactions in the stream and 
only transmitting when either flush is called or I have enough data to pass my 
threshold. This means I have a byte buffer in my class which I fill up as new 
transactions come in. When I transmit, I set this buffer as an entry to 
bookkeeper. I.e. N whole namenode transactions will be contained in 1 single bk 
entry. 

The problem is this byte buffer (DataOutputBuffer in this case). I reuse the 
same buffer over and over. But this buffer has a fixed size. If I transmit 
before it is full, the whole buffer size will be transmitted anyhow. If the 
buffer is being reused, this will retransmit old transactions out of order. For 
example, in the first use, the buffer fills with, [a,b,c,d,e] and adds this as 
an entry and resets the byte buffer. Then transaction f is  added and flushed, 
in this case [f,b,c,d,e] is not transmitted. 

What I need is the ability to set offset and length in the byte[] passed to 
addEntry. Is there a reason this wasn't added in the initial implementation? If 
not, and if you agree this is a valid usecase, ill open a JIRA and add this 
functionality. Im getting around this now by doing an extra Array.copyOf which 
is less than ideal.

-Ivan



[1] 
https://github.com/ivankelly/hadoop-common/blob/HDFS-1580-BK/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/bkjournal/BookKeeperEditLogOutputStream.java#L149

Re: 3.4 update.

[Ivan Kelly]

Is there any estimated date for the release of 3.4?

-Ivan

Re: Review of public APIs

[Ivan Kelly]

Yes, I agree on the 4.0 release thing. It will make it clear that there's been 
a big change.

I've created two JIRAs for the changes, BOOKKEEPER-89 for bookkeeper, 
BOOKKEEPER-90 for hedwig. Please have a look [1][2].

BOOKKEEPER-89 is complete.

For BOOKKEEPER-90, I still need to decide what to do about ByteString if we do 
anything at all. Having ByteString in the API tries us to google protobufs. Im 
not sure this is a huge issue, but it means that this API will never be able to 
move away from it.

-Ivan

[1] https://reviews.apache.org/r/2543/
[2] https://reviews.apache.org/r/2546/

On 22 Oct 2011, at 00:27, Benjamin Reed wrote:

> i also agee. we should also call it a 4.0 release.
> 
> ben
> 
> On Fri, Oct 21, 2011 at 2:17 AM, Ivan Kelly  wrote:
>> I agree with Utkarsh. Initial release is as good of an excuse we'll ever 
>> have for breaking stuff. The changes themselves shouldn't be too big though. 
>> Ill have a go later today.
>> 
>> -Ivan
>> 
>> On 19 Oct 2011, at 19:04, Utkarsh Srivastava wrote:
>> 
>>> I think it makes sense to delay the release if required for this
>>> cleanup. Such cleanup will get only harder after the release.
>>> 
>>> On Wed, Oct 19, 2011 at 9:42 AM, Flavio Junqueira  
>>> wrote:
>>>> Hi Ivan, Thanks for putting this list together. I don't have a good sense 
>>>> of
>>>> how many changes the modifications you're proposing would induce. My 
>>>> concern
>>>> is delaying the release further, even though I agree that cleaning up the
>>>> interfaces is important.
>>>> 
>>>> Also, I was thinking that some public calls are related to our package
>>>> structure. In particular, I'm referring to BookKeeperTools, which is in 
>>>> this
>>>> tools package and I've needed to make some calls public to be able to 
>>>> access
>>>> from there. It might be a good idea to review those as well.
>>>> 
>>>> -Flavio
>>>> 
>>>> On Oct 19, 2011, at 5:40 PM, Ivan Kelly wrote:
>>>> 
>>>>> I've just gone through the public apis for BK and HW and have found the
>>>>> following issues. Most of them are issues with protection being too loose.
>>>>> There's only 2-3 breaks here which should require code change on the 
>>>>> users'
>>>>> part. The rest are things that they really shouldn't be using in any case.
>>>>> 
>>>>> BookKeeper#createLedger, parameter is named passwd, "Key" used in
>>>>> LedgerHandle api
>>>>> BookKeeper#getBookieClient shouldn't be public
>>>>> BookKeeper#createComplete shouldn't be public
>>>>> BookKeeper#openComplete shouldn't be public
>>>>> BookKeeper#deleteComplete shouldn't be public
>>>>> BookKeeper#halt could be changed to close(), should throw a BKException
>>>>> 
>>>>> LedgerHandle#getLedgerKey passwd is used in BookKeeper, should possibly be
>>>>> private
>>>>> LedgerHandle#getLedgerMetadata shouldn't be public
>>>>> LedgerHandle#getDigestManager shouldn't be public
>>>>> LedgerHandle#getDistributionSchedule shouldn't be public
>>>>> LedgerHandle#writeLedgerConfig shouldn't be public
>>>>> LedgerHandle#addEntry should return void, errors should go in an Exception
>>>>> LedgerHandle#readComplete should not be public
>>>>> LedgerHandle#addComplete should not be public
>>>>> LedgerHandle#readLastConfirmedCompelte should not be public
>>>>> LedgerHandle#closeComplete should not be public
>>>>> 
>>>>> ASyncCallback#RecoverCallback shouldn't be public
>>>>> 
>>>>> HedwigClient#getSslFactory shouldn't be public
>>>>> HedwigClient#getConsumeCallback shouldn't be public
>>>>> HedwigClient#doConnect shouldn't be public
>>>>> HedwigClient#getHostFromChannel shouldn't be public
>>>>> HedwigClient#getResponseHandlerFromChannel shouldn't be public
>>>>> HedwigClient#getHostForTopic shouldn't be public
>>>>> HedwigClient#clearAllTopicsForHost shouldn't be public
>>>>> HedwigClient#getClientTimer shoulnd't be public
>>>>> HedwigClient#stop should throw some sort of Exception in the case of
>>>>> errors
>>>>> 
>>>>> HedwigPu

Re: Review of public APIs

[Ivan Kelly]

> For BOOKKEEPER-90, I still need to decide what to do about ByteString if we 
> do anything at all. Having ByteString in the API tries us to google 
> protobufs. Im not sure this is a huge issue, but it means that this API will 
> never be able to move away from it.

Another open issue is that HedwigClient#stop doesn't throw any exception at the 
moment. Should we add an exception? Also, I think it should be renamed to 
close().

Btw, "mvn compile site" will generate javadoc in target/site/apidocs/index.html

-Ivan

Re: [VOTE] Release ZooKeeper 3.4.0 (candidate 0)

[Ivan Kelly]

-1

The distributed binaries work for starting a server client etc.

I get compile warnings:

compile:
[javac] Compiling 145 source files to 
/Users/ivank/Junk/zookeeper-3.4.0/build/classes
[javac] 
/Users/ivank/Junk/zookeeper-3.4.0/src/java/main/org/apache/zookeeper/Shell.java:276:
 warning: [serial] serializable class 
org.apache.zookeeper.Shell.ExitCodeException has no definition of 
serialVersionUID
[javac]   public static class ExitCodeException extends IOException {
[javac] ^
[javac] 
/Users/ivank/Junk/zookeeper-3.4.0/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java:573:
 warning: [deprecation] org.apache.zookeeper.server.quorum.LeaderElection in 
org.apache.zookeeper.server.quorum has been deprecated
[javac] le = new LeaderElection(this);
[javac]  ^
[javac] 
/Users/ivank/Junk/zookeeper-3.4.0/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java:576:
 warning: [deprecation] 
org.apache.zookeeper.server.quorum.AuthFastLeaderElection in 
org.apache.zookeeper.server.quorum has been deprecated
[javac] le = new AuthFastLeaderElection(this);
[javac]  ^
[javac] 
/Users/ivank/Junk/zookeeper-3.4.0/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java:579:
 warning: [deprecation] 
org.apache.zookeeper.server.quorum.AuthFastLeaderElection in 
org.apache.zookeeper.server.quorum has been deprecated
[javac] le = new AuthFastLeaderElection(this, true);
[javac]  ^
[javac] 
/Users/ivank/Junk/zookeeper-3.4.0/src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java:600:
 warning: [deprecation] org.apache.zookeeper.server.quorum.LeaderElection in 
org.apache.zookeeper.server.quorum has been deprecated
[javac] electionAlg = new LeaderElection(this);
[javac]   ^
[javac] 5 warnings

The first one can be fixed with "static final long serialVersionUID = 
2977863941654513113L;". The deprecation may be a bit more involved though. You 
could suppress the warning, but really you should just remove the use of a 
deprecated class. 

Also, while looking into that, I saw QuorumPeer.java has tabs in it.

org.apache.zookeeper.test.ClientPortBindTest fails, consistently.

This is on Mac OS 10.6.8
$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03-384-10M3425)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02-384, mixed mode)

$ uname -a
Darwin spokegrown-lm 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun  7 16:33:36 
PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386

-Ivan


On 25 Oct 2011, at 10:12, Mahadev Konar wrote:

> Hi all,
> I have created a 3.4.0 release candidate. 3.4 is the first major
> release after ZooKeeper became a TLP.
> The last major release 3.3.0 was done in March, 2010.
> 3.4.0 includes a lot of features, improvements and bug fixes. Please
> look at Release Notes for more information.
> 
> *** Please download, test and VOTE before the
> *** vote closes 1am PDT on Tuesday, Nov1***
> 
> http://people.apache.org/~mahadev/zookeeper-3.4.0-candidate-0/
> 
> Should we release this?
> 
> Please make sure, you download and test the release ASAP. Early
> feedback would be very helpful.
> 
> thanks
> mahadev

Re: [VOTE] Release ZooKeeper 3.4.0 (candidate 0)

[Ivan Kelly]

> the compile warnings are a known issue (not something that should
> block the release IMO), see this jira I filed:
> https://issues.apache.org/jira/browse/ZOOKEEPER-1236
That only refers to the proprietary sun apis. Is there a JIRA that covers the 
use of Zookeeper's own deprecated classes also?
> 
> The test failure on macos is unfortunate, but give it's not a
> "production" platform I don't think it should be considered a blocker
> (although I may not be in the majority here...)
> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_systemReq
Sure, I've just tested on a linux box and it doesn't seem to happen.

I've also just notices that this actual tarball isn't actually gzipped. 

$ file zookeeper-3.4.0.tar.gz 
zookeeper-3.4.0.tar.gz: POSIX tar archive

-Ivan

[VOTE] BookKeeper 4.0.0 Release Candidate 0

[Ivan Kelly]

This is the first candidate release for Apache BookKeeper, version
4.0.0. 

It fixes the following issues:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311293&version=12317843

*** Please download, test and vote by December 7th 2011, 08:00 GMT.

Note that we are voting upon the source (tag), binaries are provided
for convenience.

Source and binary files:
http://people.apache.org/~ivank/bookkeeper-4.0.0-candidate-0/

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachebookkeeper-285/

The tag to be voted upon:
https://svn.apache.org/repos/asf/zookeeper/bookkeeper/tags/release-4.0.0/

BookKeeper's KEYS file containing PGP keys we use to sign the release:
https://svn.apache.org/repos/asf/zookeeper/bookkeeper/tags/release-4.0.0/KEYS

Please download the the source package, and follow the README to build
and run a bookkeeper and hedwig service.

[VOTE] BookKeeper 4.0.0 Release Candidate 1

[Ivan Kelly]

This is the second candidate release for Apache BookKeeper, version
4.0.0. 

It fixes the following issues:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311293&version=12317843

*** Please download, test and vote by December 5th 2011, 21:00 GMT.

Note that we are voting upon the source (tag), binaries are provided
for convenience.

Source and binary files:
http://people.apache.org/~ivank/bookkeeper-4.0.0-candidate-1/

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachebookkeeper-287/

The tag to be voted upon:
https://svn.apache.org/repos/asf/zookeeper/bookkeeper/tags/release-4.0.0/

BookKeeper's KEYS file containing PGP keys we use to sign the release:
https://svn.apache.org/repos/asf/zookeeper/bookkeeper/dist/KEYS

Please download the the source package, and follow the README to build
and run a bookkeeper and hedwig service.

Re: [VOTE] BookKeeper 4.0.0 Release Candidate 1

[Ivan Kelly]

I've updated the binary packages to include licenses for all included
jars. Attached for convenience.

> I'm not sure if this changed btw when Hadoop/ZK started and current
> documentation, notice that it says to include in the license file.
> What hadoop/zk do is to include individual license files in the "lib"
> directory, along side the jars themselves. We probably should change
> this in zk to be in compliance.
Individual license files will probably be hard to do in maven, as the
lib directory will not exist in the source.

-Ivan

Re: [VOTE] BookKeeper 4.0.0 Release Candidate 1

[Ivan Kelly]

The vote has passed with three binding +1 from phunt, breed & fpj.

I'll push out the list to dist and maven. Once the mirrors have
synced, I'll update the website and send out an announcement.

Thanks for taking a look guys,
-Ivan

On Fri, Dec 02, 2011 at 09:13:48PM +0100, Ivan Kelly wrote:
> This is the second candidate release for Apache BookKeeper, version
> 4.0.0. 
> 
> It fixes the following issues:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311293&version=12317843
> 
> *** Please download, test and vote by December 5th 2011, 21:00 GMT.
> 
> Note that we are voting upon the source (tag), binaries are provided
> for convenience.
> 
> Source and binary files:
> http://people.apache.org/~ivank/bookkeeper-4.0.0-candidate-1/
> 
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachebookkeeper-287/
> 
> The tag to be voted upon:
> https://svn.apache.org/repos/asf/zookeeper/bookkeeper/tags/release-4.0.0/
> 
> BookKeeper's KEYS file containing PGP keys we use to sign the release:
> https://svn.apache.org/repos/asf/zookeeper/bookkeeper/dist/KEYS
> 
> Please download the the source package, and follow the README to build
> and run a bookkeeper and hedwig service.

[ANNOUNCE] Apache BookKeeper 4.0.0 released

[Ivan Kelly]

The Apache BookKeeper team is proud to announce Apache BookKeeper
version 4.0.0

This is the first release of the Apache BookKeeper subproject of
ZooKeeper.
The BookKeeper project is made up of a distributed logging service
called BookKeeper and a distributed publish/subscribe system build
on top of BookKeeper called Hedwig.

For BookKeeper release details and downloads, visit:

http://zookeeper.apache.org/bookkeeper/releases.html

BookKeeper 4.0.0 Release Notes are at:
http://zookeeper.apache.org/bookkeeper/docs/r4.0.0/releaseNotes.html

We would like to thank the contributors that made the release
possible.

Regards,

The BookKeeper Team

Re: ZooKeeper 3.4.0 release.

[Ivan Kelly]

>  As Ted mentioned, 3.4 client should be fine with 3.3 server. But
> again, Ill be removing the 3.4.0 release from download and maven
> repo's so it might impact you if you depend on the artifacts.
Since the client should be fine, what is the advantage of removing
from maven? Does anyone actually run a server from what is in maven?
Hbase maybe?

In any case, could you ensure ZOOKEEPER-1311 has been committed to the
3.4 branch before rolling the new release.

-Ivan

Re: Review Request: BOOKKEEPER-95: extends zookeeper JMX to monitor and manage bookie server

[Ivan Kelly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3451/#review4424
---


The bulk of the code in this patch could be removed if zookeeper's 
ZKMBeanRegistry was updated to allow easy extension. Have you looked at 
changing the ZK code? Of course, we'd need to continue to use the code in this 
patch until a version of zookeeper with the change is released, but i think it 
would make things cleaner.


bookkeeper-server/src/main/java/org/apache/bookkeeper/jmx/BKMBeanRegistry.java


Why is this static? 


- Ivan


On 2012-01-11 01:04:57, Sijie Guo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3451/
> ---
> 
> (Updated 2012-01-11 01:04:57)
> 
> 
> Review request for bookkeeper.
> 
> 
> Summary
> ---
> 
> we can extends/reuses zookeeper JMX until to monitor and manage bookie server
> 
> 
> This addresses bug BOOKKEEPER-95.
> https://issues.apache.org/jira/browse/BOOKKEEPER-95
> 
> 
> Diffs
> -
> 
>   bookkeeper-server/src/main/java/org/apache/bookkeeper/jmx/BKMBeanInfo.java 
> PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/jmx/BKMBeanRegistry.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/3451/diff
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sijie
> 
>

Re: Review Request: BOOKKEEPER-98: collect add/read statistics on bookie server

[Ivan Kelly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3452/#review4425
---


In general the code looks good. One issue with it though, is that it doesn't 
allow for JMX to be enabled one multiple bookies running in a single process. 
So observe this, run:

bookkeeper-server/bin/bookkeeper localbookie 3

and then run jconsole. You would expect 3 hierarchies for bookkeeper but there 
is only one. This would be easily resolved by appending the port to the name in 
the BookieServerBean.


- Ivan


On 2012-01-11 01:04:36, Sijie Guo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3452/
> ---
> 
> (Updated 2012-01-11 01:04:36)
> 
> 
> Review request for bookkeeper.
> 
> 
> Summary
> ---
> 
> collect add/read statistics on bookie server
> 
> 
> This addresses bug BOOKKEEPER-98.
> https://issues.apache.org/jira/browse/BOOKKEEPER-98
> 
> 
> Diffs
> -
> 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieServer.java 
> beab5e8 
>   bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BKStats.java 
> PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java
>  12cdcec 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerCacheMXBean.java
>  PRE-CREATION 
>   bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java 
> cb3bb26 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieBean.java 
> PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieMXBean.java
>  PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerCache.java 
> d2f959b 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerCacheBean.java
>  PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieServerBean.java
>  PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieServerMXBean.java
>  PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/NIOServerFactory.java
>  2a8df38 
> 
> Diff: https://reviews.apache.org/r/3452/diff
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sijie
> 
>

Review Request: BOOKKEEPER-23 Timeout requests

[Ivan Kelly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3662/
---

Review request for bookkeeper.


Summary
---

If a client remains connected to a bookie, requests do not timeout. This is an 
undesirable behavior if a bookie is up but not responding to a request for some 
reason, like it is slow or has not handled an exception appropriately and did 
not respond to the client. I propose to add a timeout mechanism.


This addresses bug BOOKKEEPER-23.
https://issues.apache.org/jira/browse/BOOKKEEPER-23


Diffs
-

  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java 
cb3bb26 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java
 5e1c964 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/NIOServerFactory.java
 2a8df38 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java
 2fa79bb 
  
bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestReadTimeout.java
 PRE-CREATION 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BaseTestCase.java 
7403604 

Diff: https://reviews.apache.org/r/3662/diff


Testing
---


Thanks,

Ivan

Review Request: BOOKKEEPER-157 For small packets, increasing number of bookies actually degrades performance.

[Ivan Kelly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3696/
---

Review request for bookkeeper.


Summary
---

When benchmarking with packets smaller than 1k, performance will degrade when 
the ensemble contains more than 3 bookies. See attached diagram.


This addresses bug BOOKKEEPER-157.
https://issues.apache.org/jira/browse/BOOKKEEPER-157


Diffs
-

  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/CRC32DigestManager.java
 fb7c8bc 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java 
547e240 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/MacDigestManager.java
 a3ed9f3 

Diff: https://reviews.apache.org/r/3696/diff


Testing
---


Thanks,

Ivan

Review Request: BOOKKEEPER-158 Move latest benchmarking code into trunk

[Ivan Kelly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3706/
---

Review request for bookkeeper.


Summary
---

Move the code we've been using for benchmarking into trunk.


This addresses bug BOOKKEEPER-158.
https://issues.apache.org/jira/browse/BOOKKEEPER-158


Diffs
-

  bookkeeper-benchmark/bin/benchmark PRE-CREATION 
  bookkeeper-benchmark/conf/log4j.properties PRE-CREATION 
  bookkeeper-benchmark/pom.xml f005531 
  
bookkeeper-benchmark/src/main/java/org/apache/bookkeeper/benchmark/BenchBookie.java
 PRE-CREATION 
  
bookkeeper-benchmark/src/main/java/org/apache/bookkeeper/benchmark/BenchReadThroughputLatency.java
 PRE-CREATION 
  
bookkeeper-benchmark/src/main/java/org/apache/bookkeeper/benchmark/BenchThroughputLatency.java
 PRE-CREATION 
  
bookkeeper-benchmark/src/test/java/org/apache/bookkeeper/benchmark/TestBenchmark.java
 PRE-CREATION 
  bookkeeper-benchmark/src/test/resources/log4j.properties PRE-CREATION 
  bookkeeper-server/bin/bookkeeper b0629bd 

Diff: https://reviews.apache.org/r/3706/diff


Testing
---


Thanks,

Ivan

Review Request: BOOKKEEPER-165 Add versioning support for journal files

[Ivan Kelly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3776/
---

Review request for bookkeeper.


Summary
---

Add versioning in the journal so that we can add new features to the journal 
without breaking backward compatibility.


This addresses bug BOOKKEEPER-165.
https://issues.apache.org/jira/browse/BOOKKEEPER-165


Diffs
-

  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java 
99b797f 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/JournalChannel.java
 PRE-CREATION 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerCache.java 
0fc5206 
  
bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieJournalTest.java
 PRE-CREATION 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/client/ClientUtil.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/3776/diff


Testing
---


Thanks,

Ivan

Re: Review Request: BOOKKEEPER-137 Do not create Ledger index files until absolutely necessary.

[Ivan Kelly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3661/
---

(Updated 2012-02-07 18:13:15.812700)


Review request for bookkeeper.


Changes
---

New diff, addresses Sijie's comments. Diffed against trunk with BOOKKEEPER-165 
applied.


Summary
---

This is an optimization to speed up the case where we have many ledgers and are 
writing to them at random (a benchmark case we currently have). Currently, we 
create the ledger index file and write the first 1k of it to disk immediately. 
With a lot of ledgers being randomly written to, this means a lot of random 
writes on the ledger disk. This fix postpones the creation of the index file 
and writing of the first 1k until the first flush of the ledger.

This patch includes BOOKKEEPER-136, as they both deal in the same area, and I 
found it difficult to separate them.

BOOKKEEPER-135 is not required for this patch, and will need modifications 
after this goes in.


This addresses bug BOOKKEEPER-137.
https://issues.apache.org/jira/browse/BOOKKEEPER-137


Diffs (updated)
-

  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java 
99b797f 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/FileInfo.java 
fa713c8 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/JournalChannel.java
 PRE-CREATION 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerCache.java 
0fc5206 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerDescriptor.java
 728d729 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieServer.java 
2228ab4 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/util/LocalBookKeeper.java 
5706dd8 
  
bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieJournalTest.java
 PRE-CREATION 
  
bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/BookieLayoutVersionTest.java
 c7b07e6 
  
bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryTest.java
 8526db5 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BaseTestCase.java 
db1a763 

Diff: https://reviews.apache.org/r/3661/diff


Testing
---


Thanks,

Ivan

Re: Review Request: BOOKKEEPER-152 Can't recover a ledger whose current ensemble contain failed bookie.

[Ivan Kelly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3737/
---

(Updated 2012-02-10 09:47:35.723185)


Review request for bookkeeper.


Summary
---

Proposed fix ensures that at least one of each quorum replies to 
ReadLastConfirmed.

Refactors code a bit to make the read last confirmed common for recovery and 
standalone read last confirmed.

The bug here was actually that we were waiting for quorumSize responses, from 
the bookies, when really all we need to get a response from one bookie in each 
possible quorum. in the 2/2 case as above this means only 1 bookie need 
response.

There's a fix for the timeouts and an improvement in fencing which fixing this 
uncovered.


This addresses bug BOOKKEEPER-152.
https://issues.apache.org/jira/browse/BOOKKEEPER-152


Diffs (updated)
-

  
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java
 a68fc8c 
  
bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryTest.java
 cbd2277 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BaseTestCase.java 
da52ca5 
  
bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieFailureTest.java
 5873255 
  
bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LedgerRecoveryTest.java
 77a2f69 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RoundRobinDistributionSchedule.java
 4a88747 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/DistributionSchedule.java
 f2ed6bd 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java 
e3d1847 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerRecoveryOp.java
 4625bbb 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedOp.java
 43e999d 

Diff: https://reviews.apache.org/r/3737/diff


Testing
---


Thanks,

Ivan

Re: [ANNOUNCE] Sijie Guo is now a bookkeeper committer

[Ivan Kelly]

This is great news :D

-Ivan

On 13 Feb 2012, at 21:08, Benjamin Reed wrote:

> because of the great contributions (and the expectation of even more
> contributions :) of Sijie Guo to the bookkeeper project. the ZooKeeper
> PMC has voted to make sijie a committer, and he has accepted.
> 
> congratulations sijie! thank you for the work you put into bookkeeper.
> 
> ben

Re: Review Request: BOOKKEEPER-135 Fencing does not check the ledger masterPasswd

[Ivan Kelly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3642/
---

(Updated 2012-03-20 18:13:56.162926)


Review request for bookkeeper.


Changes
---

New patch is much simpler on the server side. A lot of the changes are just 
added the new error path for unauthorized fencing.


Summary
---

When fencing, the ledger handle is not checked before the fencing is applied. 
Currently the openLedger does fail, on because it will addEntry and fail at 
that point, but by this stage, fencing has already been applied. The check 
should be earlier.


This addresses bug BOOKKEEPER-135.
https://issues.apache.org/jira/browse/BOOKKEEPER-135


Diffs (updated)
-

  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java 
ad41ba5 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BKException.java 
911c660 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerOpenOp.java 
56186ab 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerRecoveryOp.java
 c67a79c 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingAddOp.java 
7aad751 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingReadOp.java 
539d6b2 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadLastConfirmedOp.java
 7dd5363 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieClient.java 
8a32c64 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtocol.java 
8598c08 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieServer.java 
1a315e1 
  
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java
 75a8e8c 
  
bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryTest.java
 b8923e8 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestFencing.java 
7de1c10 
  
bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieClientTest.java
 99d6ef0 

Diff: https://reviews.apache.org/r/3642/diff


Testing
---


Thanks,

Ivan

Re: Review Request: BOOKKEEPER-218: Provide journal manager to manage journal related operations

[Ivan Kelly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4737/#review6973
---

Ship it!


lgtm +1

- Ivan


On 2012-04-17 05:50:11, Sijie Guo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4737/
> ---
> 
> (Updated 2012-04-17 05:50:11)
> 
> 
> Review request for bookkeeper.
> 
> 
> Summary
> ---
> 
> Currently we put all journal related operations in Bookie class. It would be 
> better to provide a journal manager to provide journal related operations. It 
> would make Bookie logic more clearly.
> 
> Besides that, some admin tools like BOOKKEEPER-183 needs to provide could use 
> JournalManager to read/check journal files directly.
> 
> 
> This addresses bug BOOKKEEPER-218.
> https://issues.apache.org/jira/browse/BOOKKEEPER-218
> 
> 
> Diffs
> -
> 
>   bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java 
> 00cba69 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieBean.java 
> fa5171b 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieMXBean.java
>  8418469 
>   bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Journal.java 
> PRE-CREATION 
>   
> bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java
>  c172224 
>   
> bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieJournalRollingTest.java
>  e795fce 
> 
> Diff: https://reviews.apache.org/r/4737/diff
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sijie
> 
>

Re: Review Request: BOOKKEEPER-218: Provide journal manager to manage journal related operations

[Ivan Kelly]

> On 2012-04-16 14:50:22, Ivan Kelly wrote:
> >
> 
> Sijie Guo wrote:
> Thanks Ivan for reviewing. would like to update them in new patch.
> 
> only one place about LastLogMark class I would like to explain. I would 
> like to provide a tool to read journal file in BOOKKEEPER-183, including 
> printing current LastLogMark. so admin guys could know the status of journal.
> 
> so I leave LastLogMark as package accessible. does it make sense for you?

re: LastLogMark, thats fine. I simply didn't see why it needed to be accessible 
in the current code base, but if you have a plan for it there's no issue.


- Ivan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4737/#review6935
---


On 2012-04-17 05:50:11, Sijie Guo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4737/
> ---
> 
> (Updated 2012-04-17 05:50:11)
> 
> 
> Review request for bookkeeper.
> 
> 
> Summary
> ---
> 
> Currently we put all journal related operations in Bookie class. It would be 
> better to provide a journal manager to provide journal related operations. It 
> would make Bookie logic more clearly.
> 
> Besides that, some admin tools like BOOKKEEPER-183 needs to provide could use 
> JournalManager to read/check journal files directly.
> 
> 
> This addresses bug BOOKKEEPER-218.
> https://issues.apache.org/jira/browse/BOOKKEEPER-218
> 
> 
> Diffs
> -
> 
>   bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java 
> 00cba69 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieBean.java 
> fa5171b 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieMXBean.java
>  8418469 
>   bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Journal.java 
> PRE-CREATION 
>   
> bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java
>  c172224 
>   
> bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieJournalRollingTest.java
>  e795fce 
> 
> Diff: https://reviews.apache.org/r/4737/diff
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sijie
> 
>

Re: Welcome Ivan Kelly to the ZooKeeper PMC!

[Ivan Kelly]

Thanks guys :)


On Tue, Aug 14, 2012 at 08:38:28AM +, Uma Maheswara Rao G wrote:
> Great! Hearty Congratulations Ivan.
> 
> ( It may not be a special for you, because you are already playing that role 
> :-)  )
> 
> Regards,
> Uma
> 
> 
> From: Rakesh R [rake...@huawei.com]
> Sent: Tuesday, August 14, 2012 1:45 PM
> To: 
> Cc: Ivan Kelly; dev@zookeeper.apache.org; 
> ; 
> Subject: RE: Welcome Ivan Kelly to the ZooKeeper PMC!
> 
> Its great news. Congrats Ivan!
> 
> From: Jordan Zimmerman [jzimmer...@netflix.com]
> Sent: Tuesday, August 14, 2012 1:34 PM
> To: 
> Cc: ; ; 
> ; Ivan Kelly
> Subject: Re: Welcome Ivan Kelly to the ZooKeeper PMC!
> 
> gratz
> 
> On Aug 14, 2012, at 12:53 AM, Flavio Junqueira 
>  wrote:
> 
> > Folks,
> >
> > I'm happy to inform that the PMC has voted and Ivan has happily
> > accepted to become a ZooKeeper PMC member. Welcome aboard, Ivan!
> >
> > -Flavio
> >

Re: Problem in setting up Single Server and Developer Setup

[Ivan Kelly]

With windows, I believe you should try zkServer.cmd and zkCli.cmd. 

-Ivan

On Fri, Nov 09, 2012 at 12:23:33AM +0530, Guruprasad B wrote:
> Hi,
> 
> I am Guruprasad from Bangalore working for HP. I am very much new to
> Zookeeper have started learning about this from two days to implement it in
> our project.
> Now I am trying to set up "Single Server and Developer Setup". I am
> following the steps given in below pointer:
> 
> http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html#sc_InstallingSingleMode
> 
> 1) have created conf/zoo.cfg
> 2) ran bin/zkServer.sh start command
> 
> got below log in command prompt:
> C:\zookeeper\zookeeper-3.4.4\bin>zkServer.sh start
> Welcome to Git (version 1.7.10-preview20120409)
> 
> 
> Run 'git help git' to display the help index.
> Run 'git help ' to display help for specific commands.
> JMX enabled by default
> Using config: /c/zookeeper/zookeeper-3.4.4/bin/../conf/zoo.cfg
> Starting zookeeper ... STARTED
> 
> 3) ran bin/zkCli.sh -server 127.0.0.1:2181
> 
> got below error in command prompt:
> C:\zookeeper\zookeeper-3.4.4\bin>zkCli.sh -server 127.0.0.1:2181
> Welcome to Git (version 1.7.10-preview20120409)
> 
> 
> Run 'git help git' to display the help index.
> Run 'git help ' to display help for specific commands.
> C:\zookeeper\zookeeper-3.4.4\bin\zkCli.sh: line 39: C:\Program: No such
> file or
> directory
> 
> Please help me in fixing this or suggest me some pointers. I have installed
> zookeeper (zookeeper-3.4.4) in windows xp
> 
> Regards,
> 
> Guruprasad

Re: Review Request 47354: ZOOKEEPER-1045 : Quorum mutual authentication using SASL mechanism

[Ivan Kelly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47354/#review134333
---


Ship it!




Ship It!

- Ivan Kelly


On May 20, 2016, 3:10 a.m., Rakesh R wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47354/
> ---
> 
> (Updated May 20, 2016, 3:10 a.m.)
> 
> 
> Review request for zookeeper, fpj, Ivan Kelly, Patrick Hunt, and Raul 
> Gutierrez Segales.
> 
> 
> Bugs: ZOOKEEPER-1045
> https://issues.apache.org/jira/browse/ZOOKEEPER-1045
> 
> 
> Repository: zookeeper-git
> 
> 
> Description
> ---
> 
> Quorum mutual authentication using SASL mechanism - Digest/Kerberos
> 
> 
> Diffs
> -
> 
>   build.xml ab254b2 
>   ivy.xml 95b0e5a 
>   src/java/main/org/apache/zookeeper/Login.java a214c9c 
>   src/java/main/org/apache/zookeeper/client/ZooKeeperSaslClient.java 21ef0fa 
>   src/java/main/org/apache/zookeeper/server/ZooKeeperSaslServer.java 71870ce 
>   
> src/java/main/org/apache/zookeeper/server/auth/SaslServerCallbackHandler.java 
> 2fbd6ed 
>   src/java/main/org/apache/zookeeper/server/quorum/Leader.java 40c6748 
>   src/java/main/org/apache/zookeeper/server/quorum/Learner.java c73a8ee 
>   src/java/main/org/apache/zookeeper/server/quorum/LearnerHandler.java 
> 8a748c7 
>   src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java 
> 20e5f16 
>   src/java/main/org/apache/zookeeper/server/quorum/QuorumPeer.java 2f0f21b 
>   src/java/main/org/apache/zookeeper/server/quorum/QuorumPeerConfig.java 
> 8ae820d 
>   src/java/main/org/apache/zookeeper/server/quorum/QuorumPeerMain.java 
> e9c8007 
>   
> src/java/main/org/apache/zookeeper/server/quorum/auth/NullQuorumAuthClient.java
>  PRE-CREATION 
>   
> src/java/main/org/apache/zookeeper/server/quorum/auth/NullQuorumAuthServer.java
>  PRE-CREATION 
>   src/java/main/org/apache/zookeeper/server/quorum/auth/QuorumAuth.java 
> PRE-CREATION 
>   src/java/main/org/apache/zookeeper/server/quorum/auth/QuorumAuthClient.java 
> PRE-CREATION 
>   src/java/main/org/apache/zookeeper/server/quorum/auth/QuorumAuthServer.java 
> PRE-CREATION 
>   src/java/main/org/apache/zookeeper/server/quorum/auth/README.md 
> PRE-CREATION 
>   
> src/java/main/org/apache/zookeeper/server/quorum/auth/SaslQuorumAuthClient.java
>  PRE-CREATION 
>   
> src/java/main/org/apache/zookeeper/server/quorum/auth/SaslQuorumAuthServer.java
>  PRE-CREATION 
>   src/java/main/org/apache/zookeeper/util/SecurityUtils.java PRE-CREATION 
>   src/java/test/data/kerberos/minikdc-krb5.conf PRE-CREATION 
>   src/java/test/data/kerberos/minikdc.ldiff PRE-CREATION 
>   src/java/test/org/apache/zookeeper/server/quorum/CnxManagerTest.java 
> 831d3ed 
>   
> src/java/test/org/apache/zookeeper/server/quorum/FLEBackwardElectionRoundTest.java
>  c1259d1 
>   src/java/test/org/apache/zookeeper/server/quorum/FLECompatibilityTest.java 
> 72e4fc9 
>   src/java/test/org/apache/zookeeper/server/quorum/FLEDontCareTest.java 
> a4c0cb0 
>   src/java/test/org/apache/zookeeper/server/quorum/FLELostMessageTest.java 
> 39a53ca 
>   src/java/test/org/apache/zookeeper/server/quorum/LearnerTest.java 2ae57ce 
>   src/java/test/org/apache/zookeeper/server/quorum/QuorumCnxManagerTest.java 
> PRE-CREATION 
>   src/java/test/org/apache/zookeeper/server/quorum/QuorumPeerTestBase.java 
> ef552db 
>   src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java ab8ce42 
>   
> src/java/test/org/apache/zookeeper/server/quorum/auth/KerberosSecurityTestcase.java
>  PRE-CREATION 
>   
> src/java/test/org/apache/zookeeper/server/quorum/auth/KerberosTestUtils.java 
> PRE-CREATION 
>   src/java/test/org/apache/zookeeper/server/quorum/auth/MiniKdc.java 
> PRE-CREATION 
>   src/java/test/org/apache/zookeeper/server/quorum/auth/MiniKdcTest.java 
> PRE-CREATION 
>   
> src/java/test/org/apache/zookeeper/server/quorum/auth/QuorumAuthTestBase.java 
> PRE-CREATION 
>   
> src/java/test/org/apache/zookeeper/server/quorum/auth/QuorumAuthUpgradeTest.java
>  PRE-CREATION 
>   
> src/java/test/org/apache/zookeeper/server/quorum/auth/QuorumDigestAuthTest.java
>  PRE-CREATION 
>   
> src/java/test/org/apache/zookeeper/server/quorum/auth/QuorumKerberosAuthTest.java
>  PRE-CREATION 
>   src/java/test/org/apache/zookeeper/test/FLEPredicateTest.java 8088505 
>   src/zookeeper.jute 6521e54 
> 
> Diff: https://reviews.apache.org/r/47354/diff/
> 
> 
> Testing
> ---
> 
> Added unit test cases to verify the changes.
> 
> 
> Thanks,
> 
> Rakesh R
> 
>

Re: [VOTE] Apache ZooKeeper release 3.5.1-alpha candidate 3

[Ivan Kelly]

Hi Michi,

-1

The tarball contains a bunch of jars(under lib), but there is no mention of
these in the NOTICE file. Perhaps phunt knows whether this is ok, but my
understanding is that if you are distributing the binaries, you need to
mention them in the NOTICE.

Apart from that the release looks good. I tested against midonet (my
companies product), wrote a small test app, ran the tests. All looked fine.

-Ivan



On Mon, Jul 13, 2015 at 8:15 PM Michi Mutsuzaki  wrote:

> We have received 2 binding +1s so far (Flavio, Michi). I'm extending
> the voting period to give other PMCs some more time to vote. Please
> vote by July 20th 2015, 23:59 UTC+0.
>
> Thanks!
> --Michi
>
>
> On Thu, Jul 9, 2015 at 11:44 PM, Rakesh R  wrote:
> >
> > +1
> >
> > - Built from source.
> > - Ran unit tests and tested few shell cmds, JMX attributes, basic
> watcher behaviors.
> >
> > Thanks Michi for making the release!
> >
> > Best Regards,
> > Rakesh
> >
> > -Original Message-
> > From: Raúl Gutiérrez Segalés [mailto:r...@itevenworks.net]
> > Sent: 10 July 2015 11:42
> > To: dev@zookeeper.apache.org
> > Subject: Re: [VOTE] Apache ZooKeeper release 3.5.1-alpha candidate 3
> >
> > +1
> >
> > My testing was:
> > * 3 participants + 2 observers using systemd-nspawn
> > * triggered elections
> > * ran some smoke tests with zk-shell along with more elections
> >
> > All looks good. Thanks Michi!
> >
> >
> > -rgs
> >
> >
> > On 9 July 2015 at 15:23, Hongchao Deng  wrote:
> >
> >> +1.Ran unit test and some basic programs. It went pretty smooth.Thanks
> >> Michi!
> >>
> >> - Hongchao Deng
> >>
> >> > Date: Thu, 9 Jul 2015 12:09:44 -0700
> >> > Subject: Re: [VOTE] Apache ZooKeeper release 3.5.1-alpha candidate 3
> >> > From: mutsuz...@gmail.com
> >> > To: dev@zookeeper.apache.org
> >> >
> >> > Just a reminder, the vote is closing in about two days, and we
> >> > haven't received enough binding votes yet. Please vote when you get a
> chance.
> >> > Thanks!
> >> >
> >> > On Mon, Jul 6, 2015 at 10:58 PM, Michi Mutsuzaki
> >> > 
> >> wrote:
> >> > > +1
> >> > >
> >> > > - Ran ant test on ubuntu.
> >> > > - Compiled the c client on mac.
> >> > >
> >> > > On Mon, Jul 6, 2015 at 2:26 AM, Flavio Junqueira
> >> > >  wrote:
> >> > >> +1, I ran tests, checked license and notice files, ran release
> >> > >> +audit
> >> to make sure that rat had no issues to report.
> >> > >>
> >> > >> -Flavio
> >> > >>
> >> > >>> On 28 Jun 2015, at 08:47, Michi Mutsuzaki 
> >> wrote:
> >> > >>>
> >> > >>> This is the fourth release candidate for 3.5.1-alpha. This
> >> > >>> candidate fixes the c client compilation error on os x found in
> >> > >>> the previous candidate. The full release notes is available at:
> >> > >>>
> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310
> >> 801&version=12326786
> >> > >>>
> >> > >>> *** Please download, test and vote by July 11th 2015, 23:59 UTC+0.
> >> ***
> >> > >>>
> >> > >>> Source files:
> >> > >>> http://people.apache.org/~michim/zookeeper-3.5.1-alpha-candidate
> >> > >>> -3/
> >> > >>>
> >> > >>> Maven staging repo:
> >> > >>>
> >> https://repository.apache.org/content/groups/staging/org/apache/zookee
> >> per/zookeeper/3.5.1-alpha/
> >> > >>>
> >> > >>> The tag to be voted upon:
> >> > >>> https://svn.apache.org/repos/asf/zookeeper/tags/release-3.5.1-rc
> >> > >>> 3/
> >> > >>>
> >> > >>> ZooKeeper's KEYS file containing PGP keys we use to sign the
> release:
> >> > >>> http://www.apache.org/dist/zookeeper/KEYS
> >> > >>>
> >> > >>> Should we release this candidate?
> >> > >>
> >>
> >>
>

Re: [VOTE] Apache ZooKeeper release 3.5.1-alpha candidate 3

[Ivan Kelly]

I you have time to check this release, and dont foresee that you'll have
time to check the next rc as in depth, I'd go ahead and check this one. If
the licensing issue is the only problem (it seems to be all that has been
raised this vote), the all that needs to be fixed and checked in the next
rc is the licensing, which is a quick check.

-Ivan

On Sun, Jul 19, 2015 at 4:20 PM Flavio P JUNQUEIRA  wrote:

> Please see the discussion here, Camille:
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-2235
>
> -Flavio
>
>
> On Sun, Jul 19, 2015 at 2:44 PM, Camille Fournier 
> wrote:
>
> > I'm checking out the tag to do a check. Do we need to do anything with
> > Ivan's -1? I don't want to spend time checking a release that can't move
> > forward due to a -1.
> >
> > Thanks,
> > C
> >
> > On Thu, Jul 16, 2015 at 8:56 AM, Flavio Junqueira <
> > flavio.junque...@gmail.com> wrote:
> >
> > > That's a good observation. The description here about what NOTICE
> should
> > > contain:
> > >
> > > http://www.apache.org/dev/licensing-howto.html <
> > > http://www.apache.org/dev/licensing-howto.html>
> > >
> > > says to be careful and not add unnecessarily. Having said that, the
> jars
> > > are under Apache 2.0, except these:
> > >
> > > - jline which has a license file under lib, I don't think we need to
> add
> > > it to the NOTICE file
> > > - slf4j has an MIT license and the license says that the copyright
> notice
> > > and the permission notice need to be included. We need to copy the
> > license
> > > under lib just like we did for jline, but again I don't think we need
> to
> > > copy it over as per the description I cited above
> > > - servlet-api needs an addition to NOTICE, see the discussion in
> > > LUCENE-4431
> > >
> > > If I haven't missed anything, we need to add two license files and make
> > > one addition to NOTICE.txt.
> > >
> > > That's the longest release process ever in this project!
> > >
> > > -Flavio
> > >
> > > > On 16 Jul 2015, at 12:41, Ivan Kelly  wrote:
> > > >
> > > > Hi Michi,
> > > >
> > > > -1
> > > >
> > > > The tarball contains a bunch of jars(under lib), but there is no
> > mention
> > > of
> > > > these in the NOTICE file. Perhaps phunt knows whether this is ok, but
> > my
> > > > understanding is that if you are distributing the binaries, you need
> to
> > > > mention them in the NOTICE.
> > > >
> > > > Apart from that the release looks good. I tested against midonet (my
> > > > companies product), wrote a small test app, ran the tests. All looked
> > > fine.
> > > >
> > > > -Ivan
> > > >
> > > >
> > > >
> > > > On Mon, Jul 13, 2015 at 8:15 PM Michi Mutsuzaki  >
> > > wrote:
> > > >
> > > >> We have received 2 binding +1s so far (Flavio, Michi). I'm extending
> > > >> the voting period to give other PMCs some more time to vote. Please
> > > >> vote by July 20th 2015, 23:59 UTC+0.
> > > >>
> > > >> Thanks!
> > > >> --Michi
> > > >>
> > > >>
> > > >> On Thu, Jul 9, 2015 at 11:44 PM, Rakesh R 
> wrote:
> > > >>>
> > > >>> +1
> > > >>>
> > > >>> - Built from source.
> > > >>> - Ran unit tests and tested few shell cmds, JMX attributes, basic
> > > >> watcher behaviors.
> > > >>>
> > > >>> Thanks Michi for making the release!
> > > >>>
> > > >>> Best Regards,
> > > >>> Rakesh
> > > >>>
> > > >>> -Original Message-
> > > >>> From: Raúl Gutiérrez Segalés [mailto:r...@itevenworks.net]
> > > >>> Sent: 10 July 2015 11:42
> > > >>> To: dev@zookeeper.apache.org
> > > >>> Subject: Re: [VOTE] Apache ZooKeeper release 3.5.1-alpha candidate
> 3
> > > >>>
> > > >>> +1
> > > >>>
> > > >>> My testing was:
> > > >>> * 3 participants + 2 observers using systemd-nspawn
> > > >>> * triggered elections
> > > >>> * ran some smoke tests 

Re: [VOTE] Apache ZooKeeper release 3.5.1-alpha candidate 4

[Ivan Kelly]

+1

lgtm

I checked the checksums and sig.
Built and tested with ant jar
Checked the licenses
Ran it against our internal QA system.

-Ivan

On Tue, Jul 28, 2015 at 10:03 AM Michi Mutsuzaki 
wrote:

> This is the fifth release candidate for 3.5.1-alpha. There are 6
> commits since the last release candidate, including the commit to add
> license information:
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-1423
> https://issues.apache.org/jira/browse/ZOOKEEPER-2140
> https://issues.apache.org/jira/browse/ZOOKEEPER-2221
> https://issues.apache.org/jira/browse/ZOOKEEPER-2223
> https://issues.apache.org/jira/browse/ZOOKEEPER-2224
> https://issues.apache.org/jira/browse/ZOOKEEPER-2235
>
> The full release notes is available at:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310801&version=12326786
>
> *** Please download, test and vote by August 15th 2015, 23:59 UTC+0. ***
>
> Source files:
> http://people.apache.org/~michim/zookeeper-3.5.1-alpha-candidate-4/
>
> Maven staging repo:
>
> https://repository.apache.org/content/groups/staging/org/apache/zookeeper/zookeeper/3.5.1-alpha/
>
> The tag to be voted upon:
> http://svn.apache.org/viewvc/zookeeper/tags/release-3.5.1-rc4/
>
> ZooKeeper's KEYS file containing PGP keys we use to sign the release:
> http://www.apache.org/dist/zookeeper/KEYS
>
> Should we release this candidate?
>

Re: QOP SASL property

[Ivan Kelly]

Is auth-int necessary if we have SSL on the client (as there is in trunk)?
My understanding is that all comms would have to be wrapped by sasl if you
have QOP enabled.

-Ivan

On Fri, Oct 9, 2015 at 9:42 AM Flavio Junqueira  wrote:

> Hi Chris,
>
> Yeah, I was thinking along the same lines, so sounds like a plan. I know
> Raul is going to hate me for this, but I'd really like to have this in
> 3.4.7. It sounds like a simple enough change that we can have in shortly,
> does it sound right?
>
> Please go ahead with the jira if you have time, and if you don't have time
> to work on the patch, just assign it to me.
>
> -Flavio
>
>
> > On 08 Oct 2015, at 23:16, Chris Nauroth 
> wrote:
> >
> > Hi Flavio,
> >
> > It appears that the current code doesn't give us any way to control the
> > QOP, so it must be always using the default QOP of "auth" (authentication
> > only).  This is because the calls to Sasl#createSaslClient and
> > Sasl#createSaslServer pass a hard-coded null for the properties map.
> >
> >
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> > keeper/client/ZooKeeperSaslClient.java#L240
> >
> >
> >
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> > keeper/client/ZooKeeperSaslClient.java#L288
> >
> >
> >
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> > keeper/server/ZooKeeperSaslServer.java#L118
> >
> >
> >
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> > keeper/server/ZooKeeperSaslServer.java#L144
> >
> >
> > If we want to support setting QOP to "auth-int" (authentication +
> > integrity/man-in-the-middle tampering protection) or "auth-conf"
> > (authentication + integrity + confidentiality/encryption), then I think
> > we'll need to make code changes to read a new QOP configuration property,
> > put it into a Map using Sasl#QOP as the key, and then pass it along to
> the
> > Sasl#createSaslClient and Sasl#createSaslServer calls.
> >
> > Is this what you need?  If so, then I'd be happy to write up the proposal
> > in a new JIRA.  I didn't find any existing open JIRAs that look relevant.
> >
> > --Chris Nauroth
> >
> >
> >
> >
> > On 10/8/15, 2:06 PM, "Flavio Junqueira"  wrote:
> >
> >> Has anyone tried to use the QOP (Quality of Protection) property for
> SASL
> >> when running ZooKeeper?
> >>
> >> -Flavio
> >
>
>

Re: QOP SASL property

[Ivan Kelly]

IMO, adding QOP to 3.4 would be a fairly large and invasive change, which
is something which shouldn't be done on the stable branch.

-Ivan

On Fri, Oct 9, 2015 at 4:02 PM Flavio Junqueira  wrote:

> Not in the 3.4 branch, which is the latest stable branch at the moment.
>
> -Flavio
>
> > On 09 Oct 2015, at 15:00, Ivan Kelly  wrote:
> >
> > Is auth-int necessary if we have SSL on the client (as there is in
> trunk)?
> > My understanding is that all comms would have to be wrapped by sasl if
> you
> > have QOP enabled.
> >
> > -Ivan
> >
> > On Fri, Oct 9, 2015 at 9:42 AM Flavio Junqueira  wrote:
> >
> >> Hi Chris,
> >>
> >> Yeah, I was thinking along the same lines, so sounds like a plan. I know
> >> Raul is going to hate me for this, but I'd really like to have this in
> >> 3.4.7. It sounds like a simple enough change that we can have in
> shortly,
> >> does it sound right?
> >>
> >> Please go ahead with the jira if you have time, and if you don't have
> time
> >> to work on the patch, just assign it to me.
> >>
> >> -Flavio
> >>
> >>
> >>> On 08 Oct 2015, at 23:16, Chris Nauroth 
> >> wrote:
> >>>
> >>> Hi Flavio,
> >>>
> >>> It appears that the current code doesn't give us any way to control the
> >>> QOP, so it must be always using the default QOP of "auth"
> (authentication
> >>> only).  This is because the calls to Sasl#createSaslClient and
> >>> Sasl#createSaslServer pass a hard-coded null for the properties map.
> >>>
> >>>
> >>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> >>> keeper/client/ZooKeeperSaslClient.java#L240
> >>>
> >>>
> >>>
> >>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> >>> keeper/client/ZooKeeperSaslClient.java#L288
> >>>
> >>>
> >>>
> >>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> >>> keeper/server/ZooKeeperSaslServer.java#L118
> >>>
> >>>
> >>>
> >>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> >>> keeper/server/ZooKeeperSaslServer.java#L144
> >>>
> >>>
> >>> If we want to support setting QOP to "auth-int" (authentication +
> >>> integrity/man-in-the-middle tampering protection) or "auth-conf"
> >>> (authentication + integrity + confidentiality/encryption), then I think
> >>> we'll need to make code changes to read a new QOP configuration
> property,
> >>> put it into a Map using Sasl#QOP as the key, and then pass it along to
> >> the
> >>> Sasl#createSaslClient and Sasl#createSaslServer calls.
> >>>
> >>> Is this what you need?  If so, then I'd be happy to write up the
> proposal
> >>> in a new JIRA.  I didn't find any existing open JIRAs that look
> relevant.
> >>>
> >>> --Chris Nauroth
> >>>
> >>>
> >>>
> >>>
> >>> On 10/8/15, 2:06 PM, "Flavio Junqueira"  wrote:
> >>>
> >>>> Has anyone tried to use the QOP (Quality of Protection) property for
> >> SASL
> >>>> when running ZooKeeper?
> >>>>
> >>>> -Flavio
> >>>
> >>
> >>
>
>

Re: QOP SASL property

[Ivan Kelly]

To protect the integrity of each packet, each packet needs to be wrapped by
SaslClient/SaslServer (see wrap/unwrap in
http://docs.oracle.com/javase/8/docs/api/javax/security/sasl/SaslClient.html).
Currently sasl is only used to initialize the connection, and then we send
over the raw socket. This changes the lifetime of the sasl components, as
it needs to be used with all communication. It's not like SSL, where we
negotiate SSL at the start and then the SSL engine provides a socket which
we use to send data.

-Ivan

On Fri, Oct 9, 2015 at 4:33 PM Flavio Junqueira  wrote:

> I'm not sure based on what you say that it'd be invasive. Enabling
> different types of QOP seems to be relatively straightforward, unless I'm
> missing something here. Chris did a good job describing what needs to be
> done, and this far I have the same understanding of the changes.
>
> -Flavio
>
> > On 09 Oct 2015, at 15:30, Ivan Kelly  wrote:
> >
> > IMO, adding QOP to 3.4 would be a fairly large and invasive change, which
> > is something which shouldn't be done on the stable branch.
> >
> > -Ivan
> >
> > On Fri, Oct 9, 2015 at 4:02 PM Flavio Junqueira  wrote:
> >
> >> Not in the 3.4 branch, which is the latest stable branch at the moment.
> >>
> >> -Flavio
> >>
> >>> On 09 Oct 2015, at 15:00, Ivan Kelly  wrote:
> >>>
> >>> Is auth-int necessary if we have SSL on the client (as there is in
> >> trunk)?
> >>> My understanding is that all comms would have to be wrapped by sasl if
> >> you
> >>> have QOP enabled.
> >>>
> >>> -Ivan
> >>>
> >>> On Fri, Oct 9, 2015 at 9:42 AM Flavio Junqueira 
> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Yeah, I was thinking along the same lines, so sounds like a plan. I
> know
> >>>> Raul is going to hate me for this, but I'd really like to have this in
> >>>> 3.4.7. It sounds like a simple enough change that we can have in
> >> shortly,
> >>>> does it sound right?
> >>>>
> >>>> Please go ahead with the jira if you have time, and if you don't have
> >> time
> >>>> to work on the patch, just assign it to me.
> >>>>
> >>>> -Flavio
> >>>>
> >>>>
> >>>>> On 08 Oct 2015, at 23:16, Chris Nauroth 
> >>>> wrote:
> >>>>>
> >>>>> Hi Flavio,
> >>>>>
> >>>>> It appears that the current code doesn't give us any way to control
> the
> >>>>> QOP, so it must be always using the default QOP of "auth"
> >> (authentication
> >>>>> only).  This is because the calls to Sasl#createSaslClient and
> >>>>> Sasl#createSaslServer pass a hard-coded null for the properties map.
> >>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> >>>>> keeper/client/ZooKeeperSaslClient.java#L240
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> >>>>> keeper/client/ZooKeeperSaslClient.java#L288
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> >>>>> keeper/server/ZooKeeperSaslServer.java#L118
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> >>>>> keeper/server/ZooKeeperSaslServer.java#L144
> >>>>>
> >>>>>
> >>>>> If we want to support setting QOP to "auth-int" (authentication +
> >>>>> integrity/man-in-the-middle tampering protection) or "auth-conf"
> >>>>> (authentication + integrity + confidentiality/encryption), then I
> think
> >>>>> we'll need to make code changes to read a new QOP configuration
> >> property,
> >>>>> put it into a Map using Sasl#QOP as the key, and then pass it along
> to
> >>>> the
> >>>>> Sasl#createSaslClient and Sasl#createSaslServer calls.
> >>>>>
> >>>>> Is this what you need?  If so, then I'd be happy to write up the
> >> proposal
> >>>>> in a new JIRA.  I didn't find any existing open JIRAs that look
> >> relevant.
> >>>>>
> >>>>> --Chris Nauroth
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 10/8/15, 2:06 PM, "Flavio Junqueira"  wrote:
> >>>>>
> >>>>>> Has anyone tried to use the QOP (Quality of Protection) property for
> >>>> SASL
> >>>>>> when running ZooKeeper?
> >>>>>>
> >>>>>> -Flavio
> >>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Re: QOP SASL property

[Ivan Kelly]

I think it's a fairly big change, especially since we'd then have a lot of
conditional if (sasl) { wrap_bytes } else { dont_wrap }. And then it
affects all communication between server and client, which is quite risky.

On Fri, Oct 9, 2015 at 4:54 PM Flavio Junqueira  wrote:

> Ok, got it, but it sounds like we can just wrap and unwrap the bytes we
> are sending, no? Do you think that will end up being a lot of changes?
>
> -Flavio
>
> > On 09 Oct 2015, at 15:38, Ivan Kelly  wrote:
> >
> > To protect the integrity of each packet, each packet needs to be wrapped
> by
> > SaslClient/SaslServer (see wrap/unwrap in
> >
> http://docs.oracle.com/javase/8/docs/api/javax/security/sasl/SaslClient.html
> ).
> > Currently sasl is only used to initialize the connection, and then we
> send
> > over the raw socket. This changes the lifetime of the sasl components, as
> > it needs to be used with all communication. It's not like SSL, where we
> > negotiate SSL at the start and then the SSL engine provides a socket
> which
> > we use to send data.
> >
> > -Ivan
> >
> > On Fri, Oct 9, 2015 at 4:33 PM Flavio Junqueira  wrote:
> >
> >> I'm not sure based on what you say that it'd be invasive. Enabling
> >> different types of QOP seems to be relatively straightforward, unless
> I'm
> >> missing something here. Chris did a good job describing what needs to be
> >> done, and this far I have the same understanding of the changes.
> >>
> >> -Flavio
> >>
> >>> On 09 Oct 2015, at 15:30, Ivan Kelly  wrote:
> >>>
> >>> IMO, adding QOP to 3.4 would be a fairly large and invasive change,
> which
> >>> is something which shouldn't be done on the stable branch.
> >>>
> >>> -Ivan
> >>>
> >>> On Fri, Oct 9, 2015 at 4:02 PM Flavio Junqueira 
> wrote:
> >>>
> >>>> Not in the 3.4 branch, which is the latest stable branch at the
> moment.
> >>>>
> >>>> -Flavio
> >>>>
> >>>>> On 09 Oct 2015, at 15:00, Ivan Kelly  wrote:
> >>>>>
> >>>>> Is auth-int necessary if we have SSL on the client (as there is in
> >>>> trunk)?
> >>>>> My understanding is that all comms would have to be wrapped by sasl
> if
> >>>> you
> >>>>> have QOP enabled.
> >>>>>
> >>>>> -Ivan
> >>>>>
> >>>>> On Fri, Oct 9, 2015 at 9:42 AM Flavio Junqueira 
> >> wrote:
> >>>>>
> >>>>>> Hi Chris,
> >>>>>>
> >>>>>> Yeah, I was thinking along the same lines, so sounds like a plan. I
> >> know
> >>>>>> Raul is going to hate me for this, but I'd really like to have this
> in
> >>>>>> 3.4.7. It sounds like a simple enough change that we can have in
> >>>> shortly,
> >>>>>> does it sound right?
> >>>>>>
> >>>>>> Please go ahead with the jira if you have time, and if you don't
> have
> >>>> time
> >>>>>> to work on the patch, just assign it to me.
> >>>>>>
> >>>>>> -Flavio
> >>>>>>
> >>>>>>
> >>>>>>> On 08 Oct 2015, at 23:16, Chris Nauroth 
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Hi Flavio,
> >>>>>>>
> >>>>>>> It appears that the current code doesn't give us any way to control
> >> the
> >>>>>>> QOP, so it must be always using the default QOP of "auth"
> >>>> (authentication
> >>>>>>> only).  This is because the calls to Sasl#createSaslClient and
> >>>>>>> Sasl#createSaslServer pass a hard-coded null for the properties
> map.
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> >>>>>>> keeper/client/ZooKeeperSaslClient.java#L240
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> >>>>>>> keeper/client/ZooKeeperSaslClient.java#L288
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> >>>>>>> keeper/server/ZooKeeperSaslServer.java#L118
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zoo
> >>>>>>> keeper/server/ZooKeeperSaslServer.java#L144
> >>>>>>>
> >>>>>>>
> >>>>>>> If we want to support setting QOP to "auth-int" (authentication +
> >>>>>>> integrity/man-in-the-middle tampering protection) or "auth-conf"
> >>>>>>> (authentication + integrity + confidentiality/encryption), then I
> >> think
> >>>>>>> we'll need to make code changes to read a new QOP configuration
> >>>> property,
> >>>>>>> put it into a Map using Sasl#QOP as the key, and then pass it along
> >> to
> >>>>>> the
> >>>>>>> Sasl#createSaslClient and Sasl#createSaslServer calls.
> >>>>>>>
> >>>>>>> Is this what you need?  If so, then I'd be happy to write up the
> >>>> proposal
> >>>>>>> in a new JIRA.  I didn't find any existing open JIRAs that look
> >>>> relevant.
> >>>>>>>
> >>>>>>> --Chris Nauroth
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 10/8/15, 2:06 PM, "Flavio Junqueira"  wrote:
> >>>>>>>
> >>>>>>>> Has anyone tried to use the QOP (Quality of Protection) property
> for
> >>>>>> SASL
> >>>>>>>> when running ZooKeeper?
> >>>>>>>>
> >>>>>>>> -Flavio
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Re: QOP SASL property

[Ivan Kelly]

Just to add, digest-md5 is obselete now[1], due to vulnerability to MITM
attacks, so that case would need SSL in any case.

krb is still safe though.

[1] http://tools.ietf.org/html/rfc6331

On Fri, Oct 9, 2015 at 7:10 PM Chris Nauroth 
wrote:

> Ivan is right.  Thank you!  I forgot about this part.
>
> When QOP is "auth" (the default), then SASL is just a one-time handshake
> at the front of the connection, and the subsequent transfer of bytes
> between client and server remain the same.  When QOP is "auth-int", SASL
> needs to inspect the transferred bytes for digest-MD5 verification to
> guard against man-in-the-middle tampering.  When QOP is "auth-conf", SASL
> needs to encrypt the bytes for privacy against a man-in-the-middle.  The
> latter two cases require the wrap/unwrap calls that Ivan mentioned.
>
> In Hadoop, we encapsulate the wrap/unwrap behind special stream subclasses
> that wrap over the underlying "raw" stream.
>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/ha
> doop-common/src/main/java/org/apache/hadoop/security/SaslInputStream.java
> <https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslInputStream.java>
>
>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/ha
> doop-common/src/main/java/org/apache/hadoop/security/SaslOutputStream.java
> <https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslOutputStream.java>
>
>
> That's nice, because the code consuming the streams doesn't need to worry
> about repeated wrapping and unwrapping.  However, even if we set up
> something like these classes in ZooKeeper, it appears the ZooKeeper
> codebase isn't structured to make inserting those wrappers easy.
>
> This does look like it would be more invasive than I originally described.
>
> --Chris Nauroth
>
>
>
>
> On 10/9/15, 7:57 AM, "Ivan Kelly"  wrote:
>
> >I think it's a fairly big change, especially since we'd then have a lot of
> >conditional if (sasl) { wrap_bytes } else { dont_wrap }. And then it
> >affects all communication between server and client, which is quite risky.
> >
> >On Fri, Oct 9, 2015 at 4:54 PM Flavio Junqueira  wrote:
> >
> >> Ok, got it, but it sounds like we can just wrap and unwrap the bytes we
> >> are sending, no? Do you think that will end up being a lot of changes?
> >>
> >> -Flavio
> >>
> >> > On 09 Oct 2015, at 15:38, Ivan Kelly  wrote:
> >> >
> >> > To protect the integrity of each packet, each packet needs to be
> >>wrapped
> >> by
> >> > SaslClient/SaslServer (see wrap/unwrap in
> >> >
> >>
> >>
> http://docs.oracle.com/javase/8/docs/api/javax/security/sasl/SaslClient.h
> >>tml
> >> ).
> >> > Currently sasl is only used to initialize the connection, and then we
> >> send
> >> > over the raw socket. This changes the lifetime of the sasl
> >>components, as
> >> > it needs to be used with all communication. It's not like SSL, where
> >>we
> >> > negotiate SSL at the start and then the SSL engine provides a socket
> >> which
> >> > we use to send data.
> >> >
> >> > -Ivan
> >> >
> >> > On Fri, Oct 9, 2015 at 4:33 PM Flavio Junqueira 
> >>wrote:
> >> >
> >> >> I'm not sure based on what you say that it'd be invasive. Enabling
> >> >> different types of QOP seems to be relatively straightforward, unless
> >> I'm
> >> >> missing something here. Chris did a good job describing what needs
> >>to be
> >> >> done, and this far I have the same understanding of the changes.
> >> >>
> >> >> -Flavio
> >> >>
> >> >>> On 09 Oct 2015, at 15:30, Ivan Kelly  wrote:
> >> >>>
> >> >>> IMO, adding QOP to 3.4 would be a fairly large and invasive change,
> >> which
> >> >>> is something which shouldn't be done on the stable branch.
> >> >>>
> >> >>> -Ivan
> >> >>>
> >> >>> On Fri, Oct 9, 2015 at 4:02 PM Flavio Junqueira 
> >> wrote:
> >> >>>
> >> >>>> Not in the 3.4 branch, which is the latest stable branch at the
> >> moment.
> >> >>>>
> >> >>

Re: QOP SASL property

[Ivan Kelly]

What's the driving use case behind all this? It's going to have to wait for
a release in any case, so why not wait for the next release from trunk, and
then use SSL + SASL auth?


On Sun, Oct 11, 2015 at 3:00 AM Andrew Purtell 
wrote:

> Because different authentication mechanisms can be plugged in and are used
> on demand when znodes are accessed according to whatever the ACL specifies,
> this is why we opted to tunnel SASL auth within the zookeeper protocol
> instead of wrap the entire connection when contributing this feature. SASL
> is just one of several possible auth methods that a client might use, even
> on a single connection. Of course this is distinctly unconventional, but at
> least at the time of contribution was preferred by the community.
>
>
> > On Oct 10, 2015, at 3:53 PM, Chris Nauroth 
> wrote:
> >
> > I filed ZOOKEEPER-2289 to track adding support for the full set of QOP
> > settings.  I did not target any specific fix version since there isn't
> > consensus yet on that.  Thank you, Flavio and Ivan.
> >
> > --Chris Nauroth
> >
> >
> >
> >
> >> On 10/9/15, 11:55 AM, "Ivan Kelly"  wrote:
> >>
> >> Just to add, digest-md5 is obselete now[1], due to vulnerability to MITM
> >> attacks, so that case would need SSL in any case.
> >>
> >> krb is still safe though.
> >>
> >> [1] http://tools.ietf.org/html/rfc6331
> >>
> >> On Fri, Oct 9, 2015 at 7:10 PM Chris Nauroth 
> >> wrote:
> >>
> >>> Ivan is right.  Thank you!  I forgot about this part.
> >>>
> >>> When QOP is "auth" (the default), then SASL is just a one-time
> handshake
> >>> at the front of the connection, and the subsequent transfer of bytes
> >>> between client and server remain the same.  When QOP is "auth-int",
> SASL
> >>> needs to inspect the transferred bytes for digest-MD5 verification to
> >>> guard against man-in-the-middle tampering.  When QOP is "auth-conf",
> >>> SASL
> >>> needs to encrypt the bytes for privacy against a man-in-the-middle.
> The
> >>> latter two cases require the wrap/unwrap calls that Ivan mentioned.
> >>>
> >>> In Hadoop, we encapsulate the wrap/unwrap behind special stream
> >>> subclasses
> >>> that wrap over the underlying "raw" stream.
> >>>
> >>>
> >>>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/
> >>> ha
> >>>
> >>>
> doop-common/src/main/java/org/apache/hadoop/security/SaslInputStream.java
> >>>
> >>> <
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project
> >>>
> /hadoop-common/src/main/java/org/apache/hadoop/security/SaslInputStream.j
> >>> ava>
> >>>
> >>>
> >>>
> >>>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/
> >>> ha
> >>>
> >>>
> doop-common/src/main/java/org/apache/hadoop/security/SaslOutputStream.jav
> >>> a
> >>>
> >>> <
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project
> >>>
> /hadoop-common/src/main/java/org/apache/hadoop/security/SaslOutputStream.
> >>> java>
> >>>
> >>>
> >>> That's nice, because the code consuming the streams doesn't need to
> >>> worry
> >>> about repeated wrapping and unwrapping.  However, even if we set up
> >>> something like these classes in ZooKeeper, it appears the ZooKeeper
> >>> codebase isn't structured to make inserting those wrappers easy.
> >>>
> >>> This does look like it would be more invasive than I originally
> >>> described.
> >>>
> >>> --Chris Nauroth
> >>>
> >>>
> >>>
> >>>
> >>>> On 10/9/15, 7:57 AM, "Ivan Kelly"  wrote:
> >>>>
> >>>> I think it's a fairly big change, especially since we'd then have a
> >>> lot of
> >>>> conditional if (sasl) { wrap_bytes } else { dont_wrap }. And then it
> >>>> affects all communication between server and client, which is quite
> >>> risky.
> >>>>
> >>>>> On Fri, Oct 9, 2015 at 4:54 PM Flavio Junqueira 
> wrote:
> >>>>>
> >>>>> Ok, got it, but it sounds like we can 

Re: [VOTE] Apache ZooKeeper release 3.4.7 candidate 0

[Ivan Kelly]

+1

Checked checksums/sig. Ran release-audit/test (passed). Eyeballed
licences, they seem ok. Put it through the CI for our application
(passed).

Good job Raul!

-Ivan

On Tue, Nov 17, 2015 at 11:41 PM, Patrick Hunt  wrote:
> I just noticed that the netty jar includes it's own license/notice w/in the
> jar, so regardless we're good there.
>
> +1 - xsum/sigs verify correctly, license/notice seem right to me, RAT ran
> clean.
>
> Patrick
>
> On Tue, Nov 17, 2015 at 2:30 PM, Flavio Junqueira  wrote:
>
>> We didn't have for all and I believe the one for log4j was already there
>> so we didn't touch it. As I understand it, the license file for log4j
>> doesn't have to be under /lib because it is covered by the same license
>> that zookeeper is and it has a license file in the jar. I don't think it
>> matters if it is there, but to make it uniform, we could remove it for
>> future releases. It shouldn't block this release, though.
>>
>> -Flavio
>>
>>
>> > On 17 Nov 2015, at 22:05, Patrick Hunt  wrote:
>> >
>> > -1 - the license file is missing for netty in the lib directory.
>> >
>> > Patrick
>> >
>> > On Mon, Nov 16, 2015 at 2:06 AM, Edward Ribeiro <
>> edward.ribe...@gmail.com>
>> > wrote:
>> >
>> >> +1 (non binding)
>> >>
>> >> Compiled from sources, ran ant test on ubuntu 15.04, reviewed docs. Set
>> up
>> >> a small ensemble (3 nodes) and ran some zkcli commands and four letter
>> >> words.
>> >>
>> >> Thanks Raul! Very nice work.
>> >>
>> >> Regards,
>> >> Edward
>> >> Em 15/11/2015 22:36, "Michi Mutsuzaki"  escreveu:
>> >>
>> >>> +1 (binding)
>> >>>
>> >>> Thanks Raul!
>> >>>
>> >>> On Sun, Nov 15, 2015 at 4:29 PM, Raúl Gutiérrez Segalés
>> >>>  wrote:
>>  Yup, Michi is right! I signed with zookeeper-3.4.7.tar.gz.asc with the
>>  wrong key. I have no updated that files. Thanks Michi!
>> 
>> 
>>  -rgs
>> 
>>  On 15 November 2015 at 15:59, Michi Mutsuzaki 
>> >>> wrote:
>> 
>> > But that's not the key that's listed here, right?
>> > http://www.apache.org/dist/zookeeper/KEYS
>> >
>> > On Sat, Nov 14, 2015 at 2:43 PM, Flavio Junqueira 
>> >>> wrote:
>> >> It works for me:
>> >>
>> >> $ gpg2 --verify zookeeper-3.4.7.tar.gz.asc
>> >> gpg: Signature made Wed Nov 11 06:47:13 2015 GMT using RSA key ID
>> > 9DED6870
>> >> gpg: Good signature from "Raul Gutierrez Segales <
>> >> r...@itevenworks.net
>>  "
>> >> gpg: WARNING: This key is not certified with a trusted signature!
>> >> gpg:  There is no indication that the signature belongs to
>> >> the
>> > owner.
>> >> Primary key fingerprint: D9FE 7869 EF41 C33C 707B  2FF7 4F0D 80FB
>> >> 9DED
>> > 6870
>> >>
>> >>
>> >>> On 14 Nov 2015, at 22:20, Michi Mutsuzaki 
>> >>> wrote:
>> >>>
>> >>> It looks like all the other files are signed with the right key.
>> >> I'll
>> >>> +1 this once zookeeper-3.4.7.tar.gz.asc gets updated.
>> >>>
>> >>> - verified the signature and md5/sha1 checksums for
>> >>> zookeeper-3.4.7.jar
>> >>> - verified the signatures and md5/sha1 checksums for all the files
>> >>> under dist-maven/
>> >>> - reviewed docs/releasenotes.html
>> >>> - ran ant test on ubuntu 14.04
>> >>>
>> >>>
>> >>> On Sat, Nov 14, 2015 at 1:40 PM, Michi Mutsuzaki <
>> >>> mutsuz...@gmail.com>
>> > wrote:
>>  I'm getting this error verifying the signature:
>> 
>>  % gpg --verify zookeeper-3.4.7.tar.gz.asc
>>  gpg: Signature made Tue 10 Nov 2015 10:47:13 PM PST using RSA key
>> >> ID
>> > 9DED6870
>>  gpg: Can't check signature: public key not found
>> 
>>  Isn't the key ID supposed to be 92BC2F2B?
>>  http://www.apache.org/dist/zookeeper/KEYS
>> 
>> 
>>  On Sat, Nov 14, 2015 at 10:03 AM, Rakesh Radhakrishnan
>>   wrote:
>> > +1(non-binding). I've built and tested with ant jar, ran few
>> >> zkcli
>> > commands, four letter words, tested against Hadoop-2.7.1 small
>> > cluster env.
>> >
>> > On Wed, Nov 11, 2015 at 3:29 PM, Flavio Junqueira <
>> >> f...@apache.org>
>> > wrote:
>> >
>> >> +1 (binding)
>> >>
>> >> I have checked LICENSE, NOTICE, hashes, and signature. I have
>> >> run
>> > tests
>> >> and some simple smoke tests locally.
>> >>
>> >> -Flavio
>> >>
>> >>> On 11 Nov 2015, at 07:17, Raúl Gutiérrez Segalés <
>> > r...@itevenworks.net>
>> >> wrote:
>> >>>
>> >>> This is a bugfix release candidate for 3.4.7. It fixes 79
>> >> issues,
>> >> including
>> >>> issues that affect followers after elections, being unable to
>> > delete a
>> >> node
>> >>> when it has no children, crashes with random input from the
>> >>> network
>> > on
>> >> the
>> >>> QuorumCnxManager, deadlocks during bad network conditions 

Re: Documentation For Writing ZooKeeper Client

[Ivan Kelly]

> I'm trying to find documentation about the ZooKeeper protocol from the
> perspective of a client talking to a server cluster. Is there a suggested
> place to find that kind of documentation, or would I be better off just
> reading other client's source code?
I've never seen nor heard of such a document, so reading the source
code is probably your best option.

-Ivan

Re: [DISCUSS] Using Apache Yetus Pre-Commit for Zookeeper

[Ivan Kelly]

On Sun, Dec 13, 2015 at 12:18 AM, Chris Nauroth
 wrote:
> I'd like to propose that we migrate the ZooKeeper pre-commit process to use 
> Apache Yetus [1].
+1. This looks very cool.

-Ivan

Re: [DISCUSS] Using Apache Yetus Pre-Commit for Zookeeper

[Ivan Kelly]

> - We should move away from ant and start using either maven or gradle. There 
> is a patch (ZOOKEEPER-1078) for maven by phunt, but hasn't been able to 
> finish it. I'm very supportive of making this move.
+1000 for this.

> - I'd rather be using git at this point, so someone would need to the process 
> of requesting infra and such to make the change.
This would also be great.

I think one of the things which scares off developers when they look
at ZK is the circa 2005 tech being used. I know every time I download
a branch, I need to try to figure out where I have ant installed.

-Ivan

Re: [DISCUSS] Remove 3.4.7 from mirrors and the Web site references

[Ivan Kelly]

> The specific issue is described and it is being discussed in ZOOKEEPER-2347. 
> Please share your opinion about the 3.4.7 release in this thread.
It looks like there's already a fix for the issue and very little has
gone into 3.4 since the release, so why no release a 3.4.8?

-Ivan

Re: [DISCUSS] Remove 3.4.7 from mirrors and the Web site references

[Ivan Kelly]

> A slightly separate issue is what to do with 3.4.7. It is a bad release and 
> we don't want folks using it, so I started this discussion to determine what 
> to do with it. I'm suggesting we remove it from mirrors and from the web 
> site, just like we did with 3.3.1 back in the day.

+1 for this.

-Ivan

Re: [DISCUSS] Remove 3.4.7 from mirrors and the Web site references

[Ivan Kelly]

> +1 on removing 3.4.7. If others speak up soon I can go ahead and remove
> 3.4.7 from the website/archive and get started with the RC..
Just to clarify, you'll only remove the archive and not touch the
maven repo? The issue is only in the server, and people are already
using the client via maven so deleting from maven would break them.
Also, the client has some fixes in 3.4.7 which we consider critical.

-Ivan

Re: [DISCUSS] Remove 3.4.7 from mirrors and the Web site references

[Ivan Kelly]

> The same fixes that are in 3.4.7 will be in 3.4.8, and we'll add the one to 
> the deadlock. I was thinking that we should remove it from the maven repo too 
> and ask people to move their dependencies back to 3.4.6. I'm fine if you want 
> to hold removing 3.4.7 from the maven repo until we have 3.4.8 out, but I 
> think we should do it as a way of really preventing folks from using it in 
> the future.

This will break people. We've already moved all versions of our
software to 3.4.7 (due to a client hitting
https://issues.apache.org/jira/browse/ZOOKEEPER-706). If 3.4.7 is
removed from the repo it will force a change for us.

Maven is used primarily to pull zookeeper to be used a a client.
There's no (known) problem with the client, so there's no problem with
people continuing to use it as a client, even when 3.4.8 is out.

-Ivan

Re: [VOTE] Remove release 3.4.7 from mirrors and Web site references

[Ivan Kelly]

+1

On Sun, Dec 20, 2015 at 7:21 PM, Chris Nauroth  wrote:
> +1 non-binding (assuming this is like a "Product Release" vote, where only
> PMC votes are binding)
>
> --Chris Nauroth
>
>
>
>
> On 12/20/15, 7:29 AM, "Flavio Junqueira"  wrote:
>
>>As per our discussion in the [DISCUSS] thread, I propose we remove
>>release 3.4.7 from Apache mirrors and the references to the release on
>>the Web site due to the serious bug we found in the release. The release
>>will remain available in maven to avoid breaking users that already rely
>>on it as a dependency for client applications. To our knowledge, there
>>isn't any serious issue with the client and it has been requested on the
>>list that we keep 3.4.7 available in maven.
>>
>>The vote ends on Wed Dec. 30 at 15:00 GMT. I'm giving more time for this
>>vote so that folks have a chance to check it out. Please vote within this
>>window.
>>
>>The bylaws do not describe the case of withdrawing a release, only
>>accepting a release. Consequently, I'm proposing that the approval
>>process be the same of accepting a release, which is lazy consensus (see
>>definition in the project bylaws). If there is disagreement about it,
>>then we'll need to run a vote within the PMC before approving/rejecting
>>this vote, but that's independent of this vote thread.
>>
>>Shall we Remove release 3.4.7 from mirrors and Web site references?
>>
>>-Flavio
>>
>>
>

Re: [VOTE] Apache ZooKeeper release 3.4.8 candidate 0

[Ivan Kelly]

+1

- Verified checksums and sig
- Licensing looks good (though there are jars without attribution in
contrib/, but this isn't anything new)
- ant test ran cleanly
- Ran client and server code through our CI. Nothing broke.
- ant releaseaudit ran cleanly

Good work Raul!

-Ivan

On Wed, Feb 17, 2016 at 3:35 AM, Michi Mutsuzaki  wrote:
> +1
>
> - Verified the checksums and the signature.
> - Reviewed the release note.
> - Ran 'ant test' on ubuntu 14.04.
>
> On Tue, Feb 16, 2016 at 5:04 PM, Cameron McKenzie
>  wrote:
>> +1 (non binding)
>>
>> Have built ZK 3.4.8 RC0 from source and tested that ZOOKEEPER-706 is fixed.
>>
>> On Wed, Feb 17, 2016 at 11:59 AM, Jerry He  wrote:
>>
>>> I have built the Zookeeper 3.4.8 RC0 from the source,
>>> updated the HBase 1.2.0RC3 (HBase latest release RC) pom to use the new
>>> zookeeper version,
>>> built HBase and ran HBase mvn test suite twice. All are successful.
>>>
>>> +1
>>>
>>> Jerry
>>>
>>> On Tue, Feb 16, 2016 at 7:17 AM, Flavio P JUNQUEIRA 
>>> wrote:
>>>
>>> > I wanted to add that it is very important that all users test the release
>>> > candidate, not only committers or PMC members.  That's the way to gain
>>> > confidence that the RC is good for everyone.
>>> >
>>> > If you have a chance to check it out, especially if you have an app to
>>> test
>>> > it against, please do it this week. The community appreciates it.
>>> >
>>> > -Flavio
>>> > On 16 Feb 2016 4:20 am, "Raúl Gutiérrez Segalés" 
>>> > wrote:
>>> >
>>> > > Hi,
>>> > >
>>> > > Quick reminder that the vote closes up on the 20th, it would be great
>>> to
>>> > > have more votes and feedback this week. Thanks!
>>> > >
>>> > >
>>> > > -rgs
>>> > >
>>> > > On 5 February 2016 at 20:00, Raúl Gutiérrez Segalés <
>>> r...@itevenworks.net
>>> > >
>>> > > wrote:
>>> > >
>>> > > > This is a bugfix release candidate for 3.4.8. It fixes 9 issues, most
>>> > > > notably a deadlock when shutting down ZooKeeper.
>>> > > >
>>> > > > The full release notes is available at:
>>> > > >
>>> > > >
>>> > > >
>>> > >
>>> >
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310801&version=12326517
>>> > > >
>>> > > > *** Please download, test and vote by February 20th 2016, 23:59
>>> UTC+0.
>>> > > ***
>>> > > >
>>> > > > Source files:
>>> > > > http://people.apache.org/~rgs/zookeeper-3-4-8-rc0/
>>> > > >
>>> > > > Maven staging repo:
>>> > > >
>>> > > >
>>> > >
>>> >
>>> https://repository.apache.org/content/groups/staging/org/apache/zookeeper/zookeeper/3.4.8/
>>> > > >
>>> > > > The tag to be voted upon:
>>> > > > https://svn.apache.org/repos/asf/zookeeper/tags/release-3.4.8-rc0
>>> > > >
>>> > > > ZooKeeper's KEYS file containing PGP keys we use to sign the release:
>>> > > > http://www.apache.org/dist/zookeeper/KEYS
>>> > > >
>>> > > > Should we release this candidate?
>>> > > >
>>> > > > Note that the approval is by lazy majority according to the bylaws
>>> and
>>> > > > only PMC votes are binding.
>>> > > >
>>> > > >
>>> > > > -rgs
>>> > > >
>>> > >
>>> >
>>>

[VOTE] Taking Bookkeeper Subproject to a Top Level Project

[Ivan Kelly]

Hi folks,

I'd like to run a community vote on converting the Bookkeeper
subproject of Apache Zookeeper into an Apache top level project.

To refresh you on the status of the subproject currently.
- We've made 7 releases since becoming a subproject
- We have production use cases in 3 major companies (that we know of,
there may be more)
- We have active committers in 4 major companies

Going top level is an oppurtunity to increase our visibility which
will help attract more committers and usecases.

We will be roughly following the incubator process, though it will be
slightly different as we're a subproject and not in incubator itself.
We still need to work out the details but in any case, a community
vote is the first step.

The vote will be a lazy concensus vote. Zookeeper PMC members have
veto. The vote will be open until Wednesday 22nd October, 18:00 GMT.

Regards
Ivan

Re: [VOTE] Taking Bookkeeper Subproject to a Top Level Project

[Ivan Kelly]

With 9 +1, no -1, 4 PMC +1, the community vote passes.

In accordance with
http://incubator.apache.org/guides/graduation.html#process, the next
step is to prepare a charter for presentation to the board. This will
also need to be voted on by a PMC. In the case of incubator projects
this is the IPMC, so in our case I guess it would be the Zookeeper
PMC. Perhaps @phunt, @tdunning or @mahadev could give guidance here,
given their position on IPMC?

-Ivan

On 17 October 2014 15:26, Ivan Kelly  wrote:
> Hi folks,
>
> I'd like to run a community vote on converting the Bookkeeper
> subproject of Apache Zookeeper into an Apache top level project.
>
> To refresh you on the status of the subproject currently.
> - We've made 7 releases since becoming a subproject
> - We have production use cases in 3 major companies (that we know of,
> there may be more)
> - We have active committers in 4 major companies
>
> Going top level is an oppurtunity to increase our visibility which
> will help attract more committers and usecases.
>
> We will be roughly following the incubator process, though it will be
> slightly different as we're a subproject and not in incubator itself.
> We still need to work out the details but in any case, a community
> vote is the first step.
>
> The vote will be a lazy concensus vote. Zookeeper PMC members have
> veto. The vote will be open until Wednesday 22nd October, 18:00 GMT.
>
> Regards
> Ivan

Configuration service for zookeeper

[Ivan Kelly]

Hey folks,

I've just had an discussion about how zookeeper is harder to configure
than etcd or consul for example.

See: https://coreos.com/docs/cluster-management/setup/cluster-discovery/

As I understand it, they have a central quorum that they use for
recording requests for new node instantiation within a quorum.

I think I discussed the problem this solves on twitter a few months
ago with Flavio, about how to ensure that a node which deletes it's
disk contents can't join a quorum.

It may be a nice GSoC projecet to do something similar for ZK.

-Ivan

Re: Review Request 17352: BOOKKEEPER-654: Bookkeeper client operations are allowed even after its closure, bk#close()

[Ivan Kelly]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17352/#review32830
---


I don't like how this patch deals with the fundamental problem, which is that 
we're not keeping track of our state in our code. PCBC, BookieClient, 
LedgerHandle, BookKeeper, LedgerManager are all state machines. PCBC is 
complex, but most have the states (READY, CLOSING, CLOSED).

Basically, while in the READY state they should accept new user requests. In 
the CLOSING state they should start shutting down, and then when all resources 
have been freed they should move the the CLOSED state.

For example, with PCBC, when in the READY state (which is CONNECTED in the 
code), if it recieves a close request, it should move to the CLOSING state, 
error out all requests and close the channel, when all these ops are complete 
it should move to CLOSED.

Similarly, in READY state, BookieClient should serve requests. A close request 
moves it to CLOSING during which it waits for all PCBC to close before moving 
into CLOSED.

In the patch you did something similar for CleanupLedgerManager, except you 
used a boolean rather than a explicit state field.

It's BookKeeper and LedgerHandle which are problematic here, since the state is 
scattered all over the place. I think if we cleanup BookieClient & PCBC as 
stated above and use the CleanupLedgerManager, we should be able to stop 
catching the runtimeexceptions. Actually the BookieClient changes go a lot of 
the way, but the problem is addEntry which reach into PCBC then reach back out 
around again to call into PCBC.

This is a little rambly, but to summarize, I think the patch deals with the 
problem from the wrong direction in places. We shouldn't be catching 
RejectedExectionExceptions because we should be tracking our state enough not 
to need to.


bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookieWatcher.java
<https://reviews.apache.org/r/17352/#comment61835>

It'd be better to only catch RejectedExecutionException, and only log at 
debug level since it occurs quite often.



bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
<https://reviews.apache.org/r/17352/#comment61836>

Again, i think rejectedexecutionexception would be better. Another thing we 
need to be careful about is if errorOutPendingAdds() would try to kick off 
anything else. For example, if there are recovery adds occurring, where do the 
callbacks go? I don't have a direct answer, because it's hard to follow now, 
since internal callbacks and client callbacks are mixed and never actually 
tracked.



bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java
<https://reviews.apache.org/r/17352/#comment61837>

    this is just replicating the state variable above.


- Ivan Kelly


On Jan. 25, 2014, 7:59 a.m., Sijie Guo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17352/
> ---
> 
> (Updated Jan. 25, 2014, 7:59 a.m.)
> 
> 
> Review request for bookkeeper, fpj and Ivan Kelly.
> 
> 
> Bugs: BOOKKEEPER-654
> https://issues.apache.org/jira/browse/BOOKKEEPER-654
> 
> 
> Repository: bookkeeper-git
> 
> 
> Description
> ---
> 
> the correct close sequence should be:
> 
> 1) close the bookie client to error out all pending bookie requests, and 
> after bookie client is close, all following request would be rejected.
> 2) close the ledger manager which erred out all pending all metadata 
> requests, and after ledger manager is close, all metadata request would be 
> rejected.
> 3) close scheduler.
> 
> attach a patch following this sequence.
> 
> 
> Diffs
> -
> 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BKException.java 
> ace1409 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeper.java 
> a91861c 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeperAdmin.java
>  c5f5233 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookieWatcher.java
>  cfb6022 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java
>  cfb9128 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerFragmentReplicator.java
>  4a4eb49 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
>  bf4bd97 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerOpenOp.java
>  5b8a703 
>   
> bookkeeper-

Re: Review Request 17352: BOOKKEEPER-654: Bookkeeper client operations are allowed even after its closure, bk#close()

[Ivan Kelly]

> On Jan. 27, 2014, 2:43 p.m., Ivan Kelly wrote:
> > I don't like how this patch deals with the fundamental problem, which is 
> > that we're not keeping track of our state in our code. PCBC, BookieClient, 
> > LedgerHandle, BookKeeper, LedgerManager are all state machines. PCBC is 
> > complex, but most have the states (READY, CLOSING, CLOSED).
> > 
> > Basically, while in the READY state they should accept new user requests. 
> > In the CLOSING state they should start shutting down, and then when all 
> > resources have been freed they should move the the CLOSED state.
> > 
> > For example, with PCBC, when in the READY state (which is CONNECTED in the 
> > code), if it recieves a close request, it should move to the CLOSING state, 
> > error out all requests and close the channel, when all these ops are 
> > complete it should move to CLOSED.
> > 
> > Similarly, in READY state, BookieClient should serve requests. A close 
> > request moves it to CLOSING during which it waits for all PCBC to close 
> > before moving into CLOSED.
> > 
> > In the patch you did something similar for CleanupLedgerManager, except you 
> > used a boolean rather than a explicit state field.
> > 
> > It's BookKeeper and LedgerHandle which are problematic here, since the 
> > state is scattered all over the place. I think if we cleanup BookieClient & 
> > PCBC as stated above and use the CleanupLedgerManager, we should be able to 
> > stop catching the runtimeexceptions. Actually the BookieClient changes go a 
> > lot of the way, but the problem is addEntry which reach into PCBC then 
> > reach back out around again to call into PCBC.
> > 
> > This is a little rambly, but to summarize, I think the patch deals with the 
> > problem from the wrong direction in places. We shouldn't be catching 
> > RejectedExectionExceptions because we should be tracking our state enough 
> > not to need to.
> 
> Sijie Guo wrote:
> I don't know if you really understand what the patch is doing or if you 
> read the comments in the jira. it is fixing the fundamental root cause that 
> we shutdown client in wrong sequence, which cause the problematic in ledger 
> handle and bookkeeper. All the operations in ledger handle and bookkeeper are 
> either metadata operations, or data operations, or the callback chained with 
> metadata/data operations. if we keep tracked the state of ledger manager and 
> bookie client (PCBC) and fixing the wrong closing sequence issue, why it is a 
> wrong direction?
> 
> For catching rejected exceptions, I think they are inherited from 
> previous patches (which is indeed dealing with the issue in wrong direction). 
> I think they are good to be kept there, so we would have logs that could help 
> us debugging if we are not dealing with the states correctly.
> 
> As current design, the states in pcbc are channel states, not client 
> state. if a channel is in closed state, it just means the channel is broken 
> and not connected, it doesn't mean that the client is closed. the flag that I 
> added is used for the client state which we could reject any incoming request 
> if the client is to be closed permanently (by calling #close). distinguishing 
> a channel state from client state is good if we want to move support multiple 
> channels to a bookie. 
> 
> >>> I think if we cleanup BookieClient & PCBC as stated above and use the 
> CleanupLedgerManager, we should be able to stop catching the 
> runtimeexceptions.
> 
> again, if you read the patch correctly, isn't the patch to make 
> BookieClient/PCBC handling close correctly and use CleanupLedgerManager? so I 
> don't know what is you real point?
> 
> fpj wrote:
> I like the idea of reasoning about this problem using state machines; 
> however, I'm not convinced that considering four separate state machines is 
> the right thing to do. We can think of an overall state machine for the 
> client (e.g., CONNECTING, READY, CLOSED) and state machines per component.
> 
> Sijie Guo wrote:
> is there any difference between using a flag than a state for the whole 
> client? or stepping back, do we really need so many state for the whole 
> client? 
> 
> will the state in ledger manager and the state in bookie client (pcbc) 
> not be enough to define the state of the whole client?
> 
> fpj wrote:
> It wouldn't be surprising if we can do with one state machine for the 
> whole client. My thinking is that it might be useful for one global client to 
> translate into multiple states for a component. For example, the 

Re: Review Request 17895: BOOKKEEPER-582: protobuf support for bookkeeper

[Ivan Kelly]

> On April 24, 2014, 12:19 p.m., Ivan Kelly wrote:
> > bookkeeper-server/src/main/proto/BookkeeperProtocol.proto, line 81
> > <https://reviews.apache.org/r/17895/diff/2/?file=563033#file563033line81>
> >
> > ledgerId and entryId should be optional in all requests. It may be the 
> > case, that how we specify them changes in the future (like when we flatten 
> > the metadata), so it would be good to leave that possibility open.
> 
> Sijie Guo wrote:
> for all existing reads/writes protocol, the ledgerId and entryId are 
> required. I am not sure how u will change the protocol by flatten the 
> metadata. but I guess that will be pretty new protocol. if so, it would be 
> better to add new request type, so we will not break any existing systems or 
> making it being complicated.
> 
> Ivan Kelly wrote:
> Im not sure how it will change either. What I'm requesting is that the 
> protobuf protocol doesnt lock us in to how we are doing it now forever.

actually for a concrete example, lets say we want to do a read request for a 
whole ledger, from bookie A we request all entries, but it doesnt have every 
3rd entry due to striping. In the case we can request all entrys from the 
ledger with entryid modulo 3 from bookie B. In this case, what would I put in 
the _required_ entry id firld for the read to bookie B?


- Ivan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17895/#review41281
---


On April 24, 2014, 7:43 a.m., Sijie Guo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17895/
> ---
> 
> (Updated April 24, 2014, 7:43 a.m.)
> 
> 
> Review request for bookkeeper and Ivan Kelly.
> 
> 
> Bugs: BOOKKEEPER-582
> https://issues.apache.org/jira/browse/BOOKKEEPER-582
> 
> 
> Repository: bookkeeper-git
> 
> 
> Description
> ---
> 
> - introducing protobuf support for bookkeeper
> - for server: introduce packet processor / EnDecoder for different protocol 
> supports
> - for client: change PCBC to use protobuf to send requests
> - misc changes for protobuf support
> 
> (bookie server is able for backward compatibility) 
> 
> 
> Diffs
> -
> 
>   bookkeeper-server/pom.xml ebc1198 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/IndexInMemPageMgr.java
>  56487aa 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java
>  28e23d6 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingReadOp.java
>  fb36b90 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/processor/RequestProcessor.java
>  241f369 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieProtoEncoding.java
>  1154047 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieRequestHandler.java
>  b922a82 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieRequestProcessor.java
>  8155b22 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookkeeperProtocol.java
>  PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PacketProcessorBase.java
>  PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PacketProcessorBaseV3.java
>  PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java
>  a10f7d5 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessor.java
>  PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessorV3.java
>  PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/WriteEntryProcessor.java
>  PRE-CREATION 
>   
> bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/WriteEntryProcessorV3.java
>  PRE-CREATION 
>   bookkeeper-server/src/main/proto/BookkeeperProtocol.proto PRE-CREATION 
>   bookkeeper-server/src/main/resources/findbugsExclude.xml 97a6156 
>   
> bookkeeper-server/src/test/java/org/apache/bookkeeper/proto/TestProtoVersions.java
>  5fcc445 
>   
> bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/AuditorPeriodicCheckTest.java
>  3f8496f 
>   
> bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieClientTest.java
>  bc05229 
>   
> bookkeeper-server/src/test/java/org/apache/bookkeeper/test/TestBackwardCompat.java
>  8376b46 

Re: intern project idea: decouple zab from zookeeper

[Ivan Kelly]

On Sat, May 31, 2014 at 02:29:34PM -0700, Michi Mutsuzaki wrote:
> I'm hosting an intern this summer. One project I've been thinking
> about is to decouple zab from zookeeper. There are many use cases
> where you need a quorum based replication, but the hierarchical data
> model doesn't work well. A smallish (~1GB?) replicated key-value store
> with millions of entires is one such example. The goal of the project
> is to decouple the consensus algorithm (zab) from the data model
> (zookeeper) more cleanly so that the users can define their own data
> models and use zab to replicate the data.
So you want a replicated log which give you the guarantees of zab. How
would this be very different from Bookkeeper?

-Ivan

Re: intern project idea: decouple zab from zookeeper

[Ivan Kelly]

> 1. BookKeeper is pretty heavyweight, as you need to deploy ZooKeeper
> and bookies. I think there are use cases where you don't need the
> horizontal scalability BookKeeper provides, and you prefer to have a
> light-weight library for replicating state. ZooKeeper is one such
> example :)
I was thinking from the point of view that if you want to provide ZAB
as a library, then the library will have to provide an RPC mechanism
for talking to other members of the quorum, and a means to persist
updates to disk before responding, and _then_ provide a ZAB
implementation somewhere in between. This doesn't seem much lighter
than BK.

I think it's a worthwhile thing to pursue, but I disagree that a
separate project is a better way to doing it. If this is an intern
project, expecting them to reimplement ZAB might be a bit of a large
ask (depending on the internship length and the intern
themselves). An investigation into splitting the user interface layer
of zookeeper and ZAB seems itself to be a nice chunk to work on, and
it has the advantage that even if the changes don't get merged into
trunk, there will be a clearer picture as to why they can't be
split.

> 2. Please correct me if I'm wrong, but BookKeeper is not designed for
> maintaining multiple in-memory replicas. A ledger can't be opened for
> reading if it's already open for writing, and you need to recover by
> restoring from a snapshot and replaying log entries if the writer goes
> down.
You can read from a ledger while it is being written to, but right now
it's polling. Twitter are working on some changes to make it more
notification like to reduce latency between the primary writing and
the secondary reading.

-Ivan

Re: Bookkeeper June board report (draft)

[Ivan Kelly]

4.2.2 was 10 Oct 2013

-Ivan

On Tue, Jun 10, 2014 at 09:32:56PM +0100, Flavio Junqueira wrote:
> There are a couple of things missing:
> 
> - The date of the last release, 4.2.2.
> - We need to say that no committer has been added during this period.
> 
> -Flavio
> 
> On 10 Jun 2014, at 16:47, Ivan Kelly  wrote:
> 
> > How's this.
> > 
> > Bookkeeper is a distributed, reliable, and high performance logging
> > service. The project also includes Hedwig which is a highly scalable
> > Pub/Sub service built on top of ZooKeeper and Bookkeeper with strong
> > durability guarantees.
> > 
> > We continue to work towards the 4.3.0 release, which has slowly been
> > moving towards a release. We project that it will be released in
> > August. This release includes a lot of improvements to the bookie
> > on-disk performance, a new statistics framework, and protobuffer
> > protocol support along with numerous bugfixes.
> > 
> > A release candidate for 4.2.3 is currently being voted on. This
> > release's focus was mainly to remove some operational annoyances along
> > with other bugfixes.
> > 
> > Infrastructure issues:
> > 
> > No issues.
> > 
> > Community:
> > 
> > 51 subscribers to bookkeeper-dev
> > 66 subscribers to bookkeeper-user
> > 
> > 757 issues opened to date, 27 since 2014-03-12
> > 517 issues resolved to date, 21 since 2014-03-12
> > 45 people have reported issues, 5 since 2014-03-12
> > 19 people have contributed patches, 6 since 2014-03-12
> 

Re: 3.5.7

[Ivan Kelly]

We discovered a split brain bug last week, which should probably be
fixed before any new release. ZOOKEEPER-3701.

-Ivan

On Wed, Jan 22, 2020 at 11:13 AM Andor Molnar  wrote:
>
> Makes perfect sense. I just committed both.
>
> Thanks,
> Andor
>
>
>
> > On 2020. Jan 22., at 6:57, Szalay-Bekő Máté  
> > wrote:
> >
> > Hi Andor,
> >
> > Thanks for the efforts, I also think there are some important bugfixes to
> > release on 3.5.
> >
> > I think ZOOKEEPER-3482 would be nice to get in 3.5.7. It is low risk (only
> > some tests + docs update), but it would improve the documentation about
> > SASL. We had a few SASL-SSL related questions on the mailing lists
> > recently, looks like people start to use the SSL feature and it would be
> > nice for them to have the latest docs.
> >
> > I have 2 PRs in it:
> > - https://github.com/apache/zookeeper/pull/1205  is for branch-3.5, and
> > - https://github.com/apache/zookeeper/pull/1204 for the master branch. The
> > later one should also be cherry-picked to branch-3.6.
> >
> > Kind regards,
> > Mate
> >
> > On Tue, Jan 21, 2020 at 10:39 PM Enrico Olivelli 
> > wrote:
> >
> >> Thank you Andor
> >>
> >>
> >> Il mar 21 gen 2020, 22:35 Andor Molnar  ha scritto:
> >>
> >>> Hi dev folks,
> >>>
> >>> We have a few important patches on 3.5, so I'm planning to branch and
> >>> create rc0 for 3.5.7 tomorrow.
> >>>
> >>> Is there anything important in-flight that would be important to wait
> >>> for and include?
> >>>
> >>
> >> No
> >>
> >> Enrico
> >>
> >>
> >>>
> >>> Regards,
> >>> Andor
> >>>
> >>>
> >>>
> >>>
> >>
>

Re: 3.5.7

[Ivan Kelly]

Log disk filling up is a pretty easy position to get yourself into. I
labelled showstopper because split brain of any type should be a
showstopper IMO. Systems base their consistency on the assumption that
zookeeper is consistent. If zookeeper loses consistency, everything is
broken (fortunately recoverably in our case, but could easily have not
been).

-Ivan

On Wed, Jan 22, 2020 at 11:29 AM Andor Molnar  wrote:
>
> Thanks for the heads up Ivan. I need to look into it.
>
> “…on log disk full” - I think it’s rather a matter of robustness, not a 
> showstopper bug.
>
> Andor
>
>
>
>
> > On 2020. Jan 22., at 11:21, Ivan Kelly  wrote:
> >
> > We discovered a split brain bug last week, which should probably be
> > fixed before any new release. ZOOKEEPER-3701.
> >
> > -Ivan
> >
> > On Wed, Jan 22, 2020 at 11:13 AM Andor Molnar  wrote:
> >>
> >> Makes perfect sense. I just committed both.
> >>
> >> Thanks,
> >> Andor
> >>
> >>
> >>
> >>> On 2020. Jan 22., at 6:57, Szalay-Bekő Máté  
> >>> wrote:
> >>>
> >>> Hi Andor,
> >>>
> >>> Thanks for the efforts, I also think there are some important bugfixes to
> >>> release on 3.5.
> >>>
> >>> I think ZOOKEEPER-3482 would be nice to get in 3.5.7. It is low risk (only
> >>> some tests + docs update), but it would improve the documentation about
> >>> SASL. We had a few SASL-SSL related questions on the mailing lists
> >>> recently, looks like people start to use the SSL feature and it would be
> >>> nice for them to have the latest docs.
> >>>
> >>> I have 2 PRs in it:
> >>> - https://github.com/apache/zookeeper/pull/1205  is for branch-3.5, and
> >>> - https://github.com/apache/zookeeper/pull/1204 for the master branch. The
> >>> later one should also be cherry-picked to branch-3.6.
> >>>
> >>> Kind regards,
> >>> Mate
> >>>
> >>> On Tue, Jan 21, 2020 at 10:39 PM Enrico Olivelli 
> >>> wrote:
> >>>
> >>>> Thank you Andor
> >>>>
> >>>>
> >>>> Il mar 21 gen 2020, 22:35 Andor Molnar  ha scritto:
> >>>>
> >>>>> Hi dev folks,
> >>>>>
> >>>>> We have a few important patches on 3.5, so I'm planning to branch and
> >>>>> create rc0 for 3.5.7 tomorrow.
> >>>>>
> >>>>> Is there anything important in-flight that would be important to wait
> >>>>> for and include?
> >>>>>
> >>>>
> >>>> No
> >>>>
> >>>> Enrico
> >>>>
> >>>>
> >>>>>
> >>>>> Regards,
> >>>>> Andor
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
>

Re: 3.5.7

[Ivan Kelly]

> "Log disk filling up is a pretty easy position to get yourself into.” - 
> Agreed, which can be easily avoided with proper monitoring.

We had monitoring set up to alarm at 80%, but for some reason it
didn't trigger. Each node only hit 100% for less than an hour, as the
PurgeTxnLog kicked in then. The processes didn't recover though
because the exception on truncateLog had left them in a bad state.

> Don’t get me wrong (once it’s confirmed) it’s a major bug that has to be 
> fixed. I’m just saying we should not block 3.5.7 with an issue which doesn’t 
> even have a PR yet. Let the community take its time to review and submit the 
> proper fix which will be included in the upcoming maintenance releases.

I disagree, but it's your call as a maintainer. Feel free to drop the
priority on the JIRA.

-Ivan

Re: 3.5.7

[Ivan Kelly]

> Would you have time for a quick fix ?

The measures to avoid the problem are listed at the end of the JIRA
description. I can't submit a PR until I get permission from my
company legal to push to ZK.

-Ivan

[VOTE] BookKeeper 4.2.0 Release Candidate 0

[Ivan Kelly]

This is the first release candidate for Apache BookKeeper, version 4.2.0.

It fixes the following issues:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12320244&styleName=Html&projectId=12311293

*** Please download, test and vote by January 18th 2013, 10:00 GMT.

Note that we are voting upon the source (tag), binaries are provided
for convenience.

Source and binary files:
http://people.apache.org/~ivank/bookkeeper-4.2.0-candidate-0/

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachebookkeeper-127/

The tag to be voted upon:
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/tags/release-4.2.0/

BookKeeper's KEYS file containing PGP keys we use to sign the release:
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/dist/KEYS

Please download the the source package, and follow the README to build
and run a bookkeeper and hedwig service.

Re: [VOTE] BookKeeper 4.2.0 Release Candidate 0

[Ivan Kelly]

As we have 6 +1s, including the necessary 3 PMC +1s, this vote has
passed. As such, I will push this candidate as the release today and
announce tomorrow once the mirrors have been updated.

-Ivan

On Mon, Jan 14, 2013 at 05:35:28PM +, Ivan Kelly wrote:
> This is the first release candidate for Apache BookKeeper, version 4.2.0.
> 
> It fixes the following issues:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12320244&styleName=Html&projectId=12311293
> 
> *** Please download, test and vote by January 18th 2013, 10:00 GMT.
> 
> Note that we are voting upon the source (tag), binaries are provided
> for convenience.
> 
> Source and binary files:
> http://people.apache.org/~ivank/bookkeeper-4.2.0-candidate-0/
> 
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachebookkeeper-127/
> 
> The tag to be voted upon:
> http://svn.apache.org/repos/asf/zookeeper/bookkeeper/tags/release-4.2.0/
> 
> BookKeeper's KEYS file containing PGP keys we use to sign the release:
> http://svn.apache.org/repos/asf/zookeeper/bookkeeper/dist/KEYS
> 
> Please download the the source package, and follow the README to build
> and run a bookkeeper and hedwig service.

Re: [Discussion] Read/Write semantic for BookKeeper and session fencing

[Ivan Kelly]

On Wed, Jan 23, 2013 at 10:22:31AM -0800, Sijie Guo wrote:
> I had explained this case when replying Flavio's comments.
> 
> Use "b) distinguish entries added before fencing at time T and entries
> added after that." rule I mentioned before, session id is the evidence to
> distinguish entries added in different sessions.
> 
> for your example,
> 
> session 1 (client A) : m1
> session 2 (client B) : m2
> 
> this entry boundaries for different sessions is used by client to
> distinguish entries added by different session. so when b1 serving reads
> for m2, it should not respond its 'm2' entry, since it was added in session
> 1 not session 2.
So, the metadata update happens after recovery, rather than before
recovery as it does now for fencing? i.e. it happens when we know what
the LAC is but before we write any new entries of our own? But the
fence occurs at the very start of recovery. 

So, session 1 may not see the fencing (fences quorums go down, so it
replaces them in the metadata). If this is the case, when session 2
tries to update the metadata, it should fails as version changed. 

This seems ok to me. It'll need some extensions to the wire protocol
to carry session id with the messages, but I think this is ok.

-Ivan

Re: [ANNOUNCE] New BookKeeper committer: Uma Maheswara Rao G

[Ivan Kelly]

Congratulations and welcome aboard Uma! 

-Ivan

On Mon, Feb 04, 2013 at 10:24:08AM +0100, Flavio Junqueira wrote:
> 
> The Apache ZooKeeper PMC recently extended committer karma to Uma and he has 
> accepted.  Congratulations and welcome aboard, Uma!
> 
> -Flavio
> 
> 

[VOTE] BookKeeper 4.2.1 Release Candidate 0

[Ivan Kelly]

This is the first release candidate for Apache BookKeeper, version 4.2.1.

This is a bugfix release to address the performance issues caused by 
BOOKKEEPER-569
https://issues.apache.org/jira/browse/BOOKKEEPER-569

The full release notes is available at:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12323840&styleName=Html&projectId=12311293

*** Please download, test and vote by February 26th 2013, 10:00 UTC+0. ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
http://people.apache.org/~ivank/bookkeeper-4.2.1-candidate-0/

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachebookkeeper-271/

The tag to be voted upon:
https://svn.apache.org/repos/asf/zookeeper/bookkeeper/tags/release-4.2.1

BookKeeper's KEYS file containing PGP keys we use to sign the release:
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/dist/KEYS

Please download the the source package, and follow the README to build
and run a bookkeeper and hedwig service.

Re: [VOTE] BookKeeper 4.2.1 Release Candidate 0

[Ivan Kelly]

As with have 5 +1 (Uma, Sijie, Ben, Flavio, mine) including the
required 3 PMC +1, this vote has passed. I will copy the artifacts
into place and make the release once the mirrors have synced.

Thanks for taking a look at this guys.

-Ivan

On Tue, Feb 26, 2013 at 10:25:13PM +0530, Uma Maheswara Rao G wrote:
> +1, I have ran the tests and they are passing with NN. Thanks a lot,
> Ivan for your efforts in making this release.
> 
> Regards,
> Uma
> 
> On Wed, Feb 20, 2013 at 12:22 AM, Ivan Kelly  wrote:
> > This is the first release candidate for Apache BookKeeper, version 4.2.1.
> >
> > This is a bugfix release to address the performance issues caused by 
> > BOOKKEEPER-569
> > https://issues.apache.org/jira/browse/BOOKKEEPER-569
> >
> > The full release notes is available at:
> >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12323840&styleName=Html&projectId=12311293
> >
> > *** Please download, test and vote by February 26th 2013, 10:00 UTC+0. ***
> >
> > Note that we are voting upon the source (tag), binaries are provided for
> > convenience.
> >
> > Source and binary files:
> > http://people.apache.org/~ivank/bookkeeper-4.2.1-candidate-0/
> >
> > Maven staging repo:
> > https://repository.apache.org/content/repositories/orgapachebookkeeper-271/
> >
> > The tag to be voted upon:
> > https://svn.apache.org/repos/asf/zookeeper/bookkeeper/tags/release-4.2.1
> >
> > BookKeeper's KEYS file containing PGP keys we use to sign the release:
> > http://svn.apache.org/repos/asf/zookeeper/bookkeeper/dist/KEYS
> >
> > Please download the the source package, and follow the README to build
> > and run a bookkeeper and hedwig service.

Re: Dumping and loading Zookeeper data

[Ivan Kelly]

On Wed, May 01, 2013 at 11:16:54AM +0900, Dave Cahill wrote:
> I then tried guano [3]. This tool dumped the zookeeper data to a folder
> structure rather than XML, but it did seem to work. If I don't find other
> options, I'll go with this, but since it's outside zookeeper core and was
> last updated 10 months ago, I wasn't sure about it.
If it uses the stock ZK client, it doesn't really matter if its
outside core. The main api will stay compatible, if there are
compatibility issues, you can just bump the version number in the pom.

-Ivan

Next RC for 4.2.2

[Ivan Kelly]

Hi guys,

I'm going to be on vacation for the next 10 days, so the next 4.2.2 rc
isn't going to be released in that time. I plan to get it out as soon
as I'm back at work.

Regards
Ivan

[VOTE] Bookkeeper 4.2.2 release candidate 1

[Ivan Kelly]

This is the second release candidate for Apache Bookkeeper, version
4.2.2.

This is a bugfix release for 4.2.1. There are some minor API
improvements. Notably, it is now possible to check whether a ledger is
closed without opening it, and it is now possible to get a list of
ledgers available in the cluster.

The full release notes is available at:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12324601&styleName=Text&projectId=12311293

*** Please download, test and vote by October 8th 2013, 10:00 UTC+0. ***

Note that we are voting upon the source (tag), binaries are provided for
convenience.

Source and binary files:
http://people.apache.org/~ivank/bookkeeper-4.2.2-candidate-1/

Maven staging repo:
https://repository.apache.org/content/repositories/orgapachebookkeeper-120/

The tag to be voted upon:
https://svn.apache.org/repos/asf/zookeeper/bookkeeper/tags/release-4.2.2

Bookkeeper's KEYS file containing PGP keys we use to sign the release:
http://svn.apache.org/repos/asf/zookeeper/bookkeeper/dist/KEYS

Please download the the source package, and follow the README to build
and run a bookkeeper and hedwig service.

-Ivan

[jira] [Commented] (ZOOKEEPER-3066) Expose on JMX of Followers the id of the current leader

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16517117#comment-16517117
 ] 

Ivan Kelly commented on ZOOKEEPER-3066:
---

It would also be useful to have this available via http.

For bonus points, create a znode with the leader when it changes, and clients 
can register to be notified when the leader changes.

> Expose on JMX of Followers the id of the current leader
> ---
>
> Key: ZOOKEEPER-3066
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3066
> Project: ZooKeeper
>  Issue Type: New Feature
>  Components: jmx, leaderElection, quorum
>Affects Versions: 3.5.4, 3.6.0
>Reporter: Enrico Olivelli
>Assignee: Enrico Olivelli
>Priority: Major
>
> It will be useful to add to JMX beans published on Follower Peers to have an 
> information about the current "leader".
> This information is only available using 4 letter words



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] Created: (ZOOKEEPER-958) Flag to turn off autoconsume in hedwig c++ client

[Ivan Kelly (JIRA)]

Flag to turn off autoconsume in hedwig c++ client
-

 Key: ZOOKEEPER-958
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-958
 Project: ZooKeeper
  Issue Type: Bug
  Components: contrib-hedwig
Affects Versions: 3.3.2
Reporter: Ivan Kelly
Assignee: Ivan Kelly
 Fix For: 3.3.3


Currently the hedwig cpp client will automatically send a consume message to 
the server when the calling client indicated that it has received the message. 
If the client wants to queue the messages and not acknowledge them to the 
server immediately, they need to block, which means interfering with any other 
running callbacks. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-958) Flag to turn off autoconsume in hedwig c++ client

[Ivan Kelly (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Kelly updated ZOOKEEPER-958:
-

Attachment: ZOOKEEPER-958.diff

> Flag to turn off autoconsume in hedwig c++ client
> -
>
> Key: ZOOKEEPER-958
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-958
> Project: ZooKeeper
>  Issue Type: Bug
>  Components: contrib-hedwig
>Affects Versions: 3.3.2
>    Reporter: Ivan Kelly
>Assignee: Ivan Kelly
> Fix For: 3.3.3
>
> Attachments: ZOOKEEPER-958.diff
>
>
> Currently the hedwig cpp client will automatically send a consume message to 
> the server when the calling client indicated that it has received the 
> message. If the client wants to queue the messages and not acknowledge them 
> to the server immediately, they need to block, which means interfering with 
> any other running callbacks. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-704) GSoC 2010: Read-Only Mode

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987033#action_12987033
 ] 

Ivan Kelly commented on ZOOKEEPER-704:
--

Whats the status with this JIRA? Has it been put on a backburner or is someone 
planning to come back to it fairly soon?

> GSoC 2010: Read-Only Mode
> -
>
> Key: ZOOKEEPER-704
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-704
> Project: ZooKeeper
>  Issue Type: Wish
>Reporter: Henry Robinson
>Assignee: Sergey Doroshenko
>
> Read-only mode
> Possible Mentor
> Henry Robinson (henry at apache dot org)
> Requirements
> Java and TCP/IP networking
> Description
> When a ZooKeeper server loses contact with over half of the other servers in 
> an ensemble ('loses a quorum'), it stops responding to client requests 
> because it cannot guarantee that writes will get processed correctly. For 
> some applications, it would be beneficial if a server still responded to read 
> requests when the quorum is lost, but caused an error condition when a write 
> request was attempted.
> This project would implement a 'read-only' mode for ZooKeeper servers (maybe 
> only for Observers) that allowed read requests to be served as long as the 
> client can contact a server.
> This is a great project for getting really hands-on with the internals of 
> ZooKeeper - you must be comfortable with Java and networking otherwise you'll 
> have a hard time coming up to speed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-704) GSoC 2010: Read-Only Mode

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987670#action_12987670
 ] 

Ivan Kelly commented on ZOOKEEPER-704:
--

Cool. I came across a usecase for this yesterday and was going to pick it up if 
it wasn't being worked on. 

Looks like you have it in hand though :). 

> GSoC 2010: Read-Only Mode
> -
>
> Key: ZOOKEEPER-704
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-704
> Project: ZooKeeper
>  Issue Type: Wish
>Reporter: Henry Robinson
>Assignee: Sergey Doroshenko
>
> Read-only mode
> Possible Mentor
> Henry Robinson (henry at apache dot org)
> Requirements
> Java and TCP/IP networking
> Description
> When a ZooKeeper server loses contact with over half of the other servers in 
> an ensemble ('loses a quorum'), it stops responding to client requests 
> because it cannot guarantee that writes will get processed correctly. For 
> some applications, it would be beneficial if a server still responded to read 
> requests when the quorum is lost, but caused an error condition when a write 
> request was attempted.
> This project would implement a 'read-only' mode for ZooKeeper servers (maybe 
> only for Observers) that allowed read requests to be served as long as the 
> client can contact a server.
> This is a great project for getting really hands-on with the internals of 
> ZooKeeper - you must be comfortable with Java and networking otherwise you'll 
> have a hard time coming up to speed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (ZOOKEEPER-1016) TeaKeeper: Hot standby support using bookkeeper

[Ivan Kelly (JIRA)]

TeaKeeper: Hot standby support using bookkeeper
---

 Key: ZOOKEEPER-1016
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1016
 Project: ZooKeeper
  Issue Type: Improvement
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Priority: Minor
 Attachments: tledger.pdf

Currently Bookkeeper provides functionality for cold backups. If the entity 
logging to bookkeeper fails, its replacement must recover the ledgers which had 
been used for backup before becoming available. This is acceptable in some 
cases, such as HBase Wals where a small delay in recovery only results in a 
small percentage of data being unavailable. 

However, systems such as the HDFS namenode, this delay can be unacceptable, 
such as cases where data is being served to customers. Secondary namenodes 
should be ready to go the instant the primary goes down.

TeaKeeper proposes a wrapper library around Bookkeeper providing T-Junction 
like functionality for logging. It also provides for primary/secondary election 
and automated hot failover. 

HDFS namenode is primary target of this work.

The attached design doc contains more details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (ZOOKEEPER-1016) TeaKeeper: Hot standby support using bookkeeper

[Ivan Kelly (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Kelly updated ZOOKEEPER-1016:
--

Attachment: tledger.pdf

> TeaKeeper: Hot standby support using bookkeeper
> ---
>
> Key: ZOOKEEPER-1016
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1016
> Project: ZooKeeper
>  Issue Type: Improvement
>    Reporter: Ivan Kelly
>    Assignee: Ivan Kelly
>Priority: Minor
> Attachments: tledger.pdf
>
>
> Currently Bookkeeper provides functionality for cold backups. If the entity 
> logging to bookkeeper fails, its replacement must recover the ledgers which 
> had been used for backup before becoming available. This is acceptable in 
> some cases, such as HBase Wals where a small delay in recovery only results 
> in a small percentage of data being unavailable. 
> However, systems such as the HDFS namenode, this delay can be unacceptable, 
> such as cases where data is being served to customers. Secondary namenodes 
> should be ready to go the instant the primary goes down.
> TeaKeeper proposes a wrapper library around Bookkeeper providing T-Junction 
> like functionality for logging. It also provides for primary/secondary 
> election and automated hot failover. 
> HDFS namenode is primary target of this work.
> The attached design doc contains more details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (ZOOKEEPER-1016) TeaKeeper: Hot standby support using bookkeeper

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006857#comment-13006857
 ] 

Ivan Kelly commented on ZOOKEEPER-1016:
---

They have definite overlap, both may or may not solve the same problem. This 
could be implemented using ZOOKEEPER-1001 as the mechanism for getting the 
replicated logs to the secondaries. This also proposes a mechanism where the 
primary sends the entries directly to the secondaries. The advantage of the 
second approach is that less data will be sent over the network. As I 
understand it, with ZOOKEEPER-1001 each reader has to read from all bookies and 
merge the entries returned. This will result in 100% more data transfer than 
required, as each entry will be read from at least 2 bookies. With the solution 
above, each entry is only sent to the secondary once. 

> TeaKeeper: Hot standby support using bookkeeper
> ---
>
> Key: ZOOKEEPER-1016
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1016
> Project: ZooKeeper
>  Issue Type: Improvement
>Reporter: Ivan Kelly
>    Assignee: Ivan Kelly
>Priority: Minor
> Attachments: tledger.pdf
>
>
> Currently Bookkeeper provides functionality for cold backups. If the entity 
> logging to bookkeeper fails, its replacement must recover the ledgers which 
> had been used for backup before becoming available. This is acceptable in 
> some cases, such as HBase Wals where a small delay in recovery only results 
> in a small percentage of data being unavailable. 
> However, systems such as the HDFS namenode, this delay can be unacceptable, 
> such as cases where data is being served to customers. Secondary namenodes 
> should be ready to go the instant the primary goes down.
> TeaKeeper proposes a wrapper library around Bookkeeper providing T-Junction 
> like functionality for logging. It also provides for primary/secondary 
> election and automated hot failover. 
> HDFS namenode is primary target of this work.
> The attached design doc contains more details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (ZOOKEEPER-1016) TeaKeeper: Hot standby support using bookkeeper

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007605#comment-13007605
 ] 

Ivan Kelly commented on ZOOKEEPER-1016:
---

I've try to summarise offline discussion from Tuesday. Please let me know if my 
understanding was off the mark.

Ben asked how TeaKeeper will differ from Hedwig. The main difference is that 
TeaKeeper will be a library colocated with the service writing to it. There'll 
be no extra server running the service, not even a daemon. Only one service 
will write to teakeeper at a time. As such, TeaKeeper is a subset of the 
functionally of Hedwig, aimed at the specific problem of hot standby.

Mahadev raised concern that TeaKeeper would need knowledge of the service to be 
effective. For example, in the namenode case, teakeeper would have to know that 
only the primary can make progress, and so only the primary could log. The 
design addresses this by stipulating that only primary can indeed log. 
TeaKeeper manages who is primary and who are the secondaries through zookeeper. 
There will need to be a notifier for the situation where the primary changes, 
but apart from that, I don't see this as a big issue.


> TeaKeeper: Hot standby support using bookkeeper
> ---
>
> Key: ZOOKEEPER-1016
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1016
> Project: ZooKeeper
>  Issue Type: Improvement
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
>Priority: Minor
> Attachments: tledger.pdf
>
>
> Currently Bookkeeper provides functionality for cold backups. If the entity 
> logging to bookkeeper fails, its replacement must recover the ledgers which 
> had been used for backup before becoming available. This is acceptable in 
> some cases, such as HBase Wals where a small delay in recovery only results 
> in a small percentage of data being unavailable. 
> However, systems such as the HDFS namenode, this delay can be unacceptable, 
> such as cases where data is being served to customers. Secondary namenodes 
> should be ready to go the instant the primary goes down.
> TeaKeeper proposes a wrapper library around Bookkeeper providing T-Junction 
> like functionality for logging. It also provides for primary/secondary 
> election and automated hot failover. 
> HDFS namenode is primary target of this work.
> The attached design doc contains more details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (ZOOKEEPER-1016) TeaKeeper: Hot standby support using bookkeeper

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008396#comment-13008396
 ] 

Ivan Kelly commented on ZOOKEEPER-1016:
---

HDFS-1623 is looking at high availability from the HDFS perspective. May be 
interesting to anyone interested in this JIRA.

> TeaKeeper: Hot standby support using bookkeeper
> ---
>
> Key: ZOOKEEPER-1016
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1016
> Project: ZooKeeper
>  Issue Type: Improvement
>Reporter: Ivan Kelly
>    Assignee: Ivan Kelly
>Priority: Minor
> Attachments: tledger.pdf
>
>
> Currently Bookkeeper provides functionality for cold backups. If the entity 
> logging to bookkeeper fails, its replacement must recover the ledgers which 
> had been used for backup before becoming available. This is acceptable in 
> some cases, such as HBase Wals where a small delay in recovery only results 
> in a small percentage of data being unavailable. 
> However, systems such as the HDFS namenode, this delay can be unacceptable, 
> such as cases where data is being served to customers. Secondary namenodes 
> should be ready to go the instant the primary goes down.
> TeaKeeper proposes a wrapper library around Bookkeeper providing T-Junction 
> like functionality for logging. It also provides for primary/secondary 
> election and automated hot failover. 
> HDFS namenode is primary target of this work.
> The attached design doc contains more details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1016) TeaKeeper: Hot standby support using bookkeeper

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009152#comment-13009152
 ] 

Ivan Kelly commented on ZOOKEEPER-1016:
---

As mahadev said, this does not replace logging to bookkeeper. Every update 
written to the writer is written to the bookies and then only after it has been 
committed to bookkeeper is it sent to the secondaries. How it is "sent" to the 
secondaries is up for discussion. It could be sent directly, a signal could be 
sent to the secondaries that updates are available at the ledger.

Regarding an API for named ledgers, this is the core of what TeaKeeper is, an 
API around ZooKeeper and BookKeeper adding the concept of transactions to 
logging to ledgers. The part about keeping secondaries up to date is just an 
extension of this.

I'll update the doc to address all the comments here in the next day or two 
hopefully.

> TeaKeeper: Hot standby support using bookkeeper
> ---
>
> Key: ZOOKEEPER-1016
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1016
> Project: ZooKeeper
>  Issue Type: Improvement
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
>Priority: Minor
> Attachments: tledger.pdf
>
>
> Currently Bookkeeper provides functionality for cold backups. If the entity 
> logging to bookkeeper fails, its replacement must recover the ledgers which 
> had been used for backup before becoming available. This is acceptable in 
> some cases, such as HBase Wals where a small delay in recovery only results 
> in a small percentage of data being unavailable. 
> However, systems such as the HDFS namenode, this delay can be unacceptable, 
> such as cases where data is being served to customers. Secondary namenodes 
> should be ready to go the instant the primary goes down.
> TeaKeeper proposes a wrapper library around Bookkeeper providing T-Junction 
> like functionality for logging. It also provides for primary/secondary 
> election and automated hot failover. 
> HDFS namenode is primary target of this work.
> The attached design doc contains more details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1001) Read from open ledger

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009158#comment-13009158
 ] 

Ivan Kelly commented on ZOOKEEPER-1001:
---

Is "Unsafe" a good name for this? Once you've written the last committed id, 
the operation is the same as reading from a ledger which had been closed at 
that last committed id. Isn't this safe?


> Read from open ledger
> -
>
> Key: ZOOKEEPER-1001
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1001
> Project: ZooKeeper
>  Issue Type: New Feature
>  Components: contrib-bookkeeper
>Reporter: Flavio Junqueira
> Attachments: zk-1001-design-doc.pdf
>
>
> The BookKeeper client currently does not allow a client to read from an open 
> ledger. That is, if the creator of a ledger is still writing to it (and the 
> ledger is not closed), then an attempt to open the same ledger for reading 
> will execute the code to recover the ledger, assuming that the ledger has not 
> been correctly closed.
> It seems that there are applications that do require the ability to read from 
> a ledger while it is being written to, and the main goal of this jira is to 
> discuss possible implementations of this feature.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1001) Read from open ledger

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009226#comment-13009226
 ] 

Ivan Kelly commented on ZOOKEEPER-1001:
---

Ah, I thought you were going put those checks inside the client. i.e. Something 
like create a subclass of LedgerHandle, which puts a watcher on the 
lastConfirmed value and if the client tries to read past it, throw an 
exception. That does raise the issue of how do you see if new data becomes 
available. Perhaps you could have a watchLastAddConfirmed() call on 
LedgerHandle that triggers a callback when it comes available.

> Read from open ledger
> -
>
> Key: ZOOKEEPER-1001
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1001
> Project: ZooKeeper
>  Issue Type: New Feature
>  Components: contrib-bookkeeper
>Reporter: Flavio Junqueira
> Attachments: zk-1001-design-doc.pdf, zk-1001-design-doc.pdf
>
>
> The BookKeeper client currently does not allow a client to read from an open 
> ledger. That is, if the creator of a ledger is still writing to it (and the 
> ledger is not closed), then an attempt to open the same ledger for reading 
> will execute the code to recover the ledger, assuming that the ledger has not 
> been correctly closed.
> It seems that there are applications that do require the ability to read from 
> a ledger while it is being written to, and the main goal of this jira is to 
> discuss possible implementations of this feature.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (ZOOKEEPER-1016) TeaKeeper: Hot standby support using bookkeeper

[Ivan Kelly (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Kelly updated ZOOKEEPER-1016:
--

Attachment: tledger.pdf

> TeaKeeper: Hot standby support using bookkeeper
> ---
>
> Key: ZOOKEEPER-1016
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1016
> Project: ZooKeeper
>  Issue Type: Improvement
>    Reporter: Ivan Kelly
>    Assignee: Ivan Kelly
>Priority: Minor
> Attachments: tledger.pdf, tledger.pdf
>
>
> Currently Bookkeeper provides functionality for cold backups. If the entity 
> logging to bookkeeper fails, its replacement must recover the ledgers which 
> had been used for backup before becoming available. This is acceptable in 
> some cases, such as HBase Wals where a small delay in recovery only results 
> in a small percentage of data being unavailable. 
> However, systems such as the HDFS namenode, this delay can be unacceptable, 
> such as cases where data is being served to customers. Secondary namenodes 
> should be ready to go the instant the primary goes down.
> TeaKeeper proposes a wrapper library around Bookkeeper providing T-Junction 
> like functionality for logging. It also provides for primary/secondary 
> election and automated hot failover. 
> HDFS namenode is primary target of this work.
> The attached design doc contains more details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1016) TeaKeeper: Hot standby support using bookkeeper

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011158#comment-13011158
 ] 

Ivan Kelly commented on ZOOKEEPER-1016:
---

Added a new design doc. Focus has changes from creating a T to providing 
transaction id and named ledger support. Streaming has been removed for the 
moment. The content and api has changed completely, so its worth taking another 
look even if you've read the previous version.

> TeaKeeper: Hot standby support using bookkeeper
> ---
>
> Key: ZOOKEEPER-1016
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1016
> Project: ZooKeeper
>  Issue Type: Improvement
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
>Priority: Minor
> Attachments: tledger.pdf, tledger.pdf
>
>
> Currently Bookkeeper provides functionality for cold backups. If the entity 
> logging to bookkeeper fails, its replacement must recover the ledgers which 
> had been used for backup before becoming available. This is acceptable in 
> some cases, such as HBase Wals where a small delay in recovery only results 
> in a small percentage of data being unavailable. 
> However, systems such as the HDFS namenode, this delay can be unacceptable, 
> such as cases where data is being served to customers. Secondary namenodes 
> should be ready to go the instant the primary goes down.
> TeaKeeper proposes a wrapper library around Bookkeeper providing T-Junction 
> like functionality for logging. It also provides for primary/secondary 
> election and automated hot failover. 
> HDFS namenode is primary target of this work.
> The attached design doc contains more details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1016) TeaKeeper: Hot standby support using bookkeeper

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011288#comment-13011288
 ] 

Ivan Kelly commented on ZOOKEEPER-1016:
---

It's an interesting idea. It would allow you also to log to HDFS. One issue is 
that FileSystem doesn't have any sort of notification system, so the standby 
would have to poll for changes to a directory to get new logs files. 

HDFS's EditLogFileOutputStream is currently implemented with java.io.File.

> TeaKeeper: Hot standby support using bookkeeper
> ---
>
> Key: ZOOKEEPER-1016
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1016
> Project: ZooKeeper
>  Issue Type: Improvement
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
>Priority: Minor
> Attachments: tledger.pdf, tledger.pdf
>
>
> Currently Bookkeeper provides functionality for cold backups. If the entity 
> logging to bookkeeper fails, its replacement must recover the ledgers which 
> had been used for backup before becoming available. This is acceptable in 
> some cases, such as HBase Wals where a small delay in recovery only results 
> in a small percentage of data being unavailable. 
> However, systems such as the HDFS namenode, this delay can be unacceptable, 
> such as cases where data is being served to customers. Secondary namenodes 
> should be ready to go the instant the primary goes down.
> TeaKeeper proposes a wrapper library around Bookkeeper providing T-Junction 
> like functionality for logging. It also provides for primary/secondary 
> election and automated hot failover. 
> HDFS namenode is primary target of this work.
> The attached design doc contains more details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1016) TeaKeeper: Hot standby support using bookkeeper

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011307#comment-13011307
 ] 

Ivan Kelly commented on ZOOKEEPER-1016:
---

Actually, HDFS-1742 does propose watchers for FS events, but I don't think it's 
aimed at FileSystem.


> TeaKeeper: Hot standby support using bookkeeper
> ---
>
> Key: ZOOKEEPER-1016
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1016
> Project: ZooKeeper
>  Issue Type: Improvement
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
>Priority: Minor
> Attachments: tledger.pdf, tledger.pdf
>
>
> Currently Bookkeeper provides functionality for cold backups. If the entity 
> logging to bookkeeper fails, its replacement must recover the ledgers which 
> had been used for backup before becoming available. This is acceptable in 
> some cases, such as HBase Wals where a small delay in recovery only results 
> in a small percentage of data being unavailable. 
> However, systems such as the HDFS namenode, this delay can be unacceptable, 
> such as cases where data is being served to customers. Secondary namenodes 
> should be ready to go the instant the primary goes down.
> TeaKeeper proposes a wrapper library around Bookkeeper providing T-Junction 
> like functionality for logging. It also provides for primary/secondary 
> election and automated hot failover. 
> HDFS namenode is primary target of this work.
> The attached design doc contains more details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1016) TeaKeeper: Hot standby support using bookkeeper

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012892#comment-13012892
 ] 

Ivan Kelly commented on ZOOKEEPER-1016:
---

*Pros*

[1] Provide a common interface for developers writing journalling code.
[2] Journals can be inspected using hadoop fs calls.
[3] file:// implementation already exists.
[4] You could also store images in BK.
  
*Cons*

[5] Developer must know that "lock" is a special file. Implementation must also 
know that lock is a special file. This is putting application logic into the 
FileSystem logic. It would be better to have a lock() type call on FileSystem. 
Also, locking in general won't work with all implementations of FileSystem. 

[6] No notifications of newly available ledgers. I know the HDFS guys have said 
this isn't important to them, but we want to use this on another project also.

[7] NN is not currently using FileSystem to write it's file:// based files. 
This is the biggest issue here. NN uses StorageDirectories and a Storage 
implementation. The Storage interface is shared by namenode and datanode. You 
could replace this with FileSystem, but you would need to add locking and 
VERSION management to the API. Also, it's an enormous amount of work. 

Otherwise you could just use FileSystem in namenode, wrapping it around 
StorageDirectory. You still need StorageDirectory for managing VERSIONS files 
and such. Again though, this is a massive amount of work. FSImage and FSEditLog 
would need to be gutted. 

It would be simpler to create a BK/ZK implementation of StorageDirectory, but 
this would be HDFS specific, so doesn't belong in BookKeeper.

[8] Someone will undoubtedly start logging to hdfs:// for extra super high 
availability. 

[9] Creates a dependency from bk to hadoop.

*Conclusion*

Given the cons, especially [7], I dont think the effort required for this 
solution is justified by the payoff. The changes to namenode would be too 
fundamental, getting a patch of that size committed would be a nightmare. 

-1 from me.


> TeaKeeper: Hot standby support using bookkeeper
> ---
>
> Key: ZOOKEEPER-1016
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1016
> Project: ZooKeeper
>      Issue Type: Improvement
>    Reporter: Ivan Kelly
>Assignee: Ivan Kelly
>Priority: Minor
> Attachments: tledger.pdf, tledger.pdf
>
>
> Currently Bookkeeper provides functionality for cold backups. If the entity 
> logging to bookkeeper fails, its replacement must recover the ledgers which 
> had been used for backup before becoming available. This is acceptable in 
> some cases, such as HBase Wals where a small delay in recovery only results 
> in a small percentage of data being unavailable. 
> However, systems such as the HDFS namenode, this delay can be unacceptable, 
> such as cases where data is being served to customers. Secondary namenodes 
> should be ready to go the instant the primary goes down.
> TeaKeeper proposes a wrapper library around Bookkeeper providing T-Junction 
> like functionality for logging. It also provides for primary/secondary 
> election and automated hot failover. 
> HDFS namenode is primary target of this work.
> The attached design doc contains more details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (ZOOKEEPER-1028) In python bindings, zookeeper.set2() should return a stat dict but instead returns None

[Ivan Kelly (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Kelly updated ZOOKEEPER-1028:
--

Attachment: ZOOKEEPER-1028.diff

> In python bindings, zookeeper.set2() should return a stat dict but instead 
> returns None
> ---
>
> Key: ZOOKEEPER-1028
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1028
> Project: ZooKeeper
>  Issue Type: Bug
>  Components: contrib-bindings
>Affects Versions: 3.3.3
> Environment: All environments.
>Reporter: Chris Medaglia
>Assignee: Chris Medaglia
>Priority: Minor
>  Labels: patch
> Fix For: 3.4.0
>
> Attachments: ZOOKEEPER-1028.diff, ZOOKEEPER-1028.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> There is a small bug in the python bindings, specifically with the 
> zookeeper.set2() call. This method should return a stat dictionary, but 
> actually returns None. The fix is a one-character change to zookeeper.c such 
> that the return value is '&stat' rather than 'stat'.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1028) In python bindings, zookeeper.set2() should return a stat dict but instead returns None

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012908#comment-13012908
 ] 

Ivan Kelly commented on ZOOKEEPER-1028:
---

This fix is wrong. 

Compilation gives a warning about invalid type being passed.

A good fix would be to change "struct Stat *stat = NULL"; to "struct Stat;" and 
pass &stat to zoo_set2. I've attached a patch to this effect. The current 
implementation passes a NULL pointer to zoo_set2, which means the stat never 
gets set.

> In python bindings, zookeeper.set2() should return a stat dict but instead 
> returns None
> ---
>
> Key: ZOOKEEPER-1028
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1028
> Project: ZooKeeper
>  Issue Type: Bug
>  Components: contrib-bindings
>Affects Versions: 3.3.3
> Environment: All environments.
>Reporter: Chris Medaglia
>Assignee: Chris Medaglia
>Priority: Minor
>  Labels: patch
> Fix For: 3.4.0
>
> Attachments: ZOOKEEPER-1028.diff, ZOOKEEPER-1028.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> There is a small bug in the python bindings, specifically with the 
> zookeeper.set2() call. This method should return a stat dictionary, but 
> actually returns None. The fix is a one-character change to zookeeper.c such 
> that the return value is '&stat' rather than 'stat'.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1034) perl bindings should automatically find the zookeeper c-client headers

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012911#comment-13012911
 ] 

Ivan Kelly commented on ZOOKEEPER-1034:
---

Looks good. 

+1 once ZOOKEEPER-1033 is in.


> perl bindings should automatically find the zookeeper c-client headers
> --
>
> Key: ZOOKEEPER-1034
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1034
> Project: ZooKeeper
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 3.3.3
>Reporter: Nicholas Harteau
>Assignee: Nicholas Harteau
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: ZOOKEEPER-1034-trunk.patch
>
>
> Installing Net::ZooKeeper from cpan or the zookeeper distribution tarballs 
> will always fail due to not finding c-client header files.  In conjunction 
> with ZOOKEEPER-1033 update perl bindings to look for c-client header files in 
> INCDIR/zookeeper/
> a.k.a. make installs of Net::ZooKeeper via cpan/cpanm/whatever *just work*, 
> assuming you've already got the zookeeper c client installed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1033) c client should install includes into INCDIR/zookeeper, not INCDIR/c-client-src

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012913#comment-13012913
 ] 

Ivan Kelly commented on ZOOKEEPER-1033:
---

Actual changes are good. However, you've reformatted the ChangeLog with tabs. 
Could you undo this formatting and use spaces for your own changelog entry 
instead.

> c client should install includes into INCDIR/zookeeper, not 
> INCDIR/c-client-src
> ---
>
> Key: ZOOKEEPER-1033
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1033
> Project: ZooKeeper
>  Issue Type: Bug
>  Components: c client
>Affects Versions: 3.3.3
>Reporter: Nicholas Harteau
>Priority: Minor
> Attachments: ZOOKEEPER-1033-trunk.patch
>
>
> header files are installed into foo/include/c-client-src/, which doesn't 
> indicate a relationship with zookeeper and doesn't correspond to 
> foo/lib/libzookeeper*
> header files should be installed into foo/include/zookeeper/ as this is the 
> common practice.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1033) c client should install includes into INCDIR/zookeeper, not INCDIR/c-client-src

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012987#comment-13012987
 ] 

Ivan Kelly commented on ZOOKEEPER-1033:
---

Could you resubmit with everything from the line "@@ -2,3 +8,3 @@" removed 
(inclusive). Ill +1 once you do. 

> c client should install includes into INCDIR/zookeeper, not 
> INCDIR/c-client-src
> ---
>
> Key: ZOOKEEPER-1033
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1033
> Project: ZooKeeper
>  Issue Type: Bug
>  Components: c client
>Affects Versions: 3.3.3
>Reporter: Nicholas Harteau
>Priority: Minor
> Attachments: ZOOKEEPER-1033-trunk.patch
>
>
> header files are installed into foo/include/c-client-src/, which doesn't 
> indicate a relationship with zookeeper and doesn't correspond to 
> foo/lib/libzookeeper*
> header files should be installed into foo/include/zookeeper/ as this is the 
> common practice.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1033) c client should install includes into INCDIR/zookeeper, not INCDIR/c-client-src

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013003#comment-13013003
 ] 

Ivan Kelly commented on ZOOKEEPER-1033:
---

yes, your patch should only contain your changes.

> c client should install includes into INCDIR/zookeeper, not 
> INCDIR/c-client-src
> ---
>
> Key: ZOOKEEPER-1033
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1033
> Project: ZooKeeper
>  Issue Type: Bug
>  Components: c client
>Affects Versions: 3.3.3
>Reporter: Nicholas Harteau
>Priority: Minor
> Attachments: ZOOKEEPER-1033-notabs-trunk.patch
>
>
> header files are installed into foo/include/c-client-src/, which doesn't 
> indicate a relationship with zookeeper and doesn't correspond to 
> foo/lib/libzookeeper*
> header files should be installed into foo/include/zookeeper/ as this is the 
> common practice.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1033) c client should install includes into INCDIR/zookeeper, not INCDIR/c-client-src

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013009#comment-13013009
 ] 

Ivan Kelly commented on ZOOKEEPER-1033:
---

+1

Good work!

> c client should install includes into INCDIR/zookeeper, not 
> INCDIR/c-client-src
> ---
>
> Key: ZOOKEEPER-1033
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1033
> Project: ZooKeeper
>  Issue Type: Bug
>  Components: c client
>Affects Versions: 3.3.3
>Reporter: Nicholas Harteau
>Priority: Minor
> Attachments: ZOOKEEPER-1033-notidy.patch
>
>
> header files are installed into foo/include/c-client-src/, which doesn't 
> indicate a relationship with zookeeper and doesn't correspond to 
> foo/lib/libzookeeper*
> header files should be installed into foo/include/zookeeper/ as this is the 
> common practice.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1038) Move bookkeeper and hedwig code in subversion

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013930#comment-13013930
 ] 

Ivan Kelly commented on ZOOKEEPER-1038:
---

I'm having trouble compiling ZK locally because zk/src/contrib/bookkeeper/ 
still exists. This should be removed.

> Move bookkeeper and hedwig code in subversion
> -
>
> Key: ZOOKEEPER-1038
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1038
> Project: ZooKeeper
>  Issue Type: Sub-task
>  Components: contrib-bookkeeper, contrib-hedwig
>Reporter: Benjamin Reed
>
> need to do an svn move of the hedwig and bookkeeper code to the bookkeeper 
> subversion

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (ZOOKEEPER-1042) Generate zookeeper test jar for maven installation

[Ivan Kelly (JIRA)]

Generate zookeeper test jar for maven installation
--

 Key: ZOOKEEPER-1042
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1042
 Project: ZooKeeper
  Issue Type: Sub-task
Reporter: Ivan Kelly
Assignee: Ivan Kelly


Bookkeeper and hedwig both need access to zookeeper test classes. This JIRA is 
to provide that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (ZOOKEEPER-1042) Generate zookeeper test jar for maven installation

[Ivan Kelly (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Kelly updated ZOOKEEPER-1042:
--

Attachment: ZOOKEEPER-1042.diff

> Generate zookeeper test jar for maven installation
> --
>
> Key: ZOOKEEPER-1042
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1042
> Project: ZooKeeper
>  Issue Type: Sub-task
>  Components: contrib-bookkeeper, contrib-hedwig
>    Reporter: Ivan Kelly
>    Assignee: Ivan Kelly
> Attachments: ZOOKEEPER-1042.diff
>
>
> Bookkeeper and hedwig both need access to zookeeper test classes. This JIRA 
> is to provide that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1042) Generate zookeeper test jar for maven installation

[Ivan Kelly (JIRA)]

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014025#comment-13014025
 ] 

Ivan Kelly commented on ZOOKEEPER-1042:
---

This patch also deletes the straggler build.xml left over from bookkeeper.

> Generate zookeeper test jar for maven installation
> --
>
> Key: ZOOKEEPER-1042
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1042
> Project: ZooKeeper
>  Issue Type: Sub-task
>  Components: contrib-bookkeeper, contrib-hedwig
>    Reporter: Ivan Kelly
>Assignee: Ivan Kelly
> Attachments: ZOOKEEPER-1042.diff
>
>
> Bookkeeper and hedwig both need access to zookeeper test classes. This JIRA 
> is to provide that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira