[jira] [Commented] (BOOKKEEPER-799) Distribution schedule coverage sets don't take gaps in response lists into account when writequorum > ackquorum
[ https://issues.apache.org/jira/browse/BOOKKEEPER-799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14220866#comment-14220866 ] Flavio Junqueira commented on BOOKKEEPER-799: - Good catch! Write quorums are defined in a round-robin fashion so there is no gap in the sequence of ordered server ids for a write quorum, but since an ack quorum can be a subset of a write quorum, an ack quorum might have gaps in the sequence of bookies ids. > Distribution schedule coverage sets don't take gaps in response lists into > account when writequorum > ackquorum > --- > > Key: BOOKKEEPER-799 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-799 > Project: Bookkeeper > Issue Type: Bug >Reporter: Ivan Kelly >Assignee: Ivan Kelly >Priority: Blocker > Fix For: 4.4.0, 4.2.4 > > Attachments: > 0001-BOOKKEEPER-799-Distribution-schedule-coverage-sets-d.patch > > > The algorithm now be used to check if all quorums are being covered when > sending a read lac or fencing message is broken when writeQuorum >= ackQuorum. > The purpose of the algorithm is to tell us when we have heard a response from > enough nodes, that an ack quorum could not possibly have been formed without > at least one of the nodes that we have heard responses from. > The current algorithm works when writeQuorum == ackQuorum, as we consider all > quorums covered if the first |ackQuorum| nodes in the writeQuorum are > covered. However, this doesn't work in the case that it's the middle node in > the quorum that we have heard. > Take the example, e:4, w:3, a:2, and we've heard from node 0, and node 2. In > this case, it is possible for the write quorum, 1,2,3 to get an ack quorum if > 1 and 3 response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-788) Provide release inspector script
[
https://issues.apache.org/jira/browse/BOOKKEEPER-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14193770#comment-14193770
]
Flavio Junqueira commented on BOOKKEEPER-788:
-
I like the idea of the script. I'm also not sure what [~ikelly] means with
integrating with the buildscripts. Is it about QA or maven build?
The build of hedwig-protocol fails for me when running your script against
trunk and it doesn't complete the initialization.
I also noticed that it doesn't spit out an error message, so I had to look at
the logs to see what happened. It is not a big deal, but it would be nice to
spit out success or failure at the end.
A simple performance test is this:
{noformat}
bookkeeper-benchmark/bin/benchmark bookie -zookeeper -host
{noformat}
> Provide release inspector script
>
>
> Key: BOOKKEEPER-788
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-788
> Project: Bookkeeper
> Issue Type: Improvement
>Reporter: Jiannan Wang
>Assignee: Jiannan Wang
>Priority: Trivial
> Fix For: 4.4.0
>
> Attachments: BOOKKEEPER-788.patch
>
>
> For each release, we need to type a sequence of commands to validate release
> candidate tag. This process could be achieved by a single script.
> Here is the behavior of the script in my mind:
>1. Collecting build environment info (Operation System, JDK, Maven)
>2. Run 'mvn install -DskipTests'
>3. Checking code (findbugs, rat).
>4. Running unit test
>5. Start standalone Bookie and Hedwig service
>6. Run simple Bookie test: bookkeeper-server/bin/bookkeeper simpletest
> -ensemble 3 -writeQuorum 3
>7. Run simple Hedwig test: hedwig-server/bin/hedwig console; pubsub
> topic subscriber 10 message
>8. Tar all above logs when error/exit
> In addition, this script could also be provided to user which could collect
> related information to diagnose compile/UT failure issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-634) Provide admin tool to rename bookie identifier in ledger metadata
[ https://issues.apache.org/jira/browse/BOOKKEEPER-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14193768#comment-14193768 ] Flavio Junqueira commented on BOOKKEEPER-634: - [~rakeshr], have you had a chance to consider the last batch of comments [~ikelly] has put up? I can see QA posts, but no response from you. Let's try to wrap up this issue soon. > Provide admin tool to rename bookie identifier in ledger metadata > - > > Key: BOOKKEEPER-634 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-634 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-client, bookkeeper-server >Affects Versions: 4.2.1 >Reporter: Rakesh R >Assignee: Rakesh R > Fix For: 4.3.1 > > Attachments: 0001-BOOKKEEPER-634-initial-draft-version.patch, > 001-BOOKKEEPER-634-rename-bookieid-in-ledger.patch, > 002-BOOKKEEPER-634-rename-bookieid-in-ledger.patch, > BOOKKEEPER-634-rename-bookieid-in-ledger.patch, BOOKKEEPER-634.patch, > BOOKKEEPER-634.patch > > > This JIRA to discuss about admin tool for changing the bookie's IP to > hostname. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-793) Move to java 7
[ https://issues.apache.org/jira/browse/BOOKKEEPER-793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14169604#comment-14169604 ] Flavio Junqueira commented on BOOKKEEPER-793: - +1 > Move to java 7 > -- > > Key: BOOKKEEPER-793 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-793 > Project: Bookkeeper > Issue Type: Bug >Reporter: Ivan Kelly > Fix For: 4.4.0 > > > Oracle ended support for java 6 last march. We should follow their lead. As > an added bonus we can start to use async io apis with 7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-790) Add JNA license in NOTICE files
[ https://issues.apache.org/jira/browse/BOOKKEEPER-790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14162006#comment-14162006 ] Flavio Junqueira commented on BOOKKEEPER-790: - +1 > Add JNA license in NOTICE files > --- > > Key: BOOKKEEPER-790 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-790 > Project: Bookkeeper > Issue Type: Task >Reporter: Sijie Guo >Assignee: Sijie Guo >Priority: Blocker > Fix For: 4.3.0 > > Attachments: BOOKKEEPER-790.patch > > > Add JNA in NOTICE file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-589) Release notes update for non-BC changes
[ https://issues.apache.org/jira/browse/BOOKKEEPER-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159513#comment-14159513 ] Flavio Junqueira commented on BOOKKEEPER-589: - Makes sense, sounds good. > Release notes update for non-BC changes > --- > > Key: BOOKKEEPER-589 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-589 > Project: Bookkeeper > Issue Type: Sub-task > Components: Documentation >Reporter: Ivan Kelly >Assignee: Ivan Kelly >Priority: Blocker > Fix For: 4.3.0 > > > 4.3.0 client will not be able to connect to 4.2.0 servers, due to something > stupid in the protocol version handling. 4.2.0 and lower will blindly reject > any messages which have a protocolVersion higher than that it's current > version, even if the data format has not changed. The release notes should > reflect this. Also, think about how the protocolVersion is used some more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-589) Release notes update for non-BC changes
[ https://issues.apache.org/jira/browse/BOOKKEEPER-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143117#comment-14143117 ] Flavio Junqueira commented on BOOKKEEPER-589: - Understood that we generate the release notes from jira, but unless your view of jira is different from mine, there is no "release note" field per jira, so the release notes will contain only jira number and title. > Release notes update for non-BC changes > --- > > Key: BOOKKEEPER-589 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-589 > Project: Bookkeeper > Issue Type: Sub-task > Components: Documentation >Reporter: Ivan Kelly >Assignee: Ivan Kelly >Priority: Blocker > Fix For: 4.3.0 > > > 4.3.0 client will not be able to connect to 4.2.0 servers, due to something > stupid in the protocol version handling. 4.2.0 and lower will blindly reject > any messages which have a protocolVersion higher than that it's current > version, even if the data format has not changed. The release notes should > reflect this. Also, think about how the protocolVersion is used some more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-589) Release notes update for non-BC changes
[ https://issues.apache.org/jira/browse/BOOKKEEPER-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143086#comment-14143086 ] Flavio Junqueira commented on BOOKKEEPER-589: - I'm a bit confused here. This issue is a sub-task of BOOKKEEPER-575, and BOOKKEEPER-575 is marked for 4.4.0. Do we need this release note on 4.3.0 because of some sub-task of BOOKKEEPER-575 that is already in? I'd really appreciate some update here from [~iv...@yahoo-inc.com]. I also had a look at the jira configuration and compared with the zookeeper one, and I really can't understand why we can't see the release note field. We might need to contact infra and ask. > Release notes update for non-BC changes > --- > > Key: BOOKKEEPER-589 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-589 > Project: Bookkeeper > Issue Type: Sub-task > Components: Documentation >Reporter: Ivan Kelly >Assignee: Ivan Kelly >Priority: Blocker > Fix For: 4.3.0 > > > 4.3.0 client will not be able to connect to 4.2.0 servers, due to something > stupid in the protocol version handling. 4.2.0 and lower will blindly reject > any messages which have a protocolVersion higher than that it's current > version, even if the data format has not changed. The release notes should > reflect this. Also, think about how the protocolVersion is used some more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-589) Release notes update for non-BC changes
[ https://issues.apache.org/jira/browse/BOOKKEEPER-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14139115#comment-14139115 ] Flavio Junqueira commented on BOOKKEEPER-589: - Which jira caused this change? I think we need to update the release notes on that jira. However, I just tried to resolve this jira to check if the release notes box is there (like we have in zookeeper jiras), and it isn't. There is problem some configuration we need to do to get jira to show the release notes text box when we resolve an issue. > Release notes update for non-BC changes > --- > > Key: BOOKKEEPER-589 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-589 > Project: Bookkeeper > Issue Type: Sub-task > Components: Documentation >Reporter: Ivan Kelly >Assignee: Ivan Kelly >Priority: Blocker > Fix For: 4.3.0 > > > 4.3.0 client will not be able to connect to 4.2.0 servers, due to something > stupid in the protocol version handling. 4.2.0 and lower will blindly reject > any messages which have a protocolVersion higher than that it's current > version, even if the data format has not changed. The release notes should > reflect this. Also, think about how the protocolVersion is used some more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-783) Avoid running out of fds in MutlipleThreadReadTest
[
https://issues.apache.org/jira/browse/BOOKKEEPER-783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14137167#comment-14137167
]
Flavio Junqueira commented on BOOKKEEPER-783:
-
[~hustlmsp], I see, it got in with BOOKKEEPER-429. No objection on my end.
> Avoid running out of fds in MutlipleThreadReadTest
> --
>
> Key: BOOKKEEPER-783
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-783
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Labels: test
> Fix For: 4.3.0
>
> Attachments: BOOKKEEPER-783.patch
>
>
> {code}
> org.jboss.netty.channel.ChannelException: Failed to create a selector.
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:100)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:52)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
> at
> org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
> at
> org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:143)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:81)
> at
> org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39)
> at
> org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClientSocketChannelFactory.java:151)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClientSocketChannelFactory.java:116)
> at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:204)
> at
> org.apache.bookkeeper.client.BookKeeperTestClient.(BookKeeperTestClient.java:50)
> at
> org.apache.bookkeeper.test.MultipleThreadReadTest.createClients(MultipleThreadReadTest.java:73)
> at
> org.apache.bookkeeper.test.MultipleThreadReadTest.multiLedgerMultiThreadRead(MultipleThreadReadTest.java:282)
> at
> org.apache.bookkeeper.test.MultipleThreadReadTest.test1Ledger50ThreadsRead(MultipleThreadReadTest.java:326)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
> Caused by: java.io.IOException: Too many open files
> at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
> at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:69)
> at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:52)
> at
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> at java.nio.channels.Selector.open(Selector.java:209)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.open(SelectorUtil.java:63)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:341)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-783) Avoid running out of fds in MutlipleThreadReadTest
[
https://issues.apache.org/jira/browse/BOOKKEEPER-783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131472#comment-14131472
]
Flavio Junqueira commented on BOOKKEEPER-783:
-
Declaring readBkc and removing clients sounds like a bit cheating. This test
seems to simulating a number of concurrent clients performing operations over
BK, and readBkc still enables multiple app threads, but over the same client. I
wonder if it isn't a better approach to simply reduce the value of numThreads
to no more than 20 instead. Would it do the work? Currently we use 20, 50, and
200.
> Avoid running out of fds in MutlipleThreadReadTest
> --
>
> Key: BOOKKEEPER-783
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-783
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Labels: test
> Fix For: 4.3.0
>
> Attachments: BOOKKEEPER-783.patch
>
>
> {code}
> org.jboss.netty.channel.ChannelException: Failed to create a selector.
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:100)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:52)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
> at
> org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
> at
> org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:143)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:81)
> at
> org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39)
> at
> org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClientSocketChannelFactory.java:151)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClientSocketChannelFactory.java:116)
> at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:204)
> at
> org.apache.bookkeeper.client.BookKeeperTestClient.(BookKeeperTestClient.java:50)
> at
> org.apache.bookkeeper.test.MultipleThreadReadTest.createClients(MultipleThreadReadTest.java:73)
> at
> org.apache.bookkeeper.test.MultipleThreadReadTest.multiLedgerMultiThreadRead(MultipleThreadReadTest.java:282)
> at
> org.apache.bookkeeper.test.MultipleThreadReadTest.test1Ledger50ThreadsRead(MultipleThreadReadTest.java:326)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
> Caused by: java.io.IOException: Too many open files
> at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
> at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:69)
> at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:52)
> at
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> at java.nio.channels.Selector.open(Selector.java:209)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.open(SelectorUtil.java:63)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:341)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-775) Improve MultipleThreadReadTest to reduce flakiness
[ https://issues.apache.org/jira/browse/BOOKKEEPER-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130171#comment-14130171 ] Flavio Junqueira commented on BOOKKEEPER-775: - I'm not sure about the timestamps here. My committed comment above says 00:41, while build-718 apparently finished at 21:45. I don't know if the time zone is consistent. We probably need to wait for 719. > Improve MultipleThreadReadTest to reduce flakiness > -- > > Key: BOOKKEEPER-775 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-775 > Project: Bookkeeper > Issue Type: Test > Components: bookkeeper-server >Reporter: Sijie Guo >Assignee: Sijie Guo > Labels: test > Fix For: 4.3.0 > > Attachments: BOOKKEEPER-775.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-775) Improve MultipleThreadReadTest to reduce flakiness
[ https://issues.apache.org/jira/browse/BOOKKEEPER-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129177#comment-14129177 ] Flavio Junqueira commented on BOOKKEEPER-775: - The patch seems good to me, but I really don't know how to read "flakiness" here. I understand that it is failing for you guys, but what is causing the failure and how do we know that it is fixing the problem? I'll get it in because it won't cause any harm and the improvements seem ok, but it is unclear to me whether we have solved the problem or not. > Improve MultipleThreadReadTest to reduce flakiness > -- > > Key: BOOKKEEPER-775 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-775 > Project: Bookkeeper > Issue Type: Test > Components: bookkeeper-server >Reporter: Sijie Guo >Assignee: Sijie Guo > Labels: test > Fix For: 4.3.0 > > Attachments: BOOKKEEPER-775.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-775) Improve MultipleThreadReadTest to reduce flakiness
[ https://issues.apache.org/jira/browse/BOOKKEEPER-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127680#comment-14127680 ] Flavio Junqueira commented on BOOKKEEPER-775: - The patch looks ok, but I'm not sure what the original is with the test case, the description is empty. > Improve MultipleThreadReadTest to reduce flakiness > -- > > Key: BOOKKEEPER-775 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-775 > Project: Bookkeeper > Issue Type: Test > Components: bookkeeper-server >Reporter: Sijie Guo >Assignee: Sijie Guo > Labels: test > Fix For: 4.3.0 > > Attachments: BOOKKEEPER-775.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-773) Provide admin tool to rename bookie identifier in Cookies
[ https://issues.apache.org/jira/browse/BOOKKEEPER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126948#comment-14126948 ] Flavio Junqueira commented on BOOKKEEPER-773: - It really depends on how urgent having this tool is. If it is a blocker, then we should hold the release until this is in. If this isn't a blocker and isn't ready for 4.3.0, why not push it to 4.4.0? > Provide admin tool to rename bookie identifier in Cookies > - > > Key: BOOKKEEPER-773 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-773 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-server >Reporter: Rakesh R >Assignee: Rakesh R > Fix For: 4.3.0 > > Attachments: 001-BOOKKEEPER-773-rename-bookieid-in-cookies.patch, > 002-BOOKKEEPER-773-rename-bookieid-in-cookies.patch, > 003-BOOKKEEPER-773-rename-bookieid-in-cookies.patch, > 004-BOOKKEEPER-773-rename-bookieid.patch > > > The idea of this JIRA to implement a mechanism to efficiently rename the > bookie identifier present in the Cookies. Cookie information will be present > in: > - ledger & journal directories in each Bookie server > - cookies znode in ZooKeeper -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-781) Fix OOM on Hedwig Tests
[
https://issues.apache.org/jira/browse/BOOKKEEPER-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126771#comment-14126771
]
Flavio Junqueira commented on BOOKKEEPER-781:
-
+1, looks good.
> Fix OOM on Hedwig Tests
> ---
>
> Key: BOOKKEEPER-781
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-781
> Project: Bookkeeper
> Issue Type: Improvement
> Components: hedwig-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Labels: test
> Fix For: 4.3.0
>
> Attachments: BOOKKEEPER-781.patch
>
>
> {code}
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:632)
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:97)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
> at
> org.jboss.netty.channel.socket.nio.SocketSendBufferPool$Preallocation.(SocketSendBufferPool.java:155)
> at
> org.jboss.netty.channel.socket.nio.SocketSendBufferPool.(SocketSendBufferPool.java:42)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:84)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.(NioServerSocketPipelineSink.java:66)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.(NioServerSocketChannelFactory.java:138)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.(NioServerSocketChannelFactory.java:109)
> at
> org.apache.bookkeeper.proto.BookieNettyServer.(BookieNettyServer.java:76)
> at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:97)
> at org.apache.bookkeeper.proto.BookieServer.(BookieServer.java:86)
> at
> org.apache.hedwig.server.persistence.BookKeeperTestBase$TestBookieServer.(BookKeeperTestBase.java:86)
> at
> org.apache.hedwig.server.persistence.BookKeeperTestBase.startBookie(BookKeeperTestBase.java:234)
> at
> org.apache.hedwig.server.persistence.BookKeeperTestBase.startUpNewBookieServer(BookKeeperTestBase.java:222)
> at
> org.apache.hedwig.server.persistence.BookKeeperTestBase.setUp(BookKeeperTestBase.java:159)
> at
> org.apache.hedwig.server.HedwigHubTestBase.setUp(HedwigHubTestBase.java:178)
> at
> org.apache.hedwig.server.integration.TestHedwigHub.setUp(TestHedwigHub.java:226)
> at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Meth
[jira] [Commented] (BOOKKEEPER-776) Flaky test BookieRecoveryTest
[ https://issues.apache.org/jira/browse/BOOKKEEPER-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126758#comment-14126758 ] Flavio Junqueira commented on BOOKKEEPER-776: - OK, I'm +1 for this one then. > Flaky test BookieRecoveryTest > - > > Key: BOOKKEEPER-776 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-776 > Project: Bookkeeper > Issue Type: Test > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Ivan Kelly > Labels: test > Fix For: 4.3.0 > > Attachments: 0001-BOOKKEEPER-776-Flaky-test-BookieRecoveryTest.patch, > TEST-org.apache.bookkeeper.client.BookieRecoveryTest.xml, > TEST-org.apache.bookkeeper.client.BookieRecoveryTest.xml.fail1, > TEST-org.apache.bookkeeper.client.BookieRecoveryTest.xml.fail2 > > > testMetadataConflictWithRecovery[0](org.apache.bookkeeper.client.BookieRecoveryTest): > Not fully replicated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-773) Provide admin tool to rename bookie identifier in Cookies
[ https://issues.apache.org/jira/browse/BOOKKEEPER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126148#comment-14126148 ] Flavio Junqueira commented on BOOKKEEPER-773: - I also don't understand why you're failing to validate. I do understand that there are two groups of parameters, but it isn't uncommon that tools/applications have different commands (e.g., svn). The execution of a parameter becomes dependent on the command. bq. Now I also feels to move this JIRA to 4.3.1 4.3.1 is supposed to be a bug fix release, and I believe this is part of a new feature... not ideal. > Provide admin tool to rename bookie identifier in Cookies > - > > Key: BOOKKEEPER-773 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-773 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-server >Reporter: Rakesh R >Assignee: Rakesh R > Fix For: 4.3.0 > > Attachments: 001-BOOKKEEPER-773-rename-bookieid-in-cookies.patch, > 002-BOOKKEEPER-773-rename-bookieid-in-cookies.patch, > 003-BOOKKEEPER-773-rename-bookieid-in-cookies.patch, > 004-BOOKKEEPER-773-rename-bookieid.patch > > > The idea of this JIRA to implement a mechanism to efficiently rename the > bookie identifier present in the Cookies. Cookie information will be present > in: > - ledger & journal directories in each Bookie server > - cookies znode in ZooKeeper -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-776) Flaky test BookieRecoveryTest
[ https://issues.apache.org/jira/browse/BOOKKEEPER-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124882#comment-14124882 ] Flavio Junqueira commented on BOOKKEEPER-776: - The bit that confused me is that in a comment from Ivan above, he says that that the problem was introduced when we upgraded to "4.2.9.Final", but this patch sets the netty version to 3.9.4. The discrepancy confused me. > Flaky test BookieRecoveryTest > - > > Key: BOOKKEEPER-776 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-776 > Project: Bookkeeper > Issue Type: Test > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Ivan Kelly > Labels: test > Fix For: 4.3.0 > > Attachments: 0001-BOOKKEEPER-776-Flaky-test-BookieRecoveryTest.patch, > TEST-org.apache.bookkeeper.client.BookieRecoveryTest.xml, > TEST-org.apache.bookkeeper.client.BookieRecoveryTest.xml.fail1, > TEST-org.apache.bookkeeper.client.BookieRecoveryTest.xml.fail2 > > > testMetadataConflictWithRecovery[0](org.apache.bookkeeper.client.BookieRecoveryTest): > Not fully replicated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-778) Flake in TestTryReadLastConfirmed
[ https://issues.apache.org/jira/browse/BOOKKEEPER-778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124880#comment-14124880 ] Flavio Junqueira commented on BOOKKEEPER-778: - Ok, +1. > Flake in TestTryReadLastConfirmed > - > > Key: BOOKKEEPER-778 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-778 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-client >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Labels: tests > Fix For: 4.3.0 > > Attachments: > 0001-BOOKKEEPER-778-Flake-in-TestTryReadLastConfirmed.patch, > TEST-org.apache.bookkeeper.client.TestTryReadLastConfirmed.xml.fail1, > TEST-org.apache.bookkeeper.client.TestTryReadLastConfirmed.xml.fail2, > TEST-org.apache.bookkeeper.client.TestTryReadLastConfirmed.xml.fail3 > > > Logs attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-780) Findbug issue in trunk
[ https://issues.apache.org/jira/browse/BOOKKEEPER-780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-780: Component/s: bookkeeper-server > Findbug issue in trunk > -- > > Key: BOOKKEEPER-780 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-780 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-server >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 4.3.0 > > Attachments: 0001-BOOKKEEPER-780-Findbug-issue-in-trunk.patch > > > [INFO] --- findbugs-maven-plugin:2.5.2:check (default-cli) @ > bookkeeper-server --- > [INFO] BugInstance size is 1 > [INFO] Error size is 0 > [INFO] Total bugs: 1 > [INFO] Dead store to conn in > org.apache.bookkeeper.proto.WriteEntryProcessorV3$1.writeComplete(int, long, > long, BookieSocketAddress, Object) > ["org.apache.bookkeeper.proto.WriteEntryProcessorV3$1"] At > WriteEntryProcessorV3.java:[lines 68-98] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-778) Flake in TestTryReadLastConfirmed
[ https://issues.apache.org/jira/browse/BOOKKEEPER-778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119855#comment-14119855 ] Flavio Junqueira commented on BOOKKEEPER-778: - The test modified in the patch seems a bit odd to me overall. For example, I don't understand why we are killing bookies in every iteration of the outer for loop. Perhaps the original intention was to open lh in each iteration? Also, looping numEntries times seems arbitrary because all this seems to be checking is that the callback result make sense for some arbitrary number of calls. > Flake in TestTryReadLastConfirmed > - > > Key: BOOKKEEPER-778 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-778 > Project: Bookkeeper > Issue Type: Bug >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 4.3.0 > > Attachments: > 0001-BOOKKEEPER-778-Flake-in-TestTryReadLastConfirmed.patch, > TEST-org.apache.bookkeeper.client.TestTryReadLastConfirmed.xml.fail1, > TEST-org.apache.bookkeeper.client.TestTryReadLastConfirmed.xml.fail2, > TEST-org.apache.bookkeeper.client.TestTryReadLastConfirmed.xml.fail3 > > > Logs attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-777) Flake in LedgerCloseTest
[ https://issues.apache.org/jira/browse/BOOKKEEPER-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119829#comment-14119829 ] Flavio Junqueira commented on BOOKKEEPER-777: - +1, looks good. > Flake in LedgerCloseTest > > > Key: BOOKKEEPER-777 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-777 > Project: Bookkeeper > Issue Type: Bug >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 4.3.0 > > Attachments: 0001-BOOKKEEPER-777-Flake-in-LedgerCloseTest.patch, > TEST-org.apache.bookkeeper.client.LedgerCloseTest.xml, > TEST-org.apache.bookkeeper.client.LedgerCloseTest.xml.fail1, > TEST-org.apache.bookkeeper.client.LedgerCloseTest.xml.fail2 > > > This test fails every so often. Logs attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-776) Flaky test BookieRecoveryTest
[ https://issues.apache.org/jira/browse/BOOKKEEPER-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119827#comment-14119827 ] Flavio Junqueira commented on BOOKKEEPER-776: - It looks good, but I'm not sure about your choice of the netty version. This page: http://netty.io/downloads.html recommends 4.0.23 instead of 3.9.4, although the latter is also stable. Is it related to ZK? > Flaky test BookieRecoveryTest > - > > Key: BOOKKEEPER-776 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-776 > Project: Bookkeeper > Issue Type: Test > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Ivan Kelly > Labels: test > Fix For: 4.3.0 > > Attachments: 0001-BOOKKEEPER-776-Flaky-test-BookieRecoveryTest.patch, > TEST-org.apache.bookkeeper.client.BookieRecoveryTest.xml, > TEST-org.apache.bookkeeper.client.BookieRecoveryTest.xml.fail1, > TEST-org.apache.bookkeeper.client.BookieRecoveryTest.xml.fail2 > > > testMetadataConflictWithRecovery[0](org.apache.bookkeeper.client.BookieRecoveryTest): > Not fully replicated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-773) Provide admin tool to rename bookie identifier in Cookies
[
https://issues.apache.org/jira/browse/BOOKKEEPER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091716#comment-14091716
]
Flavio Junqueira commented on BOOKKEEPER-773:
-
I forgot why we need to to do the two commands separately. I was actually
thinking that you could do something like:
{noformat}
bkrename -ledger
bkrename -cookie -useHostNameAsBookieId
{noformat}
This is cleaner, but it opens up to people doing "bkrename -ledger
-cookie", which we could of course fail if it is really needed. Just a
suggestion...
> Provide admin tool to rename bookie identifier in Cookies
> -
>
> Key: BOOKKEEPER-773
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-773
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-server
>Reporter: Rakesh R
>Assignee: Rakesh R
> Fix For: 4.3.0
>
> Attachments: 001-BOOKKEEPER-773-rename-bookieid-in-cookies.patch,
> 002-BOOKKEEPER-773-rename-bookieid-in-cookies.patch,
> 003-BOOKKEEPER-773-rename-bookieid-in-cookies.patch
>
>
> The idea of this JIRA to implement a mechanism to efficiently rename the
> bookie identifier present in the Cookies. Cookie information will be present
> in:
> - ledger & journal directories in each Bookie server
> - cookies znode in ZooKeeper
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (BOOKKEEPER-773) Provide admin tool to rename bookie identifier in Cookies
[
https://issues.apache.org/jira/browse/BOOKKEEPER-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14086736#comment-14086736
]
Flavio Junqueira commented on BOOKKEEPER-773:
-
I'm curious to know how often one needs to execute such a command. This is a
rare command to use, yes? I also don't like such long names for commands, what
about "bkrename" or "renamebookie"?
The order of execution in the patch:
{noformat}
+// creates new cookie and write to zookeeper
+Cookie newCookie = cookie.copy();
+newCookie.writeToZooKeeper(zk, newConf);
+
+LOG.info("Updating cookie file present in journalDirectory
{}", journalDirectory);
+LOG.info("Updating cookie file present in ledgerDirectories
{}", ledgerDirectories);
+newCookie.writeToDirectory(journalDirectory);
+for (File dir : ledgerDirectories) {
+newCookie.writeToDirectory(dir);
+}
+cookie.deleteFromZooKeeper(zk, conf);
{noformat}
seems to swap 3 and 4 in this sequence:
{noformat}
3. update new cookie into local directories.
4. create new cookie with new zk path with new bookie id
5. delete the old one.
{noformat}
> Provide admin tool to rename bookie identifier in Cookies
> -
>
> Key: BOOKKEEPER-773
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-773
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-server
>Reporter: Rakesh R
>Assignee: Rakesh R
> Fix For: 4.3.0
>
> Attachments: 001-BOOKKEEPER-773-rename-bookieid-in-cookies.patch,
> 002-BOOKKEEPER-773-rename-bookieid-in-cookies.patch,
> 003-BOOKKEEPER-773-rename-bookieid-in-cookies.patch
>
>
> The idea of this JIRA to implement a mechanism to efficiently rename the
> bookie identifier present in the Cookies. Cookie information will be present
> in:
> - ledger & journal directories in each Bookie server
> - cookies znode in ZooKeeper
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (BOOKKEEPER-697) stats collection on bookkeeper server
[
https://issues.apache.org/jira/browse/BOOKKEEPER-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083200#comment-14083200
]
Flavio Junqueira commented on BOOKKEEPER-697:
-
Here is one:
{noformat}
testMetadataConflictWithRecovery[0](org.apache.bookkeeper.client.BookieRecoveryTest):
Not fully replicated
{noformat}
and another:
{noformat}
Failed tests:
test10Ledgers200ThreadsRead(org.apache.bookkeeper.test.MultipleThreadReadTest):
Test failed because we couldn't read entries
{noformat}
> stats collection on bookkeeper server
> -
>
> Key: BOOKKEEPER-697
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-697
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Aniruddha
> Fix For: 4.3.0
>
> Attachments: BOOKKEEPER-697.patch
>
>
> Stats collection on bookkeeper server.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (BOOKKEEPER-697) stats collection on bookkeeper server
[ https://issues.apache.org/jira/browse/BOOKKEEPER-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077517#comment-14077517 ] Flavio Junqueira commented on BOOKKEEPER-697: - The patch looks good, I just have a couple of comments: # I'm not sure what's going on with the use of NullStatsLogger in some constructors. That means that we won't be collecting stats for those classes, no? What am I missing? # I've tried to run the tests a number of times and it fails for me every time, but a different test case. It doesn't like this patch is causing it, so the tests might be a bit unstable with the recent changes. Are you observing the same? > stats collection on bookkeeper server > - > > Key: BOOKKEEPER-697 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-697 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-server >Reporter: Sijie Guo >Assignee: Aniruddha > Fix For: 4.3.0 > > Attachments: BOOKKEEPER-697.patch > > > Stats collection on bookkeeper server. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-690) Add state diagram in document
[ https://issues.apache.org/jira/browse/BOOKKEEPER-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075587#comment-14075587 ] Flavio Junqueira commented on BOOKKEEPER-690: - I noticed that you have two users registered, can you check now? Let me know offline which user you prefer and consider disabling the other one. > Add state diagram in document > - > > Key: BOOKKEEPER-690 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-690 > Project: Bookkeeper > Issue Type: Task > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Rakesh R >Priority: Minor > Fix For: 4.3.0 > > Attachments: BookieClient connection state transitions.docx, Screen > Shot 2014-07-26 at 11.24.47 PM.png, bookie client state diagram.png > > > Add client state diagram for bookkeeper client in document. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-690) Add state diagram in document
[ https://issues.apache.org/jira/browse/BOOKKEEPER-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075505#comment-14075505 ] Flavio Junqueira commented on BOOKKEEPER-690: - I just checked and you have admin rights. I'm not sure why you're ending up on the apache mesos space... > Add state diagram in document > - > > Key: BOOKKEEPER-690 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-690 > Project: Bookkeeper > Issue Type: Task > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Rakesh R >Priority: Minor > Fix For: 4.3.0 > > Attachments: BookieClient connection state transitions.docx, bookie > client state diagram.png > > > Add client state diagram for bookkeeper client in document. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-589) Release notes update for non-BC changes
[ https://issues.apache.org/jira/browse/BOOKKEEPER-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075302#comment-14075302 ] Flavio Junqueira commented on BOOKKEEPER-589: - I think he was referring to the release notes we publish with a release: http://zookeeper.apache.org/bookkeeper/docs/r4.2.3/releaseNotes.html Here I haven't seen us taking advantage of that field, but when we resolve an issue, there is a "release notes" box. I believe the content of that box shows up when we generate the release notes page. That's the thing that comes to mind, but I don't know what [~ikelly] had in mind exactly. > Release notes update for non-BC changes > --- > > Key: BOOKKEEPER-589 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-589 > Project: Bookkeeper > Issue Type: Sub-task >Reporter: Ivan Kelly >Assignee: Ivan Kelly >Priority: Blocker > Fix For: 4.3.0 > > > 4.3.0 client will not be able to connect to 4.2.0 servers, due to something > stupid in the protocol version handling. 4.2.0 and lower will blindly reject > any messages which have a protocolVersion higher than that it's current > version, even if the data format has not changed. The release notes should > reflect this. Also, think about how the protocolVersion is used some more. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-582) Make bookie and client use protobuf for requests (non-wire part)
[ https://issues.apache.org/jira/browse/BOOKKEEPER-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073764#comment-14073764 ] Flavio Junqueira commented on BOOKKEEPER-582: - +1, thanks [~hustlmsp]. Committed revision 1613315. > Make bookie and client use protobuf for requests (non-wire part) > > > Key: BOOKKEEPER-582 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-582 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-client, bookkeeper-server >Reporter: Ivan Kelly >Assignee: Aniruddha > Fix For: 4.3.0 > > Attachments: > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0003-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > BOOKKEEPER-582.diff, BOOKKEEPER-582.v2.diff, BOOKKEEPER-582.v3.diff, > BOOKKEEPER-582.v4.diff > > > Make the client and the bookie use protobufs internally. This is the first > step to using protobufs on the wire, but for the moment, BookieRequestHandler > decodes the old wire protocol into the protobuf messages. Once this is in, > enabling on the wire will be very simple, and the old manual serialization > can be made "legacy" (still supported, but deprecated). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-575) Bookie SSL support
[ https://issues.apache.org/jira/browse/BOOKKEEPER-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-575: Issue Type: New Feature (was: Bug) > Bookie SSL support > -- > > Key: BOOKKEEPER-575 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-575 > Project: Bookkeeper > Issue Type: New Feature >Reporter: Ivan Kelly > Fix For: 4.3.0 > > > Support SSL for Bookies. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-704) reconnectable zookeeper client wrapper
[ https://issues.apache.org/jira/browse/BOOKKEEPER-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066296#comment-14066296 ] Flavio Junqueira commented on BOOKKEEPER-704: - The rationale for [~iv...@yahoo-inc.com] proposal sounds right, but what's the concrete proposal? That we have multiple handles sprinkled across the code? > reconnectable zookeeper client wrapper > -- > > Key: BOOKKEEPER-704 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-704 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-client, bookkeeper-server >Reporter: Sijie Guo >Assignee: Sijie Guo > Fix For: 4.4.0 > > Attachments: BOOKKEEPER-704.diff > > > create a reconnectable zookkeeper client wrapper to handle session expire > event. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-582) Make bookie and client use protobuf for requests (non-wire part)
[ https://issues.apache.org/jira/browse/BOOKKEEPER-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062021#comment-14062021 ] Flavio Junqueira commented on BOOKKEEPER-582: - bq. in zookeeper, EPHEMERAL / EPHEMERAL_SEQUENTIAL / PERSISTENT / PERSISTENT_SEQUENTIAL are used for representing create mode. what do you think about this? does it introduce any troubles in zookeeper for extension or maintenance? I think this has been mentioned before or possibly I thought of that and didn't say it, but it is right that we do it in ZK. The difference is that those aren't error codes and there are really only two options, ephemeral and sequential. Those options have never changed, while I'd expect code error codes to evolve. I still prefer to use enums over booleans, and I hope java incorporates eventually this same flag feature for enums that C# has. bq. the packet-level error isn't as what you described for server errors. Ok, got it. I think calling it packet-level error is a bit confusing, though. Just so that we come to a closure, based on all the comments, do you still feel that this patch should go in unchanged or would you like to change anything? > Make bookie and client use protobuf for requests (non-wire part) > > > Key: BOOKKEEPER-582 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-582 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-client, bookkeeper-server >Reporter: Ivan Kelly >Assignee: Aniruddha > Fix For: 4.3.0 > > Attachments: > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0003-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > BOOKKEEPER-582.diff, BOOKKEEPER-582.v2.diff, BOOKKEEPER-582.v3.diff > > > Make the client and the bookie use protobufs internally. This is the first > step to using protobufs on the wire, but for the moment, BookieRequestHandler > decodes the old wire protocol into the protobuf messages. Once this is in, > enabling on the wire will be very simple, and the old manual serialization > can be made "legacy" (still supported, but deprecated). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-582) Make bookie and client use protobuf for requests (non-wire part)
[ https://issues.apache.org/jira/browse/BOOKKEEPER-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14042815#comment-14042815 ] Flavio Junqueira commented on BOOKKEEPER-582: - I actually like the way the patch is done. I also prefer to use enums rather than booleans. I agree though that it would be nice to send a bit vector of flags rather than a single enum to avoid the combinatorial explosion. We should check how other people have done with protobufs, I'm sure we are not the first to encounter this problem. I did a bit of search, but couldn't find anything. I'm also good with separating "packet-level errors" from "app-level errors". I'm just not sure I agree with the codes in StatusCodes. For example, I don't think "NOLEDGER" is a packet-level error. A packet-level error (or differently stated, a server error as opposed to application error) is one that reflects a problem that didn't allow the request to complete. No ledger means that the request execution has completed, but it turns out that the ledger doesn't exist, so report it. This is different from say having an IO exception that prevents the processing of the request from completing. I think that if we could separate those a bit better, then it would make more sense the separation. About the use of enum for errors, I was checking the code and the valueOf method it produces has a default value, which is null. I haven't actually checked. If the default value propagates up, then we can spot that there is an unknown value possibly because of a newer version. I haven't checked, though, how it works on the decoder side. The last point of "too much required" is a bit obscure to me, I'm not sure what to do with it, so I'm ignoring. If you just want a straight vote, then I'd stick to my previous +1, but my preference is to work a bit on those error codes and perhaps investigate if protobufs has a better way of dealing with enums and flags. It doesn't sound hard to take an enumset and transform it into a bit vector, so maybe protobufs should support it or provide similar functionality. > Make bookie and client use protobuf for requests (non-wire part) > > > Key: BOOKKEEPER-582 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-582 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-client, bookkeeper-server >Reporter: Ivan Kelly >Assignee: Aniruddha > Fix For: 4.3.0 > > Attachments: > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0003-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > BOOKKEEPER-582.diff, BOOKKEEPER-582.v2.diff, BOOKKEEPER-582.v3.diff > > > Make the client and the bookie use protobufs internally. This is the first > step to using protobufs on the wire, but for the moment, BookieRequestHandler > decodes the old wire protocol into the protobuf messages. Once this is in, > enabling on the wire will be very simple, and the old manual serialization > can be made "legacy" (still supported, but deprecated). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (BOOKKEEPER-768) fix typo 'seconds' to milliseconds in benchmark output
[ https://issues.apache.org/jira/browse/BOOKKEEPER-768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039052#comment-14039052 ] Flavio Junqueira edited comment on BOOKKEEPER-768 at 6/21/14 9:19 AM: -- It is only mandatory for committers. It is better if contributors sign an ICLA, but it is not strictly required. Committers need to make sure that there are no IP issues. The fact that a patch has been uploaded to jira is already an indication that the contributor wishes to grant it to ASF. That's how I understand it at least. was (Author: fpj): It is only mandatory for committers. It is better if contributors sign an ICLA, but it is not strictly required. Committers need to make sure that there are no IP issues. The fact that a patch has been uploaded to jira is already an indication that the contributor wishes to grant it to ASF. That's who I understand it at least. > fix typo 'seconds' to milliseconds in benchmark output > -- > > Key: BOOKKEEPER-768 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-768 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-benchmark >Affects Versions: 4.2.2 >Reporter: jialin >Assignee: jialin >Priority: Trivial > Fix For: 4.3.0 > > Attachments: benchmark-seconds-typo.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > fix typo 'seconds' to 'milliseconds' in benchmark output -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-768) fix typo 'seconds' to milliseconds in benchmark output
[ https://issues.apache.org/jira/browse/BOOKKEEPER-768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039052#comment-14039052 ] Flavio Junqueira commented on BOOKKEEPER-768: - It is only mandatory for committers. It is better if contributors sign an ICLA, but it is not strictly required. Committers need to make sure that there are no IP issues. The fact that a patch has been uploaded to jira is already an indication that the contributor wishes to grant it to ASF. That's who I understand it at least. > fix typo 'seconds' to milliseconds in benchmark output > -- > > Key: BOOKKEEPER-768 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-768 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-benchmark >Affects Versions: 4.2.2 >Reporter: jialin >Assignee: jialin >Priority: Trivial > Fix For: 4.3.0 > > Attachments: benchmark-seconds-typo.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > fix typo 'seconds' to 'milliseconds' in benchmark output -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-769) Remove hedwig c++ client from source tree
[ https://issues.apache.org/jira/browse/BOOKKEEPER-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038687#comment-14038687 ] Flavio Junqueira commented on BOOKKEEPER-769: - [~jiannan], you need to be a bit more explicit about how you feel about this. Is your vote a 0-, meaning that you don't have a strong preference although you have a bias, or is it -1, meaning that you really don't agree with this happening? We got feedback on the legal list, which I have forwarded to the dev list. It seems ok to have it as is, but the issue that this part of the code is not maintained still remains. > Remove hedwig c++ client from source tree > - > > Key: BOOKKEEPER-769 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-769 > Project: Bookkeeper > Issue Type: Bug >Reporter: Ivan Kelly > Fix For: 4.3.0, 4.2.3 > > > The hedwig c++ client has not had any real code change since november 2012. > It is not built as part of the main build. It is not tested as part of the > main build. It is effectively unmaintained. > In addition to this, it contains m4 files whose license status is unclear. > I propose we simple remove it from trunk and branch-4.2. The license issue is > currently blocking the release of 4.2.3. [~jiannan], [~farrellee], you were > the last people to submit anything for the client. Are you using in > production? Any objection to its removal from the main code tree? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-770) Please delete old releases from mirroring system
[ https://issues.apache.org/jira/browse/BOOKKEEPER-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037847#comment-14037847 ] Flavio Junqueira commented on BOOKKEEPER-770: - I've moved this issue to the bookkeeper queue. Given that 4.2.2 is our current release, I think it would be ok to have just that one mirrored, Does it sound right to everyone else? > Please delete old releases from mirroring system > > > Key: BOOKKEEPER-770 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-770 > Project: Bookkeeper > Issue Type: Bug > Environment: http://www.apache.org/dist/zookeeper/bookkeeper/ >Reporter: Sebb > > To reduce the load on the ASF mirrors, projects are required to delete old > releases. > Please can you remove all non-current releases? > [Note that older releases are always available from the ASF archive server] > Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (BOOKKEEPER-770) Please delete old releases from mirroring system
[ https://issues.apache.org/jira/browse/BOOKKEEPER-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira moved ZOOKEEPER-1822 to BOOKKEEPER-770: Key: BOOKKEEPER-770 (was: ZOOKEEPER-1822) Project: Bookkeeper (was: ZooKeeper) > Please delete old releases from mirroring system > > > Key: BOOKKEEPER-770 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-770 > Project: Bookkeeper > Issue Type: Bug > Environment: http://www.apache.org/dist/zookeeper/bookkeeper/ >Reporter: Sebb > > To reduce the load on the ASF mirrors, projects are required to delete old > releases. > Please can you remove all non-current releases? > [Note that older releases are always available from the ASF archive server] > Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-769) Remove hedwig c++ client from source tree
[ https://issues.apache.org/jira/browse/BOOKKEEPER-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037315#comment-14037315 ] Flavio Junqueira commented on BOOKKEEPER-769: - [~jiannan], this comment doesn't make a lot of sense to me. Your comment doesn't imply a use case and there has been no investment in that code that is visible to this community. Given that it is in the way because this license issue, Ivan's suggestion might be the best choice. I'd give a few more days to see if there is any feedback here, but without use cases and investment in the code, I'd say +1 here. > Remove hedwig c++ client from source tree > - > > Key: BOOKKEEPER-769 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-769 > Project: Bookkeeper > Issue Type: Bug >Reporter: Ivan Kelly > Fix For: 4.3.0, 4.2.3 > > > The hedwig c++ client has not had any real code change since november 2012. > It is not built as part of the main build. It is not tested as part of the > main build. It is effectively unmaintained. > In addition to this, it contains m4 files whose license status is unclear. > I propose we simple remove it from trunk and branch-4.2. The license issue is > currently blocking the release of 4.2.3. [~jiannan], [~farrellee], you were > the last people to submit anything for the client. Are you using in > production? Any objection to its removal from the main code tree? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-582) Make bookie and client use protobuf for requests (non-wire part)
[ https://issues.apache.org/jira/browse/BOOKKEEPER-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036600#comment-14036600 ] Flavio Junqueira commented on BOOKKEEPER-582: - are you guys tired of this issue? let's try to come to a resolution, this is issue is important, yes? > Make bookie and client use protobuf for requests (non-wire part) > > > Key: BOOKKEEPER-582 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-582 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-client, bookkeeper-server >Reporter: Ivan Kelly >Assignee: Aniruddha > Fix For: 4.3.0 > > Attachments: > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0003-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > BOOKKEEPER-582.diff, BOOKKEEPER-582.v2.diff, BOOKKEEPER-582.v3.diff > > > Make the client and the bookie use protobufs internally. This is the first > step to using protobufs on the wire, but for the moment, BookieRequestHandler > decodes the old wire protocol into the protobuf messages. Once this is in, > enabling on the wire will be very simple, and the old manual serialization > can be made "legacy" (still supported, but deprecated). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-582) Make bookie and client use protobuf for requests (non-wire part)
[ https://issues.apache.org/jira/browse/BOOKKEEPER-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032047#comment-14032047 ] Flavio Junqueira commented on BOOKKEEPER-582: - bq. Use of an enum for flags Ideally, we use something like an enumset here, but I'm not sure if it works well with protobufs. Another option is a list of enum instances. I don't like very much the use of combinations of flags, although I understand Sijie is proposing it for efficient processing. bq. Sending error messages twice I'm assuming that a packet-level error is something like a message not parsing and an app level error is something like an entry doesn't exist in a ledger. If this interpretation is right, then I don't see us sending both types of error codes in the same message. If the message goes through ok, but the operation doesn't process correctly (e.g., entry doesn't exist), then we only need the application-level error. In the case of a packet-level error, then we return only that error and the client produces a return code if needed. Also, the error codes seem to be logically separated, so they should be perhaps in different enums. bq. Use of enum for errors Introducing logic for backward compatibility is tricky. It is not hard to forget something or simply not get it 100% right. Using integers, however, is less clear, less structure, and opens up to errors due to unused values. In general, I prefer the use of enums, but I'm not sure what the protobufs code does if it finds an unknown value. Is there a way of telling it to assign a default value? bq. Too much required Sijie's assessment about ledgerid, entryid seems right to me. I wonder if there is something else here that Ivan is concerned about. > Make bookie and client use protobuf for requests (non-wire part) > > > Key: BOOKKEEPER-582 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-582 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-client, bookkeeper-server >Reporter: Ivan Kelly >Assignee: Aniruddha > Fix For: 4.3.0 > > Attachments: > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0003-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > BOOKKEEPER-582.diff, BOOKKEEPER-582.v2.diff, BOOKKEEPER-582.v3.diff > > > Make the client and the bookie use protobufs internally. This is the first > step to using protobufs on the wire, but for the moment, BookieRequestHandler > decodes the old wire protocol into the protobuf messages. Once this is in, > enabling on the wire will be very simple, and the old manual serialization > can be made "legacy" (still supported, but deprecated). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-582) Make bookie and client use protobuf for requests (non-wire part)
[ https://issues.apache.org/jira/browse/BOOKKEEPER-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019676#comment-14019676 ] Flavio Junqueira commented on BOOKKEEPER-582: - Ok, awesome, thanks for both summaries. Let me go through them and get back to you two asap. > Make bookie and client use protobuf for requests (non-wire part) > > > Key: BOOKKEEPER-582 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-582 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-client, bookkeeper-server >Reporter: Ivan Kelly >Assignee: Aniruddha > Fix For: 4.3.0 > > Attachments: > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0003-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > BOOKKEEPER-582.diff, BOOKKEEPER-582.v2.diff, BOOKKEEPER-582.v3.diff > > > Make the client and the bookie use protobufs internally. This is the first > step to using protobufs on the wire, but for the moment, BookieRequestHandler > decodes the old wire protocol into the protobuf messages. Once this is in, > enabling on the wire will be very simple, and the old manual serialization > can be made "legacy" (still supported, but deprecated). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-767) Allow loopback in tests
[
https://issues.apache.org/jira/browse/BOOKKEEPER-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019360#comment-14019360
]
Flavio Junqueira commented on BOOKKEEPER-767:
-
I have checked the 4.2 patch as well. +1, looks good.
> Allow loopback in tests
> ---
>
> Key: BOOKKEEPER-767
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-767
> Project: Bookkeeper
> Issue Type: Bug
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
> Fix For: 4.3.0, 4.2.3
>
> Attachments: 0002-BOOKKEEPER-767-Allow-loopback-in-tests.trunk.patch,
> 0003-BOOKKEEPER-767-Allow-loopback-in-tests.branch42.patch
>
>
> It's not uncommon to have something like the following in /etc/hosts:
> {quote}
> 127.0.1.1
> {quote}
> This breaks the bookkeeper tests because getAllowLoopback() defaults to
> false. We should set allow loopback to true to allow tests to run out of the
> box on linux machines.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (BOOKKEEPER-767) Allow loopback in tests
[
https://issues.apache.org/jira/browse/BOOKKEEPER-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14018834#comment-14018834
]
Flavio Junqueira commented on BOOKKEEPER-767:
-
Ok, I was able to repro this one in an ubuntu vm. I'm not able to complete the
test, though, it crashes due to oom. Has anyone tried to run this on a vm? In
any case, it seems to be a separate issue, so I'm fine with the trunk patch. I
still need to go through the 4.2 patch.
> Allow loopback in tests
> ---
>
> Key: BOOKKEEPER-767
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-767
> Project: Bookkeeper
> Issue Type: Bug
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
> Fix For: 4.3.0, 4.2.3
>
> Attachments: 0002-BOOKKEEPER-767-Allow-loopback-in-tests.trunk.patch,
> 0003-BOOKKEEPER-767-Allow-loopback-in-tests.branch42.patch
>
>
> It's not uncommon to have something like the following in /etc/hosts:
> {quote}
> 127.0.1.1
> {quote}
> This breaks the bookkeeper tests because getAllowLoopback() defaults to
> false. We should set allow loopback to true to allow tests to run out of the
> box on linux machines.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (BOOKKEEPER-765) bookkeeper script should fall back to java in path if JAVA_HOME is not set
[ https://issues.apache.org/jira/browse/BOOKKEEPER-765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14018679#comment-14018679 ] Flavio Junqueira commented on BOOKKEEPER-765: - I'm wondering if falling back to "which java" could cause any problem. In this machine at least, "which java" points to the java executable in /usr/bin but not the real java home. Just to confirm, is the java executable all we need? > bookkeeper script should fall back to java in path if JAVA_HOME is not set > -- > > Key: BOOKKEEPER-765 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-765 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-server >Reporter: Ivan Kelly >Assignee: Ivan Kelly >Priority: Blocker > Fix For: 4.3.0, 4.2.3 > > Attachments: > 0001-BOOKKEEPER-765-bookkeeper-script-should-fall-b.trunk.patch, > 0001-BOOKKEEPER-765-bookkeeper-script-should-fall-back-to.patch > > > The current behaviour requires JAVA_HOME to be set, which isn't set by > default on all machines. So machines on which bookkeeper ran fine, no longer > work. > Solution is to fallback to java in path is JAVA_HOME not set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-767) Allow loopback in tests
[
https://issues.apache.org/jira/browse/BOOKKEEPER-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017711#comment-14017711
]
Flavio Junqueira commented on BOOKKEEPER-767:
-
yes, I did try on a mac, let me try on ubuntu to see if I can reproduce it
there.
> Allow loopback in tests
> ---
>
> Key: BOOKKEEPER-767
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-767
> Project: Bookkeeper
> Issue Type: Bug
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
> Fix For: 4.3.0, 4.2.3
>
> Attachments: 0002-BOOKKEEPER-767-Allow-loopback-in-tests.trunk.patch,
> 0003-BOOKKEEPER-767-Allow-loopback-in-tests.branch42.patch
>
>
> It's not uncommon to have something like the following in /etc/hosts:
> {quote}
> 127.0.1.1
> {quote}
> This breaks the bookkeeper tests because getAllowLoopback() defaults to
> false. We should set allow loopback to true to allow tests to run out of the
> box on linux machines.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (BOOKKEEPER-767) Allow loopback in tests
[
https://issues.apache.org/jira/browse/BOOKKEEPER-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017691#comment-14017691
]
Flavio Junqueira commented on BOOKKEEPER-767:
-
Patch looks good to me, but I couldn't reproduce the problem. I've added an
127.0.1.1 entry to my /etc/hosts but I couldn't get the tests to fail without
the patch.
> Allow loopback in tests
> ---
>
> Key: BOOKKEEPER-767
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-767
> Project: Bookkeeper
> Issue Type: Bug
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
> Fix For: 4.3.0, 4.2.3
>
> Attachments: 0002-BOOKKEEPER-767-Allow-loopback-in-tests.trunk.patch,
> 0003-BOOKKEEPER-767-Allow-loopback-in-tests.branch42.patch
>
>
> It's not uncommon to have something like the following in /etc/hosts:
> {quote}
> 127.0.1.1
> {quote}
> This breaks the bookkeeper tests because getAllowLoopback() defaults to
> false. We should set allow loopback to true to allow tests to run out of the
> box on linux machines.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (BOOKKEEPER-582) Make bookie and client use protobuf for requests (non-wire part)
[ https://issues.apache.org/jira/browse/BOOKKEEPER-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14017194#comment-14017194 ] Flavio Junqueira commented on BOOKKEEPER-582: - I have made a pass to the comments in the review board. It is dense, so I'm not sure I'm getting the key points. My understanding is that the main source of contention is the use of enums. Ivan's point is that if we update an enum and update bookies before clients, then we risk breaking clients if we are not careful. Being careful may imply code complexity. Is this the main issue or is there something else? [~hustlmsp], could you please summarize your view here? It would help me understand the discussion better and possibly enable meaningful suggestions. > Make bookie and client use protobuf for requests (non-wire part) > > > Key: BOOKKEEPER-582 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-582 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-client, bookkeeper-server >Reporter: Ivan Kelly >Assignee: Aniruddha > Fix For: 4.3.0 > > Attachments: > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0003-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > BOOKKEEPER-582.diff, BOOKKEEPER-582.v2.diff, BOOKKEEPER-582.v3.diff > > > Make the client and the bookie use protobufs internally. This is the first > step to using protobufs on the wire, but for the moment, BookieRequestHandler > decodes the old wire protocol into the protobuf messages. Once this is in, > enabling on the wire will be very simple, and the old manual serialization > can be made "legacy" (still supported, but deprecated). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-582) Make bookie and client use protobuf for requests (non-wire part)
[ https://issues.apache.org/jira/browse/BOOKKEEPER-582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016336#comment-14016336 ] Flavio Junqueira commented on BOOKKEEPER-582: - I'll check it out today, thanks for the pings. > Make bookie and client use protobuf for requests (non-wire part) > > > Key: BOOKKEEPER-582 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-582 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-client, bookkeeper-server >Reporter: Ivan Kelly >Assignee: Aniruddha > Fix For: 4.3.0 > > Attachments: > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0002-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > 0003-BOOKKEEPER-582-Make-bookie-and-client-use-protobuf-f.patch, > BOOKKEEPER-582.diff, BOOKKEEPER-582.v2.diff, BOOKKEEPER-582.v3.diff > > > Make the client and the bookie use protobufs internally. This is the first > step to using protobufs on the wire, but for the moment, BookieRequestHandler > decodes the old wire protocol into the protobuf messages. Once this is in, > enabling on the wire will be very simple, and the old manual serialization > can be made "legacy" (still supported, but deprecated). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-745) Fix for false reports of ledger unreplication during rolling restarts.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014822#comment-14014822 ] Flavio Junqueira commented on BOOKKEEPER-745: - The patch looks good to me. The latest patch does not apply to the 4.2 branch, but I noticed that there is an older patch for 4.2. Did you mean to update it? I also don't quite understand the javadoc warning in the latest QA build. Do you know why it started showing up now? > Fix for false reports of ledger unreplication during rolling restarts. > -- > > Key: BOOKKEEPER-745 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-745 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-auto-recovery >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 4.3.0, 4.2.3 > > Attachments: > 0001-BOOKKEEPER-745-Fix-for-false-reports-of-ledger-unrep.patch, > 0001-Fix-for-false-reports-of-ledger-unreplication-.trunk.patch, > 0001-Fix-for-false-reports-of-ledger-unreplication-.trunk.patch, > 0002-Fix-for-false-reports-of-ledger-unreplication-.trunk.patch, > 0004-Fix-for-false-reports-of-ledger-unreplication-.trunk.patch, > 0006-Fix-for-false-reports-of-ledger-unreplicat.branch4.2.patch > > > The bug occurred because there was no check if rereplication was enabled or > not when the auditor came online. When the auditor comes online it does a > check of which bookies are up and marks the ledgers on missing bookies as > underreplicated. In the false report case, the auditor was running after each > bookie was bounced due to the way leader election for the auditor works. And > since one bookie was down since you're bouncing the server, all ledgers on > that bookie will get marked as underreplicated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-745) Fix for false reports of ledger unreplication during rolling restarts.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013046#comment-14013046 ] Flavio Junqueira commented on BOOKKEEPER-745: - bq. Failure seems to be in part of the code that this change doesn't touch, kicking off test again. The test that failed is org.apache.bookkeeper.client.BookieWriteLedgerTest.testWithMultipleBookieFailuresInLastEnsemble[4]. Do you know if there is a jira for this? We probably want to have a look at it before releasing 4.2.3, no? > Fix for false reports of ledger unreplication during rolling restarts. > -- > > Key: BOOKKEEPER-745 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-745 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-auto-recovery >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 4.3.0, 4.2.3 > > Attachments: > 0001-Fix-for-false-reports-of-ledger-unreplication-.trunk.patch, > 0001-Fix-for-false-reports-of-ledger-unreplication-.trunk.patch, > 0002-Fix-for-false-reports-of-ledger-unreplication-.trunk.patch, > 0004-Fix-for-false-reports-of-ledger-unreplication-.trunk.patch, > 0006-Fix-for-false-reports-of-ledger-unreplicat.branch4.2.patch > > > The bug occurred because there was no check if rereplication was enabled or > not when the auditor came online. When the auditor comes online it does a > check of which bookies are up and marks the ledgers on missing bookies as > underreplicated. In the false report case, the auditor was running after each > bookie was bounced due to the way leader election for the auditor works. And > since one bookie was down since you're bouncing the server, all ledgers on > that bookie will get marked as underreplicated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-758) Add TryReadLastAddConfirmed API
[ https://issues.apache.org/jira/browse/BOOKKEEPER-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011710#comment-14011710 ] Flavio Junqueira commented on BOOKKEEPER-758: - bq. our callbacks thread model isn't well defined Sounds like a potential area of improvement... > Add TryReadLastAddConfirmed API > --- > > Key: BOOKKEEPER-758 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-758 > Project: Bookkeeper > Issue Type: Improvement > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Sijie Guo > Fix For: 4.3.0 > > Attachments: BOOKKEEPER-758.diff, BOOKKEEPER-758.v2.diff > > > add TryReadLastConfirmed to read last confirmed without coverage checking, as > for readers which polls LAC, they just need LAC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-690) Add state diagram in document
[ https://issues.apache.org/jira/browse/BOOKKEEPER-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011707#comment-14011707 ] Flavio Junqueira commented on BOOKKEEPER-690: - Thanks for doing it, Rakesh. As I understand it, this is trying to capture the state machine in PCBC, right? The difficulty here is that a connection does not map so nicely to the behavior the app observes like with ZooKeeper. I guess what I'm trying to say is that this is not very useful from a high level if we can't represent what the state transitions mean to the application. > Add state diagram in document > - > > Key: BOOKKEEPER-690 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-690 > Project: Bookkeeper > Issue Type: Task > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Rakesh R >Priority: Minor > Fix For: 4.3.0 > > Attachments: BookieClient connection state transitions.docx, bookie > client state diagram.png > > > Add client state diagram for bookkeeper client in document. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-760) Don't close PCBC proactively if bookies disappeared from zookeeper znodes.
[
https://issues.apache.org/jira/browse/BOOKKEEPER-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14009090#comment-14009090
]
Flavio Junqueira commented on BOOKKEEPER-760:
-
If we really don't need this part of the code, I'd rather remove it altogether.
Some questions:
bq. // a. the dead bookies will be removed from topology, which will not be
used in new ensemble.
It depends on whether the session of the bookie has already expired, no?
bq. // b. the read sequence will be reordered based on znode availability, so
most of the reads will not be sent to them.
I'm not sure what reordered means here. Do you mean to say rerouted or resent?
{quote}
// c. the close here is just to disconnect the channel, which doesn't remove
the channel
//from from pcbc map. we don't really need to disconnect the channel here,
since if a bookie is
//really down, PCBC will disconnect itself based on netty callback. if we
try to disconnect
//here, it actually introduces side-effects on case d.
{quote}
Ok, this one sounds like a good reason for removing... but what are the
side-effects?
{quote}
// d. closing the client here will affect latency if the bookie is alive but
just being flaky
//on its znode registration due zookeeper session expire.
{quote}
I don't understand how it affects latency.
bq. // e. if we want to permanently remove a bookkeeper client, we should watch
on the cookies' list.
I'm a bit confused here. Do we even want to remove a bookkeeper client?
> Don't close PCBC proactively if bookies disappeared from zookeeper znodes.
> --
>
> Key: BOOKKEEPER-760
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-760
> Project: Bookkeeper
> Issue Type: Improvement
> Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.3.0, 4.2.3
>
> Attachments: BOOKKEEPER-760.diff
>
>
> Don't close PCBC proactively if bookies disappeared from zookeeper znodes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (BOOKKEEPER-762) CookieTest failure
[
https://issues.apache.org/jira/browse/BOOKKEEPER-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14008083#comment-14008083
]
Flavio Junqueira commented on BOOKKEEPER-762:
-
This is a bit odd because now it passes consistently. I guess there has been
some issue with my network stack. I wait a bit longer to see if I observe this
again and otherwise I'll mark it as not an issue.
For the record, the return value of cookie.isBookieHostCreatedFromIp() is false.
> CookieTest failure
> --
>
> Key: BOOKKEEPER-762
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-762
> Project: Bookkeeper
> Issue Type: Bug
>Reporter: Flavio Junqueira
>
> I'm getting this test failure in trunk consistently:
> {noformat}
> ---
> Test set: org.apache.bookkeeper.bookie.CookieTest
> ---
> Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.208 sec
> <<< FAILURE!
> testRestartWithIpAddressAsBookieID(org.apache.bookkeeper.bookie.CookieTest)
> Time elapsed: 0.155 sec <<< FAILURE!
> java.lang.AssertionError: Fails to recognize bookie which was started with
> HostName as ID
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at
> org.apache.bookkeeper.bookie.CookieTest.testRestartWithIpAddressAsBookieID(CookieTest.java:405)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (BOOKKEEPER-762) CookieTest failure
[
https://issues.apache.org/jira/browse/BOOKKEEPER-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14007769#comment-14007769
]
Flavio Junqueira commented on BOOKKEEPER-762:
-
Should the assert be assertFalse instead of assertTrue here:
{noformat}
conf.setUseHostNameAsBookieID(false);
b = new Bookie(conf);
b.start();
assertTrue("Fails to recognize bookie which was started with HostName as ID",
conf.getUseHostNameAsBookieID());
{noformat}
> CookieTest failure
> --
>
> Key: BOOKKEEPER-762
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-762
> Project: Bookkeeper
> Issue Type: Bug
>Reporter: Flavio Junqueira
>
> I'm getting this test failure in trunk consistently:
> {noformat}
> ---
> Test set: org.apache.bookkeeper.bookie.CookieTest
> ---
> Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.208 sec
> <<< FAILURE!
> testRestartWithIpAddressAsBookieID(org.apache.bookkeeper.bookie.CookieTest)
> Time elapsed: 0.155 sec <<< FAILURE!
> java.lang.AssertionError: Fails to recognize bookie which was started with
> HostName as ID
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at
> org.apache.bookkeeper.bookie.CookieTest.testRestartWithIpAddressAsBookieID(CookieTest.java:405)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (BOOKKEEPER-762) CookieTest failure
Flavio Junqueira created BOOKKEEPER-762:
---
Summary: CookieTest failure
Key: BOOKKEEPER-762
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-762
Project: Bookkeeper
Issue Type: Bug
Reporter: Flavio Junqueira
I'm getting this test failure in trunk consistently:
{noformat}
---
Test set: org.apache.bookkeeper.bookie.CookieTest
---
Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.208 sec <<<
FAILURE!
testRestartWithIpAddressAsBookieID(org.apache.bookkeeper.bookie.CookieTest)
Time elapsed: 0.155 sec <<< FAILURE!
java.lang.AssertionError: Fails to recognize bookie which was started with
HostName as ID
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.bookkeeper.bookie.CookieTest.testRestartWithIpAddressAsBookieID(CookieTest.java:405)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (BOOKKEEPER-758) Add TryReadLastAddConfirmed API
[ https://issues.apache.org/jira/browse/BOOKKEEPER-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14007193#comment-14007193 ] Flavio Junqueira commented on BOOKKEEPER-758: - Just a ping... I'm happy to have this one in, waiting for [~ikelly]. > Add TryReadLastAddConfirmed API > --- > > Key: BOOKKEEPER-758 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-758 > Project: Bookkeeper > Issue Type: Improvement > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Sijie Guo > Fix For: 4.3.0 > > Attachments: BOOKKEEPER-758.diff, BOOKKEEPER-758.v2.diff > > > add TryReadLastConfirmed to read last confirmed without coverage checking, as > for readers which polls LAC, they just need LAC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-758) Add TryReadLastAddConfirmed API
[ https://issues.apache.org/jira/browse/BOOKKEEPER-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14005296#comment-14005296 ] Flavio Junqueira commented on BOOKKEEPER-758: - Sounds right, it is not propagated up to the application. > Add TryReadLastAddConfirmed API > --- > > Key: BOOKKEEPER-758 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-758 > Project: Bookkeeper > Issue Type: Improvement > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Sijie Guo > Fix For: 4.3.0 > > Attachments: BOOKKEEPER-758.diff, BOOKKEEPER-758.v2.diff > > > add TryReadLastConfirmed to read last confirmed without coverage checking, as > for readers which polls LAC, they just need LAC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-751) Ensure all the bookkeeper callbacks not run under ledger handle lock
[ https://issues.apache.org/jira/browse/BOOKKEEPER-751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004022#comment-14004022 ] Flavio Junqueira commented on BOOKKEEPER-751: - This test case failed for me, could it be related: Failed tests: test10Ledgers200ThreadsRead(org.apache.bookkeeper.test.MultipleThreadReadTest): Test failed because we couldn't read entries > Ensure all the bookkeeper callbacks not run under ledger handle lock > > > Key: BOOKKEEPER-751 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-751 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Sijie Guo > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-751.diff > > > we are running bookkeeper callbacks under ledger handle lock, which would > possibly introduce deadlock if application call bookkeeper functions in those > callbacks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-758) Add TryReadLastAddConfirmed API
[ https://issues.apache.org/jira/browse/BOOKKEEPER-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003963#comment-14003963 ] Flavio Junqueira commented on BOOKKEEPER-758: - But this code does not benefit at all from the fact that the callback can be called multiple times, no? I don't think it is necessarily a big deal, but it isn't very clear the precise semantics and how applications can benefit from potentially multiple calls to the callback. > Add TryReadLastAddConfirmed API > --- > > Key: BOOKKEEPER-758 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-758 > Project: Bookkeeper > Issue Type: Improvement > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Sijie Guo > Fix For: 4.3.0 > > Attachments: BOOKKEEPER-758.diff, BOOKKEEPER-758.v2.diff > > > add TryReadLastConfirmed to read last confirmed without coverage checking, as > for readers which polls LAC, they just need LAC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-629) Support hostname based ledger metadata to help users to change IP with existing installation
[ https://issues.apache.org/jira/browse/BOOKKEEPER-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994247#comment-13994247 ] Flavio Junqueira commented on BOOKKEEPER-629: - I suggest we don´t commit anything else until the asf mail server stabilizes. [~rakeshr], ok with focusing on the admin tool aka BOOKKEEPER-634. > Support hostname based ledger metadata to help users to change IP with > existing installation > > > Key: BOOKKEEPER-629 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-629 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-auto-recovery, bookkeeper-client, > bookkeeper-server >Affects Versions: 4.2.1 >Reporter: Vinayakumar B >Assignee: Rakesh R > Fix For: 4.3.0 > > Attachments: 1-BOOKKEEPER-629.patch, 10-BOOKKEEPER-629.patch, > 11-BOOKKEEPER-629.patch, 2-BOOKKEEPER-629.patch, 3-BOOKKEEPER-629.patch, > 4-BOOKKEEPER-629.patch, 5-BOOKKEEPER-629.patch, 6-BOOKKEEPER-629.patch, > 7-BOOKKEEPER-629.patch, 9-BOOKKEEPER-629.patch > > > Register the bookie with *hostname:port* and also store the bookie addresses > as *hostname:port* in ledger metadata files instead of *ip:port* > This will help users to change the machine IP if they want without loosing > their data. > Supporting hostname based installation/functionality is one of the important > requirement of users. > Any thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Attachment: (was: BOOKKEEPER-747.patch) > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Attachment: BOOKKEEPER-747.patch > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-755) Incorrect number of seconds specified in a day
[ https://issues.apache.org/jira/browse/BOOKKEEPER-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-755: Fix Version/s: 4.2.3 4.3.0 > Incorrect number of seconds specified in a day > -- > > Key: BOOKKEEPER-755 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-755 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-server >Reporter: Joseph Redfern >Assignee: Joseph Redfern >Priority: Minor > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-755.patch > > > Pretty minor bug, but: > https://github.com/apache/bookkeeper/blob/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java#L1182 > specifies that there are 84600 seconds in a day, when there are in fact > 86400. The default should be changed to to the correct value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Attachment: BOOKKEEPER-747-4.2.patch > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993576#comment-13993576 ] Flavio Junqueira commented on BOOKKEEPER-747: - I have another test case for the removal option and fixed the log debug message issue. > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-755) Incorrect number of seconds specified in a day
[ https://issues.apache.org/jira/browse/BOOKKEEPER-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996900#comment-13996900 ] Flavio Junqueira commented on BOOKKEEPER-755: - +1, looks good. > Incorrect number of seconds specified in a day > -- > > Key: BOOKKEEPER-755 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-755 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-server >Reporter: Joseph Redfern >Assignee: Joseph Redfern >Priority: Minor > Attachments: BOOKKEEPER-755.patch > > > Pretty minor bug, but: > https://github.com/apache/bookkeeper/blob/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java#L1182 > specifies that there are 84600 seconds in a day, when there are in fact > 86400. The default should be changed to to the correct value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-758) Add TryReadLastAddConfirmed API
[ https://issues.apache.org/jira/browse/BOOKKEEPER-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996511#comment-13996511 ] Flavio Junqueira commented on BOOKKEEPER-758: - +1, looks good. Since this is an extension of the API, it would be good to have another +1 here. [~ikelly]? > Add TryReadLastAddConfirmed API > --- > > Key: BOOKKEEPER-758 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-758 > Project: Bookkeeper > Issue Type: Improvement > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Sijie Guo > Fix For: 4.3.0 > > Attachments: BOOKKEEPER-758.diff, BOOKKEEPER-758.v2.diff > > > add TryReadLastConfirmed to read last confirmed without coverage checking, as > for readers which polls LAC, they just need LAC. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-755) Incorrect number of seconds specified in a day
[ https://issues.apache.org/jira/browse/BOOKKEEPER-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-755: Assignee: Joseph Redfern > Incorrect number of seconds specified in a day > -- > > Key: BOOKKEEPER-755 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-755 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-server >Reporter: Joseph Redfern >Assignee: Joseph Redfern >Priority: Minor > > Pretty minor bug, but: > https://github.com/apache/bookkeeper/blob/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java#L1182 > specifies that there are 84600 seconds in a day, when there are in fact > 86400. The default should be changed to to the correct value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-758) Add TryReadLastAddConfirmed API
[
https://issues.apache.org/jira/browse/BOOKKEEPER-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995075#comment-13995075
]
Flavio Junqueira commented on BOOKKEEPER-758:
-
It looks good. It sounds like you want to have a weaker version of
ReadLastAddConfirmed that simply returns some valid value, not necessarily the
last.
It looks like the callback can be called multiple times. For example, we can
return immediately here:
{noformat}
// callback immediately
cb.readLastConfirmedDataComplete(BKException.Code.OK, maxRecoveredData);
{noformat}
and later on here:
{noformat}
if (numResponsesPending == 0 && !completed) {
if (!hasValidResponse) {
// no success called
cb.readLastConfirmedDataComplete(BKException.Code.LedgerRecoveryException,
maxRecoveredData);
} else {
// callback
cb.readLastConfirmedDataComplete(BKException.Code.OK,
maxRecoveredData);
}
completed = true;
}
{noformat}
In the test cases, don't we have to wait until the bookie is down when we kill
a bookie?
(typo) Also, in the last test case, it shouldn't be numEntries here:
{noformat}
for (int i = 0; i < numEntries; i++) {
killBookie(lh.getLedgerMetadata().currentEnsemble.get(i));
}
{noformat}
> Add TryReadLastAddConfirmed API
> ---
>
> Key: BOOKKEEPER-758
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-758
> Project: Bookkeeper
> Issue Type: Improvement
> Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.3.0
>
> Attachments: BOOKKEEPER-758.diff
>
>
> add TryReadLastConfirmed to read last confirmed without coverage checking, as
> for readers which polls LAC, they just need LAC.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Attachment: BOOKKEEPER-747.patch Ok, fair enough. I have made the change you suggested. I have also realized that we are processing NodeDeleted in AbstractZK. I have made a few changes to the processing of removed metadata to make the two implementations match. > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Attachment: (was: BOOKKEEPER-747-4.2.patch) > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Attachment: BOOKKEEPER-747.patch > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Attachment: BOOKKEEPER-747-4.2.patch > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992613#comment-13992613 ] Flavio Junqueira commented on BOOKKEEPER-747: - The reason for synchronizing listenerSet in MSLedgerManagerFactory is that I'm removing the listeners (and notifying) when the metadata is removed. The same functionality doesn't seem to be present in the AbstractZK version, so I think in the worst case we will be processing a slightly stale list if we don't synchronize. Given this context, there are two questions I'd like to ask: # Should we have the "REMOVED" functionality in this patch given that we don't have it in AbstractZK? # What can go wrong if we don't synchronize listenerSet in AbstractZK? > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Attachment: BOOKKEEPER-747-4.2.patch > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, > BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Fix Version/s: 4.2.3 4.3.0 > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Attachment: BOOKKEEPER-747.patch > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira reassigned BOOKKEEPER-747: --- Assignee: Flavio Junqueira > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira >Assignee: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Attachment: BOOKKEEPER-747-4.2.patch > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira > Fix For: 4.3.0, 4.2.3 > > Attachments: BOOKKEEPER-747-4.2.patch, BOOKKEEPER-747.patch, > BOOKKEEPER-747.patch, BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Attachment: BOOKKEEPER-747.patch Some more progress with this patch. > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira > Attachments: BOOKKEEPER-747.patch, BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-744) Run the auditor bookie check periodically
[ https://issues.apache.org/jira/browse/BOOKKEEPER-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-744: Attachment: BOOKKEEPER-744-4.2.patch AuditorPeriodicBookieCheckTest is missing BookieSocketAddress, so I've reverted it back to InetSocketAddress, since BOOKKEEPER-644 is not in the 4.2 branch. > Run the auditor bookie check periodically > - > > Key: BOOKKEEPER-744 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-744 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-auto-recovery >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 4.3.0, 4.2.3 > > Attachments: > 0001-BOOKKEEPER-744-Run-the-bookie-check-periodical.trunk.patch, > 0001-Run-the-bookie-check-periodically.trunk.patch, > 0003-Run-the-bookie-check-periodically.trunk.patch, > 0005-Run-the-bookie-check-periodically.branch4.2.patch, > BOOKKEEPER-744-4.2.patch > > > Previous the bookie check only runs when a bookie fails. If for some reason > this doesn't pick up a failure, the failure will go undetected until the next > time a bookie fails, in which case quorum could have been lost. This fix > makes the bookie check run periodically, by default once a day. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-744) Run the auditor bookie check periodically
[ https://issues.apache.org/jira/browse/BOOKKEEPER-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988144#comment-13988144 ] Flavio Junqueira commented on BOOKKEEPER-744: - Actually, the patch works fine with trunk (tests pass for me too), but it fails to apply to 4.2. > Run the auditor bookie check periodically > - > > Key: BOOKKEEPER-744 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-744 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-auto-recovery >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 4.3.0, 4.2.3 > > Attachments: > 0001-BOOKKEEPER-744-Run-the-bookie-check-periodical.trunk.patch, > 0001-Run-the-bookie-check-periodically.trunk.patch, > 0003-Run-the-bookie-check-periodically.trunk.patch, > 0005-Run-the-bookie-check-periodically.branch4.2.patch > > > Previous the bookie check only runs when a bookie fails. If for some reason > this doesn't pick up a failure, the failure will go undetected until the next > time a bookie fails, in which case quorum could have been lost. This fix > makes the bookie check run periodically, by default once a day. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-744) Run the auditor bookie check periodically
[ https://issues.apache.org/jira/browse/BOOKKEEPER-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987730#comment-13987730 ] Flavio Junqueira commented on BOOKKEEPER-744: - The patch looks good to me, but it isn't applying cleanly to trunk for me, I'm not sure what's happening. It complains about bk_server.conf and ServerConfiguration.java, and when I tried to make those changes manually, it doesn't compile. > Run the auditor bookie check periodically > - > > Key: BOOKKEEPER-744 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-744 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-auto-recovery >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 4.3.0, 4.2.3 > > Attachments: > 0001-BOOKKEEPER-744-Run-the-bookie-check-periodical.trunk.patch, > 0001-Run-the-bookie-check-periodically.trunk.patch, > 0003-Run-the-bookie-check-periodically.trunk.patch, > 0005-Run-the-bookie-check-periodically.branch4.2.patch > > > Previous the bookie check only runs when a bookie fails. If for some reason > this doesn't pick up a failure, the failure will go undetected until the next > time a bookie fails, in which case quorum could have been lost. This fix > makes the bookie check run periodically, by default once a day. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-629) Support hostname based ledger metadata to help users to change IP with existing installation
[ https://issues.apache.org/jira/browse/BOOKKEEPER-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986056#comment-13986056 ] Flavio Junqueira commented on BOOKKEEPER-629: - The backward compatibility point is a good one. I suppose we could rewrite the znode once we release 4.3.0, no? One option actually is that we get this patch in 4.2.3, and do the znode rewrite in 4.3.0. This way we have an upgrade path. Otherwise, it looks good to me. > Support hostname based ledger metadata to help users to change IP with > existing installation > > > Key: BOOKKEEPER-629 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-629 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-auto-recovery, bookkeeper-client, > bookkeeper-server >Affects Versions: 4.2.1 >Reporter: Vinayakumar B >Assignee: Rakesh R > Fix For: 4.3.0 > > Attachments: 1-BOOKKEEPER-629.patch, 10-BOOKKEEPER-629.patch, > 11-BOOKKEEPER-629.patch, 2-BOOKKEEPER-629.patch, 3-BOOKKEEPER-629.patch, > 4-BOOKKEEPER-629.patch, 5-BOOKKEEPER-629.patch, 6-BOOKKEEPER-629.patch, > 7-BOOKKEEPER-629.patch, 9-BOOKKEEPER-629.patch > > > Register the bookie with *hostname:port* and also store the bookie addresses > as *hostname:port* in ledger metadata files instead of *ip:port* > This will help users to change the machine IP if they want without loosing > their data. > Supporting hostname based installation/functionality is one of the important > requirement of users. > Any thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13968514#comment-13968514 ] Flavio Junqueira commented on BOOKKEEPER-747: - yeah, it complicates things a bit. I suppose the general idea is to poll the metastore in the case it doesn't support notifications. if we can have metastore options that support notifications and others that don't support it, then it sounds like we need to have the metastore implementation polling and notifying up the MSLedgerManager. otherwise the ledger manager needs to support both ways of notifying of metadata changes. > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira > Attachments: BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
[ https://issues.apache.org/jira/browse/BOOKKEEPER-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated BOOKKEEPER-747: Attachment: BOOKKEEPER-747.patch Here is a preliminary patch for branch 4.2. There is a fair amount of code duplication, which I'm not very happy with. I was also thinking about moving MSLedgerManager to an independent file. If you people have any high-level comments, please let me know. Otherwise I'll keep making progress. > Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory > -- > > Key: BOOKKEEPER-747 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 > Project: Bookkeeper > Issue Type: Improvement >Affects Versions: 4.3.0, 4.2.3 >Reporter: Flavio Junqueira > Attachments: BOOKKEEPER-747.patch > > > Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (BOOKKEEPER-747) Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
Flavio Junqueira created BOOKKEEPER-747: --- Summary: Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory Key: BOOKKEEPER-747 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-747 Project: Bookkeeper Issue Type: Improvement Affects Versions: 4.3.0, 4.2.3 Reporter: Flavio Junqueira Check TODOs in MSLedgerManagerFactory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-742) Fix for empty ledgers losing quorum.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967200#comment-13967200 ] Flavio Junqueira commented on BOOKKEEPER-742: - Could you make the changes so that I can check this in, please? > Fix for empty ledgers losing quorum. > > > Key: BOOKKEEPER-742 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-742 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-auto-recovery >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 4.3.0, 4.2.3 > > Attachments: 0001-Fix-for-empty-ledgers-using-quorum.trunk.patch, > 0003-Fix-for-empty-ledgers-using-quorum.branch4.2.patch > > > If a ledger is open and empty, when a bookie in the ensemble crashes no > recovery will take place (because there's nothing to recover). This open > empty unrepaired ledger can persist for a long time. If it loses another > bookie, it can lose quorum. At this point it's impossible for the bookie to > know that its an empty ledger, and the admin gets notified of missing data. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-710) OpenLedgerNoRecovery should watch ensemble change.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966330#comment-13966330 ] Flavio Junqueira commented on BOOKKEEPER-710: - About the question on watches, I knew I had looked into it, but I couldn't remember where the discussion. In Chapter 4 of the ZK book, there is a discussion about the scalability of watches. Each new watch consumes around 300 bytes on the server, so again watches induce memory utilization on the server and possibly bursts of traffic depending on the number of notifications triggered. One way to spread watches across servers is to use observers > OpenLedgerNoRecovery should watch ensemble change. > -- > > Key: BOOKKEEPER-710 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-710 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Sijie Guo >Priority: Blocker > Fix For: 4.3.0, 4.2.3 > > Attachments: > 0001-BOOKKEEPER-710-OpenLedgerNoRecovery-should-watch-ens.patch, > BOOKKEEPER-710.diff > > > LedgerHandle opened by openLedgerNoRecovery should watch ensemble change. > otherwise, readLastConfirmed & readEntries will stuck if there are ensemble > changes in the ledger. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-742) Fix for empty ledgers losing quorum.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963141#comment-13963141 ] Flavio Junqueira commented on BOOKKEEPER-742: - bq. isLastSegmentOpenAndMissingBookies(LedgerHandle) yes, this is good. > Fix for empty ledgers losing quorum. > > > Key: BOOKKEEPER-742 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-742 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-auto-recovery >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 4.3.0, 4.2.3 > > Attachments: 0001-Fix-for-empty-ledgers-using-quorum.trunk.patch, > 0003-Fix-for-empty-ledgers-using-quorum.branch4.2.patch > > > If a ledger is open and empty, when a bookie in the ensemble crashes no > recovery will take place (because there's nothing to recover). This open > empty unrepaired ledger can persist for a long time. If it loses another > bookie, it can lose quorum. At this point it's impossible for the bookie to > know that its an empty ledger, and the admin gets notified of missing data. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-733) Improve ReplicationWorker to handle the urLedgers which already have same leder replica in hand
[
https://issues.apache.org/jira/browse/BOOKKEEPER-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963076#comment-13963076
]
Flavio Junqueira commented on BOOKKEEPER-733:
-
bq. As a component, I already find it hard to test and understand. I'd
actually prefer to take some logic out of it.
+1
> Improve ReplicationWorker to handle the urLedgers which already have same
> leder replica in hand
> ---
>
> Key: BOOKKEEPER-733
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-733
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-auto-recovery
>Affects Versions: 4.2.2, 4.3.0
>Reporter: Rakesh R
>Assignee: Rakesh R
>
> +Scenario:+
> Step1 : Have three bookies BK1, BK2, BK3
> Step2 : Have written ledgers with quorum 2
> Step3 : Unfortunately BK2 and BK3 both went down for few moments.
> The following logs are flooded in BK1 autorecovery logs. RW is trying to
> replicate the ledgers, but it simply skip this fragment and moves to next
> cycle when it sees a replica found in his hand. IMO, we should have a
> mechanism in place to avoid unnecessary cycles.
> {code}
> 2014-02-18 21:47:55,140 - ERROR - [New I/O client boss
> #2-1:PerChannelBookieClient$1@230] - Could not connect to bookie: [id:
> 0x00ba679e]/10.18.170.130:15002, current state CONNECTING :
> java.net.ConnectException: Connection refused: no further information
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> 2014-02-18 21:47:55,140 - INFO - 2014-02-18 21:59:33,215 - DEBUG -
> [ReplicationWorker:ReplicationWorker@182] - Target
> Bookie[10.18.170.130:15003] found in the fragment ensemble:
> [10.18.170.130:15003, 10.18.170.130:15001, 10.18.170.130:15002]
> [ReplicationWorker:PerChannelBookieClient@194] - Connecting to bookie:
> 10.18.170.130:15002
> 2014-02-18 21:47:56,162 - ERROR - [New I/O client boss
> #2-1:PerChannelBookieClient$1@230] - Could not connect to bookie: [id:
> 0x0003f377]/10.18.170.130:15002, current state CONNECTING :
> java.net.ConnectException: Connection refused: no further information
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> 2014-02-18 21:59:33,215 - DEBUG - [ReplicationWorker:ReplicationWorker@182]
> - Target Bookie[10.18.170.130:15003] found in the fragment ensemble:
> [10.18.170.130:15003, 10.18.170.130:15001, 10.18.170.130:15002]
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (BOOKKEEPER-742) Fix for empty ledgers losing quorum.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962385#comment-13962385 ] Flavio Junqueira edited comment on BOOKKEEPER-742 at 4/8/14 7:05 AM: - bq. shouldCloseLedger is too generic shouldCloseLedgerWhenLedgerOpenAndEnsembleAvailable ;-) or shouldCloseLedgerIfLedgerOpenAndEnsembleAvailable was (Author: fpj): bq. shouldCloseLedger is too generic shouldCloseLedgerWhenLedgerOpenAndEnsembleAvailable ;-) > Fix for empty ledgers losing quorum. > > > Key: BOOKKEEPER-742 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-742 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-auto-recovery >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 4.3.0, 4.2.3 > > Attachments: 0001-Fix-for-empty-ledgers-using-quorum.trunk.patch, > 0003-Fix-for-empty-ledgers-using-quorum.branch4.2.patch > > > If a ledger is open and empty, when a bookie in the ensemble crashes no > recovery will take place (because there's nothing to recover). This open > empty unrepaired ledger can persist for a long time. If it loses another > bookie, it can lose quorum. At this point it's impossible for the bookie to > know that its an empty ledger, and the admin gets notified of missing data. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-742) Fix for empty ledgers losing quorum.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962385#comment-13962385 ] Flavio Junqueira commented on BOOKKEEPER-742: - bq. shouldCloseLedger is too generic shouldCloseLedgerWhenLedgerOpenAndEnsembleAvailable ;-) > Fix for empty ledgers losing quorum. > > > Key: BOOKKEEPER-742 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-742 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-auto-recovery >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 4.3.0, 4.2.3 > > Attachments: 0001-Fix-for-empty-ledgers-using-quorum.trunk.patch, > 0003-Fix-for-empty-ledgers-using-quorum.branch4.2.patch > > > If a ledger is open and empty, when a bookie in the ensemble crashes no > recovery will take place (because there's nothing to recover). This open > empty unrepaired ledger can persist for a long time. If it loses another > bookie, it can lose quorum. At this point it's impossible for the bookie to > know that its an empty ledger, and the admin gets notified of missing data. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (BOOKKEEPER-742) Fix for empty ledgers losing quorum.
[
https://issues.apache.org/jira/browse/BOOKKEEPER-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961141#comment-13961141
]
Flavio Junqueira commented on BOOKKEEPER-742:
-
I remember this discussion on the list, but I can't remember the conclusion.
This is a comment in the patch:
{noformat}
* For example, if in a E3Q2, only 1 entry is written and the last bookie
* in the ensemble fails, nothing has been written to it, so nothing needs
to be
* recovered. But if the second to last bookie fails, we've now lost quorum
for
* the second entry, so it's impossible to see if the second has been
written or
* not.
{noformat}
The problem here is that you've lost too many bookies, so it is possible that
you've lost data and consequently the ledger is bad. Say that entry has been
successfully written and acknowledged. In this case, we can't recover 1 and the
ledger is bad, but we can't distinguish the case you're describing from the one
I just presented.
Also, for completeness, I just wanted to confirm that if 3 or more entries have
been written, then we would be able to spot that the ledger is really bad
because we would be able to see entry 2, but not 1.
One small point about the patch. This method isFinalEnsembleOpenAndAvailable,
which seems to be key to this patch, returns true in the case the ledger
metadata says closed. Returning true in this case is a bit misleading, and
perhaps we could rename the method to something like shouldCloseLedger and
negate the return values.
> Fix for empty ledgers losing quorum.
>
>
> Key: BOOKKEEPER-742
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-742
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-auto-recovery
>Reporter: Ivan Kelly
>Assignee: Ivan Kelly
> Fix For: 4.3.0, 4.2.3
>
> Attachments: 0001-Fix-for-empty-ledgers-using-quorum.trunk.patch,
> 0003-Fix-for-empty-ledgers-using-quorum.branch4.2.patch
>
>
> If a ledger is open and empty, when a bookie in the ensemble crashes no
> recovery will take place (because there's nothing to recover). This open
> empty unrepaired ledger can persist for a long time. If it loses another
> bookie, it can lose quorum. At this point it's impossible for the bookie to
> know that its an empty ledger, and the admin gets notified of missing data.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (BOOKKEEPER-629) Support hostname based ledger metadata to help users to change IP with existing installation
[ https://issues.apache.org/jira/browse/BOOKKEEPER-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13959303#comment-13959303 ] Flavio Junqueira commented on BOOKKEEPER-629: - I'm sorry for chiming in late, but I was wondering if it wouldn't be best to recreate the znode instead of inverting the value of the parameter. It may be a bit counterintuitive. > Support hostname based ledger metadata to help users to change IP with > existing installation > > > Key: BOOKKEEPER-629 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-629 > Project: Bookkeeper > Issue Type: Sub-task > Components: bookkeeper-auto-recovery, bookkeeper-client, > bookkeeper-server >Affects Versions: 4.2.1 >Reporter: Vinayakumar B >Assignee: Rakesh R > Fix For: 4.3.0 > > Attachments: 1-BOOKKEEPER-629.patch, 10-BOOKKEEPER-629.patch, > 11-BOOKKEEPER-629.patch, 2-BOOKKEEPER-629.patch, 3-BOOKKEEPER-629.patch, > 4-BOOKKEEPER-629.patch, 5-BOOKKEEPER-629.patch, 6-BOOKKEEPER-629.patch, > 7-BOOKKEEPER-629.patch, 9-BOOKKEEPER-629.patch > > > Register the bookie with *hostname:port* and also store the bookie addresses > as *hostname:port* in ledger metadata files instead of *ip:port* > This will help users to change the machine IP if they want without loosing > their data. > Supporting hostname based installation/functionality is one of the important > requirement of users. > Any thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)
