ZooKeeper_branch34_solaris - Build # 576 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch34_solaris/576/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 1171 lines...] [junit] 2013-07-02 07:36:04,371 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@57] - FINISHED TEST METHOD testSingleSerialize [junit] 2013-07-02 07:36:04,371 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testSingleSerialize [junit] 2013-07-02 07:36:04,371 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testSingleSerialize [junit] 2013-07-02 07:36:04,375 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testWideSerialize [junit] 2013-07-02 07:36:04,375 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@50] - RUNNING TEST METHOD testWideSerialize [junit] 2013-07-02 07:36:04,987 [myid:] - INFO [main:SerializationPerfTest@73] - Serialized 10005 nodes in 156 ms (15us/node), depth=2 width=1 datalen=20 [junit] 2013-07-02 07:36:04,988 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@57] - FINISHED TEST METHOD testWideSerialize [junit] 2013-07-02 07:36:04,988 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testWideSerialize [junit] 2013-07-02 07:36:04,988 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testWideSerialize [junit] 2013-07-02 07:36:04,989 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING testDeepSerialize [junit] 2013-07-02 07:36:04,989 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@50] - RUNNING TEST METHOD testDeepSerialize [junit] 2013-07-02 07:36:05,183 [myid:] - INFO [main:SerializationPerfTest@73] - Serialized 404 nodes in 9 ms (22us/node), depth=400 width=1 datalen=20 [junit] 2013-07-02 07:36:05,183 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@57] - FINISHED TEST METHOD testDeepSerialize [junit] 2013-07-02 07:36:05,185 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED testDeepSerialize [junit] 2013-07-02 07:36:05,185 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED testDeepSerialize [junit] 2013-07-02 07:36:05,186 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING test10Wide5DeepSerialize [junit] 2013-07-02 07:36:05,186 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@50] - RUNNING TEST METHOD test10Wide5DeepSerialize [junit] 2013-07-02 07:36:05,420 [myid:] - INFO [main:SerializationPerfTest@73] - Serialized 5 nodes in 55 ms (5us/node), depth=5 width=10 datalen=20 [junit] 2013-07-02 07:36:05,420 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@57] - FINISHED TEST METHOD test10Wide5DeepSerialize [junit] 2013-07-02 07:36:05,420 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED test10Wide5DeepSerialize [junit] 2013-07-02 07:36:05,420 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED test10Wide5DeepSerialize [junit] 2013-07-02 07:36:05,421 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING test15Wide5DeepSerialize [junit] 2013-07-02 07:36:05,421 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@50] - RUNNING TEST METHOD test15Wide5DeepSerialize [junit] 2013-07-02 07:36:06,469 [myid:] - INFO [main:SerializationPerfTest@73] - Serialized 54245 nodes in 118 ms (2us/node), depth=5 width=15 datalen=20 [junit] 2013-07-02 07:36:06,469 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@57] - FINISHED TEST METHOD test15Wide5DeepSerialize [junit] 2013-07-02 07:36:06,469 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED test15Wide5DeepSerialize [junit] 2013-07-02 07:36:06,469 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED test15Wide5DeepSerialize [junit] 2013-07-02 07:36:06,470 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING test25Wide4DeepSerialize [junit] 2013-07-02 07:36:06,470 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@50] - RUNNING TEST METHOD test25Wide4DeepSerialize [junit] 2013-07-02 07:36:06,709 [myid:] - INFO [main:SerializationPerfTest@73] - Serialized 16280 nodes in 16 ms (1us/node), depth=4 width=25 datalen=20 [junit] 2013-07-02 07:36:06,709 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@57] - FINISHED TEST METHOD test25Wide4DeepSerialize [junit] 2013-07-02 07:36:06,710 [myid:] - INFO [main:ZKTestCase$1@60] - SUCCEEDED test25Wide4DeepSerialize [junit] 2013-07-02 07:36:06,710 [myid:] - INFO [main:ZKTestCase$1@55] - FINISHED test25Wide4DeepSerialize [junit] 2013-07-02 07:36:06,710 [myid:] - INFO [main:ZKTestCase$1@50] - STARTING test40Wide4DeepSerialize [junit] 2013-07-02 07:36:06,710 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@50] - RUNNING TEST METHOD test40Wide4DeepSerialize [junit] 2013-07-02 07:36:07,460 [myid:] - INFO [main:SerializationPerfTest@73] - Serialized 65645 nodes in 78 ms (1us/node), depth=4 width=40 datalen=20 [junit] 2013-07-02 07:36:07,460
ZooKeeper-trunk-WinVS2008 - Build # 914 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk-WinVS2008/914/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 515 lines...] .\src\cli.c(402): error C2065: 'p' : undeclared identifier .\src\cli.c(403): error C2065: 'p' : undeclared identifier .\src\cli.c(404): error C2065: 'p' : undeclared identifier .\src\cli.c(405): error C2065: 'mode' : undeclared identifier .\src\cli.c(405): error C2065: 'p' : undeclared identifier .\src\cli.c(406): error C2065: 'syntaxError' : undeclared identifier .\src\cli.c(409): error C2065: 'mode' : undeclared identifier .\src\cli.c(410): error C2275: 'FILE' : illegal use of this type as an expression .\src\cli.c(410): error C2065: 'fp' : undeclared identifier .\src\cli.c(410): error C2065: 'p' : undeclared identifier .\src\cli.c(411): error C2065: 'fp' : undeclared identifier .\src\cli.c(412): error C2065: 'p' : undeclared identifier .\src\cli.c(413): error C2065: 'syntaxError' : undeclared identifier .\src\cli.c(416): error C2065: 'fp' : undeclared identifier .\src\cli.c(417): error C2065: 'members_size' : undeclared identifier .\src\cli.c(417): error C2065: 'fp' : undeclared identifier .\src\cli.c(418): error C2065: 'fp' : undeclared identifier .\src\cli.c(419): error C2065: 'members' : undeclared identifier .\src\cli.c(419): error C2065: 'members_size' : undeclared identifier .\src\cli.c(420): error C2065: 'members' : undeclared identifier .\src\cli.c(422): error C2065: 'p' : undeclared identifier .\src\cli.c(423): error C2065: 'syntaxError' : undeclared identifier .\src\cli.c(424): error C2065: 'fp' : undeclared identifier .\src\cli.c(434): error C2065: 'members' : undeclared identifier .\src\cli.c(434): error C2065: 'members_size' : undeclared identifier .\src\cli.c(434): error C2065: 'fp' : undeclared identifier .\src\cli.c(435): error C2065: 'p' : undeclared identifier .\src\cli.c(436): error C2065: 'syntaxError' : undeclared identifier .\src\cli.c(437): error C2065: 'fp' : undeclared identifier .\src\cli.c(440): error C2065: 'fp' : undeclared identifier .\src\cli.c(441): error C2065: 'p' : undeclared identifier .\src\cli.c(442): error C2065: 'p' : undeclared identifier .\src\cli.c(443): error C2065: 'version' : undeclared identifier .\src\cli.c(443): error C2065: 'p' : undeclared identifier .\src\cli.c(444): error C2065: 'syntaxError' : undeclared identifier .\src\cli.c(447): error C2065: 'version' : undeclared identifier .\src\cli.c(447): error C2065: 'p' : undeclared identifier .\src\cli.c(448): error C2065: 'version' : undeclared identifier .\src\cli.c(449): error C2065: 'syntaxError' : undeclared identifier .\src\cli.c(453): error C2065: 'syntaxError' : undeclared identifier .\src\cli.c(456): error C2065: 'p' : undeclared identifier .\src\cli.c(458): error C2065: 'syntaxError' : undeclared identifier .\src\cli.c(460): error C2065: 'joining' : undeclared identifier .\src\cli.c(460): error C2065: 'leaving' : undeclared identifier .\src\cli.c(460): error C2065: 'members' : undeclared identifier .\src\cli.c(460): error C2065: 'version' : undeclared identifier .\src\cli.c(461): error C2065: 'joining' : undeclared identifier .\src\cli.c(462): error C2065: 'leaving' : undeclared identifier .\src\cli.c(463): error C2065: 'members' : undeclared identifier 104 Warning(s) 102 Error(s) Time Elapsed 00:00:15.99 f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008exit 1 Build step 'Execute Windows batch command' marked build as failure Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
ZooKeeper-trunk-solaris - Build # 599 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk-solaris/599/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 142542 lines...] [junit] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:292) [junit] at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) [junit] 2013-07-02 09:18:41,762 [myid:] - INFO [main:ZooKeeperServer@422] - shutting down [junit] 2013-07-02 09:18:41,763 [myid:] - INFO [main:SessionTrackerImpl@180] - Shutting down [junit] 2013-07-02 09:18:41,763 [myid:] - INFO [main:PrepRequestProcessor@929] - Shutting down [junit] 2013-07-02 09:18:41,762 [myid:] - INFO [LearnerCnxAcceptor-0.0.0.0/0.0.0.0:12472:Leader$LearnerCnxAcceptor@398] - exception while shutting down acceptor: java.net.SocketException: Socket closed [junit] 2013-07-02 09:18:41,763 [myid:] - INFO [main:ProposalRequestProcessor@88] - Shutting down [junit] 2013-07-02 09:18:41,764 [myid:] - INFO [main:CommitProcessor@333] - Shutting down [junit] 2013-07-02 09:18:41,763 [myid:] - INFO [ProcessThread(sid:5 cport:-1)::PrepRequestProcessor@156] - PrepRequestProcessor exited loop! [junit] 2013-07-02 09:18:41,765 [myid:] - INFO [CommitProcessor:5:CommitProcessor@218] - CommitProcessor exited loop! [junit] 2013-07-02 09:18:41,765 [myid:] - INFO [main:Leader$ToBeAppliedRequestProcessor@886] - Shutting down [junit] 2013-07-02 09:18:41,766 [myid:] - INFO [main:FinalRequestProcessor@427] - shutdown of request processor complete [junit] 2013-07-02 09:18:41,766 [myid:] - INFO [main:SyncRequestProcessor@175] - Shutting down [junit] 2013-07-02 09:18:41,766 [myid:] - INFO [SyncThread:5:SyncRequestProcessor@155] - SyncRequestProcessor exited! [junit] 2013-07-02 09:18:41,766 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method [junit] 2013-07-02 09:18:41,766 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11482:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method [junit] 2013-07-02 09:18:41,766 [myid:] - INFO [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method [junit] 2013-07-02 09:18:41,948 [myid:] - INFO [main:QuorumBase@342] - Shutting down leader election QuorumPeer[myid=5]/0.0.0.0:11482 [junit] 2013-07-02 09:18:41,948 [myid:] - WARN [QuorumPeer[myid=5]/0.0.0.0:11482:QuorumPeer@952] - Unexpected exception [junit] java.lang.InterruptedException: sleep interrupted [junit] at java.lang.Thread.sleep(Native Method) [junit] at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:545) [junit] at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:949) [junit] 2013-07-02 09:18:41,949 [myid:] - INFO [QuorumPeer[myid=5]/0.0.0.0:11482:Leader@585] - Shutting down [junit] 2013-07-02 09:18:41,949 [myid:] - WARN [QuorumPeer[myid=5]/0.0.0.0:11482:QuorumPeer@979] - PeerState set to LOOKING [junit] 2013-07-02 09:18:41,949 [myid:] - WARN [QuorumPeer[myid=5]/0.0.0.0:11482:QuorumPeer@965] - QuorumPeer main thread exited [junit] 2013-07-02 09:18:41,948 [myid:] - INFO [/127.0.0.1:12477:QuorumCnxManager$Listener@565] - Leaving listener [junit] 2013-07-02 09:18:41,949 [myid:] - INFO [main:QuorumBase@347] - Waiting for QuorumPeer[myid=5]/0.0.0.0:11482 to exit thread [junit] 2013-07-02 09:18:41,950 [myid:] - INFO [main:FourLetterWordMain@43] - connecting to 127.0.0.1 11478 [junit] 2013-07-02 09:18:41,950 [myid:] - INFO [main:QuorumBase@323] - 127.0.0.1:11478 is no longer accepting client connections [junit] 2013-07-02 09:18:41,951 [myid:] - INFO [main:FourLetterWordMain@43] - connecting to 127.0.0.1 11479 [junit] 2013-07-02 09:18:41,951 [myid:] - INFO [main:QuorumBase@323] - 127.0.0.1:11479 is no longer accepting client connections [junit] 2013-07-02 09:18:41,951 [myid:] - INFO [main:FourLetterWordMain@43] - connecting to 127.0.0.1 11480 [junit] 2013-07-02 09:18:41,952 [myid:] - INFO [main:QuorumBase@323] - 127.0.0.1:11480 is no longer accepting client connections [junit] 2013-07-02 09:18:41,952 [myid:] - INFO [main:FourLetterWordMain@43] - connecting to 127.0.0.1 11481 [junit] 2013-07-02 09:18:41,952 [myid:] - INFO [main:QuorumBase@323] - 127.0.0.1:11481 is no longer accepting client
[jira] [Commented] (ZOOKEEPER-1702) ZooKeeper client may write operation packets before receiving successful response to connection request, can cause TCP RST
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13697689#comment-13697689
]
Hudson commented on ZOOKEEPER-1702:
---
Integrated in ZooKeeper-trunk #1978 (See
[https://builds.apache.org/job/ZooKeeper-trunk/1978/])
ZOOKEEPER-1702. ZooKeeper client may write operation packets before
receiving successful response to connection request, can cause TCP RST (Chris
Nauroth via phunt) (Revision 1498732)
Result = SUCCESS
phunt : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1498732
Files :
* /zookeeper/trunk/CHANGES.txt
* /zookeeper/trunk/src/java/main/org/apache/zookeeper/ClientCnxnSocketNIO.java
ZooKeeper client may write operation packets before receiving successful
response to connection request, can cause TCP RST
--
Key: ZOOKEEPER-1702
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1702
Project: ZooKeeper
Issue Type: Bug
Components: java client
Affects Versions: 3.4.2
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 3.5.0, 3.4.6
Attachments: ZOOKEEPER-1702.1.patch
The problem occurs when a connection attempt is pending and there are
multiple outbound packets in the queue for other operations. In
{{ClientCnxnSocketNIO#doIO}}, it is possible to receive notification that the
socket is writable for the next operation packet before receiving
notification that the socket is readable for the connection response from the
server. If the server decides that the session is expired, then it responds
by immediately closing the socket on its side. If the client has written
packets after the server has closed its end of the socket, then the TCP stack
may choose to abort the connection with an RST. When this happens, the
client doesn't receive an orderly shutdown, and ultimately it fails to
deliver a session expired event to the application.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1702) ZooKeeper client may write operation packets before receiving successful response to connection request, can cause TCP RST
[
https://issues.apache.org/jira/browse/ZOOKEEPER-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13697982#comment-13697982
]
Chris Nauroth commented on ZOOKEEPER-1702:
--
Thanks very much for reviewing and committing this, Patrick!
ZooKeeper client may write operation packets before receiving successful
response to connection request, can cause TCP RST
--
Key: ZOOKEEPER-1702
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1702
Project: ZooKeeper
Issue Type: Bug
Components: java client
Affects Versions: 3.4.2
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 3.5.0, 3.4.6
Attachments: ZOOKEEPER-1702.1.patch
The problem occurs when a connection attempt is pending and there are
multiple outbound packets in the queue for other operations. In
{{ClientCnxnSocketNIO#doIO}}, it is possible to receive notification that the
socket is writable for the next operation packet before receiving
notification that the socket is readable for the connection response from the
server. If the server decides that the session is expired, then it responds
by immediately closing the socket on its side. If the client has written
packets after the server has closed its end of the socket, then the TCP stack
may choose to abort the connection with an RST. When this happens, the
client doesn't receive an orderly shutdown, and ultimately it fails to
deliver a session expired event to the application.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-636) Latest txn logs might be deleted in a race condition which is not recoverable if BK goes down before next txn log created.
[
https://issues.apache.org/jira/browse/BOOKKEEPER-636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Kelly updated BOOKKEEPER-636:
--
Attachment: 0001-BOOKKEEPER-636-Latest-txn-logs-might-be-deleted-in-a.patch
The createNewFile change is very small, and ensures two processes don't make a
mess of each others journals. I'm not sure two processes could even get this
far, but it's better to be safe than sorry. (i've attached a patch with
includes this change)
Latest txn logs might be deleted in a race condition which is not recoverable
if BK goes down before next txn log created.
--
Key: BOOKKEEPER-636
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-636
Project: Bookkeeper
Issue Type: Bug
Components: bookkeeper-server
Affects Versions: 4.2.1, 4.3.0
Reporter: Vinay
Assignee: Vinay
Priority: Blocker
Fix For: 4.2.2, 4.3.0
Attachments:
0001-BOOKKEEPER-636-Latest-txn-logs-might-be-deleted-in-a.patch,
BOOKKEEPER-636.diff, BOOKKEEPER-636.diff, BOOKKEEPER-636.patch,
BOOKKEEPER-636.patch
With the following scenario latest transaction log can be deleted.
1. more than {{journalMaxBackups}} txn logs are there in journal dir.
2. BK machine was up for long time and the latest txn log id is some what
huge number
3. Now reboot the machine.
4. after reboot BK restarted.
5. Now, Immediately after startup, One entry is added, due to which Synthread
rolled the lastMark in ledger dirs before the lastLogId updated by Journal
thread. (this lastMark was having the old logId which was before reboot).
6. Now after roll, old journal txn logs were gc'ed. *Now latest created the
txn log was deleted.*
7. After this Journal thread updated the lastLogMark, also some more rolls
happened.
8. Now BK restarted again. But BK was not able to start because it was not
able to find the latest txn log file in journal directory.
{noformat}java.io.IOException: Recovery log 264564 is missing
at org.apache.bookkeeper.bookie.Journal.replay(Journal.java:424)
at org.apache.bookkeeper.bookie.Bookie.readJournal(Bookie.java:547)
at org.apache.bookkeeper.bookie.Bookie.start(Bookie.java:603)
at
org.apache.bookkeeper.proto.BookieServer.start(BookieServer.java:111){noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1147) Add support for local sessions
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13697996#comment-13697996 ] Raul Gutierrez Segales commented on ZOOKEEPER-1147: --- fwiw, we are using this patch in prod at Twitter so it would be awesome to have this merged. Besides what I mentioned in my previous comment (having a way to do rolling upgrades to enable local sessions) is there anything else that's left to get this merged? Add support for local sessions -- Key: ZOOKEEPER-1147 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1147 Project: ZooKeeper Issue Type: Improvement Components: server Affects Versions: 3.3.3 Reporter: Vishal Kathuria Assignee: Thawan Kooburat Labels: api-change, scaling Fix For: 3.5.0 Attachments: ZOOKEEPER-1147.patch, ZOOKEEPER-1147.patch, ZOOKEEPER-1147.patch, ZOOKEEPER-1147.patch, ZOOKEEPER-1147.patch, ZOOKEEPER-1147.patch, ZOOKEEPER-1147.patch, ZOOKEEPER-1147.patch Original Estimate: 840h Remaining Estimate: 840h This improvement is in the bucket of making ZooKeeper work at a large scale. We are planning on having about a 1 million clients connect to a ZooKeeper ensemble through a set of 50-100 observers. Majority of these clients are read only - ie they do not do any updates or create ephemeral nodes. In ZooKeeper today, the client creates a session and the session creation is handled like any other update. In the above use case, the session create/drop workload can easily overwhelm an ensemble. The following is a proposal for a local session, to support a larger number of connections. 1. The idea is to introduce a new type of session - local session. A local session doesn't have a full functionality of a normal session. 2. Local sessions cannot create ephemeral nodes. 3. Once a local session is lost, you cannot re-establish it using the session-id/password. The session and its watches are gone for good. 4. When a local session connects, the session info is only maintained on the zookeeper server (in this case, an observer) that it is connected to. The leader is not aware of the creation of such a session and there is no state written to disk. 5. The pings and expiration is handled by the server that the session is connected to. With the above changes, we can make ZooKeeper scale to a much larger number of clients without making the core ensemble a bottleneck. In terms of API, there are two options that are being considered 1. Let the client specify at the connect time which kind of session do they want. 2. All sessions connect as local sessions and automatically get promoted to global sessions when they do an operation that requires a global session (e.g. creating an ephemeral node) Chubby took the approach of lazily promoting all sessions to global, but I don't think that would work in our case, where we want to keep sessions which never create ephemeral nodes as always local. Option 2 would make it more broadly usable but option 1 would be easier to implement. We are thinking of implementing option 1 as the first cut. There would be a client flag, IsLocalSession (much like the current readOnly flag) that would be used to determine whether to create a local session or a global session. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-625) On OutOfMemoryError in NIOServerFactory thread bookie should shutdown
[ https://issues.apache.org/jira/browse/BOOKKEEPER-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13697999#comment-13697999 ] Ivan Kelly commented on BOOKKEEPER-625: --- With this patch the process will just exit silently, no? Couldn't we also change the catch on line 158 to take a Throwable rather than an Exception. This way, it will at least attempt to inform the user of the problem. On OutOfMemoryError in NIOServerFactory thread bookie should shutdown - Key: BOOKKEEPER-625 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-625 Project: Bookkeeper Issue Type: Bug Components: bookkeeper-server Affects Versions: 4.2.1 Reporter: Vinay Assignee: Vinay Fix For: 4.2.2 Attachments: BOOKKEEPER-625-branch-4.2.patch Observed OutOfMemoryError in NIOServerFactory, but it didnt bring down the bookie and it continued to run without serving. On OOME in any thread, bookie should shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ZOOKEEPER-1718) Support JLine 2
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shiju K babu updated ZOOKEEPER-1718: Description: not fixed Support JLine 2 --- Key: ZOOKEEPER-1718 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1718 Project: ZooKeeper Issue Type: Test Reporter: Christopher Tubbs Priority: Critical not fixed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-629) Support hostname based ledger metadata to help users to change IP with exitsting installation
[ https://issues.apache.org/jira/browse/BOOKKEEPER-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13697565#comment-13697565 ] Rakesh R commented on BOOKKEEPER-629: - There are some cases where users would like to ship pre-installed clusters. One use case which I come across is, after installing and testing, user wants to ship the nodes and update /etc/hosts by reconfigure the IP addresses. Very rarely changing the server ports and hostname. IMHO, we can discuss all the possibilities and choose better approach. Thanks Sijie for the detailed analysis. bq.information, you don't need to change 'hostname:port' identifier format. during first start time, you could generate the identifier based on 'hostname:port'. so the identifier is still backward compatible with old version clients. After that, the bookie keeps using this identifier, you could let new bookie inject So if I understand correctly, here the idea is to keep the identifier as 'hostname:port' format. For example, 'HOST-10-18-40-30:8020' later when user change the hostname to 'HOST-10-18-40-35', still this bookie will publish 'HOST-10-18-40-12:8020' and content as the actual 'HOST-10-18-40-35:8020'. Here I could see one conflicts, say when a new bookie joins with actual hostname as 'HOST-10-18-40-12' and port 8020. Support hostname based ledger metadata to help users to change IP with exitsting installation - Key: BOOKKEEPER-629 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-629 Project: Bookkeeper Issue Type: Sub-task Components: bookkeeper-auto-recovery, bookkeeper-client, bookkeeper-server Affects Versions: 4.2.1, 4.3.0 Reporter: Vinay Assignee: Rakesh R Fix For: 4.3.0 Attachments: 1-BOOKKEEPER-629.patch, 2-BOOKKEEPER-629.patch, 3-BOOKKEEPER-629.patch, 4-BOOKKEEPER-629.patch, 5-BOOKKEEPER-629.patch, 6-BOOKKEEPER-629.patch Register the bookie with *hostname:port* and also store the bookie addresses as *hostname:port* in ledger metadata files instead of *ip:port* This will help users to change the machine IP if they want without loosing their data. Supporting hostname based installation/functionality is one of the important requirement of users. Any thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-630) Add tag to o.a.b.net.* to indict which release of hadoop they came from, move DNS to o.a.b.net.* and indent
[ https://issues.apache.org/jira/browse/BOOKKEEPER-630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated BOOKKEEPER-630: -- Fix Version/s: 4.2.2 Add tag to o.a.b.net.* to indict which release of hadoop they came from, move DNS to o.a.b.net.* and indent --- Key: BOOKKEEPER-630 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-630 Project: Bookkeeper Issue Type: Improvement Reporter: Flavio Junqueira Assignee: Ivan Kelly Priority: Trivial Fix For: 4.2.2, 4.3.0 We didn't reformat the code before bringing it in from hadoop commons in the patch of BOOKKEEPER-618. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-629) Support hostname based ledger metadata to help users to change IP with exitsting installation
[ https://issues.apache.org/jira/browse/BOOKKEEPER-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13697640#comment-13697640 ] Rakesh R commented on BOOKKEEPER-629: - Sorry, please ignore the above example, its not correct. So if I understand correctly, here the idea is to keep the identifier as 'hostname:port' format. For example, one bookie starts with the id 'HOST-10-18-40-30:8020', when user change the hostname to 'HOST-10-18-40-35', this bookie will publish 'HOST-10-18-40-30:8020' and content as the actual 'HOST-10-18-40-35:8020'. Here I could see one conflicts, say when a new bookie wants to joins with its actual hostname as 'HOST-10-18-40-30' and port 8020, he will get ZNodeExistsException. Support hostname based ledger metadata to help users to change IP with exitsting installation - Key: BOOKKEEPER-629 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-629 Project: Bookkeeper Issue Type: Sub-task Components: bookkeeper-auto-recovery, bookkeeper-client, bookkeeper-server Affects Versions: 4.2.1, 4.3.0 Reporter: Vinay Assignee: Rakesh R Fix For: 4.3.0 Attachments: 1-BOOKKEEPER-629.patch, 2-BOOKKEEPER-629.patch, 3-BOOKKEEPER-629.patch, 4-BOOKKEEPER-629.patch, 5-BOOKKEEPER-629.patch, 6-BOOKKEEPER-629.patch Register the bookie with *hostname:port* and also store the bookie addresses as *hostname:port* in ledger metadata files instead of *ip:port* This will help users to change the machine IP if they want without loosing their data. Supporting hostname based installation/functionality is one of the important requirement of users. Any thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-630) Add tag to o.a.b.net.* to indict which release of hadoop they came from, move DNS to o.a.b.net.* and indent
[ https://issues.apache.org/jira/browse/BOOKKEEPER-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13697656#comment-13697656 ] Ivan Kelly commented on BOOKKEEPER-630: --- my release, i mean the hadoop-commons release btw. Add tag to o.a.b.net.* to indict which release of hadoop they came from, move DNS to o.a.b.net.* and indent --- Key: BOOKKEEPER-630 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-630 Project: Bookkeeper Issue Type: Improvement Reporter: Flavio Junqueira Assignee: Ivan Kelly Priority: Trivial Fix For: 4.2.2, 4.3.0 We didn't reformat the code before bringing it in from hadoop commons in the patch of BOOKKEEPER-618. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-588) SSL support
[
https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13697654#comment-13697654
]
Ivan Kelly commented on BOOKKEEPER-588:
---
{quote}if you checked any existed services
(https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers), they would
support both secure and non-secure ports. for example, imap port is 143, while
SSL/TLS encrypted IMAP used port 993.{quote}
imaps, pops and smtps are older implementations which simply tunnel over an ssl
socket. starttls versions superceded them.
The TLS with IMAP and POP rfc gives the rationale for this:
https://tools.ietf.org/html/rfc2595#section-7
Not all apply to us, but the url scheme issue is just another way to stating
the ids problem.
{quote}I don't think mixing non-ssl port with ssl port together is a good
practice, since it would make debugging and troubleshooting very
complicated.{quote}
How is it harder to debug? We currently don't decode bk wire transmissions, and
doing so with any form of SSL would be a pain anyhow. Once it hits the bookie,
debugging is no more difficult. In fact, I would argue that is makes debugging
and troubleshooting easier, as it halves the number of ports you need to check
are working.
{quote}
But if separating non-ssl port from ssl port, it's straightforward for any
clients to disable ssl port without paying any costs.
{quote}
The client will have a useSSL config option no matter which ssl option we go
with.
{quote}
no, if a bookie has been non-ssl only in the past? it should start with
previous installation since the cookie already has its previous identifier. so
any bookie client connects to this bookie would only use its non-ssl port.
if a bookie wants to upgrade to enable ssl support, it needs to run an admin
tool provided in BOOKKEEPER-634 to change its identifier. Changing the
identifier is somehow needed by BOOKKEEPER-639, we could leverage the tasks in
BOOKKEEPER-634 to achieve it.
{quote}
I think this is a unnecessary complication when we have the starttls option
available to us. I'm not saying we shouldn't have them for BOOKKEEPER-639, but
their use should be minimized to avoid side effects.
SSL support
---
Key: BOOKKEEPER-588
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-588
Project: Bookkeeper
Issue Type: Sub-task
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Fix For: 4.3.0
Attachments: 0004-BOOKKEEPER-588-SSL-support-for-bookkeeper.patch
SSL support using startTLS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-629) Support hostname based ledger metadata to help users to change IP with exitsting installation
[ https://issues.apache.org/jira/browse/BOOKKEEPER-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13697671#comment-13697671 ] Ivan Kelly commented on BOOKKEEPER-629: --- Actually what we could do is register the bookie as /ledgers/active/uuid-host/ip-port. This removes the extra lookup. The uuid could then be used in the ledger metadata. uuid and port could be used in the cookie. The issues Sijie raised about BC is valid. However, fixing for this should be the same as fixing for hostname. However, in this case, the main usecase is VM appliances, so BC shouldn't be a huge issue (we should provide tools to change a existing ip/hostname in any case). Support hostname based ledger metadata to help users to change IP with exitsting installation - Key: BOOKKEEPER-629 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-629 Project: Bookkeeper Issue Type: Sub-task Components: bookkeeper-auto-recovery, bookkeeper-client, bookkeeper-server Affects Versions: 4.2.1, 4.3.0 Reporter: Vinay Assignee: Rakesh R Fix For: 4.3.0 Attachments: 1-BOOKKEEPER-629.patch, 2-BOOKKEEPER-629.patch, 3-BOOKKEEPER-629.patch, 4-BOOKKEEPER-629.patch, 5-BOOKKEEPER-629.patch, 6-BOOKKEEPER-629.patch Register the bookie with *hostname:port* and also store the bookie addresses as *hostname:port* in ledger metadata files instead of *ip:port* This will help users to change the machine IP if they want without loosing their data. Supporting hostname based installation/functionality is one of the important requirement of users. Any thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-637) NoSuchEntry exception when reading an entry from a bookie should not print ERROR level message
[ https://issues.apache.org/jira/browse/BOOKKEEPER-637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated BOOKKEEPER-637: -- Fix Version/s: 4.3.0 4.2.2 NoSuchEntry exception when reading an entry from a bookie should not print ERROR level message -- Key: BOOKKEEPER-637 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-637 Project: Bookkeeper Issue Type: Improvement Affects Versions: 4.2.1, 4.3.0 Reporter: Matteo Merli Assignee: Matteo Merli Priority: Trivial Fix For: 4.2.2, 4.3.0 Attachments: BOOKKEEPER-637.diff The NoSuchEntry is an internal error that is recoverable within the BK client library by issuing a read to a different bookie. I think that INFO level should be more appropriate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-632) AutoRecovery should consider read only bookies
[ https://issues.apache.org/jira/browse/BOOKKEEPER-632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698005#comment-13698005 ] Ivan Kelly commented on BOOKKEEPER-632: --- The whitespace is actually in the comment for BookKeeperAdmin#getReadOnlyBookie. I've actually added a tool to find this, BOOKKEEPER-635. It should make this easier to find in future. The patch adds a new public api to read the readonly bookies, then manually reads them in Auditor#getAvailableBookies(). It would be much better if this method read the bookies through BookKeeperAdmin's apis, so that if we change the format in future, we have less places to update. AutoRecovery should consider read only bookies -- Key: BOOKKEEPER-632 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-632 Project: Bookkeeper Issue Type: Bug Components: bookkeeper-auto-recovery, bookkeeper-server Affects Versions: 4.2.1, 4.3.0 Reporter: Vinay Assignee: Vinay Fix For: 4.2.2, 4.3.0 Attachments: BOOKKEEPER-632.patch, BOOKKEEPER-632.patch Autorecovery Auditor should consider the readonly bookies as Available Bookies while publishing the under-replicated ledgers. Also AutoRecoveryDaemon should shutdown if the local bookie is readonly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-636) Latest txn logs might be deleted in a race condition which is not recoverable if BK goes down before next txn log created.
[
https://issues.apache.org/jira/browse/BOOKKEEPER-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698024#comment-13698024
]
Hadoop QA commented on BOOKKEEPER-636:
--
Testing JIRA BOOKKEEPER-636
Patch
[0001-BOOKKEEPER-636-Latest-txn-logs-might-be-deleted-in-a.patch|https://issues.apache.org/jira/secure/attachment/12590465/0001-BOOKKEEPER-636-Latest-txn-logs-might-be-deleted-in-a.patch]
downloaded at Tue Jul 2 17:11:30 UTC 2013
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
120
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 FINDBUGS{color}
.{color:green}+1{color} the patch does not seem to introduce new Findbugs
warnings
{color:green}+1 TESTS{color}
.Tests run: 853
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/bookkeeper-trunk-precommit-build/401/
Latest txn logs might be deleted in a race condition which is not recoverable
if BK goes down before next txn log created.
--
Key: BOOKKEEPER-636
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-636
Project: Bookkeeper
Issue Type: Bug
Components: bookkeeper-server
Affects Versions: 4.2.1, 4.3.0
Reporter: Vinay
Assignee: Vinay
Priority: Blocker
Fix For: 4.2.2, 4.3.0
Attachments:
0001-BOOKKEEPER-636-Latest-txn-logs-might-be-deleted-in-a.patch,
BOOKKEEPER-636.diff, BOOKKEEPER-636.diff, BOOKKEEPER-636.patch,
BOOKKEEPER-636.patch
With the following scenario latest transaction log can be deleted.
1. more than {{journalMaxBackups}} txn logs are there in journal dir.
2. BK machine was up for long time and the latest txn log id is some what
huge number
3. Now reboot the machine.
4. after reboot BK restarted.
5. Now, Immediately after startup, One entry is added, due to which Synthread
rolled the lastMark in ledger dirs before the lastLogId updated by Journal
thread. (this lastMark was having the old logId which was before reboot).
6. Now after roll, old journal txn logs were gc'ed. *Now latest created the
txn log was deleted.*
7. After this Journal thread updated the lastLogMark, also some more rolls
happened.
8. Now BK restarted again. But BK was not able to start because it was not
able to find the latest txn log file in journal directory.
{noformat}java.io.IOException: Recovery log 264564 is missing
at org.apache.bookkeeper.bookie.Journal.replay(Journal.java:424)
at org.apache.bookkeeper.bookie.Bookie.readJournal(Bookie.java:547)
at org.apache.bookkeeper.bookie.Bookie.start(Bookie.java:603)
at
org.apache.bookkeeper.proto.BookieServer.start(BookieServer.java:111){noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-618) Better resolution of bookie address
[
https://issues.apache.org/jira/browse/BOOKKEEPER-618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698049#comment-13698049
]
Hadoop QA commented on BOOKKEEPER-618:
--
Testing JIRA BOOKKEEPER-618
Patch
[BOOKKEEPER-618.branch4-2.diff|https://issues.apache.org/jira/secure/attachment/12590472/BOOKKEEPER-618.branch4-2.diff]
downloaded at Tue Jul 2 18:01:22 UTC 2013
{color:red}-1{color} Patch failed to apply to head of branch
Better resolution of bookie address
---
Key: BOOKKEEPER-618
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-618
Project: Bookkeeper
Issue Type: Improvement
Components: bookkeeper-server
Affects Versions: 4.2.1
Reporter: Bryan Beaudreault
Assignee: Ivan Kelly
Priority: Minor
Fix For: 4.2.2, 4.3.0
Attachments:
0001-BOOKKEEPER-618-Better-resolution-of-bookie-address.patch,
BOOKKEEPER-618.branch4-2.diff, BOOKKEEPER-618.branch4-2.diff
Bookie#getBookieAddress uses the following code:
{code:title=Bookie.java}
/**
* Return the configured address of the bookie.
*/
public static InetSocketAddress getBookieAddress(ServerConfiguration conf)
throws UnknownHostException {
return new InetSocketAddress(InetAddress.getLocalHost()
.getHostAddress(), conf.getBookiePort());
}
{code}
This code is subject to the contents of one's /etc/hosts file, in that if
they have an entry like {{127.0.0.1 myhostname}}, this method will return the
same 127.0.0.1 address on all bookie servers. This causes conflicts due to
the way bookies register in zookeeper.
There should be an optional bk_server.conf setting to allow one to select
their preferred network interface to use for the bookie. Then you could use
something like
{{NetworkInterface.getByName(PREFERRED_INTERFACE).getInetAddresses()}}
instead. This method is not effected by the /etc/hosts.
An alternative method of registering the bookie that does not rely on the
local address would be another possible solution, such as using the DNS like
other apache projects (hbase).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-641) DeathWatcher thread is unnecessarily running even after bookie shutdown
[ https://issues.apache.org/jira/browse/BOOKKEEPER-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sijie Guo updated BOOKKEEPER-641: - Fix Version/s: 4.2.2 DeathWatcher thread is unnecessarily running even after bookie shutdown --- Key: BOOKKEEPER-641 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-641 Project: Bookkeeper Issue Type: Bug Components: bookkeeper-server Affects Versions: 4.2.1 Reporter: Rakesh R Assignee: Rakesh R Priority: Minor Fix For: 4.2.2, 4.3.0 Attachments: 001-BOOKKEEPER-641.patch I've seen in our testcases, DeathWatcher threads are continue running even after bkServer#shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-631) RequestProcessor Proposal
[
https://issues.apache.org/jira/browse/BOOKKEEPER-631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698456#comment-13698456
]
Sijie Guo commented on BOOKKEEPER-631:
--
{code}
To me, it looks like if you get an exception in a middleware, you won't get any
response.
Also, are requests sent to all middlewares every time?
{code}
if there is any exception throwing in processing the request, it stopped
propagate the requests and triggered exception callback and respond the
response.
RequestProcessor Proposal
-
Key: BOOKKEEPER-631
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-631
Project: Bookkeeper
Issue Type: New Feature
Components: bookkeeper-client, bookkeeper-server
Reporter: Sijie Guo
This JIRA tries to propose a *Middleware* concept, which is a kind of
request/response handler to intercept bookie request/response flow for
different purposes.
Different *middleware*s serve different purpose:
- *StatsMiddleware* is to collect stats of requests
- *AuthMiddleware* is to authenticate requests
- *ACLMiddleware* is to serve authorization.
- any customized middleware could be added to serve their request
intercepting.
the middlewares are loaded from configuration to process request/response in
order.
request - (middleware 1) - (middleware 2) - (middleware N) - response
each middleware could decide: whether it could process the request or not? if
it can't process, it passes the request to its downstream middleware. if it
could, processes the request and decide whether to pass the request to
downstream or not.
for example, an auth bookie could load two middlewares:
request - AuthMiddleware - BookieMiddleware - response
The *AuthMiddleware* could intercept *authenticate* requests or requests with
*authenticate* information (such as Token). if the request is authenticated,
pass it to BookieMiddleware to process the requests; otherwise, it stopped
and respond with EUA response.
A non-auth bookie could just load BookieMiddleware without any authentication.
prototype of this idea in github. the interface in the prototype is not
finalized, since the middleware concept is quite similar as netty channel
handler. I am thinking how to consolidate them.
https://github.com/sijie/bookkeeper/commit/d23df97b209170852f2ce6676a49c97e72ecb2ee
a token-based authentication middleware example:
https://github.com/sijie/bookkeeper/tree/middlewares/bookkeeper-server/src/main/java/org/apache/bookkeeper/security/token
or if you want to make authentication flow like this:
- client instantiates a connection
- client sends credential first
- after client verified the credential, all the following requests are
authenticated.
you could implement a middleware maintaining all incoming requests, if the
first message is not credential, respond EUA and close the channel, if the
first message is credential message and it is authenticated, mark this
channel as authenticated and bypass all its following requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (BOOKKEEPER-629) Support hostname based ledger metadata to help users to change IP with exitsting installation
[
https://issues.apache.org/jira/browse/BOOKKEEPER-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698459#comment-13698459
]
Sijie Guo edited comment on BOOKKEEPER-629 at 7/3/13 12:48 AM:
---
{quote}
So if I understand correctly, here the idea is to keep the identifier as
'hostname:port' format. For example, one bookie starts with the id
'HOST-10-18-40-30:8020', when user change the hostname to 'HOST-10-18-40-35',
this bookie will publish 'HOST-10-18-40-30:8020' and content as the actual
'HOST-10-18-40-35:8020'. Here I could see one conflicts, say when a new bookie
wants to joins with its actual hostname as 'HOST-10-18-40-30' and port 8020, he
will get ZNodeExistsException.
{quote}
unless you had a global id assignment, you had resolved this issue in uuid case.
for me, this solution is making confused about the identifier, you had no idea
about it is an old version bookie or a new version bookie. actually I don't
like the proposal I raised.
so if there is no clear solution for backward compatibility, I would suggest
keeping current identifier format and providing an admin tool to change
identifier. Let's do the generic solution util there is a big requirement and
have to do it. for now keep things simple.
was (Author: hustlmsp):
{quote}
So if I understand correctly, here the idea is to keep the identifier as
'hostname:port' format. For example, one bookie starts with the id
'HOST-10-18-40-30:8020', when user change the hostname to 'HOST-10-18-40-35',
this bookie will publish 'HOST-10-18-40-30:8020' and content as the actual
'HOST-10-18-40-35:8020'. Here I could see one conflicts, say when a new bookie
wants to joins with its actual hostname as 'HOST-10-18-40-30' and port 8020, he
will get ZNodeExistsException.
{quote}
unless you had a global id assignment, you had resolved this issue in uuid case.
for me, this solution is making confused about the identifier, you had no idea
about it is an old version bookie or a new version bookie. actually I don't
like the proposal I raised.
so if there is no clear solution for backward compatibility, I would suggest
keeping current identifier format and providing an admin tool to change
identifier. Let's do the generic solution util there is a big requirement and
have to do it. keep things simple.
Support hostname based ledger metadata to help users to change IP with
exitsting installation
-
Key: BOOKKEEPER-629
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-629
Project: Bookkeeper
Issue Type: Sub-task
Components: bookkeeper-auto-recovery, bookkeeper-client,
bookkeeper-server
Affects Versions: 4.2.1, 4.3.0
Reporter: Vinay
Assignee: Rakesh R
Fix For: 4.3.0
Attachments: 1-BOOKKEEPER-629.patch, 2-BOOKKEEPER-629.patch,
3-BOOKKEEPER-629.patch, 4-BOOKKEEPER-629.patch, 5-BOOKKEEPER-629.patch,
6-BOOKKEEPER-629.patch
Register the bookie with *hostname:port* and also store the bookie addresses
as *hostname:port* in ledger metadata files instead of *ip:port*
This will help users to change the machine IP if they want without loosing
their data.
Supporting hostname based installation/functionality is one of the important
requirement of users.
Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-629) Support hostname based ledger metadata to help users to change IP with exitsting installation
[
https://issues.apache.org/jira/browse/BOOKKEEPER-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698459#comment-13698459
]
Sijie Guo commented on BOOKKEEPER-629:
--
{quote}
So if I understand correctly, here the idea is to keep the identifier as
'hostname:port' format. For example, one bookie starts with the id
'HOST-10-18-40-30:8020', when user change the hostname to 'HOST-10-18-40-35',
this bookie will publish 'HOST-10-18-40-30:8020' and content as the actual
'HOST-10-18-40-35:8020'. Here I could see one conflicts, say when a new bookie
wants to joins with its actual hostname as 'HOST-10-18-40-30' and port 8020, he
will get ZNodeExistsException.
{quote}
unless you had a global id assignment, you had resolved this issue in uuid case.
for me, this solution is making confused about the identifier, you had no idea
about it is an old version bookie or a new version bookie. actually I don't
like the proposal I raised.
so if there is no clear solution for backward compatibility, I would suggest
keeping current identifier format and providing an admin tool to change
identifier. Let's do the generic solution util there is a big requirement and
have to do it. keep things simple.
Support hostname based ledger metadata to help users to change IP with
exitsting installation
-
Key: BOOKKEEPER-629
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-629
Project: Bookkeeper
Issue Type: Sub-task
Components: bookkeeper-auto-recovery, bookkeeper-client,
bookkeeper-server
Affects Versions: 4.2.1, 4.3.0
Reporter: Vinay
Assignee: Rakesh R
Fix For: 4.3.0
Attachments: 1-BOOKKEEPER-629.patch, 2-BOOKKEEPER-629.patch,
3-BOOKKEEPER-629.patch, 4-BOOKKEEPER-629.patch, 5-BOOKKEEPER-629.patch,
6-BOOKKEEPER-629.patch
Register the bookie with *hostname:port* and also store the bookie addresses
as *hostname:port* in ledger metadata files instead of *ip:port*
This will help users to change the machine IP if they want without loosing
their data.
Supporting hostname based installation/functionality is one of the important
requirement of users.
Any thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-588) SSL support
[
https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13698470#comment-13698470
]
Sijie Guo commented on BOOKKEEPER-588:
--
{code}
imaps, pops and smtps are older implementations which simply tunnel over an ssl
socket. starttls versions superceded them.
The TLS with IMAP and POP rfc gives the rationale for this:
https://tools.ietf.org/html/rfc2595#section-7
{code}
ok. sounds true for IMAP and POP.
{code}
How is it harder to debug? We currently don't decode bk wire transmissions, and
doing so with any form of SSL would be a pain anyhow. Once it hits the bookie,
debugging is no more difficult. In fact, I would argue that is makes debugging
and troubleshooting easier, as it halves the number of ports you need to check
are working.
{code}
1. it is hard to figure out what bookie is running ssl or what bookie is not
running ssl, when you are looking into a ledger metadata or /ledger/available
znode (if using different port, it is easy to know).
2. it isn't straightforward to dump ssl non-ssl mixed stream. that's the part
what I meant for troubleshooting.
as my view, an additional port might make things clear and manageable. but just
my view. I don't have any strong preference.
SSL support
---
Key: BOOKKEEPER-588
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-588
Project: Bookkeeper
Issue Type: Sub-task
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Fix For: 4.3.0
Attachments: 0004-BOOKKEEPER-588-SSL-support-for-bookkeeper.patch
SSL support using startTLS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
