[jira] [Commented] (ZOOKEEPER-1366) Zookeeper should be tolerant of clock adjustments
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13575664#comment-13575664 ] Benjamin Reed commented on ZOOKEEPER-1366: -- i was just reviewing this issue and reading my comments that i had recently made. and they sounded like me but i couldn't remember making them last month. then i realized that i made them in 2012!!! we really need to get this patch in. is someone working on it? Zookeeper should be tolerant of clock adjustments - Key: ZOOKEEPER-1366 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1366 Project: ZooKeeper Issue Type: Bug Reporter: Ted Dunning Assignee: Ted Dunning Fix For: 3.5.0 Attachments: ZOOKEEPER-1366-3.3.3.patch, ZOOKEEPER-1366.patch, ZOOKEEPER-1366.patch, ZOOKEEPER-1366.patch, ZOOKEEPER-1366.patch If you want to wreak havoc on a ZK based system just do [date -s +1hour] and watch the mayhem as all sessions expire at once. This shouldn't happen. Zookeeper could easily know handle elapsed times as elapsed times rather than as differences between absolute times. The absolute times are subject to adjustment when the clock is set while a timer is not subject to this problem. In Java, System.currentTimeMillis() gives you absolute time while System.nanoTime() gives you time based on a timer from an arbitrary epoch. I have done this and have been running tests now for some tens of minutes with no failures. I will set up a test machine to redo the build again on Ubuntu and post a patch here for discussion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1382) Zookeeper server holds onto dead/expired session ids in the watch data structures
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13575670#comment-13575670 ] Michael Morello commented on ZOOKEEPER-1382: Since we monitor sessions and watches we note that we encounter this problem from time to time. This is not a big issue but it causes a memory leak. I'll try to see if I can work on it but it does seem hard to reproduce in a unit test. Zookeeper server holds onto dead/expired session ids in the watch data structures - Key: ZOOKEEPER-1382 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1382 Project: ZooKeeper Issue Type: Bug Components: server Affects Versions: 3.3.4 Reporter: Neha Narkhede Assignee: Neha Narkhede Priority: Critical Fix For: 3.4.6 Attachments: ZOOKEEPER-1382_3.3.4.patch I've observed that zookeeper server holds onto expired session ids in the watcher data structures. The result is the wchp command reports session ids that cannot be found through cons/dump and those expired session ids sit there maybe until the server is restarted. Here are snippets from the client and the server logs that lead to this state, for one particular session id 0x134485fd7bcb26f - There are 4 servers in the zookeeper cluster - 223, 224, 225 (leader), 226 and I'm using ZkClient to connect to the cluster From the application log - application.log.2012-01-26-325.gz:2012/01/26 04:56:36.177 INFO [ClientCnxn] [main-SendThread(223.prod:12913)] [application Session establishment complete on server 223.prod/172.17.135.38:12913, sessionid = 0x134485fd7bcb26f, negotiated timeout = 6000 application.log.2012-01-27.gz:2012/01/27 09:52:37.714 INFO [ClientCnxn] [main-SendThread(223.prod:12913)] [application] Client session timed out, have not heard from server in 9827ms for sessionid 0x134485fd7bcb26f, closing socket connection and attempting reconnect application.log.2012-01-27.gz:2012/01/27 09:52:38.191 INFO [ClientCnxn] [main-SendThread(226.prod:12913)] [application] Unable to reconnect to ZooKeeper service, session 0x134485fd7bcb26f has expired, closing socket connection On the leader zk, 225 - zookeeper.log.2012-01-27-leader-225.gz:2012-01-27 09:52:34,010 - INFO [SessionTracker:ZooKeeperServer@314] - Expiring session 0x134485fd7bcb26f, timeout of 6000ms exceeded zookeeper.log.2012-01-27-leader-225.gz:2012-01-27 09:52:34,010 - INFO [ProcessThread:-1:PrepRequestProcessor@391] - Processed session termination for sessionid: 0x134485fd7bcb26f On the server, the client was initially connected to, 223 - zookeeper.log.2012-01-26-223.gz:2012-01-26 04:56:36,173 - INFO [CommitProcessor:1:NIOServerCnxn@1580] - Established session 0x134485fd7bcb26f with negotiated timeout 6000 for client /172.17.136.82:45020 zookeeper.log.2012-01-27-223.gz:2012-01-27 09:52:34,018 - INFO [CommitProcessor:1:NIOServerCnxn@1435] - Closed socket connection for client /172.17.136.82:45020 which had sessionid 0x134485fd7bcb26f Here are the log snippets from 226, which is the server, the client reconnected to, before getting session expired event - 2012-01-27 09:52:38,190 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:12913:NIOServerCnxn@770] - Client attempting to renew session 0x134485fd7bcb26f at /172.17.136.82:49367 2012-01-27 09:52:38,191 - INFO [QuorumPeer:/0.0.0.0:12913:NIOServerCnxn@1573] - Invalid session 0x134485fd7bcb26f for client /172.17.136.82:49367, probably expired 2012-01-27 09:52:38,191 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:12913:NIOServerCnxn@1435] - Closed socket connection for client /172.17.136.82:49367 which had sessionid 0x134485fd7bcb26f wchp output from 226, taken on 01/30 - nnarkhed-ld:zk-cons-wchp-2012013000 nnarkhed$ grep 0x134485fd7bcb26f *226.*wchp* | wc -l 3 wchp output from 223, taken on 01/30 - nnarkhed-ld:zk-cons-wchp-2012013000 nnarkhed$ grep 0x134485fd7bcb26f *223.*wchp* | wc -l 0 cons output from 223 and 226, taken on 01/30 - nnarkhed-ld:zk-cons-wchp-2012013000 nnarkhed$ grep 0x134485fd7bcb26f *226.*cons* | wc -l 0 nnarkhed-ld:zk-cons-wchp-2012013000 nnarkhed$ grep 0x134485fd7bcb26f *223.*cons* | wc -l 0 So, what seems to have happened is that the client was able to re-register the watches on the new server (226), after it got disconnected from 223, inspite of having an expired session id. In NIOServerCnxn, I saw that after suspecting that a session is expired, a server removes the cnxn and its watches from its internal data structures. But before that it allows more requests to be processed even if the session is expired - // Now that the session is ready we can start receiving packets
ZooKeeper-trunk-solaris - Build # 467 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk-solaris/467/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 163819 lines...] [junit] 2013-02-11 10:02:26,924 [myid:] - INFO [main:SessionTrackerImpl@180] - Shutting down [junit] 2013-02-11 10:02:26,925 [myid:] - INFO [main:PrepRequestProcessor@804] - Shutting down [junit] 2013-02-11 10:02:26,925 [myid:] - INFO [main:SyncRequestProcessor@175] - Shutting down [junit] 2013-02-11 10:02:26,925 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@144] - PrepRequestProcessor exited loop! [junit] 2013-02-11 10:02:26,925 [myid:] - INFO [SyncThread:0:SyncRequestProcessor@155] - SyncRequestProcessor exited! [junit] 2013-02-11 10:02:26,925 [myid:] - INFO [main:FinalRequestProcessor@421] - shutdown of request processor complete [junit] 2013-02-11 10:02:26,926 [myid:] - INFO [main:FourLetterWordMain@43] - connecting to 127.0.0.1 11221 [junit] 2013-02-11 10:02:26,926 [myid:] - INFO [main:JMXEnv@133] - ensureOnly:[] [junit] 2013-02-11 10:02:26,927 [myid:] - INFO [main:ClientBase@414] - STARTING server [junit] 2013-02-11 10:02:26,927 [myid:] - INFO [main:ZooKeeperServer@149] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 6 datadir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/trunk/build/test/tmp/test1210265611741651528.junit.dir/version-2 snapdir /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/trunk/build/test/tmp/test1210265611741651528.junit.dir/version-2 [junit] 2013-02-11 10:02:26,928 [myid:] - INFO [main:NIOServerCnxnFactory@663] - Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 16 worker threads, and 64 kB direct buffers. [junit] 2013-02-11 10:02:26,928 [myid:] - INFO [main:NIOServerCnxnFactory@676] - binding to port 0.0.0.0/0.0.0.0:11221 [junit] 2013-02-11 10:02:26,930 [myid:] - INFO [main:FileSnap@83] - Reading snapshot /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/trunk/build/test/tmp/test1210265611741651528.junit.dir/version-2/snapshot.b [junit] 2013-02-11 10:02:26,932 [myid:] - INFO [main:FileTxnSnapLog@270] - Snapshotting: 0xb to /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/ZooKeeper-trunk-solaris/trunk/build/test/tmp/test1210265611741651528.junit.dir/version-2/snapshot.b [junit] 2013-02-11 10:02:26,933 [myid:] - INFO [main:FourLetterWordMain@43] - connecting to 127.0.0.1 11221 [junit] 2013-02-11 10:02:26,933 [myid:] - INFO [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11221:NIOServerCnxnFactory$AcceptThread@289] - Accepted socket connection from /127.0.0.1:33378 [junit] 2013-02-11 10:02:26,934 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@829] - Processing stat command from /127.0.0.1:33378 [junit] 2013-02-11 10:02:26,934 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn$StatCommand@678] - Stat command output [junit] 2013-02-11 10:02:26,935 [myid:] - INFO [NIOWorkerThread-1:NIOServerCnxn@1000] - Closed socket connection for client /127.0.0.1:33378 (no session established for client) [junit] 2013-02-11 10:02:26,935 [myid:] - INFO [main:JMXEnv@133] - ensureOnly:[InMemoryDataTree, StandaloneServer_port] [junit] 2013-02-11 10:02:26,936 [myid:] - INFO [main:JMXEnv@105] - expect:InMemoryDataTree [junit] 2013-02-11 10:02:26,936 [myid:] - INFO [main:JMXEnv@108] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=InMemoryDataTree [junit] 2013-02-11 10:02:26,936 [myid:] - INFO [main:JMXEnv@105] - expect:StandaloneServer_port [junit] 2013-02-11 10:02:26,936 [myid:] - INFO [main:JMXEnv@108] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port-1 [junit] 2013-02-11 10:02:26,937 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@57] - FINISHED TEST METHOD testQuota [junit] 2013-02-11 10:02:26,937 [myid:] - INFO [main:ClientBase@451] - tearDown starting [junit] 2013-02-11 10:02:27,000 [myid:] - INFO [SessionTracker:SessionTrackerImpl@135] - SessionTrackerImpl exited loop! [junit] 2013-02-11 10:02:27,001 [myid:] - INFO [SessionTracker:SessionTrackerImpl@135] - SessionTrackerImpl exited loop! [junit] 2013-02-11 10:02:27,012 [myid:] - INFO [main:ZooKeeper@744] - Session: 0x13cc8b4a0a2 closed [junit] 2013-02-11 10:02:27,012 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down [junit] 2013-02-11 10:02:27,012 [myid:] - INFO [main:ClientBase@421] - STOPPING server [junit] 2013-02-11 10:02:27,013 [myid:] - INFO
ZooKeeper-trunk-WinVS2008 - Build # 719 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk-WinVS2008/719/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 92 lines...] generate_jute_parser: [mkdir] Created dir: f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\build\jute_compiler\org\apache\jute\compiler\generated [ivy:artifactproperty] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead [ivy:artifactproperty] :: loading settings :: file = f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\ivysettings.xml [move] Moving 1 file to f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\build\lib [javacc] Java Compiler Compiler Version 5.0 (Parser Generator) [javacc] (type javacc with no arguments for help) [javacc] Reading from file f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\java\main\org\apache\jute\compiler\generated\rcc.jj . . . [javacc] File TokenMgrError.java does not exist. Will create one. [javacc] File ParseException.java does not exist. Will create one. [javacc] File Token.java does not exist. Will create one. [javacc] File SimpleCharStream.java does not exist. Will create one. [javacc] Parser generated successfully. jute: [javac] Compiling 39 source files to f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\build\classes compile_jute_uptodate: compile_jute: [mkdir] Created dir: f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\java\generated [mkdir] Created dir: f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\c\generated [java] ../../zookeeper.jute Parsed Successfully [java] ../../zookeeper.jute Parsed Successfully [touch] Creating f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\java\generated\.generated BUILD SUCCESSFUL Total time: 7 seconds [ZooKeeper-trunk-WinVS2008] $ cmd /c call C:\Users\hudson\AppData\Local\Temp\hudson7171710232071093615.bat f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008set ZOOKEEPER_HOME=f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008msbuild trunk/src/c/zookeeper.sln /p:Configuration=Release Microsoft (R) Build Engine Version 3.5.30729.1 [Microsoft .NET Framework, Version 2.0.50727.4223] Copyright (C) Microsoft Corporation 2007. All rights reserved. Build started 2/11/2013 10:11:51 AM. Project f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\c\zookeeper.sln on node 0 (default targets). Building solution configuration Release|Win32. f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\c\zookeeper.vcxproj(3,14): error MSB4066: The attribute Label in element ItemGroup is unrecognized. Done Building Project f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\c\zookeeper.sln (default targets) -- FAILED. Build FAILED. f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\c\zookeeper.sln (default target) (1) - (zookeeper target) - f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\c\zookeeper.vcxproj(3,14): error MSB4066: The attribute Label in element ItemGroup is unrecognized. 0 Warning(s) 1 Error(s) Time Elapsed 00:00:04.26 f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008exit 1 Build step 'Execute Windows batch command' marked build as failure Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
ZooKeeper-trunk-WinVS2008 - Build # 720 - Still Failing
See https://builds.apache.org/job/ZooKeeper-trunk-WinVS2008/720/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 94 lines...] generate_jute_parser: [mkdir] Created dir: f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\build\jute_compiler\org\apache\jute\compiler\generated [ivy:artifactproperty] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead [ivy:artifactproperty] :: loading settings :: file = f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\ivysettings.xml [move] Moving 1 file to f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\build\lib [javacc] Java Compiler Compiler Version 5.0 (Parser Generator) [javacc] (type javacc with no arguments for help) [javacc] Reading from file f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\java\main\org\apache\jute\compiler\generated\rcc.jj . . . [javacc] File TokenMgrError.java does not exist. Will create one. [javacc] File ParseException.java does not exist. Will create one. [javacc] File Token.java does not exist. Will create one. [javacc] File SimpleCharStream.java does not exist. Will create one. [javacc] Parser generated successfully. jute: [javac] Compiling 39 source files to f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\build\classes compile_jute_uptodate: compile_jute: [mkdir] Created dir: f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\java\generated [mkdir] Created dir: f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\c\generated [java] ../../zookeeper.jute Parsed Successfully [java] ../../zookeeper.jute Parsed Successfully [touch] Creating f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\java\generated\.generated BUILD SUCCESSFUL Total time: 6 seconds [ZooKeeper-trunk-WinVS2008] $ cmd /c call C:\Users\hudson\AppData\Local\Temp\hudson395269925251531586.bat f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008set ZOOKEEPER_HOME=f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008msbuild trunk/src/c/zookeeper.sln /p:Configuration=Release Microsoft (R) Build Engine Version 3.5.30729.1 [Microsoft .NET Framework, Version 2.0.50727.4223] Copyright (C) Microsoft Corporation 2007. All rights reserved. Build started 2/11/2013 11:54:14 AM. Project f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\c\zookeeper.sln on node 0 (default targets). Building solution configuration Release|Win32. f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\c\zookeeper.vcxproj(3,14): error MSB4066: The attribute Label in element ItemGroup is unrecognized. Done Building Project f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\c\zookeeper.sln (default targets) -- FAILED. Build FAILED. f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\c\zookeeper.sln (default target) (1) - (zookeeper target) - f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008\trunk\src\c\zookeeper.vcxproj(3,14): error MSB4066: The attribute Label in element ItemGroup is unrecognized. 0 Warning(s) 1 Error(s) Time Elapsed 00:00:02.81 f:\hudson\hudson-slave\workspace\ZooKeeper-trunk-WinVS2008exit 1 Build step 'Execute Windows batch command' marked build as failure Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Commented] (ZOOKEEPER-1643) Windows: fetch_and_add not 64bit-compatible, may not be correct
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576008#comment-13576008 ] Michi Mutsuzaki commented on ZOOKEEPER-1643: I didn't write the code, but I agree with Erik and Ben. - For windows, we should use InterlockedExchangeAdd. - For non-windows, we should use __sync_fetch_and_ad Erik, would you be interested in providing a patch? Thanks! --Michi Windows: fetch_and_add not 64bit-compatible, may not be correct --- Key: ZOOKEEPER-1643 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1643 Project: ZooKeeper Issue Type: Bug Components: c client Affects Versions: 3.3.3 Environment: Windows 7 Microsoft Visual Studio 2005 Reporter: Erik Anderson Note: While I am using a really old version of ZK, I did do enough SVN Blame operations to realize that this code hasn't changed. I am currently attempting to compile the C client under MSVC 2005 arch=x64. There are three things I can see with fetch_and_add() inside of /src/c/src/mt_adapter.c (1) MSVC 2005 64bit will not compile inline _asm sections. I'm moderately sure this code is x86-specific so I'm unsure whether it should attempt to either. (2) The Windows intrinsic InterlockedExchangeAdd [http://msdn.microsoft.com/en-us/library/windows/desktop/ms683597(v=vs.85).aspx] appears to do the same thing this code is attempting to do (3) I'm really rusty on my assembly, but why are we doing two separate XADD operations here, and is the code as-written anything approaching atomicity? If you want an official patch I likely can do an SVN checkout and submit a patch the replaces the entire #else on lines 495-505 with a return InterlockedExchangeAdd(operand, incr); Usually when I'm scratching my head this badly there's something I'm missing though. As far as I can tell there has been no prior discussion on this code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1642) Leader loading database twice
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576109#comment-13576109 ] Flavio Junqueira commented on ZOOKEEPER-1642: - Here is how I currently see it. The leader doesn't have to change its database instance after being elected, but a follower might need to change it, though. ZKDatabase is actually cleared in Learner#syncWithLeader(), ZooKeeperServer#shutdown(), ZKDatabase#deserializeSnapshot(), ZKDatabase#truncateLog(). In the case the follower requires changes, it will end up clearing the database and reloading it with the corresponding changes. About committedLog(), it is cleared in ZKDatabase#clear() and ZKDatabase#addCommittedProposal(). addCommittedProposal is invoked in ZKDatabase#loadDatabase() and FinalRequestProcessor#processRequest(). With the patch I'm proposing, these methods would be invoked in the same way when a follower needs to snap, take a diff, or truncate, so I don't expect a change of behavior. About a test, I'm happy to have it, but I'm still not very sure about what to test. Should we test that a server contains all commits it should? If so, I'm sure we have tests that do it already. Should we instead just check for the content of the committedLog? Leader loading database twice - Key: ZOOKEEPER-1642 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1642 Project: ZooKeeper Issue Type: Bug Reporter: Flavio Junqueira Fix For: 3.5.0, 3.4.6 Attachments: ZOOKEEPER-1642.patch The leader server currently loads the database before running leader election when trying to figure out the zxid it needs to use for the election and again when it starts leading. This is problematic for larger databases so we should remove the redundant load if possible. The code references are: # getLastLoggedZxid() in QuorumPeer; # loadData() in ZooKeeperServer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (ZOOKEEPER-1644) Add support for compressed SetWatches packet
Thawan Kooburat created ZOOKEEPER-1644: -- Summary: Add support for compressed SetWatches packet Key: ZOOKEEPER-1644 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1644 Project: ZooKeeper Issue Type: Improvement Components: c client, java client, server Reporter: Thawan Kooburat On reconnect with a server to restore its session, a client have to send all watched paths via SetWatches packet to the server. This can be potentially large and exceeded server-side buffer (jute.maxbuffer) causing the session to fail. We have 2 concerns. 1. We can increase jute.maxbuffer to arbitrarily size as a simple workaround, but, in our use case, the number of watch is going to keep growing 2. If a large number of clients get disconnected at once, the server may receive a large amount data over network because of the flood of SetWatches packet. In our case, the watch paths should by highly compressible. So our current plan is to add a new type of request which is a compressed set watch request. It should be possible to support multiple compression schemes. We are probably going to use snappy compression but may add support for gzip as a default to minimize external dependency requirement. Feel free to comment if you have any suggestion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1495) ZK client hangs when using a function not available on the server.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576219#comment-13576219 ] Ted Yu commented on ZOOKEEPER-1495: --- What about 1495.br33.v3.patch ? It would really useful for clients to see whether certain operation is supported. ZK client hangs when using a function not available on the server. -- Key: ZOOKEEPER-1495 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1495 Project: ZooKeeper Issue Type: Bug Components: server Affects Versions: 3.4.2, 3.3.5 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 3.5.0, 3.4.6 Attachments: 1495.br33.v3.patch, ZOOKEEPER-1495.2.patch, ZOOKEEPER-1495_branch34.patch, ZOOKEEPER-1495.patch This happens for example when using zk#multi with a 3.4 client but a 3.3 server. The issue seems to be on the server side: the servers drops the packets with an unknown OpCode in ZooKeeperServer#submitRequest {noformat} public void submitRequest(Request si) { // snip try { touch(si.cnxn); boolean validpacket = Request.isValid(si.type); // === Check on case OpCode.* if (validpacket) { // snip } else { LOG.warn(Dropping packet at server of type + si.type); // if invalid packet drop the packet. } } catch (MissingSessionException e) { if (LOG.isDebugEnabled()) { LOG.debug(Dropping request: + e.getMessage()); } } } {noformat} The solution discussed in ZOOKEEPER-1381 would be to get an exception on the client side then close the session. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1334) Zookeeper 3.4.x is not OSGi compliant - MANIFEST.MF is flawed
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576413#comment-13576413 ] Arnoud Glimmerveen commented on ZOOKEEPER-1334: --- I've tested this patch on a locally build zookeeper (3.4.5 + patch from ZOOKEEPER-1334) using Karaf 2.2.8 and 2.3.0 (both Equinox and Felix) with a simple test bundle that creates a ZooKeeper connection upon the bundle being started. In my tests this patch only works for the combination Karaf 2.2.8 + Equinox. In the other scenario's I ran into some NoClassDefFoundErrors on classes located in the packages {{javax.security.auth.callback}} , {{javax.security.auth.login}} and {{javax.security.sasl}} . I resolved this by adding these packages to the Import-Package section of the ZooKeeper bundle. @Claus: should these packages be added to the Import-Package section of the MANIFEST of the ZooKeeper bundle or are users expected to expose these packages to ZooKeeper bundle through OSGi framework configuration? Zookeeper 3.4.x is not OSGi compliant - MANIFEST.MF is flawed - Key: ZOOKEEPER-1334 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1334 Project: ZooKeeper Issue Type: Bug Affects Versions: 3.4.0 Reporter: Claus Ibsen Assignee: Claus Ibsen Fix For: 3.5.0, 3.4.6 Attachments: zookeeper-1334-osgi.patch, zookeeper-1334-osgi.patch, ZOOKEEPER-1334.patch In Zookeeper 3.3.x you use log4j for logging, and the maven dep is eg from 3.3.4 {code} dependency groupIdlog4j/groupId artifactIdlog4j/artifactId version1.2.15/version scopecompile/scope /dependency {code} Now in 3.4.0 or better you changed to use slf4j also/instead. The maven pom.xml now includes: {code} dependency groupIdorg.slf4j/groupId artifactIdslf4j-api/artifactId version1.6.1/version scopecompile/scope /dependency dependency groupIdorg.slf4j/groupId artifactIdslf4j-log4j12/artifactId version1.6.1/version scopecompile/scope /dependency dependency groupIdlog4j/groupId artifactIdlog4j/artifactId version1.2.15/version scopecompile/scope /dependency {code} But the META-INF/MANIFEST.MF file in the distribution did not change to reflect this. The 3.3.4 MANIFEST.MF, import packages {code} Import-Package: javax.management,org.apache.log4j,org.osgi.framework;v ersion=[1.4,2.0),org.osgi.util.tracker;version=[1.1,2.0) {code} And the 3.4.1 MANIFEST.MF, import packages: {code} Import-Package: javax.management,org.apache.log4j,org.osgi.framework;v ersion=[1.4,2.0),org.osgi.util.tracker;version=[1.1,2.0) {code} This makes using zookeeper 3.4.x in OSGi environments not possible, as we get NoClassDefFoundException for slf4j classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1334) Zookeeper 3.4.x is not OSGi compliant - MANIFEST.MF is flawed
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576415#comment-13576415 ] Patrick Hunt commented on ZOOKEEPER-1334: - Hi [~arnoud]. Please create a new jira for this issue, the original one has already been resolved. Thanks! Zookeeper 3.4.x is not OSGi compliant - MANIFEST.MF is flawed - Key: ZOOKEEPER-1334 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1334 Project: ZooKeeper Issue Type: Bug Affects Versions: 3.4.0 Reporter: Claus Ibsen Assignee: Claus Ibsen Fix For: 3.5.0, 3.4.6 Attachments: zookeeper-1334-osgi.patch, zookeeper-1334-osgi.patch, ZOOKEEPER-1334.patch In Zookeeper 3.3.x you use log4j for logging, and the maven dep is eg from 3.3.4 {code} dependency groupIdlog4j/groupId artifactIdlog4j/artifactId version1.2.15/version scopecompile/scope /dependency {code} Now in 3.4.0 or better you changed to use slf4j also/instead. The maven pom.xml now includes: {code} dependency groupIdorg.slf4j/groupId artifactIdslf4j-api/artifactId version1.6.1/version scopecompile/scope /dependency dependency groupIdorg.slf4j/groupId artifactIdslf4j-log4j12/artifactId version1.6.1/version scopecompile/scope /dependency dependency groupIdlog4j/groupId artifactIdlog4j/artifactId version1.2.15/version scopecompile/scope /dependency {code} But the META-INF/MANIFEST.MF file in the distribution did not change to reflect this. The 3.3.4 MANIFEST.MF, import packages {code} Import-Package: javax.management,org.apache.log4j,org.osgi.framework;v ersion=[1.4,2.0),org.osgi.util.tracker;version=[1.1,2.0) {code} And the 3.4.1 MANIFEST.MF, import packages: {code} Import-Package: javax.management,org.apache.log4j,org.osgi.framework;v ersion=[1.4,2.0),org.osgi.util.tracker;version=[1.1,2.0) {code} This makes using zookeeper 3.4.x in OSGi environments not possible, as we get NoClassDefFoundException for slf4j classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1334) Zookeeper 3.4.x is not OSGi compliant - MANIFEST.MF is flawed
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576427#comment-13576427 ] Claus Ibsen commented on ZOOKEEPER-1334: [~arnoud.buitenh...@sogeti.nl] No the MANIFEST.MF should contain all the packages that this bundle will or may use. So the 3 packages you found should be added as well. Zookeeper 3.4.x is not OSGi compliant - MANIFEST.MF is flawed - Key: ZOOKEEPER-1334 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1334 Project: ZooKeeper Issue Type: Bug Affects Versions: 3.4.0 Reporter: Claus Ibsen Assignee: Claus Ibsen Fix For: 3.5.0, 3.4.6 Attachments: zookeeper-1334-osgi.patch, zookeeper-1334-osgi.patch, ZOOKEEPER-1334.patch In Zookeeper 3.3.x you use log4j for logging, and the maven dep is eg from 3.3.4 {code} dependency groupIdlog4j/groupId artifactIdlog4j/artifactId version1.2.15/version scopecompile/scope /dependency {code} Now in 3.4.0 or better you changed to use slf4j also/instead. The maven pom.xml now includes: {code} dependency groupIdorg.slf4j/groupId artifactIdslf4j-api/artifactId version1.6.1/version scopecompile/scope /dependency dependency groupIdorg.slf4j/groupId artifactIdslf4j-log4j12/artifactId version1.6.1/version scopecompile/scope /dependency dependency groupIdlog4j/groupId artifactIdlog4j/artifactId version1.2.15/version scopecompile/scope /dependency {code} But the META-INF/MANIFEST.MF file in the distribution did not change to reflect this. The 3.3.4 MANIFEST.MF, import packages {code} Import-Package: javax.management,org.apache.log4j,org.osgi.framework;v ersion=[1.4,2.0),org.osgi.util.tracker;version=[1.1,2.0) {code} And the 3.4.1 MANIFEST.MF, import packages: {code} Import-Package: javax.management,org.apache.log4j,org.osgi.framework;v ersion=[1.4,2.0),org.osgi.util.tracker;version=[1.1,2.0) {code} This makes using zookeeper 3.4.x in OSGi environments not possible, as we get NoClassDefFoundException for slf4j classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (ZOOKEEPER-1645) ZooKeeper OSGi package imports not complete
Arnoud Glimmerveen created ZOOKEEPER-1645: - Summary: ZooKeeper OSGi package imports not complete Key: ZOOKEEPER-1645 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1645 Project: ZooKeeper Issue Type: Bug Affects Versions: 3.5.0, 3.4.6 Reporter: Arnoud Glimmerveen The ZooKeeper bundle relies on three packages it currently does not declare in the Import-Package MANIFEST header: {{javax.security.auth.callback}} , {{javax.security.auth.login}} and {{javax.security.sasl}} . By adding these the ZooKeeper jar will be a valid OSGi bundle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ZOOKEEPER-1334) Zookeeper 3.4.x is not OSGi compliant - MANIFEST.MF is flawed
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576443#comment-13576443 ] Arnoud Glimmerveen commented on ZOOKEEPER-1334: --- Okay, I've created a new issue for this: ZOOKEEPER-1645 Zookeeper 3.4.x is not OSGi compliant - MANIFEST.MF is flawed - Key: ZOOKEEPER-1334 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1334 Project: ZooKeeper Issue Type: Bug Affects Versions: 3.4.0 Reporter: Claus Ibsen Assignee: Claus Ibsen Fix For: 3.5.0, 3.4.6 Attachments: zookeeper-1334-osgi.patch, zookeeper-1334-osgi.patch, ZOOKEEPER-1334.patch In Zookeeper 3.3.x you use log4j for logging, and the maven dep is eg from 3.3.4 {code} dependency groupIdlog4j/groupId artifactIdlog4j/artifactId version1.2.15/version scopecompile/scope /dependency {code} Now in 3.4.0 or better you changed to use slf4j also/instead. The maven pom.xml now includes: {code} dependency groupIdorg.slf4j/groupId artifactIdslf4j-api/artifactId version1.6.1/version scopecompile/scope /dependency dependency groupIdorg.slf4j/groupId artifactIdslf4j-log4j12/artifactId version1.6.1/version scopecompile/scope /dependency dependency groupIdlog4j/groupId artifactIdlog4j/artifactId version1.2.15/version scopecompile/scope /dependency {code} But the META-INF/MANIFEST.MF file in the distribution did not change to reflect this. The 3.3.4 MANIFEST.MF, import packages {code} Import-Package: javax.management,org.apache.log4j,org.osgi.framework;v ersion=[1.4,2.0),org.osgi.util.tracker;version=[1.1,2.0) {code} And the 3.4.1 MANIFEST.MF, import packages: {code} Import-Package: javax.management,org.apache.log4j,org.osgi.framework;v ersion=[1.4,2.0),org.osgi.util.tracker;version=[1.1,2.0) {code} This makes using zookeeper 3.4.x in OSGi environments not possible, as we get NoClassDefFoundException for slf4j classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-555) Make BookieServer use Netty rather than a custom IO server
[ https://issues.apache.org/jira/browse/BOOKKEEPER-555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated BOOKKEEPER-555: -- Attachment: 0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch New patch addresses Rakesh's comments except for the import thing. What do you mean by organise? Alphabetize? Make BookieServer use Netty rather than a custom IO server -- Key: BOOKKEEPER-555 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-555 Project: Bookkeeper Issue Type: Bug Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 4.3.0 Attachments: 0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, BOOKKEEPER-555.patch Move from the custom NIO server to netty. This will make it easier to do things like add more server side threads and support SSL. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-555) Make BookieServer use Netty rather than a custom IO server
[ https://issues.apache.org/jira/browse/BOOKKEEPER-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13575765#comment-13575765 ] Hadoop QA commented on BOOKKEEPER-555: -- Testing JIRA BOOKKEEPER-555 Patch [0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch|https://issues.apache.org/jira/secure/attachment/12568814/0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch] downloaded at Mon Feb 11 12:11:12 UTC 2013 {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:green}+1 RAW_PATCH_ANALYSIS{color} .{color:green}+1{color} the patch does not introduce any @author tags .{color:green}+1{color} the patch does not introduce any tabs .{color:green}+1{color} the patch does not introduce any trailing spaces .{color:green}+1{color} the patch does not introduce any line longer than 120 .{color:green}+1{color} the patch does adds/modifies 2 testcase(s) {color:green}+1 RAT{color} .{color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} .{color:green}+1{color} the patch does not seem to introduce new Javadoc warnings {color:green}+1 COMPILE{color} .{color:green}+1{color} HEAD compiles .{color:green}+1{color} patch compiles .{color:green}+1{color} the patch does not seem to introduce new javac warnings {color:green}+1 FINDBUGS{color} .{color:green}+1{color} the patch does not seem to introduce new Findbugs warnings {color:green}+1 TESTS{color} .Tests run: 815 {color:green}+1 DISTRO{color} .{color:green}+1{color} distro tarball builds with the patch {color:green}*+1 Overall result, good!, no -1s*{color} The full output of the test-patch run is available at . https://builds.apache.org/job/bookkeeper-trunk-precommit-build/264/ Make BookieServer use Netty rather than a custom IO server -- Key: BOOKKEEPER-555 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-555 Project: Bookkeeper Issue Type: Bug Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 4.3.0 Attachments: 0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, BOOKKEEPER-555.patch Move from the custom NIO server to netty. This will make it easier to do things like add more server side threads and support SSL. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-555) Make BookieServer use Netty rather than a custom IO server
[ https://issues.apache.org/jira/browse/BOOKKEEPER-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13575781#comment-13575781 ] Rakesh R commented on BOOKKEEPER-555: - Thanks [~iv...@yahoo-inc.com] for the patch. {quote}What do you mean by organise? Alphabetize?{quote} Unused imports exists in these classes(BookieNettyServer, BookieServer, BookieServerBean), just wants to clear it:) Make BookieServer use Netty rather than a custom IO server -- Key: BOOKKEEPER-555 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-555 Project: Bookkeeper Issue Type: Bug Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 4.3.0 Attachments: 0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, BOOKKEEPER-555.patch Move from the custom NIO server to netty. This will make it easier to do things like add more server side threads and support SSL. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-554) fd leaking when move ledger index file
[ https://issues.apache.org/jira/browse/BOOKKEEPER-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated BOOKKEEPER-554: -- Attachment: 0001-BOOKKEEPER-554.patch Added @VisibleForTesting annotation. Once latest patch gets +1, ill push it in. fd leaking when move ledger index file -- Key: BOOKKEEPER-554 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-554 Project: Bookkeeper Issue Type: Bug Components: bookkeeper-server Affects Versions: 4.2.0 Reporter: Sijie Guo Assignee: Sijie Guo Fix For: 4.3.0 Attachments: 0001-BOOKKEEPER-554.patch, BOOKKEEPER-554.diff a file info is get when moving ledger index, but it doesn't release after use. so the reference counting for file info stays more than zero, the file channel would never be closed even the file is evicted from ledger cache. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-554) fd leaking when move ledger index file
[ https://issues.apache.org/jira/browse/BOOKKEEPER-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13575806#comment-13575806 ] Hadoop QA commented on BOOKKEEPER-554: -- Testing JIRA BOOKKEEPER-554 Patch [0001-BOOKKEEPER-554.patch|https://issues.apache.org/jira/secure/attachment/12568821/0001-BOOKKEEPER-554.patch] downloaded at Mon Feb 11 13:55:54 UTC 2013 {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:green}+1 RAW_PATCH_ANALYSIS{color} .{color:green}+1{color} the patch does not introduce any @author tags .{color:green}+1{color} the patch does not introduce any tabs .{color:green}+1{color} the patch does not introduce any trailing spaces .{color:green}+1{color} the patch does not introduce any line longer than 120 .{color:green}+1{color} the patch does adds/modifies 1 testcase(s) {color:green}+1 RAT{color} .{color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} .{color:green}+1{color} the patch does not seem to introduce new Javadoc warnings {color:green}+1 COMPILE{color} .{color:green}+1{color} HEAD compiles .{color:green}+1{color} patch compiles .{color:green}+1{color} the patch does not seem to introduce new javac warnings {color:green}+1 FINDBUGS{color} .{color:green}+1{color} the patch does not seem to introduce new Findbugs warnings {color:green}+1 TESTS{color} .Tests run: 816 {color:green}+1 DISTRO{color} .{color:green}+1{color} distro tarball builds with the patch {color:green}*+1 Overall result, good!, no -1s*{color} The full output of the test-patch run is available at . https://builds.apache.org/job/bookkeeper-trunk-precommit-build/265/ fd leaking when move ledger index file -- Key: BOOKKEEPER-554 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-554 Project: Bookkeeper Issue Type: Bug Components: bookkeeper-server Affects Versions: 4.2.0 Reporter: Sijie Guo Assignee: Sijie Guo Fix For: 4.3.0 Attachments: 0001-BOOKKEEPER-554.patch, BOOKKEEPER-554.diff a file info is get when moving ledger index, but it doesn't release after use. so the reference counting for file info stays more than zero, the file channel would never be closed even the file is evicted from ledger cache. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-555) Make BookieServer use Netty rather than a custom IO server
[ https://issues.apache.org/jira/browse/BOOKKEEPER-555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated BOOKKEEPER-555: -- Attachment: 0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch Ah, had completely missed those. Fixed now. Make BookieServer use Netty rather than a custom IO server -- Key: BOOKKEEPER-555 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-555 Project: Bookkeeper Issue Type: Bug Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 4.3.0 Attachments: 0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, BOOKKEEPER-555.patch Move from the custom NIO server to netty. This will make it easier to do things like add more server side threads and support SSL. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-555) Make BookieServer use Netty rather than a custom IO server
[ https://issues.apache.org/jira/browse/BOOKKEEPER-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13575832#comment-13575832 ] Hadoop QA commented on BOOKKEEPER-555: -- Testing JIRA BOOKKEEPER-555 Patch [0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch|https://issues.apache.org/jira/secure/attachment/12568824/0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch] downloaded at Mon Feb 11 14:51:12 UTC 2013 {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:green}+1 RAW_PATCH_ANALYSIS{color} .{color:green}+1{color} the patch does not introduce any @author tags .{color:green}+1{color} the patch does not introduce any tabs .{color:green}+1{color} the patch does not introduce any trailing spaces .{color:green}+1{color} the patch does not introduce any line longer than 120 .{color:green}+1{color} the patch does adds/modifies 2 testcase(s) {color:green}+1 RAT{color} .{color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} .{color:green}+1{color} the patch does not seem to introduce new Javadoc warnings {color:green}+1 COMPILE{color} .{color:green}+1{color} HEAD compiles .{color:green}+1{color} patch compiles .{color:green}+1{color} the patch does not seem to introduce new javac warnings {color:green}+1 FINDBUGS{color} .{color:green}+1{color} the patch does not seem to introduce new Findbugs warnings {color:green}+1 TESTS{color} .Tests run: 815 {color:green}+1 DISTRO{color} .{color:green}+1{color} distro tarball builds with the patch {color:green}*+1 Overall result, good!, no -1s*{color} The full output of the test-patch run is available at . https://builds.apache.org/job/bookkeeper-trunk-precommit-build/266/ Make BookieServer use Netty rather than a custom IO server -- Key: BOOKKEEPER-555 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-555 Project: Bookkeeper Issue Type: Bug Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 4.3.0 Attachments: 0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, BOOKKEEPER-555.patch Move from the custom NIO server to netty. This will make it easier to do things like add more server side threads and support SSL. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-49) bookkeeper - parallel async read same entry of same ledger will fail
[ https://issues.apache.org/jira/browse/BOOKKEEPER-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13575867#comment-13575867 ] Matteo Merli commented on BOOKKEEPER-49: Rakesh, I think that there should be an Id on the request and that should be addressed in BOOKKEEPER-558. Once that jira is resolved we'll just need to keep the map with the key + reqId. bookkeeper - parallel async read same entry of same ledger will fail Key: BOOKKEEPER-49 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-49 Project: Bookkeeper Issue Type: Bug Components: bookkeeper-client Affects Versions: 4.0.0, 4.1.0 Reporter: Sijie Guo Assignee: Sijie Guo Fix For: 4.3.0 Attachments: 0001-BOOKKEEPER-49-bookkeeper-parallel-async-read-same-en.patch all ledgers shared a PerChannelBookieClient. PerChannelBookieClient put all the read requests in a ConcurrentHashMapCompletionKey, ReadCompletion map called readCompletions, which is indexed by CompletionKey. If two read requests have same entryId and same ledgerId, they have the same CompletionKey, the latter one will overwrite the previous one. So a read request's callback will not be invoked. we may need to chain the callbacks for same completion keys. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-49) bookkeeper - parallel async read same entry of same ledger will fail
[ https://issues.apache.org/jira/browse/BOOKKEEPER-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13575900#comment-13575900 ] Rakesh R commented on BOOKKEEPER-49: Oh ok. Once BOOKKEEPER-558 is in, will revisit again. Could you please link with BOOKKEEPER-558. bookkeeper - parallel async read same entry of same ledger will fail Key: BOOKKEEPER-49 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-49 Project: Bookkeeper Issue Type: Bug Components: bookkeeper-client Affects Versions: 4.0.0, 4.1.0 Reporter: Sijie Guo Assignee: Sijie Guo Fix For: 4.3.0 Attachments: 0001-BOOKKEEPER-49-bookkeeper-parallel-async-read-same-en.patch all ledgers shared a PerChannelBookieClient. PerChannelBookieClient put all the read requests in a ConcurrentHashMapCompletionKey, ReadCompletion map called readCompletions, which is indexed by CompletionKey. If two read requests have same entryId and same ledgerId, they have the same CompletionKey, the latter one will overwrite the previous one. So a read request's callback will not be invoked. we may need to chain the callbacks for same completion keys. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (BOOKKEEPER-568) NPE during GC with HierarchicalLedgerManager
Matteo Merli created BOOKKEEPER-568: --- Summary: NPE during GC with HierarchicalLedgerManager Key: BOOKKEEPER-568 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-568 Project: Bookkeeper Issue Type: Bug Components: bookkeeper-server Affects Versions: 4.2.0 Reporter: Matteo Merli Priority: Minor {noformat} 2013-02-11 14:06:28,904 - WARN - [GarbageCollectorThread:ScanAndCompareGarbageCollector@103] - Exception when iterating over the metadata {} java.io.IOException: Error when check more elements at org.apache.bookkeeper.meta.HierarchicalLedgerManager$HierarchicalLedgerRangeIterator.hasNext(HierarchicalLedgerManager.java:423) at org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector.gc(ScanAndCompareGarbageCollector.java:75) at org.apache.bookkeeper.bookie.GarbageCollectorThread.doGcLedgers(GarbageCollectorThread.java:302) at org.apache.bookkeeper.bookie.GarbageCollectorThread.run(GarbageCollectorThread.java:271) Caused by: java.lang.NullPointerException at org.apache.bookkeeper.meta.HierarchicalLedgerManager$HierarchicalLedgerRangeIterator.hasNext(HierarchicalLedgerManager.java:419) ... 3 more {noformat} In the code below, l2NodesIter appears to be null. {code} public boolean hasNext() throws IOException { try { if (l1NodesIter == null) { l1NodesIter = zk.getChildren(ledgerRootPath, null).iterator(); hasMoreElement = nextL1Node(); } else if (!l2NodesIter.hasNext()) { hasMoreElement = nextL1Node(); } } catch (Exception e) { throw new IOException(Error when check more elements, e); } return hasMoreElement; } {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-568) NPE during GC with HierarchicalLedgerManager
[ https://issues.apache.org/jira/browse/BOOKKEEPER-568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576182#comment-13576182 ] Matteo Merli commented on BOOKKEEPER-568: - I'm not sure if doing {code} l2NodesIter == null || !l2NodesIter.hasNext() {code} would be the correct check. NPE during GC with HierarchicalLedgerManager Key: BOOKKEEPER-568 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-568 Project: Bookkeeper Issue Type: Bug Components: bookkeeper-server Affects Versions: 4.2.0 Reporter: Matteo Merli Priority: Minor {noformat} 2013-02-11 14:06:28,904 - WARN - [GarbageCollectorThread:ScanAndCompareGarbageCollector@103] - Exception when iterating over the metadata {} java.io.IOException: Error when check more elements at org.apache.bookkeeper.meta.HierarchicalLedgerManager$HierarchicalLedgerRangeIterator.hasNext(HierarchicalLedgerManager.java:423) at org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector.gc(ScanAndCompareGarbageCollector.java:75) at org.apache.bookkeeper.bookie.GarbageCollectorThread.doGcLedgers(GarbageCollectorThread.java:302) at org.apache.bookkeeper.bookie.GarbageCollectorThread.run(GarbageCollectorThread.java:271) Caused by: java.lang.NullPointerException at org.apache.bookkeeper.meta.HierarchicalLedgerManager$HierarchicalLedgerRangeIterator.hasNext(HierarchicalLedgerManager.java:419) ... 3 more {noformat} In the code below, l2NodesIter appears to be null. {code} public boolean hasNext() throws IOException { try { if (l1NodesIter == null) { l1NodesIter = zk.getChildren(ledgerRootPath, null).iterator(); hasMoreElement = nextL1Node(); } else if (!l2NodesIter.hasNext()) { hasMoreElement = nextL1Node(); } } catch (Exception e) { throw new IOException(Error when check more elements, e); } return hasMoreElement; } {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-555) Make BookieServer use Netty rather than a custom IO server
[ https://issues.apache.org/jira/browse/BOOKKEEPER-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576362#comment-13576362 ] Rakesh R commented on BOOKKEEPER-555: - Thanks [~ikelly], latest patch looks nice. Apart from the following point, the patch is ready to go in +1. Just one clarification, latest patch BookieRequestHandler is not having @Sharable annotation. In my previous comment I mentioned to replace the @ChannelPipelineCoverage(which is deprecated) with @Sharable. Am I missing anything? Make BookieServer use Netty rather than a custom IO server -- Key: BOOKKEEPER-555 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-555 Project: Bookkeeper Issue Type: Bug Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 4.3.0 Attachments: 0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0001-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, 0002-BOOKKEEPER-555-Netty-Server-for-Bookie.patch, BOOKKEEPER-555.patch Move from the custom NIO server to netty. This will make it easier to do things like add more server side threads and support SSL. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-568) NPE during GC with HierarchicalLedgerManager
[ https://issues.apache.org/jira/browse/BOOKKEEPER-568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576374#comment-13576374 ] Sijie Guo commented on BOOKKEEPER-568: -- ah, it seems that this code doesn't handle calling hasNext twice. so second time call hasNext would fail with null pointer. the case would happened when there is no ledgers existed in bookkeeper. [~merlimat] it is OK to fix as your suggestion. could you generate a patch for it? thanks. NPE during GC with HierarchicalLedgerManager Key: BOOKKEEPER-568 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-568 Project: Bookkeeper Issue Type: Bug Components: bookkeeper-server Affects Versions: 4.2.0 Reporter: Matteo Merli Priority: Minor {noformat} 2013-02-11 14:06:28,904 - WARN - [GarbageCollectorThread:ScanAndCompareGarbageCollector@103] - Exception when iterating over the metadata {} java.io.IOException: Error when check more elements at org.apache.bookkeeper.meta.HierarchicalLedgerManager$HierarchicalLedgerRangeIterator.hasNext(HierarchicalLedgerManager.java:423) at org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector.gc(ScanAndCompareGarbageCollector.java:75) at org.apache.bookkeeper.bookie.GarbageCollectorThread.doGcLedgers(GarbageCollectorThread.java:302) at org.apache.bookkeeper.bookie.GarbageCollectorThread.run(GarbageCollectorThread.java:271) Caused by: java.lang.NullPointerException at org.apache.bookkeeper.meta.HierarchicalLedgerManager$HierarchicalLedgerRangeIterator.hasNext(HierarchicalLedgerManager.java:419) ... 3 more {noformat} In the code below, l2NodesIter appears to be null. {code} public boolean hasNext() throws IOException { try { if (l1NodesIter == null) { l1NodesIter = zk.getChildren(ledgerRootPath, null).iterator(); hasMoreElement = nextL1Node(); } else if (!l2NodesIter.hasNext()) { hasMoreElement = nextL1Node(); } } catch (Exception e) { throw new IOException(Error when check more elements, e); } return hasMoreElement; } {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-554) fd leaking when move ledger index file
[ https://issues.apache.org/jira/browse/BOOKKEEPER-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576376#comment-13576376 ] Sijie Guo commented on BOOKKEEPER-554: -- +1 for the new patch. thanks Ivan. will commit it. fd leaking when move ledger index file -- Key: BOOKKEEPER-554 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-554 Project: Bookkeeper Issue Type: Bug Components: bookkeeper-server Affects Versions: 4.2.0 Reporter: Sijie Guo Assignee: Sijie Guo Fix For: 4.3.0 Attachments: 0001-BOOKKEEPER-554.patch, BOOKKEEPER-554.diff a file info is get when moving ledger index, but it doesn't release after use. so the reference counting for file info stays more than zero, the file channel would never be closed even the file is evicted from ledger cache. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-554) fd leaking when move ledger index file
[ https://issues.apache.org/jira/browse/BOOKKEEPER-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576397#comment-13576397 ] Hudson commented on BOOKKEEPER-554: --- Integrated in bookkeeper-trunk #98 (See [https://builds.apache.org/job/bookkeeper-trunk/98/]) BOOKKEEPER-554: fd leaking when move ledger index file (sijie, ivank via sijie) (Revision 1445033) Result = SUCCESS sijie : Files : * /zookeeper/bookkeeper/trunk/CHANGES.txt * /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/FileInfo.java * /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerCacheImpl.java * /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/LedgerCacheTest.java fd leaking when move ledger index file -- Key: BOOKKEEPER-554 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-554 Project: Bookkeeper Issue Type: Bug Components: bookkeeper-server Affects Versions: 4.2.0 Reporter: Sijie Guo Assignee: Sijie Guo Fix For: 4.3.0 Attachments: 0001-BOOKKEEPER-554.patch, BOOKKEEPER-554.diff a file info is get when moving ledger index, but it doesn't release after use. so the reference counting for file info stays more than zero, the file channel would never be closed even the file is evicted from ledger cache. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira