[
https://issues.apache.org/jira/browse/ZOOKEEPER-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632641#comment-16632641
]
Hudson commented on ZOOKEEPER-1177:
-----------------------------------
FAILURE: Integrated in Jenkins build ZooKeeper-trunk #208 (See
[https://builds.apache.org/job/ZooKeeper-trunk/208/])
ZOOKEEPER-1177: Add the memory optimized watch manager for concentrate (hanm:
rev fdde8b006458f7b989c894af0eac7e124d271a1e)
* (edit) src/java/test/config/findbugsExcludeFile.xml
* (edit) src/java/main/org/apache/zookeeper/server/DataTree.java
* (add) src/java/test/org/apache/zookeeper/server/util/BitMapTest.java
* (edit)
zookeeper-contrib/zookeeper-contrib-fatjar/src/main/resources/mainClasses
* (edit) ivy.xml
* (add) src/java/test/org/apache/zookeeper/server/watch/WatchesSummaryTest.java
* (edit) build.xml
* (add) src/java/main/org/apache/zookeeper/server/util/BitMap.java
* (delete) src/java/main/org/apache/zookeeper/server/WatchesPathReport.java
* (delete) src/java/test/org/apache/zookeeper/server/WatchesReportTest.java
* (add) src/java/main/org/apache/zookeeper/server/watch/WatchManagerFactory.java
* (delete) src/java/main/org/apache/zookeeper/server/WatchesSummary.java
* (add) src/java/main/org/apache/zookeeper/server/watch/WatchesPathReport.java
* (delete) src/java/main/org/apache/zookeeper/server/WatchManager.java
* (add) src/java/test/org/apache/zookeeper/server/watch/WatchManagerTest.java
* (add)
src/java/test/org/apache/zookeeper/server/watch/WatchesPathReportTest.java
* (delete) src/java/test/org/apache/zookeeper/server/WatchesSummaryTest.java
* (add) src/java/main/org/apache/zookeeper/server/watch/WatchesReport.java
* (add) src/java/main/org/apache/zookeeper/server/watch/WatchesSummary.java
* (edit) src/java/main/org/apache/zookeeper/server/ServerCnxn.java
* (add)
src/java/main/org/apache/zookeeper/server/watch/WatchManagerOptimized.java
* (edit) src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java
* (edit) src/java/main/org/apache/zookeeper/server/ZKDatabase.java
* (add)
src/java/main/org/apache/zookeeper/server/watch/IDeadWatcherListener.java
* (add) src/java/main/org/apache/zookeeper/server/watch/WatchManager.java
* (add) src/java/test/org/apache/zookeeper/server/watch/WatchesReportTest.java
* (add) src/test/java/bench/org/apache/zookeeper/BenchMain.java
* (add) src/java/main/org/apache/zookeeper/server/util/BitHashSet.java
* (add) src/test/java/bench/org/apache/zookeeper/server/watch/WatchBench.java
* (add) src/java/main/org/apache/zookeeper/server/watch/WatcherCleaner.java
* (add) src/java/test/org/apache/zookeeper/server/watch/WatcherOrBitSetTest.java
* (delete) src/java/test/org/apache/zookeeper/server/WatchesPathReportTest.java
* (add) src/java/test/org/apache/zookeeper/server/util/BitHashSetTest.java
* (add) src/java/main/org/apache/zookeeper/server/watch/WatcherOrBitSet.java
* (edit) zookeeper-docs/src/documentation/content/xdocs/zookeeperAdmin.xml
* (add) src/java/main/org/apache/zookeeper/server/DumbWatcher.java
* (add) src/java/main/org/apache/zookeeper/server/watch/IWatchManager.java
* (edit) src/java/main/org/apache/zookeeper/server/NIOServerCnxn.java
* (add) src/java/test/org/apache/zookeeper/server/watch/WatcherCleanerTest.java
* (delete) src/java/main/org/apache/zookeeper/server/WatchesReport.java
> Enabling a large number of watches for a large number of clients
> ----------------------------------------------------------------
>
> Key: ZOOKEEPER-1177
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1177
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Affects Versions: 3.3.3
> Reporter: Vishal Kathuria
> Assignee: Fangmin Lv
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.6.0
>
> Attachments: ZOOKEEPER-1177.patch, ZOOKEEPER-1177.patch,
> ZooKeeper-with-fix-for-findbugs-warning.patch, ZooKeeper.patch,
> Zookeeper-after-resolving-merge-conflicts.patch
>
> Time Spent: 13h 50m
> Remaining Estimate: 0h
>
> In my ZooKeeper, I see watch manager consuming several GB of memory and I dug
> a bit deeper.
> In the scenario I am testing, I have 10K clients connected to an observer.
> There are about 20K znodes in ZooKeeper, each is about 1K - so about 20M data
> in total.
> Each client fetches and puts watches on all the znodes. That is 200 million
> watches.
> It seems a single watch takes about 100 bytes. I am currently at 14528037
> watches and according to the yourkit profiler, WatchManager has 1.2 G
> already. This is not going to work as it might end up needing 20G of RAM just
> for the watches.
> So we need a more compact way of storing watches. Here are the possible
> solutions.
> 1. Use a bitmap instead of the current hashmap. In this approach, each znode
> would get a unique id when its gets created. For every session, we can keep
> track of a bitmap that indicates the set of znodes this session is watching.
> A bitmap, assuming a 100K znodes, would be 12K. For 10K sessions, we can keep
> track of watches using 120M instead of 20G.
> 2. This second idea is based on the observation that clients watch znodes in
> sets (for example all znodes under a folder). Multiple clients watch the same
> set and the total number of sets is a couple of orders of magnitude smaller
> than the total number of znodes. In my scenario, there are about 100 sets. So
> instead of keeping track of watches at the znode level, keep track of it at
> the set level. It may mean that get may also need to be implemented at the
> set level. With this, we can save the watches in 100M.
> Are there any other suggestions of solutions?
> Thanks
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
