I agree with Mahadev that it sounds like a stretch. I just wanted to
point out that we have been working on a new feature that would
certainly help in this case: Observers (ZOOKEEPER-368). I don't think
we have decided yet in which release we will include it, but my
current guess is 4.0.0 at the latest, and it might be possible to have
it for 3.3.0.
-Flavio
On Aug 30, 2009, at 4:32 AM, Mahadev Konar wrote:
Hi Avinash,
100K clients would be a stretch. We have never tested at that scale.
100K watches should not be a problem at all. I am more concerened
about the
number of client connections that would result to each of the
zookeeper
servers.
In case of 5 servers, that would be 20K persistent clients to each
of the
zookeeper servers which seems really high and not really feasible.
With a
high level of quorums servers like 13 or so you should have a more
reasonable number of connections per server but again you would just
be
running at the edge (in case some amount of servers went down).
Is 100K machines all running in the same data center? If not (which
I think
is more likely the case?), I would suggest running different zookeeper
ensembles in different data centers and using a bridge to keep them
synchronized amongst themselves.
If you can shed more light on the setup and use case of these 100K
machines, I think we can work out a reasonable solution.
Thanks
mahadev
On 8/29/09 6:16 PM, "Ted Dunning" <ted.dunn...@gmail.com> wrote:
That is probably a bit beyond reasonable levels of scaling. For
one thing,
putting 100,000 machines close together in a network is a bit
tricky. The
two major limitations are likely to be memory for keeping the
watches on the
server side and bandwidth for publishing the notifications.
That said, ZK is solid enough that I would not be surprised if it
scaled to
that level with sufficient memory and low enough update rate.
On Sat, Aug 29, 2009 at 2:51 PM, Avinash Lakshman <
avinash.laksh...@gmail.com> wrote:
Hi All
Is it possible to have 100K machines register for a watch on a
znode? I
mean
theoritically yes it should work but ZK scale to these many
instances when
it comes to delivering watch notifications? Perhaps no one has
practical
experience in dealing with this but is there any fundamental
limitation
that
I should be aware of? These 100K machines are only interested in
receiving
notifications.
Cheers
Avinash
