[
https://issues.apache.org/jira/browse/ZOOKEEPER-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161075#comment-13161075
]
Daniel Lord commented on ZOOKEEPER-1312:
----------------------------------------
Ben/Ted what do you think the performance characteristics would be of doing
lots of quick reads would be? The reason that I'm requesting this feature is
to avoid the "large directory problem" but also to avoid having to do a read
for each child. Currently, there is a use case in my application that has
cropped up with a very large directory and that has caused issues. I could use
this operation (or the multi perhaps) to restructure the data that I'm writing
to zk and avoid the issue. At my present scale to do this with a
multi/individual stats would cost me approximately 1 getChildren followed by
2000 individual stats. I'm hoping to make our clusters larger and not smaller
and if that costs too much I won't be able to continue to scale out.
My current algorithm requires two getChildren operations. The first one will
return approximately 2000 children and the second one could return several
times that (lets say 10 times for the sake of arugment). If I can switch the
way that the data is written and leverage a "getChildrenWithStat" operation
then I would change the second getChildren to "getChildrenWithStat" and I would
receive approximately 2000 children with stats in that case as well. Having
the ability to getChildrenWithStat would also allow me to change what data
needs to be encoded in the name of the znodes too.
Using my current algorithm my two getChildren calls have roughly this cost
associated:
The first getChildren returns 2000 EPHEMERAL children with each child having a
name with length 39. This could possibly be changed to a name with a length of
23 but that would limit my flexibility and would lose a small amount of
information. That makes the total size of the response approximately 78k
characters.
The second getChildren returns 20000 EPHEMERAL_SEQUENTIAL children with each
child having a name with length 89. This could be shortened to a length of 58
with minimal data loss or a length of 27 with some acceptable loss of data.
That makes the total size of the response approximately 1780k characters. If
the number of children in this call approaches 45k then I will exceed the
default jute max buffer size and the whole cluster spins out of control. This
number can grow without human intervention (it's not necessarily a function of
the cluster size).
Switching my algorithm how I would if I had getChildrenWithStat would have
roughly this cost:
The first getChildren call would remain identical.
The second getChildren call would return 2000 PERSISTENT children with Stats.
For the sake of argument let's say that the payload size is the same, per
child, for these children as it was above. That puts this call at 178k.
Additionally, this would scale as a function of the size of the cluster and not
as something that is up to the behavior of the application.
If my worries about doing 2000+ reads in order to accomplish this task are
completely unfounded then we can probably close this issue and I'll either
implement an async reads solution or use a multi-read solution. However, I
fear that won't scale well. I have a large number of clients that all need to
do this process fairly frequently. As long as my number of children has stayed
less than 10k I have been having no problems. However, a bug was discovered
(outside of this code but affecting this process) that caused the number of
children to shoot up to 45k and cause the jute max buffer size to be exceeded.
I'm trying to protect myself from that issue.
> Add a "getChildrenWithStat" operation
> -------------------------------------
>
> Key: ZOOKEEPER-1312
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1312
> Project: ZooKeeper
> Issue Type: New Feature
> Reporter: Daniel Lord
> Labels: newbie
>
> It would be extremely useful to be able to have a "getChildrenWithStat"
> method. This method would behave exactly the same as getChildren but in
> addition to returning the list of all child znode names it would also return
> a Stat for each child. I'm sure there are quite a few use cases for this but
> it could save a lot of extra reads for my application.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
