[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500437#comment-14500437 ] Ashish K Singh commented on KAFKA-1994: --- [~junrao], [~gwenshap], does this look on now? > Evaluate performance effect of chroot check on Topic creation > - > > Key: KAFKA-1994 > URL: https://issues.apache.org/jira/browse/KAFKA-1994 > Project: Kafka > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Attachments: KAFKA-1994.patch, KAFKA-1994_2015-03-03_18:19:45.patch > > > KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks > if namespace exists before trying to create a path in ZK. This raises a > concern that checking namespace for each path creation might be unnecessary > and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379959#comment-14379959 ] Ashish K Singh commented on KAFKA-1994: --- [~gwenshap] My apologies for the delay. Kindly find the comparison numbers below. The numbers are averaged over 1000 iterations. w/o any check: 635438 nanoseconds w/ check from KAFKA-1664: 908189 nanoseconds (42.92% up) w/ check suggested on this JIRA: 642484 nanoseonds (1.1% up) The above comparison numbers are expected and in accordance to the discussion on this JIRA. > Evaluate performance effect of chroot check on Topic creation > - > > Key: KAFKA-1994 > URL: https://issues.apache.org/jira/browse/KAFKA-1994 > Project: Kafka > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Attachments: KAFKA-1994.patch, KAFKA-1994_2015-03-03_18:19:45.patch > > > KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks > if namespace exists before trying to create a path in ZK. This raises a > concern that checking namespace for each path creation might be unnecessary > and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363349#comment-14363349 ] Gwen Shapira commented on KAFKA-1994: - New patch looks good. [~singhashish], can you share how does the performance of the code with new patch compares to that of the older solution and to that of createPersistent() without any of the checks? > Evaluate performance effect of chroot check on Topic creation > - > > Key: KAFKA-1994 > URL: https://issues.apache.org/jira/browse/KAFKA-1994 > Project: Kafka > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Attachments: KAFKA-1994.patch, KAFKA-1994_2015-03-03_18:19:45.patch > > > KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks > if namespace exists before trying to create a path in ZK. This raises a > concern that checking namespace for each path creation might be unnecessary > and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346253#comment-14346253 ] Ashish K Singh commented on KAFKA-1994: --- Updated reviewboard https://reviews.apache.org/r/31711/ against branch trunk > Evaluate performance effect of chroot check on Topic creation > - > > Key: KAFKA-1994 > URL: https://issues.apache.org/jira/browse/KAFKA-1994 > Project: Kafka > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Attachments: KAFKA-1994.patch, KAFKA-1994_2015-03-03_18:19:45.patch > > > KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks > if namespace exists before trying to create a path in ZK. This raises a > concern that checking namespace for each path creation might be unnecessary > and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346125#comment-14346125 ] Ashish K Singh commented on KAFKA-1994: --- Created reviewboard https://reviews.apache.org/r/31711/ against branch trunk > Evaluate performance effect of chroot check on Topic creation > - > > Key: KAFKA-1994 > URL: https://issues.apache.org/jira/browse/KAFKA-1994 > Project: Kafka > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Attachments: KAFKA-1994.patch > > > KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks > if namespace exists before trying to create a path in ZK. This raises a > concern that checking namespace for each path creation might be unnecessary > and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[
https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14345891#comment-14345891
]
Ashish K Singh commented on KAFKA-1994:
---
[~junrao] Your suggestion makes sense. Profiled {{createPersistent()}} with and
without the patch using {{ZooKeeperTestHarness}}, there is ~44% jump in exec
time. Will shortly attach a patch to fix it.
[~gwenshap] let me know if you think running on a real zk cluster is still
required.
> Evaluate performance effect of chroot check on Topic creation
> -
>
> Key: KAFKA-1994
> URL: https://issues.apache.org/jira/browse/KAFKA-1994
> Project: Kafka
> Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks
> if namespace exists before trying to create a path in ZK. This raises a
> concern that checking namespace for each path creation might be unnecessary
> and can potentially make creations expensive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344572#comment-14344572 ] Jun Rao commented on KAFKA-1994: Ashish, The code path for creating a topic may not be optimized. Could you just test the cost of createPersistent() with and w/o the patch? Thanks, > Evaluate performance effect of chroot check on Topic creation > - > > Key: KAFKA-1994 > URL: https://issues.apache.org/jira/browse/KAFKA-1994 > Project: Kafka > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks > if namespace exists before trying to create a path in ZK. This raises a > concern that checking namespace for each path creation might be unnecessary > and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14342774#comment-14342774 ] Gwen Shapira commented on KAFKA-1994: - Thanks for checking. This looks good :) I think a 3-node zk cluster not on localhost will be more realistic for production. Can you try that? > Evaluate performance effect of chroot check on Topic creation > - > > Key: KAFKA-1994 > URL: https://issues.apache.org/jira/browse/KAFKA-1994 > Project: Kafka > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks > if namespace exists before trying to create a path in ZK. This raises a > concern that checking namespace for each path creation might be unnecessary > and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[
https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14342769#comment-14342769
]
Ashish K Singh commented on KAFKA-1994:
---
Ran quick test of creating a topic with 1000 partitions with and without
KAFKA-1664's patch. Results are averaged over 100 iterations.
W/o patch (100 iterations):
real2m4.689s
user1m45.213s
sys 0m6.893s
user + sys = 1121.06 ms for each topic creation
W patch (100 iterations):
real2m4.327s
user1m45.497s
sys 0m7.004s
user + sys = 1125.01 ms for each topic creation
Effect due to the patch is ~0.35%. Below is the simple test I ran. Let me know
if you have a better test in mind.
{code}
time for i in `seq 1 100`; do bin/kafka-topics.sh --zookeeper localhost:2181
--create --topic topic$i --partitions 2000 --replication-factor 1; done
{code}
> Evaluate performance effect of chroot check on Topic creation
> -
>
> Key: KAFKA-1994
> URL: https://issues.apache.org/jira/browse/KAFKA-1994
> Project: Kafka
> Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks
> if namespace exists before trying to create a path in ZK. This raises a
> concern that checking namespace for each path creation might be unnecessary
> and can potentially make creations expensive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
