These results are about what is expected although the might be a little more
extreme.
I doubt very much that hbase is mutating zk nodes fast enough for this to
matter much.
Sent from my iPhone
On Feb 14, 2012, at 8:00, N Keywal <nkey...@gmail.com> wrote:
> Hi,
>
> I've done a test with Zookeeper 3.4.2 to compare the performances of
> synchronous vs. asynchronous vs. multi when creating znode (variations
> around:
> calling 10000 times zk.create("/dummyTest", "dummy".getBytes(),
> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);) The code is at the
> end of the mail.
>
> I've tested different environments:
> - 1 linux server with the client and 1 zookeeper node on the same machine
> - 1 linux server for the client, 1 for 1 zookeeper node.
> - 6 linux servers, 1 for the client, 5 for 5 zookeeper nodes.
>
> Server are middle range, with 4*2 cores, jdk 1.6. ZK was on its own HD.
>
> But the results are comparable:
>
> Using the sync API, it takes 200 seconds for 10K creations, so around 0.02
> second per call.
> Using the async API, it takes 2 seconds for 10K (including waiting for the
> last callback message)
> Using the "multi" available since 3.4, it takes less than 1 second, again
> for 10K.
>
> I'm surprised by the time taken by the sync operation, I was not expecting
> it to be that slow. The gap between async & sync is quite huge.
>
> Is this something expected? Zookeeper is used in critical functions in
> Hadoop/Hbase, I was looking at the possible benefits of using "multi", but
> it seems low compared to async (well ~3 times faster :-). There are many
> small data creations/deletions with the sync API in the existing hbase
> algorithms, it would not be simple to replace them all by asynchronous
> calls...
>
> Cheers,
>
> N.
>
> --
>
> public class ZookeeperTest {
> static ZooKeeper zk;
> static int nbTests = 10000;
>
> private ZookeeperTest() {
> }
>
> public static void test11() throws Exception {
> for (int i = 0; i < nbTests; ++i) {
> zk.create("/dummyTest_" + i, "dummy".getBytes(),
> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
> }
> }
>
>
> public static void test51() throws Exception {
> final AtomicInteger counter = new AtomicInteger(0);
>
> for (int i = 0; i < nbTests; ++i) {
> zk.create("/dummyTest_" + i, "dummy".getBytes(),
> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT,
> new AsyncCallback.StringCallback() {
> public void processResult(int i, String s, Object o, String s1) {
> counter.incrementAndGet();
> }
> }
> , null);
> }
>
> while (counter.get() != nbTests) {
> Thread.sleep(1);
> }
> }
>
> public static void test41() throws Exception {
> ArrayList<Op> ops = new ArrayList<Op>(nbTests);
> for (int i = 0; i < nbTests; ++i) {
> ops.add(
> Op.create("/dummyTest_" + i, "dummy".getBytes(),
> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)
> );
> }
>
> zk.multi(ops);
> }
>
> public static void delete() throws Exception {
> ArrayList<Op> ops = new ArrayList<Op>(nbTests);
>
> for (int i = 0; i < nbTests; ++i) {
> ops.add(
> Op.delete("/dummyTest_" + i,-1)
> );
> }
>
> zk.multi(ops);
> }
>
>
> public static void test(String connection, String testName) throws
> Throwable{
> Method m = ZookeeperTest.class.getMethod(testName);
>
> zk = new ZooKeeper(connection, 20000, new Watcher() {
> public void process(WatchedEvent watchedEvent) {
> }
> });
>
> final long start = System.currentTimeMillis();
>
> try {
> m.invoke(null);
> } catch (IllegalAccessException e) {
> throw e;
> } catch (InvocationTargetException e) {
> throw e.getTargetException();
> }
>
> final long end = System.currentTimeMillis();
>
> zk.close();
>
> final long endClose = System.currentTimeMillis();
>
> System.out.println(testName+": ExeTime= " + (end - start) );
> }
>
> public static void main(String... args) throws Throwable {
> test(args[0], args[1]);
> }
> }
