On Sep 25, 2011, at 9:02 AM, OZAWA Tsuyoshi wrote:
(2011/09/25 6:43), Flavio Junqueira wrote:Thanks for sending this reference to the list, it sounds very interesting. I have a few questions and comments, if you don't mind:1- I was wondering if you can give more detail on the setup you used togenerate the numbers you show in the graphs on your Accord page. The ZooKeeper values are way too low, and I suspect that you're using asingle hard drive. It could be because you expect to use a single hard drive with an Accord server, and you wanted to make the comparison fair.Is this correct?No, it isn't. Both ZooKeeper and Accord use the dedicated hard drive for logging. Setting file I used is here: https://gist.github.com/1240291 Please tell me if I have a mistake.
I gave a cursory look, and I can't see any obvious problem. It is intriguing that the numbers are so low. Have you tried with different numbers of servers? I'm not sure if I just missed this information, but what version of ZooKeeper are you looking at? Also, if it is not too much trouble, could you please report on your read performance?
2- The previous observation leads me to the next question: could you saymore about your use of disk with persistence on?ZooKeeper returns ACK after writing the disks of the over half machines.Accord returns ACK after writing the disk of just one machine, whichaccepted a request. However, at the same time, the ACK assures that allservers receive the messages in the same order.The difference of the semantics means that this measurement is not fair. I would like to measure the under fair situation, but not yet. If there are requests from users, I'm going to implement it and measure it. Notethat the benchmark of in-memory is fair.
I'm not sure I understand this part. You say that an operation is ACKed after being written to one disk, but also that it is guaranteed to be delivered in the same order in all servers. Does it mean that Accord still replicates on other servers before ACKing but the other servers do not write to disk? Otherwise, the first server may crash and never come back, and the message cannot possibly be delivered by other servers.
One question related to this point: with Accord, do you replicate the original request message or the result of operation? Do you guarantee that each server executes a request or applies the result of a request exactly once? If not, what kind of semantics does Accord provide?
3- The limitation on the message size in ZooKeeper is not a fundamentallimitation. We have chosen to limit for the reasons we explain in thewiki page that is linked in the Accord page. Do you have any particularuse case in mind for which you think it would be useful to have very large messages?Some developers use ZooKeeper as storage. For example, Onix developer, aimplementation of open flow switch, says that : "for most the object size limitations of Zookeeper and convenience of accessing the configuration state directly through the NIB are a reason to favor the transactional database." http://www.usenix.org/event/osdi10/tech/full_papers/Koponen.pdf
The comment in the paper is exactly right, we instruct our users to store metadata in ZooKeeper and data elsewhere. There are systems designed to store bulk data, and ZooKeeper shouldn't try to compete with such storage systems, it is not our goal.
4- If I understand the group communication substrate Accord uses, itenables Accord to process client requests in any server. ZooKeeper has aleader for a few reasons, one being the ability of managing clientsessions. Ephemeral nodes, for example, are bound to sessions. Are theresimilar abstractions in Accord? If the answer is positive, could youexplain it a bit? If not, is it doable with the substrate you're using?Yes, Accord has abstractions like Ephemeral nodes. We use Corosync cluster engine, which provides Virtual Synchronysemantics. It assures of having consensus of the message ordering and aserver-failure ordering among all servers(conductor daemons).
Good to know that you also support ephemerals. Could you say a little more about how you decide to eliminate an ephemeral node? I suppose that an ephemeral is bound to the client that created it somehow, and it is deleted if the client crashes or disconnects. What's the exact mechanism?
5- I'm not sure where we say that 8 bytes is a typical value in thedocumentation. I actually remember writing in one of our papers that atypical value is around 1k bytes.The benchmark assumes the lock. I'm going to measure various message sizes. I'll report it.
Sounds good, thanks. -Flavio
-Flavio On Sep 23, 2011, at 4:22 PM, OZAWA Tsuyoshi wrote:Hi, Sending zookeeper-users and hbase-users ml since there may be somecluster developers interested in participating in this project there.I am pleased to announce the initial release of Accord, yet another coordination service like Apache ZooKeeper. ZooKeeper is a de facto standard coordination kernel as you know at present.Accord provides ZK-like features as a coordination service. Concretelyspeaking, it features:- Accord is a distributed, transactional, and fully-replicated (No SPoF)Key-Value Store with strong consistency. - Accord can be scale-out up to tens of nodes. - Accord servers can handle tens or thousands of clients.- The changes for a write request from a client can be notified to theother clients. - Accord detects events of client's joining/leaving, and notifies joined/left client information to the other clients. There are some problems in ZK, however, as follows:- ZK cannot handle write-intensive workloads well. ZK forwards all writerequests to a master server. It may be bottleneck in write-intensive workload. - ZK is optimized for disk-persistence mode, not for in-memory mode. ZOOKEEPER-866 shows that ZK has the other bottleneck outside diskpersistence, though there are some needs of a fully-replicated storagewith both strong consistency and low latency. https://issues.apache.org/jira/browse/ZOOKEEPER-866- Limited Transaction APIs. ZK can only issue write operations (write,del) in a transaction(multi-update). These restriction limit the capability of the coordination kernel. Accord solves such problems. 1. Accord uses Corosync Cluster Engine as a total-order messaginginfrastructure instead of Zab, an atomic broadcast protocol ZK uses. Theengine enable any servers to accept and process requests. 2. Accord supports in-memory mode.3. More flexible transaction support. Not only write, del operations,but also cmp, copy, read operations are supported in transaction operation.These differences of the core engine (1, 2) enable us to avoid master bottleneck. Benchmark demonstrates that the write-operation throughputof Accord is much higher than one of ZooKeeper(up to 20 times better throughput at persistent mode, and up to 18 timesbetter throughput at in-memory mode).The high performance kernel can extend the application ranges. Assumedapplications are as follows, for instance : - Distributed Lock Manager whose lock operations occur at a high frequency from thousands of clients.I assume that the lock manager for Hbase in particluar. The coordinationservice enables HBase to update multiple rows with ACID properties.Hbase acts as distributed DB with ACID properties until the coordination service becomes the bottleneck. The new coordination kernel, Accord, canhandle 18 times better throughput than ZK. As a result, Accord can dramatically improve the scalability of Hbase with ACID properties. - Metadata management service for large-scale distributed storage, including HDFS, Ceph and Sheepdog etc. Replicated-master can be implemented easily. - Replicated Message Queue or logger (For instance, replicatedRabbitMQ).and so on. The other distributed systems can use Accord features easily because Accord provides general-purpose APIs (read/write/del/more flexible transaction).More information including getting started, benchmarks, and API docs areavailable from our project page : http://www.osrg.net/accord and all code is available from: http://github.com/collie/accord Please try it out, and let me know any opinions or problems. Best regards, OZAWA Tsuyoshi <[email protected]>flavio junqueira research scientist [email protected] direct +34 93-183-8828 avinguda diagonal 177, 8th floor, barcelona, 08018, es phone (408) 349 3300 fax (408) 349 3301
flavio junqueira research scientist [email protected] direct +34 93-183-8828 avinguda diagonal 177, 8th floor, barcelona, 08018, es phone (408) 349 3300 fax (408) 349 3301
