Re: Upgrading an ensemble from 3.3.5 to 3.4.4

2012-10-03 Thread Patrick Hunt
That's also fine, but also not extensively tested. Many users do rolling upgrades, but typically that would cover "one hop" server version upgrade, but given we only do bug fixes in fix releases that shouldn't be an issue here. Patrick On Tue, Oct 2, 2012 at 8:29 AM, Jordan Zimmerman wrote: > Wh

Re: zookeeper on SSD

2012-10-03 Thread Ted Dunning
This is a good observation. On Thu, Oct 4, 2012 at 5:39 AM, Andrew Purtell wrote: > Even so, I've seen in notes from attendees of Amazon's "DynamoDB For > Developers" talks that Amazon says they found it necessary to work > "extensively" with their SSD vendor (not stated publicly AFAIK) to > eng

Re: zookeeper on SSD

2012-10-03 Thread Andrew Purtell
Even so, I've seen in notes from attendees of Amazon's "DynamoDB For Developers" talks that Amazon says they found it necessary to work "extensively" with their SSD vendor (not stated publicly AFAIK) to engineer out latency spikes. I'd imagine they started with a strong vendor and not a low end dev

Re: zookeeper on SSD

2012-10-03 Thread Milind Parikh
It does seem so. "The most performance critical part of ZooKeeper is the transaction log. ZooKeeper syncs transactions to media before it returns a response. A dedicated transaction log device is key to consistent good performance. Putting the log on a busy device will adversely effect performance

Re: zookeeper on SSD

2012-10-03 Thread Patrick Hunt
On Wed, Oct 3, 2012 at 9:12 PM, Jun Rao wrote: > Patrick, > > Thanks for the info. Does each ZK write wait for log being flushed to disk? > Yes (it's necessary for the guarantees we provide), although the servers do batching of writes to improve throughput. This doc gives some insight: http://zo

Re: zookeeper on SSD

2012-10-03 Thread Ted Dunning
Yes. And Patrick's experience is not unexpected. There is, however, a huge variation with different types of flash memory. The software driving the flash can also result in very different experience. The experiences that he alludes to are likely with a conventional SSD packaging of flash driven

Re: zookeeper on SSD

2012-10-03 Thread Jun Rao
Patrick, Thanks for the info. Does each ZK write wait for log being flushed to disk? Jun On Wed, Oct 3, 2012 at 6:13 PM, Patrick Hunt wrote: > My experience with SSDs and ZK has been discouraging. SSDs have some > really terrible corner cases for latency. I've seen them take 40+ > seconds (tha

Re: zookeeper on SSD

2012-10-03 Thread Patrick Hunt
My experience with SSDs and ZK has been discouraging. SSDs have some really terrible corner cases for latency. I've seen them take 40+ seconds (that's not a mistake - seconds) for fsync to complete. When this happened (every few hours) all of the sessions would timeout. See this article: http://st

Announcing availability of Net::ZKMon perl wrapper module for ZooKeeper monitoring

2012-10-03 Thread Nikhil
Hi, I have put together a perl module named 'Net::ZKMon', a wrapper module around the ZooKeeper's 4-letter word commands meant for zookeeper monitoring. This module is written and meant to the serve the purpose of - providing a perl interface to executing 4-letter command as just another perl

Re: identify the cause of a large quantity of zookeeper threads

2012-10-03 Thread Camille Fournier
Are you properly reusing your ZooKeeper clients? You should (probably) create one client and share it throughout your process to send requests to the server, not create a new client per usage. Seems like you're not creating and reusing clients appropriately. C On Wed, Oct 3, 2012 at 2:21 PM, Reyn

identify the cause of a large quantity of zookeeper threads

2012-10-03 Thread Reynold Xin
I am trying to debug a problem in which when I run jstack, I see a very large number of zookeeper threads, and as the program continues running, more threads like the following are created. I suspect somewhere in the program it is not releasing some resource properly, but I am not sure where it is.