Re: Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-17 Thread Erik Onnen
Unfortunately, the previous AMI we used to provision the 7.5 version is no longer available. More unfortunately, the two test nodes we spun up in each AZ did not get Nehalem architectures so the only things I can say for certain after running Mike's test 10x on each test node are: 1) I could not r

Re: Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-14 Thread Mike Malone
That's interesting. For us, the 7.5 version of libc was causing problems. Either way, I'm looking forward to hearing about anything you find. Mike On Thu, Jan 13, 2011 at 11:47 PM, Erik Onnen wrote: > Too similar to be a coincidence I'd say: > > Good node (old AZ): 2.11.1-0ubuntu7.5 > Bad node

Re: Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-13 Thread Erik Onnen
Too similar to be a coincidence I'd say: Good node (old AZ): 2.11.1-0ubuntu7.5 Bad node (new AZ): 2.11.1-0ubuntu7.6 You beat me to the punch with the test program. I was working on something similar to test it out and got side tracked. I'll try the test app tomorrow and verify the versions of t

Re: Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-13 Thread Mike Malone
Erik, the scenario you're describing is almost identical to what we've been experiencing. Sounds like you've been pulling your hair out too! You're also running the same distro and kernel as us. And we also run without swap. Which begs the question... what version of libc6 are you running!? Here's

Re: Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-13 Thread Erik Onnen
Forgot one critical point, we use zero swap on any of these hosts.

Re: Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-13 Thread Erik Onnen
May or may not be related but I thought I'd recount a similar experience we had in EC2 in hopes it helps someone else. As background, we had been running several servers in a 0.6.8 ring with no Cassandra issues (some EC2 issues, but none related to Cassandra) on multiple EC2 XL instances in a sing

Cassandra freezes under load when using libc6 2.11.1-0ubuntu7.5

2011-01-13 Thread Mike Malone
Hey folks, We've discovered an issue on Ubuntu/Lenny with libc6 2.11.1-0ubuntu7.5 (it may also affect versions between 2.11.1-0ubuntu7.1 and 2.11.1-0ubuntu7.4). The bug affects systems when a large number of threads (or processes) are created rapidly. Once triggered, the system will become complet