Re: Using a separate commit log drive was 4x slower

2010-08-10 Thread Peter Schuller
I have no explanation for the slower reads, but I have an hypothesis on the writes. Your iostat shows: Device: rrqm/s   wrqm/s r/s w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util cciss/c0d0    0.00   908.50    0.00  110.50 0.00  8152.00    73.77

Re: non blocking Cassandra with Tornado

2010-08-10 Thread Ryan Daum
Barring this we (place where I work, Chango) will probably eventually fork Cassandra to have a RESTful interface and use the Jetty async HTTP client to connect to it. It's just ridiculous for us to have threads and associated resources tied up on I/O-blocked operations. We've done exactly

Re: Growing commit log directory.

2010-08-10 Thread Jonathan Ellis
having 3 digit pending counts in both RRS and RMS is a danger sign. It looks like you are i/o bound on reads, and possibly on writes as well. (commitlog not on separate disk?) On Mon, Aug 9, 2010 at 10:53 PM, Edward Capriolo edlinuxg...@gmail.com wrote: On Mon, Aug 9, 2010 at 8:20 PM, Jonathan

Re: COMMIT-LOG_WRITER Assertion Error

2010-08-10 Thread Jonathan Ellis
Can you create a ticket for this? On Mon, Aug 9, 2010 at 8:42 PM, Arya Goudarzi agouda...@gaiaonline.com wrote: I've never run 0.6. I have been running of trunc with automatic svn update and build everyday at 2pm. One of my nodes got this error which lead to the same last error prior to

why does it take 60-90 seconds for a new node to get up?

2010-08-10 Thread S Ahmed
Why is it that, if you set AutoBootStrap = false that it takes 60-90 seconds for the node to announce itself? I just want to understand what is going on during that time, and why that specific timeframe (if there is a reason?)

Re: explanation of generated files and ops

2010-08-10 Thread Peter Schuller
Is this pretty much all the files that Cassandra generates? (have I missed any) I believe so. There will also be some temporary files (*-tmp*) during compaction, and you'll see some *.Compacted marker files (empty IIRC). What exactly is stored in the -Filter.db files? Those are bloom

Re: COMMIT-LOG_WRITER Assertion Error

2010-08-10 Thread Arya Goudarzi
Sure. https://issues.apache.org/jira/browse/CASSANDRA-1376 - Original Message - From: Jonathan Ellis jbel...@gmail.com To: user@cassandra.apache.org Sent: Tuesday, August 10, 2010 7:05:31 AM Subject: Re: COMMIT-LOG_WRITER Assertion Error Can you create a ticket for this? On Mon, Aug 9,

Re: Using a separate commit log drive was 4x slower

2010-08-10 Thread Jeremy Davis
Yeah, it has a BBU, and it is charged and on.. Very odd behavior, I'm stumped. -JD On Tue, Aug 10, 2010 at 12:28 AM, Peter Schuller peter.schul...@infidyne.com wrote: I have no explanation for the slower reads, but I have an hypothesis on the writes. Your iostat shows: Device:

Re: Using a separate commit log drive was 4x slower

2010-08-10 Thread Jonathan Ellis
Other activity, e.g. syslog? Journaling at the FS level? you could try making a small partition formatted as ext2. On Tue, Aug 10, 2010 at 5:10 PM, Jeremy Davis jerdavis.cassan...@gmail.com wrote: Yeah, it has a BBU, and it is charged and on.. Very odd behavior, I'm stumped. -JD On Tue,

Re: Using a separate commit log drive was 4x slower

2010-08-10 Thread Peter Schuller
Yeah, it has a BBU, and it is charged and on.. Very odd behavior, I'm stumped. I advise double-checking raid volume settings and ensuring that policy is truly such that the write cache is used. This may also be a function of kernel driver settings depending on what RAID controller/kernel

Re: explanation of generated files and ops

2010-08-10 Thread Robert Coli
On 8/9/10 9:00 PM, S Ahmed wrote: Is this pretty much all the files that Cassandra generates? (have I missed any) If you are running cassandra via the linux init scripts, you are setting outfile for jsvc to be : -outfile /var/log/$NAME/output.log \ And with : -errfile 1 \

Soliciting thoughts on possible read optimization

2010-08-10 Thread Arya Asemanfar
I mentioned this today to a couple folks at Cassandra Summit, and thought I'd solicit some more thoughts here. Currently, the read stage includes checking row cache. So if your concurrent reads is N and you have N reads reading from disk, the next read will block until a disk read finishes, even

Re: explanation of generated files and ops

2010-08-10 Thread Robert Coli
On 8/9/10 9:00 PM, S Ahmed wrote: What exactly is stored in the -Filter.db files? Oh, didn't see this line. http://wiki.apache.org/cassandra/ArchitectureOverview Bloom filter (all keys in data file). A Bloom filter, is a space-efficient probabilistic data structure that is used to test