I did some analysis using iostat and vmstat and those are the results: When the node freezes (I'm not running on a vm, I'm running on 2 cpu 8 cores machine with 12G RAM):
sda 0.00 9791.20 0.00 93.60 0.00 94080.00 1005.13 140.46 1589.95 10.69 100.02 does that mean that the device just can't handle the io produced by casandra? Do you think it will help if I 'll use several devices and not one? On Thu, Feb 25, 2010 at 4:56 PM, Jonathan Ellis <jbel...@gmail.com> wrote: > Are you swapping? > http://spyced.blogspot.com/2010/01/linux-performance-basics.html > > otherwise there's something wrong w/ your vm (?), disk i/o doesn't > block incoming writes in cassandra > > On Thu, Feb 25, 2010 at 8:49 AM, Boris Shulman <shulm...@gmail.com> wrote: >> I don't think it is gc related issue. There is no correlation between >> gc times and the freeze times. More over I don't see any gc activity >> that lasts for omre than o.03 sec. But there is a correlation between >> disk flushing operations. I've noticed that the system freezes each >> time when my commit log reaches 1.1G. I have 1024M memtable size so I >> assume this is when the data flushing occurs. >> >> >> >> On Thu, Feb 25, 2010 at 4:13 PM, Jonathan Ellis <jbel...@gmail.com> wrote: >>> Then you should check GC timing with -Xverbose:gc option (see: >>> http://wiki.apache.org/cassandra/RunningCassandra for how to modify >>> jvm options) for a correlation. >>> >>> On Thu, Feb 25, 2010 at 8:09 AM, Boris Shulman <shulm...@gmail.com> wrote: >>>> In these tests I perform only write operations, no reads. >>>> >>>> On Thu, Feb 25, 2010 at 4:07 PM, Jonathan Ellis <jbel...@gmail.com> wrote: >>>>> The only kind of "freeze" that makes sense there is your reads are i/o >>>>> bound and the extra disk activity is killing you. In that case the >>>>> fix is to add more RAM, or give less to the JVM so the OS can use more >>>>> for buffer cache. >>>>> >>>>> On Thu, Feb 25, 2010 at 8:01 AM, Boris Shulman <shulm...@gmail.com> wrote: >>>>>> In my case the cassandra node freezes while memtable flush operation >>>>>> is performed or compactation operation is performed. How can I >>>>>> optimize the cassandra configuration in order to avoid this behavior? >>>>>> I've tried both using large memtable size (1G) and small (128M) but in >>>>>> every case I have some sort of freezes when the data is flushed to the >>>>>> disk. >>>>>> >>>>>> Please advice. >>>>>> >>>>> >>>> >>> >> >
