One pointer to look at would be the memory snapshot and try tuning GC around that if you find something fishy. Could be that it tries to load everything in memory and then it has to do garbage collection which adds pause times to it.
Turn on the GC logging by changing the parameters in conf/cassandra-env.sh Second pointer would be to change the file system if u can to XFS or ext4. On Fri, Jul 20, 2012 at 7:37 AM, Edward Capriolo <edlinuxg...@gmail.com>wrote: > Local storage is more the just the norm. Unless you have a very good > reason you should not be using NFS. > > Edward > > On Fri, Jul 20, 2012 at 4:55 AM, aaron morton <aa...@thelastpickle.com> > wrote: > > 45 minutes for 90GB is high. > > > > The odd one out here is using NFS, local storage is the norm. > > > > I would look into the NFS first, low network IO and low CPU would > suggest it > > is waiting on disk IO. The simple thing would be to try starting from > local > > disk and see how much faster it is. Or look at the await time in iostat. > > > > Hope that helps. > > > > ----------------- > > Aaron Morton > > Freelance Developer > > @aaronmorton > > http://www.thelastpickle.com > > > > On 19/07/2012, at 4:54 PM, Ben Kaehne wrote: > > > > Good evening, > > > > I am interested in improving the startup time of our cassandra cluster. > > > > We have a 3 node cluster (replication factor of 3) in which our > application > > requires quorum reads and writes to function. > > > > Each machine is well specced with 24gig of ram, 10 cores, jna enabled > etc. > > > > On each server our keyspace files are so far around 90 Gb (stored on NFS > > although I am not seeing signs that we have much network io). This size > will > > grow in future. > > > > Our startup time for 1 server at the moment is greater then half an hour > (45 > > minutes to 50 minutes even) which is putting a risk factor on the > resiliance > > of our service. I have tried version 1.09 to latest 1.12. > > > > I do not see too much system utilization while starting either. > > > > I gazed apon an article suggesting increased speed in 1.2 although when I > > set it up, it did not seem to be any faster at all (if not slower). > > > > I was observing what was happening during startup and I noticed (via > > strace), cassandra was doing lots of 8 byte reads from: > > > > > > > /var/lib/cassandra/data/XXXXXX/YYYYYY/XXXXXXX-YYYYYYY-hc-1871-CompressionInfo.db > > > > > /var/lib/cassandra/data/XXXXXX/YYYYYY/XXXXXXX-YYYYYYY-hc-1874-CompressionInfo.db > > > > Also... Is there someone I can change the 8 byte reads to something > greater? > > 8 byte reads across NFS is terribly inefficient (and I am guessing the > cause > > of our terribly slow startup times). > > > > Regards, > > > > -- > > -Ben > > > > >
