It's a tricky thing to get right. Do you have wide rows ?
Try enabling the GC logging in cassandra-env.sh. You may want to try changing: JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=4" JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=4" This will slow down objects being promoted to the tenured heap, and GC logging will show a distribution of how long objects are staying in the new heap. If you have wide rows you may want to run XX:MaxTenuringThreshold 2. Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 11/05/2013, at 5:56 AM, Bao Le <[email protected]> wrote: > My cluster of 11 nodes running Casandra 1.1-5 is pausing too long for ParNew > GC, which increases our response latency, Is it a good idea to have a a > smaller HEAP_NEWSIZE so that we can collect more often, but not pause that > long? > > INFO [ScheduledTasks:1] 2013-05-10 01:00:17,245 GCInspector.java (line 122) > GC for ParNew: 252 ms for 1 collections, 3238218616 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 02:15:20,650 GCInspector.java (line 122) > GC for ParNew: 445 ms for 1 collections, 4810760088 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 02:30:19,932 GCInspector.java (line 122) > GC for ParNew: 419 ms for 2 collections, 5210373288 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 02:40:23,201 GCInspector.java (line 122) > GC for ParNew: 333 ms for 1 collections, 2172614912 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 03:45:19,975 GCInspector.java (line 122) > GC for ParNew: 201 ms for 1 collections, 4134399864 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 03:55:20,345 GCInspector.java (line 122) > GC for ParNew: 685 ms for 1 collections, 4696326432 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 04:40:21,957 GCInspector.java (line 122) > GC for ParNew: 379 ms for 1 collections, 4051166216 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 04:50:23,057 GCInspector.java (line 122) > GC for ParNew: 334 ms for 1 collections, 4695497128 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 05:05:20,304 GCInspector.java (line 122) > GC for ParNew: 222 ms for 1 collections, 5527026728 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 05:35:21,848 GCInspector.java (line 122) > GC for ParNew: 279 ms for 1 collections, 3138206504 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 05:45:19,939 GCInspector.java (line 122) > GC for ParNew: 353 ms for 1 collections, 3445606832 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 05:55:21,326 GCInspector.java (line 122) > GC for ParNew: 344 ms for 1 collections, 4331945664 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 06:05:20,424 GCInspector.java (line 122) > GC for ParNew: 214 ms for 1 collections, 4787806520 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 07:00:21,402 GCInspector.java (line 122) > GC for ParNew: 256 ms for 1 collections, 5119566040 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 07:15:20,747 GCInspector.java (line 122) > GC for ParNew: 512 ms for 2 collections, 2068901896 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 07:30:19,081 GCInspector.java (line 122) > GC for ParNew: 267 ms for 1 collections, 2614774320 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 08:10:22,440 GCInspector.java (line 122) > GC for ParNew: 305 ms for 1 collections, 4042611368 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 08:15:20,482 GCInspector.java (line 122) > GC for ParNew: 371 ms for 1 collections, 4365244824 used; max is 8422162432 > INFO [ScheduledTasks:1] 2013-05-10 08:25:20,047 GCInspector.java (line 122) > GC for ParNew: 251 ms for 1 collections, 4900957800 used; max is 8422162432 > > Thanks > Bao
