What heap size are you running with? and Which version of Cassandra? Thanks, Naren
On Thu, Mar 17, 2011 at 3:45 AM, ruslan usifov <ruslan.usi...@gmail.com>wrote: > Hello > > Some times i have very long GC pauses: > > > Total time for which application threads were stopped: 0.0303150 seconds > 2011-03-17T13:19:56.476+0300: 33295.671: [GC 33295.671: [ParNew: > 678855K->20708K(737280K), 0.0271230 secs] 1457643K->806795K(4112384K), > 0.027305 > 0 secs] [Times: user=0.33 sys=0.00, real=0.03 secs] > Total time for which application threads were stopped: 0.0291820 seconds > 2011-03-17T13:20:32.962+0300: 33332.157: [GC 33332.157: [ParNew: > 676068K->23527K(737280K), 0.0302180 secs] 1462155K->817599K(4112384K), > 0.030402 > 0 secs] [Times: user=0.31 sys=0.00, real=0.03 secs] > Total time for which application threads were stopped: 0.1270270 seconds > 2011-03-17T13:21:11.908+0300: 33371.103: [GC 33371.103: [ParNew: > 678887K->21564K(737280K), 0.0268160 secs] 1472959K->823191K(4112384K), > 0.027011 > 0 secs] [Times: user=0.28 sys=0.00, real=0.03 secs] > Total time for which application threads were stopped: 0.0293330 seconds > 2011-03-17T13:21:50.482+0300: 33409.677: [GC 33409.677: [ParNew: > 676924K->21115K(737280K), 0.0281720 secs] 1478551K->829900K(4112384K), > 0.028363 > 0 secs] [Times: user=0.27 sys=0.00, real=0.03 secs] > Total time for which application threads were stopped: 0.0339610 seconds > 2011-03-17T13:22:32.849+0300: 33452.044: [GC 33452.044: [ParNew: > 676475K->25948K(737280K), 0.0317600 secs] 1485260K->842061K(4112384K), > 0.031952 > 0 secs] [Times: user=0.22 sys=0.00, real=0.03 secs] > Total time for which application threads were stopped: 0.0344430 seconds > 2011-03-17T13:23:14.924+0300: 33494.119: [GC 33494.119: [ParNew: > 681308K->25087K(737280K), 0.0282600 secs] 1497421K->848300K(4112384K), > 0.028436 > 0 secs] [Times: user=0.32 sys=0.00, real=0.03 secs] > Total time for which application threads were stopped: 0.0309160 seconds > 2011-03-17T13:23:57.192+0300: 33536.387: [GC 33536.387: [ParNew: > 680447K->24805K(737280K), 0.0299910 secs] 1503660K->855829K(4112384K), > 0.030167 > 0 secs] [Times: user=0.29 sys=0.01, real=0.03 secs] > Total time for which application threads were stopped: 0.0324200 seconds > 2011-03-17T13:24:01.553+0300: 33540.748: [GC 33540.749: [ParNew: > 680165K->31886K(737280K), 0.0495620 secs] 1511189K->936503K(4112384K), > 0.049742 > 0 secs] [Times: user=0.57 sys=0.00, real=0.05 secs] > Total time for which application threads were stopped: 0.0507030 seconds > 2011-03-17T13:37:56.009+0300: 34375.204: [GC 34375.204: [ParNew: > 687246K->28727K(737280K), 0.0244720 secs] 1591863K->942459K(4112384K), > 0.024690 > 0 secs] [Times: user=0.18 sys=0.00, real=0.02 secs] > Total time for which application threads were stopped: 806.7442720 seconds > Total time for which application threads were stopped: 0.0006590 seconds > Total time for which application threads were stopped: 0.0004360 seconds > Total time for which application threads were stopped: 0.0004630 seconds > Total time for which application threads were stopped: 0.0008120 seconds > 2011-03-17T13:37:59.018+0300: 34378.213: [GC 34378.213: [ParNew: > 676678K->21640K(737280K), 0.0137740 secs] 1590410K->949991K(4112384K), > 0.013961 > 0 secs] [Times: user=0.13 sys=0.02, real=0.01 secs] > Total time for which application threads were stopped: 0.0145920 seconds > Total time for which application threads were stopped: 0.1036080 seconds > Total time for which application threads were stopped: 0.0585600 seconds > Total time for which application threads were stopped: 0.0600550 seconds > Total time for which application threads were stopped: 0.0008560 seconds > Total time for which application threads were stopped: 0.0006770 seconds > Total time for which application threads were stopped: 0.0005910 seconds > Total time for which application threads were stopped: 0.0351330 seconds > Total time for which application threads were stopped: 0.0329020 seconds > Total time for which application threads were stopped: 0.0728490 seconds > Total time for which application threads were stopped: 0.0480990 seconds > Total time for which application threads were stopped: 0.0804250 seconds > 2011-03-17T13:38:04.394+0300: 34383.589: [GC 34383.589: [ParNew: > 677000K->8375K(737280K), 0.0218310 secs] 1605351K->944271K(4112384K), > 0.0220300 > secs] > > > > > I have follow nodetoll cfstats on hung node: > > Keyspace: fishdom_tuenti > Read Count: 4970999 > Read Latency: 1.0267005945887335 ms. > Write Count: 1441619 > Write Latency: 0.013146585887117193 ms. > Pending Tasks: 0 > Column Family: decor > SSTable count: 3 > Space used (live): 1296203532 > Space used (total): 1302520037 > Memtable Columns Count: 1066 > Memtable Data Size: 121742 > Memtable Switch Count: 11 > Read Count: 108125 > Read Latency: 2.809 ms. > Write Count: 11261 > Write Latency: 0.006 ms. > Pending Tasks: 0 > Key cache capacity: 300000 > Key cache size: 46470 > Key cache hit rate: 0.40384615384615385 > Row cache: disabled > Compacted row minimum size: 36 > Compacted row maximum size: 73457 > Compacted row mean size: 958 > > Column Family: adopt > SSTable count: 1 > Space used (live): 3285283 > Space used (total): 6585649 > Memtable Columns Count: 14 > Memtable Data Size: 322 > Memtable Switch Count: 19 > Read Count: 208 > Read Latency: 0.009 ms. > Write Count: 248 > Write Latency: 0.005 ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 171 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 87 > Compacted row maximum size: 103 > Compacted row mean size: 103 > > Column Family: user_store > SSTable count: 0 > Space used (live): 0 > Space used (total): 0 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 0 > Read Count: 0 > Read Latency: NaN ms. > Write Count: 0 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 0 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 0 > Compacted row maximum size: 0 > Compacted row mean size: 0 > > Column Family: gifts > SSTable count: 1 > Space used (live): 8767135 > Space used (total): 17581781 > Memtable Columns Count: 9 > Memtable Data Size: 5993 > Memtable Switch Count: 11 > Read Count: 230 > Read Latency: NaN ms. > Write Count: 306 > Write Latency: 0.004 ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 186 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 125 > Compacted row maximum size: 14237 > Compacted row mean size: 498 > > Column Family: user_awards > SSTable count: 4 > Space used (live): 285142885 > Space used (total): 285142885 > Memtable Columns Count: 399 > Memtable Data Size: 21265 > Memtable Switch Count: 11 > Read Count: 602884 > Read Latency: 0.075 ms. > Write Count: 3741 > Write Latency: 0.003 ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 9692 > Key cache hit rate: 0.9929742388758782 > Row cache: disabled > Compacted row minimum size: 104 > Compacted row maximum size: 770 > Compacted row mean size: 262 > > Column Family: aquarium > SSTable count: 4 > Space used (live): 945412589 > Space used (total): 970196634 > Memtable Columns Count: 10973 > Memtable Data Size: 1805059 > Memtable Switch Count: 11 > Read Count: 737198 > Read Latency: 0.419 ms. > Write Count: 259785 > Write Latency: 0.011 ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 68960 > Key cache hit rate: 0.9497041420118343 > Row cache: disabled > Compacted row minimum size: 87 > Compacted row maximum size: 1916 > Compacted row mean size: 539 > > Column Family: user_messages > SSTable count: 1 > Space used (live): 4648148 > Space used (total): 9311688 > Memtable Columns Count: 23 > Memtable Data Size: 2525 > Memtable Switch Count: 19 > Read Count: 8782 > Read Latency: 0.012 ms. > Write Count: 246 > Write Latency: 0.012 ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 2707 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 180 > Compacted row maximum size: 9887 > Compacted row mean size: 380 > > Column Family: user_stat > SSTable count: 4 > Space used (live): 872814042 > Space used (total): 872814042 > Memtable Columns Count: 36852 > Memtable Data Size: 902102 > Memtable Switch Count: 11 > Read Count: 1663028 > Read Latency: 0.446 ms. > Write Count: 406925 > Write Latency: 0.010 ms. > Pending Tasks: 0 > Key cache capacity: 400000 > Key cache size: 78611 > Key cache hit rate: 0.9583333333333334 > Row cache: disabled > Compacted row minimum size: 51 > Compacted row maximum size: 770 > Compacted row mean size: 516 > > Column Family: user_stat_friend > SSTable count: 4 > Space used (live): 614446488 > Space used (total): 614446488 > Memtable Columns Count: 9288 > Memtable Data Size: 250169 > Memtable Switch Count: 11 > Read Count: 237287 > Read Latency: 1.147 ms. > Write Count: 116691 > Write Latency: 0.006 ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 39410 > Key cache hit rate: 0.8387096774193549 > Row cache: disabled > Compacted row minimum size: 61 > Compacted row maximum size: 258 > Compacted row mean size: 199 > > Column Family: aquarium_friend > SSTable count: 4 > Space used (live): 7135867112 > Space used (total): 7145863363 > Memtable Columns Count: 6061 > Memtable Data Size: 618699 > Memtable Switch Count: 11 > Read Count: 333994 > Read Latency: 1.358 ms. > Write Count: 168818 > Write Latency: 0.010 ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 47973 > Key cache hit rate: 0.8808664259927798 > Row cache: disabled > Compacted row minimum size: 104 > Compacted row maximum size: 924 > Compacted row mean size: 304 > > Column Family: aquarium_messages > SSTable count: 1 > Space used (live): 5949016 > Space used (total): 11912616 > Memtable Columns Count: 11 > Memtable Data Size: 1355 > Memtable Switch Count: 19 > Read Count: 26441 > Read Latency: 0.010 ms. > Write Count: 279 > Write Latency: 0.003 ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 4095 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 180 > Compacted row maximum size: 14237 > Compacted row mean size: 382 > > Column Family: user_params > SSTable count: 0 > Space used (live): 0 > Space used (total): 0 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 0 > Read Count: 0 > Read Latency: NaN ms. > Write Count: 0 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 0 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 0 > Compacted row maximum size: 0 > Compacted row mean size: 0 > > Column Family: gems > SSTable count: 4 > Space used (live): 582170315 > Space used (total): 582170315 > Memtable Columns Count: 8713 > Memtable Data Size: 285047 > Memtable Switch Count: 11 > Read Count: 97307 > Read Latency: 0.733 ms. > Write Count: 35400 > Write Latency: 0.009 ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 21120 > Key cache hit rate: 0.8691588785046729 > Row cache: disabled > Compacted row minimum size: 36 > Compacted row maximum size: 2759 > Compacted row mean size: 691 > > Column Family: adopt_accept > SSTable count: 1 > Space used (live): 3531249 > Space used (total): 7077144 > Memtable Columns Count: 15 > Memtable Data Size: 433 > Memtable Switch Count: 19 > Read Count: 271 > Read Latency: 0.011 ms. > Write Count: 249 > Write Latency: 0.004 ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 325 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 87 > Compacted row maximum size: 179 > Compacted row mean size: 109 > > Column Family: gems_friend > SSTable count: 5 > Space used (live): 1620588533 > Space used (total): 1620588533 > Memtable Columns Count: 21441 > Memtable Data Size: 577929 > Memtable Switch Count: 11 > Read Count: 265787 > Read Latency: 0.974 ms. > Write Count: 135310 > Write Latency: 0.011 ms. > Pending Tasks: 0 > Key cache capacity: 200000 > Key cache size: 49702 > Key cache hit rate: 0.8641304347826086 > Row cache: disabled > Compacted row minimum size: 36 > Compacted row maximum size: 1916 > Compacted row mean size: 536 > > Column Family: fish > SSTable count: 5 > Space used (live): 3424896182 > Space used (total): 3543066763 > Memtable Columns Count: 88706 > Memtable Data Size: 10660109 > Memtable Switch Count: 11 > Read Count: 889507 > Read Latency: 0.718 ms. > Write Count: 302385 > Write Latency: 0.024 ms. > Pending Tasks: 0 > Key cache capacity: 400000 > Key cache size: 90802 > Key cache hit rate: 0.9509594882729211 > Row cache: disabled > Compacted row minimum size: 51 > Compacted row maximum size: 51012 > Compacted row mean size: 1890 > > ---------------- >
