I'm facing the same issue with swap. It only occurs when I perform read operations (write are very fast :)). So I can't help you with the memory probleme.
But to balance the load evenly between nodes in cluster just manually fix their token.(the "formula" is i * 2^127 / nb_nodes). Jordzn On Tue, May 4, 2010 at 8:20 AM, Ran Tavory <ran...@gmail.com> wrote: > I'm looking into performance issues on a 0.6.1 cluster. I see two symptoms: > 1. Reads and writes are slow > 2. One of the hosts is doing a lot of GC. > > 1 is slow in the sense that in normal state the cluster used to make around > 3-5k read and writes per second (6-10k operations per second), but how it's > in the order of 200-400 ops per second, sometimes even less. > 2 looks like this: > $ tail -f /outbrain/cassandra/log/system.log > INFO [GC inspection] 2010-05-04 00:42:18,636 GCInspector.java (line 110) > GC for ParNew: 672 ms, 166482384 reclaimed leaving 2872087208 used; max is > 4432068608 > INFO [GC inspection] 2010-05-04 00:42:28,638 GCInspector.java (line 110) > GC for ParNew: 498 ms, 166493352 reclaimed leaving 2836049448 used; max is > 4432068608 > INFO [GC inspection] 2010-05-04 00:42:38,640 GCInspector.java (line 110) > GC for ParNew: 327 ms, 166091528 reclaimed leaving 2796888424 used; max is > 4432068608 > ... and it goes on and on for hours, no stopping... > > The cluster is made of 6 hosts, 3 in one DC and 3 in another. > Each host has 8G RAM. > -Xmx=4G > > For some reason, the load isn't distributed evenly b/w the hosts, although > I'm not sure this is the cause for slowness > $ nodetool -h localhost -p 9004 ring > Address Status Load Range > Ring > > 144413773383729447702215082383444206680 > 192.168.252.99Up 15.94 GB > 66002764663998929243644931915471302076 |<--| > 192.168.254.57Up 19.84 GB > 81288739225600737067856268063987022738 | ^ > 192.168.254.58Up 973.78 MB > 86999744104066390588161689990810839743 v | > 192.168.252.62Up 5.18 GB > 88308919879653155454332084719458267849 | ^ > 192.168.254.59Up 10.57 GB > 142482163220375328195837946953175033937 v | > 192.168.252.61Up 11.36 GB > 144413773383729447702215082383444206680 |-->| > > The slow host is 192.168.252.61 and it isn't the most loaded one. > > The host is waiting a lot on IO and the load average is usually 6-7 > $ w > 00:42:56 up 11 days, 13:22, 1 user, load average: 6.21, 5.52, 3.93 > > $ vmstat 5 > procs -----------memory---------- ---swap-- -----io---- --system-- > -----cpu------ > r b swpd free buff cache si so bi bo in cs us sy id > wa st > 0 8 2147844 45744 1816 4457384 6 5 66 32 5 2 1 1 > 96 2 0 > 0 8 2147164 49020 1808 4451596 385 0 2345 58 3372 9957 2 2 > 78 18 0 > 0 3 2146432 45704 1812 4453956 342 0 2274 108 3937 10732 2 2 > 78 19 0 > 0 1 2146252 44696 1804 4453436 345 164 1939 294 3647 7833 2 2 > 78 18 0 > 0 1 2145960 46924 1744 4451260 158 0 2423 122 4354 14597 2 2 > 77 18 0 > 7 1 2138344 44676 952 4504148 1722 403 1722 406 1388 439 87 0 > 10 2 0 > 7 2 2137248 45652 956 4499436 1384 655 1384 658 1356 392 87 0 > 10 3 0 > 7 1 2135976 46764 956 4495020 1366 718 1366 718 1395 380 87 0 > 9 4 0 > 0 8 2134484 46964 956 4489420 1673 555 1814 586 1601 215590 14 > 2 68 16 0 > 0 1 2135388 47444 972 4488516 785 833 2390 995 3812 8305 2 2 > 77 20 0 > 0 10 2135164 45928 980 4488796 788 543 2275 626 36 > > So, the host is swapping like crazy... > > top shows that it's using a lot of memory. As noted before -Xmx=4G and > nothing else seems to be using a lot of memory on the host except for the > cassandra process, however, of the 8G ram on the host, 92% is used by > cassandra. How's that? > Top shows there's 3.9g Shared and 7.2g Resident and *15.9g Virtual*. Why > does it have 15g virtual? And why 7.2 RES? This can explain the slowness in > swapping. > > $ top > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > 20281 cassandr 25 0 15.9g 7.2g 3.9g S 33.3 92.6 175:30.27 java > > So, can the total memory be controlled? > Or perhaps I'm looking in the wrong direction... > > I've looked at all the cassandra JMX counts and nothing seemed suspicious > so far. By suspicious i mean a large number of pending tasks - there were > always very small numbers in each pool. > About read and write latencies, I'm not sure what the normal state is, but > here's an example of what I see on the problematic host: > > #mbean = org.apache.cassandra.service:type=StorageProxy: > RecentReadLatencyMicros = 30105.888180684495; > TotalReadLatencyMicros = 78543052801; > TotalWriteLatencyMicros = 4213118609; > RecentWriteLatencyMicros = 1444.4809201925639; > ReadOperations = 4779553; > RangeOperations = 0; > TotalRangeLatencyMicros = 0; > RecentRangeLatencyMicros = NaN; > WriteOperations = 4740093; > > And the only pool that I do see some pending tasks is the ROW-READ-STAGE, > but it doesn't look like much, usually around 6-8: > #mbean = org.apache.cassandra.concurrent:type=ROW-READ-STAGE: > ActiveCount = 8; > PendingTasks = 8; > CompletedTasks = 5427955; > > Any help finding the solution is appreciated, thanks... > > Below are a few more JMXes I collected from the system that may be > interesting. > > #mbean = java.lang:type=Memory: > Verbose = false; > > HeapMemoryUsage = { > committed = 3767279616; > init = 134217728; > max = 4293656576; > used = 1237105080; > }; > > NonHeapMemoryUsage = { > committed = 35061760; > init = 24313856; > max = 138412032; > used = 23151320; > }; > > ObjectPendingFinalizationCount = 0; > > #mbean = java.lang:name=ParNew,type=GarbageCollector: > LastGcInfo = { > GcThreadCount = 11; > duration = 136; > endTime = 42219272; > id = 11719; > memoryUsageAfterGc = { > ( CMS Perm Gen ) = { > key = CMS Perm Gen; > value = { > committed = 29229056; > init = 21757952; > max = 88080384; > used = 17648848; > }; > }; > ( Code Cache ) = { > key = Code Cache; > value = { > committed = 5832704; > init = 2555904; > max = 50331648; > used = 5563520; > }; > }; > ( CMS Old Gen ) = { > key = CMS Old Gen; > value = { > committed = 3594133504; > init = 112459776; > max = 4120510464; > used = 964565720; > }; > }; > ( Par Eden Space ) = { > key = Par Eden Space; > value = { > committed = 171835392; > init = 21495808; > max = 171835392; > used = 0; > }; > }; > ( Par Survivor Space ) = { > key = Par Survivor Space; > value = { > committed = 1310720; > init = 131072; > max = 1310720; > used = 0; > }; > }; > }; > memoryUsageBeforeGc = { > ( CMS Perm Gen ) = { > key = CMS Perm Gen; > value = { > committed = 29229056; > init = 21757952; > max = 88080384; > used = 17648848; > }; > }; > ( Code Cache ) = { > key = Code Cache; > value = { > committed = 5832704; > init = 2555904; > max = 50331648; > used = 5563520; > }; > }; > ( CMS Old Gen ) = { > key = CMS Old Gen; > value = { > committed = 3594133504; > init = 112459776; > max = 4120510464; > used = 959221872; > }; > }; > ( Par Eden Space ) = { > key = Par Eden Space; > value = { > committed = 171835392; > init = 21495808; > max = 171835392; > used = 171835392; > }; > }; > ( Par Survivor Space ) = { > key = Par Survivor Space; > value = { > committed = 1310720; > init = 131072; > max = 1310720; > used = 0; > }; > }; > }; > startTime = 42219136; > }; > CollectionCount = 11720; > CollectionTime = 4561730; > Name = ParNew; > Valid = true; > MemoryPoolNames = [ Par Eden Space, Par Survivor Space ]; > > #mbean = java.lang:type=OperatingSystem: > MaxFileDescriptorCount = 63536; > OpenFileDescriptorCount = 75; > CommittedVirtualMemorySize = 17787711488; > FreePhysicalMemorySize = 45522944; > FreeSwapSpaceSize = 2123968512; > ProcessCpuTime = 12251460000000; > TotalPhysicalMemorySize = 8364417024; > TotalSwapSpaceSize = 4294959104; > Name = Linux; > AvailableProcessors = 8; > Arch = amd64; > SystemLoadAverage = 4.36; > Version = 2.6.18-164.15.1.el5; > > #mbean = java.lang:type=Runtime: > Name = 20...@ob1061.nydc1.outbrain.com; > > ClassPath = > /outbrain/cassandra/apache-cassandra-0.6.1/bin/../conf:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../build/classes:/outbrain/cassandra/apache-cassandra-0.6.1/bin/.. > > /lib/antlr-3.1.3.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/apache-cassandra-0.6.1.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/avro-1.2.0-dev.jar:/outb > > rain/cassandra/apache-cassandra-0.6.1/bin/../lib/clhm-production.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/commons-cli-1.1.jar:/outbrain/cassandra/apache-cassandra- > > 0.6.1/bin/../lib/commons-codec-1.2.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/commons-collections-3.2.1.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/com > > mons-lang-2.4.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/google-collections-1.0.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/hadoop-core-0.20.1.jar:/out > > brain/cassandra/apache-cassandra-0.6.1/bin/../lib/high-scale-lib.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/ivy-2.1.0.jar:/outbrain/cassandra/apache-cassandra-0.6.1/ > > bin/../lib/jackson-core-asl-1.4.0.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/jackson-mapper-asl-1.4.0.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/jline > > -0.9.94.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/json-simple-1.1.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/libthrift-r917130.jar:/outbrain/cassandr > > a/apache-cassandra-0.6.1/bin/../lib/log4j-1.2.14.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/slf4j-api-1.5.8.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib > /slf4j-log4j12-1.5.8.jar; > > BootClassPath = > /usr/java/jdk1.6.0_17/jre/lib/alt-rt.jar:/usr/java/jdk1.6.0_17/jre/lib/resources.jar:/usr/java/jdk1.6.0_17/jre/lib/rt.jar:/usr/java/jdk1.6.0_17/jre/lib/sunrsasign.j > > ar:/usr/java/jdk1.6.0_17/jre/lib/jsse.jar:/usr/java/jdk1.6.0_17/jre/lib/jce.jar:/usr/java/jdk1.6.0_17/jre/lib/charsets.jar:/usr/java/jdk1.6.0_17/jre/classes; > > LibraryPath = > /usr/java/jdk1.6.0_17/jre/lib/amd64/server:/usr/java/jdk1.6.0_17/jre/lib/amd64:/usr/java/jdk1.6.0_17/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib; > > VmName = Java HotSpot(TM) 64-Bit Server VM; > > VmVendor = Sun Microsystems Inc.; > > VmVersion = 14.3-b01; > > BootClassPathSupported = true; > > InputArguments = [ -ea, -Xms128M, -Xmx4G, -XX:TargetSurvivorRatio=90, > -XX:+AggressiveOpts, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, > -XX:+CMSParallelRemarkEnabled, -XX:+HeapDumpOnOutOfMemoryError, > -XX:SurvivorRatio=128, -XX:MaxTenuringThreshold=0, > -Dcom.sun.management.jmxremote.port=9004, > -Dcom.sun.management.jmxremote.ssl=false, > -Dcom.sun.management.jmxremote.authenticate=false, > -Dstorage-config=/outbrain/cassandra/apache-cassandra-0.6.1/bin/../conf, > -Dcassandra-pidfile=/var/run/cassandra.pid ]; > > ManagementSpecVersion = 1.2; > > SpecName = Java Virtual Machine Specification; > > SpecVendor = Sun Microsystems Inc.; > > SpecVersion = 1.0; > > StartTime = 1272911001415; > ... >