After investigating it deeper, I suspect it's native memory leak of JVM. The large anonymous map on lower address space should be the native heap of JVM, but not java object heap. Has anybody met it before?
I'll try to upgrade the JVM tonight. best regards, hanzhu On Thu, Dec 16, 2010 at 10:50 AM, Zhu Han <schumi....@gmail.com> wrote: > Hi, > > I have a test node with apache-cassandra-0.6.8 on ubuntu 10.4. The > hardware environment is an OpenVZ container. JVM settings is > # java -Xmx128m -version > java version "1.6.0_18" > OpenJDK Runtime Environment (IcedTea6 1.8.2) (6b18-1.8.2-4ubuntu2) > OpenJDK 64-Bit Server VM (build 16.0-b13, mixed mode) > > This is the memory settings: > > "/usr/bin/java -ea -Xms1G -Xmx1G ..." > > And the ondisk footprint of sstables is very small: > > "#du -sh data/ > "9.8M data/" > > The node was infrequently accessed in the last three weeks. After that, I > observe the abnormal memory utilization by top: > > PID USER PR NI *VIRT* *RES* SHR S %CPU %MEM TIME+ > COMMAND > > 7836 root 15 0 *3300m* *2.4g* 13m S 0 26.0 2:58.51 > java > > The jvm heap utilization is quite normal: > > #sudo jstat -gc -J"-Xmx128m" 7836 > S0C S1C S0U S1U *EC* *EU* *OC* * > OU* *PC PU* YGC YGCT FGC FGCT GCT > > 8512.0 8512.0 372.8 0.0 *68160.0* *5225.7* *963392.0 508200.7 > 30604.0 18373.4* 480 3.979 2 0.005 3.984 > > And then I try "pmap" to see the native memory mapping. *There is two > large anonymous mmap regions.* > > 00000000080dc000 1573568K rw--- [ anon ] > 00002b2afc900000 1079180K rw--- [ anon ] > > The second one should be JVM heap. What is the first one? Mmap of sstable > should never be anonymous mmap, but file based mmap. *Is it a native > memory leak? *Does cassandra allocate any DirectByteBuffer? > > best regards, > hanzhu >
