No, I use only malloc env var, and I set it (as suggested before) into hbase-env.sh, and it looks like it eats more less memory (in my case 4.7G vs 3.3G with 2Gheap)
2011/1/12 Friso van Vollenhoven <[email protected]> > Thanks. > > I went back to hbase 0.89 with 0.1 LZO, which works fine and does not show > this issue. > > I tried with a newer Hbase and LZO version, also with the MALLOC... setting > but without max direct memory set, so I was wondering whether you need a > combination of the two to fix things (apparently not). > > Now i am wondering whether I did something wrong setting the env var. It > should just be picked up when it's in hbase-env.sh, right? > > > Friso > > > > On 12 jan 2011, at 10:59, Andrey Stepachev wrote: > > > with MALLOC_ARENA_MAX=2 > > > > I check -XX:MaxDirectMemorySize=256m, before, but it doesn't affect > anything > > (even no OOM > > exceptions or so on). > > > > But it looks like i have exactly the same issue (it looks like). I have > many > > 64Mb anon memory blocks. > > (sometimes they 132MB). And on heavy load i have rapidly growing rss size > of > > jvm process. > > > > 2011/1/12 Friso van Vollenhoven <[email protected]> > > > >> Just to clarify: you fixed it by setting the MALLOC_MAX_ARENA=? in > >> hbase-env.sh? > >> > >> Did you also use the -XX:MaxDirectMemorySize=256m ? > >> > >> It would be nice to check that this is a different than the leakage with > >> LZO... > >> > >> > >> Thanks, > >> Friso > >> > >> > >> On 12 jan 2011, at 07:46, Andrey Stepachev wrote: > >> > >>> My bad. All things work. Thanks for Todd Lipcon :) > >>> > >>> 2011/1/11 Andrey Stepachev <[email protected]> > >>> > >>>> I tried to set MALLOC_ARENA_MAX=2. But still the same issue like in > LZO > >>>> problem thread. All those 65M blocks here. And JVM continues to eat > >> memory > >>>> on heavy write load. And yes, I use "improved" kernel > >>>> Linux 2.6.34.7-0.5. > >>>> > >>>> 2011/1/11 Xavier Stevens <[email protected]> > >>>> > >>>> Are you using a newer linux kernel with the new and "improved" memory > >>>>> allocator? > >>>>> > >>>>> If so try setting this in hadoop-env.sh: > >>>>> > >>>>> export MALLOC_ARENA_MAX=<number of cores you want to use> > >>>>> > >>>>> Maybe start by setting it to 4. You can thank Todd Lipcon if this > >> works > >>>>> for you. > >>>>> > >>>>> Cheers, > >>>>> > >>>>> > >>>>> -Xavier > >>>>> > >>>>> On 1/11/11 7:24 AM, Andrey Stepachev wrote: > >>>>>> No. I don't use LZO. I tried even remove any native support (i.e. > all > >>>>> .so > >>>>>> from class path) > >>>>>> and use java gzip. But nothing. > >>>>>> > >>>>>> > >>>>>> 2011/1/11 Friso van Vollenhoven <[email protected]> > >>>>>> > >>>>>>> Are you using LZO by any chance? If so, which version? > >>>>>>> > >>>>>>> Friso > >>>>>>> > >>>>>>> > >>>>>>> On 11 jan 2011, at 15:57, Andrey Stepachev wrote: > >>>>>>> > >>>>>>>> After starting the hbase in jroсkit found the same memory leakage. > >>>>>>>> > >>>>>>>> After the launch > >>>>>>>> > >>>>>>>> Every 2,0 s: date & & ps - sort =- rss-eopid, rss, vsz, pcpu | > head > >>>>>>>> Tue Jan 11 16:49:31 2011 > >>>>>>>> > >>>>>>>> 11 16:49:31 MSK 2011 > >>>>>>>> PID RSS VSZ% CPU > >>>>>>>> 7863 2547760 5576744 78.7 > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> JR dumps: > >>>>>>>> > >>>>>>>> Total mapped 5576740KB (reserved = 2676404KB) - Java heap > 2048000KB > >>>>>>>> (reserved = 1472176KB) - GC tables 68512KB - Thread stacks 37236KB > >> (# > >>>>>>>> threads = 111) - Compiled code 1048576KB (used = 2599KB) - > Internal > >>>>>>>> 1224KB - OS 549688KB - Other 1800976KB - Classblocks 1280KB > >> (malloced > >>>>>>>> = 1110KB # 3285) - Java class data 20224KB (malloced = 20002KB # > >> 15134 > >>>>>>>> in 3285 classes) - Native memory tracking 1024KB (malloced = 325KB > >> +10 > >>>>>>>> KB # 20) > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> After running the mr which make high write load (~1hour) > >>>>>>>> > >>>>>>>> Every 2,0 s: date & & ps - sort =- rss-eopid, rss, vsz, pcpu | > head > >>>>>>>> Tue Jan 11 17:08:56 2011 > >>>>>>>> > >>>>>>>> 11 17:08:56 MSK 2011 > >>>>>>>> PID RSS VSZ% CPU > >>>>>>>> 7863 4072396 5459572 100 > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> JR said not important below specify why) > >>>>>>>> > >>>>>>>> http://paste.ubuntu.com/552820/ > >>>>>>>> <http://paste.ubuntu.com/552820/> > >>>>>>>> > >>>>>>>> > >>>>>>>> 7863: > >>>>>>>> Total mapped 5742628KB +165888KB > >> (reserved=1144000KB > >>>>>>>> -1532404KB) > >>>>>>>> - Java heap 2048000KB (reserved=0KB > >>>>>>> -1472176KB) > >>>>>>>> - GC tables 68512KB > >>>>>>>> - Thread stacks 38028KB +792KB (#threads=114 > +3) > >>>>>>>> - Compiled code 1048576KB (used=3376KB > >> +776KB) > >>>>>>>> - Internal 1480KB +256KB > >>>>>>>> - OS 517944KB -31744KB > >>>>>>>> - Other 1996792KB +195816KB > >>>>>>>> - Classblocks 1280KB (malloced=1156KB > >>>>>>>> +45KB #3421 +136) > >>>>>>>> - Java class data 20992KB +768KB > (malloced=20843KB > >>>>>>>> +840KB #15774 +640 in 3421 classes) > >>>>>>>> - Native memory tracking 1024KB (malloced=325KB > >>>>> +10KB > >>>>>>> #20) > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >>>>>>>> OS *java r x 0x0000000000400000.( > >>>>>>> 76KB) > >>>>>>>> OS *java rw 0x0000000000612000 ( > >>>>>>> 4KB) > >>>>>>>> OS *[heap] rw 0x0000000000613000.( > >>>>>>> 478712KB) > >>>>>>>> INT Poll r 0x000000007fffe000 ( > >>>>>>> 4KB) > >>>>>>>> INT Membar rw 0x000000007ffff000.( > >>>>>>> 4KB) > >>>>>>>> MSP Classblocks (1/2) rw 0x0000000082ec0000 ( > >>>>>>> 768KB) > >>>>>>>> MSP Classblocks (2/2) rw 0x0000000082f80000 ( > >>>>>>> 512KB) > >>>>>>>> HEAP Java heap rw > >>>>>>> 0x0000000083000000.(2048000KB) > >>>>>>>> rw 0x00007f2574000000 ( > >>>>>>> 65500KB) > >>>>>>>> 0x00007f2577ff7000.( > >>>>>>> 36KB) > >>>>>>>> rw 0x00007f2584000000 ( > >>>>>>> 65492KB) > >>>>>>>> 0x00007f2587ff5000.( > >>>>>>> 44KB) > >>>>>>>> rw 0x00007f258c000000 ( > >>>>>>> 65500KB) > >>>>>>>> 0x00007f258fff7000 ( > >>>>>>> 36KB) > >>>>>>>> rw 0x00007f2590000000 ( > >>>>>>> 65500KB) > >>>>>>>> 0x00007f2593ff7000 ( > >>>>>>> 36KB) > >>>>>>>> rw 0x00007f2594000000 ( > >>>>>>> 65500KB) > >>>>>>>> 0x00007f2597ff7000 ( > >>>>>>> 36KB) > >>>>>>>> rw 0x00007f2598000000 ( > >>>>>>> 131036KB) > >>>>>>>> 0x00007f259fff7000 ( > >>>>>>> 36KB) > >>>>>>>> rw 0x00007f25a0000000 ( > >>>>>>> 65528KB) > >>>>>>>> 0x00007f25a3ffe000 ( > >>>>>>> 8KB) > >>>>>>>> rw 0x00007f25a4000000 ( > >>>>>>> 65496KB) > >>>>>>>> 0x00007f25a7ff6000 ( > >>>>>>> 40KB) > >>>>>>>> rw 0x00007f25a8000000 ( > >>>>>>> 65496KB) > >>>>>>>> 0x00007f25abff6000 ( > >>>>>>> 40KB) > >>>>>>>> rw 0x00007f25ac000000 ( > >>>>>>> 65504KB) > >>>>>>>> > >>>>>>>> > >>>>>>>> So, the difference was in the pieces of memory like this: > >>>>>>>> > >>>>>>>> rw 0x00007f2590000000 (65500KB) > >>>>>>>> 0x00007f2593ff7000 (36KB) > >>>>>>>> > >>>>>>>> > >>>>>>>> Looks like HLog allocates memory (looks like HLog, becase it is > very > >>>>>>> similar > >>>>>>>> size) > >>>>>>>> > >>>>>>>> If we count this blocks we get amount of lost memory: > >>>>>>>> > >>>>>>>> 65M * 32 + 132M = 2212M > >>>>>>>> > >>>>>>>> So, it looks like HLog allcates to many memory, and question is: > how > >>>>> to > >>>>>>>> restrict it? > >>>>>>>> > >>>>>>>> 2010/12/30 Andrey Stepachev <[email protected]> > >>>>>>>> > >>>>>>>>> Hi All. > >>>>>>>>> > >>>>>>>>> After heavy load into hbase (single node, nondistributed test > >> system) > >>>>> I > >>>>>>> got > >>>>>>>>> 4Gb process size of my HBase java process. > >>>>>>>>> On 6GB machine there was no room for anything else (disk cache > and > >> so > >>>>>>> on). > >>>>>>>>> Does anybody knows, what is going on, and how you solve this. > What > >>>>> heap > >>>>>>>>> memory is set on you hosts > >>>>>>>>> and how much of RSS hbase process actually use. > >>>>>>>>> > >>>>>>>>> I don't see such things before, all tomcat and other java apps > >> don't > >>>>>>> eats > >>>>>>>>> significally more memory then -Xmx. > >>>>>>>>> > >>>>>>>>> Connection name: pid: 23476 > >> org.apache.hadoop.hbase.master.HMaster > >>>>>>>>> start Virtual Machine: Java HotSpot(TM) 64-Bit Server VM > >> version > >>>>>>>>> 17.1-b03 Vendor: Sun Microsystems Inc. Name: 23...@mars > >>>>>>> Uptime: 12 > >>>>>>>>> hours 4 minutes Process CPU time: 5 hours 45 minutes JIT > >>>>> compiler: > >>>>>>> HotSpot > >>>>>>>>> 64-Bit Server Compiler Total compile time: 19,223 seconds > >>>>>>>>> ------------------------------ > >>>>>>>>> Current heap size: 703 903 kbytes Maximum heap size: 2 > >> 030 > >>>>>>> 976kbytes Committed memory: > >>>>>>>>> 2 030 976 kbytes Pending finalization: 0 objects Garbage > >>>>>>>>> collector: Name = 'ParNew', Collections = 9 990, Total time > spent > >> = > >>>>> 5 > >>>>>>>>> minutes Garbage collector: Name = 'ConcurrentMarkSweep', > >>>>> Collections > >>>>>>> = > >>>>>>>>> 20, Total time spent = 35,754 seconds > >>>>>>>>> ------------------------------ > >>>>>>>>> Operating System: Linux 2.6.34.7-0.5-xen Architecture: > >> amd64 > >>>>>>> Number of processors: > >>>>>>>>> 8 Committed virtual memory: 4 403 512 kbytes Total > physical > >>>>>>>>> memory: 6 815 744 kbytes Free physical memory: 82 720 > >> kbytes > >>>>>>> Total swap space: > >>>>>>>>> 8 393 924 kbytes Free swap space: 8 050 880 kbytes > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>> > >>>> > >>>> > >> > >> > >
