On Wed, Jan 12, 2011 at 5:01 PM, Tatsuya Kawano <[email protected]>wrote:
> > And > > in some circumstances (like all the rigged tests I've attempted to do) > these > > get cleaned up nicely by the JVM. It seems only in pretty large heaps in > > real workloads does the leak actually end up running away. > > This issue should be circumstance dependent as we don't have direct control > on deallocating those buffers. We need them GCed but they never occupy the > Java heap to encourage the GC to run. > Thanks to reflection and use of undocumented APIs, you can actually free() a direct buffer - check out the patch referenced earlier in this thread. Of course it probably doesn't work on other JVMs... oh well. -Todd > > > On Jan 13, 2011, at 8:30 AM, Todd Lipcon <[email protected]> wrote: > > > On Wed, Jan 12, 2011 at 3:25 PM, Tatsuya Kawano <[email protected] > >wrote: > > > >> Hi Friso and everyone, > >> > >> OK. We don't have to spend time to juggle hadoop-core jars anymore since > >> Todd is working hard on enhancing hadoop-lzo behavior. > >> > >> I think your assumption is correct, but what I was trying to say was > HBase > >> doesn't change the way to use Hadoop compressors since HBase 0.20 > release > >> while Hadoop added reinit() on 0.21. I verified that ASF Hadoop 0.21 and > >> CDH3b3 have reinit() and ASF Hadoop 0.20.2 (including its append branch) > and > >> CDH3b2 don't. I saw you had no problem running HBase 0.89 on CDH3b2, so > I > >> thought HBase 0.90 would work fine on ASF Hadoop 0.20.2. Because both of > >> them don't have reinit(). > >> > >> > > Yep - but that jar isn't wire-compatible with a CDH3b3 cluster. So if you > > have a CDH3b3 cluster for one of the other features included, you need to > > use a 3b3 client jar as well, which includes the reinit stuff. > > > > > >> HBase tries to create an output compression stream on each compression > >> block, and one HFile flush will contain roughly 1000 compression blocks. > I > >> think reinit() could get called 1000 times on one flush, and if > hadoop-lzo > >> allocates 64MB block on reinit() (HBase's compression blocks is about > 64KB > >> though), it will become pretty much something you're observing now. > >> > >> > > Yep - though I think it's only leaking a 64K buffer for each in 0.4.8. > And > > in some circumstances (like all the rigged tests I've attempted to do) > these > > get cleaned up nicely by the JVM. It seems only in pretty large heaps in > > real workloads does the leak actually end up running away. > > > > -Todd > > > >> > >> On Jan 13, 2011, at 7:50 AM, Todd Lipcon <[email protected]> wrote: > >> > >>> Can someone who is having this issue try checking out the following git > >>> branch and rebuilding LZO? > >>> > >>> https://github.com/toddlipcon/hadoop-lzo/tree/realloc > >>> > >>> This definitely stems one leak of a 64KB directbuffer on every reinit. > >>> > >>> -Todd > >>> > >>> On Wed, Jan 12, 2011 at 2:12 PM, Todd Lipcon <[email protected]> > wrote: > >>> > >>>> Yea, you're definitely on the right track. Have you considered systems > >>>> programming, Friso? :) > >>>> > >>>> Hopefully should have a candidate patch to LZO later today. > >>>> > >>>> -Todd > >>>> > >>>> On Wed, Jan 12, 2011 at 1:20 PM, Friso van Vollenhoven < > >>>> [email protected]> wrote: > >>>> > >>>>> Hi, > >>>>> My guess is indeed that it has to do with using the reinit() method > on > >>>>> compressors and making them long lived instead of throwaway together > >> with > >>>>> the LZO implementation of reinit(), which magically causes NIO buffer > >>>>> objects not to be finalized and as a result not release their native > >>>>> allocations. It's just theory and I haven't had the time to properly > >> verify > >>>>> this (unfortunately, I spend most of my time writing application > code), > >> but > >>>>> Todd said he will be looking into it further. I browsed the LZO code > to > >> see > >>>>> what was going on there, but with my limited knowledge of the HBase > >> code it > >>>>> would be bald to say that this is for sure the case. It would be my > >> first > >>>>> direction of investigation. I would add some logging to the LZO code > >> where > >>>>> new direct byte buffers are created to log how often that happens and > >> what > >>>>> size they are and then redo the workload that shows the leak. > Together > >> with > >>>>> some profiling you should be able to see how long it takes for these > >> get > >>>>> finalized. > >>>>> > >>>>> Cheers, > >>>>> Friso > >>>>> > >>>>> > >>>>> > >>>>> On 12 jan 2011, at 20:08, Stack wrote: > >>>>> > >>>>>> 2011/1/12 Friso van Vollenhoven <[email protected]>: > >>>>>>> No, I haven't. But the Hadoop (mapreduce) LZO compression is not > the > >>>>> problem. Compressing the map output using LZO works just fine. The > >> problem > >>>>> is HBase LZO compression. The region server process is the one with > the > >>>>> memory leak... > >>>>>>> > >>>>>> > >>>>>> (Sorry for dumb question Friso) But HBase is leaking because we make > >>>>>> use of the Compression API in a manner that produces leaks? > >>>>>> Thanks, > >>>>>> St.Ack > >>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> Todd Lipcon > >>>> Software Engineer, Cloudera > >>>> > >>> > >>> > >>> > >>> -- > >>> Todd Lipcon > >>> Software Engineer, Cloudera > >> > > > > > > > > -- > > Todd Lipcon > > Software Engineer, Cloudera > > -- Todd Lipcon Software Engineer, Cloudera
