Hi Johannes, There were frequent Oozie OOM's with hadoop 0.23 seen at Yahoo due to https://issues.apache.org/jira/browse/HDFS-3646. You can analyze your heap dump to check whether the OOM's are caused due to the same issue as seen in HDFS-3646. If it is, you can use 0.23.3 which was recently released and has the fix. (http://hadoop.apache.org/releases.html)
Thanks, Virag On 9/19/12 4:43 AM, "Johannes Schwenk" <[email protected]> wrote: >Hello, > >does someone know, why we have a huge memory consumption with Oozie >3.1.3 (cdh4) and YARN ? Oozie needs around 4GB Java heap for just some >very short pig workflows that run under a minute and never more than 5 >simultaneously. It looks almost like a memory leak, since the memory >used slowly rises over around 1 hour to be kept constantly at around >3.8GB... This might be due to (higher) GC activity, but we didn't check >explicitly. > >After we switched back to MRv1 the memory consumption is now below 300MB >however. Plus the same jobs now complete in roughly half the time to a >tenth of the time. > >I would be curious to know if this is a known issue and maybe get some >insight on how and when we could switch to YARN for production?! > >Greetings, >Johannes Schwenk > >-- >Softwareentwickler (Reporting) >________________________________________________________ > >ADITION technologies AG >Schwarzwaldstraße 78b >79117 Freiburg > >http://www.adition.com > >T +49 / (0)761 / 88147 - 30 >F +49 / (0)761 / 88147 - 77 >SUPPORT +49 / (0)1805 - ADITION > >(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min) > >Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076 >Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus >Schlüter >Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer >UStIDNr.: DE 218 858 434 >
