Hi Virag, thanks for the link. This is exactly what is happening in our case! We will go back to MRv1 for the moment.
Thanks, Johannes Am 19.09.2012 19:19, schrieb Virag Kothari: > Hi Johannes, > > There were frequent Oozie OOM's with hadoop 0.23 seen at Yahoo due to > https://issues.apache.org/jira/browse/HDFS-3646. You can analyze your heap > dump to check whether the OOM's are caused due to the same issue as seen > in HDFS-3646. If it is, you can use 0.23.3 which was recently released and > has the fix. (http://hadoop.apache.org/releases.html) > > Thanks, > Virag > > On 9/19/12 4:43 AM, "Johannes Schwenk" <[email protected]> > wrote: > >> Hello, >> >> does someone know, why we have a huge memory consumption with Oozie >> 3.1.3 (cdh4) and YARN ? Oozie needs around 4GB Java heap for just some >> very short pig workflows that run under a minute and never more than 5 >> simultaneously. It looks almost like a memory leak, since the memory >> used slowly rises over around 1 hour to be kept constantly at around >> 3.8GB... This might be due to (higher) GC activity, but we didn't check >> explicitly. >> >> After we switched back to MRv1 the memory consumption is now below 300MB >> however. Plus the same jobs now complete in roughly half the time to a >> tenth of the time. >> >> I would be curious to know if this is a known issue and maybe get some >> insight on how and when we could switch to YARN for production?! >> >> Greetings, >> Johannes Schwenk >> >> -- >> Softwareentwickler (Reporting) >> ________________________________________________________ >> >> ADITION technologies AG >> Schwarzwaldstraße 78b >> 79117 Freiburg >> >> http://www.adition.com >> >> T +49 / (0)761 / 88147 - 30 >> F +49 / (0)761 / 88147 - 77 >> SUPPORT +49 / (0)1805 - ADITION >> >> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min) >> >> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076 >> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus >> Schlüter >> Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer >> UStIDNr.: DE 218 858 434 >> > Johannes Schwenk -- Softwareentwickler (Reporting) ________________________________________________________ ADITION technologies AG Schwarzwaldstraße 78b 79117 Freiburg http://www.adition.com T +49 / (0)761 / 88147 - 30 F +49 / (0)761 / 88147 - 77 SUPPORT +49 / (0)1805 - ADITION (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min) Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076 Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer UStIDNr.: DE 218 858 434
signature.asc
Description: OpenPGP digital signature
