Hi, I think reducing the frequency of the checkpoints and decreasing parallelism of the things using the S3AOutputStream class, would help to mitigate the issue.
I don’t know about other solutions. I would suggest to ask this question directly to Steve L. in the bug ticket [1], as he is the one that fixed the issue. If there is no workaround, maybe it would be possible to put a pressure on the Hadoop guys to back port the fix to older versions? Piotrek [1] https://issues.apache.org/jira/browse/HADOOP-15658 <https://issues.apache.org/jira/browse/HADOOP-15658> > On 27 Jan 2020, at 15:41, Cliff Resnick <cre...@gmail.com> wrote: > > I know from experience that Flink's shaded S3A FileSystem does not reference > core-site.xml, though I don't remember offhand what file (s) it does > reference. However since it's shaded, maybe this could be fixed by building a > Flink FS referencing 3.3.0? Last I checked I think it referenced 3.1.0. > > On Mon, Jan 27, 2020, 8:48 AM David Magalhães <speeddra...@gmail.com > <mailto:speeddra...@gmail.com>> wrote: > Does StreamingFileSink use core-site.xml ? When I was using it, it didn't > load any configurations from core-site.xml. > > On Mon, Jan 27, 2020 at 12:08 PM Mark Harris <mark.har...@hivehome.com > <mailto:mark.har...@hivehome.com>> wrote: > Hi Piotr, > > Thanks for the link to the issue. > > Do you know if there's a workaround? I've tried setting the following in my > core-site.xml: > > fs.s3a.fast.upload.buffer=true > > To try and avoid writing the buffer files, but the taskmanager breaks with > the same problem. > > Best regards, > > Mark > From: Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> on behalf of Piotr Nowojski > <pi...@ververica.com <mailto:pi...@ververica.com>> > Sent: 22 January 2020 13:29 > To: Till Rohrmann <trohrm...@apache.org <mailto:trohrm...@apache.org>> > Cc: Mark Harris <mark.har...@hivehome.com <mailto:mark.har...@hivehome.com>>; > flink-u...@apache.org <mailto:flink-u...@apache.org> <flink-u...@apache.org > <mailto:flink-u...@apache.org>>; kkloudas <kklou...@apache.org > <mailto:kklou...@apache.org>> > Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks > for S3a files > > Hi, > > This is probably a known issue of Hadoop [1]. Unfortunately it was only fixed > in 3.3.0. > > Piotrek > > [1] https://issues.apache.org/jira/browse/HADOOP-15658 > <https://issues.apache.org/jira/browse/HADOOP-15658> > >> On 22 Jan 2020, at 13:56, Till Rohrmann <trohrm...@apache.org >> <mailto:trohrm...@apache.org>> wrote: >> >> Thanks for reporting this issue Mark. I'm pulling Klou into this >> conversation who knows more about the StreamingFileSink. @Klou does the >> StreamingFileSink relies on DeleteOnExitHooks to clean up files? >> >> Cheers, >> Till >> >> On Tue, Jan 21, 2020 at 3:38 PM Mark Harris <mark.har...@hivehome.com >> <mailto:mark.har...@hivehome.com>> wrote: >> Hi, >> >> We're using flink 1.7.2 on an EMR cluster v emr-5.22.0, which runs hadoop v >> "Amazon 2.8.5". We've recently noticed that some TaskManagers fail (causing >> all the jobs running on them to fail) with an "java.lang.OutOfMemoryError: >> GC overhead limit exceeded”. The taskmanager (and jobs that should be >> running on it) remain down until manually restarted. >> >> I managed to take and analyze a memory dump from one of the afflicted >> taskmanagers. >> >> It showed that 85% of the heap was made up of the >> java.io.DeleteOnExitHook.files hashset. The majority of the strings in that >> hashset (9041060 out of ~9041100) pointed to files that began >> /tmp/hadoop-yarn/s3a/s3ablock >> >> The problem seems to affect jobs that make use of the StreamingFileSink - >> all of the taskmanager crashes have been on the taskmaster running at least >> one job using this sink, and a cluster running only a single taskmanager / >> job that uses the StreamingFileSink crashed with the GC overhead limit >> exceeded error. >> >> I've had a look for advice on handling this error more broadly without luck. >> >> Any suggestions or advice gratefully received. >> >> Best regards, >> >> Mark Harris >> >> >> >> The information contained in or attached to this email is intended only for >> the use of the individual or entity to which it is addressed. If you are not >> the intended recipient, or a person responsible for delivering it to the >> intended recipient, you are not authorised to and must not disclose, copy, >> distribute, or retain this message or any part of it. It may contain >> information which is confidential and/or covered by legal professional or >> other privilege under applicable law. >> >> The views expressed in this email are not necessarily the views of Centrica >> plc or its subsidiaries, and the company, its directors, officers or >> employees make no representation or accept any liability for its accuracy or >> completeness unless expressly stated to the contrary. >> >> Additional regulatory disclosures may be found here: >> https://www.centrica.com/privacy-cookies-and-legal-disclaimer#email >> <https://www.centrica.com/privacy-cookies-and-legal-disclaimer#email> >> >> PH Jones is a trading name of British Gas Social Housing Limited. British >> Gas Social Housing Limited (company no: 01026007), British Gas Trading >> Limited (company no: 03078711), British Gas Services Limited (company no: >> 3141243), British Gas Insurance Limited (company no: 06608316), British Gas >> New Heating Limited (company no: 06723244), British Gas Services >> (Commercial) Limited (company no: 07385984) and Centrica Energy (Trading) >> Limited (company no: 02877397) are all wholly owned subsidiaries of Centrica >> plc (company no: 3033654). Each company is registered in England and Wales >> with a registered office at Millstream, Maidenhead Road, Windsor, Berkshire >> SL4 5GD. >> >> British Gas Insurance Limited is authorised by the Prudential Regulation >> Authority and regulated by the Financial Conduct Authority and the >> Prudential Regulation Authority. British Gas Services Limited and Centrica >> Energy (Trading) Limited are authorised and regulated by the Financial >> Conduct Authority. British Gas Trading Limited is an appointed >> representative of British Gas Services Limited which is authorised and >> regulated by the Financial Conduct Authority. > > > > The information contained in or attached to this email is intended only for > the use of the individual or entity to which it is addressed. If you are not > the intended recipient, or a person responsible for delivering it to the > intended recipient, you are not authorised to and must not disclose, copy, > distribute, or retain this message or any part of it. It may contain > information which is confidential and/or covered by legal professional or > other privilege under applicable law. > > The views expressed in this email are not necessarily the views of Centrica > plc or its subsidiaries, and the company, its directors, officers or > employees make no representation or accept any liability for its accuracy or > completeness unless expressly stated to the contrary. > > Additional regulatory disclosures may be found here: > https://www.centrica.com/privacy-cookies-and-legal-disclaimer#email > <https://www.centrica.com/privacy-cookies-and-legal-disclaimer#email> > > PH Jones is a trading name of British Gas Social Housing Limited. British Gas > Social Housing Limited (company no: 01026007), British Gas Trading Limited > (company no: 03078711), British Gas Services Limited (company no: 3141243), > British Gas Insurance Limited (company no: 06608316), British Gas New Heating > Limited (company no: 06723244), British Gas Services (Commercial) Limited > (company no: 07385984) and Centrica Energy (Trading) Limited (company no: > 02877397) are all wholly owned subsidiaries of Centrica plc (company no: > 3033654). Each company is registered in England and Wales with a registered > office at Millstream, Maidenhead Road, Windsor, Berkshire SL4 5GD. > > British Gas Insurance Limited is authorised by the Prudential Regulation > Authority and regulated by the Financial Conduct Authority and the Prudential > Regulation Authority. British Gas Services Limited and Centrica Energy > (Trading) Limited are authorised and regulated by the Financial Conduct > Authority. British Gas Trading Limited is an appointed representative of > British Gas Services Limited which is authorised and regulated by the > Financial Conduct Authority.
