Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

*From:* Kostas Kloudas >> *Sent:* 03 February 2020 15:39 >> *To:* Mark Harris >> *Cc:* Piotr Nowojski ; Cliff Resnick < >> cre...@gmail.com>; David Magalhães ; Till >> Rohrmann ; flink-u...@apache.org < >> flink-u...@apache.org> >> *Subje

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

regards, Mark From: Kostas Kloudas Sent: 03 February 2020 15:39 To: Mark Harris Cc: Piotr Nowojski ; Cliff Resnick ; David Magalhães ; Till Rohrmann ; flink-u...@apache.org Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

rk Harris > *Cc:* Piotr Nowojski ; Cliff Resnick < > cre...@gmail.com>; David Magalhães ; Till Rohrmann > ; flink-u...@apache.org > *Subject:* Re: GC overhead limit exceeded, memory full of DeleteOnExit > hooks for S3a files > > Hi Mark, > > Have you

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files Hi Mark, Have you tried to set your rolling policy to close inactive part files after some time [1]? If the part files in the buckets are inactive and there are no new part files, then the state handle

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

* Piotr Nowojski > *Cc:* Cliff Resnick ; David Magalhães < > speeddra...@gmail.com>; Till Rohrmann ; > flink-u...@apache.org ; kkloudas < > kklou...@apache.org> > *Subject:* Re: GC overhead limit exceeded, memory full of DeleteOnExit > hooks for S3a files > > Hi, &g

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

id Magalhães > ; Till Rohrmann ; > flink-u...@apache.org ; kkloudas > Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks > for S3a files > > Hi, > > Thanks for your help with this.  > > The EMR cluster has 3 15GB VMs, and the flink cluster i

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

: 30 January 2020 14:36 To: Piotr Nowojski Cc: Cliff Resnick ; David Magalhães ; Till Rohrmann ; flink-u...@apache.org ; kkloudas Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files Hi, Thanks for your help with this.  The EMR cluster has 3 15GB VMs

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

s Cc: Cliff Resnick ; David Magalhães ; Till Rohrmann ; flink-u...@apache.org ; kkloudas Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files Hi, What is your job setup? Size of the nodes, memory settings of the Flink/JVM? 9 041 060 strings is awfully s

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

uld this be a factor? > > Best regards, > > Mark > From: Piotr Nowojski > Sent: 27 January 2020 16:16 > To: Cliff Resnick > Cc: David Magalhães ; Mark Harris > ; Till Rohrmann ; > flink-u...@apache.org ; kkloudas > Subject: Re: GC overhead limit exceeded, memory ful

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

gt;; flink-u...@apache.org<mailto:flink-u...@apache.org> mailto:flink-u...@apache.org>>; kkloudas mailto:kklou...@apache.org>> Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files Hi, This is probably a known issue of Hadoop [1]. Unfortunately it was o

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

gt;>> Mark >>> ------ >>> *From:* Piotr Nowojski on behalf of Piotr >>> Nowojski >>> *Sent:* 22 January 2020 13:29 >>> *To:* Till Rohrmann >>> *Cc:* Mark Harris ; flink-u...@apache.org < >>> flink-u...@

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

.com>> > Sent: 22 January 2020 13:29 > To: Till Rohrmann mailto:trohrm...@apache.org>> > Cc: Mark Harris mailto:mark.har...@hivehome.com>>; > flink-u...@apache.org <mailto:flink-u...@apache.org> <mailto:flink-u...@apache.org>>; kkloudas <mailto:kklou.

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

he taskmanager breaks >> with the same problem. >> >> Best regards, >> >> Mark >> -- >> *From:* Piotr Nowojski on behalf of Piotr >> Nowojski >> *Sent:* 22 January 2020 13:29 >> *To:* Till Rohrmann >> *Cc:* Mark Harris ; flink-u...@apache.org

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

half of Piotr > Nowojski > *Sent:* 22 January 2020 13:29 > *To:* Till Rohrmann > *Cc:* Mark Harris ; flink-u...@apache.org < > flink-u...@apache.org>; kkloudas > *Subject:* Re: GC overhead limit exceeded, memory full of DeleteOnExit > hooks for S3a files > > Hi,

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

From: Piotr Nowojski on behalf of Piotr Nowojski Sent: 22 January 2020 13:29 To: Till Rohrmann Cc: Mark Harris ; flink-u...@apache.org ; kkloudas Subject: Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files Hi, This is probably a known issue

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

Hi, This is probably a known issue of Hadoop [1]. Unfortunately it was only fixed in 3.3.0. Piotrek [1] https://issues.apache.org/jira/browse/HADOOP-15658 > On 22 Jan 2020, at 13:56, Till Rohrmann wrote: > > Thanks for reporting this

Re: GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

Thanks for reporting this issue Mark. I'm pulling Klou into this conversation who knows more about the StreamingFileSink. @Klou does the StreamingFileSink relies on DeleteOnExitHooks to clean up files? Cheers, Till On Tue, Jan 21, 2020 at 3:38 PM Mark Harris wrote: > Hi, > > We're using flink

GC overhead limit exceeded, memory full of DeleteOnExit hooks for S3a files

Hi, We're using flink 1.7.2 on an EMR cluster v emr-5.22.0, which runs hadoop v "Amazon 2.8.5". We've recently noticed that some TaskManagers fail (causing all the jobs running on them to fail) with an "java.lang.OutOfMemoryError: GC overhead limit exceeded”. The taskmanager (and jobs that