fyi after further troubleshooting logging this as
https://issues.apache.org/jira/browse/SPARK-12511
On Tuesday, 22 December 2015, 18:16, Antony Mayi
wrote:
I narrowed it down to problem described for example here:
I have streaming app (pyspark 1.5.2 on yarn) that's crashing due to driver (jvm
part, not python) OOM (no matter how big heap is assigned, eventually runs out).
When checking the heap it is all taken by "byte" items of
io.netty.buffer.PoolThreadCache. The number of
This might be related but the jmap output there looks different:
http://stackoverflow.com/questions/32537965/huge-number-of-io-netty-buffer-poolthreadcachememoryregioncacheentry-instances
On Tue, Dec 22, 2015 at 2:59 AM, Antony Mayi
wrote:
> I have streaming app
I narrowed it down to problem described for example here:
https://bugs.openjdk.java.net/browse/JDK-6293787
It is the mass finalization of zip Inflater/Deflater objects which can't keep
up with the rate of these instances being garbage collected. as the jdk bug
report (not being accepted as a
I searched code briefly.
The following uses ZipEntry, ZipOutputStream :
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
core/src/main/scala/org/apache/spark/deploy/RPackageUtils.scala
FYI
On Tue, Dec 22, 2015 at 9:16 AM, Antony Mayi