Re: "java.io.IOException: Filesystem closed" on executors

Thank you, Akhil. Actually the problem was solved last week and I did not have time to report back. The error was caused by YARN killing the container because executors use more off-heap memory that they were assigned. There was nothing in the exectuor log, but the AM log clearly states this is

Re: "java.io.IOException: Filesystem closed" on executors

Can you look a bit deeper in the executor logs? It could be filling up the memory and getting killed. Thanks Best Regards On Mon, Oct 5, 2015 at 8:55 PM, Lan Jiang wrote: > I am still facing this issue. Executor dies due to > > org.apache.avro.AvroRuntimeException:

Re: "java.io.IOException: Filesystem closed" on executors

I am still facing this issue. Executor dies due to org.apache.avro.AvroRuntimeException: java.io.IOException: Filesystem closed at org.apache.avro.file.DataFileStream.hasNextBlock(DataFileStream.java:278) at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:197) at

"java.io.IOException: Filesystem closed" on executors

Hi, there Here is the problem I ran into when executing a Spark Job (Spark 1.3). The spark job is loading a bunch of avro files using Spark SQL spark-avro 1.0.0 library. Then it does some filter/map transformation, repartition to 1 partition and then write to HDFS. It creates 2 stages. The total