>From thread dump page of Executor of WebUI, I found that there are about 1300 threads named "DataStreamer for file /test/data/test_temp/_temporary/0/_temporary/attempt_20170207172435_80750_m_000069_1/part-00069-690407af-0900-46b1-9590-a6d6c696fe68.snappy.parquet" in TIMED_WAITING state like this: <http://apache-spark-developers-list.1001551.n3.nabble.com/file/n20881/QQ20170207-212340.png>
The exceed off-heap memory may be caused by these abnormal threads. This problem occurs only when writing data to the Hadoop(tasks may be killed by Executor during writing). Could this be related to https://issues.apache.org/jira/browse/HDFS-9812 <https://issues.apache.org/jira/browse/HDFS-9812> ? It's may be a bug of Spark when killing tasks during writing data. What's the difference between Spark 1.6.x and 2.1.0 in killing tasks? This is a critical issue, I've worked on this for days. Any help? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20881.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
