[jira] [Commented] (SPARK-19532) [Core]`DataStreamer for file` threads of DFSOutputStream leak if set `spark.speculation` to true
[
https://issues.apache.org/jira/browse/SPARK-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15999016#comment-15999016
]
Abhishek Madav commented on SPARK-19532:
I am running into this issue wherein codepath similar to hiveWriterContainer is
trying to the HDFS location. I tried setting spark.speculation to false but it
doesn't seem to be the issue. Is there any workaround? This wait-time leads to
make the job run real slow.
> [Core]`DataStreamer for file` threads of DFSOutputStream leak if set
> `spark.speculation` to true
>
>
> Key: SPARK-19532
> URL: https://issues.apache.org/jira/browse/SPARK-19532
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.1.0
>Reporter: StanZhai
>Priority: Critical
>
> When set `spark.speculation` to true, from thread dump page of Executor of
> WebUI, I found that there are about 1300 threads named "DataStreamer for
> file
> /test/data/test_temp/_temporary/0/_temporary/attempt_20170207172435_80750_m_69_1/part-00069-690407af-0900-46b1-9590-a6d6c696fe68.snappy.parquet"
> in TIMED_WAITING state.
> {code}
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:564)
> {code}
> The off-heap memory exceeds a lot until Executor exited with OOM exception.
> This problem occurs only when writing data to the Hadoop(tasks may be killed
> by Executor during writing).
> Could this be related to [https://issues.apache.org/jira/browse/HDFS-9812]?
> The version of Hadoop is 2.6.4.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19532) [Core]`DataStreamer for file` threads of DFSOutputStream leak if set `spark.speculation` to true
[
https://issues.apache.org/jira/browse/SPARK-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952730#comment-15952730
]
Hyukjin Kwon commented on SPARK-19532:
--
It sounds no one except for the reporter could reproduce this. If you are going
to fix it, that's great but if not, I think it might be better to resolve this
for now. Anyone can reopen this with more details and symptoms if anyone meets
this issue.
I guess this will probably be open without more actions.
> [Core]`DataStreamer for file` threads of DFSOutputStream leak if set
> `spark.speculation` to true
>
>
> Key: SPARK-19532
> URL: https://issues.apache.org/jira/browse/SPARK-19532
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.1.0
>Reporter: StanZhai
>Priority: Critical
>
> When set `spark.speculation` to true, from thread dump page of Executor of
> WebUI, I found that there are about 1300 threads named "DataStreamer for
> file
> /test/data/test_temp/_temporary/0/_temporary/attempt_20170207172435_80750_m_69_1/part-00069-690407af-0900-46b1-9590-a6d6c696fe68.snappy.parquet"
> in TIMED_WAITING state.
> {code}
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:564)
> {code}
> The off-heap memory exceeds a lot until Executor exited with OOM exception.
> This problem occurs only when writing data to the Hadoop(tasks may be killed
> by Executor during writing).
> Could this be related to [https://issues.apache.org/jira/browse/HDFS-9812]?
> The version of Hadoop is 2.6.4.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19532) [Core]`DataStreamer for file` threads of DFSOutputStream leak if set `spark.speculation` to true
[
https://issues.apache.org/jira/browse/SPARK-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864953#comment-15864953
]
StanZhai commented on SPARK-19532:
--
I can reproduce this by split our online data to the production test cluster
use our Spark application.
Our application is a web service, sql job requests are concurrently handled by
it（like hive-thriftserver）.
It's really a bit difficult to reproduce in the development environment.
> [Core]`DataStreamer for file` threads of DFSOutputStream leak if set
> `spark.speculation` to true
>
>
> Key: SPARK-19532
> URL: https://issues.apache.org/jira/browse/SPARK-19532
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.1.0
>Reporter: StanZhai
>Priority: Critical
>
> When set `spark.speculation` to true, from thread dump page of Executor of
> WebUI, I found that there are about 1300 threads named "DataStreamer for
> file
> /test/data/test_temp/_temporary/0/_temporary/attempt_20170207172435_80750_m_69_1/part-00069-690407af-0900-46b1-9590-a6d6c696fe68.snappy.parquet"
> in TIMED_WAITING state.
> {code}
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:564)
> {code}
> The off-heap memory exceeds a lot until Executor exited with OOM exception.
> This problem occurs only when writing data to the Hadoop(tasks may be killed
> by Executor during writing).
> Could this be related to [https://issues.apache.org/jira/browse/HDFS-9812]?
> The version of Hadoop is 2.6.4.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19532) [Core]`DataStreamer for file` threads of DFSOutputStream leak if set `spark.speculation` to true
[
https://issues.apache.org/jira/browse/SPARK-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863340#comment-15863340
]
Herman van Hovell commented on SPARK-19532:
---
I understand that this is a blocker for you; however this is not a release
blocker.
The problem with this ticket is that it is almost impossible to reproduce. Is
there any chance that you create a reproducible error?
> [Core]`DataStreamer for file` threads of DFSOutputStream leak if set
> `spark.speculation` to true
>
>
> Key: SPARK-19532
> URL: https://issues.apache.org/jira/browse/SPARK-19532
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.1.0
>Reporter: StanZhai
>Priority: Critical
>
> When set `spark.speculation` to true, from thread dump page of Executor of
> WebUI, I found that there are about 1300 threads named "DataStreamer for
> file
> /test/data/test_temp/_temporary/0/_temporary/attempt_20170207172435_80750_m_69_1/part-00069-690407af-0900-46b1-9590-a6d6c696fe68.snappy.parquet"
> in TIMED_WAITING state.
> {code}
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:564)
> {code}
> The off-heap memory exceeds a lot until Executor exited with OOM exception.
> This problem occurs only when writing data to the Hadoop(tasks may be killed
> by Executor during writing).
> Could this be related to [https://issues.apache.org/jira/browse/HDFS-9812]?
> The version of Hadoop is 2.6.4.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19532) [Core]`DataStreamer for file` threads of DFSOutputStream leak if set `spark.speculation` to true
[
https://issues.apache.org/jira/browse/SPARK-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862625#comment-15862625
]
StanZhai commented on SPARK-19532:
--
We have been trying to upgrade our Spark from the releasing of Spark 2.1.0.
This version is not available for us because of the memory problems.
> [Core]`DataStreamer for file` threads of DFSOutputStream leak if set
> `spark.speculation` to true
>
>
> Key: SPARK-19532
> URL: https://issues.apache.org/jira/browse/SPARK-19532
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.1.0
>Reporter: StanZhai
>Priority: Blocker
>
> When set `spark.speculation` to true, from thread dump page of Executor of
> WebUI, I found that there are about 1300 threads named "DataStreamer for
> file
> /test/data/test_temp/_temporary/0/_temporary/attempt_20170207172435_80750_m_69_1/part-00069-690407af-0900-46b1-9590-a6d6c696fe68.snappy.parquet"
> in TIMED_WAITING state.
> {code}
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:564)
> {code}
> The off-heap memory exceeds a lot until Executor exited with OOM exception.
> This problem occurs only when writing data to the Hadoop(tasks may be killed
> by Executor during writing).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19532) [Core]`DataStreamer for file` threads of DFSOutputStream leak if set `spark.speculation` to true
[
https://issues.apache.org/jira/browse/SPARK-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862443#comment-15862443
]
Hyukjin Kwon commented on SPARK-19532:
--
Is it a blocker BTW?
> [Core]`DataStreamer for file` threads of DFSOutputStream leak if set
> `spark.speculation` to true
>
>
> Key: SPARK-19532
> URL: https://issues.apache.org/jira/browse/SPARK-19532
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.1.0
>Reporter: StanZhai
>Priority: Blocker
>
> When set `spark.speculation` to true, from thread dump page of Executor of
> WebUI, I found that there are about 1300 threads named "DataStreamer for
> file
> /test/data/test_temp/_temporary/0/_temporary/attempt_20170207172435_80750_m_69_1/part-00069-690407af-0900-46b1-9590-a6d6c696fe68.snappy.parquet"
> in TIMED_WAITING state.
> {code}
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:564)
> {code}
> The off-heap memory exceeds a lot until Executor exited with OOM exception.
> This problem occurs only when writing data to the Hadoop(tasks may be killed
> by Executor during writing).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
