[
https://issues.apache.org/jira/browse/SPARK-11101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-11101.
-------------------------------
Resolution: Invalid
If it's a question, you should ask as u...@spark.apache.org, not make a JIRA.
It may have nothing to do with your process, though you do need to verify how
much it uses. There is little margin in the YARN allocation for off-heap
memory, so you probably have to increase this value, yes.
> pipe() operation OOM
> --------------------
>
> Key: SPARK-11101
> URL: https://issues.apache.org/jira/browse/SPARK-11101
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.4.1
> Environment: spark on yarn
> Reporter: hotdog
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> when using pipe() operation with large data(10TB), the pipe() operation
> always OOM.
> I use pipe() to calling a external c++ process. I'm sure the c++ program only
> use little memory(about 1MB).
> my parameters:
> executor-memory 16g
> executor-cores 4
> num-executors 400
> "spark.yarn.executor.memoryOverhead", "8192"
> partition number: 60000
> does pipe() operation use many off-heap memory?
> the log is :
> killed by YARN for exceeding memory limits. 24.4 GB of 24 GB physical memory
> used. Consider boosting spark.yarn.executor.memoryOverhead.
> should I continue boosting spark.yarn.executor.memoryOverhead? Or there are
> some bugs in the pipe() operation?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
