[
https://issues.apache.org/jira/browse/SPARK-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-20975.
-------------------------------
Resolution: Invalid
There's no useful detail here. It's normal for executors to not be released in
some cases, like if the min number are running, or if there is cached data.
Questions should to to the mailing list.
> Excutors are no released if speculation + dynamic allocation enabled
> ---------------------------------------------------------------------
>
> Key: SPARK-20975
> URL: https://issues.apache.org/jira/browse/SPARK-20975
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.6.2
> Reporter: harish chandra
>
> Whenever user enable spark speculation + dynamic allocation then after
> running few jobs in spark context, few executors keeps running like forever
> and never get free.
> *Configuration*
> {code:java}
> - "spark.master=yarn-client"
> - "spark.yarn.am.extraJavaOptions=-Dhdp.version=2.5.3.0-37"
> - "spark.sql.sources.maxConcurrentWrites=1"
> - "parquet.memory.pool.ratio=0.1"
> - "hive.map.aggr=true"
> - "spark.sql.shuffle.partitions=1200"
> - "spark.scheduler.mode=FAIR"
> -
> "spark.scheduler.allocation.file=/etc/spark/conf/fairscheduler.xml.template"
> - "spark.speculation=true"
> - "spark.dynamicAllocation.enabled=true"
> - "spark.shuffle.service.enabled=true"
> - "spark.dynamicAllocation.executorIdleTimeout=15s"
> - "spark.dynamicAllocation.cachedExecutorIdleTimeout=15s"
> - "spark.dynamicAllocation.initialExecutors=1"
> - "spark.dynamicAllocation.maxExecutors=900"
> - "spark.dynamicAllocation.minExecutors=1"
> - "spark.yarn.max.executor.failures=10000"
> - "spark.executor.cores=2"
> - "spark.executor.memory=8G"
> - "spark.sql.codegen=true"
> - "spark.sql.codegen.wholeStage=true"
> - "spark.sql.shuffle.partitions=75"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
