[ https://issues.apache.org/jira/browse/SPARK-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035903#comment-16035903 ]
harish chandra edited comment on SPARK-20975 at 6/3/17 9:39 AM: ---------------------------------------------------------------- Thanks [~sowen] for quick response, Since I am passing config {code:java} spark.dynamicAllocation.executorIdleTimeout=15s {code} which says after 15 seconds executor should cool down, if no job running. But I don't see this expected behavior was (Author: harry.rat...@gmail.com): Thanks [~sowen] for quick response, Since I am passing config {code:java} spark.dynamicAllocation.executorIdleTimeout=15s {code} which says after 15 seconds execution should cool down, if no job running. But I don't see this expected behavior > Excutors are no released if speculation + dynamic allocation enabled > --------------------------------------------------------------------- > > Key: SPARK-20975 > URL: https://issues.apache.org/jira/browse/SPARK-20975 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.6.2 > Reporter: harish chandra > > Whenever user enable spark speculation + dynamic allocation then after > running few jobs in spark context, few executors keeps running like forever > and never get free. > *Configuration* > {code:java} > - "spark.master=yarn-client" > - "spark.yarn.am.extraJavaOptions=-Dhdp.version=2.5.3.0-37" > - "spark.sql.sources.maxConcurrentWrites=1" > - "parquet.memory.pool.ratio=0.1" > - "hive.map.aggr=true" > - "spark.sql.shuffle.partitions=1200" > - "spark.scheduler.mode=FAIR" > - > "spark.scheduler.allocation.file=/etc/spark/conf/fairscheduler.xml.template" > - "spark.speculation=true" > - "spark.dynamicAllocation.enabled=true" > - "spark.shuffle.service.enabled=true" > - "spark.dynamicAllocation.executorIdleTimeout=15s" > - "spark.dynamicAllocation.cachedExecutorIdleTimeout=15s" > - "spark.dynamicAllocation.initialExecutors=1" > - "spark.dynamicAllocation.maxExecutors=900" > - "spark.dynamicAllocation.minExecutors=1" > - "spark.yarn.max.executor.failures=10000" > - "spark.executor.cores=2" > - "spark.executor.memory=8G" > - "spark.sql.codegen=true" > - "spark.sql.codegen.wholeStage=true" > - "spark.sql.shuffle.partitions=75" > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org