[ https://issues.apache.org/jira/browse/SPARK-29152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932436#comment-16932436 ]
Rakesh Raushan commented on SPARK-29152: ---------------------------------------- I will look into this issue. > Spark Executor Plugin API shutdown is not proper when dynamic allocation > enabled[SPARK-24918] > --------------------------------------------------------------------------------------------- > > Key: SPARK-29152 > URL: https://issues.apache.org/jira/browse/SPARK-29152 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.4.4 > Reporter: jobit mathew > Priority: Major > > *Issue Description* > Spark Executor Plugin API *shutdown handling is not proper*, when dynamic > allocation enabled .Plugin shutdown method is not processed-while dynamic > allocation is enabled and *executors become dead* after inactive time. > *Test Precondition* > 1.Prepared 4 spark applications with executor plugin interface. > First application-SparkExecutorplugin.jar > import org.apache.spark.ExecutorPlugin; > public class ExecutoTest1 implements ExecutorPlugin{ > public void init(){ > System.out.println("Executor Plugin Initialised."); > } > public void shutdown(){ > System.out.println("Executor plugin closed successfully."); > } > } > 2. Create the jars with the same and put it in folder /spark/examples/jars > *Test Steps* > 1. launch bin/spark-sql with dynamic allocation enabled > ./spark-sql --master yarn --conf spark.executor.plugins=ExecutoTest1 > --conf="spark.executor.extraClassPath=/opt/HA/C10/install/spark/spark/examples/jars/*" > --conf spark.dynamicAllocation.enabled=true --conf > spark.dynamicAllocation.initialExecutors=2 --conf > spark.dynamicAllocation.minExecutors=0 > 2 create a table , insert the data and select * from tablename > 3.Check the spark UI Jobs tab/SQL tab > 4. Check all Executors(executor tab will give all executors details) > application log file for Executor plugin Initialization and Shutdown messages > or operations. > Example > /yarn/logdir/application_1567156749079_0025/container_e02_1567156749079_0025_01_000005/ > stdout > 5. Wait for the executor to be dead after the inactive time and check the > same container log > 6. Kill the spark sql and check the container log for executor plugin > shutdown. > *Expect Output* > 1. Job should be success. Create table ,insert and select query should be > success. > 2.While running query All Executors log should contain the executor plugin > Init and shutdown messages or operations. > "Executor Plugin Initialised. > 3.Once the executors are dead ,executor shutdown should call shutdown > message should be there in log file. > “ Executor plugin closed successfully. > 4.Once the sql application closed ,executor shutdown should call shutdown > message should be there in log. > “ Executor plugin closed successfully". > *Actual Output* > Shutdown message is not called when executor is dead or after closing the > application > *Observation* > Without dynamic allocation Executor plugin is working fine.But after enabling > dynamic allocation,Executor shutdown is not processed. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org