I would also check firewall rules. Is communication allowed on all the required port ranges and hosts ?
On Thu, Jul 18, 2019 at 3:56 AM Amit Sharma <[email protected]> wrote: > Do you have dynamic resource allocation enabled? > > > On Wednesday, July 17, 2019, zenglong chen <[email protected]> > wrote: > >> Hi,all, >> My standalone mode has two slaves.When I submit my job,the >> localhost slave is working well,but second slave do add and remove executor >> action always!The log are below: >> 2019-07-17 10:51:38,889 INFO >> client.StandaloneAppClient$ClientEndpoint: Executor updated: >> app-20190717105135-0008/2 is now EXITED (Command exited with code 1) >> 2019-07-17 10:51:38,890 INFO cluster.StandaloneSchedulerBackend: Executor >> app-20190717105135-0008/2 removed: Command exited with code 1 >> 2019-07-17 10:51:38,890 INFO storage.BlockManagerMasterEndpoint: Trying >> to remove executor 2 from BlockManagerMaster. >> 2019-07-17 10:51:38,890 INFO storage.BlockManagerMaster: Removal of >> executor 2 requested >> 2019-07-17 10:51:38,891 INFO >> cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove >> non-existent executor 2 >> 2019-07-17 10:51:38,892 INFO client.StandaloneAppClient$ClientEndpoint: >> Executor added: app-20190717105135-0008/3 on >> worker-20190717093045-172.22.9.179-40573 (172.22.9.179:40573) with 8 >> core(s) >> 2019-07-17 10:51:38,892 INFO cluster.StandaloneSchedulerBackend: Granted >> executor ID app-20190717105135-0008/3 on hostPort 172.22.9.179:40573 >> with 8 core(s), 12.0 GB RAM >> 2019-07-17 10:51:38,893 INFO client.StandaloneAppClient$ClientEndpoint: >> Executor updated: app-20190717105135-0008/3 is now RUNNING >> 2019-07-17 10:51:40,521 INFO client.StandaloneAppClient$ClientEndpoint: >> Executor updated: app-20190717105135-0008/3 is now EXITED (Command exited >> with code 1) >> 2019-07-17 10:51:40,521 INFO cluster.StandaloneSchedulerBackend: Executor >> app-20190717105135-0008/3 removed: Command exited with code 1 >> 2019-07-17 10:51:40,521 INFO client.StandaloneAppClient$ClientEndpoint: >> Executor added: app-20190717105135-0008/4 on >> worker-20190717093045-172.22.9.179-40573 (172.22.9.179:40573) with 8 >> core(s) >> 2019-07-17 10:51:40,521 INFO storage.BlockManagerMaster: Removal of >> executor 3 requested >> 2019-07-17 10:51:40,521 INFO >> cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove >> non-existent executor 3 >> 2019-07-17 10:51:40,521 INFO storage.BlockManagerMasterEndpoint: Trying >> to remove executor 3 from BlockManagerMaster. >> 2019-07-17 10:51:40,521 INFO cluster.StandaloneSchedulerBackend: Granted >> executor ID app-20190717105135-0008/4 on hostPort 172.22.9.179:40573 >> with 8 core(s), 12.0 GB RAM >> 2019-07-17 10:51:40,523 INFO client.StandaloneAppClient$ClientEndpoint: >> Executor updated: app-20190717105135-0008/4 is now RUNNING >> >> >> And the slave output are below: >> 19/07/17 10:47:12 INFO ExecutorRunner: Launch command: >> "/home/ubuntu/data/jdk/jre/bin/java" "-cp" >> "/home/ubuntu/spark-2.4.3/conf/:/home/ubuntu/spark-2.4.3/jars/*" >> "-Xmx12288M" "-Dspark.driver.port=40335" >> "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" >> "spark://CoarseGrainedScheduler@iZk1a7vdbutmi6eluaskecZ:40335" >> "--executor-id" "18" "--hostname" "172.22.9.179" "--cores" "8" "--app-id" >> "app-20190717104645-0007" "--worker-url" "spark:// >> [email protected]:40573" >> 19/07/17 10:47:13 INFO Worker: Executor app-20190717104645-0007/18 >> finished with state EXITED message Command exited with code 1 exitStatus 1 >> 19/07/17 10:47:13 INFO ExternalShuffleBlockResolver: Clean up non-shuffle >> files associated with the finished executor 18 >> 19/07/17 10:47:13 INFO ExternalShuffleBlockResolver: Executor is not >> registered (appId=app-20190717104645-0007, execId=18) >> 19/07/17 10:47:13 INFO Worker: Asked to launch executor >> app-20190717104645-0007/19 for ph_user_pre_level >> 19/07/17 10:47:13 INFO SecurityManager: Changing view acls to: ubuntu >> 19/07/17 10:47:13 INFO SecurityManager: Changing modify acls to: ubuntu >> 19/07/17 10:47:13 INFO SecurityManager: Changing view acls groups to: >> 19/07/17 10:47:13 INFO SecurityManager: Changing modify acls groups to: >> 19/07/17 10:47:13 INFO SecurityManager: SecurityManager: authentication >> disabled; ui acls disabled; users with view permissions: Set(ubuntu); >> groups with view permissions: Set(); users with modify permissions: >> Set(ubuntu); groups with modify permissions: Set() >> 19/07/17 10:47:14 INFO ExecutorRunner: Launch command: >> "/home/ubuntu/data/jdk/jre/bin/java" "-cp" >> "/home/ubuntu/spark-2.4.3/conf/:/home/ubuntu/spark-2.4.3/jars/*" >> "-Xmx12288M" "-Dspark.driver.port=40335" >> "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" >> "spark://CoarseGrainedScheduler@iZk1a7vdbutmi6eluaskecZ:40335" >> "--executor-id" "19" "--hostname" "172.22.9.179" "--cores" "8" "--app-id" >> "app-20190717104645-0007" "--worker-url" "spark:// >> [email protected]:40573" >> >> I guest that may be "Dspark.driver.port=40335" problem. >> Any suggests will help me a lot! >> >
