In production, i'd suggest you having a High availability cluster with
minimum of 3 nodes (data nodes in your case).
Now lets examine your scenario:
- When you suddenly brings down one of the node which has 2 executors
running on it, what happens is that the node (DN2) will be having your jobs
Hi,
We are observing a hung spark application when one of the yarn datanode
(running multiple spark executors) go down.
Setup details:
* Spark: 1.2.1
* Hadoop: 2.4.0
* Spark Application Mode: yarn-client
* 2 datanodes (DN1, DN2)
* 6 spark executors (initially 3 executors on
