Re: Spark production scenario

2018-03-08 Thread yncxcw
hi, Passion I don't know an exact solution. But yes, the port each executor chosen to communicate with driver is random. I am wondering if it's possible that you can have a node has two ethernet card, configure one card for intranet for Spark and configure one card for WAN. Then connect the

Spark production scenario

2018-03-08 Thread रविशंकर नायर
Hi all, We are going to move to production with an 8 node Spark cluster. Request some help for below We are running on YARN cluster manager.That means YARN is installed with SSH between the nodes. When we run a standalone Spark program with spark-submit, YARN initializes a resource manager