I think what you want to do is set spark.driver.port to a fixed port.
On Fri, May 2, 2014 at 1:52 PM, Andrew Lee <alee...@hotmail.com> wrote: > Hi All, > > I encountered this problem when the firewall is enabled between the > spark-shell and the Workers. > > When I launch spark-shell in yarn-client mode, I notice that Workers on > the YARN containers are trying to talk to the driver (spark-shell), > however, the firewall is not opened and caused timeout. > > For the Workers, it tries to open listening ports on 54xxx for each > Worker? Is the port random in such case? > What will be the better way to predict the ports so I can configure the > firewall correctly between the driver (spark-shell) and the Workers? Is > there a range of ports we can specify in the firewall/iptables? > > Any ideas? >
