Hi All,
I encountered this problem when the firewall is enabled between the spark-shell
and the Workers.
When I launch spark-shell in yarn-client mode, I notice that Workers on the
YARN containers are trying to talk to the driver (spark-shell), however, the
firewall is not opened and caused timeout.
For the Workers, it tries to open listening ports on 54xxx for each Worker? Is
the port random in such case?What will be the better way to predict the ports
so I can configure the firewall correctly between the driver (spark-shell) and
the Workers? Is there a range of ports we can specify in the firewall/iptables?
Any ideas?
