Re: Port to open for submitting Spark on Yarn application

[Satoshi Yamada]

Jerry,

Thanks for your comment.

On Mon, Sep 4, 2017 at 10:43 AM, Saisai Shao  wrote:

> I think spark.yarn.am.port is not used any more, so you don't need to
> consider this.
>
> If you're running Spark on YARN, I think some YARN RM port to submit
> applications should also be reachable via firewall, as well as HDFS port to
> upload resources.
>
> Also in the Spark side, executors will be connected to driver via
> spark.driver.port, maybe you should also set a fixed port number for this
> and add to white list of firewall.
>
> Thanks
> Jerry
>
>
> On Mon, Sep 4, 2017 at 8:50 AM, Satoshi Yamada <
> satoshi.yamada@gmail.com> wrote:
>
>> Hi,
>>
>> In case we run Spark on Yarn in client mode, we have firewall for Hadoop 
>> cluster,
>> and the client node is outside firewall, I think I have to open some ports
>> that Application Master uses.
>>
>>
>> I think the ports is specified by "spark.yarn.am.port" as document says.
>> https://spark.apache.org/docs/latest/running-on-yarn.html
>>
>> But, according to the source code, spark.yarn.am.port is deprecated since 
>> 2.0.
>> https://github.com/apache/spark/commit/829cd7b8b70e65a91aa66e6d626bd45f18e0ad97
>>
>> Does this mean we do not need to open particular ports of firewall for
>>
>> Spark on Yarn?
>>
>>
>> Thanks,
>>
>>
>

Re: Port to open for submitting Spark on Yarn application

[Saisai Shao]

I think spark.yarn.am.port is not used any more, so you don't need to
consider this.

If you're running Spark on YARN, I think some YARN RM port to submit
applications should also be reachable via firewall, as well as HDFS port to
upload resources.

Also in the Spark side, executors will be connected to driver via
spark.driver.port, maybe you should also set a fixed port number for this
and add to white list of firewall.

Thanks
Jerry


On Mon, Sep 4, 2017 at 8:50 AM, Satoshi Yamada  wrote:

> Hi,
>
> In case we run Spark on Yarn in client mode, we have firewall for Hadoop 
> cluster,
> and the client node is outside firewall, I think I have to open some ports
> that Application Master uses.
>
>
> I think the ports is specified by "spark.yarn.am.port" as document says.
> https://spark.apache.org/docs/latest/running-on-yarn.html
>
> But, according to the source code, spark.yarn.am.port is deprecated since 2.0.
> https://github.com/apache/spark/commit/829cd7b8b70e65a91aa66e6d626bd45f18e0ad97
>
> Does this mean we do not need to open particular ports of firewall for
>
> Spark on Yarn?
>
>
> Thanks,
>
>

Port to open for submitting Spark on Yarn application

[Satoshi Yamada]

Hi,

In case we run Spark on Yarn in client mode, we have firewall for
Hadoop cluster,
and the client node is outside firewall, I think I have to open some ports
that Application Master uses.


I think the ports is specified by "spark.yarn.am.port" as document says.
https://spark.apache.org/docs/latest/running-on-yarn.html

But, according to the source code, spark.yarn.am.port is deprecated since 2.0.
https://github.com/apache/spark/commit/829cd7b8b70e65a91aa66e6d626bd45f18e0ad97

Does this mean we do not need to open particular ports of firewall for

Spark on Yarn?


Thanks,