Re: Securely discovering Application Master's metadata or sending a secret to Application Master at submission

[Mingyu Kim]

Ah, I see. I thought the RPC port was used for YARN’s own communication, but 
that field in ApplicationReport is exactly there for exposing some interfaces 
to the client, which is what I was looking for. I’ll take a look at MRAppMaster 
for an example. Thanks a lot for help!

 

Mingyu

 

From: Sunil Govind 
Date: Friday, June 10, 2016 at 5:37 AM
To: Rohith Sharma K S , Mingyu Kim 
, "user@hadoop.apache.org" 
Cc: Matt Cheah 
Subject: Re: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

 

Hi Mike 

 

Adding to what Rohith has mentioned, you can refer to below interface to know 
what all information which you can get from Yarn w.r.t one application. 
https://hadoop.apache.org/docs/r2.7.0/api/org/apache/hadoop/yarn/api/records/ApplicationReport.html

 

This has RPC port ApplicationMaster, and you can try to interact AM through 
that. Being said this, its upto ApplicationMaster to expose interfaces which 
you are looking. And YARN doesnt have any control on same as mentioned by 
Rohith.

 

- Sunil

 

 

On Fri, Jun 10, 2016 at 11:26 AM Rohith Sharma K S  
wrote:

Hi

 

Basically I see you have multiple questions

1.   How to get AM RPC port ?

>>> This you can get it via YarnClient# getApplicationReport(). This gives 
>>> common/generic application specific details. Note that RM does not maintain 
>>> any custom details for applications.

2.   How can you get metadata of AM?

>>> Basically AM design should be such that bind an interface to AM RPC. And 
>>> AM-RPC host and port can be obtained from ResourceManager. Using host:port 
>>> of AM from application submitter,  connect to AM and get required details 
>>> from AM only. To achieve this , YARN does not provide any interface since 
>>> AM are written users. Essentially, user can design AM to expose client 
>>> interface to their clients. For your better understanding , see MapReduce 
>>> framework MRAppMaster. 

3.   About the authenticity of job-submitter to AM 

>>> Use secured hadoop cluster with Kerberos enabled. Note that AM also should 
>>> be implemented for handling Kerberos.

 

Thanks & Regards

Rohith Sharma K S

 

From: Mingyu Kim [mailto:m...@palantir.com] 
Sent: 10 June 2016 03:47


To: Rohith Sharma K S; user@hadoop.apache.org
Cc: Matt Cheah
Subject: Re: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

 

Hi Rohith,

 

Thanks for the pointers. I checked the Hadoop documentation you linked, but 
it’s not clear how I can expose client interface for providing metadata. By 
“YARN internal communications”, I was referring to the endpoints that are 
exposed by AM on the RPC port as reported in ApplicationReport. I assume either 
RM or containers will communicate with AM through these endpoints.

 

I believe your suggestion is to expose additional endpoints to the AM RPC port. 
Can you clarify how I can do that? Is there an interface/class I need to 
extend? How can I register the extra endpoints for providing metadata on the 
existing AM RPC port?

 

Mingyu

 

From: Rohith Sharma K S 
Date: Wednesday, June 8, 2016 at 11:15 PM
To: Mingyu Kim , "user@hadoop.apache.org" 

Cc: Matt Cheah 
Subject: RE: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

 

Hi

 

Do you know how I can extend the client interface of the RPC port?

>>> YARN provides YARNClIent library that uses ApplicationClientProtocol. For 
>>> your more understanding refer 
>>> https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html#Writing_a_simple_Client

 

I know AM has some endpoints exposed through the RPC port for internal YARN 
communications, but was not sure how I can extend it to expose a custom 
endpoint.

>>> I am not sure what you mean here internal YARN communication? AM can 
>>> connect to RM only via AM-RM interface for register/unregister and 
>>> heartbeat and details sent to RM are limited.  It is up to the AM’s to 
>>> expose client interface for providing metadata.

Thanks & Regards

Rohith Sharma K S

From: Mingyu Kim [mailto:m...@palantir.com] 
Sent: 09 June 2016 11:21
To: Rohith Sharma K S; user@hadoop.apache.org
Cc: Matt Cheah
Subject: Re: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

 

Hi Rohith,

 

Thanks for the quick response. That sounds promising. Do you know how I can 
extend the client interface of the RPC port? I know AM has some endpoints 
exposed through the RPC port for internal YARN communications, but was not sure 
how I can extend it to expose a custom endpoint. Any pointer would be 
appreciated!

 

Mingyu

 

Re: Securely discovering Application Master's metadata or sending a secret to Application Master at submission

[Sunil Govind]

Hi Mike

Adding to what Rohith has mentioned, you can refer to below interface to
know what all information which you can get from Yarn w.r.t one
application.
https://hadoop.apache.org/docs/r2.7.0/api/org/apache/hadoop/yarn/api/records/ApplicationReport.html

This has RPC port ApplicationMaster, and you can try to interact AM through
that. Being said this, its upto ApplicationMaster to expose interfaces
which you are looking. And YARN doesnt have any control on same as
mentioned by Rohith.

- Sunil


On Fri, Jun 10, 2016 at 11:26 AM Rohith Sharma K S <
rohithsharm...@huawei.com> wrote:

> Hi
>
>
>
> Basically I see you have multiple questions
>
> 1.   How to get AM RPC port ?
>
> >>> This you can get it via YarnClient# getApplicationReport(). This
> gives common/generic application specific details. Note that RM does not
> maintain any custom details for applications.
>
> 2.   How can you get metadata of AM?
>
> >>> Basically AM design should be such that bind an interface to AM RPC.
> And AM-RPC host and port can be obtained from ResourceManager. Using
> host:port of AM from application submitter,  connect to AM and get required
> details from AM only. To achieve this , YARN does not provide any interface
> since AM are written users. Essentially, user can design AM to expose
> client interface to their clients. For your better understanding , see
> MapReduce framework MRAppMaster.
>
> 3.   About the authenticity of job-submitter to AM
>
> >>> Use secured hadoop cluster with Kerberos enabled. Note that AM also
> should be implemented for handling Kerberos.
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
>
>
> *From:* Mingyu Kim [mailto:m...@palantir.com]
> *Sent:* 10 June 2016 03:47
>
>
> *To:* Rohith Sharma K S; user@hadoop.apache.org
> *Cc:* Matt Cheah
> *Subject:* Re: Securely discovering Application Master's metadata or
> sending a secret to Application Master at submission
>
>
>
> Hi Rohith,
>
>
>
> Thanks for the pointers. I checked the Hadoop documentation you linked,
> but it’s not clear how I can expose client interface for providing
> metadata. By “YARN internal communications”, I was referring to the
> endpoints that are exposed by AM on the RPC port as reported in
> ApplicationReport. I assume either RM or containers will communicate with
> AM through these endpoints.
>
>
>
> I believe your suggestion is to expose additional endpoints to the AM RPC
> port. Can you clarify how I can do that? Is there an interface/class I need
> to extend? How can I register the extra endpoints for providing metadata on
> the existing AM RPC port?
>
>
>
> Mingyu
>
>
>
> *From: *Rohith Sharma K S 
> *Date: *Wednesday, June 8, 2016 at 11:15 PM
> *To: *Mingyu Kim , "user@hadoop.apache.org" <
> user@hadoop.apache.org>
> *Cc: *Matt Cheah 
> *Subject: *RE: Securely discovering Application Master's metadata or
> sending a secret to Application Master at submission
>
>
>
> Hi
>
>
>
> Do you know how I can extend the client interface of the RPC port?
>
> >>> YARN provides YARNClIent library that uses ApplicationClientProtocol.
> For your more understanding refer
> https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html#Writing_a_simple_Client
> 
>
>
>
> I know AM has some endpoints exposed through the RPC port for internal
> YARN communications, but was not sure how I can extend it to expose a
> custom endpoint.
>
> >>> I am not sure what you mean here internal YARN communication? AM can
> connect to RM only via AM-RM interface for register/unregister and
> heartbeat and details sent to RM are limited.  It is up to the AM’s to
> expose client interface for providing metadata.
>
> Thanks & Regards
>
> Rohith Sharma K S
>
> *From:* Mingyu Kim [mailto:m...@palantir.com ]
> *Sent:* 09 June 2016 11:21
> *To:* Rohith Sharma K S; user@hadoop.apache.org
> *Cc:* Matt Cheah
> *Subject:* Re: Securely discovering Application Master's metadata or
> sending a secret to Application Master at submission
>
>
>
> Hi Rohith,
>
>
>
> Thanks for the quick response. That sounds promising. Do you know how I
> can extend the client interface of the RPC port? I know AM has some
> endpoints exposed through the RPC port for internal YARN communications,
> but was not sure how I can extend it to expose a custom endpoint. Any
> pointer would be appreciated!
>
>
>
> Mingyu
>
>
>
> *From: *Rohith Sharma K S 
> *Date: *Wednesday, June 8, 2016 at 10:39 PM
> *To: *Mingyu Kim , "user@hadoop.apache.org" <
> 

RE: Securely discovering Application Master's metadata or sending a secret to Application Master at submission

[Rohith Sharma K S]

Hi

Basically I see you have multiple questions

1.   How to get AM RPC port ?

>>> This you can get it via YarnClient# getApplicationReport(). This gives 
>>> common/generic application specific details. Note that RM does not maintain 
>>> any custom details for applications.

2.   How can you get metadata of AM?

>>> Basically AM design should be such that bind an interface to AM RPC. And 
>>> AM-RPC host and port can be obtained from ResourceManager. Using host:port 
>>> of AM from application submitter,  connect to AM and get required details 
>>> from AM only. To achieve this , YARN does not provide any interface since 
>>> AM are written users. Essentially, user can design AM to expose client 
>>> interface to their clients. For your better understanding , see MapReduce 
>>> framework MRAppMaster.

3.   About the authenticity of job-submitter to AM

>>> Use secured hadoop cluster with Kerberos enabled. Note that AM also should 
>>> be implemented for handling Kerberos.


Thanks & Regards
Rohith Sharma K S

From: Mingyu Kim [mailto:m...@palantir.com]
Sent: 10 June 2016 03:47
To: Rohith Sharma K S; user@hadoop.apache.org
Cc: Matt Cheah
Subject: Re: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

Hi Rohith,

Thanks for the pointers. I checked the Hadoop documentation you linked, but 
it’s not clear how I can expose client interface for providing metadata. By 
“YARN internal communications”, I was referring to the endpoints that are 
exposed by AM on the RPC port as reported in ApplicationReport. I assume either 
RM or containers will communicate with AM through these endpoints.

I believe your suggestion is to expose additional endpoints to the AM RPC port. 
Can you clarify how I can do that? Is there an interface/class I need to 
extend? How can I register the extra endpoints for providing metadata on the 
existing AM RPC port?

Mingyu

From: Rohith Sharma K S 
>
Date: Wednesday, June 8, 2016 at 11:15 PM
To: Mingyu Kim >, 
"user@hadoop.apache.org" 
>
Cc: Matt Cheah >
Subject: RE: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

Hi

Do you know how I can extend the client interface of the RPC port?
>>> YARN provides YARNClIent library that uses ApplicationClientProtocol. For 
>>> your more understanding refer 
>>> https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html#Writing_a_simple_Client

I know AM has some endpoints exposed through the RPC port for internal YARN 
communications, but was not sure how I can extend it to expose a custom 
endpoint.
>>> I am not sure what you mean here internal YARN communication? AM can 
>>> connect to RM only via AM-RM interface for register/unregister and 
>>> heartbeat and details sent to RM are limited.  It is up to the AM’s to 
>>> expose client interface for providing metadata.
Thanks & Regards
Rohith Sharma K S
From: Mingyu Kim [mailto:m...@palantir.com]
Sent: 09 June 2016 11:21
To: Rohith Sharma K S; user@hadoop.apache.org
Cc: Matt Cheah
Subject: Re: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

Hi Rohith,

Thanks for the quick response. That sounds promising. Do you know how I can 
extend the client interface of the RPC port? I know AM has some endpoints 
exposed through the RPC port for internal YARN communications, but was not sure 
how I can extend it to expose a custom endpoint. Any pointer would be 
appreciated!

Mingyu

From: Rohith Sharma K S 
>
Date: Wednesday, June 8, 2016 at 10:39 PM
To: Mingyu Kim >, 
"user@hadoop.apache.org" 
>
Cc: Matt Cheah >
Subject: RE: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

Hi

Apart from AM address and tracking URL, no other meta data of applicationMaster 
are stored in YARN. May be AM can expose client interface so that AM clients 
can interact with Running AM to retrieve specific AM details.

RPC port of AM can be get from YARN client interface such as 

Re: Securely discovering Application Master's metadata or sending a secret to Application Master at submission

[Mingyu Kim]

Hi Rohith,

 

Thanks for the pointers. I checked the Hadoop documentation you linked, but 
it’s not clear how I can expose client interface for providing metadata. By 
“YARN internal communications”, I was referring to the endpoints that are 
exposed by AM on the RPC port as reported in ApplicationReport. I assume either 
RM or containers will communicate with AM through these endpoints.

 

I believe your suggestion is to expose additional endpoints to the AM RPC port. 
Can you clarify how I can do that? Is there an interface/class I need to 
extend? How can I register the extra endpoints for providing metadata on the 
existing AM RPC port?

 

Mingyu

 

From: Rohith Sharma K S 
Date: Wednesday, June 8, 2016 at 11:15 PM
To: Mingyu Kim , "user@hadoop.apache.org" 

Cc: Matt Cheah 
Subject: RE: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

 

Hi

 

Do you know how I can extend the client interface of the RPC port?

>>> YARN provides YARNClIent library that uses ApplicationClientProtocol. For 
>>> your more understanding refer 
>>> https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html#Writing_a_simple_Client

 

I know AM has some endpoints exposed through the RPC port for internal YARN 
communications, but was not sure how I can extend it to expose a custom 
endpoint.

>>> I am not sure what you mean here internal YARN communication? AM can 
>>> connect to RM only via AM-RM interface for register/unregister and 
>>> heartbeat and details sent to RM are limited.  It is up to the AM’s to 
>>> expose client interface for providing metadata.

Thanks & Regards

Rohith Sharma K S

From: Mingyu Kim [mailto:m...@palantir.com] 
Sent: 09 June 2016 11:21
To: Rohith Sharma K S; user@hadoop.apache.org
Cc: Matt Cheah
Subject: Re: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

 

Hi Rohith,

 

Thanks for the quick response. That sounds promising. Do you know how I can 
extend the client interface of the RPC port? I know AM has some endpoints 
exposed through the RPC port for internal YARN communications, but was not sure 
how I can extend it to expose a custom endpoint. Any pointer would be 
appreciated!

 

Mingyu

 

From: Rohith Sharma K S 
Date: Wednesday, June 8, 2016 at 10:39 PM
To: Mingyu Kim , "user@hadoop.apache.org" 

Cc: Matt Cheah 
Subject: RE: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

 

Hi

 

Apart from AM address and tracking URL, no other meta data of applicationMaster 
are stored in YARN. May be AM can expose client interface so that AM clients 
can interact with Running AM to retrieve specific AM details. 

 

RPC port of AM can be get from YARN client interface such as 
ApplicationClientProtocol# getApplicationReport() OR ApplicationClientProtocol 
#getApplicationAttemptReport().

 

Thanks & Regards

Rohith Sharma K S

 

From: Mingyu Kim [mailto:m...@palantir.com] 
Sent: 09 June 2016 10:36
To: user@hadoop.apache.org
Cc: Matt Cheah
Subject: Securely discovering Application Master's metadata or sending a secret 
to Application Master at submission

 

Hi all,

 

To provide a bit of background, I’m trying to deploy a REST server on 
Application Master and discover the randomly assigned port number securely. I 
can easily discover the host name of AM through YARN REST API, but the port 
number needs to be discovered separately. (Port number is assigned within a 
specified range with retries to avoid port conflicts) An easy solution would be 
to have Application Master make a callback with the port number, but I’d like 
to design it such that YARN nodes don’t talk back to the node that submitted 
the YARN application. So, this problem reduces to securely discovering a small 
metadata of Application Master. To be clear, by being secure, I’m less 
concerned about exposing the information to others, but more concerned about 
the integrity of data (e.g. the metadata actually originated from the 
Application Master.)

 

I was hoping that there is a way to register some Application Master metadata 
to Resource Manager, but there doesn’t seem to be a way. Another option I 
considered was to write the information to a HDFS file, but in order to verify 
the integrity of the content, I need a way to securely send a private key to 
Application Master, which I’m not sure what the best is.

 

To recap, does anyone know if there is a way

· To register small metadata securely from Application Master to 
Resource Manager so that it can be discovered by the YARN application submitter?

· Or, to securely send a private key to Application Master at the 
application submission time?

 

Thanks 

RE: Securely discovering Application Master's metadata or sending a secret to Application Master at submission

[Rohith Sharma K S]

Hi

Do you know how I can extend the client interface of the RPC port?
>>> YARN provides YARNClIent library that uses ApplicationClientProtocol. For 
>>> your more understanding refer 
>>> https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html#Writing_a_simple_Client

I know AM has some endpoints exposed through the RPC port for internal YARN 
communications, but was not sure how I can extend it to expose a custom 
endpoint.
>>> I am not sure what you mean here internal YARN communication? AM can 
>>> connect to RM only via AM-RM interface for register/unregister and 
>>> heartbeat and details sent to RM are limited.  It is up to the AM’s to 
>>> expose client interface for providing metadata.
Thanks & Regards
Rohith Sharma K S
From: Mingyu Kim [mailto:m...@palantir.com]
Sent: 09 June 2016 11:21
To: Rohith Sharma K S; user@hadoop.apache.org
Cc: Matt Cheah
Subject: Re: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

Hi Rohith,

Thanks for the quick response. That sounds promising. Do you know how I can 
extend the client interface of the RPC port? I know AM has some endpoints 
exposed through the RPC port for internal YARN communications, but was not sure 
how I can extend it to expose a custom endpoint. Any pointer would be 
appreciated!

Mingyu

From: Rohith Sharma K S 
>
Date: Wednesday, June 8, 2016 at 10:39 PM
To: Mingyu Kim >, 
"user@hadoop.apache.org" 
>
Cc: Matt Cheah >
Subject: RE: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

Hi

Apart from AM address and tracking URL, no other meta data of applicationMaster 
are stored in YARN. May be AM can expose client interface so that AM clients 
can interact with Running AM to retrieve specific AM details.

RPC port of AM can be get from YARN client interface such as 
ApplicationClientProtocol# getApplicationReport() OR ApplicationClientProtocol 
#getApplicationAttemptReport().

Thanks & Regards
Rohith Sharma K S

From: Mingyu Kim [mailto:m...@palantir.com]
Sent: 09 June 2016 10:36
To: user@hadoop.apache.org
Cc: Matt Cheah
Subject: Securely discovering Application Master's metadata or sending a secret 
to Application Master at submission

Hi all,

To provide a bit of background, I’m trying to deploy a REST server on 
Application Master and discover the randomly assigned port number securely. I 
can easily discover the host name of AM through YARN REST API, but the port 
number needs to be discovered separately. (Port number is assigned within a 
specified range with retries to avoid port conflicts) An easy solution would be 
to have Application Master make a callback with the port number, but I’d like 
to design it such that YARN nodes don’t talk back to the node that submitted 
the YARN application. So, this problem reduces to securely discovering a small 
metadata of Application Master. To be clear, by being secure, I’m less 
concerned about exposing the information to others, but more concerned about 
the integrity of data (e.g. the metadata actually originated from the 
Application Master.)

I was hoping that there is a way to register some Application Master metadata 
to Resource Manager, but there doesn’t seem to be a way. Another option I 
considered was to write the information to a HDFS file, but in order to verify 
the integrity of the content, I need a way to securely send a private key to 
Application Master, which I’m not sure what the best is.

To recap, does anyone know if there is a way

• To register small metadata securely from Application Master to 
Resource Manager so that it can be discovered by the YARN application submitter?

• Or, to securely send a private key to Application Master at the 
application submission time?

Thanks a lot,
Mingyu

Re: Securely discovering Application Master's metadata or sending a secret to Application Master at submission

[Mingyu Kim]

Hi Rohith,

 

Thanks for the quick response. That sounds promising. Do you know how I can 
extend the client interface of the RPC port? I know AM has some endpoints 
exposed through the RPC port for internal YARN communications, but was not sure 
how I can extend it to expose a custom endpoint. Any pointer would be 
appreciated!

 

Mingyu

 

From: Rohith Sharma K S 
Date: Wednesday, June 8, 2016 at 10:39 PM
To: Mingyu Kim , "user@hadoop.apache.org" 

Cc: Matt Cheah 
Subject: RE: Securely discovering Application Master's metadata or sending a 
secret to Application Master at submission

 

Hi

 

Apart from AM address and tracking URL, no other meta data of applicationMaster 
are stored in YARN. May be AM can expose client interface so that AM clients 
can interact with Running AM to retrieve specific AM details. 

 

RPC port of AM can be get from YARN client interface such as 
ApplicationClientProtocol# getApplicationReport() OR ApplicationClientProtocol 
#getApplicationAttemptReport().

 

Thanks & Regards

Rohith Sharma K S

 

From: Mingyu Kim [mailto:m...@palantir.com] 
Sent: 09 June 2016 10:36
To: user@hadoop.apache.org
Cc: Matt Cheah
Subject: Securely discovering Application Master's metadata or sending a secret 
to Application Master at submission

 

Hi all,

 

To provide a bit of background, I’m trying to deploy a REST server on 
Application Master and discover the randomly assigned port number securely. I 
can easily discover the host name of AM through YARN REST API, but the port 
number needs to be discovered separately. (Port number is assigned within a 
specified range with retries to avoid port conflicts) An easy solution would be 
to have Application Master make a callback with the port number, but I’d like 
to design it such that YARN nodes don’t talk back to the node that submitted 
the YARN application. So, this problem reduces to securely discovering a small 
metadata of Application Master. To be clear, by being secure, I’m less 
concerned about exposing the information to others, but more concerned about 
the integrity of data (e.g. the metadata actually originated from the 
Application Master.)

 

I was hoping that there is a way to register some Application Master metadata 
to Resource Manager, but there doesn’t seem to be a way. Another option I 
considered was to write the information to a HDFS file, but in order to verify 
the integrity of the content, I need a way to securely send a private key to 
Application Master, which I’m not sure what the best is.

 

To recap, does anyone know if there is a way

· To register small metadata securely from Application Master to 
Resource Manager so that it can be discovered by the YARN application submitter?

· Or, to securely send a private key to Application Master at the 
application submission time?

 

Thanks a lot,

Mingyu



smime.p7s
Description: S/MIME cryptographic signature

RE: Securely discovering Application Master's metadata or sending a secret to Application Master at submission

[Rohith Sharma K S]

Hi

Apart from AM address and tracking URL, no other meta data of applicationMaster 
are stored in YARN. May be AM can expose client interface so that AM clients 
can interact with Running AM to retrieve specific AM details.

RPC port of AM can be get from YARN client interface such as 
ApplicationClientProtocol# getApplicationReport() OR ApplicationClientProtocol 
#getApplicationAttemptReport().

Thanks & Regards
Rohith Sharma K S

From: Mingyu Kim [mailto:m...@palantir.com]
Sent: 09 June 2016 10:36
To: user@hadoop.apache.org
Cc: Matt Cheah
Subject: Securely discovering Application Master's metadata or sending a secret 
to Application Master at submission

Hi all,

To provide a bit of background, I’m trying to deploy a REST server on 
Application Master and discover the randomly assigned port number securely. I 
can easily discover the host name of AM through YARN REST API, but the port 
number needs to be discovered separately. (Port number is assigned within a 
specified range with retries to avoid port conflicts) An easy solution would be 
to have Application Master make a callback with the port number, but I’d like 
to design it such that YARN nodes don’t talk back to the node that submitted 
the YARN application. So, this problem reduces to securely discovering a small 
metadata of Application Master. To be clear, by being secure, I’m less 
concerned about exposing the information to others, but more concerned about 
the integrity of data (e.g. the metadata actually originated from the 
Application Master.)

I was hoping that there is a way to register some Application Master metadata 
to Resource Manager, but there doesn’t seem to be a way. Another option I 
considered was to write the information to a HDFS file, but in order to verify 
the integrity of the content, I need a way to securely send a private key to 
Application Master, which I’m not sure what the best is.

To recap, does anyone know if there is a way

• To register small metadata securely from Application Master to 
Resource Manager so that it can be discovered by the YARN application submitter?

• Or, to securely send a private key to Application Master at the 
application submission time?

Thanks a lot,
Mingyu