Re: Review Request 46282: SAMZA-928 document Kerberos on YARN

[Chen Song]

> On May 26, 2016, 9:37 p.m., Yi Pan (Data Infrastructure) wrote:
> > docs/learn/documentation/versioned/yarn/yarn-security.md, line 66
> > 
> >
> > nit: trailing white space

Done


> On May 26, 2016, 9:37 p.m., Yi Pan (Data Infrastructure) wrote:
> > docs/learn/documentation/versioned/yarn/yarn-security.md, line 68
> > 
> >
> > Here as well.

Done


- Chen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46282/#review135081
---


On June 15, 2016, 2:25 a.m., Chen Song wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46282/
> ---
> 
> (Updated June 15, 2016, 2:25 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-928 document Kerberos on YARN
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/index.html c2a7b29 
>   docs/learn/documentation/versioned/jobs/configuration-table.html c1c822e 
>   docs/learn/documentation/versioned/jobs/yarn-jobs.md 827cc14 
>   docs/learn/documentation/versioned/yarn/isolation.md 1eb3bf5 
>   docs/learn/documentation/versioned/yarn/yarn-security.md PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/46282/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Chen Song
> 
>

Re: Review Request 46282: SAMZA-928 document Kerberos on YARN

[Chen Song]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46282/
---

(Updated June 15, 2016, 2:25 a.m.)


Review request for samza.


Repository: samza


Description
---

SAMZA-928 document Kerberos on YARN


Diffs (updated)
-

  docs/learn/documentation/versioned/index.html c2a7b29 
  docs/learn/documentation/versioned/jobs/configuration-table.html c1c822e 
  docs/learn/documentation/versioned/jobs/yarn-jobs.md 827cc14 
  docs/learn/documentation/versioned/yarn/isolation.md 1eb3bf5 
  docs/learn/documentation/versioned/yarn/yarn-security.md PRE-CREATION 

Diff: https://reviews.apache.org/r/46282/diff/


Testing
---


Thanks,

Chen Song

Re: Review Request 46282: SAMZA-928 document Kerberos on YARN

[Yi Pan (Data Infrastructure)]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46282/#review135081
---



It seems that the following configuration variable from SAMZA-727 is still 
missing: yarn.job.staging.directory. Thanks!


docs/learn/documentation/versioned/yarn/yarn-security.md (line 28)


If we have test this step, let's not wait for the YARN community responses.



docs/learn/documentation/versioned/yarn/yarn-security.md (line 66)


nit: trailing white space



docs/learn/documentation/versioned/yarn/yarn-security.md (line 68)


Here as well.


- Yi Pan (Data Infrastructure)


On May 25, 2016, 8:11 p.m., Chen Song wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46282/
> ---
> 
> (Updated May 25, 2016, 8:11 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-928 document Kerberos on YARN
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/jobs/configuration-table.html c1c822e 
>   docs/learn/documentation/versioned/jobs/yarn-jobs.md 827cc14 
>   docs/learn/documentation/versioned/yarn/isolation.md 1eb3bf5 
>   docs/learn/documentation/versioned/yarn/yarn-security.md PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/46282/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Chen Song
> 
>

Re: Review Request 46282: SAMZA-928 document Kerberos on YARN

[Chen Song]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46282/
---

(Updated May 25, 2016, 8:11 p.m.)


Review request for samza.


Repository: samza


Description
---

SAMZA-928 document Kerberos on YARN


Diffs (updated)
-

  docs/learn/documentation/versioned/jobs/configuration-table.html c1c822e 
  docs/learn/documentation/versioned/jobs/yarn-jobs.md 827cc14 
  docs/learn/documentation/versioned/yarn/isolation.md 1eb3bf5 
  docs/learn/documentation/versioned/yarn/yarn-security.md PRE-CREATION 

Diff: https://reviews.apache.org/r/46282/diff/


Testing
---


Thanks,

Chen Song

Re: Review Request 46282: SAMZA-928 document Kerberos on YARN

[Chen Song]

> On April 26, 2016, 7:33 p.m., Yi Pan (Data Infrastructure) wrote:
> > docs/learn/documentation/versioned/yarn/yarn-security.md, line 34
> > 
> >
> > Question: wouldn't the running container needs a HDFS delegation token 
> > to access secured HDFS to read the credential files as well? How is the 
> > initial HDFS delegation token passed to the container? Via launch context 
> > from RM? It would be good to add some explanation, or pointing to some 
> > online docs for general YARN APP launch sequence w/ Kerberos.
> 
> Chen Song wrote:
> Yes, that is my understanding. Each container, once initialized, will use 
> whatever HDFS delegation token passed in the launch context. However, once it 
> expires, the container won't get renewed or refreshed token from RM and that 
> is why we need to manage renewal of HDFS delegation tokens ourselves. Same 
> rule applies to AM too. Let me summarize this a bit, and provide some 
> detailed explanations once confirm from the Hadoop community.

I summarized what we understood on Yarn when running long lived applications 
and posted on hadoop user list. Hopefully, someone will clarify and confirm our 
understanding is correct.

http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201605.mbox/%3ccagf+3ryx_nqohkqa1ot+jmvnanahonefs8dzgjq8lmy7ygg...@mail.gmail.com%3E


- Chen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46282/#review130660
---


On April 15, 2016, 10:09 p.m., Chen Song wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46282/
> ---
> 
> (Updated April 15, 2016, 10:09 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-928 document Kerberos on YARN
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/jobs/yarn-jobs.md 827cc14 
>   docs/learn/documentation/versioned/yarn/isolation.md 1eb3bf5 
>   docs/learn/documentation/versioned/yarn/yarn-security.md PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/46282/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Chen Song
> 
>

Re: Review Request 46282: SAMZA-928 document Kerberos on YARN

[Yi Pan (Data Infrastructure)]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46282/#review134509
---



Thanks for answering all the questions. There is one more point I missed last 
time. Thanks!


docs/learn/documentation/versioned/yarn/yarn-security.md (line 48)


One more point: please add them in the configuration table as in 
docs/learn/documentation/versioned/jobs/configuration-table.html


- Yi Pan (Data Infrastructure)


On April 15, 2016, 10:09 p.m., Chen Song wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46282/
> ---
> 
> (Updated April 15, 2016, 10:09 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-928 document Kerberos on YARN
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/jobs/yarn-jobs.md 827cc14 
>   docs/learn/documentation/versioned/yarn/isolation.md 1eb3bf5 
>   docs/learn/documentation/versioned/yarn/yarn-security.md PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/46282/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Chen Song
> 
>

Re: Review Request 46282: SAMZA-928 document Kerberos on YARN

[Chen Song]

> On April 26, 2016, 7:33 p.m., Yi Pan (Data Infrastructure) wrote:
> > docs/learn/documentation/versioned/yarn/yarn-security.md, line 28
> > 
> >
> > nit: remove *config*, we can directly refer to the coordinator stream
> 
> Yi Pan (Data Infrastructure) wrote:
> One more: it would be good to elaborate the reason not to put the keytab 
> file in the coordinator stream as well. I assume that the reason is that 
> secured HDFS file system is protected via kerberos, while coordinator stream 
> is not??

For both keytab and refreshed credentials, HDFS is the only place that is 
secure via Kerberos for now.

Regarding keytab file, the Hadoop log in API can only deal with a keytab file. 
If being stored in a coordinator stream, the Application Master will want to 
localize the bytes stream into a local file and has to deal with this at 
application level. So it adds a bit complexity.

Regarding refreshed credentials, it can be potentially exchanged on the 
coordinator stream, as Hadoop supports both APIs to read credentials from HDFS 
and a generic stream. Once we have coordinator stream secured, we can revisit 
this.


- Chen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46282/#review130660
---


On April 15, 2016, 10:09 p.m., Chen Song wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46282/
> ---
> 
> (Updated April 15, 2016, 10:09 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-928 document Kerberos on YARN
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/jobs/yarn-jobs.md 827cc14 
>   docs/learn/documentation/versioned/yarn/isolation.md 1eb3bf5 
>   docs/learn/documentation/versioned/yarn/yarn-security.md PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/46282/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Chen Song
> 
>

Re: Review Request 46282: SAMZA-928 document Kerberos on YARN

[Chen Song]

> On April 26, 2016, 7:33 p.m., Yi Pan (Data Infrastructure) wrote:
> > docs/learn/documentation/versioned/yarn/yarn-security.md, line 48
> > 
> >
> > nit: trailing white space

done.


> On April 26, 2016, 7:33 p.m., Yi Pan (Data Infrastructure) wrote:
> > docs/learn/documentation/versioned/yarn/yarn-security.md, line 62
> > 
> >
> > nit: trailing white space

done.


> On April 26, 2016, 7:33 p.m., Yi Pan (Data Infrastructure) wrote:
> > docs/learn/documentation/versioned/yarn/yarn-security.md, line 30
> > 
> >
> > nit: config stream --> coordinator stream

done.


> On April 26, 2016, 7:33 p.m., Yi Pan (Data Infrastructure) wrote:
> > docs/learn/documentation/versioned/yarn/yarn-security.md, line 34
> > 
> >
> > Question: wouldn't the running container needs a HDFS delegation token 
> > to access secured HDFS to read the credential files as well? How is the 
> > initial HDFS delegation token passed to the container? Via launch context 
> > from RM? It would be good to add some explanation, or pointing to some 
> > online docs for general YARN APP launch sequence w/ Kerberos.

Yes, that is my understanding. Each container, once initialized, will use 
whatever HDFS delegation token passed in the launch context. However, once it 
expires, the container won't get renewed or refreshed token from RM and that is 
why we need to manage renewal of HDFS delegation tokens ourselves. Same rule 
applies to AM too. Let me summarize this a bit, and provide some detailed 
explanations once confirm from the Hadoop community.


- Chen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46282/#review130660
---


On April 15, 2016, 10:09 p.m., Chen Song wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46282/
> ---
> 
> (Updated April 15, 2016, 10:09 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-928 document Kerberos on YARN
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/jobs/yarn-jobs.md 827cc14 
>   docs/learn/documentation/versioned/yarn/isolation.md 1eb3bf5 
>   docs/learn/documentation/versioned/yarn/yarn-security.md PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/46282/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Chen Song
> 
>

Re: Review Request 46282: SAMZA-928 document Kerberos on YARN

[Yi Pan (Data Infrastructure)]

> On April 26, 2016, 7:33 p.m., Yi Pan (Data Infrastructure) wrote:
> > docs/learn/documentation/versioned/yarn/yarn-security.md, line 28
> > 
> >
> > nit: remove *config*, we can directly refer to the coordinator stream

One more: it would be good to elaborate the reason not to put the keytab file 
in the coordinator stream as well. I assume that the reason is that secured 
HDFS file system is protected via kerberos, while coordinator stream is not??


- Yi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46282/#review130660
---


On April 15, 2016, 10:09 p.m., Chen Song wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46282/
> ---
> 
> (Updated April 15, 2016, 10:09 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-928 document Kerberos on YARN
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/jobs/yarn-jobs.md 827cc14 
>   docs/learn/documentation/versioned/yarn/isolation.md 1eb3bf5 
>   docs/learn/documentation/versioned/yarn/yarn-security.md PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/46282/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Chen Song
> 
>

Re: Review Request 46282: SAMZA-928 document Kerberos on YARN

[Yi Pan (Data Infrastructure)]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46282/#review130660
---


Fix it, then Ship it!




Lgtm overall! Thanks a lot!


docs/learn/documentation/versioned/yarn/yarn-security.md (line 28)


nit: remove *config*, we can directly refer to the coordinator stream



docs/learn/documentation/versioned/yarn/yarn-security.md (line 30)


nit: config stream --> coordinator stream



docs/learn/documentation/versioned/yarn/yarn-security.md (line 34)


Question: wouldn't the running container needs a HDFS delegation token to 
access secured HDFS to read the credential files as well? How is the initial 
HDFS delegation token passed to the container? Via launch context from RM? It 
would be good to add some explanation, or pointing to some online docs for 
general YARN APP launch sequence w/ Kerberos.



docs/learn/documentation/versioned/yarn/yarn-security.md (line 48)


nit: trailing white space



docs/learn/documentation/versioned/yarn/yarn-security.md (line 62)


nit: trailing white space



docs/learn/documentation/versioned/yarn/yarn-security.md (line 63)


Cool! If this works, feel free to close SAMZA-929!


- Yi Pan (Data Infrastructure)


On April 15, 2016, 10:09 p.m., Chen Song wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46282/
> ---
> 
> (Updated April 15, 2016, 10:09 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-928 document Kerberos on YARN
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/jobs/yarn-jobs.md 827cc14 
>   docs/learn/documentation/versioned/yarn/isolation.md 1eb3bf5 
>   docs/learn/documentation/versioned/yarn/yarn-security.md PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/46282/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Chen Song
> 
>

Re: Review Request 46282: SAMZA-928 document Kerberos on YARN

[Chen Song]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46282/
---

(Updated April 15, 2016, 10:09 p.m.)


Review request for samza.


Repository: samza


Description
---

SAMZA-928 document Kerberos on YARN


Diffs (updated)
-

  docs/learn/documentation/versioned/jobs/yarn-jobs.md 827cc14 
  docs/learn/documentation/versioned/yarn/isolation.md 1eb3bf5 
  docs/learn/documentation/versioned/yarn/yarn-security.md PRE-CREATION 

Diff: https://reviews.apache.org/r/46282/diff/


Testing
---


Thanks,

Chen Song

Review Request 46282: SAMZA-928 document Kerberos on YARN

[Chen Song]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46282/
---

Review request for samza.


Repository: samza


Description
---

SAMZA-928 document Kerberos on YARN


Diffs
-

  docs/learn/documentation/versioned/jobs/yarn-jobs.md 827cc14 
  docs/learn/documentation/versioned/yarn/isolation.md 1eb3bf5 
  docs/learn/documentation/versioned/yarn/yarn-security.md PRE-CREATION 

Diff: https://reviews.apache.org/r/46282/diff/


Testing
---


Thanks,

Chen Song