[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661417#comment-16661417 ] Hudson commented on MAPREDUCE-4669: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15299 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15299/]) MAPREDUCE-4669. MRAM web UI does not work with HTTPS. (Contributed by (haibochen: rev 823bb5dda88b2a9a1374c2a52af8d0e9b6bb801a) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRWebAppUtil.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebApp.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * (edit) hadoop-project/pom.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Fix For: 3.3.0 > > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch, > MAPREDUCE-4669.003.patch, MAPREDUCE-4669.004.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661391#comment-16661391 ] Haibo Chen commented on MAPREDUCE-4669: --- +1 on the latest patch. Checking it in shortly. > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch, > MAPREDUCE-4669.003.patch, MAPREDUCE-4669.004.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[
https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657566#comment-16657566
]
Hadoop QA commented on MAPREDUCE-4669:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 31s{color} | {color:orange} root: The patch generated 4 new + 662 unchanged
- 0 fixed = 666 total (was 662) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
2s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
17s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
7s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
0s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
24s{color} | {color:green}
[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[
https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657444#comment-16657444
]
Robert Kanter commented on MAPREDUCE-4669:
--
Thanks [~haibochen] for the review.
The 004 patch:
- Rebased on latest trunk
- Renames {{withNeedsClientAuth}} to just {{needsClientAuth}}
-- [~haibochen] I went with this instead of {{withClientAuth}} to keep it
consistent with HttpServer2 and SSL, while still making it more normal sounding
- Now only adds the truststore if {{needsClientAuth}} is {{true}}
- Updated {{TestAMWebApp#testMRWebAppSSLEnabledWithClientAuth}} to also try a
wrong client cert
-- I kept this as part of the same test because that way it shows that a client
with the right cert works, while with the wrong cert does not
- Minor refactoring in {{TestAMWebApp}}
> MRAM web UI does not work with HTTPS
>
>
> Key: MAPREDUCE-4669
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.0.3-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Robert Kanter
>Priority: Major
> Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch,
> MAPREDUCE-4669.003.patch, MAPREDUCE-4669.004.patch
>
>
> With Kerberos enable, the MRAM runs as the user that submitted the job, thus
> the MRAM process cannot read the cluster keystore files to get the
> certificates to start its HttpServer using HTTPS.
> We need to decouple the keystore used by RM/NM/NN/DN (which are cluster
> provided) from the keystore used by AMs (which ought to be user provided).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657337#comment-16657337 ] Haibo Chen commented on MAPREDUCE-4669: --- Thanks [~rkanter] for the patch! I have a few comments/questions. 1) Can we rename withNeedsClientAuth() to withClientAuth(boolean)? 2) In WebApps, I think we can skip the truststore part if clientAuth is false. 3) Can we add another unit test that queries the AM web server with a un-trusted certificate when client authorization is on? > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch, > MAPREDUCE-4669.003.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[
https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652847#comment-16652847
]
Hadoop QA commented on MAPREDUCE-4669:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 55s{color} | {color:orange} root: The patch generated 4 new + 662 unchanged
- 0 fixed = 666 total (was 662) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
25s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
15s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
56s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
25s{color} | {color:green}
[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652670#comment-16652670 ] Robert Kanter commented on MAPREDUCE-4669: -- The 003 patch: - Rebased on latest trunk (including YARN-8448) - Added configs to mapred-default > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch, > MAPREDUCE-4669.003.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647088#comment-16647088 ] Robert Kanter commented on MAPREDUCE-4669: -- The 002 patch: * Rebased on the latest version of YARN-8448 (the 008 patch) * Renamed OFF, OPTIONAL, REQUIRED to NONE, LENIENT, and STRICT as per YARN-8448 > MRAM web UI does not work with HTTPS > > > Key: MAPREDUCE-4669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Robert Kanter >Priority: Major > Attachments: MAPREDUCE-4669.001.patch, MAPREDUCE-4669.002.patch > > > With Kerberos enable, the MRAM runs as the user that submitted the job, thus > the MRAM process cannot read the cluster keystore files to get the > certificates to start its HttpServer using HTTPS. > We need to decouple the keystore used by RM/NM/NN/DN (which are cluster > provided) from the keystore used by AMs (which ought to be user provided). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-4669) MRAM web UI does not work with HTTPS
[
https://issues.apache.org/jira/browse/MAPREDUCE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556366#comment-16556366
]
Robert Kanter commented on MAPREDUCE-4669:
--
See [this
comment|https://issues.apache.org/jira/browse/YARN-8448?focusedCommentId=16556364=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16556364]
in YARN-8448 and the design doc in YARN-6586 for more background on the patch.
This patch (MAPREDUCE-4669.001.patch) contains the MR changes that rely on
YARN-8448.001.patch.
Some notes on the patch:
- The {{yarn.app.mapreduce.am.webapp.https.enabled}} property controls if the
MR AM should try to use the Yarn-provided keystore (when set to {{true}}); this
will also cause it to provide an HTTPS tracking URL to the RM. It defaults to
{{false}}.
- The {{yarn.app.mapreduce.am.webapp.https.client.auth}} property controls if
the MR AM should require client authentication (when set to {{true}}). It
defaults to {{false}}. In this case, the MR AM is the server and the RM is the
client, so this requires that the RM present its certificate to the AM when it
connects to the AM - the AM can then verify this certificate with the
Yarn-provided truststore.
> MRAM web UI does not work with HTTPS
>
>
> Key: MAPREDUCE-4669
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4669
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.0.3-alpha
>Reporter: Alejandro Abdelnur
>Assignee: Robert Kanter
>Priority: Major
> Attachments: MAPREDUCE-4669.001.patch
>
>
> With Kerberos enable, the MRAM runs as the user that submitted the job, thus
> the MRAM process cannot read the cluster keystore files to get the
> certificates to start its HttpServer using HTTPS.
> We need to decouple the keystore used by RM/NM/NN/DN (which are cluster
> provided) from the keystore used by AMs (which ought to be user provided).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
