[jira] [Commented] (YARN-5089) Improve "yarn log" command-line "logFiles" option to support regex
[
https://issues.apache.org/jira/browse/YARN-5089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295364#comment-15295364
]
Hadoop QA commented on YARN-5089:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 53s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 34s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 25 new +
137 unchanged - 4 fixed = 162 total (was 141) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 45s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 54s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 2 new + 592 unchanged - 0 fixed = 594 total (was 592) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 5s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 22s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 52s
{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 6s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 111m 20s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| | Found reliance on default encoding in
org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices$1.write(OutputStream):in
org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices$1.write(OutputStream):
String.getBytes() At
[jira] [Commented] (YARN-5089) Improve "yarn log" command-line "logFiles" option to support regex
[ https://issues.apache.org/jira/browse/YARN-5089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295344#comment-15295344 ] Xuan Gong commented on YARN-5089: - Upload a new patch which does several re-factory work and remove several useless code. > Improve "yarn log" command-line "logFiles" option to support regex > --- > > Key: YARN-5089 > URL: https://issues.apache.org/jira/browse/YARN-5089 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5089.1.patch, YARN-5089.2.patch > > > Right now, we only support matching the exact file name. We need to support > Java regex as well -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5089) Improve "yarn log" command-line "logFiles" option to support regex
[ https://issues.apache.org/jira/browse/YARN-5089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5089: Attachment: YARN-5089.2.patch > Improve "yarn log" command-line "logFiles" option to support regex > --- > > Key: YARN-5089 > URL: https://issues.apache.org/jira/browse/YARN-5089 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-5089.1.patch, YARN-5089.2.patch > > > Right now, we only support matching the exact file name. We need to support > Java regex as well -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295281#comment-15295281
]
Hadoop QA commented on YARN-5109:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 31s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 13s
{color} | {color:red}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/target/findbugsXml.xml)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 11s
{color} | {color:red}
patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/target/findbugsXml.xml)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 39s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed
with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 0s
{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the
patch passed with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 47s
{color} | {color:green}
[jira] [Updated] (YARN-5124) Modify AMRMClient to set the ExecutionType in the ResourceRequest
[
https://issues.apache.org/jira/browse/YARN-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5124:
--
Summary: Modify AMRMClient to set the ExecutionType in the ResourceRequest
(was: Modify AMRMClient to actually send the ExecutionType)
> Modify AMRMClient to set the ExecutionType in the ResourceRequest
> -
>
> Key: YARN-5124
> URL: https://issues.apache.org/jira/browse/YARN-5124
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>
> Currently the {{ContainerRequest}} allows the AM to set the {{ExecutionType}}
> in the AMRMClient, but it is not being set in the actual {{ResourceRequest}}
> that is sent to the RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5124) Modify AMRMClient to actually send the ExecutionType
Arun Suresh created YARN-5124:
-
Summary: Modify AMRMClient to actually send the ExecutionType
Key: YARN-5124
URL: https://issues.apache.org/jira/browse/YARN-5124
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Arun Suresh
Assignee: Arun Suresh
Currently the {{ContainerRequest}} allows the AM to set the {{ExecutionType}}
in the AMRMClient, but it is not being set in the actual {{ResourceRequest}}
that is sent to the RM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5109:
---
Attachment: YARN-5109-YARN-2928.003.patch
> timestamps are stored unencoded causing parse errors
>
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5109-YARN-2928.003.patch,
> YARN-5109-YARN-2928.01.patch, YARN-5109-YARN-2928.02.patch,
> YARN-5109-YARN-2928.03.patch
>
>
> When we store timestamps (for example as part of the row key or part of the
> column name for an event), the bytes are used as is without any encoding. If
> the byte value happens to contain a separator character we use (e.g. "!" or
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
> incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event)
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5109:
---
Attachment: (was: YARN-5109-YARN-2928.003.patch)
> timestamps are stored unencoded causing parse errors
>
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5109-YARN-2928.003.patch,
> YARN-5109-YARN-2928.01.patch, YARN-5109-YARN-2928.02.patch,
> YARN-5109-YARN-2928.03.patch
>
>
> When we store timestamps (for example as part of the row key or part of the
> column name for an event), the bytes are used as is without any encoding. If
> the byte value happens to contain a separator character we use (e.g. "!" or
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
> incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event)
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295257#comment-15295257
]
Hadoop QA commented on YARN-5109:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 39s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 13s
{color} | {color:red}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/target/findbugsXml.xml)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 25s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: patch
generated 8 new + 2 unchanged - 0 fixed = 10 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 12s
{color} | {color:red}
patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/target/findbugsXml.xml)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 40s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed
with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 1s
{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the
patch passed with JDK v1.8.0_91. {color} |
|
[jira] [Commented] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295242#comment-15295242
]
Varun Saxena commented on YARN-5109:
Consistently getting build failures on Jenkins with following error
{noformat}
ERROR: a previous rebase failed. Aborting it.
HEAD is now at d491ef0 YARN-3367. Replace starting a separate thread for post
entity with event loop in TimelineClient (Naganarasimha G R via sjlee)
Switched to branch 'trunk'
Your branch is up-to-date with 'origin/trunk'.
Current branch trunk is up to date.
Switched to branch 'YARN-2928'
Your branch and 'origin/YARN-2928' have diverged,
and have 85 and 782 different commits each, respectively.
(use "git pull" to merge the remote branch into yours)
First, rewinding head to replay your work on top of it...
Applying: YARN-3063. Bootstrapping TimelineServer next generation module.
Contributed by Zhijie Shen.
Using index info to reconstruct a base tree...
M hadoop-project/pom.xml
A hadoop-yarn-project/CHANGES.txt
M hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/pom.xml
:47: trailing whitespace.
Trunk - Unreleased
warning: 1 line adds whitespace errors.
Falling back to patching base and 3-way merge...
Auto-merging hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/pom.xml
CONFLICT (content): Merge conflict in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/pom.xml
Auto-merging
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/pom.xml
CONFLICT (add/add): Merge conflict in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/pom.xml
CONFLICT (modify/delete): hadoop-yarn-project/CHANGES.txt deleted in HEAD and
modified in YARN-3063. Bootstrapping TimelineServer next generation module.
Contributed by Zhijie Shen.. Version YARN-3063. Bootstrapping TimelineServer
next generation module. Contributed by Zhijie Shen. of
hadoop-yarn-project/CHANGES.txt left in tree.
Auto-merging hadoop-project/pom.xml
CONFLICT (content): Merge conflict in hadoop-project/pom.xml
Failed to merge in the changes.
Patch failed at 0001 YARN-3063. Bootstrapping TimelineServer next generation
module. Contributed by Zhijie Shen.
The copy of the patch that failed is found in:
/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/.git/rebase-apply/patch
When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".
ERROR: git pull is failing
{noformat}
Submitted the JIRA manually twice, together so that another build can start
before first submission fails. Then it works. It seems workspace on one of the
machines has a problem.
> timestamps are stored unencoded causing parse errors
>
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5109-YARN-2928.003.patch,
> YARN-5109-YARN-2928.01.patch, YARN-5109-YARN-2928.02.patch,
> YARN-5109-YARN-2928.03.patch
>
>
> When we store timestamps (for example as part of the row key or part of the
> column name for an event), the bytes are used as is without any encoding. If
> the byte value happens to contain a separator character we use (e.g. "!" or
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
> incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event)
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5095) flow activities and flow runs are populated with wrong timestamp when RM restarts w/ recovery enabled
[
https://issues.apache.org/jira/browse/YARN-5095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295238#comment-15295238
]
Hadoop QA commented on YARN-5095:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 23s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 6s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 4s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 4s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 11s
{color} | {color:red} root: patch generated 1 new + 153 unchanged - 1 fixed =
154 total (was 154) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 29m 21s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hadoop-archive-logs in the patch passed with JDK
v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 44s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 34s
{color} | {color:green} hadoop-archive-logs in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black}
[jira] [Commented] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295203#comment-15295203
]
Varun Saxena commented on YARN-5109:
Updating a patch trying to fix checkstyle issues.
> timestamps are stored unencoded causing parse errors
>
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5109-YARN-2928.003.patch,
> YARN-5109-YARN-2928.01.patch, YARN-5109-YARN-2928.02.patch,
> YARN-5109-YARN-2928.03.patch
>
>
> When we store timestamps (for example as part of the row key or part of the
> column name for an event), the bytes are used as is without any encoding. If
> the byte value happens to contain a separator character we use (e.g. "!" or
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
> incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event)
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5109:
---
Attachment: YARN-5109-YARN-2928.003.patch
> timestamps are stored unencoded causing parse errors
>
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5109-YARN-2928.003.patch,
> YARN-5109-YARN-2928.01.patch, YARN-5109-YARN-2928.02.patch,
> YARN-5109-YARN-2928.03.patch
>
>
> When we store timestamps (for example as part of the row key or part of the
> column name for an event), the bytes are used as is without any encoding. If
> the byte value happens to contain a separator character we use (e.g. "!" or
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
> incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event)
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5115) Security risk by using CONTENT-DISPOSITION header in the container-logs web-service
[
https://issues.apache.org/jira/browse/YARN-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295199#comment-15295199
]
Hadoop QA commented on YARN-5115:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m 3s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 43s
{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 31s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805006/YARN-5115.1.patch |
| JIRA Issue | YARN-5115 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 23596f181d11 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d8c1fd1 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/11611/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
| Console output |
[jira] [Commented] (YARN-5105) entire time series is returned for YARN container system metrics (CPU and memory)
[
https://issues.apache.org/jira/browse/YARN-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295179#comment-15295179
]
Varun Saxena commented on YARN-5105:
We cannot really set the number of versions to fetch on each column qualifier /
column family. We can only set it on Scan/Get.
We do not really need to fetch anything other than latest version for each
column other than metrics.
So we can set max versions to 1 each time and set it to -1(unlimited) if we
want a time series. We can support this by indicating so in fields to retrieve.
If fields to retrieve is METRICS, it would mean single value and
METRICS_TIME_SERIES
Also, regarding query by time range, there is an issue, that we again cannot
set time range at qualifier level. It can only be set on Scan/Get which is an
issue for YARN-4455 implementation. So instead of time range we can support
limit for metric values returned in time series.
[~sjlee0], thoughts ?
> entire time series is returned for YARN container system metrics (CPU and
> memory)
> -
>
> Key: YARN-5105
> URL: https://issues.apache.org/jira/browse/YARN-5105
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
> Labels: yarn-2928-1st-milestone
>
> I see that the entire time series of the CPU and memory metrics are returned
> for the YARN containers REST query. This has a potential of bloating the
> output big time.
> {noformat}
> "metrics": [
> {
> "type": "TIME_SERIES",
> "id": "MEMORY",
> "values":
> {
> "1463518173363": 407539712,
> "1463518170347": 407539712,
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5095) flow activities and flow runs are populated with wrong timestamp when RM restarts w/ recovery enabled
[
https://issues.apache.org/jira/browse/YARN-5095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295175#comment-15295175
]
Varun Saxena commented on YARN-5095:
Modified RMAppImpl constructor and set startTime from start time retrieved from
state store.
> flow activities and flow runs are populated with wrong timestamp when RM
> restarts w/ recovery enabled
> -
>
> Key: YARN-5095
> URL: https://issues.apache.org/jira/browse/YARN-5095
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5095-YARN-2928.01.patch
>
>
> I have the RM recovery enabled. I see that upon restart the RM populates
> records into flow activity and flow runs but with *wrong* timestamps. What I
> mean by the timestamp is the part of the row key:
> - flow activity: row created with the day of the RM restart
> - flow run: row created with the RM start time as the "run id"
> The following illustrates an example flow run:
> {noformat}
> metrics: [ ],
> events: [ ],
> id: "sjlee@Sleep job/1463433569917",
> type: "YARN_FLOW_RUN",
> createdtime: 1463422860987,
> info: {
> UID: "yarn_cluster!sjlee!Sleep job!1463433569917",
> SYSTEM_INFO_FLOW_RUN_ID: 1463433569917,
> SYSTEM_INFO_FLOW_NAME: "Sleep job",
> SYSTEM_INFO_FLOW_RUN_END_TIME: 1463422865033,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> {noformat}
> The created time and the end time are correct (i.e. original time), whereas
> the timestamp in the row key (= run id: 1463433569917) is actually later than
> the end time and coincides with the RM restart.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5095) flow activities and flow runs are populated with wrong timestamp when RM restarts w/ recovery enabled
[
https://issues.apache.org/jira/browse/YARN-5095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5095:
---
Attachment: YARN-5095-YARN-2928.01.patch
> flow activities and flow runs are populated with wrong timestamp when RM
> restarts w/ recovery enabled
> -
>
> Key: YARN-5095
> URL: https://issues.apache.org/jira/browse/YARN-5095
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5095-YARN-2928.01.patch
>
>
> I have the RM recovery enabled. I see that upon restart the RM populates
> records into flow activity and flow runs but with *wrong* timestamps. What I
> mean by the timestamp is the part of the row key:
> - flow activity: row created with the day of the RM restart
> - flow run: row created with the RM start time as the "run id"
> The following illustrates an example flow run:
> {noformat}
> metrics: [ ],
> events: [ ],
> id: "sjlee@Sleep job/1463433569917",
> type: "YARN_FLOW_RUN",
> createdtime: 1463422860987,
> info: {
> UID: "yarn_cluster!sjlee!Sleep job!1463433569917",
> SYSTEM_INFO_FLOW_RUN_ID: 1463433569917,
> SYSTEM_INFO_FLOW_NAME: "Sleep job",
> SYSTEM_INFO_FLOW_RUN_END_TIME: 1463422865033,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> {noformat}
> The created time and the end time are correct (i.e. original time), whereas
> the timestamp in the row key (= run id: 1463433569917) is actually later than
> the end time and coincides with the RM restart.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5121) fix some container-executor portability issues
[
https://issues.apache.org/jira/browse/YARN-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295171#comment-15295171
]
Hadoop QA commented on YARN-5121:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
3s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m 30s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 33s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 39s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 124m 14s
{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 181m 33s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805475/YARN-5121.00.patch |
| JIRA Issue | YARN-5121 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml cc |
| uname | Linux bda5ec503747 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 500e946 |
| Default Java | 1.8.0_91 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/11609/artifact/patchprocess/patch-unit-root.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/11609/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/11609/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
. U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/11609/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> fix some container-executor portability issues
> --
>
> Key: YARN-5121
>
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295172#comment-15295172
]
Larry McCay commented on YARN-4006:
---
I would say that whether certain deployments decide to use Knox or not for
proxying UIs shouldn't really impact whether custom authentication handlers are
available in ATS. I think that the full pattern of configured authentication
handlers should be present anywhere that the configured values will affect a
given endpoint. In this respect the current implementation within ATS is broken.
My original thoughts about Knox weren't targeting the UIs but instead the
external non-browser clients that could go through Knox for API calls. I don't
think that tricking an AltKerberos impl into treating a useragent as a browser
is a great practice. While it might work for some AltKerberos impls, it could
easily not work for others without additional changes to the API request. Using
something like Knox as an adapter to the kerberos expectations is more or less
guaranteed to work with AltKerberos for non-browser agents.
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5123) SQL based RM state store
[ https://issues.apache.org/jira/browse/YARN-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295166#comment-15295166 ] Lavkesh Lahngir commented on YARN-5123: --- RM_DB_TYPE -> it's the name of jdbc driver, in our case it would be "postgresql". RM_DB_HOST -> DB host name. RM_DB_DBNAME -> Database name. RM_DB_USERNAME -> Database user. RM_DB_PASSWORD -> Database password. RM_DB_PASSWORD_FILE -> Password file from which the state store will read the password. In our setup we have password file with special permission. If the password file is not provided, the value of RM_DB_PASSWORD will be used. RM_DB_RETRIES -> number of retries before any DB operation is given up. If these retries are exhausted RM will go on stand by. DEFAULT_RM_DB_RETRIES -> Default retries RM_DB_RETRIES_INTERVAL -> Fixed interval between retries DEFAULT_RM_DB_RETRIES_INTERVAL -> Default interval for retries in seconds. RM_DB_VERIFICATION_TIMEOUT -> A timeout interval in milliseconds for the verification thread which keeps checking the status of active RM. DEFAULT_RM_DB_VERIFICATION_TIMEOUT -> Default verification timeout > SQL based RM state store > > > Key: YARN-5123 > URL: https://issues.apache.org/jira/browse/YARN-5123 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.6.0 >Reporter: Lavkesh Lahngir > Attachments: sqlstatestore.patch > > > In our setup, zookeeper based RM state store didn't work. We ended up > implementing our own SQL based state store. Here is a patch, if anybody else > wants to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5123) SQL based RM state store
[ https://issues.apache.org/jira/browse/YARN-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295166#comment-15295166 ] Lavkesh Lahngir edited comment on YARN-5123 at 5/21/16 6:02 PM: Config definitions. RM_DB_TYPE -> it's the name of jdbc driver, in our case it would be "postgresql". RM_DB_HOST -> DB host name. RM_DB_DBNAME -> Database name. RM_DB_USERNAME -> Database user. RM_DB_PASSWORD -> Database password. RM_DB_PASSWORD_FILE -> Password file from which the state store will read the password. In our setup we have password file with special permission. If the password file is not provided, the value of RM_DB_PASSWORD will be used. RM_DB_RETRIES -> number of retries before any DB operation is given up. If these retries are exhausted RM will go on stand by. DEFAULT_RM_DB_RETRIES -> Default retries RM_DB_RETRIES_INTERVAL -> Fixed interval between retries DEFAULT_RM_DB_RETRIES_INTERVAL -> Default interval for retries in seconds. RM_DB_VERIFICATION_TIMEOUT -> A timeout interval in milliseconds for the verification thread which keeps checking the status of active RM. DEFAULT_RM_DB_VERIFICATION_TIMEOUT -> Default verification timeout was (Author: lavkesh): RM_DB_TYPE -> it's the name of jdbc driver, in our case it would be "postgresql". RM_DB_HOST -> DB host name. RM_DB_DBNAME -> Database name. RM_DB_USERNAME -> Database user. RM_DB_PASSWORD -> Database password. RM_DB_PASSWORD_FILE -> Password file from which the state store will read the password. In our setup we have password file with special permission. If the password file is not provided, the value of RM_DB_PASSWORD will be used. RM_DB_RETRIES -> number of retries before any DB operation is given up. If these retries are exhausted RM will go on stand by. DEFAULT_RM_DB_RETRIES -> Default retries RM_DB_RETRIES_INTERVAL -> Fixed interval between retries DEFAULT_RM_DB_RETRIES_INTERVAL -> Default interval for retries in seconds. RM_DB_VERIFICATION_TIMEOUT -> A timeout interval in milliseconds for the verification thread which keeps checking the status of active RM. DEFAULT_RM_DB_VERIFICATION_TIMEOUT -> Default verification timeout > SQL based RM state store > > > Key: YARN-5123 > URL: https://issues.apache.org/jira/browse/YARN-5123 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.6.0 >Reporter: Lavkesh Lahngir > Attachments: sqlstatestore.patch > > > In our setup, zookeeper based RM state store didn't work. We ended up > implementing our own SQL based state store. Here is a patch, if anybody else > wants to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5121) fix some container-executor portability issues
[ https://issues.apache.org/jira/browse/YARN-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295160#comment-15295160 ] Allen Wittenauer commented on YARN-5121: Unit test failures are known. The increase in errors is directly correlated with the build no longer prematurely exiting. :) > fix some container-executor portability issues > -- > > Key: YARN-5121 > URL: https://issues.apache.org/jira/browse/YARN-5121 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: YARN-5121.00.patch > > > container-executor has some issues that are preventing it from even compiling > on the OS X jenkins instance. Let's fix those. While we're there, let's > also try to take care of some of the other portability problems that have > crept in over the years, since it used to work great on Solaris but now > doesn't. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5123) SQL based RM state store
[ https://issues.apache.org/jira/browse/YARN-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lavkesh Lahngir updated YARN-5123: -- Description: In our setup, zookeeper based RM state store didn't work. We ended up implementing our own SQL based state store. Here is a patch, if anybody else wants to use it. (was: In our setup, zookeeper based RM state store didn't work. We ended up implementing our own SQL based state store. This is a patch, if anybody else want to use it. ) > SQL based RM state store > > > Key: YARN-5123 > URL: https://issues.apache.org/jira/browse/YARN-5123 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.6.0 >Reporter: Lavkesh Lahngir > Attachments: sqlstatestore.patch > > > In our setup, zookeeper based RM state store didn't work. We ended up > implementing our own SQL based state store. Here is a patch, if anybody else > wants to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295159#comment-15295159
]
Hadoop QA commented on YARN-5109:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 57s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
12s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} YARN-2928 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 13s
{color} | {color:red}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/target/findbugsXml.xml)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: patch
generated 7 new + 2 unchanged - 0 fixed = 9 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 11s
{color} | {color:red}
patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/target/findbugsXml.xml)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 41s
{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed
with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 27s
{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the
patch passed with JDK v1.8.0_91. {color} |
|
[jira] [Commented] (YARN-5121) fix some container-executor portability issues
[
https://issues.apache.org/jira/browse/YARN-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295158#comment-15295158
]
Hadoop QA commented on YARN-5121:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 00s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
00s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 49s
{color} | {color:red} root in trunk failed. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 46s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 7m 46s {color} |
{color:red} root generated 11 new + 17 unchanged - 9 fixed = 28 total (was 26)
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 46s {color}
| {color:red} root generated 526 new + 172 unchanged - 0 fixed = 698 total (was
172) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 24s {color}
| {color:red} root in the patch failed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.TestSymlinkLocalFSFileContext |
| | hadoop.fs.TestSymlinkLocalFSFileSystem |
| | hadoop.net.unix.TestDomainSocket |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805475/YARN-5121.00.patch |
| JIRA Issue | YARN-5121 |
| Optional Tests | compile javac mvninstall unit cc |
| uname | Darwin Gavins-Mac-mini.local 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 |
| Build tool | maven |
| Personality |
/Users/jenkins/jenkins-home/workspace/Precommit-HADOOP-OSX/patchprocess/apache-yetus-21ed107/precommit/personality/hadoop.sh
|
| git revision | trunk / d8c1fd1 |
| Default Java | 1.8.0_74 |
| compile |
https://builds.apache.org/job/Precommit-HADOOP-OSX/18/artifact/patchprocess/branch-compile-root.txt
|
| cc |
https://builds.apache.org/job/Precommit-HADOOP-OSX/18/artifact/patchprocess/diff-compile-cc-root.txt
|
| javac |
https://builds.apache.org/job/Precommit-HADOOP-OSX/18/artifact/patchprocess/diff-compile-javac-root.txt
|
| unit |
https://builds.apache.org/job/Precommit-HADOOP-OSX/18/artifact/patchprocess/patch-unit-root.txt
|
| unit test logs |
https://builds.apache.org/job/Precommit-HADOOP-OSX/18/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/Precommit-HADOOP-OSX/18/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
. U: . |
| Console output |
https://builds.apache.org/job/Precommit-HADOOP-OSX/18/console |
| Powered by | Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> fix some container-executor portability issues
> --
>
> Key: YARN-5121
> URL: https://issues.apache.org/jira/browse/YARN-5121
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: YARN-5121.00.patch
>
>
> container-executor has some issues that are preventing it from even compiling
> on the OS X jenkins instance. Let's fix those. While we're there, let's
> also try to take care of some of the other portability problems that have
> crept in over the years, since it used to work great on Solaris but now
> doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295156#comment-15295156
]
Larry McCay commented on YARN-4006:
---
There are some outstanding issues with UIs colocated in the same topology that
we are ironing out for a 0.9.1 release targeting a release date of ~6/10th.
Let's move that conversation over to the Knox dev@ list as not to clutter
Hadoop conversation with those details. We should try and make sure what is
needed for your usecases is available in 0.9.1 or at least a roadmap that
includes it beyond that.
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5123) SQL based RM state store
[ https://issues.apache.org/jira/browse/YARN-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295155#comment-15295155 ] Lavkesh Lahngir commented on YARN-5123: --- Few comments: 1. We applied this in hdp2.2.4 which is essentially hadoop-2.6.0 or similar. 2. Before applying this to higher versions, we need to check, if there are any API changes. 3. For this patch to work, postgres jar should be present in the RM classpath. 4. See YarnConfiguration for new configurations. > SQL based RM state store > > > Key: YARN-5123 > URL: https://issues.apache.org/jira/browse/YARN-5123 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.6.0 >Reporter: Lavkesh Lahngir > Attachments: sqlstatestore.patch > > > In our setup, zookeeper based RM state store didn't work. We ended up > implementing our own SQL based state store. This is a patch, if anybody else > want to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5113) Refactoring and other clean-up for distributed scheduling
[
https://issues.apache.org/jira/browse/YARN-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295153#comment-15295153
]
Arun Suresh commented on YARN-5113:
---
Also you maybe want to update the javadoc plugin in
{{hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml}}
like so :
{noformat}
org.apache.maven.plugins
maven-javadoc-plugin
org.apache.hadoop.yarn.proto
{noformat}
> Refactoring and other clean-up for distributed scheduling
> -
>
> Key: YARN-5113
> URL: https://issues.apache.org/jira/browse/YARN-5113
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5113.001.patch, YARN-5113.002.patch,
> YARN-5113.003.patch
>
>
> This JIRA focuses on the refactoring of classes related to Distributed
> Scheduling
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5123) SQL based RM state store
[ https://issues.apache.org/jira/browse/YARN-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lavkesh Lahngir updated YARN-5123: -- Attachment: sqlstatestore.patch > SQL based RM state store > > > Key: YARN-5123 > URL: https://issues.apache.org/jira/browse/YARN-5123 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.6.0 >Reporter: Lavkesh Lahngir > Attachments: sqlstatestore.patch > > > In our setup, zookeeper based RM state store didn't work. We ended up > implementing our own SQL based state store. This is a patch, if anybody else > want to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5123) SQL based RM state store
Lavkesh Lahngir created YARN-5123: - Summary: SQL based RM state store Key: YARN-5123 URL: https://issues.apache.org/jira/browse/YARN-5123 Project: Hadoop YARN Issue Type: New Feature Affects Versions: 2.6.0 Reporter: Lavkesh Lahngir In our setup, zookeeper based RM state store didn't work. We ended up implementing our own SQL based state store. This is a patch, if anybody else want to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[ https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295150#comment-15295150 ] Greg Senia commented on YARN-4006: -- I would definitely agree Knox is the long term answer. At my new employer we are rolling out Knox for users interacting with Hadoop outside of the cluster network.. I've relayed this back to my remaining contacts at my previous employer that it is best to ditch AltAuth code all together. The whole reason for the AltAuth handler use was born out of Data Scientists and Hadoop Endusers being upset that they couldn't view their jobs logs while things were running. This was because we decided to protect the HTTP/S UI's with kerberos as folks started to leak PII/PHI in custom Mapreduce/Yarn job logs in debug mode and kerberizing the HTTP Endpoints was a way to contain it. I think the things blocking Knox rollout in both my current and previous employer are the status of the remaining JIRAs around the UIs I know when reading the HWX docy supposedly all the UI's are supported [~vinodkv] and [~lmccay] do you know if these are all supported in HA mode? If so this JIRA can be closed out in my book as it really is dead ended unless folks don't want to use Knox.. Knox Overview What Knox Does How Knox Works Hortonworks Focus for Knox Gateway Recent Progress in Knox Gateway Knox Tutorials Knox in the Blog Webinars & Presentations Knox Community Apache Project Page WHAT KNOX DOES With YARN as its architectural center, Apache Hadoop continues to attract new engines to run within the data platform, as organizations want to efficiently store their data in a single repository and interact with it for batch, interactive and real-time streaming use cases. More and more independent software vendors (ISVs) are developing applications to run in Hadoop via YARN. This increases the number of users and processing engines that operate simultaneously across a Hadoop cluster, on the same data, at the same time. The Apache Knox Gateway (“Knox”) provides perimeter security so that the enterprise can confidently extend Hadoop access to more of those new users while also maintaining compliance with enterprise security policies. Knox also simplifies Hadoop security for users who access the cluster data and execute jobs. It integrates with prevalent identity management and SSO systems and allows identities from those enterprise systems to be used for seamless, secure access to Hadoop clusters. Knox provides perimeter security for Hadoop clusters, with these advantages: Advantage Description Simplified access Entend Hadoop’s REST/HTTP services by encapsulating Kerberos within the cluster Enhanced security Expose Hadoop’s REST/HTTP services without revealing network details, with SSL provided out of box Centralized control Centrally enforce REST API security and route requests to multiple Hadoop clusters Enterprise integration Support LDAP and Active Directory The following Apache Hadoop services have integrations with the Knox Gateway: http://hortonworks.com/apache/knox-gateway/ Supported Apache Hadoop Services Ambari WebHDFS (HDFS) Templeton (HCatalog) Stargate (HBase) Oozie Hive/JDBC Yarn RM Storm Supported Apache Hadoop UIs Name Node UI Job History UI Oozie UI HBase UI Yarn UI Spark UI Ambari UI Ranger Admin Console Not Resolved: Having Knox Supporting Hadoop/Yarn/HBase/Hive High Availability - https://issues.apache.org/jira/browse/KNOX-567 Support Zeppelin UI through Knox - https://issues.apache.org/jira/browse/KNOX-710 NameNode UI through Knox has various tabs not working - https://issues.apache.org/jira/browse/KNOX-626 Views in Ambari UI don't render when proxied by the AMBARIUI service - https://issues.apache.org/jira/browse/KNOX-705 HBase Master UI through Knox is missing JS and CSS resources - https://issues.apache.org/jira/browse/KNOX-627 Oozie Web UI doesn't render when proxied using Knox - https://issues.apache.org/jira/browse/KNOX-628 Fixed: Proxy support for Ranger UI - https://issues.apache.org/jira/browse/KNOX-668 Knox support for HiveServer2 HA - https://issues.apache.org/jira/browse/KNOX-570 Proxy support for Ambari UI - https://issues.apache.org/jira/browse/KNOX-673 Provide a template topology file for UI proxy services - https://issues.apache.org/jira/browse/KNOX-625 > YARN ATS Alternate Kerberos HTTP Authentication Changes > --- > > Key: YARN-4006 > URL: https://issues.apache.org/jira/browse/YARN-4006 > Project: Hadoop YARN > Issue Type: Improvement > Components: security, timelineserver >Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2 >Reporter: Greg Senia >Assignee: Greg Senia >Priority: Blocker > Attachments: YARN-4006-branch-trunk.patch, >
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295142#comment-15295142
]
Larry McCay commented on YARN-4006:
---
Okay...
I think that we can boil this down to just wanting to add custom authentication
handler support then.
Which is perfectly reasonable.
However, pulling AltKerberos into the discussion without details is adding
confusion/ambiguity that makes review impossible.
* I would like to see a common builder class created to add handlers to the
FilterConfig based on this hadoop.http.authentication.type configuration
pattern.
* I would also suggest that something other than AltKerberos be considered and
non-browser clients setting useragent strings to trick them into acting like
browser agents. This could be configured exactly where you need them with via
hadoop.http.authentication.type or specifically for ATS with the PREFIX.
* Additionally, external clients that require holes punched should seriously
consider using Apache Knox. These are exactly the sort of usecases targeted by
Knox and will work with kerberos internally and whatever authentication
mechanism that you want at the gateway. Whether we need to add ATS API support
is a separate question but service definitions are pretty easy in Knox these
days.
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295138#comment-15295138
]
Greg Senia commented on YARN-4006:
--
[~vinodkv], [~aw] and [~lmccay], the goal of my original patch was to allow the
ATS UI component operate in a similar fashion to the Mapreduce History Server
so users were able to view their YARN job history and such as my previous
employer disabled kerberos in web browsers why I couldn't tell you it was a
fight just to allow them to kerberize the cluster using Active Directory and
allowing service principal creation in a nonstandard AD child forest.. My
former employer is still using this patch with a recompiled version of HDP 2.4
so users can view the job logs and such.
-Greg
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295134#comment-15295134
]
Allen Wittenauer commented on YARN-4006:
bq. Is the usecase description part of a proprietary solution that we don't
want to expose for some reason?
Yes.
bq. From what I can see, if a custom authentication handler (forget AltKerberos
for a moment) is configured than the TimelineAuthenticationFilterInitializer
will not add it or the Pseudo or Kerberos handlers to the FilterConfig.
... which would break folks who are using, e.g., SAML. There are definitely
handlers out in the wild for all sorts of weird things.
bq. If we want to be able to use custom authentication handlers here than we
need the change proposed by this patch.
I'm left with the question of "why should timeline server be handled
differently than every other serving process in Hadoop?" Even the 2NN allows
for a custom authentication handler. It basically makes the TS seem like a
significantly lower quality service if it can only use two auth handlers when
it only supports REST as a connection point!
bq. What are the different client types for this endpoint and how (if at all)
does a proxy come into play for each?
Automated processes that are outside the network to build reporting and what
not. They will not have access to a kdc to use for authentication. A network
hole will be punched that will allow for IP redirection. Also note that they
can add custom user-agent strings...
bq. What would be examples of the non-kerberos side of the AltKerberos
JWT is definitely one of them. The other is an OTP string in a custom HTTP
header.
FWIW: I completely agree that this whole mess would go away if either
AltKerberos was smarter and/or there was a generic AuthenticationHandler hook
that every daemon used. But Hadoop being Hadoop, the latter was never an
option given how often individual teams working on individual components seem
to be stuck permanently in NIH mode. "Oh, the NN needs to do this little bit
different for the 2NN" is what started the downward spiral here. :(
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail:
[jira] [Commented] (YARN-5087) Expose API to allow AM to request for change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295133#comment-15295133 ] Arun Suresh commented on YARN-5087: --- [~vinodkv], [~leftnoteasy], [~jianhe], [~mding] We were thinking of reusing / piggy-backing some of the stuff done in YARN-1197. Essentially, we would like the AM to issue a request to the RM update the container type in an allocate request and we would like to be notified of when that request is satisfied in the response. We were wondering if it is possible to combine the increase resources, decrease resources and change container type requests into a single *UpdateContainerRequest* type. The RM can then decide what type of change request it is. Also, in a similar vein, we can merge the Increased/Decreased Containers and the Updated containerType response into the same list in the AllocateResponse. To be honest, I feel even that is superfluous, since we already have a List allocatedContainers list and the ContainerToken containers all the information about a container (Container Resource and ExecutionType). The AM can then figure out if its request has been satisfied. adding [~subru], [~curino], [~kkaranasos] in the conversation. > Expose API to allow AM to request for change of container ExecutionType > --- > > Key: YARN-5087 > URL: https://issues.apache.org/jira/browse/YARN-5087 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295124#comment-15295124
]
Larry McCay commented on YARN-4006:
---
[~aw] - this is entirely dependent on the proxy and whether it can be
configured as a trusted proxy in hadoop. Knox is certainly a proxy that
authenticates via kerberos and asserts the doas identity.
That aside, I am still unclear on how AltKerberos can help that particular
usecase. Non-browsers are intended to use kerberos in AltKeberos extensions.
This may already be understood by you but I can't tease out the usecase being
addressed here without an end to end description.
Is the usecase description part of a proprietary solution that we don't want to
expose for some reason?
Not trying to be snarky here - just would like to understand the reluctance.
* From what I can see, if a custom authentication handler (forget AltKerberos
for a moment) is configured than the TimelineAuthenticationFilterInitializer
will not add it or the Pseudo or Kerberos handlers to the FilterConfig.
* If we don't care about custom authentication handlers here then we could
change the patch to default to kerberos when there is a custom authentication
handler configured. This would be assuming that any custom handler would be an
AltKerberos impl and would likely require there to be no browser useragents in
play at this endpoint or that they have SPNEGO enabled if there are.
* If we want to be able to use custom authentication handlers here than we need
the change proposed by this patch.
* If we don't have browser based useragents in play here then we don't - by
definition - need AltKerberos extensions here but by having custom
authentication handlers available you could use them but only the kerberos side
would be used.
Let's try simplified questions:
1. What are the different client types for this endpoint and how (if at all)
does a proxy come into play for each?
2. What would be examples of the non-kerberos side of the AltKerberos (can we
use JWT based cookie for example)?
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Commented] (YARN-4887) AM-RM protocol changes for identifying resource-requests explicitly
[
https://issues.apache.org/jira/browse/YARN-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295122#comment-15295122
]
Chris Nauroth commented on YARN-4887:
-
Hello [~subru]. I think the YARN build would need configuration to exclude
protobuf-generated sources, which do tend to generate a lot of Javadoc warnings
that we can't do anything about. For example, here is an exclusion from
hadoop-hdfs-project/hadoop-hdfs/pom.xml:
{code}
org.apache.maven.plugins
maven-javadoc-plugin
org.apache.hadoop.hdfs.protocol.proto
{code}
I don't see a similar exclusion in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml.
> AM-RM protocol changes for identifying resource-requests explicitly
> ---
>
> Key: YARN-4887
> URL: https://issues.apache.org/jira/browse/YARN-4887
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: applications, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-4887-v1.patch, YARN-4887-v2.patch
>
>
> YARN-4879 proposes the addition of a simple delta allocate protocol. This
> JIRA is to track the changes in AM-RM protocol to accomplish it. The crux is
> the addition of ID field in ResourceRequest and Container. The detailed
> proposal is in the parent JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295110#comment-15295110
]
Allen Wittenauer commented on YARN-4006:
Non-browsers going through proxies _cannot_ use Kerberos for authentication.
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295105#comment-15295105
]
Larry McCay commented on YARN-4006:
---
bq. a) split configurations are a pain in the ass to manage
agreed.
bq. b) timeline server is expected to be used by end users for querying. so if
your altkerberos has, say, JWT support for end user applications
Please excuse my ignorance here. When you say end user applications what
useragents would be in play: browser or non-browser. JWT is a great example. We
have a AltKerberos handler that will accept a cookie that contains a JWT token.
It will only be presented by browsers however and even if it were presented by
a non-browser the kerberos side of the AltKerberos would be engaged not the JWT
side.
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295097#comment-15295097
]
Allen Wittenauer commented on YARN-4006:
bq. if we don't really need AltKerberos here then we could set this to just be
kerberos.
Two issues:
a) split configurations are a pain in the ass to manage
b) timeline server is expected to be used by end users for querying. so if
your altkerberos has, say, JWT support for end user applications
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295093#comment-15295093
]
Larry McCay commented on YARN-4006:
---
Relative to 2.b. in my previous comment, it seems that the authentication type
can also be overridden using the PREFIX defined in the initializer. So that,
yarn.timeline-service.http-authentication.http.authentication.type can set it
to whatever you like for ATS. So, if we don't really need AltKerberos here then
we could set this to just be kerberos.
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5121) fix some container-executor portability issues
[ https://issues.apache.org/jira/browse/YARN-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295063#comment-15295063 ] Allen Wittenauer edited comment on YARN-5121 at 5/21/16 3:23 PM: - -00: * add some missing routines for OS X 10.9 * replace get_executable with something portable * lock out cgroup support on non-Linux platforms * add necessary license and RAT exclusion handling * change the optional defines so they are actually legal for clang * fix a ton of whitespace issues was (Author: aw): -00: * add some missing routines for OS X 10.9 * replace get_executable with something portable * lock out cgroup support on non-Linux platforms * add necessary license and RAT exclusion handling * change the optional defines so they are actually legal for clang > fix some container-executor portability issues > -- > > Key: YARN-5121 > URL: https://issues.apache.org/jira/browse/YARN-5121 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: YARN-5121.00.patch > > > container-executor has some issues that are preventing it from even compiling > on the OS X jenkins instance. Let's fix those. While we're there, let's > also try to take care of some of the other portability problems that have > crept in over the years, since it used to work great on Solaris but now > doesn't. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5121) fix some container-executor portability issues
[ https://issues.apache.org/jira/browse/YARN-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295063#comment-15295063 ] Allen Wittenauer edited comment on YARN-5121 at 5/21/16 3:15 PM: - -00: * add some missing routines for OS X 10.9 * replace get_executable with something portable * lock out cgroup support on non-Linux platforms * add necessary license and RAT exclusion handling * change the optional defines so they are actually legal for clang was (Author: aw): -00: * add some missing routines for OS X 10.9 * replace get_executable with something portable * lock out cgroup support on non-Linux platforms * add necessary license and RAT exclusion handling > fix some container-executor portability issues > -- > > Key: YARN-5121 > URL: https://issues.apache.org/jira/browse/YARN-5121 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: YARN-5121.00.patch > > > container-executor has some issues that are preventing it from even compiling > on the OS X jenkins instance. Let's fix those. While we're there, let's > also try to take care of some of the other portability problems that have > crept in over the years, since it used to work great on Solaris but now > doesn't. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5121) fix some container-executor portability issues
[ https://issues.apache.org/jira/browse/YARN-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-5121: --- Description: container-executor has some issues that are preventing it from even compiling on the OS X jenkins instance. Let's fix those. While we're there, let's also try to take care of some of the other portability problems that have crept in over the years, since it used to work great on Solaris but now doesn't. (was: container-executor has some issues that are preventing it from even compiling on the OS X jenkins instance. Let's fix those.) > fix some container-executor portability issues > -- > > Key: YARN-5121 > URL: https://issues.apache.org/jira/browse/YARN-5121 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: YARN-5121.00.patch > > > container-executor has some issues that are preventing it from even compiling > on the OS X jenkins instance. Let's fix those. While we're there, let's > also try to take care of some of the other portability problems that have > crept in over the years, since it used to work great on Solaris but now > doesn't. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295068#comment-15295068
]
Larry McCay commented on YARN-4006:
---
Okay - I am starting to understand where the ambiguity complaints are coming
from
Can we step back and answer a few questions:
1. AltKerberos is generally used to split the authentication mechanisms used
for browsers/UIs and non-browsers (java, curl, wget, perl, etc) in order to not
require the burden of SPNEGO being enabled for the browsers. The inability to
enable AltKerberos at this endpoint should just make browsers not be able to
use something other than SPNEGO. For instance, a cookie maybe available from
webapps that can be used to communicate the previous authentication event and
identity.
a. How does enabling AltKerberos for other endpoints break the Yarn client
- which should have a useragent of java - when it should be expecting SPNEGO -
given that it is using REST?
b. I guess the fact that the same property
(hadoop.http.authentication.type) is being for the existing integration points
as well as in ATS that the current code creates no authentication handler and
results in a NPE? If the case is that there is no UI aspect at the ATS
integration point then we don't necessarily have to support AltKerberos here
but instead just make sure that kerberos is properly instantiated there. Which
would also require a simple change here.
2. Without seeing any changes related to the need to super() up to the parent
AltKerberos class, I am having trouble understanding why that change is needed.
3. If we do indeed need to have the exact same pattern for enabling custom
authentication handlers here (in multiple places in general) then it seems that
we should actually be factoring that logic out into a common
AutheneticationHandlerBuilder class that can be plugged in anywhere.
I think if we can articulate answers to the above questions it will make the
usecase and problem much more clear.
I apologize if it is already in the comments above and I just can't get my head
around it.
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail:
[jira] [Commented] (YARN-5121) fix some container-executor portability issues
[ https://issues.apache.org/jira/browse/YARN-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295064#comment-15295064 ] Allen Wittenauer commented on YARN-5121: Ping [~cnauroth], since he and I were talking about this problem. > fix some container-executor portability issues > -- > > Key: YARN-5121 > URL: https://issues.apache.org/jira/browse/YARN-5121 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer > Attachments: YARN-5121.00.patch > > > container-executor has some issues that are preventing it from even compiling > on the OS X jenkins instance. Let's fix those. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5121) fix some container-executor portability issues
[ https://issues.apache.org/jira/browse/YARN-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-5121: --- Attachment: YARN-5121.00.patch -00: * add some missing routines for OS X 10.9 * replace get_executable with something portable * lock out cgroup support on non-Linux platforms * add necessary license and RAT exclusion handling > fix some container-executor portability issues > -- > > Key: YARN-5121 > URL: https://issues.apache.org/jira/browse/YARN-5121 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer > Attachments: YARN-5121.00.patch > > > container-executor has some issues that are preventing it from even compiling > on the OS X jenkins instance. Let's fix those. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5109:
---
Attachment: YARN-5109-YARN-2928.03.patch
> timestamps are stored unencoded causing parse errors
>
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5109-YARN-2928.01.patch,
> YARN-5109-YARN-2928.02.patch, YARN-5109-YARN-2928.03.patch
>
>
> When we store timestamps (for example as part of the row key or part of the
> column name for an event), the bytes are used as is without any encoding. If
> the byte value happens to contain a separator character we use (e.g. "!" or
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
> incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event)
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5109:
---
Attachment: (was: YARN-5109-YARN-2928.03.patch)
> timestamps are stored unencoded causing parse errors
>
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5109-YARN-2928.01.patch,
> YARN-5109-YARN-2928.02.patch
>
>
> When we store timestamps (for example as part of the row key or part of the
> column name for an event), the bytes are used as is without any encoding. If
> the byte value happens to contain a separator character we use (e.g. "!" or
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
> incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event)
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295033#comment-15295033
]
Allen Wittenauer commented on YARN-4006:
bq. Manual testing is acceptable if it is deemed that this is not testable via
unit-tests. I don't see any comments from anyone saying that.
Then I'll say it. It's pretty clear just looking at
TestTimelineAuthenticationFilterInitializer.java that it doesn't test nearly
enough because of the difficulty involved. There are *zero* Kerberos tests,
for example. Whoever committed clearly understood the difficulty invovled,
otherwise it wouldn't have made it in right? ;)
bq. we still don't completely understand the patch
The yarn client expects to be able to ask the auth handler to get a delegation
token over REST if security is enabled. The problem, however, is that if
AltKerberos is enabled, there's no way for the AltKerberos implementation to
call the DelegationToken manager because the security check is built around the
idea that there are only two auth handlers (none or Kerberos). So yarn job
submission fails because the token fetch fails on the server side because the
handler in use isn't either of those.
It's pretty obviously broken if one sets up an AltKerb handler and actually
tries to use it.
The patch here *does* require a change to AltKerberos implementations (they'll
likely need to super() up to call the token management routines), but there's
really no other choice. It could be easily argued that the AltKerberos
interface/design specs were incomplete and this forces that hole closed.
There's also the whole side-thing around timeline server not really being
deployed at a lot of sites (for various reasons) due to being optional. Hadoop
2.x has introduced significantly worse incompatibilities throughout it's
lifecycle, but at least this one is realistic and understandable.
It's worth pointing out that this is probably the first time that I'm aware of
that REST is being used instead of RPC inside Java code in Hadoop. It's not,
however, the first time that AltKerberos was completely broken because authors
of code didn't realize that there were was more than one handler. The HDFS
browser fiasco effectively killed 2.5 and 2.6 upgrades for the exact same
reason. (although, with 2.6, nothing of value was lost.) Now that both HDFS
and YARN have broken something in this space, let's hope there isn't a
threepeat.
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal
[jira] [Updated] (YARN-5109) timestamps are stored unencoded causing parse errors
[
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-5109:
---
Attachment: YARN-5109-YARN-2928.03.patch
> timestamps are stored unencoded causing parse errors
>
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5109-YARN-2928.01.patch,
> YARN-5109-YARN-2928.02.patch, YARN-5109-YARN-2928.03.patch
>
>
> When we store timestamps (for example as part of the row key or part of the
> column name for an event), the bytes are used as is without any encoding. If
> the byte value happens to contain a separator character we use (e.g. "!" or
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
> incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event)
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5093) created time shows 0 in most REST output
[
https://issues.apache.org/jira/browse/YARN-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294870#comment-15294870
]
Varun Saxena commented on YARN-5093:
[~sjlee0], kindly review
> created time shows 0 in most REST output
>
>
> Key: YARN-5093
> URL: https://issues.apache.org/jira/browse/YARN-5093
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
> Labels: yarn-2928-1st-milestone
> Attachments: YARN-5093-YARN-2928.01.patch
>
>
> When querying the REST API, I find that the created time value is returned as
> "0" for most of the output. It includes:
> - flow activity and flow runs in the flow activity page
> - apps in the application page
> - entities in the entity page
> For example, in the flow activity page,
> {noformat}
> {
> metrics: [ ],
> events: [ ],
> id: "yarn_cluster/146335680/sjlee@ds-date",
> type: "YARN_FLOW_ACTIVITY",
> createdtime: 0,
> flowruns: [
> {
> metrics: [ ],
> events: [ ],
> id: "sjlee@ds-date/1463435661428",
> type: "YARN_FLOW_RUN",
> createdtime: 0,
> info: {
> SYSTEM_INFO_FLOW_VERSION: "1",
> SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> ],
> info: {
> SYSTEM_INFO_CLUSTER: "yarn_cluster",
> UID: "yarn_cluster!sjlee!ds-date",
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_DATE: 146335680,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> {noformat}
> The only page that appears to show the proper created time value is the flow
> run page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4837) User facing aspects of 'AM blacklisting' feature need fixing
[
https://issues.apache.org/jira/browse/YARN-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294832#comment-15294832
]
Hadoop QA commented on YARN-4837:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 3s
{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 53s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 56s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 1 new +
1494 unchanged - 17 fixed = 1495 total (was 1511) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 40s
{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api
generated 0 new + 5385 unchanged - 21 fixed = 5385 total (was 5406) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 9s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 59s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294804#comment-15294804
]
Vinod Kumar Vavilapalli commented on YARN-4006:
---
Manual testing is acceptable if it is deemed that this is not testable via
unit-tests. I don't see any comments from anyone saying that.
bq. There isn't anything that is specific to AltKeberos about this patch but
such extensions should work fine.
I missed this. In this case, it is entirely possible to add tests for this.
Other comments on the patch, forgetting for a moment that we still don't
completely understand the patch given the lack of even a basic explanation
despite multiple requests
- No need to log the auth-type three times, the first one should suffice?
- Fix lines longer than 80 characters, Jenkins will complain
- Use CommonConfigurationKeyPublic constants instead of hardcoding
"hadoop.security.authentication"
- Spurious conditional statements - you can collapse the following into a
simple if-else statement
{code}
if (a || b) {
if (!a) { }
else { }
}
{code}
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4837) User facing aspects of 'AM blacklisting' feature need fixing
[ https://issues.apache.org/jira/browse/YARN-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-4837: -- Attachment: YARN-4837-20160520.1.txt Updated patch fixing the checkstyle issues and the two tests broken by the patch - TestAppSchedulingInfo & TestFSAppAttempt. > User facing aspects of 'AM blacklisting' feature need fixing > > > Key: YARN-4837 > URL: https://issues.apache.org/jira/browse/YARN-4837 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Critical > Attachments: YARN-4837-20160515.txt, YARN-4837-20160520.1.txt, > YARN-4837-20160520.txt > > > Was reviewing the user-facing aspects that we are releasing as part of 2.8.0. > Looking at the 'AM blacklisting feature', I see several things to be fixed > before we release it in 2.8.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294775#comment-15294775
]
Allen Wittenauer commented on YARN-4006:
Yes, actually, given we've personally tested the code under the exact situation
that causes yarn application submission to fail because of assumptions made by
the ATS authentication handler.
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294764#comment-15294764
]
Vinod Kumar Vavilapalli commented on YARN-4006:
---
So, we are good to commit the patch with no test-cases or explanation of what's
wrong with the current code and how the patch is fixing it?
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294757#comment-15294757
]
Allen Wittenauer commented on YARN-4006:
Actually, I take that back.
The log lines need to be cleaned up so that they don't show up at INFO level.
DEBUG is probably better.
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4837) User facing aspects of 'AM blacklisting' feature need fixing
[
https://issues.apache.org/jira/browse/YARN-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294759#comment-15294759
]
Hadoop QA commented on YARN-4837:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 53s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 11 new +
1494 unchanged - 17 fixed = 1505 total (was 1511) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 53s
{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api
generated 0 new + 5385 unchanged - 21 fixed = 5385 total (was 5406) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 20s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green}
[jira] [Commented] (YARN-4006) YARN ATS Alternate Kerberos HTTP Authentication Changes
[
https://issues.apache.org/jira/browse/YARN-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294752#comment-15294752
]
Allen Wittenauer commented on YARN-4006:
... and now you know why Hortonworks never fixed your issue. ;)
[~lmccay], I was mainly just making sure I wasn't missing anything wrong with
the patch since, yeah, it's fairly simple change but as you know the security
code in Hadoop is full of pitfalls.
I'm +1 on the patch then and will commit here in a sec.
> YARN ATS Alternate Kerberos HTTP Authentication Changes
> ---
>
> Key: YARN-4006
> URL: https://issues.apache.org/jira/browse/YARN-4006
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, timelineserver
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.6.1, 2.8.0, 2.7.1, 2.7.2
>Reporter: Greg Senia
>Assignee: Greg Senia
>Priority: Blocker
> Attachments: YARN-4006-branch-trunk.patch,
> YARN-4006-branch2.6.0.patch, sample-ats-alt-auth.patch
>
>
> When attempting to use The Hadoop Alternate Authentication Classes. They do
> not exactly work with what was built with YARN-1935.
> I went ahead and made the following changes to support using a Custom
> AltKerberos DelegationToken custom class.
> Changes to: TimelineAuthenticationFilterInitializer.class
> {code}
>String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE);
> LOG.info("AuthType Configured: "+authType);
> if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> PseudoDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: PseudoDelegationTokenAuthenticationHandler");
> } else if (authType.equals(KerberosAuthenticationHandler.TYPE) ||
> (UserGroupInformation.isSecurityEnabled() &&
> conf.get("hadoop.security.authentication").equals(KerberosAuthenticationHandler.TYPE)))
> {
> if (!(authType.equals(KerberosAuthenticationHandler.TYPE))) {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> authType);
> LOG.info("AuthType: "+authType);
> } else {
> filterConfig.put(AuthenticationFilter.AUTH_TYPE,
> KerberosDelegationTokenAuthenticationHandler.class.getName());
> LOG.info("AuthType: KerberosDelegationTokenAuthenticationHandler");
> }
> // Resolve _HOST into bind address
> String bindAddress = conf.get(HttpServer2.BIND_ADDRESS);
> String principal =
> filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL);
> if (principal != null) {
> try {
> principal = SecurityUtil.getServerPrincipal(principal, bindAddress);
> } catch (IOException ex) {
> throw new RuntimeException(
> "Could not resolve Kerberos principal name: " + ex.toString(),
> ex);
> }
> filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL,
> principal);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
