[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[
https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930646#comment-15930646
]
Hadoop QA commented on YARN-1547:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
14s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
15s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
43s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 43s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 96 new + 205 unchanged - 0 fixed = 301 total (was 205) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
16s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
15s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 182 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
17s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 13 new + 231 unchanged - 0 fixed = 244 total (was 231) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 25s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 16s{color}
| {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 16s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit te
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[
https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427327#comment-15427327
]
Hadoop QA commented on YARN-1547:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 30s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 48s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 44s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 96
new + 208 unchanged - 0 fixed = 304 total (was 208) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 182 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 48s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 25s {color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 38s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 1s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 17s
{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 48s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
|
| | Boxed value is unboxed and then immediately reboxed in
org.apache.hadoop.yarn.server.preventdos.SlotBasedAccumulator.getCounts() At
SlotBasedAccumulator.java:then immediately reboxed in
org.apache.hadoop.yarn.server.preventdos.SlotBasedAccumulator.getCounts() At
SlotBasedAccumulator.java:[line 85] |
| |
org.apache.hadoop.yarn.server.preventdos.SlotBasedAccumulator.updateTotal(Object,
long) invokes inefficient new Long(long) constructor; use Long.valueOf(long)
instead At SlotBasedAccumulator.java:Long(long) constructor; use
Long.valueOf(long) instead At SlotBasedAccumulator.java:[line 122] |
| Failed junit tests | hadoop.yarn.conf.TestYarnConfig
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[
https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406853#comment-15406853
]
Hadoop QA commented on YARN-1547:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 19s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 35 unchanged -
0 fixed = 36 total (was 35) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 13
new + 208 unchanged - 0 fixed = 221 total (was 208) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 22s {color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 11s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 0s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804094/YARN-1547.prototype.v0.patch
|
| JIRA Issue | YARN-1547 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 02b02dd84b00 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a1f6564 |
| Defau
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[
https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284086#comment-15284086
]
Hadoop QA commented on YARN-1547:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 41s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s
{color} | {color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 7s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 43s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common in
trunk has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 1s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 49s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 13 new +
211 unchanged - 0 fixed = 224 total (was 211) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s
{color} | {color:red} hadoop-yarn-server-common in the patch failed with JDK
v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 6m 43s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdk1.7.0_95
with JDK v1.7.0_95 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 21s {color}
| {color:red} hadoop-yarn-api in the patch failed with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-yarn-server-common in the patch passed with JDK
v1.8.0_91. {color} |
|
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[ https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284052#comment-15284052 ] Giovanni Matteo Fumarola commented on YARN-1547: [~vinodkv] I just pushed a first prototype version. Just to get feedback on it. Some few comments from my side: 1) To prevent DoS attacks, this new class performs 3 checks: on singular request, over the entire lifetime of the application, and in a short time using a sliding window. 2) The sliding window's code is taken from an other OSS project: Storm. I made some modification on top of it. I don't know how it works with shared code between OSS Apache projects. 3) Right now, the class is an interceptor for AMRMProxy, as we discussed we should move this class in a common place to be used by RM - maybe yarn-server-common. 4) I inserted some default values on YarnConfiguration. They are inaccurate. We should get those data from a real cluster. On top of it, I would like to create a new YarnConfiguration for DoS values. 5) In this patch, jUnits are missing. I tested it with some simple UTCs in the same style of [YARN-4117|https://issues.apache.org/jira/browse/YARN-4117]. As I said before this patch is just a prototype to get feedback :) in the next versions I will insert real UTCs. 6) A full complete JavaDocs are missing. 7) When the application passes a limit or gives a wrong values, the interceptor should reject the call. I inserted couple of TODO because I was not sure what to do: return null; or throw a YarnException. > Prevent DoS of ApplicationMasterProtocol by putting in limits > - > > Key: YARN-1547 > URL: https://issues.apache.org/jira/browse/YARN-1547 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-1547.pdf, YARN-1547.prototype.v0.patch, > YARN-1547.v0.pdf > > > Points of DoS in ApplicationMasterProtocol > - Host and trackingURL in RegisterApplicationMasterRequest > - Diagnostics, final trackingURL in FinishApplicationMasterRequest > - Unlimited number of resourceAsks, containersToBeReleased and > resourceBlacklistRequest in AllocateRequest > -- Unbounded number of priorities and/or resourceRequests in each ask. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[ https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281856#comment-15281856 ] Vinod Kumar Vavilapalli commented on YARN-1547: --- [~giovanni.fumarola] any update? > Prevent DoS of ApplicationMasterProtocol by putting in limits > - > > Key: YARN-1547 > URL: https://issues.apache.org/jira/browse/YARN-1547 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-1547.pdf, YARN-1547.v0.pdf > > > Points of DoS in ApplicationMasterProtocol > - Host and trackingURL in RegisterApplicationMasterRequest > - Diagnostics, final trackingURL in FinishApplicationMasterRequest > - Unlimited number of resourceAsks, containersToBeReleased and > resourceBlacklistRequest in AllocateRequest > -- Unbounded number of priorities and/or resourceRequests in each ask. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[ https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218681#comment-15218681 ] Vinod Kumar Vavilapalli commented on YARN-1547: --- Looked at the updated document, thanks [~giovanni.fumarola] and [~subru]! Few comments bq. Keeps tracks of all the requests In addition to the sliding window, we should have an upper limit on all requests possible from any single application in its entire life-time. The sliding window avoids unreasonable spikes while the across-the-lifetime limit avoids slow abuse of unreasonable amount of resources. bq. We can apply an additional control to check if the list-size of a single request overflows a specific threshold. +1, we must. To me this is an equally pressing concern compared to the spikes in list-sizes. It will be good if we can implement the enforcers / limiters in such a way that we can easily reuse them as the APIs themselves change over time from lists to maps etc. Overall, you have covered significant ground in the latest doc. We should get going with code etc. Thanks for taking care of this! > Prevent DoS of ApplicationMasterProtocol by putting in limits > - > > Key: YARN-1547 > URL: https://issues.apache.org/jira/browse/YARN-1547 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-1547.pdf, YARN-1547.v0.pdf > > > Points of DoS in ApplicationMasterProtocol > - Host and trackingURL in RegisterApplicationMasterRequest > - Diagnostics, final trackingURL in FinishApplicationMasterRequest > - Unlimited number of resourceAsks, containersToBeReleased and > resourceBlacklistRequest in AllocateRequest > -- Unbounded number of priorities and/or resourceRequests in each ask. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[ https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196152#comment-15196152 ] Giovanni Matteo Fumarola commented on YARN-1547: Cool, reattached as v0. > Prevent DoS of ApplicationMasterProtocol by putting in limits > - > > Key: YARN-1547 > URL: https://issues.apache.org/jira/browse/YARN-1547 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-1547.pdf, YARN-1547.v0.pdf > > > Points of DoS in ApplicationMasterProtocol > - Host and trackingURL in RegisterApplicationMasterRequest > - Diagnostics, final trackingURL in FinishApplicationMasterRequest > - Unlimited number of resourceAsks, containersToBeReleased and > resourceBlacklistRequest in AllocateRequest > -- Unbounded number of priorities and/or resourceRequests in each ask. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[ https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196013#comment-15196013 ] Vinod Kumar Vavilapalli commented on YARN-1547: --- [~giovanni.fumarola], a process feedback, we usually version the attachments and don't delete the old ones for the sake of keeping history. Will be good if you can do the same here. > Prevent DoS of ApplicationMasterProtocol by putting in limits > - > > Key: YARN-1547 > URL: https://issues.apache.org/jira/browse/YARN-1547 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-1547.pdf > > > Points of DoS in ApplicationMasterProtocol > - Host and trackingURL in RegisterApplicationMasterRequest > - Diagnostics, final trackingURL in FinishApplicationMasterRequest > - Unlimited number of resourceAsks, containersToBeReleased and > resourceBlacklistRequest in AllocateRequest > -- Unbounded number of priorities and/or resourceRequests in each ask. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[ https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195981#comment-15195981 ] Giovanni Matteo Fumarola commented on YARN-1547: Thank [~vinodkv] for the feedbacks, and [~subru] for helping me out with the new design document. I just pushed it. > Prevent DoS of ApplicationMasterProtocol by putting in limits > - > > Key: YARN-1547 > URL: https://issues.apache.org/jira/browse/YARN-1547 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-1547.pdf > > > Points of DoS in ApplicationMasterProtocol > - Host and trackingURL in RegisterApplicationMasterRequest > - Diagnostics, final trackingURL in FinishApplicationMasterRequest > - Unlimited number of resourceAsks, containersToBeReleased and > resourceBlacklistRequest in AllocateRequest > -- Unbounded number of priorities and/or resourceRequests in each ask. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[ https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187942#comment-15187942 ] Vinod Kumar Vavilapalli commented on YARN-1547: --- Looked at the PDF attached. Few comments: As I mentioned before, we should organize code such that we can reuse your proposed _DoSInterceptor_ (and may be even the _AbstractRequestInterceptor_ framework) to enforce limits on the server-side. bq. Its request contains information about host, port, and tracking URL. We need to validate them. One possible solution can be adding a validate inside RMWebApp, by using Jetty DoSFilter. I don't quite understand this. We should make sure that things like tracking URL are limited in length at the time of accepting them into RM - that'd mean checks-and-validations in ApplicationMasterService component (the first one in RM interacting with AM) itself. bq. To do this, we can implement a sliding window system that counts the number of request per minute and determinates if in a time slot we AM exceeds the number or requests. For most of our limits, sliding-windows won't work. What we need we are static limits per application, per ResourceRequest etc. A general ask: instead of a textual representation of the proposal, I think it is much better to reason/discuss about this in a tabular format. For e.g. ||Object||Object Type||Type of Limit to be put||Behavior on limit-overflow|| |RegisterApplicationMasterRequest.trackingURL|String|Size of String|String truncated| |AllocateRequest.ResourceBlacklistRequest.BlacklistAdditions|List of Strings|Size of the list, Size of each string|Rejected if list-size overflow, truncated strings| > Prevent DoS of ApplicationMasterProtocol by putting in limits > - > > Key: YARN-1547 > URL: https://issues.apache.org/jira/browse/YARN-1547 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-1547.pdf > > > Points of DoS in ApplicationMasterProtocol > - Host and trackingURL in RegisterApplicationMasterRequest > - Diagnostics, final trackingURL in FinishApplicationMasterRequest > - Unlimited number of resourceAsks, containersToBeReleased and > resourceBlacklistRequest in AllocateRequest > -- Unbounded number of priorities and/or resourceRequests in each ask. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[ https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176343#comment-15176343 ] Vinod Kumar Vavilapalli commented on YARN-1547: --- bq. Thanks for raising this Vinod Kumar Vavilapalli. I was wondering if I might take this up, if you are not actively working on it. Tx [~giovanni.fumarola], please go ahead and assign it to yourselves! We can discuss after you have a design, but wanted to bring up one point of note w.r.t this ticket and the larger YARN-1545 itself. It is likely that we can solve 60-70% of our use-case of avoiding accidental DoS'ing by well-behaved apps by way of putting limits in the client, but it is imperative that we handle this on the server-side instead of on client-side, lest an abusive client can circumvent any client-side restrictions. > Prevent DoS of ApplicationMasterProtocol by putting in limits > - > > Key: YARN-1547 > URL: https://issues.apache.org/jira/browse/YARN-1547 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli > > Points of DoS in ApplicationMasterProtocol > - Host and trackingURL in RegisterApplicationMasterRequest > - Diagnostics, final trackingURL in FinishApplicationMasterRequest > - Unlimited number of resourceAsks, containersToBeReleased and > resourceBlacklistRequest in AllocateRequest > -- Unbounded number of priorities and/or resourceRequests in each ask. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[ https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149085#comment-15149085 ] Giovanni Matteo Fumarola commented on YARN-1547: Thanks for raising this [~vinodkv]. I was wondering if I might take this up, if you are not actively working on it. I, [~subru] and [~kishorch] we brainstorming it and we came up with a first approach. I will share a small design document with you to receive feedback on it. > Prevent DoS of ApplicationMasterProtocol by putting in limits > - > > Key: YARN-1547 > URL: https://issues.apache.org/jira/browse/YARN-1547 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli > > Points of DoS in ApplicationMasterProtocol > - Host and trackingURL in RegisterApplicationMasterRequest > - Diagnostics, final trackingURL in FinishApplicationMasterRequest > - Unlimited number of resourceAsks, containersToBeReleased and > resourceBlacklistRequest in AllocateRequest > -- Unbounded number of priorities and/or resourceRequests in each ask. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1547) Prevent DoS of ApplicationMasterProtocol by putting in limits
[ https://issues.apache.org/jira/browse/YARN-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232326#comment-14232326 ] Carlo Curino commented on YARN-1547: This is related to YARN-2884, which give us a structurally sound location where to enforce throttling/bounding of the misbehaving AMs. > Prevent DoS of ApplicationMasterProtocol by putting in limits > - > > Key: YARN-1547 > URL: https://issues.apache.org/jira/browse/YARN-1547 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli > > Points of DoS in ApplicationMasterProtocol > - Host and trackingURL in RegisterApplicationMasterRequest > - Diagnostics, final trackingURL in FinishApplicationMasterRequest > - Unlimited number of resourceAsks, containersToBeReleased and > resourceBlacklistRequest in AllocateRequest > -- Unbounded number of priorities and/or resourceRequests in each ask. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
