subject:"\[jira\] \[Commented\] \(YARN\-9629\) Support configurable MIN_LOG_ROLLING

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-04 Thread Adam Antal (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878427#comment-16878427
 ] 

Adam Antal commented on YARN-9629:
--

Thanks [~snemeth] for uploading the patches.

Looks like the two test failures are unrelated:
- {{TestFederationInterceptor.testMultipleSubClusters}}: There's been thrown a 
YarnRuntimeException: saying "amrmToken from UAM SC-1 should be null here". 
This patch does nothing around tokens.
- {{TestContainerManager.testLocalingResourceWhileContainerRunning}} is a 
timeout, and it is also unrelated

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch, YARN-9629.004.patch, YARN-9629.005.patch, 
> YARN-9629.006.patch, YARN-9629.branch-3.1.001.patch, 
> YARN-9629.branch-3.2.001.patch, YARN-9629.branch-3.2.001.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-03 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878092#comment-16878092
 ] 

Hadoop QA commented on YARN-9629:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
54s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 
41s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
41s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
21s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
49s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
17m 49s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
36s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
33s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
29s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 383 unchanged - 1 fixed = 383 total (was 384) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 20s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
57s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
49s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 23s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
47s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 55s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor |
|   | hadoop.yarn.server.nodemanager.containermanager.TestContainerManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.5 Server=18.09.5 Image:yetus/hadoop:63396beab41 |
| JIRA Issue | YARN-9629 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12973579/YARN-9629.branch-3.2.001.patch
 |
| Optional Tests

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-03 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877887#comment-16877887
 ] 

Hadoop QA commented on YARN-9629:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
11s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
25s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 4s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
3s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 33s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
45s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
38s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 3s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 385 unchanged - 1 fixed = 385 total (was 386) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 11s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
42s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
28s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m  
6s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
39s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 13s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:080e9d0 |
| JIRA Issue | YARN-9629 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12973563/YARN-9629.branch-3.1.001.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  xml  |
| uname | Linux 7d8921d5df41 4.4.0-139-generic #165-Ubuntu SMP

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-03 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877796#comment-16877796
 ] 

Szilard Nemeth commented on YARN-9629:
--

Hi [~adam.antal]!
Thanks for the latest patch, committed to trunk!
Thanks [~sunilg] for the reviews!
Pending Jenkins on branch-3.2 / branch-3.1 patches.

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch, YARN-9629.004.patch, YARN-9629.005.patch, 
> YARN-9629.006.patch, YARN-9629.branch-3.1.001.patch, 
> YARN-9629.branch-3.2.001.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-03 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1684#comment-1684
 ] 

Hudson commented on YARN-9629:
--

FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16857 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/16857/])
YARN-9629. Support configurable MIN_LOG_ROLLING_INTERVAL. Contributed by 
(snemeth: rev a2a8be18cb5e912c8de0ea6beec1de4a99de656b)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java


> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch, YARN-9629.004.patch, YARN-9629.005.patch, 
> YARN-9629.006.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-03 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877757#comment-16877757
 ] 

Hadoop QA commented on YARN-9629:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 53s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
33s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 382 unchanged - 1 fixed = 382 total (was 383) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 53s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
9s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
14s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 22m 
44s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
46s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}124m 38s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.5 Server=18.09.5 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9629 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12973542/YARN-9629.006.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  xml  |
| uname | Linux 732a94dae498 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 
08:28:49 UTC 2019 x86_64 x86_64 x86_64

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-03 Thread Adam Antal (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877668#comment-16877668
 ] 

Adam Antal commented on YARN-9629:
--

Thanks for the reply [~sunilg].

Agreed on #2. Removed the plus information from the debug mode, and moved the 
"rollingMonitorInterval is set as ... " part to {{serviceInit}} where the 
actual setting is happening. Also reorganised a bit using the LOG interface 
provided String formatting.

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch, YARN-9629.004.patch, YARN-9629.005.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-02 Thread Sunil Govindan (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877018#comment-16877018
 ] 

Sunil Govindan commented on YARN-9629:
--

Thanks [~adam.antal]

First point makes sense if the default value is -1 and this code will always 
kick in if this is never configured

For #2, given a configuration is done, its always better to cleanly say like in 
that last else condition, Hence taking it out and show same log in both kind of 
scenario still makes sense to me. in first log, you can minimize content if its 
duplicated, however skipping else conditions looks more cleaner and generic.

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch, YARN-9629.004.patch, YARN-9629.005.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-02 Thread Adam Antal (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877015#comment-16877015
 ] 

Adam Antal commented on YARN-9629:
--

Thanks for the review [~sunilg].
1) 
Firstly if the user does not override default configs this message is displayed 
and can be concerning if WARN level logs are displayed during standard startup. 
Secondly I think this does not falls into WARN level category. We want to 
inform the users about that the rolling feature is not enabled and not warning 
them about this.
Thirdly it was in INFO level log before this patch, so I'd keep it that way if 
there's no big reason to raise level.
Taking into account all of this I suggest to keep this in INFO level.

About adding an extra information: this is a static method and also we have 
only a Configuration object during {{serviceInit}}. That part is also not bound 
to a specific application, so we can not really add more information beside the 
NM's id (but it's the log of that NM where this message is displayed, so it 
does not really make sense).

2) 
This information is included in other branches of that if condition. If the 
configuration value does not pass the minimum constraint in case of debug mode 
we just say that this has been set to that value, otherwise we should warn the 
customer that the the configured value violates the minimum constraint. In 
either way, we provide the information, so there's no need to display is again 
(so there's no need to move that last log out of that else cause).

Does these rationale make sense to you [~sunilg]?

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch, YARN-9629.004.patch, YARN-9629.005.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-02 Thread Sunil Govindan (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876693#comment-16876693
 ] 

Sunil Govindan commented on YARN-9629:
--

Hi [~adam.antal] 

Thanks for the patch.

Couple of minor nits

1.
{code:java}
+  LOG.info("rollingMonitorInterval is set as " + interval
+  + ". The log rolling monitoring interval is disabled. "
+  + "The logs will be aggregated after this application is 
finished.");{code}
Could this be log to considered as warn ?.  is it possible to print appId or 
some more identifications details for better easy understanding?

2. 
{code:java}
+  if (lowerThanHardLimit) {
+if (logAggregationDebugMode) {
+  LOG.info("Log aggregation debug mode enabled. " +
+  "rollingMonitorInterval = " + interval);
+} else {
+  LOG.warn("rollingMonitorInterval should be more than " +
+  "or equal to {} seconds. Using {} seconds instead.",
+  minRollingMonitorInterval, minRollingMonitorInterval);
+  interval = minRollingMonitorInterval;
+}
+  } else {
+LOG.info("rollingMonitorInterval is set as " + interval
++ ". The logs will be aggregated every " + interval
++ " seconds");
+  }{code}
Last log which is in the else block can be taken out and kept a common one. 
since interval is set in internal if..else, its better we have the common log 
outside. (warn log what you added is correct, and that can be there)

 

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch, YARN-9629.004.patch, YARN-9629.005.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-01 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876199#comment-16876199
 ] 

Hadoop QA commented on YARN-9629:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
17m 31s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
24s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 382 unchanged - 1 fixed = 382 total (was 383) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 12s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
57s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
48s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m  
0s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
38s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 12s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.5 Server=18.09.5 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9629 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12973335/YARN-9629.005.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  xml  |
| uname | Linux fbbac63a0bcb 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 
08:28:49 UTC 2019 x86_64 x86_64 x86_64

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-01 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876171#comment-16876171
 ] 

Szilard Nemeth commented on YARN-9629:
--

Hi [~adam.antal]!
+1 for the latest patch!

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch, YARN-9629.004.patch, YARN-9629.005.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-01 Thread Adam Antal (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876128#comment-16876128
 ] 

Adam Antal commented on YARN-9629:
--

Ah indeed, thanks for the suggestion [~snemeth].
I must have missed it. Fixed it in patch v5.

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch, YARN-9629.004.patch, YARN-9629.005.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-07-01 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875976#comment-16875976
 ] 

Szilard Nemeth commented on YARN-9629:
--

Hi [~adam.antal]!
Thanks for the update!

The code changes you made for patch004 regarding documentation in 
yarn-default.xml looks good to me.
One minor thing left: Could you please remove any "suggested" references 
(variable names, log messages) in method 
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService#calculateRollingMonitorInterval?
 

Thanks!

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch, YARN-9629.004.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-06-27 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874265#comment-16874265
 ] 

Hadoop QA commented on YARN-9629:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
17m 53s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
22s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 382 unchanged - 1 fixed = 382 total (was 383) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 36s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
57s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
55s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 
17s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
40s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 10s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.5 Server=18.09.5 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9629 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12973085/YARN-9629.004.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  xml  |
| uname | Linux a1452de6d9ee 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 
08:28:49 UTC 2019 x86_64 x86_64 x86_64

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-06-27 Thread Adam Antal (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874150#comment-16874150
 ] 

Adam Antal commented on YARN-9629:
--

Thanks for the suggestions [~snemeth]. Rephrased the texts in patch v4.

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch, YARN-9629.004.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-06-25 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872282#comment-16872282
 ] 

Szilard Nemeth commented on YARN-9629:
--

Hi [~adam.antal]!
Thanks for the updated patch!

Couple of comments: 
1. In yarn-default.xml: 
Could you please rephrase these sentences? 
a. 
{code:java}
If this configuration is set less than the default 3600 a NodeManager may raise 
a warning.
{code}
b. 
{code:java}
By setting this configure, logs can be uploaded periodically when the 
application is running.
{code}

c. 

{code:java}
Defines the hard positive minimum limit for
{code}
should be 

{code:java}
Defines the positive minimum hard limit for
{code}

2. I don't like the name of 
org.apache.hadoop.yarn.conf.YarnConfiguration#MIN_LOG_ROLLING_INTERVAL_SECONDS_SUGGESTED.
 
The problem is I can't think of any better name, but maybe you can.

Apart from those, the code changes that I suggested in my last comment look 
good for me!




> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-06-24 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871071#comment-16871071
 ] 

Hadoop QA commented on YARN-9629:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 21s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
13s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 382 unchanged - 1 fixed = 382 total (was 383) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 18s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
51s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
42s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m 
52s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
37s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m  5s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.5 Server=18.09.5 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9629 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12972701/YARN-9629.003.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  xml  |
| uname | Linux cdad548db0b5 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 
08:28:49 UTC 2019 x86_64 x86_64 x86_64

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-06-24 Thread Adam Antal (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871003#comment-16871003
 ] 

Adam Antal commented on YARN-9629:
--

Thanks for the review [~snemeth].
Please find your concerns addressed in patch v3.

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch, 
> YARN-9629.003.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-06-21 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869664#comment-16869664
 ] 

Hadoop QA commented on YARN-9629:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 12s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 25s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
56s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
47s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 
17s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
36s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 21s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.5 Server=18.09.5 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | YARN-9629 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12972449/YARN-9629.002.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  xml  |
| uname | Linux 007404a92edd 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 
08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-06-21 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869568#comment-16869568
 ] 

Szilard Nemeth commented on YARN-9629:
--

Hi [~adam.antal]!
Thanks for this patch, looks good in overall!
You could improve the method serviceInit in class LogAggregationService if you 
have a bit time.
The if-else chain is becoming quite difficult to read.

I would extract some functions for clarity: 
1. 
{code:java}
if (rollingMonitorInterval <= 0) {
{code}
Could be a method named "isRollingMonitorIntervalDisabled" or something like 
this.

2. 
{code:java}
if (!logAggregationDebugMode && minRollingMonitorInterval <
  YarnConfiguration.MIN_LOG_ROLLING_INTERVAL_SECONDS_SUGGESTED) {
{code}
This could be a method named 
"logWarningIfRollingMonitorIntervalIsLessThanMinimum" or something like this.

3. 

{code:java}
if (rollingMonitorInterval < minRollingMonitorInterval) {
{code}
This could be isRollingMonitorIntervalLessThanMinimum.

I think you will have the idea based on my examples.

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-06-21 Thread Adam Antal (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869540#comment-16869540
 ] 

Adam Antal commented on YARN-9629:
--

Thanks for the review [~shuzirra].
Renamed the constant MIN_LOG_ROLLING_INTERVAL_SECONDS. I tried to save some 
characters from the original 
"NM_LOG_AGGREGATION_ROLL_MONITORING_INTERVAL_SECONDS_MINIMUM".

Fixed unit test and checkstyle errors. The fix of the unit test was to add this 
config to yarn-site.xml along with a new description.

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch, YARN-9629.002.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-06-21 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869413#comment-16869413
 ] 

Hadoop QA commented on YARN-9629:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 16s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 11s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 5 new + 383 unchanged - 0 fixed = 388 total (was 383) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 53s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 57s{color} 
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 
30s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
45s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 30s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | TEST-TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9629 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12972426/YARN-9629.001.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 6c80610ed2f4 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 83dcb9d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-06-21 Thread Gergely Pollak (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869379#comment-16869379
 ] 

Gergely Pollak commented on YARN-9629:
--

Hi [~adam.antal] thank you for the patch! 

The name of the constant MIN_LOG_ROLLING_INTERVAL is a bit confusing, it 
suggests that it is an interval (a number), but instead it is the name of a 
property, which is a string.

What makes it even more confusing, it used to be a numeric value, in the 
pervious patch. I'd suggest to rename the constant.

Otherwise LGTM+1 (non-binding)

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

2019-06-21 Thread Adam Antal (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-9629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869360#comment-16869360
 ] 

Adam Antal commented on YARN-9629:
--

Uploaded patch v1.
Implementation comments:
- a new config option has been added with current default value (3600), that 
can be now overridden
- added three tests for the edge cases that I wanted to cover
- had to refactor the if-else part in {{LogAggregationService#serviceInit}} so 
that a warning could be emitted if it has been set lower than the suggested 
minimum value.

Please review.

> Support configurable MIN_LOG_ROLLING_INTERVAL
> -
>
> Key: YARN-9629
> URL: https://issues.apache.org/jira/browse/YARN-9629
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-9629.001.patch
>
>
> One of the log-aggregation parameter, the minimum valid value for 
> {{yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds}} is 
> MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in 
> YARN-2583. 
> It has been empirically set as 1 hour, as lower values would too frequently 
> put the NodeManagers under pressure. For bigger clusters that is indeed a 
> valid limitation, but for smaller clusters it makes sense and a valid 
> customer usecase to use lower values, even like not so lower 30 mins. At this 
> point this can only be achieved by setting 
> {{yarn.nodemanager.log-aggregation.debug-enabled}}, which I believe should be 
> kept as debug purposes.
> I'm suggesting to make this min configurable, although a warning should be 
> logged in the NodeManager startup when this value is lower than 1 hour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

[jira] [Commented] (YARN-9629) Support configurable MIN_LOG_ROLLING_INTERVAL

25 matches

Site Navigation

Mail list logo

Footer information