[GitHub] [hadoop] zhengchenyu commented on pull request #6102: MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0.
zhengchenyu commented on PR #6102: URL: https://github.com/apache/hadoop/pull/6102#issuecomment-1741787028 @ayushtkn We can get back the ContainerLogAppender for CLA. But how about CRLA? If we have to keep our old behavior, I think we can just revert HADOOP-18649. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #6102: MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0.
zhengchenyu commented on PR #6102: URL: https://github.com/apache/hadoop/pull/6102#issuecomment-1741776757 @ayushtkn Thanks for your reply. I think what you said is reasonable. We should try to maintain old behaviour. There are two appender for regular container log: CLA, CRLA. CLA is regular appender. CRLA is roll over appender. Let's analyze the original behavior before HADOOP-18649. | Appender | behaviour | | --- | --- | | CLA | In default, yarn.app.container.log.filesize is 0, means disable cap. If yarn.app.container.log.filesize is greater than 0, the container log will show the limited size log until container is done. | | CRLA | If we use CRLA, yarn.app.container.log.filesize must be greater than 0, will roll over the log when size exceed the limit. | Then let's analyze the original behavior after HADOOP-18649. | Appender | behaviour | | --- | --- | | CLA | same with CRLA | | CRLA | If we use CRLA, yarn.app.container.log.filesize must be greater than 0, will roll over the log when size exceed the limit.| For CRLA, there is no change in behavior. I thought that when using CRLA, yarn.app.container.log.filesize might be 0, so the default value was changed. Util I see https://github.com/apache/hadoop/blob/5f47f091a227ecd1d9b1b375e9c71e01d31a264f/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java#L635. So I think your are right, there is no need to modify the default value. For CLA, I will replace with org.apache.log4j.FileAppender. But there will be no way to limit the size for FileAppender. And I think there is no need to limit the size for CLA. If we wanna limit the size, we could use CRLA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #6102: MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0.
zhengchenyu commented on PR #6102:
URL: https://github.com/apache/hadoop/pull/6102#issuecomment-1741690367
@ayushtkn
Thanks for you reply! I reproduce the problem of
'testThreadDumpOnTaskTimeout' in my laptop. Here we can see the log dir:
```
(base) zhengchenyu@zhengchenyudeMacBook-Pro hadoop % ls -l
/Users/zhengchenyu/work/git/zhengchenyu/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/org.apache.hadoop.mapreduce.v2.TestMRJobs/yarn-902198130/org.apache.hadoop.mapreduce.v2.TestMRJobs-logDir-nm-1_0/application_1696053632795_0001/container_1696053632795_0001_01_01
total 112
drwx--x--- 10 zhengchenyu staff320 9 30 14:01 ./
drwx--x--- 4 zhengchenyu staff128 9 30 14:00 ../
-rw-r--r-- 1 zhengchenyu staff 3657 9 30 14:00 directory.info
-rw-r- 1 zhengchenyu staff 39286 9 30 14:00 launch_container.sh
-rw-r--r-- 1 zhengchenyu staff 0 9 30 14:00 prelaunch.err
-rw-r--r-- 1 zhengchenyu staff100 9 30 14:00 prelaunch.out
-rw-r--r-- 1 zhengchenyu staff 1787 9 30 14:01 stderr
-rw-r--r-- 1 zhengchenyu staff 0 9 30 14:00 stdout
-rw-r--r-- 1 zhengchenyu staff 0 9 30 14:01 syslog
-rw-r--r-- 1 zhengchenyu staff297 9 30 14:01 syslog.1
(base) zhengchenyu@zhengchenyudeMacBook-Pro hadoop % cat
/Users/zhengchenyu/work/git/zhengchenyu/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/org.apache.hadoop.mapreduce.v2.TestMRJobs/yarn-902198130/org.apache.hadoop.mapreduce.v2.TestMRJobs-logDir-nm-1_0/application_1696053632795_0001/container_1696053632795_0001_01_01/syslog
(base) zhengchenyu@zhengchenyudeMacBook-Pro hadoop % cat
/Users/zhengchenyu/work/git/zhengchenyu/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/org.apache.hadoop.mapreduce.v2.TestMRJobs/yarn-902198130/org.apache.hadoop.mapreduce.v2.TestMRJobs-logDir-nm-1_0/application_1696053632795_0001/container_1696053632795_0001_01_01/syslog.1
2023-09-30 14:01:22,345 INFO [Thread-69]
org.eclipse.jetty.server.handler.ContextHandler: Stopped
o.e.j.s.ServletContextHandler@2dc995f4{static,/static,file:///Users/zhengchenyu/work/git/zhengchenyu/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/classes/webapps/static/,STOPPED}
```
We can see this is just the problem which I describe the problem in this PR.
MaxSize is 0, MaxBackup is 1. So we will always roll back the log, and show the
latest log forever.
This PR indeed solve this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #6102: MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0.
zhengchenyu commented on PR #6102: URL: https://github.com/apache/hadoop/pull/6102#issuecomment-1741619676 > Another observation, the day this PR was reverted, some tests which were broken since eternity broke again. https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1355/ & with this PR in they were passing: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1354/testReport/junit/org.apache.hadoop.mapreduce.v2/TestMRJobs/testThreadDumpOnTaskTimeout/ > > So, some way this fixes those tests, that we should anyway fix @ayushtkn It looks strange! Since we have revert it, should not fail. Do we have this error in other PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #6102: MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0.
zhengchenyu commented on PR #6102: URL: https://github.com/apache/hadoop/pull/6102#issuecomment-1741613920 @ayushtkn @virajjasani In my opinion, I think CLA should use FileAppender, and CRLA should use RollingFileAppender. FileAppender will disable the cap. Then CRLA will limited by maxFileSize and maxBackups. Users can choose CLA or CRLA they want -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #6102: MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0.
zhengchenyu commented on PR #6102: URL: https://github.com/apache/hadoop/pull/6102#issuecomment-1741612979 > ``` > 0 disables the cap. > ``` @ayushtkn Thanks for you reply! log4j.appender.CLA change s from 'org.apache.hadoop.yarn.ContainerLogAppender' to 'org.apache.log4j.RollingFileAppender'. For ContainerLogAppender, 0 means disable cap. For RollingFileAppender, 0 means maxFileSize for roll back log is 0. If we use RollingFileAppender, I think maybe there are no way to disable the cap. But we can only set the enough big value for RollingFileAppender::maxBackups and RollingFileAppender::maxFileSize. See: https://github.com/apache/logging-log4j1/blob/b7e9154128cd4ae1244c877a6fda8f834a0f2247/src/main/java/org/apache/log4j/RollingFileAppender.java#L50 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #6102: MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0.
zhengchenyu commented on PR #6102: URL: https://github.com/apache/hadoop/pull/6102#issuecomment-1741028712 @virajjasani Thanks for your reply. For CLA before HADOOP-18649: CLA is org.apache.hadoop.yarn.ContainerLogAppender which is extend from FileAppender. In default, yarn.app.container.log.filesize is set to 0, then log4j.appender.CLA.totalLogFileSize is set to 0. So maxEvents is 0, then eventBuffer is null. Then we will append directly. There are no limit. For CLA after HADOOP-18649: CLA is org.apache.log4j.RollingFileAppender. In default, yarn.app.container.log.filesize is set to 0, RollingFileAppender::maxFileSize is set to 0. This configure means that we will roll the log every time. Then will reproduce the problem which is reported by me. For CRLA before HADOOP-18649: CRLA is ContainerRollingLogAppender which is extend from org.apache.log4j.RollingFileAppender. In default, RollingFileAppender::maxFileSize is not set, so use default value 10M. The default value is comming from: https://github.com/apache/logging-log4j1/blob/b7e9154128cd4ae1244c877a6fda8f834a0f2247/src/main/java/org/apache/log4j/RollingFileAppender.java#L50 For CRLA after HADOOP-18649: CRLA is org.apache.log4j.RollingFileAppender. In default, yarn.app.container.log.filesize is set to 0, RollingFileAppender::maxFileSize is set to 0. This configure means that we will roll the log every time. Then will reproduce the problem which is reported by me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #6102: MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0.
zhengchenyu commented on PR #6102: URL: https://github.com/apache/hadoop/pull/6102#issuecomment-1736556764 @jojochuang @virajjasani @ayushtkn Can you please review this PR? I think this PR should be merged into next release. In default, for trunk, only show the latest line log. It will confuse the user. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhengchenyu commented on pull request #6102: MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0.
zhengchenyu commented on PR #6102:
URL: https://github.com/apache/hadoop/pull/6102#issuecomment-1726712035
@ayushtkn
https://github.com/apache/hadoop/pull/6042 was reverted. Can you please
review this PR?
And I still think we should change the default value.
I found that before HADOOP-18649, both log4j.appender.CLA.MaxFileSize and
log4j.appender.CLA.maximumFileSize are not set, so we will use the default
value 10M. After HADOOP-18649, log4j.appender.CLA.MaxFileSize is set to
${yarn.app.container.log.filesize}, in default for mr is 0, so only see the
latest log.
In the other way, in default, we even does not use
yarn.app.container.log.filesize, so there are no problem before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
