[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360873#comment-14360873 ] zhihai xu commented on MAPREDUCE-6265: -- [~ozawa], yes, that is a very good suggestion, I uploaded a new patch MAPREDUCE-6265.001.patch which addressed your comment. Please review it. many thanks for the review. make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers. -- Key: MAPREDUCE-6265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at {code} protected static final int INITIAL_POOL_SIZE = 10; {code} We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM. Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size. We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360963#comment-14360963 ] Hadoop QA commented on MAPREDUCE-6265: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704478/MAPREDUCE-6265.001.patch against trunk revision f446669. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5288//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5288//console This message is automatically generated. make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers. -- Key: MAPREDUCE-6265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at {code} protected static final int INITIAL_POOL_SIZE = 10; {code} We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM. Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size. We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361602#comment-14361602 ] Tsuyoshi Ozawa commented on MAPREDUCE-6265: --- I meant containerlauncher.thread-initial-size should be containerlauncher.*threadpool*-initial-size; make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers. -- Key: MAPREDUCE-6265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at {code} protected static final int INITIAL_POOL_SIZE = 10; {code} We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM. Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size. We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361601#comment-14361601 ] Tsuyoshi Ozawa commented on MAPREDUCE-6265: --- [~zxu] should we also update the following line(name of property name itself)? {code} + public static final String MR_AM_CONTAINERLAUNCHER_THREADPOOL_INITIAL_SIZE = + MR_AM_PREFIX+containerlauncher.thread-initial-size; {code} make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers. -- Key: MAPREDUCE-6265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at {code} protected static final int INITIAL_POOL_SIZE = 10; {code} We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM. Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size. We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361609#comment-14361609 ] zhihai xu commented on MAPREDUCE-6265: -- I uploaded a new patch MAPREDUCE-6265.002.patch to fix this issue. make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers. -- Key: MAPREDUCE-6265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch, MAPREDUCE-6265.002.patch make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at {code} protected static final int INITIAL_POOL_SIZE = 10; {code} We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM. Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size. We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361608#comment-14361608 ] zhihai xu commented on MAPREDUCE-6265: -- [~ozawa], good catch, I forget to update the string value. thanks for your thorough review. make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers. -- Key: MAPREDUCE-6265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch, MAPREDUCE-6265.002.patch make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at {code} protected static final int INITIAL_POOL_SIZE = 10; {code} We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM. Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size. We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361615#comment-14361615 ] Hadoop QA commented on MAPREDUCE-6265: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704570/MAPREDUCE-6265.002.patch against trunk revision 32741cf. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5289//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5289//console This message is automatically generated. make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers. -- Key: MAPREDUCE-6265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch, MAPREDUCE-6265.002.patch make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at {code} protected static final int INITIAL_POOL_SIZE = 10; {code} We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM. Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size. We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360641#comment-14360641 ] Tsuyoshi Ozawa commented on MAPREDUCE-6265: --- [~zxu] looks good to me overall. Minor nits: I think the name of the property should be yarn.app.mapreduce.am.containerlauncher.*threadpool*-initial-size and MR_AM_CONTAINERLAUNCHER_*THREADPOOL*_INITIAL_SIZE for accuracy since it's a bit difficult to understand what thread initial size means. What do you think? make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers. -- Key: MAPREDUCE-6265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6265.000.patch make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at {code} protected static final int INITIAL_POOL_SIZE = 10; {code} We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM. Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size. We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359867#comment-14359867 ] zhihai xu commented on MAPREDUCE-6265: -- I uploaded a patch MAPREDUCE-6265.000.patch for review. The initial thread pool size is a important parameter for ContainerLauncherImpl. make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers. -- Key: MAPREDUCE-6265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6265.000.patch make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at {code} protected static final int INITIAL_POOL_SIZE = 10; {code} We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM. Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size. We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359899#comment-14359899 ] Hadoop QA commented on MAPREDUCE-6265: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704343/MAPREDUCE-6265.000.patch against trunk revision 8212877. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5285//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5285//console This message is automatically generated. make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers. -- Key: MAPREDUCE-6265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6265.000.patch make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at {code} protected static final int INITIAL_POOL_SIZE = 10; {code} We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM. Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size. We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359929#comment-14359929 ] zhihai xu commented on MAPREDUCE-6265: -- Hi [~ozawa], The patch passed all the tests. Do you have time to review it? thanks make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers. -- Key: MAPREDUCE-6265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6265.000.patch make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at {code} protected static final int INITIAL_POOL_SIZE = 10; {code} We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM. Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size. We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359952#comment-14359952 ] Tsuyoshi Ozawa commented on MAPREDUCE-6265: --- Sure, I'll check it. make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers. -- Key: MAPREDUCE-6265 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6265.000.patch make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at {code} protected static final int INITIAL_POOL_SIZE = 10; {code} We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM. Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE configurable to better manage the thread pool size. We saw some issue due to the small thread pool size when some node is down. The recovery from a shutdown node take very long time due to all the ContainerLauncher threads are blocked by IPC client connection to the shutdown node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)