Siyao Meng created HADOOP-20026:
-----------------------------------
Summary: MapReduce task JVMs should not force
java.net.preferIPv4Stack=true
Key: HADOOP-20026
URL: https://issues.apache.org/jira/browse/HADOOP-20026
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Siyao Meng
Assignee: Siyao Meng
The default for {{mapreduce.admin.map.child.java.opts}} and
{{mapreduce.admin.reduce.child.java.opts}} pins task JVMs to IPv4:
{code:java}
public static final String DEFAULT_MAPRED_ADMIN_JAVA_OPTS =
"-Djava.net.preferIPv4Stack=true " +
"-Dhadoop.metrics.log.level=WARN ";
{code}
[MRJobConfig.java:1048|https://github.com/apache/hadoop/blob/9ede2e95c6fb577d7ae8664833d5e151cbfc47c9/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java#L1048]
Daemons take the stack preference from {{HADOOP_OPTS}}, which operators can now
override on both Unix and Windows (HADOOP-19935). Task JVMs cannot follow that
setting, so on an IPv6-only cluster every map and reduce task runs with
IPv4-only sockets. A task that connects to a DataNode's IPv6 address fails with
{{java.nio.channels.UnsupportedAddressTypeException}}. On the IPv6
docker-compose test cluster, WordCount map tasks failed until the pin was
removed.
*Proposed fix:* remove {{-Djava.net.preferIPv4Stack=true}} from
{{DEFAULT_MAPRED_ADMIN_JAVA_OPTS}} so task JVMs use the JVM default
(dual-stack), as daemons and the MR ApplicationMaster already do. IPv4-only
clusters keep working, because dual-stack sockets connect to IPv4 addresses.
Operators who want the old behaviour can set the property in
{{mapreduce.admin.*.child.java.opts}}. This changes a default, so it should get
a release note.
YARN services sets the same property in {{JavaCommandLineBuilder}}; that can be
handled separately.
POC:
https://github.com/smengcl/hadoop/commit/0ff1dc343907f1c6ecc622a52550967160ce055f
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
