(hadoop) branch trunk updated: HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#6537) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 555faf28ce36 HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to
3.2.5. (#6537) Contributed by Shilun Fan
555faf28ce36 is described below
commit 555faf28ce365443c17767adf0d6d570cef32ee3
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Sun Feb 11 07:41:46 2024 +0800
HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#6537)
Contributed by Shilun Fan
Reviewed-by: Steve Loughran
Signed-off-by: Shilun Fan
---
dev-support/bin/hadoop.sh | 3 ++-
hadoop-project/pom.xml| 2 +-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/dev-support/bin/hadoop.sh b/dev-support/bin/hadoop.sh
index fd9363e66bbd..c60603334feb 100755
--- a/dev-support/bin/hadoop.sh
+++ b/dev-support/bin/hadoop.sh
@@ -577,6 +577,7 @@ function shadedclient_rebuild
extra=(
"-Dtest=NoUnitTests"
+"-Dsurefire.failIfNoSpecifiedTests=false"
"-Dmaven.javadoc.skip=true"
"-Dcheckstyle.skip=true"
"-Dspotbugs.skip=true"
@@ -614,7 +615,7 @@ function shadedclient_rebuild
echo_and_redirect "${logfile}" \
"${MAVEN}" "${MAVEN_ARGS[@]}" verify -fae --batch-mode -am \
"${modules[@]}" \
- -DskipShade -Dtest=NoUnitTests -Dmaven.javadoc.skip=true
-Dcheckstyle.skip=true \
+ -DskipShade -Dtest=NoUnitTests -Dsurefire.failIfNoSpecifiedTests=false
-Dmaven.javadoc.skip=true -Dcheckstyle.skip=true \
-Dspotbugs.skip=true ${extra[*]}
count=$("${GREP}" -c '\[ERROR\]' "${logfile}")
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index e457797e3ad8..c732837dd17f 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -169,7 +169,7 @@
-Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError
-3.0.0-M1
+3.2.5
${maven-surefire-plugin.version}
${maven-surefire-plugin.version}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new 88fbe62f27e8 HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533)
Contributed by Shilun Fan
88fbe62f27e8 is described below
commit 88fbe62f27e859feda7f8d876118707993e85641
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Thu Feb 8 19:18:04 2024 +0800
HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533) Contributed by Shilun Fan
Reviewed-by: He Xiaoqiao
Signed-off-by: Shilun Fan
---
hadoop-common-project/hadoop-common/pom.xml | 2 +-
hadoop-project/pom.xml | 6 +++---
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml | 6 +++---
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index a18fdbf37f8f..d3b91de7a826 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -40,7 +40,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
org.apache.hadoop
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 2ef800e2d348..accaa2efc8f3 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -93,10 +93,10 @@
${common.protobuf2.scope}
-3.7.1
+3.21.12
${env.HADOOP_PROTOC_PATH}
-1.1.1
+1.2.0
${hadoop-thirdparty.version}
${hadoop-thirdparty.version}
org.apache.hadoop.thirdparty
@@ -250,7 +250,7 @@
org.apache.hadoop.thirdparty
-hadoop-shaded-protobuf_3_7
+hadoop-shaded-protobuf_3_21
${hadoop-thirdparty-protobuf.version}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
index 86e9b2da8e3b..ac85bead0f05 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
@@ -51,7 +51,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
@@ -69,7 +69,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
@@ -80,7 +80,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new e6eeaa05abdd HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533)
Contributed by Shilun Fan
e6eeaa05abdd is described below
commit e6eeaa05abddfcd4a55077c4ce9f71ea68cc3248
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Thu Feb 8 19:18:04 2024 +0800
HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533) Contributed by Shilun Fan
Reviewed-by: He Xiaoqiao
Signed-off-by: Shilun Fan
---
hadoop-common-project/hadoop-common/pom.xml | 2 +-
hadoop-project/pom.xml | 6 +++---
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml | 6 +++---
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index e1d1683d7278..86ae00ead72a 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -40,7 +40,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
org.apache.hadoop
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 3fc37667e1cc..a189514b9fb8 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -93,10 +93,10 @@
${common.protobuf2.scope}
-3.7.1
+3.21.12
${env.HADOOP_PROTOC_PATH}
-1.1.1
+1.2.0
${hadoop-thirdparty.version}
${hadoop-thirdparty.version}
org.apache.hadoop.thirdparty
@@ -250,7 +250,7 @@
org.apache.hadoop.thirdparty
-hadoop-shaded-protobuf_3_7
+hadoop-shaded-protobuf_3_21
${hadoop-thirdparty-protobuf.version}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
index 9bc1f1737a5a..ea01ec9cce7f 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
@@ -51,7 +51,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
@@ -69,7 +69,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
@@ -80,7 +80,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-18980. S3A credential provider remapping: make extensible (#6406)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new f4a44d2b3562 HADOOP-18980. S3A credential provider remapping: make
extensible (#6406)
f4a44d2b3562 is described below
commit f4a44d2b35622d0432c763633b4c957e87143e6f
Author: Viraj Jasani
AuthorDate: Fri Feb 9 05:29:40 2024 -0900
HADOOP-18980. S3A credential provider remapping: make extensible (#6406)
Contributed by Viraj Jasani
---
.../java/org/apache/hadoop/conf/Configuration.java | 4 +-
.../java/org/apache/hadoop/util/StringUtils.java | 40 +-
.../org/apache/hadoop/util/TestStringUtils.java| 56 +
.../java/org/apache/hadoop/fs/s3a/Constants.java | 7 ++
.../java/org/apache/hadoop/fs/s3a/S3AUtils.java| 23
.../fs/s3a/auth/CredentialProviderListFactory.java | 8 ++
.../markdown/tools/hadoop-aws/aws_sdk_upgrade.md | 49
.../src/site/markdown/tools/hadoop-aws/index.md| 22
.../fs/s3a/ITestS3AAWSCredentialsProvider.java | 31 +
.../fs/s3a/TestS3AAWSCredentialsProvider.java | 139 +
10 files changed, 376 insertions(+), 3 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 7c4f617b179e..ea3d6dc74e4a 100755
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -2339,8 +2339,8 @@ public class Configuration implements
Iterable>,
}
return StringUtils.getTrimmedStringCollection(valueString);
}
-
- /**
+
+ /**
* Get the comma delimited values of the name property as
* an array of Strings, trimmed of the leading and trailing
whitespace.
* If no such property is specified then an empty array is returned.
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
index 3debd36da78d..b8d999162d38 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
@@ -25,6 +25,7 @@ import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
+import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedHashSet;
import java.util.List;
@@ -479,7 +480,28 @@ public class StringUtils {
set.remove("");
return set;
}
-
+
+ /**
+ * Splits an "=" separated value String, trimming leading and
+ * trailing whitespace on each value after splitting by comma and new line
separator.
+ *
+ * @param str a comma separated String with values, may be null
+ * @return a Map of String keys and values, empty
+ * Collection if null String input.
+ */
+ public static Map getTrimmedStringCollectionSplitByEquals(
+ String str) {
+String[] trimmedList = getTrimmedStrings(str);
+Map pairs = new HashMap<>();
+for (String s : trimmedList) {
+ String[] splitByKeyVal = getTrimmedStringsSplitByEquals(s);
+ if (splitByKeyVal.length == 2) {
+pairs.put(splitByKeyVal[0], splitByKeyVal[1]);
+ }
+}
+return pairs;
+ }
+
/**
* Splits a comma or newline separated value String, trimming
* leading and trailing whitespace on each value.
@@ -497,6 +519,22 @@ public class StringUtils {
return str.trim().split("\\s*[,\n]\\s*");
}
+ /**
+ * Splits "=" separated value String, trimming
+ * leading and trailing whitespace on each value.
+ *
+ * @param str an "=" separated String with values,
+ *may be null
+ * @return an array of String values, empty array if null String
+ * input
+ */
+ public static String[] getTrimmedStringsSplitByEquals(String str){
+if (null == str || str.trim().isEmpty()) {
+ return emptyStringArray;
+}
+return str.trim().split("\\s*=\\s*");
+ }
+
final public static String[] emptyStringArray = {};
final public static char COMMA = ',';
final public static String COMMA_STR = ",";
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
index f05b58956760..d9bcf5842689 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
@@ -44,
(hadoop) branch branch-3.4.0 updated: HADOOP-19059. S3A: Update AWS Java SDK to 2.23.19 (#6538)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.0 by this push: new 893b9efb5398 HADOOP-19059. S3A: Update AWS Java SDK to 2.23.19 (#6538) 893b9efb5398 is described below commit 893b9efb5398d841af0b17b6835245e37fa5b2c5 Author: Adnan Hemani AuthorDate: Thu Feb 8 12:38:37 2024 -0800 HADOOP-19059. S3A: Update AWS Java SDK to 2.23.19 (#6538) Contributed by Adnan Hemani --- LICENSE-binary | 2 +- hadoop-project/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 93e38cc34ee4..c5ba9f938aef 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -363,7 +363,7 @@ org.objenesis:objenesis:2.6 org.xerial.snappy:snappy-java:1.1.10.4 org.yaml:snakeyaml:2.0 org.wildfly.openssl:wildfly-openssl:1.1.3.Final -software.amazon.awssdk:bundle:jar:2.23.5 +software.amazon.awssdk:bundle:jar:2.23.19 diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 5fbcf409f352..2ef800e2d348 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -187,7 +187,7 @@ 1.0-beta-1 900 1.12.599 -2.23.5 +2.23.19 1.0.1 2.7.1 1.11.2 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-18993. Add option fs.s3a.classloader.isolation (#6301)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new 836f0aeadc2c HADOOP-18993. Add option fs.s3a.classloader.isolation
(#6301)
836f0aeadc2c is described below
commit 836f0aeadc2cd12baf54c50cb34c9b1afa2303d4
Author: Antonio Murgia
AuthorDate: Wed Feb 7 15:30:19 2024 +0100
HADOOP-18993. Add option fs.s3a.classloader.isolation (#6301)
The option fs.s3a.classloader.isolation (default: true) can be set to false
to disable s3a classloader isolation;
This can assist in using custom credential providers and other extension
points.
Contributed by Antonio Murgia
---
.../java/org/apache/hadoop/fs/s3a/Constants.java | 17 +++
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java| 4 +-
.../java/org/apache/hadoop/fs/s3a/S3AUtils.java| 20 +++
.../src/site/markdown/tools/hadoop-aws/index.md| 24
.../s3a/ITestS3AFileSystemIsolatedClassloader.java | 134 +
5 files changed, 197 insertions(+), 2 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
index 744146ccf4f3..9ed6594244b9 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
@@ -1600,4 +1600,21 @@ public final class Constants {
*/
public static final boolean CHECKSUM_VALIDATION_DEFAULT = false;
+ /**
+ * Are extensions classes, such as {@code fs.s3a.aws.credentials.provider},
+ * going to be loaded from the same classloader that loaded
+ * the {@link S3AFileSystem}?
+ * It is useful to turn classloader isolation off for Apache Spark
applications
+ * that might load {@link S3AFileSystem} from the Spark distribution
(Launcher classloader)
+ * while users might want to provide custom extensions (loaded by Spark
MutableClassloader).
+ * Value: {@value}.
+ */
+ public static final String AWS_S3_CLASSLOADER_ISOLATION =
+"fs.s3a.classloader.isolation";
+
+ /**
+ * Default value for {@link #AWS_S3_CLASSLOADER_ISOLATION}.
+ * Value: {@value}.
+ */
+ public static final boolean DEFAULT_AWS_S3_CLASSLOADER_ISOLATION = true;
}
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
index de48c2df1569..3aec03766dac 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
@@ -559,8 +559,8 @@ public class S3AFileSystem extends FileSystem implements
StreamCapabilities,
// fix up the classloader of the configuration to be whatever
// classloader loaded this filesystem.
- // See: HADOOP-17372
- conf.setClassLoader(this.getClass().getClassLoader());
+ // See: HADOOP-17372 and follow-up on HADOOP-18993
+ S3AUtils.maybeIsolateClassloader(conf, this.getClass().getClassLoader());
// patch the Hadoop security providers
patchSecurityCredentialProviders(conf);
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
index 6a719739e720..cceaea6159d7 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
@@ -1670,4 +1670,24 @@ public final class S3AUtils {
return String.format("bytes=%d-%d", rangeStart, rangeEnd);
}
+ /**
+ * If classloader isolation is {@code true}
+ * (through {@link Constants#AWS_S3_CLASSLOADER_ISOLATION}) or not
+ * explicitly set, then the classLoader of the input configuration object
+ * will be set to the input classloader, otherwise nothing will happen.
+ * @param conf configuration object.
+ * @param classLoader isolated classLoader.
+ */
+ static void maybeIsolateClassloader(Configuration conf, ClassLoader
classLoader) {
+if (conf.getBoolean(Constants.AWS_S3_CLASSLOADER_ISOLATION,
+Constants.DEFAULT_AWS_S3_CLASSLOADER_ISOLATION)) {
+ LOG.debug("Configuration classloader set to S3AFileSystem classloader:
{}", classLoader);
+ conf.setClassLoader(classLoader);
+} else {
+ LOG.debug("Configuration classloader not changed, support classes needed
will be loaded " +
+ "from the classloader that instantiated the
Configuration object: {}",
+ conf.getClassLoader());
+}
+ }
+
}
diff --git
a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/inde
(hadoop) branch branch-3.4.0 updated: HADOOP-19045. CreateSession Timeout - followup (#6532)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.0 by this push: new c8ee14a2eb2f HADOOP-19045. CreateSession Timeout - followup (#6532) c8ee14a2eb2f is described below commit c8ee14a2eb2fd1ea435945b6740b77c73c132f8a Author: Steve Loughran AuthorDate: Wed Feb 7 12:07:54 2024 + HADOOP-19045. CreateSession Timeout - followup (#6532) This is a followup to PR: HADOOP-19045. S3A: Validate CreateSession Timeout Propagation (#6470) Remove all declarations of fs.s3a.connection.request.timeout in - hadoop-common/src/main/resources/core-default.xml - hadoop-aws/src/test/resources/core-site.xml New test in TestAwsClientConfig to verify that the value defined in fs.s3a.Constants class is used. This is brittle to someone overriding it in their test setups, but as this test is intended to verify that the option is not explicitly set, there's no workaround. Contributed by Steve Loughran --- .../hadoop-common/src/main/resources/core-default.xml| 14 -- .../src/site/markdown/tools/hadoop-aws/performance.md| 2 +- .../markdown/tools/hadoop-aws/troubleshooting_s3a.md | 2 +- .../apache/hadoop/fs/s3a/impl/TestAwsClientConfig.java | 16 hadoop-tools/hadoop-aws/src/test/resources/core-site.xml | 5 - 5 files changed, 18 insertions(+), 21 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml index 29ec06db6598..00533e362b99 100644 --- a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml +++ b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml @@ -2097,20 +2097,6 @@ - - fs.s3a.connection.request.timeout - 0s - -Time out on HTTP requests to the AWS service; 0 means no timeout. - -Important: this is the maximum duration of any AWS service call, -including upload and copy operations. If non-zero, it must be larger -than the time to upload multi-megabyte blocks to S3 from the client, -and to rename many-GB files. Use with care. - - - - fs.s3a.etag.checksum.enabled false diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md index 4c03cca17161..44c8086d4f02 100644 --- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md +++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md @@ -231,7 +231,7 @@ The choice is yours. Generally recovery is better, but sometimes fail-fast is mo | `fs.s3a.connection.acquisition.timeout` | `60s` | `*` | Timeout for waiting for a connection from the pool. | | `fs.s3a.connection.establish.timeout` | `30s` | | Time to establish the TCP/TLS connection | | `fs.s3a.connection.idle.time` | `60s` | `*` | Maximum time for idle HTTP connections in the pool| -| `fs.s3a.connection.request.timeout` | `0` | | If greater than zero, maximum duration of any request | +| `fs.s3a.connection.request.timeout` | `60s` | | If greater than zero, maximum time for a response | | `fs.s3a.connection.timeout` | `200s` | | Timeout for socket problems on a TCP channel | | `fs.s3a.connection.ttl` | `5m`| | Lifetime of HTTP connections from the pool| diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md index c1b499e3da95..e53e4a002265 100644 --- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md +++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md @@ -1510,7 +1510,7 @@ It is possible to configure a global timeout for AWS service calls using followi ```xml fs.s3a.connection.request.timeout - 0 + 5m Time out on HTTP requests to the AWS service; 0 means no timeout. Measured in seconds; the usual time suffixes are all supported diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestAwsClientConfig.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestAwsClientConfig.java index 2a9e05755ce6..eacff90ea4c8 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestAwsClientConfig.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestAwsClientConfig.java @@ -40,6 +40,7 @@ import static org.apache.hadoop.fs.s3a.Constants.DEFAULT_CONNECTION_KEEPALIVE; import static org.apache.hadoop.fs.s3a.Cons
(hadoop) branch branch-3.4.0 updated: HDFS-17370. Fix junit dependency for running parameterized tests in hadoop-hdfs-rbf (#6522)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.0 by this push: new 65674b977e7e HDFS-17370. Fix junit dependency for running parameterized tests in hadoop-hdfs-rbf (#6522) 65674b977e7e is described below commit 65674b977e7e9ed118f90e73982e399aabc08790 Author: Takanobu Asanuma AuthorDate: Tue Feb 6 11:54:14 2024 +0900 HDFS-17370. Fix junit dependency for running parameterized tests in hadoop-hdfs-rbf (#6522) (cherry picked from commit 9a7eeadaac818258b319cdb0dc19e9bb1e4fa11a) --- hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml | 5 + 1 file changed, 5 insertions(+) diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml b/hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml index 9ed29346b920..7c3da647f96c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml @@ -177,6 +177,11 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd";> junit-jupiter-params test + + org.junit.jupiter + junit-jupiter-engine + test + - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-19049. Fix StatisticsDataReferenceCleaner classloader leak (#6488)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new d6c66f76aeb1 HADOOP-19049. Fix StatisticsDataReferenceCleaner
classloader leak (#6488)
d6c66f76aeb1 is described below
commit d6c66f76aeb17a9033197b56b278f3dd5d33b67c
Author: Jia Fan
AuthorDate: Sat Feb 3 22:48:52 2024 +0800
HADOOP-19049. Fix StatisticsDataReferenceCleaner classloader leak (#6488)
Contributed by Jia Fan
---
.../src/main/java/org/apache/hadoop/fs/FileSystem.java | 1 +
.../org/apache/hadoop/fs/TestFileSystemStorageStatistics.java| 9 +
2 files changed, 10 insertions(+)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
index 0213772ab6a5..768fd5b5e1ca 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
@@ -4077,6 +4077,7 @@ public abstract class FileSystem extends Configured
STATS_DATA_CLEANER.
setName(StatisticsDataReferenceCleaner.class.getName());
STATS_DATA_CLEANER.setDaemon(true);
+ STATS_DATA_CLEANER.setContextClassLoader(null);
STATS_DATA_CLEANER.start();
}
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemStorageStatistics.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemStorageStatistics.java
index e99f0f2348b3..5710049afb10 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemStorageStatistics.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemStorageStatistics.java
@@ -34,6 +34,7 @@ import java.util.concurrent.TimeUnit;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
/**
* This tests basic operations of {@link FileSystemStorageStatistics} class.
@@ -102,6 +103,14 @@ public class TestFileSystemStorageStatistics {
}
}
+ @Test
+ public void testStatisticsDataReferenceCleanerClassLoader() {
+Thread thread = Thread.getAllStackTraces().keySet().stream()
+.filter(t ->
t.getName().contains("StatisticsDataReferenceCleaner")).findFirst().get();
+ClassLoader classLoader = thread.getContextClassLoader();
+assertNull(classLoader);
+ }
+
/**
* Helper method to retrieve the specific FileSystem.Statistics value by
name.
*
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-19044. S3A: AWS SDK V2 - Update region logic (#6479)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new fd0d0c90d964 HADOOP-19044. S3A: AWS SDK V2 - Update region logic
(#6479)
fd0d0c90d964 is described below
commit fd0d0c90d9646becdc4ca6e8098f97b89250f7ee
Author: Viraj Jasani
AuthorDate: Sat Feb 3 05:44:23 2024 -0900
HADOOP-19044. S3A: AWS SDK V2 - Update region logic (#6479)
Improves region handling in the S3A connector, including enabling
cross-region support
when that is considered necessary.
Consult the documentation in connecting.md/connecting.html for the current
resolution process.
Contributed by Viraj Jasani
---
.../hadoop/fs/s3a/DefaultS3ClientFactory.java | 53 +--
.../tools/hadoop-aws/aws_sdk_v2_changelog.md | 4 +-
.../site/markdown/tools/hadoop-aws/connecting.md | 36 +
.../src/site/markdown/tools/hadoop-aws/index.md| 4 +-
.../hadoop/fs/s3a/ITestS3AEndpointRegion.java | 163 -
.../hadoop/fs/s3a/test/PublicDatasetTestUtils.java | 7 +
6 files changed, 251 insertions(+), 16 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
index 0a3267a9fe51..284ba8e6ae5c 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
@@ -267,9 +267,10 @@ public class DefaultS3ClientFactory extends Configured
*/
private , ClientT>
void configureEndpointAndRegion(
BuilderT builder, S3ClientCreationParameters parameters, Configuration
conf) {
-URI endpoint = getS3Endpoint(parameters.getEndpoint(), conf);
+final String endpointStr = parameters.getEndpoint();
+final URI endpoint = getS3Endpoint(endpointStr, conf);
-String configuredRegion = parameters.getRegion();
+final String configuredRegion = parameters.getRegion();
Region region = null;
String origin = "";
@@ -291,15 +292,33 @@ public class DefaultS3ClientFactory extends Configured
if (endpoint != null) {
checkArgument(!fipsEnabled,
"%s : %s", ERROR_ENDPOINT_WITH_FIPS, endpoint);
- builder.endpointOverride(endpoint);
- // No region was configured, try to determine it from the endpoint.
+ boolean endpointEndsWithCentral =
+ endpointStr.endsWith(CENTRAL_ENDPOINT);
+
+ // No region was configured,
+ // determine the region from the endpoint.
if (region == null) {
-region = getS3RegionFromEndpoint(parameters.getEndpoint());
+region = getS3RegionFromEndpoint(endpointStr,
+endpointEndsWithCentral);
if (region != null) {
origin = "endpoint";
}
}
- LOG.debug("Setting endpoint to {}", endpoint);
+
+ // No need to override endpoint with "s3.amazonaws.com".
+ // Let the client take care of endpoint resolution. Overriding
+ // the endpoint with "s3.amazonaws.com" causes 400 Bad Request
+ // errors for non-existent buckets and objects.
+ // ref: https://github.com/aws/aws-sdk-java-v2/issues/4846
+ if (!endpointEndsWithCentral) {
+builder.endpointOverride(endpoint);
+LOG.debug("Setting endpoint to {}", endpoint);
+ } else {
+builder.crossRegionAccessEnabled(true);
+origin = "central endpoint with cross region access";
+LOG.debug("Enabling cross region access for endpoint {}",
+endpointStr);
+ }
}
if (region != null) {
@@ -354,20 +373,32 @@ public class DefaultS3ClientFactory extends Configured
/**
* Parses the endpoint to get the region.
- * If endpoint is the central one, use US_EAST_1.
+ * If endpoint is the central one, use US_EAST_2.
*
* @param endpoint the configure endpoint.
+ * @param endpointEndsWithCentral true if the endpoint is configured as
central.
* @return the S3 region, null if unable to resolve from endpoint.
*/
- private static Region getS3RegionFromEndpoint(String endpoint) {
+ private static Region getS3RegionFromEndpoint(final String endpoint,
+ final boolean endpointEndsWithCentral) {
-if(!endpoint.endsWith(CENTRAL_ENDPOINT)) {
+if (!endpointEndsWithCentral) {
LOG.debug("Endpoint {} is not the default; parsing", endpoint);
return AwsHostNameUtils.parseSigningRegion(endpoint,
S3_SERVICE_NAME).orElse(null);
}
-// endpoint is for US_EAST_1;
-return Region.US_EAST_1;
+// Select default region here to enable cross-region access.
+// If both "fs.s3a.endpoint" and "fs.s3a.endpoint.region" are empty,
+// Spark sets "fs.s3a.end
(hadoop) branch branch-3.4.0 updated: HADOOP-18987. Various fixes to FileSystem API docs (#6292)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new fdbc67b9a8f5 HADOOP-18987. Various fixes to FileSystem API docs (#6292)
fdbc67b9a8f5 is described below
commit fdbc67b9a8f52237acdeed7b46f442e6e597588a
Author: DieterDP <90392398+dieterdp...@users.noreply.github.com>
AuthorDate: Fri Feb 2 12:49:31 2024 +0100
HADOOP-18987. Various fixes to FileSystem API docs (#6292)
Contributed by Dieter De Paepe
---
.../src/site/markdown/filesystem/abortable.md | 3 +-
.../src/site/markdown/filesystem/filesystem.md | 97 +++---
.../site/markdown/filesystem/fsdatainputstream.md | 8 +-
.../src/site/markdown/filesystem/model.md | 28 ---
.../src/site/markdown/filesystem/notation.md | 8 +-
5 files changed, 75 insertions(+), 69 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/abortable.md
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/abortable.md
index 7e6ea01a8fe9..dc7677bd9a54 100644
---
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/abortable.md
+++
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/abortable.md
@@ -88,14 +88,13 @@ for example. output streams returned by the S3A FileSystem.
The stream MUST implement `Abortable` and `StreamCapabilities`.
```python
- if unsupported:
+if unsupported:
throw UnsupportedException
if not isOpen(stream):
no-op
StreamCapabilities.hasCapability("fs.capability.outputstream.abortable") ==
True
-
```
diff --git
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
index 5fba8a2515bb..518026876ba0 100644
---
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
+++
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
@@ -64,13 +64,13 @@ a protected directory, result in such an exception being
raised.
### `boolean isDirectory(Path p)`
-def isDirectory(FS, p)= p in directories(FS)
+def isDir(FS, p) = p in directories(FS)
### `boolean isFile(Path p)`
-def isFile(FS, p) = p in files(FS)
+def isFile(FS, p) = p in filenames(FS)
### `FileStatus getFileStatus(Path p)`
@@ -250,7 +250,7 @@ process.
changes are made to the filesystem, the result of `listStatus(parent(P))`
SHOULD
include the value of `getFileStatus(P)`.
-* After an entry at path `P` is created, and before any other
+* After an entry at path `P` is deleted, and before any other
changes are made to the filesystem, the result of `listStatus(parent(P))`
SHOULD
NOT include the value of `getFileStatus(P)`.
@@ -305,7 +305,7 @@ that they must all be listed, and, at the time of listing,
exist.
All paths must exist. There is no requirement for uniqueness.
forall p in paths :
- exists(fs, p) else raise FileNotFoundException
+ exists(FS, p) else raise FileNotFoundException
Postconditions
@@ -381,7 +381,7 @@ being completely performed.
Path `path` must exist:
-exists(FS, path) : raise FileNotFoundException
+if not exists(FS, path) : raise FileNotFoundException
Postconditions
@@ -432,7 +432,7 @@ of data which must be collected in a single RPC call.
Preconditions
-exists(FS, path) else raise FileNotFoundException
+if not exists(FS, path) : raise FileNotFoundException
### Postconditions
@@ -463,7 +463,7 @@ and 1 for file count.
Preconditions
-exists(FS, path) else raise FileNotFoundException
+if not exists(FS, path) : raise FileNotFoundException
Postconditions
@@ -567,7 +567,7 @@ when writing objects to a path in the filesystem.
Postconditions
-result = integer >= 0
+result = integer >= 0
The outcome of this operation is usually identical to `getDefaultBlockSize()`,
with no checks for the existence of the given path.
@@ -591,12 +591,12 @@ on the filesystem.
Preconditions
-if not exists(FS, p) : raise FileNotFoundException
+if not exists(FS, p) : raise FileNotFoundException
Postconditions
-if len(FS, P) > 0: getFileStatus(P).getBlockSize() > 0
+if len(FS, P) > 0 : getFileStatus(P).getBlockSize() > 0
result == getFileStatus(P).getBlockSize()
1. The outcome of this operation MUST be identical to the value of
@@ -654,12 +654,12 @@ No ancestor may be a file
forall d = ancestors(FS, p) :
if exists(FS, d) and not isDir(FS, d) :
-raise [ParentNotDirectoryException, FileAlreadyExistsException,
IOException]
+raise {ParentNotDirectoryException, FileAlreadyExistsException,
IOException}
Postconditions
-FS' where FS'.Di
(hadoop) branch branch-3.4.0 updated: HDFS-17359. EC: recheck failed streamers should only after flushing all packets. (#6503). Contributed by farmmamba.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new c8a17b09c33d HDFS-17359. EC: recheck failed streamers should only
after flushing all packets. (#6503). Contributed by farmmamba.
c8a17b09c33d is described below
commit c8a17b09c33db0f518984977ab5cf643a7ef34b2
Author: hfutatzhanghb
AuthorDate: Thu Feb 1 22:53:37 2024 +0800
HDFS-17359. EC: recheck failed streamers should only after flushing all
packets. (#6503). Contributed by farmmamba.
Signed-off-by: Takanobu Asanuma
(cherry picked from commit 4f4b84698658c42efd5f7af0c6061546f6fb7ce8)
---
.../src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
index a58c7bbb204f..8320cc9a4086 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
@@ -671,9 +671,9 @@ public class DFSStripedOutputStream extends DFSOutputStream
// for healthy streamers, wait till all of them have fetched the new
block
// and flushed out all the enqueued packets.
flushAllInternals();
+ // recheck failed streamers again after the flush
+ newFailed = checkStreamers();
}
-// recheck failed streamers again after the flush
-newFailed = checkStreamers();
while (newFailed.size() > 0) {
failedStreamers.addAll(newFailed);
coordinator.clearFailureStates();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-19046. S3A: update AWS V2 SDK to 2.23.5; v1 to 1.12.599 (#6467)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.0 by this push: new f01d33d4539f HADOOP-19046. S3A: update AWS V2 SDK to 2.23.5; v1 to 1.12.599 (#6467) f01d33d4539f is described below commit f01d33d4539f971bd7b1c04594c2294f148b4249 Author: Steve Loughran AuthorDate: Sun Jan 21 19:00:34 2024 + HADOOP-19046. S3A: update AWS V2 SDK to 2.23.5; v1 to 1.12.599 (#6467) This update ensures that the timeout set in fs.s3a.connection.request.timeout is passed down to calls to CreateSession made in the AWS SDK to get S3 Express session tokens. Contributed by Steve Loughran --- LICENSE-binary | 4 ++-- NOTICE-binary | 2 +- hadoop-project/pom.xml | 4 ++-- .../src/site/markdown/tools/hadoop-aws/testing.md | 13 ++--- 4 files changed, 15 insertions(+), 8 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 3720a7809563..93e38cc34ee4 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -215,7 +215,7 @@ com.aliyun:aliyun-java-sdk-kms:2.11.0 com.aliyun:aliyun-java-sdk-ram:3.1.0 com.aliyun:aliyun-java-sdk-sts:3.0.0 com.aliyun.oss:aliyun-sdk-oss:3.13.2 -com.amazonaws:aws-java-sdk-bundle:1.12.565 +com.amazonaws:aws-java-sdk-bundle:1.12.599 com.cedarsoftware:java-util:1.9.0 com.cedarsoftware:json-io:2.5.1 com.fasterxml.jackson.core:jackson-annotations:2.12.7 @@ -363,7 +363,7 @@ org.objenesis:objenesis:2.6 org.xerial.snappy:snappy-java:1.1.10.4 org.yaml:snakeyaml:2.0 org.wildfly.openssl:wildfly-openssl:1.1.3.Final -software.amazon.awssdk:bundle:jar:2.21.41 +software.amazon.awssdk:bundle:jar:2.23.5 diff --git a/NOTICE-binary b/NOTICE-binary index 6db51d08b42f..7389a31fd5a1 100644 --- a/NOTICE-binary +++ b/NOTICE-binary @@ -66,7 +66,7 @@ available from http://www.digip.org/jansson/. AWS SDK for Java -Copyright 2010-2022 Amazon.com, Inc. or its affiliates. All Rights Reserved. +Copyright 2010-2024 Amazon.com, Inc. or its affiliates. All Rights Reserved. This product includes software developed by Amazon Technologies, Inc (http://www.amazon.com/). diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 2c0cc3edcb6c..b79fdd576109 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -186,8 +186,8 @@ 1.3.1 1.0-beta-1 900 -1.12.565 -2.21.41 +1.12.599 +2.23.5 1.0.1 2.7.1 1.11.2 diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md index c2eafbcb8de2..62d449daeea5 100644 --- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md +++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md @@ -1142,7 +1142,7 @@ as it may take a couple of SDK updates before it is ready. 1. Identify the latest AWS SDK [available for download](https://aws.amazon.com/sdk-for-java/). 1. Create a private git branch of trunk for JIRA, and in `hadoop-project/pom.xml` update the `aws-java-sdk.version` to the new SDK version. -1. Update AWS SDK versions in NOTICE.txt. +1. Update AWS SDK versions in NOTICE.txt and LICENSE.binary 1. Do a clean build and rerun all the `hadoop-aws` tests. This includes the `-Pscale` set, with a role defined for the assumed role tests. in `fs.s3a.assumed.role.arn` for testing assumed roles, @@ -1164,11 +1164,18 @@ your IDE or via maven. `mvn dependency:tree -Dverbose > target/dependencies.txt`. Examine the `target/dependencies.txt` file to verify that no new artifacts have unintentionally been declared as dependencies - of the shaded `aws-java-sdk-bundle` artifact. + of the shaded `software.amazon.awssdk:bundle:jar` artifact. 1. Run a full AWS-test suite with S3 client-side encryption enabled by setting `fs.s3a.encryption.algorithm` to 'CSE-KMS' and setting up AWS-KMS Key ID in `fs.s3a.encryption.key`. +The dependency chain of the `hadoop-aws` module should be similar to this, albeit +with different version numbers: +``` +[INFO] +- org.apache.hadoop:hadoop-aws:jar:3.4.0-SNAPSHOT:compile +[INFO] | +- software.amazon.awssdk:bundle:jar:2.23.5:compile +[INFO] | \- org.wildfly.openssl:wildfly-openssl:jar:1.1.3.Final:compile +``` ### Basic command line regression testing We need a run through of the CLI to see if there have been changes there @@ -1365,5 +1372,5 @@ Don't be surprised if this happens, don't worry too much, and, while that rollback option is there to be used, ideally try to work forwards. If the problem is with the SDK, file issues with the - [AWS SDK Bug tracker](https://github.com/aw
(hadoop) branch branch-3.4.0 updated: HADOOP-19045. S3A: Validate CreateSession Timeout Propagation (#6470)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new 4344537aa2d1 HADOOP-19045. S3A: Validate CreateSession Timeout
Propagation (#6470)
4344537aa2d1 is described below
commit 4344537aa2d118eb751868c6fa10594df5a842df
Author: Steve Loughran
AuthorDate: Tue Jan 30 15:32:24 2024 +
HADOOP-19045. S3A: Validate CreateSession Timeout Propagation (#6470)
New test ITestCreateSessionTimeout to verify that the duration set
in fs.s3a.connection.request.timeout is passed all the way down.
This is done by adding a sleep() in a custom signer and verifying
that it is interrupted and that an AWSApiCallTimeoutException is
raised.
+ Fix testRequestTimeout()
* doesn't skip if considered cross-region
* sets a minimum duration of 0 before invocation
* resets the minimum afterwards
Contributed by Steve Loughran
---
.../java/org/apache/hadoop/fs/s3a/Constants.java | 25 ++-
.../hadoop/fs/s3a/auth/CustomHttpSigner.java | 2 +-
.../apache/hadoop/fs/s3a/impl/AWSClientConfig.java | 2 +-
.../hadoop/fs/s3a/ITestS3AConfiguration.java | 28 ++-
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 10 +
.../s3a/performance/ITestCreateSessionTimeout.java | 211 +
6 files changed, 262 insertions(+), 16 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
index 4408cf68a451..744146ccf4f3 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
@@ -337,16 +337,33 @@ public final class Constants {
public static final int DEFAULT_SOCKET_TIMEOUT =
(int)DEFAULT_SOCKET_TIMEOUT_DURATION.toMillis();
/**
- * Time until a request is timed-out: {@value}.
- * If zero, there is no timeout.
+ * How long should the SDK retry/wait on a response from an S3 store:
{@value}
+ * including the time needed to sign the request.
+ *
+ * This is time to response, so for a GET request it is "time to 200
response"
+ * not the time limit to download the requested data.
+ * This makes it different from {@link #REQUEST_TIMEOUT}, which is for total
+ * HTTP request.
+ *
+ * Default unit is milliseconds.
+ *
+ * There is a minimum duration set in {@link
#MINIMUM_NETWORK_OPERATION_DURATION};
+ * it is impossible to set a delay less than this, even for testing.
+ * Why so? Too many deployments where the configuration assumed the timeout
was in seconds
+ * and that "120" was a reasonable value rather than "too short to work
reliably"
+ *
+ * Note for anyone writing tests which need to set a low value for this:
+ * to avoid the minimum duration overrides, call
+ * {@code AWSClientConfig.setMinimumOperationDuration()} and set a low value
+ * before creating the filesystem.
*/
public static final String REQUEST_TIMEOUT =
"fs.s3a.connection.request.timeout";
/**
- * Default duration of a request before it is timed out: Zero.
+ * Default duration of a request before it is timed out: 60s.
*/
- public static final Duration DEFAULT_REQUEST_TIMEOUT_DURATION =
Duration.ZERO;
+ public static final Duration DEFAULT_REQUEST_TIMEOUT_DURATION =
Duration.ofSeconds(60);
/**
* Default duration of a request before it is timed out: Zero.
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CustomHttpSigner.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CustomHttpSigner.java
index ba1169a5e598..528414b63e32 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CustomHttpSigner.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CustomHttpSigner.java
@@ -40,7 +40,7 @@ import
software.amazon.awssdk.identity.spi.AwsCredentialsIdentity;
* fs.s3a.http.signer.class = org.apache.hadoop.fs.s3a.auth.CustomHttpSigner
*
*/
-public final class CustomHttpSigner implements
HttpSigner {
+public class CustomHttpSigner implements HttpSigner {
private static final Logger LOG = LoggerFactory
.getLogger(CustomHttpSigner.class);
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java
index f6da9d84e0a7..60729ac30866 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java
@@ -577,7 +577,7 @@ public final class AWSClientConfig {
(hadoop) branch branch-3.4.0 updated: HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple server calls (#6022)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new 92e59743c104 HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution:
prevent multiple server calls (#6022)
92e59743c104 is described below
commit 92e59743c104970fe8878f925f65dffcf2eba092
Author: Pranav Saxena <108325433+saxenapra...@users.noreply.github.com>
AuthorDate: Tue Jan 30 05:17:04 2024 -0800
HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple
server calls (#6022)
Address JDK bug JDK-8314978 related to handling of HTTP 100
responses.
https://bugs.openjdk.org/browse/JDK-8314978
In the AbfsHttpOperation, after sendRequest() we call processResponse()
method from AbfsRestOperation.
Even if the conn.getOutputStream() fails due to expect-100 error,
we consume the exception and let the code go ahead.
This may call getHeaderField() / getHeaderFields() / getHeaderFieldLong()
after
getOutputStream() has failed. These invocation all lead to server calls.
This commit aims to prevent this.
If connection.getOutputStream() fails due to an Expect-100 error,
the ABFS client does not invoke getHeaderField(), getHeaderFields(),
getHeaderFieldLong() or getInputStream().
getResponseCode() is safe as on the failure it sets the
responseCode variable in HttpUrlConnection object.
Contributed by Pranav Saxena
---
.../fs/azurebfs/constants/AbfsHttpConstants.java | 1 +
.../fs/azurebfs/services/AbfsHttpOperation.java| 41 +--
.../fs/azurebfs/services/AbfsOutputStream.java | 9 +++-
.../fs/azurebfs/services/ITestAbfsClient.java | 3 +-
.../azurebfs/services/ITestAbfsOutputStream.java | 61 ++
.../azurebfs/services/ITestAbfsRestOperation.java | 3 +-
6 files changed, 109 insertions(+), 9 deletions(-)
diff --git
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java
index 91f6bddcc1d4..63de71eb178d 100644
---
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java
+++
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java
@@ -69,6 +69,7 @@ public final class AbfsHttpConstants {
* and should qualify for retry.
*/
public static final int HTTP_CONTINUE = 100;
+ public static final String EXPECT_100_JDK_ERROR = "Server rejected
operation";
// Abfs generic constants
public static final String SINGLE_WHITE_SPACE = " ";
diff --git
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java
index 7f5df6066f1b..c0b554f60702 100644
---
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java
+++
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java
@@ -22,12 +22,14 @@ import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.HttpURLConnection;
+import java.net.ProtocolException;
import java.net.URL;
import java.util.List;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLSocketFactory;
+import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.fs.azurebfs.utils.UriUtils;
import org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory;
@@ -43,6 +45,7 @@ import
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations;
import org.apache.hadoop.fs.azurebfs.contracts.services.AbfsPerfLoggable;
import org.apache.hadoop.fs.azurebfs.contracts.services.ListResultSchema;
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EXPECT_100_JDK_ERROR;
import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.HUNDRED_CONTINUE;
import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.EXPECT;
@@ -83,6 +86,7 @@ public class AbfsHttpOperation implements AbfsPerfLoggable {
private long sendRequestTimeMs;
private long recvResponseTimeMs;
private boolean shouldMask = false;
+ private boolean connectionDisconnectedOnError = false;
public static AbfsHttpOperation getAbfsHttpOperationWithFixedResult(
final URL url,
@@ -324,14 +328,26 @@ public class AbfsHttpOperation implements
AbfsPerfLoggable {
*/
outputStream = getConnOutputStream();
} catch (IOException e) {
-/* If getOutputStream fails with an exception and expect header
- is enabled, we return back without throwing
(hadoop) branch branch-3.4.0 updated: YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.0 by this push: new 2b68496fad2b YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455) 2b68496fad2b is described below commit 2b68496fad2b9bfb2c4c11ebeecb19a62eeb49d2 Author: Ferenc Erdelyi <55103964+ferde...@users.noreply.github.com> AuthorDate: Mon Jan 22 15:41:48 2024 +0100 YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455) --- .../capacity/policy/PriorityUtilizationQueueOrderingPolicy.java| 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java index d12a5e74b8d0..f60208e04849 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java @@ -28,6 +28,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity .CapacitySchedulerConfiguration; import org.apache.hadoop.yarn.util.resource.Resources; +import java.util.ArrayList; import java.util.Comparator; import java.util.Iterator; import java.util.List; @@ -254,9 +255,9 @@ public class PriorityUtilizationQueueOrderingPolicy // partitionToLookAt is a thread local variable, therefore it is safe to mutate it. PriorityUtilizationQueueOrderingPolicy.partitionToLookAt.set(partition); -// Sort the snapshot of the queues in order to avoid breaking the prerequisites of TimSort. -// See YARN-10178 for details. -return queues.stream().map(PriorityQueueResourcesForSorting::new).sorted( +// Copy (for thread safety) and sort the snapshot of the queues in order to avoid breaking +// the prerequisites of TimSort. See YARN-10178 for details. +return new ArrayList<>(queues).stream().map(PriorityQueueResourcesForSorting::new).sorted( new PriorityQueueComparator()).map(PriorityQueueResourcesForSorting::getQueue).collect( Collectors.toList()).iterator(); } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new 6190423505f2 YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455) 6190423505f2 is described below commit 6190423505f2fc27a53e03ddbc027a53b20f5104 Author: Ferenc Erdelyi <55103964+ferde...@users.noreply.github.com> AuthorDate: Mon Jan 22 15:41:48 2024 +0100 YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455) --- .../capacity/policy/PriorityUtilizationQueueOrderingPolicy.java| 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java index d12a5e74b8d0..f60208e04849 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java @@ -28,6 +28,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity .CapacitySchedulerConfiguration; import org.apache.hadoop.yarn.util.resource.Resources; +import java.util.ArrayList; import java.util.Comparator; import java.util.Iterator; import java.util.List; @@ -254,9 +255,9 @@ public class PriorityUtilizationQueueOrderingPolicy // partitionToLookAt is a thread local variable, therefore it is safe to mutate it. PriorityUtilizationQueueOrderingPolicy.partitionToLookAt.set(partition); -// Sort the snapshot of the queues in order to avoid breaking the prerequisites of TimSort. -// See YARN-10178 for details. -return queues.stream().map(PriorityQueueResourcesForSorting::new).sorted( +// Copy (for thread safety) and sort the snapshot of the queues in order to avoid breaking +// the prerequisites of TimSort. See YARN-10178 for details. +return new ArrayList<>(queues).stream().map(PriorityQueueResourcesForSorting::new).sorted( new PriorityQueueComparator()).map(PriorityQueueResourcesForSorting::getQueue).collect( Collectors.toList()).iterator(); } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-19056. Highlight RBF features and improvements targeting version 3.4. (#6512) Contributed by Takanobu Asanuma.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new 95951e928ee5 HADOOP-19056. Highlight RBF features and improvements
targeting version 3.4. (#6512) Contributed by Takanobu Asanuma.
95951e928ee5 is described below
commit 95951e928ee5d725704e61132f4ea850c8716469
Author: Takanobu Asanuma
AuthorDate: Wed Jan 31 14:30:35 2024 +0900
HADOOP-19056. Highlight RBF features and improvements targeting version
3.4. (#6512) Contributed by Takanobu Asanuma.
Signed-off-by: Shilun Fan
---
hadoop-project/src/site/markdown/index.md.vm | 12
1 file changed, 12 insertions(+)
diff --git a/hadoop-project/src/site/markdown/index.md.vm
b/hadoop-project/src/site/markdown/index.md.vm
index 7c1b3ef5eb40..54e8055e633d 100644
--- a/hadoop-project/src/site/markdown/index.md.vm
+++ b/hadoop-project/src/site/markdown/index.md.vm
@@ -89,8 +89,20 @@ Important features and improvements are as follows:
[HDFS-15294](https://issues.apache.org/jira/browse/HDFS-15294) HDFS Federation
balance tool introduces one tool to balance data across different namespace.
+[HDFS-13522](https://issues.apache.org/jira/browse/HDFS-13522),
[HDFS-16767](https://issues.apache.org/jira/browse/HDFS-16767) Support observer
node from Router-Based Federation.
+
**Improvement**
+[HADOOP-13144](https://issues.apache.org/jira/browse/HADOOP-13144),
[HDFS-13274](https://issues.apache.org/jira/browse/HDFS-13274),
[HDFS-15757](https://issues.apache.org/jira/browse/HDFS-15757)
+
+These tickets have enhanced IPC throughput between Router and NameNode via
multiple connections per user, and optimized connection management.
+
+[HDFS-14090](https://issues.apache.org/jira/browse/HDFS-14090) RBF: Improved
isolation for downstream name nodes. {Static}
+
+Router supports assignment of the dedicated number of RPC handlers to achieve
isolation for all downstream nameservices
+it is configured to proxy. Since large or busy clusters may have relatively
higher RPC traffic to the namenode compared to other clusters namenodes,
+this feature if enabled allows admins to configure higher number of RPC
handlers for busy clusters.
+
[HDFS-17128](https://issues.apache.org/jira/browse/HDFS-17128) RBF:
SQLDelegationTokenSecretManager should use version of tokens updated by other
routers.
The SQLDelegationTokenSecretManager enhances performance by maintaining
processed tokens in memory. However, there is
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19056. Highlight RBF features and improvements targeting version 3.4. (#6512) Contributed by Takanobu Asanuma.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 02c1273ceb9b HADOOP-19056. Highlight RBF features and improvements
targeting version 3.4. (#6512) Contributed by Takanobu Asanuma.
02c1273ceb9b is described below
commit 02c1273ceb9b2104c6a891dbedc43e0f0149f5ab
Author: Takanobu Asanuma
AuthorDate: Wed Jan 31 14:30:35 2024 +0900
HADOOP-19056. Highlight RBF features and improvements targeting version
3.4. (#6512) Contributed by Takanobu Asanuma.
Signed-off-by: Shilun Fan
---
hadoop-project/src/site/markdown/index.md.vm | 12
1 file changed, 12 insertions(+)
diff --git a/hadoop-project/src/site/markdown/index.md.vm
b/hadoop-project/src/site/markdown/index.md.vm
index 7c1b3ef5eb40..54e8055e633d 100644
--- a/hadoop-project/src/site/markdown/index.md.vm
+++ b/hadoop-project/src/site/markdown/index.md.vm
@@ -89,8 +89,20 @@ Important features and improvements are as follows:
[HDFS-15294](https://issues.apache.org/jira/browse/HDFS-15294) HDFS Federation
balance tool introduces one tool to balance data across different namespace.
+[HDFS-13522](https://issues.apache.org/jira/browse/HDFS-13522),
[HDFS-16767](https://issues.apache.org/jira/browse/HDFS-16767) Support observer
node from Router-Based Federation.
+
**Improvement**
+[HADOOP-13144](https://issues.apache.org/jira/browse/HADOOP-13144),
[HDFS-13274](https://issues.apache.org/jira/browse/HDFS-13274),
[HDFS-15757](https://issues.apache.org/jira/browse/HDFS-15757)
+
+These tickets have enhanced IPC throughput between Router and NameNode via
multiple connections per user, and optimized connection management.
+
+[HDFS-14090](https://issues.apache.org/jira/browse/HDFS-14090) RBF: Improved
isolation for downstream name nodes. {Static}
+
+Router supports assignment of the dedicated number of RPC handlers to achieve
isolation for all downstream nameservices
+it is configured to proxy. Since large or busy clusters may have relatively
higher RPC traffic to the namenode compared to other clusters namenodes,
+this feature if enabled allows admins to configure higher number of RPC
handlers for busy clusters.
+
[HDFS-17128](https://issues.apache.org/jira/browse/HDFS-17128) RBF:
SQLDelegationTokenSecretManager should use version of tokens updated by other
routers.
The SQLDelegationTokenSecretManager enhances performance by maintaining
processed tokens in memory. However, there is
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
