(hadoop) branch trunk updated: HADOOP-19286: S3A: Support cross region access when S3 region/endpoint is set (ADDENDUM) (#7098)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 5ea3a1bd0a5 HADOOP-19286: S3A: Support cross region access when S3
region/endpoint is set (ADDENDUM) (#7098)
5ea3a1bd0a5 is described below
commit 5ea3a1bd0a5fb40fdca7805bafb563e12be2bf85
Author: Syed Shameerur Rahman
AuthorDate: Fri Oct 4 19:28:53 2024 +0530
HADOOP-19286: S3A: Support cross region access when S3 region/endpoint is
set (ADDENDUM) (#7098)
Contributed by Syed Shameerur Rahman
---
.../main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java | 4 +++-
.../test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java | 5 +
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
index b5e0afbf9e8..0b18ce999f8 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
@@ -271,8 +271,10 @@ public class DefaultS3ClientFactory extends Configured
* If endpoint is configured via via fs.s3a.endpoint, set it.
* If no region is configured, try to parse region from endpoint.
* If no region is configured, and it could not be parsed from the
endpoint,
- * set the default region as US_EAST_2 and enable cross region access.
+ * set the default region as US_EAST_2
* If configured region is empty, fallback to SDK resolution chain.
+ * S3 cross region is enabled by default irrespective of region or
endpoint
+ * is set or not.
*
*
* @param builder S3 client builder.
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
index 18cb985065f..80b061de031 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
@@ -356,6 +356,10 @@ public class ITestS3AEndpointRegion extends
AbstractS3ATestBase {
// skip the test if the region is sa-east-1
skipCrossRegionTest();
final Configuration newConf = new Configuration(getConfiguration());
+removeBaseAndBucketOverrides(newConf,
+ENDPOINT,
+AWS_S3_CROSS_REGION_ACCESS_ENABLED,
+AWS_REGION);
// disable cross region access
newConf.setBoolean(AWS_S3_CROSS_REGION_ACCESS_ENABLED, false);
newConf.set(AWS_REGION, SA_EAST_1);
@@ -374,6 +378,7 @@ public class ITestS3AEndpointRegion extends
AbstractS3ATestBase {
skipCrossRegionTest();
final Configuration newConf = new Configuration(getConfiguration());
removeBaseAndBucketOverrides(newConf,
+ENDPOINT,
AWS_S3_CROSS_REGION_ACCESS_ENABLED,
AWS_REGION);
// enable cross region access
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-release-support) branch main updated: HADOOP-19087 3.4.1 RC2 validation
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hadoop-release-support.git
The following commit(s) were added to refs/heads/main by this push:
new 1c53594 HADOOP-19087 3.4.1 RC2 validation
1c53594 is described below
commit 1c53594d76ec6e62bfd74481c8b7a26aabab1e0f
Author: Steve Loughran
AuthorDate: Wed Oct 2 15:38:51 2024 +0100
HADOOP-19087 3.4.1 RC2 validation
support for -Pnative in builds; docs
---
README.md | 31 ++-
build.xml | 7 +++
2 files changed, 37 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 969718e..889700d 100644
--- a/README.md
+++ b/README.md
@@ -493,6 +493,25 @@ do not do this while building/testing downstream projects
ant release.src.untar release.src.build
```
+This build does not attempt to build the native binaries.
+The `Pnative` profile can be enabled (or any other maven arguments)
+in by declaring them in the property `source.compile.maven.args`
+
+```properties
+source.compile.maven.args=-Pnative
+```
+These are added at the end of the hard-coded arguments (`clean install
-DskipTests`)
+
+Testing is also possible through the target `release.src.test`
+
+```
+ant release.src.test
+```
+Again, the options set in `source.compile.maven.args` are passed down.
+
+These targets are simply invoking maven in the source subdirectory
+of `downloads/untar/source`, for example
`downloads/untar/source/hadoop-3.4.1-src`
+
### untar site and validate.
@@ -540,6 +559,7 @@ The and build itself is successful.
## Testing ARM binaries
```bash
+# untars the `-aarch64.tar.gz` binary
ant release.arm.untar
ant release.arm.commands
```
@@ -558,7 +578,7 @@ git clone
https://github.com/apache/hadoop-release-support.git
A lot of the targets build maven projects from the staged maven artifacts.
-For this to work
+For this to work:
1. Check out the relevant projects in your local system.
2. Set their location in the `build.properties` file
@@ -587,6 +607,15 @@ Download the artifacts from maven staging repositories and
compile/test a minima
ant mvn-test
```
+Note: setting up on linux needs a number of dependencies installed (at least
on an arm64 ubuntu system):
+
+```bash
+sudo apt-get install libssl-dev openssl subversion maven gpgv openjdk-17-jdk
libjsch-java ant ant-optional ant-contrib cmake
+
+# add to .bashrc
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64/
+```
+
## Validating maven dependencies
To see the dependencies of the maven project:
diff --git a/build.xml b/build.xml
index 8edbb3d..6921428 100644
--- a/build.xml
+++ b/build.xml
@@ -168,6 +168,9 @@
+
+
+
@@ -1120,9 +1123,11 @@ Message is in file ${message.out}
+
+
@@ -1131,6 +1136,7 @@ Message is in file ${message.out}
+
@@ -1153,6 +1159,7 @@ Message is in file ${message.out}
+Executing hadoop commands from directory
target/bin-untar/${release}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (4ff0dceebda -> e9ed21c065f)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 4ff0dceebda HADOOP-19288. hadoop-client-runtime to exclude dnsjava InetAddressResolverProvider (#7070) add e9ed21c065f HADOOP-19286: S3A: Support cross region access when S3 region/endpoint is set (#7067) No new revisions were added by this update. Summary of changes: .../java/org/apache/hadoop/fs/s3a/Constants.java | 13 ++ .../hadoop/fs/s3a/DefaultS3ClientFactory.java | 14 +-- .../site/markdown/tools/hadoop-aws/connecting.md | 10 + .../hadoop/fs/s3a/ITestS3AConfiguration.java | 5 ++- .../hadoop/fs/s3a/ITestS3AEndpointRegion.java | 49 ++ 5 files changed, 85 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19288. hadoop-client-runtime to exclude dnsjava InetAddressResolverProvider (#7070)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new b0102360368 HADOOP-19288. hadoop-client-runtime to exclude dnsjava InetAddressResolverProvider (#7070) b0102360368 is described below commit b0102360368b8e66c4a8813d0a315de54fc1a091 Author: cxzl25 <3898450+cxz...@users.noreply.github.com> AuthorDate: Tue Oct 1 21:48:48 2024 +0800 HADOOP-19288. hadoop-client-runtime to exclude dnsjava InetAddressResolverProvider (#7070) Contributed by dzcxzl. --- hadoop-client-modules/hadoop-client-runtime/pom.xml | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/hadoop-client-modules/hadoop-client-runtime/pom.xml b/hadoop-client-modules/hadoop-client-runtime/pom.xml index 1a4b25e8802..93657d7e25d 100644 --- a/hadoop-client-modules/hadoop-client-runtime/pom.xml +++ b/hadoop-client-modules/hadoop-client-runtime/pom.xml @@ -229,8 +229,9 @@ jnamed* lookup* update* -META-INF/versions/21/* -META-INF/versions/21/**/* +META-INF/versions/18/* +META-INF/versions/18/**/* + META-INF/services/java.net.spi.InetAddressResolverProvider @@ -245,7 +246,7 @@ META-INF/versions/9/module-info.class META-INF/versions/11/module-info.class - META-INF/versions/21/module-info.class + META-INF/versions/18/module-info.class - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.1 updated: HADOOP-19294. NPE on maven enforcer with -Pnative on arm mac (#7082)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.1 by this push: new 1d069671ed1 HADOOP-19294. NPE on maven enforcer with -Pnative on arm mac (#7082) 1d069671ed1 is described below commit 1d069671ed1a99e6c06e21ab06af7ad07a6ab732 Author: Steve Loughran AuthorDate: Tue Oct 1 14:34:05 2024 +0100 HADOOP-19294. NPE on maven enforcer with -Pnative on arm mac (#7082) Update maven-enforcer-plugin.version to 3.5.0 Contributed by Steve Loughran --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 3d027fa6704..16a3733b36b 100644 --- a/pom.xml +++ b/pom.xml @@ -100,7 +100,7 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x 1.7 2.4 3.0.2 -3.0.0 +3.5.0 2.0.0 3.0.1 1.5 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19294. NPE on maven enforcer with -Pnative on arm mac (#7082)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new bb9bbc06e86 HADOOP-19294. NPE on maven enforcer with -Pnative on arm mac (#7082) bb9bbc06e86 is described below commit bb9bbc06e86c7f1777c82f6da590201f7200569a Author: Steve Loughran AuthorDate: Tue Oct 1 14:34:05 2024 +0100 HADOOP-19294. NPE on maven enforcer with -Pnative on arm mac (#7082) Update maven-enforcer-plugin.version to 3.5.0 Contributed by Steve Loughran --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 5df95f2ea87..280121c8a0c 100644 --- a/pom.xml +++ b/pom.xml @@ -100,7 +100,7 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x 1.7 2.4 3.0.2 -3.0.0 +3.5.0 2.0.0 3.0.1 1.5 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19294. NPE on maven enforcer with -Pnative on arm mac (#7082)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 45b1c86fe51 HADOOP-19294. NPE on maven enforcer with -Pnative on arm mac (#7082) 45b1c86fe51 is described below commit 45b1c86fe51ec09fc0ffc1083a4996c34f49719b Author: Steve Loughran AuthorDate: Tue Oct 1 14:34:05 2024 +0100 HADOOP-19294. NPE on maven enforcer with -Pnative on arm mac (#7082) Update maven-enforcer-plugin.version to 3.5.0 Contributed by Steve Loughran --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 964c23c6cb5..1d4fda50677 100644 --- a/pom.xml +++ b/pom.xml @@ -100,7 +100,7 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/x 1.7 2.4 3.0.2 -3.0.0 +3.5.0 2.0.0 3.0.1 1.5 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.1 updated: HADOOP-19288. hadoop-client-runtime to exclude dnsjava InetAddressResolverProvider (#7070)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.1 by this push: new 842dd074107 HADOOP-19288. hadoop-client-runtime to exclude dnsjava InetAddressResolverProvider (#7070) 842dd074107 is described below commit 842dd074107382285d18dd1403e3aa06a0467c9e Author: cxzl25 <3898450+cxz...@users.noreply.github.com> AuthorDate: Tue Oct 1 21:48:48 2024 +0800 HADOOP-19288. hadoop-client-runtime to exclude dnsjava InetAddressResolverProvider (#7070) Contributed by dzcxzl. --- hadoop-client-modules/hadoop-client-runtime/pom.xml | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/hadoop-client-modules/hadoop-client-runtime/pom.xml b/hadoop-client-modules/hadoop-client-runtime/pom.xml index f1dc7062a32..5a55ef01a2e 100644 --- a/hadoop-client-modules/hadoop-client-runtime/pom.xml +++ b/hadoop-client-modules/hadoop-client-runtime/pom.xml @@ -229,8 +229,9 @@ jnamed* lookup* update* -META-INF/versions/21/* -META-INF/versions/21/**/* +META-INF/versions/18/* +META-INF/versions/18/**/* + META-INF/services/java.net.spi.InetAddressResolverProvider @@ -245,7 +246,7 @@ META-INF/versions/9/module-info.class META-INF/versions/11/module-info.class - META-INF/versions/21/module-info.class + META-INF/versions/18/module-info.class - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19288. hadoop-client-runtime to exclude dnsjava InetAddressResolverProvider (#7070)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 4ff0dceebda HADOOP-19288. hadoop-client-runtime to exclude dnsjava InetAddressResolverProvider (#7070) 4ff0dceebda is described below commit 4ff0dceebdabbbe697495a4bf8c72530440c2ff7 Author: cxzl25 <3898450+cxz...@users.noreply.github.com> AuthorDate: Tue Oct 1 21:48:48 2024 +0800 HADOOP-19288. hadoop-client-runtime to exclude dnsjava InetAddressResolverProvider (#7070) Contributed by dzcxzl. --- hadoop-client-modules/hadoop-client-runtime/pom.xml | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/hadoop-client-modules/hadoop-client-runtime/pom.xml b/hadoop-client-modules/hadoop-client-runtime/pom.xml index 8c72f53e918..ed410d25546 100644 --- a/hadoop-client-modules/hadoop-client-runtime/pom.xml +++ b/hadoop-client-modules/hadoop-client-runtime/pom.xml @@ -229,8 +229,9 @@ jnamed* lookup* update* -META-INF/versions/21/* -META-INF/versions/21/**/* +META-INF/versions/18/* +META-INF/versions/18/**/* + META-INF/services/java.net.spi.InetAddressResolverProvider @@ -245,7 +246,7 @@ META-INF/versions/9/module-info.class META-INF/versions/11/module-info.class - META-INF/versions/21/module-info.class + META-INF/versions/18/module-info.class - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19280. [ABFS] Initialize client timer only if metric collection is enabled (#7061)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 9aca73481e0 HADOOP-19280. [ABFS] Initialize client timer only if
metric collection is enabled (#7061)
9aca73481e0 is described below
commit 9aca73481e037e54c14e55e1b88ebe3a785b260c
Author: Manish Bhatt <52626736+bhattmanis...@users.noreply.github.com>
AuthorDate: Mon Sep 30 08:56:18 2024 -0700
HADOOP-19280. [ABFS] Initialize client timer only if metric collection is
enabled (#7061)
Contributed by Manish Bhatt
---
.../hadoop/fs/azurebfs/services/AbfsClient.java| 18 ++-
.../fs/azurebfs/services/TestAbfsClient.java | 141 +
2 files changed, 153 insertions(+), 6 deletions(-)
diff --git
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
index ca35015b193..034a7f9e8a4 100644
---
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
+++
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
@@ -131,6 +131,7 @@ import static
org.apache.hadoop.fs.azurebfs.services.RetryReasonConstants.CONNEC
public abstract class AbfsClient implements Closeable {
public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
public static final String HUNDRED_CONTINUE_USER_AGENT = SINGLE_WHITE_SPACE
+ HUNDRED_CONTINUE + SEMICOLON;
+ public static final String ABFS_CLIENT_TIMER_THREAD_NAME =
"abfs-timer-client";
private final URL baseUrl;
private final SharedKeyCredentials sharedKeyCredentials;
@@ -149,7 +150,7 @@ public abstract class AbfsClient implements Closeable {
private AccessTokenProvider tokenProvider;
private SASTokenProvider sasTokenProvider;
private final AbfsCounters abfsCounters;
- private final Timer timer;
+ private Timer timer;
private final String abfsMetricUrl;
private boolean isMetricCollectionEnabled = false;
private final MetricFormat metricFormat;
@@ -258,9 +259,9 @@ public abstract class AbfsClient implements Closeable {
throw new IOException("Exception while initializing metric credentials
" + e);
}
}
-this.timer = new Timer(
-"abfs-timer-client", true);
if (isMetricCollectionEnabled) {
+ this.timer = new Timer(
+ ABFS_CLIENT_TIMER_THREAD_NAME, true);
timer.schedule(new TimerTaskImpl(),
metricIdlePeriod,
metricIdlePeriod);
@@ -292,9 +293,9 @@ public abstract class AbfsClient implements Closeable {
@Override
public void close() throws IOException {
-if (runningTimerTask != null) {
+if (isMetricCollectionEnabled && runningTimerTask != null) {
runningTimerTask.cancel();
- timer.purge();
+ timer.cancel();
}
if (keepAliveCache != null) {
keepAliveCache.close();
@@ -1418,7 +1419,7 @@ public abstract class AbfsClient implements Closeable {
boolean timerOrchestrator(TimerFunctionality timerFunctionality, TimerTask
timerTask) {
switch (timerFunctionality) {
case RESUME:
-if (isMetricCollectionStopped.get()) {
+if (isMetricCollectionEnabled && isMetricCollectionStopped.get()) {
synchronized (this) {
if (isMetricCollectionStopped.get()) {
resumeTimer();
@@ -1597,6 +1598,11 @@ public abstract class AbfsClient implements Closeable {
return keepAliveCache;
}
+ @VisibleForTesting
+ protected Timer getTimer() {
+return timer;
+ }
+
protected String getUserAgent() {
return userAgent;
}
diff --git
a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsClient.java
b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsClient.java
new file mode 100644
index 000..e8ab4291b32
--- /dev/null
+++
b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsClient.java
@@ -0,0 +1,141 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDIT
(hadoop-release-support) branch main updated: HADOOP-19087 3.4.1 RC2 validation
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hadoop-release-support.git
The following commit(s) were added to refs/heads/main by this push:
new 797ff51 HADOOP-19087 3.4.1 RC2 validation
797ff51 is described below
commit 797ff5127fb06cc5a4f35804087cb3ce1f9eb37d
Author: Steve Loughran
AuthorDate: Fri Sep 27 19:14:36 2024 +0100
HADOOP-19087 3.4.1 RC2 validation
---
README.md | 83 +
build.xml | 114 +++--
doc/thirdparty.md | 75 +++
src/releases/release-info-3.4.1.properties | 8 +-
4 files changed, 222 insertions(+), 58 deletions(-)
diff --git a/README.md b/README.md
index 374b564..969718e 100644
--- a/README.md
+++ b/README.md
@@ -104,7 +104,7 @@ Ant uses this to to set the property `release.info.file` to
the path
`src/releases/release-info-${release.version}.properties`
```properties
-release.info.file=src/releases/release-info-3.4.0.prpoperties
+release.info.file=src/releases/release-info-3.4.1.prpoperties
```
This is then loaded, with the build failing if it is not found.
@@ -478,8 +478,10 @@ This will import all the KEYS from
then verify the signature of each downloaded file.
If you don't yet trust the key of whoever signed the release then
-now is the time to use the keytool to declare that you trust them
--after performing whatever verification you consider is wise.
+1. In your keychain app: update the keys from the server, to see
+ if they've been signed by others.
+2. Perform whatever key verification you can and sign the key that
+ level -ideally push up the signature to the servers.
### untar source and build.
@@ -498,7 +500,8 @@ ant release.src.untar release.src.build
```bash
ant release.site.untar release.site.validate
```
-
+Validation is pretty minimal; it just looks for the existence
+of index.html files in the site root and under api/.
### untar binary release
@@ -516,7 +519,7 @@ ant release.bin.commands
```
This will fail on a platform where the native binaries don't load,
-unless the checknative command has been disabled.
+unless the `hadoop checknative` command has been disabled.
This can be done in `build.properties`
@@ -528,6 +531,12 @@ check.native.binaries=false
ant release.bin.commands -Dcheck.native.binaries=false
```
+If `check.native.binaries` is false, the `bin/hadoop checknative`
+is still executed, with the outcome printed (reporting a failure if
+the binaries are not present).
+
+The and build itself is successful.
+
## Testing ARM binaries
```bash
@@ -537,10 +546,14 @@ ant release.arm.commands
# Testing on a remote server
-Currently the way to do this is to clone the hadoop-release
+Currently the way to do this is to clone this hadoop-release-support
repository to the remote server and run the validation
commands there.
+```sh
+git clone https://github.com/apache/hadoop-release-support.git
+```
+
# Building and testing projects from the staged maven artifacts
A lot of the targets build maven projects from the staged maven artifacts.
@@ -554,7 +567,12 @@ For this to work
on their own branches.
4. Some projects need java11 or later.
-First, purge your maven repository of all hadoop- JAR files of the
+Some of these builds/tests are slow, but they can all be executed in parallel
unless
+you are actually trying to transitively build components, such as run spark
tests with
+the parquet artifact you build with the RC.
+If you find yourself doing this: you've just become a CI system without the
automation.
+
+First, purge your maven repository of all `hadoop-` JAR files of the
pending release version
```bash
@@ -576,7 +594,9 @@ To see the dependencies of the maven project:
ant mvn-validate-dependencies
```
-This saves the output to the file `target/mvndeps.txt`.
+This saves the output to the file `target/mvndeps.txt` and explicitly
+checks for some known "forbidden" artifacts that must not be exported
+as transitive dependencies.
Review this to make sure there are no unexpected artifacts coming in,
@@ -595,7 +615,6 @@ Note: this does not include the AWS V1 SDK `-Pextra`
profile.
[Big Data Interop](https://github.com/GoogleCloudPlatform/bigdata-interop).
* This is java 11+ only.
-* currently only builds against AWS v1 SDK.
Ideally, you should run the tests, or even better, run them before the RC is
up for review.
@@ -639,21 +658,9 @@ This independent module tests the s3a, gcs and abfs
connectors,
and associated committers, through the spark RDD and SQL APIs.
-[cloud integration](https://github.com/hortonworks-spark/cloud-integration)
-```bash
-ant cloud-examples.build
-ant cloud-examples.test
-```
-
-
-The test run is fairly tricky to get running; don't try and do thi
(hadoop) branch branch-3.4 updated: HADOOP-19281. MetricsSystemImpl should not print INFO message in CLI (#7071)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new afe644c88dd HADOOP-19281. MetricsSystemImpl should not print INFO
message in CLI (#7071)
afe644c88dd is described below
commit afe644c88dda89060b0f8678d12ee3bf637a30b9
Author: Sarveksha Yeshavantha Raju
<79865743+sarveksh...@users.noreply.github.com>
AuthorDate: Fri Sep 27 18:50:11 2024 +0530
HADOOP-19281. MetricsSystemImpl should not print INFO message in CLI (#7071)
Replaced all LOG.info with LOG.debug
Contributed by Sarveksha Yeshavantha Raju
---
.../apache/hadoop/metrics2/impl/MetricsConfig.java| 14 +-
.../hadoop/metrics2/impl/MetricsSystemImpl.java | 19 +--
2 files changed, 14 insertions(+), 19 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
index f4848fed519..3ebcb9ee69f 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
@@ -26,12 +26,12 @@ import static java.security.AccessController.*;
import java.nio.charset.StandardCharsets;
import java.security.PrivilegedAction;
+import java.util.Arrays;
import java.util.Iterator;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-import org.apache.hadoop.thirdparty.com.google.common.base.Joiner;
import org.apache.hadoop.thirdparty.com.google.common.base.Splitter;
import org.apache.hadoop.thirdparty.com.google.common.collect.Iterables;
import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
@@ -106,8 +106,8 @@ class MetricsConfig extends SubsetConfiguration {
/**
* Load configuration from a list of files until the first successful load
- * @param conf the configuration object
- * @param files the list of filenames to try
+ * @param prefix The prefix of the configuration.
+ * @param fileNames the list of filenames to try.
* @return the configuration object
*/
static MetricsConfig loadFirst(String prefix, String... fileNames) {
@@ -119,10 +119,7 @@ class MetricsConfig extends SubsetConfiguration {
fh.setFileName(fname);
fh.load();
Configuration cf = pcf.interpolatedConfiguration();
-LOG.info("Loaded properties from {}", fname);
-if (LOG.isDebugEnabled()) {
- LOG.debug("Properties: {}", toString(cf));
-}
+LOG.debug("Loaded properties from {}: {}", fname, cf);
MetricsConfig mc = new MetricsConfig(cf, prefix);
LOG.debug("Metrics Config: {}", mc);
return mc;
@@ -135,8 +132,7 @@ class MetricsConfig extends SubsetConfiguration {
throw new MetricsConfigException(e);
}
}
-LOG.warn("Cannot locate configuration: tried " +
- Joiner.on(",").join(fileNames));
+LOG.debug("Cannot locate configuration: tried {}",
Arrays.asList(fileNames));
// default to an empty configuration
return new MetricsConfig(new PropertiesConfiguration(), prefix);
}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
index 6c5a71a708f..09390f46194 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
@@ -155,7 +155,7 @@ public class MetricsSystemImpl extends MetricsSystem
implements MetricsSource {
++refCount;
if (monitoring) {
// in mini cluster mode
- LOG.info(this.prefix +" metrics system started (again)");
+ LOG.debug("{} metrics system started (again)", prefix);
return this;
}
switch (initMode()) {
@@ -169,7 +169,7 @@ public class MetricsSystemImpl extends MetricsSystem
implements MetricsSource {
}
break;
case STANDBY:
-LOG.info(prefix +" metrics system started in standby mode");
+LOG.debug("{} metrics system started in standby mode", prefix);
}
initSystemMBean();
return this;
@@ -188,7 +188,7 @@ public class MetricsSystemImpl extends MetricsSystem
implements MetricsSource {
configure(prefix);
startTimer();
monitoring = true;
-LOG.info(prefix +" metrics system started");
+LOG.debug("{}
(hadoop) branch trunk updated: HADOOP-19281. MetricsSystemImpl should not print INFO message in CLI (#7071)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 01401d71eff HADOOP-19281. MetricsSystemImpl should not print INFO
message in CLI (#7071)
01401d71eff is described below
commit 01401d71efff752628215e0eb5c36b7f62837847
Author: Sarveksha Yeshavantha Raju
<79865743+sarveksh...@users.noreply.github.com>
AuthorDate: Fri Sep 27 18:50:11 2024 +0530
HADOOP-19281. MetricsSystemImpl should not print INFO message in CLI (#7071)
Replaced all LOG.info with LOG.debug
Contributed by Sarveksha Yeshavantha Raju
---
.../apache/hadoop/metrics2/impl/MetricsConfig.java| 14 +-
.../hadoop/metrics2/impl/MetricsSystemImpl.java | 19 +--
2 files changed, 14 insertions(+), 19 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
index f4848fed519..3ebcb9ee69f 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
@@ -26,12 +26,12 @@ import static java.security.AccessController.*;
import java.nio.charset.StandardCharsets;
import java.security.PrivilegedAction;
+import java.util.Arrays;
import java.util.Iterator;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-import org.apache.hadoop.thirdparty.com.google.common.base.Joiner;
import org.apache.hadoop.thirdparty.com.google.common.base.Splitter;
import org.apache.hadoop.thirdparty.com.google.common.collect.Iterables;
import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
@@ -106,8 +106,8 @@ class MetricsConfig extends SubsetConfiguration {
/**
* Load configuration from a list of files until the first successful load
- * @param conf the configuration object
- * @param files the list of filenames to try
+ * @param prefix The prefix of the configuration.
+ * @param fileNames the list of filenames to try.
* @return the configuration object
*/
static MetricsConfig loadFirst(String prefix, String... fileNames) {
@@ -119,10 +119,7 @@ class MetricsConfig extends SubsetConfiguration {
fh.setFileName(fname);
fh.load();
Configuration cf = pcf.interpolatedConfiguration();
-LOG.info("Loaded properties from {}", fname);
-if (LOG.isDebugEnabled()) {
- LOG.debug("Properties: {}", toString(cf));
-}
+LOG.debug("Loaded properties from {}: {}", fname, cf);
MetricsConfig mc = new MetricsConfig(cf, prefix);
LOG.debug("Metrics Config: {}", mc);
return mc;
@@ -135,8 +132,7 @@ class MetricsConfig extends SubsetConfiguration {
throw new MetricsConfigException(e);
}
}
-LOG.warn("Cannot locate configuration: tried " +
- Joiner.on(",").join(fileNames));
+LOG.debug("Cannot locate configuration: tried {}",
Arrays.asList(fileNames));
// default to an empty configuration
return new MetricsConfig(new PropertiesConfiguration(), prefix);
}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
index 6c5a71a708f..09390f46194 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
@@ -155,7 +155,7 @@ public class MetricsSystemImpl extends MetricsSystem
implements MetricsSource {
++refCount;
if (monitoring) {
// in mini cluster mode
- LOG.info(this.prefix +" metrics system started (again)");
+ LOG.debug("{} metrics system started (again)", prefix);
return this;
}
switch (initMode()) {
@@ -169,7 +169,7 @@ public class MetricsSystemImpl extends MetricsSystem
implements MetricsSource {
}
break;
case STANDBY:
-LOG.info(prefix +" metrics system started in standby mode");
+LOG.debug("{} metrics system started in standby mode", prefix);
}
initSystemMBean();
return this;
@@ -188,7 +188,7 @@ public class MetricsSystemImpl extends MetricsSystem
implements MetricsSource {
configure(prefix);
startTimer();
monitoring = true;
-LOG.info(prefix +" metrics system started");
+LOG.debug("{}
(hadoop-thirdparty) branch trunk updated: [BUILD] upgrade GitHub actions because old versions no longer work reliably (#39)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git The following commit(s) were added to refs/heads/trunk by this push: new 95fb8f3 [BUILD] upgrade GitHub actions because old versions no longer work reliably (#39) 95fb8f3 is described below commit 95fb8f36449a170fd569c7bab7ddf6950a6008f2 Author: PJ Fanning AuthorDate: Wed Sep 25 20:14:09 2024 +0100 [BUILD] upgrade GitHub actions because old versions no longer work reliably (#39) Contributed by PJ Fanning --- .github/workflows/build.yml| 6 +++--- .github/workflows/dependency_check.yml | 8 .github/workflows/license_check.yml| 8 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index ae1d9cf..3822e73 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -24,12 +24,12 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout source - uses: actions/checkout@v2 + uses: actions/checkout@v4 - name: Set up JDK 8 - uses: actions/setup-java@v2 + uses: actions/setup-java@v4 with: java-version: '8' -distribution: 'adopt' +distribution: 'temurin' - name: Build with Maven run: mvn install diff --git a/.github/workflows/dependency_check.yml b/.github/workflows/dependency_check.yml index fca9322..d282660 100644 --- a/.github/workflows/dependency_check.yml +++ b/.github/workflows/dependency_check.yml @@ -24,16 +24,16 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout source - uses: actions/checkout@v2 + uses: actions/checkout@v4 - name: Set up JDK 8 - uses: actions/setup-java@v2 + uses: actions/setup-java@v4 with: java-version: '8' -distribution: 'adopt' +distribution: 'temurin' - name: Dependency check with Maven run: mvn dependency-check:aggregate - name: Upload the report - uses: actions/upload-artifact@v2 + uses: actions/upload-artifact@v4 with: name: dependency-check-report path: "**/target/dependency-check-report.html" diff --git a/.github/workflows/license_check.yml b/.github/workflows/license_check.yml index 7cf5734..9e440ab 100644 --- a/.github/workflows/license_check.yml +++ b/.github/workflows/license_check.yml @@ -24,16 +24,16 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout source - uses: actions/checkout@v2 + uses: actions/checkout@v4 - name: Set up JDK 8 - uses: actions/setup-java@v2 + uses: actions/setup-java@v4 with: java-version: '8' -distribution: 'adopt' +distribution: 'temurin' - name: License check with Maven run: mvn apache-rat:check - name: Upload the report - uses: actions/upload-artifact@v2 + uses: actions/upload-artifact@v4 with: name: license-check-report path: "**/target/rat.txt" - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HDFS-17381. Distcp of EC files should not be limited to DFS. (#6551)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 49a495803a9 HDFS-17381. Distcp of EC files should not be limited to
DFS. (#6551)
49a495803a9 is described below
commit 49a495803a9451850b8982317e277b605c785587
Author: Sadanand Shenoy
AuthorDate: Wed Sep 25 22:24:09 2024 +0530
HDFS-17381. Distcp of EC files should not be limited to DFS. (#6551)
Contributed by Sadanand Shenoy
---
.../main/java/org/apache/hadoop/fs/FileUtil.java | 19 +++
.../main/java/org/apache/hadoop/fs/Options.java| 5 +
.../org/apache/hadoop/fs/WithErasureCoding.java| 50
.../apache/hadoop/hdfs/DistributedFileSystem.java | 16 ++-
.../hadoop/hdfs/client/DfsPathCapabilities.java| 2 +
.../org/apache/hadoop/tools/mapred/CopyMapper.java | 9 +-
.../mapred/RetriableDirectoryCreateCommand.java| 33 --
.../tools/mapred/RetriableFileCopyCommand.java | 43 +++
.../hadoop/tools/TestDistCpWithRawXAttrs.java | 130 +
9 files changed, 271 insertions(+), 36 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
index 39dac46dd54..ebb3763f7a3 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
@@ -2108,4 +2108,23 @@ public class FileUtil {
LOG.info("Ignoring missing directory {}", path);
LOG.debug("Directory missing", e);
}
+
+ /**
+ * Return true if the FS implements {@link WithErasureCoding} and
+ * supports EC_POLICY option in {@link Options.OpenFileOptions}.
+ * A message is logged when the filesystem does not support Erasure coding.
+ * @param fs filesystem
+ * @param path path
+ * @return true if the Filesystem supports EC
+ * @throws IOException if there is a failure in hasPathCapability call
+ */
+ public static boolean checkFSSupportsEC(FileSystem fs, Path path) throws
IOException {
+if (fs instanceof WithErasureCoding &&
+fs.hasPathCapability(path,
Options.OpenFileOptions.FS_OPTION_OPENFILE_EC_POLICY)) {
+ return true;
+}
+LOG.warn("Filesystem with scheme {} does not support Erasure Coding" +
+" at path {}", fs.getScheme(), path);
+return false;
+ }
}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java
index b59d2f3be15..f473e9427ba 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java
@@ -704,5 +704,10 @@ public final class Options {
FS_OPTION_OPENFILE_READ_POLICY_WHOLE_FILE)
.collect(Collectors.toSet()));
+/**
+ * EC policy to be set on the file that needs to be created : {@value}.
+ */
+public static final String FS_OPTION_OPENFILE_EC_POLICY =
+FS_OPTION_OPENFILE + "ec.policy";
}
}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/WithErasureCoding.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/WithErasureCoding.java
new file mode 100644
index 000..5f8a7fbad6e
--- /dev/null
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/WithErasureCoding.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.io.IOException;
+
+/**
+ * Filesystems that support EC can implement this interface.
+ */
+public interface WithErasureCoding {
+
+ /**
+ * Get the EC Policy name of the given file's fileStatus.
+ * If the file is not erasure coded, this shall return null.
+ * Call
(hadoop) branch trunk updated: HADOOP-15760. Upgrade commons-collections to commons-collections4 (#7006)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new e602c601ddd HADOOP-15760. Upgrade commons-collections to commons-collections4 (#7006) e602c601ddd is described below commit e602c601ddd98edf098dd9f13811846756cac9c3 Author: Nihal Jain AuthorDate: Tue Sep 24 21:20:22 2024 +0530 HADOOP-15760. Upgrade commons-collections to commons-collections4 (#7006) This moves Hadoop to Apache commons-collections4. Apache commons-collections has been removed and is completely banned from the source code. Contributed by Nihal Jain --- LICENSE-binary| 3 +-- .../hadoop-client-minicluster/pom.xml | 4 hadoop-common-project/hadoop-auth/pom.xml | 4 hadoop-common-project/hadoop-common/pom.xml | 10 -- .../java/org/apache/hadoop/conf/Configuration.java| 6 +++--- .../src/main/java/org/apache/hadoop/fs/FileUtil.java | 2 +- .../hadoop/security/JniBasedUnixGroupsMapping.java| 2 +- .../java/org/apache/hadoop/hdfs/DFSUtilClient.java| 2 +- .../token/delegation/DelegationTokenIdentifier.java | 2 +- .../hadoop/hdfs/shortcircuit/ShortCircuitCache.java | 2 +- .../federation/resolver/order/RandomResolver.java | 2 +- .../server/federation/metrics/TestRBFMetrics.java | 2 +- .../hadoop/hdfs/server/datanode/DirectoryScanner.java | 2 +- .../fsdataset/impl/ReplicaCachingGetSpaceUsed.java| 2 +- .../hadoop/fs/TestEnhancedByteBufferAccess.java | 2 +- .../namenode/snapshot/TestSnapshotDiffReport.java | 2 +- .../hdfs/shortcircuit/TestShortCircuitCache.java | 2 +- .../main/java/org/apache/hadoop/mapred/Counters.java | 2 +- .../hadoop-mapreduce-client/pom.xml | 4 ++-- hadoop-project/pom.xml| 19 +++ .../hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java | 2 +- .../fs/aliyun/oss/AliyunOSSFileSystemStore.java | 2 +- .../hadoop/fs/s3a/impl/CopyFromLocalOperation.java| 2 +- .../java/org/apache/hadoop/yarn/sls/SLSRunner.java| 2 +- .../hadoop-yarn-applications-catalog-webapp/pom.xml | 8 .../org/apache/hadoop/yarn/client/cli/RouterCLI.java | 4 ++-- .../pom.xml | 4 ++-- .../yarn/server/timeline/LeveldbTimelineStore.java| 2 +- .../server/timeline/RollingLevelDBTimelineStore.java | 2 +- .../server/timeline/security/TimelineACLsManager.java | 2 +- .../amrmproxy/LocalityMulticastAMRMProxyPolicy.java | 2 +- .../federation/policies/dao/WeightedPolicyInfo.java | 2 +- .../utils/FederationPolicyStoreInputValidator.java| 2 +- .../federation/utils/FederationRegistryClient.java| 2 +- .../federation/utils/FederationStateStoreFacade.java | 4 ++-- .../policygenerator/LoadBasedGlobalPolicy.java| 2 +- .../yarn/server/resourcemanager/NodesListManager.java | 2 +- .../resourcemanager/ResourceTrackerService.java | 2 +- .../ProportionalCapacityPreemptionPolicy.java | 2 +- .../server/resourcemanager/rmnode/RMNodeImpl.java | 2 +- .../scheduler/activities/ActivitiesManager.java | 2 +- .../scheduler/activities/AppAllocation.java | 2 +- .../CapacitySchedulerQueueCapacityHandler.java| 2 +- .../scheduler/placement/AppPlacementAllocator.java| 2 +- .../scheduler/placement/MultiNodeSortingManager.java | 2 +- .../hadoop/yarn/server/router/RouterServerUtil.java | 2 +- .../router/rmadmin/FederationRMAdminInterceptor.java | 4 ++-- .../hadoop/yarn/server/router/webapp/AppsBlock.java | 2 +- .../router/webapp/FederationInterceptorREST.java | 2 +- .../hadoop/yarn/server/router/webapp/NodesBlock.java | 2 +- .../hadoop/yarn/server/router/webapp/RouterBlock.java | 2 +- .../router/clientrm/TestRouterYarnClientUtils.java| 2 +- .../router/clientrm/TestSequentialRouterPolicy.java | 2 +- .../rmadmin/TestableFederationRMAdminInterceptor.java | 2 +- .../yarn/server/router/secure/TestSecureLogins.java | 2 +- .../router/subcluster/TestFederationSubCluster.java | 2 +- .../pom.xml | 4 57 files changed, 101 insertions(+), 65 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 39c567fb01f..ada9deaff23 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -246,7 +246,7 @@ com.zaxxer:HikariCP:4.0.3 commons-beanutils:commons-beanutils:1.9.4 commons-cli:commons-cli:1.5.0 commons-codec:commons-codec:1.15 -commons-collections:commons-collections:3.2.2 +org.apache.commons:commons-collections4:4.4 commons-daemon:commons-daemon:1.0.13 commons-io:commons-io:2.16.1 commons-net:commons-net:3.9.0 @@ -299,7 +299,6 @@ net.java.dev.jna:jna:5.2.0
(hadoop) branch branch-3.4 updated: HADOOP-19272. S3A: AWS SDK 2.25.53 warnings logged by transfer manager (#7048)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new f3600d28ab9 HADOOP-19272. S3A: AWS SDK 2.25.53 warnings logged by
transfer manager (#7048)
f3600d28ab9 is described below
commit f3600d28ab9a396e628034bad3aabfa212fe4339
Author: Steve Loughran
AuthorDate: Thu Sep 19 13:50:06 2024 +0100
HADOOP-19272. S3A: AWS SDK 2.25.53 warnings logged by transfer manager
(#7048)
Disables all logging below error in the AWS SDK Transfer Manager.
This is done in ClientManagerImpl construction so is automatically done
during S3A FS initialization.
ITests verify that
* It is possible to restore the warning log. This verifies the validity of
the test suite, and will identify when an SDK update fixes this
regression.
* Constructing an S3A FS instance will disable the logging.
The log manipulation code is lifted from Cloudstore, where it was used to
dynamically enable logging. It uses reflection to load the Log4J binding;
all uses of the API catch and swallow exceptions.
This is needed to avoid failures when running against different log backends
This is an emergency fix -we could come up with a better design for
the reflection based code using the new DynMethods classes.
But this is based on working code, which is always good.
Contributed by Steve Loughran
---
.../hadoop/fs/s3a/impl/AwsSdkWorkarounds.java | 59 ++
.../hadoop/fs/s3a/impl/ClientManagerImpl.java | 6 +
.../fs/s3a/impl/logging/Log4JController.java | 52 +
.../hadoop/fs/s3a/impl/logging/LogControl.java | 92 +
.../fs/s3a/impl/logging/LogControllerFactory.java | 98 ++
.../hadoop/fs/s3a/impl/logging/package-info.java | 26 +++
.../hadoop/fs/s3a/impl/ITestAwsSdkWorkarounds.java | 160 +++
.../s3a/impl/logging/TestLogControllerFactory.java | 214 +
8 files changed, 707 insertions(+)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AwsSdkWorkarounds.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AwsSdkWorkarounds.java
new file mode 100644
index 000..a0673b123b2
--- /dev/null
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AwsSdkWorkarounds.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.s3a.impl.logging.LogControl;
+import org.apache.hadoop.fs.s3a.impl.logging.LogControllerFactory;
+
+/**
+ * This class exists to support workarounds for parts of the AWS SDK
+ * which have caused problems.
+ */
+public final class AwsSdkWorkarounds {
+
+ /**
+ * Transfer manager log name. See HADOOP-19272.
+ * {@value}.
+ */
+ public static final String TRANSFER_MANAGER =
+ "software.amazon.awssdk.transfer.s3.S3TransferManager";
+
+ private AwsSdkWorkarounds() {
+ }
+
+ /**
+ * Prepare logging before creating AWS clients.
+ * @return true if the log tuning operation took place.
+ */
+ public static boolean prepareLogging() {
+return LogControllerFactory.createController().
+setLogLevel(TRANSFER_MANAGER, LogControl.LogLevel.ERROR);
+ }
+
+ /**
+ * Restore all noisy logs to INFO.
+ * @return true if the restoration operation took place.
+ */
+ @VisibleForTesting
+ static boolean restoreNoisyLogging() {
+return LogControllerFactory.createController().
+setLogLevel(TRANSFER_MANAGER, LogControl.LogLevel.INFO);
+ }
+}
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ClientManagerImpl.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ClientManagerImpl.java
index 4b2fc1c599b..24c37cc564a 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ClientManagerImpl.java
+++
b/hadoop-tools/hadoop-aws/src/main/java
(hadoop) branch trunk updated (d1311e52f78 -> ee2e5ac4e41)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
from d1311e52f78 YARN-11709. NodeManager should be marked unhealthy on
localizer config issues (#7043)
add ee2e5ac4e41 HADOOP-19272. S3A: AWS SDK 2.25.53 warnings logged by
transfer manager (#7048)
No new revisions were added by this update.
Summary of changes:
.../hadoop/fs/s3a/impl/AwsSdkWorkarounds.java | 59 ++
.../hadoop/fs/s3a/impl/ClientManagerImpl.java | 6 +
.../fs/s3a/impl/logging/Log4JController.java | 52 +
.../hadoop/fs/s3a/impl/logging/LogControl.java | 92 +
.../fs/s3a/impl/logging/LogControllerFactory.java | 98 ++
.../hadoop/fs/s3a/impl/logging}/package-info.java | 8 +-
.../hadoop/fs/s3a/impl/ITestAwsSdkWorkarounds.java | 160 +++
.../s3a/impl/logging/TestLogControllerFactory.java | 214 +
8 files changed, 686 insertions(+), 3 deletions(-)
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AwsSdkWorkarounds.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/logging/Log4JController.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/logging/LogControl.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/logging/LogControllerFactory.java
copy
{hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service
=>
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/logging}/package-info.java
(85%)
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestAwsSdkWorkarounds.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/logging/TestLogControllerFactory.java
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.1 updated: Revert "HADOOP-19195. S3A: Upgrade aws sdk v2 to 2.25.53 (#6900)"
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.1 by this push: new a9fb087ff41 Revert "HADOOP-19195. S3A: Upgrade aws sdk v2 to 2.25.53 (#6900)" a9fb087ff41 is described below commit a9fb087ff4107eb49b7b12456ebccd4a03bd2d89 Author: Steve Loughran AuthorDate: Mon Sep 16 14:21:34 2024 +0100 Revert "HADOOP-19195. S3A: Upgrade aws sdk v2 to 2.25.53 (#6900)" This reverts commit fc86a52c884f15b2f2fb401bbf0baaa36a057651. This rollback is due to: HADOOP-19272. S3A: AWS SDK 2.25.53 warnings logged about transfer manager not using CRT client Change-Id: I324f75d62daa02650ff9d199a2e0fc465a2ea28a --- LICENSE-binary | 2 +- hadoop-project/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 3e5698dea96..b064b6a15d1 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -362,7 +362,7 @@ org.objenesis:objenesis:2.6 org.xerial.snappy:snappy-java:1.1.10.4 org.yaml:snakeyaml:2.0 org.wildfly.openssl:wildfly-openssl:1.1.3.Final -software.amazon.awssdk:bundle:2.25.53 +software.amazon.awssdk:bundle:jar:2.24.6 diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 6416cc8b7dc..bcba56eced6 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -187,7 +187,7 @@ 1.0-beta-1 900 1.12.720 -2.25.53 +2.24.6 1.0.1 2.7.1 1.11.2 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.1 updated: HADOOP-19201. S3A. Support external-id in assume role (#6876)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.1 by this push:
new 87d9bb60229 HADOOP-19201. S3A. Support external-id in assume role
(#6876)
87d9bb60229 is described below
commit 87d9bb602290063f324ca09702bb030bd3fbbba6
Author: Smith Cruise
AuthorDate: Tue Sep 10 22:38:32 2024 +0800
HADOOP-19201. S3A. Support external-id in assume role (#6876)
The option fs.s3a.assumed.role.external.id sets the
external id for calls of AssumeRole to the STS service
Contributed by Smith Cruise
---
.../src/main/java/org/apache/hadoop/fs/s3a/Constants.java | 5 +
.../apache/hadoop/fs/s3a/auth/AssumedRoleCredentialProvider.java | 5 +
.../src/site/markdown/tools/hadoop-aws/assumed_roles.md | 8
3 files changed, 18 insertions(+)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
index 5ce1b49864a..7e614bc11d6 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
@@ -94,6 +94,11 @@ public final class Constants {
public static final String ASSUMED_ROLE_ARN =
"fs.s3a.assumed.role.arn";
+ /**
+ * external id for assume role request: {@value}.
+ */
+ public static final String ASSUMED_ROLE_EXTERNAL_ID =
"fs.s3a.assumed.role.external.id";
+
/**
* Session name for the assumed role, must be valid characters according
* to the AWS APIs: {@value}.
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AssumedRoleCredentialProvider.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AssumedRoleCredentialProvider.java
index c2ac8fe4c81..ce20684feca 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AssumedRoleCredentialProvider.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AssumedRoleCredentialProvider.java
@@ -125,6 +125,7 @@ public final class AssumedRoleCredentialProvider implements
AwsCredentialsProvid
duration = conf.getTimeDuration(ASSUMED_ROLE_SESSION_DURATION,
ASSUMED_ROLE_SESSION_DURATION_DEFAULT, TimeUnit.SECONDS);
String policy = conf.getTrimmed(ASSUMED_ROLE_POLICY, "");
+String externalId = conf.getTrimmed(ASSUMED_ROLE_EXTERNAL_ID, "");
LOG.debug("{}", this);
@@ -132,6 +133,10 @@ public final class AssumedRoleCredentialProvider
implements AwsCredentialsProvid
AssumeRoleRequest.builder().roleArn(arn).roleSessionName(sessionName)
.durationSeconds((int) duration);
+if (StringUtils.isNotEmpty(externalId)) {
+ requestBuilder.externalId(externalId);
+}
+
if (StringUtils.isNotEmpty(policy)) {
LOG.debug("Scope down policy {}", policy);
requestBuilder.policy(policy);
diff --git
a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md
b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md
index 065a757f217..ba1bc4b362c 100644
---
a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md
+++
b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md
@@ -153,6 +153,14 @@ Here are the full set of configuration options.
+
+ fs.s3a.assumed.role.external.id
+ arbitrary value, specific by user in AWS console
+
+External id for assumed role, it's an optional configuration.
"https://aws.amazon.com/cn/blogs/security/how-to-use-external-id-when-granting-access-to-your-aws-resources/";
+
+
+
fs.s3a.assumed.role.policy
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (8c41fbcaf54 -> 6881d12da4b)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 8c41fbcaf54 Revert "YARN-11709. NodeManager should be shut down or blacklisted when it ca…" (#7028) add 6881d12da4b HADOOP-19262: Upgrade wildfly-openssl:1.1.3.Final to 2.1.4.Final to support Java17+ (#7026) No new revisions were added by this update. Summary of changes: LICENSE-binary | 2 +- hadoop-project/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.1 updated: HADOOP-19252. Upgrade hadoop-thirdparty from 1.2.0 to 1.3.0 (#7007) (#7014)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.1 by this push:
new 5db50fb995b HADOOP-19252. Upgrade hadoop-thirdparty from 1.2.0 to
1.3.0 (#7007) (#7014)
5db50fb995b is described below
commit 5db50fb995b29bb301546b98866beb7b48cc8405
Author: Steve Loughran
AuthorDate: Thu Sep 5 20:53:13 2024 +0100
HADOOP-19252. Upgrade hadoop-thirdparty from 1.2.0 to 1.3.0 (#7007) (#7014)
Update the version of hadoop-thirdparty to 1.3.0
across all shaded artifacts used.
This synchronizes the shaded protobuf library with those of
all other shaded artifacts (guava, avro)
Note: this patch moves from 1.2.0; the trunk PR moves from
1.3.0-SNAPSHOT and is slightly different
Contributed by Steve Loughran
---
LICENSE-binary | 15 ---
hadoop-common-project/hadoop-common/pom.xml | 2 +-
hadoop-project/pom.xml | 6 +++---
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml | 6 +++---
4 files changed, 15 insertions(+), 14 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index 887e7070967..3e5698dea96 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -232,19 +232,19 @@ com.google:guice:4.0
com.google:guice-servlet:4.0
com.google.api.grpc:proto-google-common-protos:1.0.0
com.google.code.gson:2.9.0
-com.google.errorprone:error_prone_annotations:2.2.0
-com.google.j2objc:j2objc-annotations:1.1
+com.google.errorprone:error_prone_annotations:2.5.1
+com.google.j2objc:j2objc-annotations:1.3
com.google.json-simple:json-simple:1.1.1
com.google.guava:failureaccess:1.0
com.google.guava:guava:20.0
-com.google.guava:guava:27.0-jre
+com.google.guava:guava:32.0.1-jre
com.google.guava:listenablefuture:.0-empty-to-avoid-conflict-with-guava
com.microsoft.azure:azure-storage:7.0.0
com.nimbusds:nimbus-jose-jwt:9.37.2
com.zaxxer:HikariCP:4.0.3
commons-beanutils:commons-beanutils:1.9.4
commons-cli:commons-cli:1.5.0
-commons-codec:commons-codec:1.11
+commons-codec:commons-codec:1.15
commons-collections:commons-collections:3.2.2
commons-daemon:commons-daemon:1.0.13
commons-io:commons-io:2.16.1
@@ -297,6 +297,7 @@ javax.inject:javax.inject:1
net.java.dev.jna:jna:5.2.0
net.minidev:accessors-smart:1.2
org.apache.avro:avro:1.9.2
+org.apache.avro:avro:1.11.3
org.apache.commons:commons-collections4:4.2
org.apache.commons:commons-compress:1.26.1
org.apache.commons:commons-configuration2:2.10.1
@@ -361,7 +362,7 @@ org.objenesis:objenesis:2.6
org.xerial.snappy:snappy-java:1.1.10.4
org.yaml:snakeyaml:2.0
org.wildfly.openssl:wildfly-openssl:1.1.3.Final
-software.amazon.awssdk:bundle:jar:2.25.53
+software.amazon.awssdk:bundle:2.25.53
@@ -394,7 +395,7 @@
hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/d3-3.5.17.min.js
leveldb v1.13
com.google.protobuf:protobuf-java:2.5.0
-com.google.protobuf:protobuf-java:3.6.1
+com.google.protobuf:protobuf-java:3.25.3
com.google.re2j:re2j:1.1
com.jcraft:jsch:0.1.55
com.thoughtworks.paranamer:paranamer:2.3
@@ -484,7 +485,7 @@ com.microsoft.sqlserver:mssql-jdbc:6.2.1.jre7
org.bouncycastle:bcpkix-jdk18on:1.78.1
org.bouncycastle:bcprov-jdk18on:1.78.1
org.bouncycastle:bcutil-jdk18on:1.78.1
-org.checkerframework:checker-qual:2.5.2
+org.checkerframework:checker-qual:3.8.0
org.codehaus.mojo:animal-sniffer-annotations:1.21
org.jruby.jcodings:jcodings:1.0.13
org.jruby.joni:joni:2.1.2
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index 31fd2923e99..bd0e6a3edc1 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -40,7 +40,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_21
+ hadoop-shaded-protobuf_3_25
org.apache.hadoop
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 780b1461857..6416cc8b7dc 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -93,10 +93,10 @@
${common.protobuf2.scope}
-3.21.12
+3.23.4
${env.HADOOP_PROTOC_PATH}
-1.2.0
+1.3.0
${hadoop-thirdparty.version}
${hadoop-thirdparty.version}
org.apache.hadoop.thirdparty
@@ -250,7 +250,7 @@
org.apache.hadoop.thirdparty
-hadoop-shaded-protobuf_3_21
+hadoop-shaded-protobuf_3_25
${hadoop-thirdparty-protobuf.version}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
index 473078225e0..2b49a0ada12 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
+++ b/hadoop-yarn-project/hadoop
(hadoop) branch branch-3.4.1 updated: HADOOP-18938. S3A: Fix endpoint region parsing for vpc endpoints. (#6466)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.1 by this push:
new 6abd818c1c9 HADOOP-18938. S3A: Fix endpoint region parsing for vpc
endpoints. (#6466)
6abd818c1c9 is described below
commit 6abd818c1c9c38e3452bbe2dd2e1b23bb7cd573b
Author: Shintaro Onuma <31045635+shintaroon...@users.noreply.github.com>
AuthorDate: Thu Sep 5 14:14:04 2024 +0100
HADOOP-18938. S3A: Fix endpoint region parsing for vpc endpoints. (#6466)
Contributed by Shintaro Onuma
---
.../hadoop/fs/s3a/DefaultS3ClientFactory.java | 16 +++-
.../hadoop/fs/s3a/ITestS3AEndpointRegion.java | 13 ++-
.../hadoop/fs/s3a/TestS3AEndpointParsing.java | 43 ++
3 files changed, 70 insertions(+), 2 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
index 7f6978e8e92..4b3db999247 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
@@ -21,6 +21,8 @@ package org.apache.hadoop.fs.s3a;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.fs.s3a.impl.AWSClientConfig;
@@ -82,6 +84,9 @@ public class DefaultS3ClientFactory extends Configured
private static final String S3_SERVICE_NAME = "s3";
+ private static final Pattern VPC_ENDPOINT_PATTERN =
+
Pattern.compile("^(?:.+\\.)?([a-z0-9-]+)\\.vpce\\.amazonaws\\.(?:com|com\\.cn)$");
+
/**
* Subclasses refer to this.
*/
@@ -380,10 +385,19 @@ public class DefaultS3ClientFactory extends Configured
* @param endpointEndsWithCentral true if the endpoint is configured as
central.
* @return the S3 region, null if unable to resolve from endpoint.
*/
- private static Region getS3RegionFromEndpoint(final String endpoint,
+ @VisibleForTesting
+ static Region getS3RegionFromEndpoint(final String endpoint,
final boolean endpointEndsWithCentral) {
if (!endpointEndsWithCentral) {
+ // S3 VPC endpoint parsing
+ Matcher matcher = VPC_ENDPOINT_PATTERN.matcher(endpoint);
+ if (matcher.find()) {
+LOG.debug("Mapping to VPCE");
+LOG.debug("Endpoint {} is vpc endpoint; parsing region as {}",
endpoint, matcher.group(1));
+return Region.of(matcher.group(1));
+ }
+
LOG.debug("Endpoint {} is not the default; parsing", endpoint);
return AwsHostNameUtils.parseSigningRegion(endpoint,
S3_SERVICE_NAME).orElse(null);
}
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
index 8403b6bd6cb..d06224df5b3 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
@@ -97,6 +97,8 @@ public class ITestS3AEndpointRegion extends
AbstractS3ATestBase {
private static final String VPC_ENDPOINT =
"vpce-1a2b3c4d-5e6f.s3.us-west-2.vpce.amazonaws.com";
+ private static final String CN_VPC_ENDPOINT =
"vpce-1a2b3c4d-5e6f.s3.cn-northwest-1.vpce.amazonaws.com.cn";
+
public static final String EXCEPTION_THROWN_BY_INTERCEPTOR = "Exception
thrown by interceptor";
/**
@@ -294,7 +296,6 @@ public class ITestS3AEndpointRegion extends
AbstractS3ATestBase {
}
@Test
- @Ignore("Pending HADOOP-18938. S3A region logic to handle vpce and non
standard endpoints")
public void testWithVPCE() throws Throwable {
describe("Test with vpc endpoint");
Configuration conf = getConfiguration();
@@ -304,6 +305,16 @@ public class ITestS3AEndpointRegion extends
AbstractS3ATestBase {
expectInterceptorException(client);
}
+ @Test
+ public void testWithChinaVPCE() throws Throwable {
+describe("Test with china vpc endpoint");
+Configuration conf = getConfiguration();
+
+S3Client client = createS3Client(conf, CN_VPC_ENDPOINT, null,
CN_NORTHWEST_1, false);
+
+expectInterceptorException(client);
+ }
+
@Test
public void testCentralEndpointAndDifferentRegionThanBucket() throws
Throwable {
describe("Access public bucket using central endpoint and region "
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AEnd
(hadoop) branch branch-3.4 updated: HADOOP-18938. S3A: Fix endpoint region parsing for vpc endpoints. (#6466)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 343bf5ffdb8 HADOOP-18938. S3A: Fix endpoint region parsing for vpc
endpoints. (#6466)
343bf5ffdb8 is described below
commit 343bf5ffdb827f070ab9e4535571a83f0a540e83
Author: Shintaro Onuma <31045635+shintaroon...@users.noreply.github.com>
AuthorDate: Thu Sep 5 14:14:04 2024 +0100
HADOOP-18938. S3A: Fix endpoint region parsing for vpc endpoints. (#6466)
Contributed by Shintaro Onuma
---
.../hadoop/fs/s3a/DefaultS3ClientFactory.java | 16 +++-
.../hadoop/fs/s3a/ITestS3AEndpointRegion.java | 13 ++-
.../hadoop/fs/s3a/TestS3AEndpointParsing.java | 43 ++
3 files changed, 70 insertions(+), 2 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
index 7f6978e8e92..4b3db999247 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
@@ -21,6 +21,8 @@ package org.apache.hadoop.fs.s3a;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.fs.s3a.impl.AWSClientConfig;
@@ -82,6 +84,9 @@ public class DefaultS3ClientFactory extends Configured
private static final String S3_SERVICE_NAME = "s3";
+ private static final Pattern VPC_ENDPOINT_PATTERN =
+
Pattern.compile("^(?:.+\\.)?([a-z0-9-]+)\\.vpce\\.amazonaws\\.(?:com|com\\.cn)$");
+
/**
* Subclasses refer to this.
*/
@@ -380,10 +385,19 @@ public class DefaultS3ClientFactory extends Configured
* @param endpointEndsWithCentral true if the endpoint is configured as
central.
* @return the S3 region, null if unable to resolve from endpoint.
*/
- private static Region getS3RegionFromEndpoint(final String endpoint,
+ @VisibleForTesting
+ static Region getS3RegionFromEndpoint(final String endpoint,
final boolean endpointEndsWithCentral) {
if (!endpointEndsWithCentral) {
+ // S3 VPC endpoint parsing
+ Matcher matcher = VPC_ENDPOINT_PATTERN.matcher(endpoint);
+ if (matcher.find()) {
+LOG.debug("Mapping to VPCE");
+LOG.debug("Endpoint {} is vpc endpoint; parsing region as {}",
endpoint, matcher.group(1));
+return Region.of(matcher.group(1));
+ }
+
LOG.debug("Endpoint {} is not the default; parsing", endpoint);
return AwsHostNameUtils.parseSigningRegion(endpoint,
S3_SERVICE_NAME).orElse(null);
}
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
index 8403b6bd6cb..d06224df5b3 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
@@ -97,6 +97,8 @@ public class ITestS3AEndpointRegion extends
AbstractS3ATestBase {
private static final String VPC_ENDPOINT =
"vpce-1a2b3c4d-5e6f.s3.us-west-2.vpce.amazonaws.com";
+ private static final String CN_VPC_ENDPOINT =
"vpce-1a2b3c4d-5e6f.s3.cn-northwest-1.vpce.amazonaws.com.cn";
+
public static final String EXCEPTION_THROWN_BY_INTERCEPTOR = "Exception
thrown by interceptor";
/**
@@ -294,7 +296,6 @@ public class ITestS3AEndpointRegion extends
AbstractS3ATestBase {
}
@Test
- @Ignore("Pending HADOOP-18938. S3A region logic to handle vpce and non
standard endpoints")
public void testWithVPCE() throws Throwable {
describe("Test with vpc endpoint");
Configuration conf = getConfiguration();
@@ -304,6 +305,16 @@ public class ITestS3AEndpointRegion extends
AbstractS3ATestBase {
expectInterceptorException(client);
}
+ @Test
+ public void testWithChinaVPCE() throws Throwable {
+describe("Test with china vpc endpoint");
+Configuration conf = getConfiguration();
+
+S3Client client = createS3Client(conf, CN_VPC_ENDPOINT, null,
CN_NORTHWEST_1, false);
+
+expectInterceptorException(client);
+ }
+
@Test
public void testCentralEndpointAndDifferentRegionThanBucket() throws
Throwable {
describe("Access public bucket using central endpoint and region "
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AEndpointPa
(hadoop) branch trunk updated: Revert "YARN-11664. Remove HDFS Binaries/Jars Dependency From Yarn (#6631)"
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 57e62ae07f1 Revert "YARN-11664. Remove HDFS Binaries/Jars Dependency
From Yarn (#6631)"
57e62ae07f1 is described below
commit 57e62ae07f1c4eb8adccd9c61fc909080ca76c53
Author: Steve Loughran
AuthorDate: Thu Sep 5 14:35:50 2024 +0100
Revert "YARN-11664. Remove HDFS Binaries/Jars Dependency From Yarn (#6631)"
This reverts commit 6c01490f14b65f43196e1f235c51749a712e7338.
---
.../org/apache/hadoop/fs/HdfsCommonConstants.java | 47 --
.../hdfs/protocol/datatransfer/package-info.java | 25
.../hdfs/protocol/datatransfer/IOStreamPair.java | 6 +--
.../delegation/DelegationTokenIdentifier.java | 9 +
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 7 +---
.../apache/hadoop/yarn/service/ServiceMaster.java | 4 +-
.../hadoop/yarn/service/client/ServiceClient.java | 6 +--
.../hadoop/yarn/service/client/TestServiceCLI.java | 4 +-
.../yarn/logaggregation/AggregatedLogFormat.java | 6 +--
.../tfile/LogAggregationTFileController.java | 4 +-
10 files changed, 18 insertions(+), 100 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HdfsCommonConstants.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HdfsCommonConstants.java
deleted file mode 100644
index f6c3ca4517d..000
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HdfsCommonConstants.java
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.fs;
-
-import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.io.Text;
-
-/**
- * This class contains constants for configuration keys and default values.
- */
-@InterfaceAudience.LimitedPrivate({"YARN", "HDFS"})
-@InterfaceStability.Evolving
-public final class HdfsCommonConstants {
-
- /**
- * Represents the kind of delegation token used for HDFS.
- * This is a constant string value "HDFS_DELEGATION_TOKEN".
- */
- public static final Text HDFS_DELEGATION_KIND =
- new Text("HDFS_DELEGATION_TOKEN");
-
- /**
- * DFS_ADMIN configuration: {@value}.
- */
- public static final String DFS_ADMIN = "dfs.cluster.administrators";
-
- private HdfsCommonConstants() {
- }
-
-}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/package-info.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/package-info.java
deleted file mode 100644
index d2b8638b96e..000
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/package-info.java
+++ /dev/null
@@ -1,25 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-/**
- * This package provides access to following class.
- * {@link org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair} class.
- */
-@InterfaceAudience.Private
-package org.apache.hadoop.hdfs.protocol.datatransfer;
-
-import org.apache.h
(hadoop) branch trunk updated: HADOOP-18938. S3A: Fix endpoint region parsing for vpc endpoints. (#6466)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 1f302e83fd9 HADOOP-18938. S3A: Fix endpoint region parsing for vpc
endpoints. (#6466)
1f302e83fd9 is described below
commit 1f302e83fd93366544ccbe2bc5ee2de305e65cb6
Author: Shintaro Onuma <31045635+shintaroon...@users.noreply.github.com>
AuthorDate: Thu Sep 5 14:14:04 2024 +0100
HADOOP-18938. S3A: Fix endpoint region parsing for vpc endpoints. (#6466)
Contributed by Shintaro Onuma
---
.../hadoop/fs/s3a/DefaultS3ClientFactory.java | 16 +++-
.../hadoop/fs/s3a/ITestS3AEndpointRegion.java | 13 ++-
.../hadoop/fs/s3a/TestS3AEndpointParsing.java | 43 ++
3 files changed, 70 insertions(+), 2 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
index ba9fc080c2c..c52454ac15c 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
@@ -21,6 +21,8 @@ package org.apache.hadoop.fs.s3a;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.fs.s3a.impl.AWSClientConfig;
@@ -85,6 +87,9 @@ public class DefaultS3ClientFactory extends Configured
private static final String S3_SERVICE_NAME = "s3";
+ private static final Pattern VPC_ENDPOINT_PATTERN =
+
Pattern.compile("^(?:.+\\.)?([a-z0-9-]+)\\.vpce\\.amazonaws\\.(?:com|com\\.cn)$");
+
/**
* Subclasses refer to this.
*/
@@ -390,10 +395,19 @@ public class DefaultS3ClientFactory extends Configured
* @param endpointEndsWithCentral true if the endpoint is configured as
central.
* @return the S3 region, null if unable to resolve from endpoint.
*/
- private static Region getS3RegionFromEndpoint(final String endpoint,
+ @VisibleForTesting
+ static Region getS3RegionFromEndpoint(final String endpoint,
final boolean endpointEndsWithCentral) {
if (!endpointEndsWithCentral) {
+ // S3 VPC endpoint parsing
+ Matcher matcher = VPC_ENDPOINT_PATTERN.matcher(endpoint);
+ if (matcher.find()) {
+LOG.debug("Mapping to VPCE");
+LOG.debug("Endpoint {} is vpc endpoint; parsing region as {}",
endpoint, matcher.group(1));
+return Region.of(matcher.group(1));
+ }
+
LOG.debug("Endpoint {} is not the default; parsing", endpoint);
return AwsHostNameUtils.parseSigningRegion(endpoint,
S3_SERVICE_NAME).orElse(null);
}
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
index 8403b6bd6cb..d06224df5b3 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
@@ -97,6 +97,8 @@ public class ITestS3AEndpointRegion extends
AbstractS3ATestBase {
private static final String VPC_ENDPOINT =
"vpce-1a2b3c4d-5e6f.s3.us-west-2.vpce.amazonaws.com";
+ private static final String CN_VPC_ENDPOINT =
"vpce-1a2b3c4d-5e6f.s3.cn-northwest-1.vpce.amazonaws.com.cn";
+
public static final String EXCEPTION_THROWN_BY_INTERCEPTOR = "Exception
thrown by interceptor";
/**
@@ -294,7 +296,6 @@ public class ITestS3AEndpointRegion extends
AbstractS3ATestBase {
}
@Test
- @Ignore("Pending HADOOP-18938. S3A region logic to handle vpce and non
standard endpoints")
public void testWithVPCE() throws Throwable {
describe("Test with vpc endpoint");
Configuration conf = getConfiguration();
@@ -304,6 +305,16 @@ public class ITestS3AEndpointRegion extends
AbstractS3ATestBase {
expectInterceptorException(client);
}
+ @Test
+ public void testWithChinaVPCE() throws Throwable {
+describe("Test with china vpc endpoint");
+Configuration conf = getConfiguration();
+
+S3Client client = createS3Client(conf, CN_VPC_ENDPOINT, null,
CN_NORTHWEST_1, false);
+
+expectInterceptorException(client);
+ }
+
@Test
public void testCentralEndpointAndDifferentRegionThanBucket() throws
Throwable {
describe("Access public bucket using central endpoint and region "
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AEndpointParsing.
(hadoop) branch trunk updated (3bbfb2be089 -> 94868446104)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 3bbfb2be089 HADOOP-19257. S3A: ITestAssumeRole.testAssumeRoleBadInnerAuth failure (#7021) add 94868446104 HADOOP-16928. Make javadoc work on Java 17 (#6976) No new revisions were added by this update. Summary of changes: .../java/org/apache/hadoop/conf/Configuration.java | 46 ++-- .../org/apache/hadoop/fs/AbstractFileSystem.java | 2 +- .../org/apache/hadoop/fs/ChecksumFileSystem.java | 2 +- .../main/java/org/apache/hadoop/fs/ChecksumFs.java | 2 +- .../java/org/apache/hadoop/fs/FileContext.java | 32 - .../main/java/org/apache/hadoop/fs/FileSystem.java | 28 .../java/org/apache/hadoop/fs/RemoteIterator.java | 4 +- .../java/org/apache/hadoop/io/EnumSetWritable.java | 18 ++--- .../java/org/apache/hadoop/io/ObjectWritable.java | 4 +- .../java/org/apache/hadoop/io/SequenceFile.java| 10 +-- .../io/compress/bzip2/CBZip2InputStream.java | 8 +-- .../io/compress/bzip2/CBZip2OutputStream.java | 82 +++--- .../io/compress/zlib/BuiltInZlibDeflater.java | 2 +- .../org/apache/hadoop/io/file/tfile/Chunk.java | 4 +- .../org/apache/hadoop/ipc/RpcClientException.java | 2 +- .../java/org/apache/hadoop/ipc/RpcException.java | 2 +- .../org/apache/hadoop/ipc/RpcServerException.java | 2 +- .../hadoop/ipc/UnexpectedServerException.java | 2 +- .../org/apache/hadoop/metrics2/package-info.java | 26 +++ .../main/java/org/apache/hadoop/net/NetUtils.java | 6 +- .../hadoop/security/AccessControlException.java| 6 +- .../security/authorize/AuthorizationException.java | 6 +- .../apache/hadoop/util/GenericOptionsParser.java | 2 +- .../apache/hadoop/util/InstrumentedReadLock.java | 2 +- .../hadoop/util/InstrumentedReadWriteLock.java | 2 +- .../apache/hadoop/util/InstrumentedWriteLock.java | 2 +- .../apache/hadoop/util/ShutdownThreadsHelper.java | 16 ++--- .../java/org/apache/hadoop/util/StringUtils.java | 2 +- .../org/apache/hadoop/ipc/MiniRPCBenchmark.java| 12 ++-- .../hdfs/client/impl/BlockReaderLocalLegacy.java | 2 +- .../server/blockmanagement/DatanodeDescriptor.java | 2 +- .../server/namenode/EncryptionZoneManager.java | 2 +- .../hadoop/hdfs/server/namenode/NameNode.java | 2 +- .../hdfs/server/namenode/snapshot/DiffList.java| 2 +- .../FileDistributionCalculator.java| 18 ++--- .../FileDistributionVisitor.java | 16 ++--- .../java/org/apache/hadoop/hdfs/TestSafeMode.java | 2 +- .../server/datanode/TestReadOnlySharedStorage.java | 6 +- .../v2/app/rm/preemption/AMPreemptionPolicy.java | 2 +- .../org/apache/hadoop/mapred/FileOutputFormat.java | 16 ++--- .../java/org/apache/hadoop/mapred/JobConf.java | 4 +- .../java/org/apache/hadoop/mapred/MapRunnable.java | 2 +- .../org/apache/hadoop/mapred/jobcontrol/Job.java | 2 +- .../hadoop/mapred/join/CompositeInputFormat.java | 12 ++-- .../hadoop/mapred/join/CompositeRecordReader.java | 4 +- .../hadoop/mapred/join/OverrideRecordReader.java | 2 +- .../java/org/apache/hadoop/mapred/join/Parser.java | 2 +- .../hadoop/mapred/lib/TotalOrderPartitioner.java | 2 +- .../mapreduce/lib/jobcontrol/ControlledJob.java| 2 +- .../mapreduce/lib/join/CompositeInputFormat.java | 12 ++-- .../mapreduce/lib/join/CompositeRecordReader.java | 4 +- .../mapreduce/lib/join/OverrideRecordReader.java | 2 +- .../apache/hadoop/mapreduce/lib/join/Parser.java | 2 +- .../hadoop/mapreduce/lib/join/TupleWritable.java | 2 +- .../mapreduce/lib/output/FileOutputFormat.java | 8 +-- .../lib/partition/TotalOrderPartitioner.java | 10 +-- .../org/apache/hadoop/fs/AccumulatingReducer.java | 8 +-- .../java/org/apache/hadoop/fs/IOMapperBase.java| 4 +- .../java/org/apache/hadoop/fs/JHLogAnalyzer.java | 42 +-- .../org/apache/hadoop/examples/pi/package.html | 71 ++- hadoop-project/pom.xml | 23 -- hadoop-tools/hadoop-aws/pom.xml| 1 - .../org/apache/hadoop/mapred/gridmix/FilePool.java | 2 +- .../hadoop/streaming/io/IdentifierResolver.java| 2 +- .../java/org/apache/hadoop/streaming/package.html | 2 +- .../java/org/apache/hadoop/typedbytes/package.html | 8 ++- .../protocolrecords/SignalContainerRequest.java| 2 +- .../timelineservice/ServiceMetricsSink.java| 2 +- .../hadoop/yarn/security/AdminACLsManager.java | 4 +- .../apache/hadoop/yarn/util/BoundedAppender.java | 2 +- .../hadoop/yarn/server/utils/LeveldbIterator.java | 2 +- .../timelineservice/storage/common/BaseTable.java | 2 +- 72 files changed, 317 insertions(+), 336 deletions(-) - To unsubscribe, e-mail: common-commits
(hadoop) branch branch-3.4.1 updated: HADOOP-19257. S3A: ITestAssumeRole.testAssumeRoleBadInnerAuth failure (#7021)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.1 by this push:
new 98ad16f1c16 HADOOP-19257. S3A:
ITestAssumeRole.testAssumeRoleBadInnerAuth failure (#7021)
98ad16f1c16 is described below
commit 98ad16f1c16ea7cdd8b750bdfae83c3bac3da1e3
Author: Steve Loughran
AuthorDate: Tue Sep 3 21:20:47 2024 +0100
HADOOP-19257. S3A: ITestAssumeRole.testAssumeRoleBadInnerAuth failure
(#7021)
Remove the error string matched on so that no future message change
from AWS will trigger a regression
Contributed by Steve Loughran
---
.../src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java
index 5aa72e69490..592529b553d 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java
@@ -283,8 +283,7 @@ public class ITestAssumeRole extends AbstractS3ATestBase {
conf.set(SECRET_KEY, "not secret");
expectFileSystemCreateFailure(conf,
AWSBadRequestException.class,
-"not a valid " +
-"key=value pair (missing equal-sign) in Authorization header");
+"");
}
@Test
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19257. S3A: ITestAssumeRole.testAssumeRoleBadInnerAuth failure (#7021)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new edfd10e0be7 HADOOP-19257. S3A:
ITestAssumeRole.testAssumeRoleBadInnerAuth failure (#7021)
edfd10e0be7 is described below
commit edfd10e0be76f9daa4b8966bae64a37bf7e49cec
Author: Steve Loughran
AuthorDate: Tue Sep 3 21:20:47 2024 +0100
HADOOP-19257. S3A: ITestAssumeRole.testAssumeRoleBadInnerAuth failure
(#7021)
Remove the error string matched on so that no future message change
from AWS will trigger a regression
Contributed by Steve Loughran
---
.../src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java
index 5aa72e69490..592529b553d 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java
@@ -283,8 +283,7 @@ public class ITestAssumeRole extends AbstractS3ATestBase {
conf.set(SECRET_KEY, "not secret");
expectFileSystemCreateFailure(conf,
AWSBadRequestException.class,
-"not a valid " +
-"key=value pair (missing equal-sign) in Authorization header");
+"");
}
@Test
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (1655acc5e2d -> 3bbfb2be089)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 1655acc5e2d HADOOP-19250. [Addendum] Fix test TestServiceInterruptHandling.testRegisterAndRaise. (#7008) add 3bbfb2be089 HADOOP-19257. S3A: ITestAssumeRole.testAssumeRoleBadInnerAuth failure (#7021) No new revisions were added by this update. Summary of changes: .../src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-thirdparty) branch trunk updated: HADOOP-19252. Release hadoop-thirdparty 1.3.0: version update on trunk
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git The following commit(s) were added to refs/heads/trunk by this push: new faa8afc HADOOP-19252. Release hadoop-thirdparty 1.3.0: version update on trunk faa8afc is described below commit faa8afc43ac467335ff111ec16ce76f6d30eea25 Author: Steve Loughran AuthorDate: Fri Aug 16 17:02:02 2024 +0100 HADOOP-19252. Release hadoop-thirdparty 1.3.0: version update on trunk --- hadoop-shaded-avro_1_11/pom.xml | 2 +- hadoop-shaded-guava/pom.xml | 2 +- hadoop-shaded-protobuf_3_25/pom.xml | 2 +- pom.xml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/hadoop-shaded-avro_1_11/pom.xml b/hadoop-shaded-avro_1_11/pom.xml index 12e4dab..5a07faf 100644 --- a/hadoop-shaded-avro_1_11/pom.xml +++ b/hadoop-shaded-avro_1_11/pom.xml @@ -23,7 +23,7 @@ hadoop-thirdparty org.apache.hadoop.thirdparty -1.3.0-SNAPSHOT +1.4.0-SNAPSHOT .. 4.0.0 diff --git a/hadoop-shaded-guava/pom.xml b/hadoop-shaded-guava/pom.xml index fcfbb60..b03646f 100644 --- a/hadoop-shaded-guava/pom.xml +++ b/hadoop-shaded-guava/pom.xml @@ -23,7 +23,7 @@ hadoop-thirdparty org.apache.hadoop.thirdparty -1.3.0-SNAPSHOT +1.4.0-SNAPSHOT ../pom.xml 4.0.0 diff --git a/hadoop-shaded-protobuf_3_25/pom.xml b/hadoop-shaded-protobuf_3_25/pom.xml index 3e9a98d..9a90d5a 100644 --- a/hadoop-shaded-protobuf_3_25/pom.xml +++ b/hadoop-shaded-protobuf_3_25/pom.xml @@ -23,7 +23,7 @@ hadoop-thirdparty org.apache.hadoop.thirdparty -1.3.0-SNAPSHOT +1.4.0-SNAPSHOT ../pom.xml 4.0.0 diff --git a/pom.xml b/pom.xml index e98cf77..98879d8 100644 --- a/pom.xml +++ b/pom.xml @@ -23,7 +23,7 @@ 4.0.0 org.apache.hadoop.thirdparty hadoop-thirdparty - 1.3.0-SNAPSHOT + 1.4.0-SNAPSHOT org.apache apache - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19250. [Addendum] Fix test TestServiceInterruptHandling.testRegisterAndRaise. (#7008)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 1655acc5e2d HADOOP-19250. [Addendum] Fix test
TestServiceInterruptHandling.testRegisterAndRaise. (#7008)
1655acc5e2d is described below
commit 1655acc5e2d5fe27e01f46ea02bd5a7dea44fe12
Author: zhengchenyu
AuthorDate: Fri Aug 30 19:05:13 2024 +0800
HADOOP-19250. [Addendum] Fix test
TestServiceInterruptHandling.testRegisterAndRaise. (#7008)
Contributed by Chenyu Zheng
---
.../apache/hadoop/service/launcher/TestServiceInterruptHandling.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/TestServiceInterruptHandling.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/TestServiceInterruptHandling.java
index c21fa8b7307..8181e07fae0 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/TestServiceInterruptHandling.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/TestServiceInterruptHandling.java
@@ -38,7 +38,7 @@ public class TestServiceInterruptHandling
@Test
public void testRegisterAndRaise() throws Throwable {
InterruptCatcher catcher = new InterruptCatcher();
-String name = IrqHandler.CONTROL_C;
+String name = "USR2";
IrqHandler irqHandler = new IrqHandler(name, catcher);
irqHandler.bind();
assertEquals(0, irqHandler.getSignalCount());
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19252. Upgrade hadoop-thirdparty to 1.3.0 (#7007)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new b404c8c8f80 HADOOP-19252. Upgrade hadoop-thirdparty to 1.3.0 (#7007)
b404c8c8f80 is described below
commit b404c8c8f80d015edf48c674463ed57a9af6c55c
Author: Steve Loughran
AuthorDate: Fri Aug 30 11:50:51 2024 +0100
HADOOP-19252. Upgrade hadoop-thirdparty to 1.3.0 (#7007)
Update the version of hadoop-thirdparty to 1.3.0
across all shaded artifacts used.
This synchronizes the shaded protobuf library with those of
all other shaded artifacts (guava, avro)
Contributed by Steve Loughran
---
LICENSE-binary | 15 ---
hadoop-project/pom.xml | 4 ++--
2 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index cc018ed265b..a716db70f72 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -233,19 +233,19 @@ com.google:guice:5.1.0
com.google:guice-servlet:5.1.0
com.google.api.grpc:proto-google-common-protos:1.0.0
com.google.code.gson:2.9.0
-com.google.errorprone:error_prone_annotations:2.2.0
-com.google.j2objc:j2objc-annotations:1.1
+com.google.errorprone:error_prone_annotations:2.5.1
+com.google.j2objc:j2objc-annotations:1.3
com.google.json-simple:json-simple:1.1.1
com.google.guava:failureaccess:1.0
com.google.guava:guava:20.0
-com.google.guava:guava:27.0-jre
+com.google.guava:guava:32.0.1-jre
com.google.guava:listenablefuture:.0-empty-to-avoid-conflict-with-guava
com.microsoft.azure:azure-storage:7.0.0
com.nimbusds:nimbus-jose-jwt:9.37.2
com.zaxxer:HikariCP:4.0.3
commons-beanutils:commons-beanutils:1.9.4
commons-cli:commons-cli:1.5.0
-commons-codec:commons-codec:1.11
+commons-codec:commons-codec:1.15
commons-collections:commons-collections:3.2.2
commons-daemon:commons-daemon:1.0.13
commons-io:commons-io:2.16.1
@@ -298,6 +298,7 @@ javax.inject:javax.inject:1
net.java.dev.jna:jna:5.2.0
net.minidev:accessors-smart:1.2
org.apache.avro:avro:1.9.2
+org.apache.avro:avro:1.11.3
org.apache.commons:commons-collections4:4.2
org.apache.commons:commons-compress:1.26.1
org.apache.commons:commons-configuration2:2.10.1
@@ -362,7 +363,7 @@ org.objenesis:objenesis:2.6
org.xerial.snappy:snappy-java:1.1.10.4
org.yaml:snakeyaml:2.0
org.wildfly.openssl:wildfly-openssl:1.1.3.Final
-software.amazon.awssdk:bundle:jar:2.25.53
+software.amazon.awssdk:bundle:2.25.53
@@ -395,7 +396,7 @@
hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/d3-3.5.17.min.js
leveldb v1.13
com.google.protobuf:protobuf-java:2.5.0
-com.google.protobuf:protobuf-java:3.21.12
+com.google.protobuf:protobuf-java:3.25.3
com.google.re2j:re2j:1.1
com.jcraft:jsch:0.1.55
com.thoughtworks.paranamer:paranamer:2.3
@@ -485,7 +486,7 @@ com.microsoft.sqlserver:mssql-jdbc:6.2.1.jre7
org.bouncycastle:bcpkix-jdk18on:1.78.1
org.bouncycastle:bcprov-jdk18on:1.78.1
org.bouncycastle:bcutil-jdk18on:1.78.1
-org.checkerframework:checker-qual:2.5.2
+org.checkerframework:checker-qual:3.8.0
org.codehaus.mojo:animal-sniffer-annotations:1.21
org.jruby.jcodings:jcodings:1.0.13
org.jruby.joni:joni:2.1.2
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 05dccb62985..33533dbbaed 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -96,8 +96,8 @@
3.23.4
${env.HADOOP_PROTOC_PATH}
-1.2.0
-
1.3.0-SNAPSHOT
+1.3.0
+
${hadoop-thirdparty.version}
${hadoop-thirdparty.version}
org.apache.hadoop.thirdparty
${hadoop-thirdparty-shaded-prefix}.protobuf
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19248. Protobuf code generate and replace should happen together (#6975)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new 6bee9db424a HADOOP-19248. Protobuf code generate and replace should happen together (#6975) 6bee9db424a is described below commit 6bee9db424a804ac08847dd53001ab39f568229e Author: Cheng Pan AuthorDate: Fri Aug 30 18:30:00 2024 +0800 HADOOP-19248. Protobuf code generate and replace should happen together (#6975) Contributed by Cheng Pan --- hadoop-project/pom.xml | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index c4aa1a39018..e92e41ac99e 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -2287,7 +2287,7 @@ replace-generated-sources - process-sources + generate-sources replace @@ -2307,7 +2307,7 @@ replace-generated-test-sources - process-test-resources + generate-test-resources replace @@ -2327,7 +2327,7 @@ replace-sources - process-sources + generate-sources replace @@ -2347,7 +2347,7 @@ replace-test-sources - process-test-sources + generate-test-sources replace - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19248. Protobuf code generate and replace should happen together (#6975)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 0aab1a29764 HADOOP-19248. Protobuf code generate and replace should happen together (#6975) 0aab1a29764 is described below commit 0aab1a297647688173a024b003e88e98d9ae92ad Author: Cheng Pan AuthorDate: Thu Aug 29 03:18:46 2024 +0800 HADOOP-19248. Protobuf code generate and replace should happen together (#6975) Contributed by Cheng Pan --- hadoop-project/pom.xml | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 4c69012f08d..05dccb62985 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -2309,7 +2309,7 @@ replace-generated-sources - process-sources + generate-sources replace @@ -2329,7 +2329,7 @@ replace-generated-test-sources - process-test-resources + generate-test-resources replace @@ -2349,7 +2349,7 @@ replace-sources - process-sources + generate-sources replace @@ -2369,7 +2369,7 @@ replace-test-sources - process-test-sources + generate-test-sources replace - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-thirdparty) annotated tag rel/release-1.3.0 updated (0fd6290 -> 1c615e7)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to annotated tag rel/release-1.3.0 in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git *** WARNING: tag rel/release-1.3.0 was modified! *** from 0fd6290 (commit) to 1c615e7 (tag) tagging 0fd62903b071b5186f31b7030ce42e1c00f6bb6a (commit) by Steve Loughran on Wed Aug 28 18:57:18 2024 +0100 - Log - HADOOP-19252. Hadoop Thirdparty 1.3.0 release -BEGIN PGP SIGNATURE- iQIzBAABCgAdFiEEOCN+5CUFAoUHfbV60iz4RtuxYqAFAmbPZP4ACgkQ0iz4Rtux YqCX+xAAqTXl1QM6se/CUFgfjko4yW8DIDfpnUMCqMaU9nl8esfn0QL0gXtgaFFK rjTDfQf1wRXj+A3bb30Fv8ByXL0pIPJcxA/YiLdFQrn7JwxwkKfbq1tMNOXoroiQ /6gLsxQk9xhYvIWwugPkG+Y4hsNt+oa/6NZo7+gphXdni+WZU8huKGXvaw+QsvGz DOPVfehRYH9YLFWRQdlH0k66n5h0BYLjv5zxj5UkX279wHbMvmj59TgjqrBv2VdR +sKnI0ohpFZHa9MxmEVVBbGaQvUiNRx3vNJSlS90KvTDB6tr6Aj2y/ruUT/KWZiy nNzi0PVY4NBzqBPAv55FIslw9Vk+Vz3hCuOYftKH20n4tEuRahZRuGUUw/S4nAY9 Pvhl8JclVXtS1QTWgdBKSBOSSSBm5OLQuTNoKfsxdh+Wb7GW3JonEUdFkGNX7lU3 Dlds3+kmjHOFdhu0K7zB6sOWPClfn/yum8rWof1BkwRrYSXXIxNHhPN29ul25oIb 5i73F6ReKvb0aj70n0FL5N6pq4usdxrhi/iaEM44YtRZ83qXQ5Pr6QpRi0p21BiB S9Zlc44WPbBYtSwLumqxWi0Ru529bgGHQpa8mvXBZO3TRI8kEt/8Zn+Uq71T6kHZ 9gsoUHwg/WVNrnKYz7lJBYVmCdpj9OmKpjes1r3KU2qkfjTzrtw= =9Pxi -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
svn commit: r71135 - /dev/hadoop/thirdparty-1.3.0-RC1/ /release/hadoop/thirdparty/thirdparty-1.3.0/
Author: stevel Date: Wed Aug 28 17:38:59 2024 New Revision: 71135 Log: HADOOP-19252. Releasing Hadoop Thirdparty 1.3.0 Added: release/hadoop/thirdparty/thirdparty-1.3.0/ - copied from r71134, dev/hadoop/thirdparty-1.3.0-RC1/ Removed: dev/hadoop/thirdparty-1.3.0-RC1/ - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
svn commit: r71122 - in /dev/hadoop: hadoop-thirdparty-1.3.0-RC1/ thirdparty-1.3.0-RC1/
Author: stevel Date: Tue Aug 27 17:43:01 2024 New Revision: 71122 Log: HADOOP-19252. preparing thirdparty release Added: dev/hadoop/thirdparty-1.3.0-RC1/ - copied from r71121, dev/hadoop/hadoop-thirdparty-1.3.0-RC1/ Removed: dev/hadoop/hadoop-thirdparty-1.3.0-RC1/ - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
svn commit: r71121 - in /dev/hadoop: hadoop-3.3.6/ hadoop-3.3.7-aws/ hadoop-3.4.0-RC3/ hadoop-thirdparty-1.2.0-RC0/ hadoop-thirdparty-1.2.0-RC1/
Author: stevel Date: Tue Aug 27 17:28:39 2024 New Revision: 71121 Log: remove all obsolete RCs Removed: dev/hadoop/hadoop-3.3.6/ dev/hadoop/hadoop-3.3.7-aws/ dev/hadoop/hadoop-3.4.0-RC3/ dev/hadoop/hadoop-thirdparty-1.2.0-RC0/ dev/hadoop/hadoop-thirdparty-1.2.0-RC1/ - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-release-support) branch main updated: HADOOP-19252. hadoop-thirdparty 1.3.0-RC1
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hadoop-release-support.git
The following commit(s) were added to refs/heads/main by this push:
new ff7669e HADOOP-19252. hadoop-thirdparty 1.3.0-RC1
ff7669e is described below
commit ff7669e03bb6e0b0bfdfc193a69784dfc2e8c2cd
Author: Steve Loughran
AuthorDate: Tue Aug 20 19:03:36 2024 +0100
HADOOP-19252. hadoop-thirdparty 1.3.0-RC1
added initial targets here
---
README.md | 51
build.xml | 90 --
pom.xml| 9 ++-
release.properties | 2 +-
.../3p-release-1.3.0.properties} | 26 +++
src/releases/release-info-3.4.1.properties | 12 +--
src/text/3p.vote.txt | 19 +
7 files changed, 165 insertions(+), 44 deletions(-)
diff --git a/README.md b/README.md
index 967bb7c..374b564 100644
--- a/README.md
+++ b/README.md
@@ -29,7 +29,7 @@ the classpath.
Installed applications/platforms
-* Java 8+. Later releases are valid for validation too.
+* Java 8+. Later releases are valid for validation too (and required for some
projects)
* Apache Ant.
* Apache maven
* gpg
@@ -82,7 +82,7 @@ the classpath.
This is an optional property file which contains all user-specific
customizations
and options to assist in the release process.
-This file is *not* SCM-managed.
+This file is *not* SCM-managed (it is explicitly ignored).
It is read before all other property files are read/ant properties
set, so can override any subsequent declarations.
@@ -97,7 +97,7 @@ path to the latest release being worked on in this branch.
2. It is read after `build.properties`
```properties
-release.version=3.4.0
+release.version=3.4.1
```
Ant uses this to to set the property `release.info.file` to the path
@@ -233,8 +233,9 @@ Update as a appropriate.
### Update `/release.properties`
-Update the value of `release.info.file` in `/release.properties` to
-point to the newly created file.
+Update the value of `release.version in `/release.properties` to
+declare the release version. This is used to determine the specific release
properties
+file for that version.
```properties
release.version=X.Y.Z
@@ -262,14 +263,13 @@ scp.hadoop.dir=hadoop
staging.dir=/Users/stevel/hadoop/release/staging
# where various modules live for build and test
-spark.dir=/Users/stevel/Projects/sparkwork/spark
-cloud-examples.dir=/Users/stevel/Projects/sparkwork/cloud-integration/cloud-examples
-cloud.test.configuration.file=/Users/stevel/Projects/config/cloud-test-configs/s3a.xml
-bigdata-interop.dir=/Users/stevel/Projects/gcs/bigdata-interop
-hboss.dir=/Users/stevel/Projects/hbasework/hbase-filesystem
-cloudstore.dir=/Users/stevel/Projects/cloudstore
-fs-api-shim.dir=/Users/stevel/Projects/Formats/fs-api-shim/
-
+spark.dir=/Users/stevel/dev/spark
+cloud-examples.dir=/Users/stevel/dev/sparkwork/cloud-integration/cloud-examples
+cloud.test.configuration.file=/Users/stevel/dev/config/test-configs/s3a.xml
+bigdata-interop.dir=/Users/stevel/dev/gcs/bigdata-interop
+hboss.dir=/Users/stevel/dev/hbasework/hbase-filesystem
+cloudstore.dir=/Users/stevel/dev/cloudstore
+fs-api-shim.dir=/Users/stevel/dev/Formats/fs-api-shim/
```
### Clean up first
@@ -561,7 +561,7 @@ pending release version
ant mvn-purge
```
-## Execute the maven test.
+## Execute a maven test run
Download the artifacts from maven staging repositories and compile/test a
minimal application
@@ -923,6 +923,29 @@ ant stage-svn-rollback
ant stage-svn-log
```
+# Releasing Hadoop Third party
+
+See wiki page [How To Release
Hadoop-Thirdparty](https://cwiki.apache.org/confluence/display/HADOOP2/How+To+Release+Hadoop-Thirdparty)
+
+
+Support for this release workflow is pretty minimal, but releasing it is
simpler
+
+* Update the branches and maven artifact versions
+* build/test. This can be done with the help of a PR to upgrade hadoop.
+* create the vote message.
+
+## Configuration options
+
+All options are prefixed `3p.`
+
+## Targets:
+
+All targets are prefixed `3p.`
+
+```
+3p.mvn-purge : remove all third party artifacts from the repo
+```
+
# Contributing to this module
There are lots of opportunities to contribute to the module
diff --git a/build.xml b/build.xml
index 1c97a33..9b8780e 100644
--- a/build.xml
+++ b/build.xml
@@ -173,6 +173,12 @@
+
+
+
+
@@ -297,10 +303,6 @@
-
-
deleting ${hadoop.artifacts}/**/${hadoop.version}/*
@@ -361,9 +363,14 @@
+
+
+
@@ -397,7 +404,7 @@
+description="copy the downloaded artifacts from incoming to release dir">
@@ -405,6 +412,7 @@
(hadoop) branch branch-3.4.1 updated: HADOOP-18542. Keep MSI tenant ID and client ID optional (#4262)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.1 by this push:
new 6e4cc8be043 HADOOP-18542. Keep MSI tenant ID and client ID optional
(#4262)
6e4cc8be043 is described below
commit 6e4cc8be04348a25e92ce2be8c61519fda6bf764
Author: Carl Levasseur
AuthorDate: Wed Aug 21 15:15:28 2024 +0200
HADOOP-18542. Keep MSI tenant ID and client ID optional (#4262)
Contributed by Carl Levasseur
---
.../hadoop/fs/azurebfs/AbfsConfiguration.java | 4 +--
.../fs/azurebfs/TestAccountConfiguration.java | 33 +-
2 files changed, 28 insertions(+), 9 deletions(-)
diff --git
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
index 3f5e7b0e69a..43923f758f9 100644
---
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
+++
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
@@ -962,9 +962,9 @@ public class AbfsConfiguration{
FS_AZURE_ACCOUNT_OAUTH_MSI_ENDPOINT,
AuthConfigurations.DEFAULT_FS_AZURE_ACCOUNT_OAUTH_MSI_ENDPOINT);
String tenantGuid =
- getMandatoryPasswordString(FS_AZURE_ACCOUNT_OAUTH_MSI_TENANT);
+ getPasswordString(FS_AZURE_ACCOUNT_OAUTH_MSI_TENANT);
String clientId =
- getMandatoryPasswordString(FS_AZURE_ACCOUNT_OAUTH_CLIENT_ID);
+ getPasswordString(FS_AZURE_ACCOUNT_OAUTH_CLIENT_ID);
String authority = getTrimmedPasswordString(
FS_AZURE_ACCOUNT_OAUTH_MSI_AUTHORITY,
AuthConfigurations.DEFAULT_FS_AZURE_ACCOUNT_OAUTH_MSI_AUTHORITY);
diff --git
a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAccountConfiguration.java
b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAccountConfiguration.java
index 17da772d081..483a7e3d5d5 100644
---
a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAccountConfiguration.java
+++
b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAccountConfiguration.java
@@ -27,6 +27,7 @@ import org.apache.hadoop.conf.Configuration;
import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.ConfigurationPropertyNotFoundException;
import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException;
+import org.apache.hadoop.fs.azurebfs.oauth2.AccessTokenProvider;
import org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider;
import org.apache.hadoop.fs.azurebfs.oauth2.CustomTokenProviderAdapter;
import org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider;
@@ -66,6 +67,7 @@ import static org.junit.Assert.assertNull;
*/
public class TestAccountConfiguration {
private static final String TEST_OAUTH_PROVIDER_CLASS_CONFIG =
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider";
+ private static final String TEST_OAUTH_MSI_TOKEN_PROVIDER_CLASS_CONFIG =
"org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider";
private static final String TEST_CUSTOM_PROVIDER_CLASS_CONFIG =
"org.apache.hadoop.fs.azurebfs.oauth2.RetryTestTokenProvider";
private static final String TEST_SAS_PROVIDER_CLASS_CONFIG_1 =
"org.apache.hadoop.fs.azurebfs.extensions.MockErrorSASTokenProvider";
private static final String TEST_SAS_PROVIDER_CLASS_CONFIG_2 =
"org.apache.hadoop.fs.azurebfs.extensions.MockSASTokenProvider";
@@ -90,11 +92,6 @@ public class TestAccountConfiguration {
FS_AZURE_ACCOUNT_OAUTH_USER_NAME,
FS_AZURE_ACCOUNT_OAUTH_USER_PASSWORD));
- private static final List MSI_TOKEN_OAUTH_CONFIG_KEYS =
- Collections.unmodifiableList(Arrays.asList(
- FS_AZURE_ACCOUNT_OAUTH_MSI_TENANT,
- FS_AZURE_ACCOUNT_OAUTH_CLIENT_ID));
-
private static final List REFRESH_TOKEN_OAUTH_CONFIG_KEYS =
Collections.unmodifiableList(Arrays.asList(
FS_AZURE_ACCOUNT_OAUTH_REFRESH_TOKEN,
@@ -410,10 +407,8 @@ public class TestAccountConfiguration {
public void testOAuthConfigPropNotFound() throws Throwable {
testConfigPropNotFound(CLIENT_CREDENTIAL_OAUTH_CONFIG_KEYS,
ClientCredsTokenProvider.class.getName());
testConfigPropNotFound(USER_PASSWORD_OAUTH_CONFIG_KEYS,
UserPasswordTokenProvider.class.getName());
-testConfigPropNotFound(MSI_TOKEN_OAUTH_CONFIG_KEYS,
MsiTokenProvider.class.getName());
testConfigPropNotFound(REFRESH_TOKEN_OAUTH_CONFIG_KEYS,
RefreshTokenBasedTokenProvider.class.getName());
testConf
(hadoop) 02/02: HADOOP-18542. Keep MSI tenant ID and client ID optional (#4262)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit e4d46d89e751f84bf09820cdeb5201af9967d3cf
Author: Carl Levasseur
AuthorDate: Wed Aug 21 15:15:28 2024 +0200
HADOOP-18542. Keep MSI tenant ID and client ID optional (#4262)
Contributed by Carl Levasseur
---
.../hadoop/fs/azurebfs/AbfsConfiguration.java | 4 +--
.../fs/azurebfs/TestAccountConfiguration.java | 33 +-
2 files changed, 28 insertions(+), 9 deletions(-)
diff --git
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
index 3f5e7b0e69a..43923f758f9 100644
---
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
+++
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
@@ -962,9 +962,9 @@ public class AbfsConfiguration{
FS_AZURE_ACCOUNT_OAUTH_MSI_ENDPOINT,
AuthConfigurations.DEFAULT_FS_AZURE_ACCOUNT_OAUTH_MSI_ENDPOINT);
String tenantGuid =
- getMandatoryPasswordString(FS_AZURE_ACCOUNT_OAUTH_MSI_TENANT);
+ getPasswordString(FS_AZURE_ACCOUNT_OAUTH_MSI_TENANT);
String clientId =
- getMandatoryPasswordString(FS_AZURE_ACCOUNT_OAUTH_CLIENT_ID);
+ getPasswordString(FS_AZURE_ACCOUNT_OAUTH_CLIENT_ID);
String authority = getTrimmedPasswordString(
FS_AZURE_ACCOUNT_OAUTH_MSI_AUTHORITY,
AuthConfigurations.DEFAULT_FS_AZURE_ACCOUNT_OAUTH_MSI_AUTHORITY);
diff --git
a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAccountConfiguration.java
b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAccountConfiguration.java
index 17da772d081..483a7e3d5d5 100644
---
a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAccountConfiguration.java
+++
b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAccountConfiguration.java
@@ -27,6 +27,7 @@ import org.apache.hadoop.conf.Configuration;
import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.ConfigurationPropertyNotFoundException;
import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException;
+import org.apache.hadoop.fs.azurebfs.oauth2.AccessTokenProvider;
import org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider;
import org.apache.hadoop.fs.azurebfs.oauth2.CustomTokenProviderAdapter;
import org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider;
@@ -66,6 +67,7 @@ import static org.junit.Assert.assertNull;
*/
public class TestAccountConfiguration {
private static final String TEST_OAUTH_PROVIDER_CLASS_CONFIG =
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider";
+ private static final String TEST_OAUTH_MSI_TOKEN_PROVIDER_CLASS_CONFIG =
"org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider";
private static final String TEST_CUSTOM_PROVIDER_CLASS_CONFIG =
"org.apache.hadoop.fs.azurebfs.oauth2.RetryTestTokenProvider";
private static final String TEST_SAS_PROVIDER_CLASS_CONFIG_1 =
"org.apache.hadoop.fs.azurebfs.extensions.MockErrorSASTokenProvider";
private static final String TEST_SAS_PROVIDER_CLASS_CONFIG_2 =
"org.apache.hadoop.fs.azurebfs.extensions.MockSASTokenProvider";
@@ -90,11 +92,6 @@ public class TestAccountConfiguration {
FS_AZURE_ACCOUNT_OAUTH_USER_NAME,
FS_AZURE_ACCOUNT_OAUTH_USER_PASSWORD));
- private static final List MSI_TOKEN_OAUTH_CONFIG_KEYS =
- Collections.unmodifiableList(Arrays.asList(
- FS_AZURE_ACCOUNT_OAUTH_MSI_TENANT,
- FS_AZURE_ACCOUNT_OAUTH_CLIENT_ID));
-
private static final List REFRESH_TOKEN_OAUTH_CONFIG_KEYS =
Collections.unmodifiableList(Arrays.asList(
FS_AZURE_ACCOUNT_OAUTH_REFRESH_TOKEN,
@@ -410,10 +407,8 @@ public class TestAccountConfiguration {
public void testOAuthConfigPropNotFound() throws Throwable {
testConfigPropNotFound(CLIENT_CREDENTIAL_OAUTH_CONFIG_KEYS,
ClientCredsTokenProvider.class.getName());
testConfigPropNotFound(USER_PASSWORD_OAUTH_CONFIG_KEYS,
UserPasswordTokenProvider.class.getName());
-testConfigPropNotFound(MSI_TOKEN_OAUTH_CONFIG_KEYS,
MsiTokenProvider.class.getName());
testConfigPropNotFound(REFRESH_TOKEN_OAUTH_CONFIG_KEYS,
RefreshTokenBasedTokenProvider.class.getName());
testConfigPropNotFound(WORKLOAD_IDENTITY_OAUTH_CONFIG_KEYS,
WorkloadIdentityTokenProvider.class.getName());
-
}
private void testConfigPropNotFound(List configKeys,
@@ -444,6 +439,30 @@ public class TestAccountConfiguration {
(hadoop) 01/02: HADOOP-19249. KMSClientProvider raises NPE with unauthed user (#6984)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit a455823e879804d839a6ee0e72a2779834729617
Author: dhavalshah9131 <35031652+dhavalshah9...@users.noreply.github.com>
AuthorDate: Tue Aug 20 18:33:05 2024 +0530
HADOOP-19249. KMSClientProvider raises NPE with unauthed user (#6984)
KMSClientProvider raises a NullPointerException when an unauthorised user
tries to perform the key operation
Contributed by Dhaval Shah
---
.../org/apache/hadoop/crypto/key/kms/KMSClientProvider.java | 12
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
index f0c912224f9..10f7b428ad1 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.crypto.key.kms;
import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.crypto.key.KeyProvider;
@@ -561,17 +562,19 @@ public class KMSClientProvider extends KeyProvider
implements CryptoExtension,
}
throw ex;
}
+
if ((conn.getResponseCode() == HttpURLConnection.HTTP_FORBIDDEN
-&& (conn.getResponseMessage().equals(ANONYMOUS_REQUESTS_DISALLOWED) ||
-conn.getResponseMessage().contains(INVALID_SIGNATURE)))
+&& (!StringUtils.isEmpty(conn.getResponseMessage())
+&& (conn.getResponseMessage().equals(ANONYMOUS_REQUESTS_DISALLOWED)
+|| conn.getResponseMessage().contains(INVALID_SIGNATURE
|| conn.getResponseCode() == HttpURLConnection.HTTP_UNAUTHORIZED) {
// Ideally, this should happen only when there is an Authentication
// failure. Unfortunately, the AuthenticationFilter returns 403 when it
// cannot authenticate (Since a 401 requires Server to send
// WWW-Authenticate header as well)..
if (LOG.isDebugEnabled()) {
-LOG.debug("Response={}({}), resetting authToken",
-conn.getResponseCode(), conn.getResponseMessage());
+LOG.debug("Response={}, resetting authToken",
+conn.getResponseCode());
}
KMSClientProvider.this.authToken =
new DelegationTokenAuthenticatedURL.Token();
@@ -798,6 +801,7 @@ public class KMSClientProvider extends KeyProvider
implements CryptoExtension,
@SuppressWarnings("rawtypes")
@Override
public KeyVersion decryptEncryptedKey(
+
EncryptedKeyVersion encryptedKeyVersion) throws IOException,
GeneralSecurityException
{
checkNotNull(encryptedKeyVersion.getEncryptionKeyVersionName(),
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated (337f2fc5660 -> e4d46d89e75)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git from 337f2fc5660 HADOOP-18962. Upgrade kafka to 3.4.0 (#6247) new a455823e879 HADOOP-19249. KMSClientProvider raises NPE with unauthed user (#6984) new e4d46d89e75 HADOOP-18542. Keep MSI tenant ID and client ID optional (#4262) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../hadoop/crypto/key/kms/KMSClientProvider.java | 12 +--- .../hadoop/fs/azurebfs/AbfsConfiguration.java | 4 +-- .../fs/azurebfs/TestAccountConfiguration.java | 33 +- 3 files changed, 36 insertions(+), 13 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (012ae9d1aa0 -> 68fcd7234ca)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 012ae9d1aa0 HDFS-17606. Do not require implementing CustomizedCallbackHandler. (#7005) add 68fcd7234ca HADOOP-18542. Keep MSI tenant ID and client ID optional (#4262) No new revisions were added by this update. Summary of changes: .../hadoop/fs/azurebfs/AbfsConfiguration.java | 4 +-- .../fs/azurebfs/TestAccountConfiguration.java | 33 +- 2 files changed, 28 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (33c9ecb6521 -> b15ed27cfbf)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
from 33c9ecb6521 HADOOP-19249. KMSClientProvider raises NPE with unauthed
user (#6984)
add b15ed27cfbf HADOOP-19187: [ABFS][FNSOverBlob] AbfsClient Refactoring
to Support Multiple Implementation of Clients. (#6879)
No new revisions were added by this update.
Summary of changes:
.../src/config/checkstyle-suppressions.xml |2 +
.../hadoop/fs/azurebfs/AbfsConfiguration.java | 83 +-
.../hadoop/fs/azurebfs/AzureBlobFileSystem.java| 56 +-
.../fs/azurebfs/AzureBlobFileSystemStore.java | 102 +-
.../fs/azurebfs/constants/AbfsHttpConstants.java |2 +
...HttpOperationType.java => AbfsServiceType.java} | 19 +-
.../fs/azurebfs/constants/ConfigurationKeys.java | 28 +-
.../fs/azurebfs/constants/FSOperationType.java |3 +-
.../constants/FileSystemConfigurations.java|1 +
.../azurebfs/constants/FileSystemUriSchemes.java |5 +-
.../InvalidConfigurationValueException.java|4 +
.../hadoop/fs/azurebfs/services/AbfsClient.java| 1249 +++
.../fs/azurebfs/services/AbfsClientHandler.java| 127 ++
.../hadoop/fs/azurebfs/services/AbfsDfsClient.java | 1302
.../apache/hadoop/fs/azurebfs/utils/UriUtils.java | 36 +
.../hadoop-azure/src/site/markdown/fns_blob.md | 82 ++
.../hadoop-azure/src/site/markdown/index.md|1 +
.../fs/azurebfs/ITestAbfsCustomEncryption.java |3 +-
.../ITestAzureBlobFileSystemCheckAccess.java |9 +-
.../ITestAzureBlobFileSystemInitAndCreate.java | 44 +-
.../fs/azurebfs/ITestGetNameSpaceEnabled.java | 14 +-
.../fs/azurebfs/services/ITestAbfsClient.java |9 +-
22 files changed, 2266 insertions(+), 915 deletions(-)
copy
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/{HttpOperationType.java
=> AbfsServiceType.java} (59%)
create mode 100644
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientHandler.java
create mode 100644
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsDfsClient.java
create mode 100644 hadoop-tools/hadoop-azure/src/site/markdown/fns_blob.md
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.1 updated (75845e66851 -> eb0732e0792)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch branch-3.4.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git from 75845e66851 HADOOP-19253. Google GCS compilation fails due to VectorIO changes (#7002) new 46336b3803c HADOOP-18962. Upgrade kafka to 3.4.0 (#6247) new eb0732e0792 HADOOP-19249. KMSClientProvider raises NPE with unauthed user (#6984) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: LICENSE-binary | 4 ++-- .../org/apache/hadoop/crypto/key/kms/KMSClientProvider.java | 12 hadoop-project/pom.xml | 2 +- 3 files changed, 11 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) 01/02: HADOOP-18962. Upgrade kafka to 3.4.0 (#6247)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 46336b3803cc7022f5a7bc081045cbedce7bf909
Author: Steve Loughran
AuthorDate: Tue Aug 20 13:54:42 2024 +0100
HADOOP-18962. Upgrade kafka to 3.4.0 (#6247)
Upgrade Kafka Client due to CVEs
* CVE-2023-25194
* CVE-2021-38153
* CVE-2018-17196
Contributed by Murali Krishna
---
LICENSE-binary | 4 ++--
hadoop-project/pom.xml | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index 252f934eac0..887e7070967 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -317,7 +317,7 @@ org.apache.htrace:htrace-core:3.1.0-incubating
org.apache.htrace:htrace-core4:4.1.0-incubating
org.apache.httpcomponents:httpclient:4.5.13
org.apache.httpcomponents:httpcore:4.4.13
-org.apache.kafka:kafka-clients:2.8.2
+org.apache.kafka:kafka-clients:3.4.0
org.apache.kerby:kerb-admin:2.0.3
org.apache.kerby:kerb-client:2.0.3
org.apache.kerby:kerb-common:2.0.3
@@ -377,7 +377,7 @@
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/com
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/util/tree.h
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/compat/{fstatat|openat|unlinkat}.h
-com.github.luben:zstd-jni:1.4.9-1
+com.github.luben:zstd-jni:1.5.2-1
dnsjava:dnsjava:3.6.1
org.codehaus.woodstox:stax2-api:4.2.1
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 9e08106bf0a..780b1461857 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -50,7 +50,7 @@
2.12.2
-2.8.2
+3.4.0
1.0.13
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) 02/02: HADOOP-19249. KMSClientProvider raises NPE with unauthed user (#6984)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit eb0732e07926bb706f2ecdc40a85c31fa814d22e
Author: dhavalshah9131 <35031652+dhavalshah9...@users.noreply.github.com>
AuthorDate: Tue Aug 20 18:33:05 2024 +0530
HADOOP-19249. KMSClientProvider raises NPE with unauthed user (#6984)
KMSClientProvider raises a NullPointerException when an unauthorised user
tries to perform the key operation
Contributed by Dhaval Shah
---
.../org/apache/hadoop/crypto/key/kms/KMSClientProvider.java | 12
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
index f0c912224f9..10f7b428ad1 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.crypto.key.kms;
import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.crypto.key.KeyProvider;
@@ -561,17 +562,19 @@ public class KMSClientProvider extends KeyProvider
implements CryptoExtension,
}
throw ex;
}
+
if ((conn.getResponseCode() == HttpURLConnection.HTTP_FORBIDDEN
-&& (conn.getResponseMessage().equals(ANONYMOUS_REQUESTS_DISALLOWED) ||
-conn.getResponseMessage().contains(INVALID_SIGNATURE)))
+&& (!StringUtils.isEmpty(conn.getResponseMessage())
+&& (conn.getResponseMessage().equals(ANONYMOUS_REQUESTS_DISALLOWED)
+|| conn.getResponseMessage().contains(INVALID_SIGNATURE
|| conn.getResponseCode() == HttpURLConnection.HTTP_UNAUTHORIZED) {
// Ideally, this should happen only when there is an Authentication
// failure. Unfortunately, the AuthenticationFilter returns 403 when it
// cannot authenticate (Since a 401 requires Server to send
// WWW-Authenticate header as well)..
if (LOG.isDebugEnabled()) {
-LOG.debug("Response={}({}), resetting authToken",
-conn.getResponseCode(), conn.getResponseMessage());
+LOG.debug("Response={}, resetting authToken",
+conn.getResponseCode());
}
KMSClientProvider.this.authToken =
new DelegationTokenAuthenticatedURL.Token();
@@ -798,6 +801,7 @@ public class KMSClientProvider extends KeyProvider
implements CryptoExtension,
@SuppressWarnings("rawtypes")
@Override
public KeyVersion decryptEncryptedKey(
+
EncryptedKeyVersion encryptedKeyVersion) throws IOException,
GeneralSecurityException
{
checkNotNull(encryptedKeyVersion.getEncryptionKeyVersionName(),
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (2fd7cf53fac -> 33c9ecb6521)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 2fd7cf53fac HADOOP-19253. Google GCS compilation fails due to VectorIO changes (#7002) add 33c9ecb6521 HADOOP-19249. KMSClientProvider raises NPE with unauthed user (#6984) No new revisions were added by this update. Summary of changes: .../org/apache/hadoop/crypto/key/kms/KMSClientProvider.java | 12 1 file changed, 8 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-18962. Upgrade kafka to 3.4.0 (#6247)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 337f2fc5660 HADOOP-18962. Upgrade kafka to 3.4.0 (#6247)
337f2fc5660 is described below
commit 337f2fc5660b9c0b3b8708baa871175b36341b9b
Author: Steve Loughran
AuthorDate: Tue Aug 20 13:54:42 2024 +0100
HADOOP-18962. Upgrade kafka to 3.4.0 (#6247)
Upgrade Kafka Client due to CVEs
* CVE-2023-25194
* CVE-2021-38153
* CVE-2018-17196
Contributed by Murali Krishna
---
LICENSE-binary | 4 ++--
hadoop-project/pom.xml | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index 252f934eac0..887e7070967 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -317,7 +317,7 @@ org.apache.htrace:htrace-core:3.1.0-incubating
org.apache.htrace:htrace-core4:4.1.0-incubating
org.apache.httpcomponents:httpclient:4.5.13
org.apache.httpcomponents:httpcore:4.4.13
-org.apache.kafka:kafka-clients:2.8.2
+org.apache.kafka:kafka-clients:3.4.0
org.apache.kerby:kerb-admin:2.0.3
org.apache.kerby:kerb-client:2.0.3
org.apache.kerby:kerb-common:2.0.3
@@ -377,7 +377,7 @@
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/com
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/util/tree.h
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/compat/{fstatat|openat|unlinkat}.h
-com.github.luben:zstd-jni:1.4.9-1
+com.github.luben:zstd-jni:1.5.2-1
dnsjava:dnsjava:3.6.1
org.codehaus.woodstox:stax2-api:4.2.1
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 5b1e55afd91..c4aa1a39018 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -50,7 +50,7 @@
2.12.2
-2.8.2
+3.4.0
1.0.13
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19253. Google GCS compilation fails due to VectorIO changes (#7002)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 2fd7cf53fac HADOOP-19253. Google GCS compilation fails due to VectorIO
changes (#7002)
2fd7cf53fac is described below
commit 2fd7cf53facec3aa649f1f2cc53f8e21c209e178
Author: Steve Loughran
AuthorDate: Mon Aug 19 19:54:47 2024 +0100
HADOOP-19253. Google GCS compilation fails due to VectorIO changes (#7002)
Fixes a compilation failure caused by HADOOP-19098
Restore original sortRanges() method signature,
FileRange[] sortRanges(List)
This ensures that google GCS connector will compile again.
It has also been marked as Stable so it is left alone
The version returning List
has been renamed sortRangeList()
Contributed by Steve Loughran
---
.../org/apache/hadoop/fs/VectoredReadUtils.java| 17 +--
.../hadoop/fs/impl/TestVectoredReadUtils.java | 35 ++
2 files changed, 44 insertions(+), 8 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
index fa0440620a4..2f99edc910c 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
@@ -308,7 +308,7 @@ public final class VectoredReadUtils {
validateRangeRequest(input.get(0));
sortedRanges = input;
} else {
- sortedRanges = sortRanges(input);
+ sortedRanges = sortRangeList(input);
FileRange prev = null;
for (final FileRange current : sortedRanges) {
validateRangeRequest(current);
@@ -341,12 +341,25 @@ public final class VectoredReadUtils {
* @param input input ranges.
* @return a new list of the ranges, sorted by offset.
*/
- public static List sortRanges(List
input) {
+ public static List sortRangeList(List input) {
final List l = new ArrayList<>(input);
l.sort(Comparator.comparingLong(FileRange::getOffset));
return l;
}
+ /**
+ * Sort the input ranges by offset; no validation is done.
+ *
+ * This method is used externally and must be retained with
+ * the signature unchanged.
+ * @param input input ranges.
+ * @return a new list of the ranges, sorted by offset.
+ */
+ @InterfaceStability.Stable
+ public static FileRange[] sortRanges(List input) {
+return sortRangeList(input).toArray(new FileRange[0]);
+ }
+
/**
* Merge sorted ranges to optimize the access from the underlying file
* system.
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
index 3fd3fe4d1f4..b08fc95279a 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
@@ -23,6 +23,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.IntBuffer;
import java.util.Collections;
+import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
@@ -47,6 +48,7 @@ import static
org.apache.hadoop.fs.VectoredReadUtils.isOrderedDisjoint;
import static org.apache.hadoop.fs.VectoredReadUtils.mergeSortedRanges;
import static org.apache.hadoop.fs.VectoredReadUtils.readRangeFrom;
import static org.apache.hadoop.fs.VectoredReadUtils.readVectored;
+import static org.apache.hadoop.fs.VectoredReadUtils.sortRangeList;
import static org.apache.hadoop.fs.VectoredReadUtils.sortRanges;
import static org.apache.hadoop.fs.VectoredReadUtils.validateAndSortRanges;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
@@ -196,7 +198,7 @@ public class TestVectoredReadUtils extends HadoopTestBase {
);
assertIsNotOrderedDisjoint(input, 100, 800);
final List outputList = mergeSortedRanges(
-sortRanges(input), 100, 1001, 2500);
+sortRangeList(input), 100, 1001, 2500);
assertRangeListSize(outputList, 1);
CombinedFileRange output = outputList.get(0);
@@ -208,7 +210,7 @@ public class TestVectoredReadUtils extends HadoopTestBase {
// the minSeek doesn't allow the first two to merge
assertIsNotOrderedDisjoint(input, 100, 100);
final List list2 = mergeSortedRanges(
-sortRanges(input),
+sortRangeList(input),
100, 1000, 2100);
assertRangeListSize(list2, 2);
assertRangeElement(list2, 0, 1000, 100);
@@ -219,7 +221,
(hadoop) branch branch-3.4.1 updated: HADOOP-19253. Google GCS compilation fails due to VectorIO changes (#7002)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.1 by this push:
new 75845e66851 HADOOP-19253. Google GCS compilation fails due to VectorIO
changes (#7002)
75845e66851 is described below
commit 75845e6685126713a01a692b47eaf77f26f62849
Author: Steve Loughran
AuthorDate: Mon Aug 19 19:54:47 2024 +0100
HADOOP-19253. Google GCS compilation fails due to VectorIO changes (#7002)
Fixes a compilation failure ncaused by HADOOP-19098
Restore original sortRanges() method signature,
FileRange[] sortRanges(List)
This ensures that google GCS connector will compile again.
It has also been marked as Stable so it is left alone
The version returning List
has been renamed sortRangeList()
Contributed by Steve Loughran
---
.../org/apache/hadoop/fs/VectoredReadUtils.java| 17 +--
.../hadoop/fs/impl/TestVectoredReadUtils.java | 35 ++
2 files changed, 44 insertions(+), 8 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
index fa0440620a4..2f99edc910c 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
@@ -308,7 +308,7 @@ public final class VectoredReadUtils {
validateRangeRequest(input.get(0));
sortedRanges = input;
} else {
- sortedRanges = sortRanges(input);
+ sortedRanges = sortRangeList(input);
FileRange prev = null;
for (final FileRange current : sortedRanges) {
validateRangeRequest(current);
@@ -341,12 +341,25 @@ public final class VectoredReadUtils {
* @param input input ranges.
* @return a new list of the ranges, sorted by offset.
*/
- public static List sortRanges(List
input) {
+ public static List sortRangeList(List input) {
final List l = new ArrayList<>(input);
l.sort(Comparator.comparingLong(FileRange::getOffset));
return l;
}
+ /**
+ * Sort the input ranges by offset; no validation is done.
+ *
+ * This method is used externally and must be retained with
+ * the signature unchanged.
+ * @param input input ranges.
+ * @return a new list of the ranges, sorted by offset.
+ */
+ @InterfaceStability.Stable
+ public static FileRange[] sortRanges(List input) {
+return sortRangeList(input).toArray(new FileRange[0]);
+ }
+
/**
* Merge sorted ranges to optimize the access from the underlying file
* system.
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
index 3fd3fe4d1f4..b08fc95279a 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
@@ -23,6 +23,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.IntBuffer;
import java.util.Collections;
+import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
@@ -47,6 +48,7 @@ import static
org.apache.hadoop.fs.VectoredReadUtils.isOrderedDisjoint;
import static org.apache.hadoop.fs.VectoredReadUtils.mergeSortedRanges;
import static org.apache.hadoop.fs.VectoredReadUtils.readRangeFrom;
import static org.apache.hadoop.fs.VectoredReadUtils.readVectored;
+import static org.apache.hadoop.fs.VectoredReadUtils.sortRangeList;
import static org.apache.hadoop.fs.VectoredReadUtils.sortRanges;
import static org.apache.hadoop.fs.VectoredReadUtils.validateAndSortRanges;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
@@ -196,7 +198,7 @@ public class TestVectoredReadUtils extends HadoopTestBase {
);
assertIsNotOrderedDisjoint(input, 100, 800);
final List outputList = mergeSortedRanges(
-sortRanges(input), 100, 1001, 2500);
+sortRangeList(input), 100, 1001, 2500);
assertRangeListSize(outputList, 1);
CombinedFileRange output = outputList.get(0);
@@ -208,7 +210,7 @@ public class TestVectoredReadUtils extends HadoopTestBase {
// the minSeek doesn't allow the first two to merge
assertIsNotOrderedDisjoint(input, 100, 100);
final List list2 = mergeSortedRanges(
-sortRanges(input),
+sortRangeList(input),
100, 1000, 2100);
assertRangeListSize(list2, 2);
assertRangeElement(list2, 0, 1000, 100)
(hadoop) branch branch-3.4 updated: HADOOP-19253. Google GCS compilation fails due to VectorIO changes (#7002)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new ed23a83b3cc HADOOP-19253. Google GCS compilation fails due to VectorIO
changes (#7002)
ed23a83b3cc is described below
commit ed23a83b3cc537f47789a5c3402425bf77e730dd
Author: Steve Loughran
AuthorDate: Mon Aug 19 19:54:47 2024 +0100
HADOOP-19253. Google GCS compilation fails due to VectorIO changes (#7002)
Fixes a compilation failure ncaused by HADOOP-19098
Restore original sortRanges() method signature,
FileRange[] sortRanges(List)
This ensures that google GCS connector will compile again.
It has also been marked as Stable so it is left alone
The version returning List
has been renamed sortRangeList()
Contributed by Steve Loughran
---
.../org/apache/hadoop/fs/VectoredReadUtils.java| 17 +--
.../hadoop/fs/impl/TestVectoredReadUtils.java | 35 ++
2 files changed, 44 insertions(+), 8 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
index fa0440620a4..2f99edc910c 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
@@ -308,7 +308,7 @@ public final class VectoredReadUtils {
validateRangeRequest(input.get(0));
sortedRanges = input;
} else {
- sortedRanges = sortRanges(input);
+ sortedRanges = sortRangeList(input);
FileRange prev = null;
for (final FileRange current : sortedRanges) {
validateRangeRequest(current);
@@ -341,12 +341,25 @@ public final class VectoredReadUtils {
* @param input input ranges.
* @return a new list of the ranges, sorted by offset.
*/
- public static List sortRanges(List
input) {
+ public static List sortRangeList(List input) {
final List l = new ArrayList<>(input);
l.sort(Comparator.comparingLong(FileRange::getOffset));
return l;
}
+ /**
+ * Sort the input ranges by offset; no validation is done.
+ *
+ * This method is used externally and must be retained with
+ * the signature unchanged.
+ * @param input input ranges.
+ * @return a new list of the ranges, sorted by offset.
+ */
+ @InterfaceStability.Stable
+ public static FileRange[] sortRanges(List input) {
+return sortRangeList(input).toArray(new FileRange[0]);
+ }
+
/**
* Merge sorted ranges to optimize the access from the underlying file
* system.
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
index 3fd3fe4d1f4..b08fc95279a 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
@@ -23,6 +23,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.IntBuffer;
import java.util.Collections;
+import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
@@ -47,6 +48,7 @@ import static
org.apache.hadoop.fs.VectoredReadUtils.isOrderedDisjoint;
import static org.apache.hadoop.fs.VectoredReadUtils.mergeSortedRanges;
import static org.apache.hadoop.fs.VectoredReadUtils.readRangeFrom;
import static org.apache.hadoop.fs.VectoredReadUtils.readVectored;
+import static org.apache.hadoop.fs.VectoredReadUtils.sortRangeList;
import static org.apache.hadoop.fs.VectoredReadUtils.sortRanges;
import static org.apache.hadoop.fs.VectoredReadUtils.validateAndSortRanges;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
@@ -196,7 +198,7 @@ public class TestVectoredReadUtils extends HadoopTestBase {
);
assertIsNotOrderedDisjoint(input, 100, 800);
final List outputList = mergeSortedRanges(
-sortRanges(input), 100, 1001, 2500);
+sortRangeList(input), 100, 1001, 2500);
assertRangeListSize(outputList, 1);
CombinedFileRange output = outputList.get(0);
@@ -208,7 +210,7 @@ public class TestVectoredReadUtils extends HadoopTestBase {
// the minSeek doesn't allow the first two to merge
assertIsNotOrderedDisjoint(input, 100, 100);
final List list2 = mergeSortedRanges(
-sortRanges(input),
+sortRangeList(input),
100, 1000, 2100);
assertRangeListSize(list2, 2);
assertRangeElement(list2, 0, 1000, 100)
(hadoop) branch branch-3.4.1 updated: HADOOP-19136. Upgrade commons-io to 2.16.1. (#6704)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.1 by this push: new 2daff9320b1 HADOOP-19136. Upgrade commons-io to 2.16.1. (#6704) 2daff9320b1 is described below commit 2daff9320b14f69a912d8d41e0cdb58fe2852cf3 Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Sat Aug 17 02:42:26 2024 +0800 HADOOP-19136. Upgrade commons-io to 2.16.1. (#6704) Contributed by Shilun Fan. --- LICENSE-binary | 2 +- hadoop-project/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 3533218abcd..252f934eac0 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -247,7 +247,7 @@ commons-cli:commons-cli:1.5.0 commons-codec:commons-codec:1.11 commons-collections:commons-collections:3.2.2 commons-daemon:commons-daemon:1.0.13 -commons-io:commons-io:2.14.0 +commons-io:commons-io:2.16.1 commons-net:commons-net:3.9.0 de.ruedigermoeller:fst:2.50 io.grpc:grpc-api:1.53.0 diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index a0e4186b801..9e08106bf0a 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -124,7 +124,7 @@ 3.2.2 1.26.1 1.9.0 -2.14.0 +2.16.1 3.12.0 1.2 3.6.1 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19136. Upgrade commons-io to 2.16.1. (#6704)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new 8a3a9509765 HADOOP-19136. Upgrade commons-io to 2.16.1. (#6704) 8a3a9509765 is described below commit 8a3a9509765fa2ed8a641700fca2c715f5c5aafb Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Sat Aug 17 02:42:26 2024 +0800 HADOOP-19136. Upgrade commons-io to 2.16.1. (#6704) Contributed by Shilun Fan. --- LICENSE-binary | 2 +- hadoop-project/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 3533218abcd..252f934eac0 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -247,7 +247,7 @@ commons-cli:commons-cli:1.5.0 commons-codec:commons-codec:1.11 commons-collections:commons-collections:3.2.2 commons-daemon:commons-daemon:1.0.13 -commons-io:commons-io:2.14.0 +commons-io:commons-io:2.16.1 commons-net:commons-net:3.9.0 de.ruedigermoeller:fst:2.50 io.grpc:grpc-api:1.53.0 diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 83fe729ef8f..5b1e55afd91 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -124,7 +124,7 @@ 3.2.2 1.26.1 1.9.0 -2.14.0 +2.16.1 3.12.0 1.2 3.6.1 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (bf804cb64be -> b5f88990b72)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from bf804cb64be HADOOP-19250. Fix test TestServiceInterruptHandling.testRegisterAndRaise (#6987) add b5f88990b72 HADOOP-19136. Upgrade commons-io to 2.16.1. (#6704) No new revisions were added by this update. Summary of changes: LICENSE-binary | 2 +- hadoop-project/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-thirdparty) branch trunk updated: Revert "HADOOP-19252. Release hadoop-thirdparty 1.3.0: version update on trunk"
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git The following commit(s) were added to refs/heads/trunk by this push: new cd3730b Revert "HADOOP-19252. Release hadoop-thirdparty 1.3.0: version update on trunk" cd3730b is described below commit cd3730b9a85657a628aeffd15bd84bd183a063dd Author: Steve Loughran AuthorDate: Fri Aug 16 18:49:18 2024 +0100 Revert "HADOOP-19252. Release hadoop-thirdparty 1.3.0: version update on trunk" Roll back so hadoop trunk builds can continue unaffected until the new artifacts are released This reverts commit bedef49c7f37ff2c7ce1ab37fa5725a925d4624a. --- hadoop-shaded-avro_1_11/pom.xml | 2 +- hadoop-shaded-guava/pom.xml | 2 +- hadoop-shaded-protobuf_3_25/pom.xml | 2 +- pom.xml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/hadoop-shaded-avro_1_11/pom.xml b/hadoop-shaded-avro_1_11/pom.xml index 5a07faf..12e4dab 100644 --- a/hadoop-shaded-avro_1_11/pom.xml +++ b/hadoop-shaded-avro_1_11/pom.xml @@ -23,7 +23,7 @@ hadoop-thirdparty org.apache.hadoop.thirdparty -1.4.0-SNAPSHOT +1.3.0-SNAPSHOT .. 4.0.0 diff --git a/hadoop-shaded-guava/pom.xml b/hadoop-shaded-guava/pom.xml index b03646f..fcfbb60 100644 --- a/hadoop-shaded-guava/pom.xml +++ b/hadoop-shaded-guava/pom.xml @@ -23,7 +23,7 @@ hadoop-thirdparty org.apache.hadoop.thirdparty -1.4.0-SNAPSHOT +1.3.0-SNAPSHOT ../pom.xml 4.0.0 diff --git a/hadoop-shaded-protobuf_3_25/pom.xml b/hadoop-shaded-protobuf_3_25/pom.xml index 9a90d5a..3e9a98d 100644 --- a/hadoop-shaded-protobuf_3_25/pom.xml +++ b/hadoop-shaded-protobuf_3_25/pom.xml @@ -23,7 +23,7 @@ hadoop-thirdparty org.apache.hadoop.thirdparty -1.4.0-SNAPSHOT +1.3.0-SNAPSHOT ../pom.xml 4.0.0 diff --git a/pom.xml b/pom.xml index 98879d8..e98cf77 100644 --- a/pom.xml +++ b/pom.xml @@ -23,7 +23,7 @@ 4.0.0 org.apache.hadoop.thirdparty hadoop-thirdparty - 1.4.0-SNAPSHOT + 1.3.0-SNAPSHOT org.apache apache - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19153. hadoop-common exports logback as a transitive dependency (#6999)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 5f93edfd707 HADOOP-19153. hadoop-common exports logback as a transitive dependency (#6999) 5f93edfd707 is described below commit 5f93edfd70784aa4f3ff392ef065c78a6fc532ea Author: Steve Loughran AuthorDate: Fri Aug 16 13:41:35 2024 +0100 HADOOP-19153. hadoop-common exports logback as a transitive dependency (#6999) - Critical: remove the obsolete exclusion list from hadoop-common. - Diligence: expand the hadoop-project exclusion list to exclude all ch.qos.logback artifacts Contributed by Steve Loughran --- hadoop-common-project/hadoop-common/pom.xml | 19 --- hadoop-project/pom.xml | 6 +- 2 files changed, 1 insertion(+), 24 deletions(-) diff --git a/hadoop-common-project/hadoop-common/pom.xml b/hadoop-common-project/hadoop-common/pom.xml index 90d66779734..06c6b06ec6a 100644 --- a/hadoop-common-project/hadoop-common/pom.xml +++ b/hadoop-common-project/hadoop-common/pom.xml @@ -330,25 +330,6 @@ org.apache.zookeeper zookeeper - - - org.jboss.netty - netty - - - - junit - junit - - - com.sun.jdmk - jmxtools - - - com.sun.jmx - jmxri - - io.netty diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 3f0a8b3a85f..8c8f675f98b 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -1484,11 +1484,7 @@ ch.qos.logback -logback-core - - -ch.qos.logback -logback-classic +* - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.1 updated: HADOOP-19153. hadoop-common exports logback as a transitive dependency (#6999)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.1 by this push: new 96a73b819bb HADOOP-19153. hadoop-common exports logback as a transitive dependency (#6999) 96a73b819bb is described below commit 96a73b819bbbaad2cd2f0a1d9e0e6356b67dfeec Author: Steve Loughran AuthorDate: Fri Aug 16 13:41:35 2024 +0100 HADOOP-19153. hadoop-common exports logback as a transitive dependency (#6999) - Critical: remove the obsolete exclusion list from hadoop-common. - Diligence: expand the hadoop-project exclusion list to exclude all ch.qos.logback artifacts Contributed by Steve Loughran --- hadoop-common-project/hadoop-common/pom.xml | 19 --- hadoop-project/pom.xml | 6 +- 2 files changed, 1 insertion(+), 24 deletions(-) diff --git a/hadoop-common-project/hadoop-common/pom.xml b/hadoop-common-project/hadoop-common/pom.xml index 66f89210d3a..31fd2923e99 100644 --- a/hadoop-common-project/hadoop-common/pom.xml +++ b/hadoop-common-project/hadoop-common/pom.xml @@ -330,25 +330,6 @@ org.apache.zookeeper zookeeper - - - org.jboss.netty - netty - - - - junit - junit - - - com.sun.jdmk - jmxtools - - - com.sun.jmx - jmxri - - io.netty diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 3e432963f4f..a0e4186b801 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -1479,11 +1479,7 @@ ch.qos.logback -logback-core - - -ch.qos.logback -logback-classic +* - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.1 updated (61f157cfc7a -> 89efe8cfc59)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch branch-3.4.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git from 61f157cfc7a HADOOP-19237. Upgrade to dnsjava 3.6.1 due to CVEs (#6961) (#6971) add 91f6840c780 HDFS-17591. RBF: Router should follow X-FRAME-OPTIONS protection setting (#6963) add ac6b2c2b657 HADOOP-19245. S3ABlockOutputStream no longer sends progress events in close() (#6974) add e2966036563 HADOOP-17609. Make SM4 support optional for OpenSSL native code. (#3019) add 5ea68e9548a HADOOP-19072. S3A: expand optimisations on stores with "fs.s3a.performance.flags" for mkdir (#6543) add dd483856f40 HADOOP-19072 S3A: Override fs.s3a.performance.flags for tests (ADDENDUM 2) (#6993) add 89efe8cfc59 HADOOP-19131. Assist reflection IO with WrappedOperations class (#6686) No new revisions were added by this update. Summary of changes: .../dev-support/findbugsExcludeFile.xml| 6 + .../org/apache/hadoop/crypto/OpensslCipher.java| 16 + .../hadoop/crypto/OpensslSm4CtrCryptoCodec.java| 4 + .../apache/hadoop/fs/CommonPathCapabilities.java | 16 + .../org/apache/hadoop/fs/FSDataInputStream.java| 8 + .../main/java/org/apache/hadoop/fs/Options.java| 65 +- .../org/apache/hadoop/fs/RawLocalFileSystem.java | 2 + .../org/apache/hadoop/io/wrappedio/WrappedIO.java | 149 - .../hadoop/io/wrappedio/WrappedStatistics.java | 357 +++ .../hadoop/io/wrappedio/impl/DynamicWrappedIO.java | 500 +++ .../wrappedio/impl/DynamicWrappedStatistics.java | 678 + .../wrappedio/impl/package-info.java} | 23 +- .../wrappedio/package-info.java} | 33 +- .../apache/hadoop/util/dynamic/BindingUtils.java | 214 +++ .../hadoop/util/dynamic/DynConstructors.java | 273 + .../org/apache/hadoop/util/dynamic/DynMethods.java | 544 + .../package-info.java} | 25 +- .../util/functional/BiFunctionRaisingIOE.java | 16 + .../hadoop/util/functional/CallableRaisingIOE.java | 19 + .../hadoop/util/functional/FunctionRaisingIOE.java | 15 + .../hadoop/util/functional/FunctionalIO.java | 23 +- .../org/apache/hadoop/util/functional/Tuples.java | 17 + .../src/org/apache/hadoop/crypto/OpensslCipher.c | 26 +- .../filesystem/fsdatainputstreambuilder.md | 95 ++- .../filesystem/fsdataoutputstreambuilder.md| 4 +- .../org/apache/hadoop/crypto/TestCryptoCodec.java | 13 +- ...tCryptoStreamsWithOpensslSm4CtrCryptoCodec.java | 2 + .../apache/hadoop/crypto/TestOpensslCipher.java| 10 + .../hadoop/fs/FileContextCreateMkdirBaseTest.java | 21 +- .../contract/AbstractContractBulkDeleteTest.java | 28 +- .../fs/contract/AbstractContractMkdirTest.java | 7 +- .../hadoop/fs/contract/ContractTestUtils.java | 18 + .../hadoop/io/wrappedio/impl/TestWrappedIO.java| 484 +++ .../io/wrappedio/impl/TestWrappedStatistics.java | 496 +++ .../apache/hadoop/util/dynamic/Concatenator.java | 85 +++ .../hadoop/util/dynamic/TestDynConstructors.java | 170 ++ .../apache/hadoop/util/dynamic/TestDynMethods.java | 320 ++ .../hadoop/util/functional/TestFunctionalIO.java | 14 + .../src/test/resources/log4j.properties| 4 +- .../server/federation/router/RouterHttpServer.java | 11 + .../router/TestRouterHttpServerXFrame.java | 65 ++ .../hadoop/fs/contract/hdfs/TestDFSWrappedIO.java | 41 +- .../hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java | 17 + .../apache/hadoop/fs/s3a/S3ABlockOutputStream.java | 3 +- .../org/apache/hadoop/fs/s3a/S3AFileSystem.java| 11 +- .../org/apache/hadoop/fs/s3a/S3AInputPolicy.java | 24 +- .../apache/hadoop/fs/s3a/impl/MkdirOperation.java | 77 ++- .../site/markdown/tools/hadoop-aws/performance.md | 21 +- .../fs/contract/s3a/ITestS3AContractCreate.java| 12 +- .../fs/contract/s3a/ITestS3AContractMkdir.java | 9 + .../s3a/ITestS3AContractMkdirWithCreatePerf.java | 68 +++ .../contract/s3a/ITestS3AContractVectoredRead.java | 4 +- .../hadoop/fs/contract/s3a/ITestS3AWrappedIO.java | 23 +- .../hadoop/fs/s3a/ITestS3AFSMainOperations.java| 6 +- .../hadoop/fs/s3a/ITestS3AFileOperationCost.java | 13 +- .../hadoop/fs/s3a/ITestS3AFileSystemContract.java | 5 +- .../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 19 + .../ITestS3AFileContextCreateMkdir.java| 9 +- ... ITestS3AFileContextCreateMkdirCreatePerf.java} | 32 +- .../ITestS3AFileContextMainOperations.java | 7 +- .../fs/s3a/fileContext/ITestS3AFileContextURI.java | 6 +- .../hadoop/fs/s3a/impl/TestOpenFileSupport.java| 43 +- .../fs/s3a/performance/ITestCreateFileCost.java| 19 +- .../fs/s3a/performance/ITestS3ADeleteCost.java | 13 +- .../fs/s3a/p
(hadoop) branch branch-3.4 updated: HADOOP-19153. hadoop-common exports logback as a transitive dependency (#6999)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new d93e0b0f02a HADOOP-19153. hadoop-common exports logback as a transitive dependency (#6999) d93e0b0f02a is described below commit d93e0b0f02a2f8949752bf3cee00e699e87ffeaa Author: Steve Loughran AuthorDate: Fri Aug 16 13:41:35 2024 +0100 HADOOP-19153. hadoop-common exports logback as a transitive dependency (#6999) - Critical: remove the obsolete exclusion list from hadoop-common. - Diligence: expand the hadoop-project exclusion list to exclude all ch.qos.logback artifacts Contributed by Steve Loughran --- hadoop-common-project/hadoop-common/pom.xml | 19 --- hadoop-project/pom.xml | 6 +- 2 files changed, 1 insertion(+), 24 deletions(-) diff --git a/hadoop-common-project/hadoop-common/pom.xml b/hadoop-common-project/hadoop-common/pom.xml index 5cd8a5e71da..4dc92419be0 100644 --- a/hadoop-common-project/hadoop-common/pom.xml +++ b/hadoop-common-project/hadoop-common/pom.xml @@ -330,25 +330,6 @@ org.apache.zookeeper zookeeper - - - org.jboss.netty - netty - - - - junit - junit - - - com.sun.jdmk - jmxtools - - - com.sun.jmx - jmxri - - io.netty diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index d328e1b650e..83fe729ef8f 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -1479,11 +1479,7 @@ ch.qos.logback -logback-core - - -ch.qos.logback -logback-classic +* - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19072 S3A: Override fs.s3a.performance.flags for tests (ADDENDUM 2) (#6993)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 54c90d70942 HADOOP-19072 S3A: Override fs.s3a.performance.flags for
tests (ADDENDUM 2) (#6993)
54c90d70942 is described below
commit 54c90d70942f27b1feeaf2a1746137c4eba6a6b3
Author: Viraj Jasani
AuthorDate: Wed Aug 14 02:57:44 2024 -0700
HADOOP-19072 S3A: Override fs.s3a.performance.flags for tests (ADDENDUM 2)
(#6993)
Second followup to #6543; all hadoop-aws integration tests complete
correctly even when
fs.s3a.performance.flags = *
Contributed by Viraj Jasani
---
.../fs/contract/s3a/ITestS3AContractCreate.java | 12
.../hadoop/fs/contract/s3a/ITestS3AContractMkdir.java | 14 --
.../s3a/ITestS3AContractMkdirWithCreatePerf.java | 13 +++--
.../hadoop/fs/s3a/ITestS3AFSMainOperations.java | 6 +-
.../hadoop/fs/s3a/ITestS3AFileOperationCost.java | 13 -
.../hadoop/fs/s3a/ITestS3AFileSystemContract.java | 5 -
.../java/org/apache/hadoop/fs/s3a/S3ATestUtils.java | 19 +++
.../fileContext/ITestS3AFileContextCreateMkdir.java | 12
.../ITestS3AFileContextCreateMkdirCreatePerf.java | 12 +++-
.../ITestS3AFileContextMainOperations.java| 7 ++-
.../fs/s3a/fileContext/ITestS3AFileContextURI.java| 6 +-
.../fs/s3a/performance/ITestCreateFileCost.java | 11 ---
.../hadoop/fs/s3a/performance/ITestS3ADeleteCost.java | 13 -
13 files changed, 69 insertions(+), 74 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java
index a1067ddc0ec..a6590e99e6c 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java
@@ -29,9 +29,7 @@ import
org.apache.hadoop.fs.contract.AbstractContractCreateTest;
import org.apache.hadoop.fs.contract.AbstractFSContract;
import org.apache.hadoop.fs.s3a.S3ATestUtils;
-import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_PERFORMANCE;
-import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
-import static
org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.setPerformanceFlags;
/**
* S3A contract tests creating files.
@@ -70,11 +68,9 @@ public class ITestS3AContractCreate extends
AbstractContractCreateTest {
@Override
protected Configuration createConfiguration() {
-final Configuration conf = super.createConfiguration();
-removeBaseAndBucketOverrides(conf,
-FS_S3A_CREATE_PERFORMANCE,
-FS_S3A_PERFORMANCE_FLAGS);
-conf.setBoolean(FS_S3A_CREATE_PERFORMANCE, createPerformance);
+final Configuration conf = setPerformanceFlags(
+super.createConfiguration(),
+createPerformance ? "create" : "");
S3ATestUtils.disableFilesystemCaching(conf);
return conf;
}
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java
index bce67ed67f3..847f6980b56 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java
@@ -22,9 +22,7 @@ import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.contract.AbstractContractMkdirTest;
import org.apache.hadoop.fs.contract.AbstractFSContract;
-import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_PERFORMANCE;
-import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
-import static
org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.setPerformanceFlags;
/**
* Test dir operations on S3A.
@@ -33,13 +31,9 @@ public class ITestS3AContractMkdir extends
AbstractContractMkdirTest {
@Override
protected Configuration createConfiguration() {
-Configuration conf = super.createConfiguration();
-removeBaseAndBucketOverrides(
-conf,
-FS_S3A_CREATE_PERFORMANCE,
-FS_S3A_PERFORMANCE_FLAGS);
-conf.set(FS_S3A_PERFORMANCE_FLAGS, "");
-return conf;
+return setPerformanceFlags(
+super.createConfiguration(),
+"");
}
@Override
diff --git
a/hadoop-tools/hadoop-aws/src/tes
(hadoop) branch trunk updated: HADOOP-19072 S3A: Override fs.s3a.performance.flags for tests (ADDENDUM 2) (#6993)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new fa83c9a8050 HADOOP-19072 S3A: Override fs.s3a.performance.flags for
tests (ADDENDUM 2) (#6993)
fa83c9a8050 is described below
commit fa83c9a805041b94b3663b773e99e8074c534770
Author: Viraj Jasani
AuthorDate: Wed Aug 14 02:57:44 2024 -0700
HADOOP-19072 S3A: Override fs.s3a.performance.flags for tests (ADDENDUM 2)
(#6993)
Second followup to #6543; all hadoop-aws integration tests complete
correctly even when
fs.s3a.performance.flags = *
Contributed by Viraj Jasani
---
.../fs/contract/s3a/ITestS3AContractCreate.java | 12
.../hadoop/fs/contract/s3a/ITestS3AContractMkdir.java | 14 --
.../s3a/ITestS3AContractMkdirWithCreatePerf.java | 13 +++--
.../hadoop/fs/s3a/ITestS3AFSMainOperations.java | 6 +-
.../hadoop/fs/s3a/ITestS3AFileOperationCost.java | 13 -
.../hadoop/fs/s3a/ITestS3AFileSystemContract.java | 5 -
.../java/org/apache/hadoop/fs/s3a/S3ATestUtils.java | 19 +++
.../fileContext/ITestS3AFileContextCreateMkdir.java | 12
.../ITestS3AFileContextCreateMkdirCreatePerf.java | 12 +++-
.../ITestS3AFileContextMainOperations.java| 7 ++-
.../fs/s3a/fileContext/ITestS3AFileContextURI.java| 6 +-
.../fs/s3a/performance/ITestCreateFileCost.java | 11 ---
.../hadoop/fs/s3a/performance/ITestS3ADeleteCost.java | 13 -
13 files changed, 69 insertions(+), 74 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java
index a1067ddc0ec..a6590e99e6c 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractCreate.java
@@ -29,9 +29,7 @@ import
org.apache.hadoop.fs.contract.AbstractContractCreateTest;
import org.apache.hadoop.fs.contract.AbstractFSContract;
import org.apache.hadoop.fs.s3a.S3ATestUtils;
-import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_PERFORMANCE;
-import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
-import static
org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.setPerformanceFlags;
/**
* S3A contract tests creating files.
@@ -70,11 +68,9 @@ public class ITestS3AContractCreate extends
AbstractContractCreateTest {
@Override
protected Configuration createConfiguration() {
-final Configuration conf = super.createConfiguration();
-removeBaseAndBucketOverrides(conf,
-FS_S3A_CREATE_PERFORMANCE,
-FS_S3A_PERFORMANCE_FLAGS);
-conf.setBoolean(FS_S3A_CREATE_PERFORMANCE, createPerformance);
+final Configuration conf = setPerformanceFlags(
+super.createConfiguration(),
+createPerformance ? "create" : "");
S3ATestUtils.disableFilesystemCaching(conf);
return conf;
}
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java
index bce67ed67f3..847f6980b56 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java
@@ -22,9 +22,7 @@ import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.contract.AbstractContractMkdirTest;
import org.apache.hadoop.fs.contract.AbstractFSContract;
-import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_PERFORMANCE;
-import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
-import static
org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.setPerformanceFlags;
/**
* Test dir operations on S3A.
@@ -33,13 +31,9 @@ public class ITestS3AContractMkdir extends
AbstractContractMkdirTest {
@Override
protected Configuration createConfiguration() {
-Configuration conf = super.createConfiguration();
-removeBaseAndBucketOverrides(
-conf,
-FS_S3A_CREATE_PERFORMANCE,
-FS_S3A_PERFORMANCE_FLAGS);
-conf.set(FS_S3A_PERFORMANCE_FLAGS, "");
-return conf;
+return setPerformanceFlags(
+super.createConfiguration(),
+"");
}
@Override
diff --git
a/hadoop-tools/hadoop-aws/src/tes
(hadoop) branch branch-3.4 updated: HADOOP-19072. S3A: expand optimisations on stores with "fs.s3a.performance.flags" for mkdir (#6543)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 0755b93ce45 HADOOP-19072. S3A: expand optimisations on stores with
"fs.s3a.performance.flags" for mkdir (#6543)
0755b93ce45 is described below
commit 0755b93ce450f269eb211fb985c8a9ff9048a328
Author: Viraj Jasani
AuthorDate: Thu Aug 8 09:48:51 2024 -0700
HADOOP-19072. S3A: expand optimisations on stores with
"fs.s3a.performance.flags" for mkdir (#6543)
If the flag list in fs.s3a.performance.flags includes "mkdir"
then the safety check of a walk up the tree to look for a parent directory,
-done to verify a directory isn't being created under a file- are skipped.
This saves the cost of multiple list operations.
Includes:
HADOOP-19072. S3A: Override fs.s3a.performance.flags for tests (ADDENDUM)
(#6985)
This is a followup to #6543 which ensures all test pass in configurations
where
fs.s3a.performance.flags is set to "*" or contains "mkdirs"
Contributed by VJ Jasani
---
.../filesystem/fsdataoutputstreambuilder.md| 4 +-
.../hadoop/fs/FileContextCreateMkdirBaseTest.java | 21 +++---
.../fs/contract/AbstractContractMkdirTest.java | 7 +-
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java| 7 +-
.../apache/hadoop/fs/s3a/impl/MkdirOperation.java | 77 +-
.../site/markdown/tools/hadoop-aws/performance.md | 21 +-
.../fs/contract/s3a/ITestS3AContractMkdir.java | 15 +
.../s3a/ITestS3AContractMkdirWithCreatePerf.java | 75 +
.../ITestS3AFileContextCreateMkdir.java| 11 +++-
.../ITestS3AFileContextCreateMkdirCreatePerf.java | 67 +++
10 files changed, 271 insertions(+), 34 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdataoutputstreambuilder.md
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdataoutputstreambuilder.md
index 5f24e755697..7dd3170036c 100644
---
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdataoutputstreambuilder.md
+++
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdataoutputstreambuilder.md
@@ -200,8 +200,8 @@ Prioritize file creation performance over safety checks for
filesystem consisten
This:
1. Skips the `LIST` call which makes sure a file is being created over a
directory.
Risk: a file is created over a directory.
-1. Ignores the overwrite flag.
-1. Never issues a `DELETE` call to delete parent directory markers.
+2. Ignores the overwrite flag.
+3. Never issues a `DELETE` call to delete parent directory markers.
It is possible to probe an S3A Filesystem instance for this capability through
the `hasPathCapability(path, "fs.s3a.create.performance")` check.
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextCreateMkdirBaseTest.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextCreateMkdirBaseTest.java
index fbd598c9deb..fcb1b6925a4 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextCreateMkdirBaseTest.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextCreateMkdirBaseTest.java
@@ -27,6 +27,7 @@ import org.junit.Test;
import static org.apache.hadoop.fs.FileContextTestHelper.*;
import static
org.apache.hadoop.fs.contract.ContractTestUtils.assertIsDirectory;
import static org.apache.hadoop.fs.contract.ContractTestUtils.assertIsFile;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
import org.apache.hadoop.test.GenericTestUtils;
import org.slf4j.event.Level;
@@ -55,7 +56,10 @@ public abstract class FileContextCreateMkdirBaseTest {
protected final FileContextTestHelper fileContextTestHelper;
protected static FileContext fc;
-
+
+ public static final String MKDIR_FILE_PRESENT_ERROR =
+ " should have failed as a file was present";
+
static {
GenericTestUtils.setLogLevel(FileSystem.LOG, Level.DEBUG);
}
@@ -128,7 +132,7 @@ public abstract class FileContextCreateMkdirBaseTest {
}
@Test
- public void testMkdirRecursiveWithExistingFile() throws IOException {
+ public void testMkdirRecursiveWithExistingFile() throws Exception {
Path f = getTestRootPath(fc, "NonExistant3/aDir");
fc.mkdir(f, FileContext.DEFAULT_PERM, true);
assertIsDirectory(fc.getFileStatus(f));
@@ -141,13 +145,12 @@ public abstract class FileContextCreateMkdirBaseTest {
// try creating another folder which conflicts with filePath
Path dirPath = new Path(filePath, "bDir/cDir");
-try {
- fc.mkdir(dirPath, FileContext.DEFAULT_PER
(hadoop) branch trunk updated (321a6cc55ed -> 74ff00705cf)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 321a6cc55ed HADOOP-19072. S3A: expand optimisations on stores with "fs.s3a.performance.flags" for mkdir (#6543) add 74ff00705cf HADOOP-19072. S3A: Override fs.s3a.performance.flags for tests (ADDENDUM) (#6985) No new revisions were added by this update. Summary of changes: .../org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java | 8 ++-- .../hadoop/fs/s3a/fileContext/ITestS3AFileContextCreateMkdir.java | 4 +++- 2 files changed, 9 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19072. S3A: expand optimisations on stores with "fs.s3a.performance.flags" for mkdir (#6543)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 321a6cc55ed HADOOP-19072. S3A: expand optimisations on stores with
"fs.s3a.performance.flags" for mkdir (#6543)
321a6cc55ed is described below
commit 321a6cc55ed2df5222bde7b5c801322e8cf68203
Author: Viraj Jasani
AuthorDate: Thu Aug 8 09:48:51 2024 -0700
HADOOP-19072. S3A: expand optimisations on stores with
"fs.s3a.performance.flags" for mkdir (#6543)
If the flag list in fs.s3a.performance.flags includes "mkdir"
then the safety check of a walk up the tree to look for a parent directory,
-done to verify a directory isn't being created under a file- are skipped.
This saves the cost of multiple list operations.
Contributed by Viraj Jasani
---
.../filesystem/fsdataoutputstreambuilder.md| 4 +-
.../hadoop/fs/FileContextCreateMkdirBaseTest.java | 21 +++---
.../fs/contract/AbstractContractMkdirTest.java | 7 +-
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java| 7 +-
.../apache/hadoop/fs/s3a/impl/MkdirOperation.java | 77 +-
.../site/markdown/tools/hadoop-aws/performance.md | 21 +-
.../fs/contract/s3a/ITestS3AContractMkdir.java | 11
.../s3a/ITestS3AContractMkdirWithCreatePerf.java | 75 +
.../ITestS3AFileContextCreateMkdir.java| 9 ++-
.../ITestS3AFileContextCreateMkdirCreatePerf.java | 67 +++
10 files changed, 265 insertions(+), 34 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdataoutputstreambuilder.md
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdataoutputstreambuilder.md
index 5f24e755697..7dd3170036c 100644
---
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdataoutputstreambuilder.md
+++
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdataoutputstreambuilder.md
@@ -200,8 +200,8 @@ Prioritize file creation performance over safety checks for
filesystem consisten
This:
1. Skips the `LIST` call which makes sure a file is being created over a
directory.
Risk: a file is created over a directory.
-1. Ignores the overwrite flag.
-1. Never issues a `DELETE` call to delete parent directory markers.
+2. Ignores the overwrite flag.
+3. Never issues a `DELETE` call to delete parent directory markers.
It is possible to probe an S3A Filesystem instance for this capability through
the `hasPathCapability(path, "fs.s3a.create.performance")` check.
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextCreateMkdirBaseTest.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextCreateMkdirBaseTest.java
index fbd598c9deb..fcb1b6925a4 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextCreateMkdirBaseTest.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextCreateMkdirBaseTest.java
@@ -27,6 +27,7 @@ import org.junit.Test;
import static org.apache.hadoop.fs.FileContextTestHelper.*;
import static
org.apache.hadoop.fs.contract.ContractTestUtils.assertIsDirectory;
import static org.apache.hadoop.fs.contract.ContractTestUtils.assertIsFile;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
import org.apache.hadoop.test.GenericTestUtils;
import org.slf4j.event.Level;
@@ -55,7 +56,10 @@ public abstract class FileContextCreateMkdirBaseTest {
protected final FileContextTestHelper fileContextTestHelper;
protected static FileContext fc;
-
+
+ public static final String MKDIR_FILE_PRESENT_ERROR =
+ " should have failed as a file was present";
+
static {
GenericTestUtils.setLogLevel(FileSystem.LOG, Level.DEBUG);
}
@@ -128,7 +132,7 @@ public abstract class FileContextCreateMkdirBaseTest {
}
@Test
- public void testMkdirRecursiveWithExistingFile() throws IOException {
+ public void testMkdirRecursiveWithExistingFile() throws Exception {
Path f = getTestRootPath(fc, "NonExistant3/aDir");
fc.mkdir(f, FileContext.DEFAULT_PERM, true);
assertIsDirectory(fc.getFileStatus(f));
@@ -141,13 +145,12 @@ public abstract class FileContextCreateMkdirBaseTest {
// try creating another folder which conflicts with filePath
Path dirPath = new Path(filePath, "bDir/cDir");
-try {
- fc.mkdir(dirPath, FileContext.DEFAULT_PERM, true);
- Assert.fail("Mkdir for " + dirPath
- + " should have failed as a file was present");
-} catch(IOException e) {
- // failed as expected
-}
+intercept(
+IOException.class,
+null,
+"Mkdir for &qu
(hadoop) branch branch-3.4 updated: HADOOP-19245. S3ABlockOutputStream no longer sends progress events in close() (#6974)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new b9b650b8437 HADOOP-19245. S3ABlockOutputStream no longer sends
progress events in close() (#6974)
b9b650b8437 is described below
commit b9b650b8437e697b201fc16ac4dddb4e41c4acc1
Author: Steve Loughran
AuthorDate: Fri Aug 2 16:01:03 2024 +0100
HADOOP-19245. S3ABlockOutputStream no longer sends progress events in
close() (#6974)
Contributed by Steve Loughran
---
.../main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java | 3 ++-
.../org/apache/hadoop/fs/s3a/performance/ITestCreateFileCost.java | 8
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
index de0f59154e9..5fe39ac6ea3 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
@@ -1100,7 +1100,8 @@ class S3ABlockOutputStream extends OutputStream implements
this.progress = progress;
}
-public void progressChanged(ProgressListenerEvent eventType, int
bytesTransferred) {
+@Override
+public void progressChanged(ProgressListenerEvent eventType, long
bytesTransferred) {
if (progress != null) {
progress.progress();
}
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestCreateFileCost.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestCreateFileCost.java
index c9a7415c181..5bd4bf412ff 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestCreateFileCost.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestCreateFileCost.java
@@ -21,6 +21,7 @@ package org.apache.hadoop.fs.s3a.performance;
import java.io.IOException;
import java.util.Arrays;
import java.util.Collection;
+import java.util.concurrent.atomic.AtomicLong;
import org.assertj.core.api.Assertions;
import org.junit.Test;
@@ -213,8 +214,11 @@ public class ITestCreateFileCost extends
AbstractS3ACostTest {
S3AFileSystem fs = getFileSystem();
Path path = methodPath();
+// increment progress events
+AtomicLong progressEvents = new AtomicLong(0);
FSDataOutputStreamBuilder builder = fs.createFile(path)
.overwrite(false)
+.progress(progressEvents::incrementAndGet)
.recursive();
// this has a broken return type; something to do with the return value of
@@ -225,6 +229,10 @@ public class ITestCreateFileCost extends
AbstractS3ACostTest {
always(NO_HEAD_OR_LIST),
with(OBJECT_BULK_DELETE_REQUEST, 0),
with(OBJECT_DELETE_REQUEST, 0));
+
+Assertions.assertThat(progressEvents.get())
+.describedAs("progress events")
+.isGreaterThanOrEqualTo(1);
}
@Test
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19245. S3ABlockOutputStream no longer sends progress events in close() (#6974)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 2cf4d638af3 HADOOP-19245. S3ABlockOutputStream no longer sends
progress events in close() (#6974)
2cf4d638af3 is described below
commit 2cf4d638af3520d60a892c94d39cf7a3a784f8f9
Author: Steve Loughran
AuthorDate: Fri Aug 2 16:01:03 2024 +0100
HADOOP-19245. S3ABlockOutputStream no longer sends progress events in
close() (#6974)
Contributed by Steve Loughran
---
.../main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java | 3 ++-
.../org/apache/hadoop/fs/s3a/performance/ITestCreateFileCost.java | 8
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
index de0f59154e9..5fe39ac6ea3 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java
@@ -1100,7 +1100,8 @@ class S3ABlockOutputStream extends OutputStream implements
this.progress = progress;
}
-public void progressChanged(ProgressListenerEvent eventType, int
bytesTransferred) {
+@Override
+public void progressChanged(ProgressListenerEvent eventType, long
bytesTransferred) {
if (progress != null) {
progress.progress();
}
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestCreateFileCost.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestCreateFileCost.java
index c9a7415c181..5bd4bf412ff 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestCreateFileCost.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestCreateFileCost.java
@@ -21,6 +21,7 @@ package org.apache.hadoop.fs.s3a.performance;
import java.io.IOException;
import java.util.Arrays;
import java.util.Collection;
+import java.util.concurrent.atomic.AtomicLong;
import org.assertj.core.api.Assertions;
import org.junit.Test;
@@ -213,8 +214,11 @@ public class ITestCreateFileCost extends
AbstractS3ACostTest {
S3AFileSystem fs = getFileSystem();
Path path = methodPath();
+// increment progress events
+AtomicLong progressEvents = new AtomicLong(0);
FSDataOutputStreamBuilder builder = fs.createFile(path)
.overwrite(false)
+.progress(progressEvents::incrementAndGet)
.recursive();
// this has a broken return type; something to do with the return value of
@@ -225,6 +229,10 @@ public class ITestCreateFileCost extends
AbstractS3ACostTest {
always(NO_HEAD_OR_LIST),
with(OBJECT_BULK_DELETE_REQUEST, 0),
with(OBJECT_DELETE_REQUEST, 0));
+
+Assertions.assertThat(progressEvents.get())
+.describedAs("progress events")
+.isGreaterThanOrEqualTo(1);
}
@Test
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19237. Upgrade to dnsjava 3.6.1 due to CVEs (#6961)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new c593c17255c HADOOP-19237. Upgrade to dnsjava 3.6.1 due to CVEs (#6961)
c593c17255c is described below
commit c593c17255c06a32b01055e2f4bb2394009bd94a
Author: PJ Fanning
AuthorDate: Thu Aug 1 20:07:36 2024 +0100
HADOOP-19237. Upgrade to dnsjava 3.6.1 due to CVEs (#6961)
Contributed by P J Fanning
---
LICENSE-binary| 2 +-
.../src/test/resources/ensure-jars-have-correct-contents.sh | 2 ++
hadoop-client-modules/hadoop-client-runtime/pom.xml | 3 +++
.../java/org/apache/hadoop/registry/server/dns/RegistryDNS.java | 2 +-
.../org/apache/hadoop/registry/server/dns/TestRegistryDNS.java| 8
hadoop-project/pom.xml| 2 +-
6 files changed, 12 insertions(+), 7 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index ff8012096a4..c0eb82f3dab 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -379,7 +379,7 @@
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/util/tree
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/compat/{fstatat|openat|unlinkat}.h
com.github.luben:zstd-jni:1.5.2-1
-dnsjava:dnsjava:2.1.7
+dnsjava:dnsjava:3.6.1
org.codehaus.woodstox:stax2-api:4.2.1
diff --git
a/hadoop-client-modules/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh
b/hadoop-client-modules/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh
index 2e927402d25..3a7c5ce7860 100644
---
a/hadoop-client-modules/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh
+++
b/hadoop-client-modules/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh
@@ -51,6 +51,8 @@ allowed_expr+="|^[^-]*-default.xml$"
allowed_expr+="|^[^-]*-version-info.properties$"
# * Hadoop's application classloader properties file.
allowed_expr+="|^org.apache.hadoop.application-classloader.properties$"
+# Comes from dnsjava, not sure if relocatable.
+allowed_expr+="|^messages.properties$"
# public suffix list used by httpcomponents
allowed_expr+="|^mozilla/$"
allowed_expr+="|^mozilla/public-suffix-list.txt$"
diff --git a/hadoop-client-modules/hadoop-client-runtime/pom.xml
b/hadoop-client-modules/hadoop-client-runtime/pom.xml
index 22c8ae00a3a..8c72f53e918 100644
--- a/hadoop-client-modules/hadoop-client-runtime/pom.xml
+++ b/hadoop-client-modules/hadoop-client-runtime/pom.xml
@@ -229,6 +229,8 @@
jnamed*
lookup*
update*
+META-INF/versions/21/*
+META-INF/versions/21/**/*
@@ -243,6 +245,7 @@
META-INF/versions/9/module-info.class
META-INF/versions/11/module-info.class
+
META-INF/versions/21/module-info.class
diff --git
a/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/dns/RegistryDNS.java
b/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/dns/RegistryDNS.java
index b6de757fc3c..e99c49f7dc6 100644
---
a/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/dns/RegistryDNS.java
+++
b/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/dns/RegistryDNS.java
@@ -1682,7 +1682,7 @@ public class RegistryDNS extends AbstractService
implements DNSOperations,
DNSSEC.sign(rRset, dnskeyRecord, privateKey,
inception, expiration);
LOG.info("Adding {}", rrsigRecord);
- rRset.addRR(rrsigRecord);
+ zone.addRecord(rrsigRecord);
//addDSRecord(zone, record.getName(), record.getDClass(),
// record.getTTL(), inception, expiration);
diff --git
a/hadoop-common-project/hadoop-registry/src/test/java/org/apache/hadoop/registry/server/dns/TestRegistryDNS.java
b/hadoop-common-project/hadoop-registry/src/test/java/org/apache/hadoop/registry/server/dns/TestRegistryDNS.java
index 56e617144ad..386cb3a196c 100644
---
a/hadoop-common-project/hadoop-registry/src/test/java/org/apache/hadoop/registry/server/dns/TestRegistryDNS.java
+++
b/hadoop-common-project/hadoop-registry/src/test/java/org/apache/hadoop/registry/server/dns/TestRegistryDNS.java
@@ -350,7 +350,7 @@ public class TestRe
(hadoop) branch trunk updated (a5806a9e7bc -> 038636a1b52)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from a5806a9e7bc HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags (#6789) add 038636a1b52 HADOOP-19238. Fix create-release script for arm64 based MacOS (#6962) No new revisions were added by this update. Summary of changes: dev-support/bin/create-release | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags (#6789) (#6966)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 414c4a2529a HADOOP-19161. S3A: option "fs.s3a.performance.flags" to
take list of performance flags (#6789) (#6966)
414c4a2529a is described below
commit 414c4a2529a6725888eea2b6740ca7bbbc8a6f1d
Author: Steve Loughran
AuthorDate: Mon Jul 29 16:06:30 2024 +0100
HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of
performance flags (#6789) (#6966)
1. Configuration adds new method `getEnumSet()` to get a set of enum values
from
a configuration string.
> EnumSet getEnumSet(String key, Class
enumClass, boolean ignoreUnknown)
Whitespace is ignored, case is ignored and the value "*" is mapped to
"all values of the enum".
If "ignoreUnknown" is true then when parsing, unknown values are ignored.
This is recommended for forward compatiblity with later versions.
2. This support is implemented in
org.apache.hadoop.fs.s3a.impl.ConfigurationHelper -it can be used
elsewhere in the hadoop codebase.
3. A new private FlagSet class in hadoop common manages a set of enum flags.
It implements StreamCapabilities and can be probed for a specific
option being set
(with a prefix)
S3A adds an option fs.s3a.performance.flags which builds a FlagSet with enum
type PerformanceFlagEnum
* which initially contains {Create, Delete, Mkdir, Open}
* the existing fs.s3a.create.performance option sets the flag "Create".
* tests which configure fs.s3a.create.performance MUST clear
fs.s3a.performance.flags in test setup.
Future performance flags are planned, with different levels of safety
and/or backwards compatibility.
Contributed by Steve Loughran
---
.../java/org/apache/hadoop/conf/Configuration.java | 22 ++
.../java/org/apache/hadoop/fs/impl/FlagSet.java| 327
.../apache/hadoop/util/ConfigurationHelper.java| 126 ++
.../org/apache/hadoop/fs/impl/TestFlagSet.java | 431 +
.../hadoop/util/TestConfigurationHelper.java | 174 +
.../java/org/apache/hadoop/fs/s3a/Constants.java | 5 +
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java| 82 +++-
.../hadoop/fs/s3a/api/PerformanceFlagEnum.java | 51 +++
.../apache/hadoop/fs/s3a/impl/StoreContext.java| 19 +-
.../hadoop/fs/s3a/impl/StoreContextBuilder.java| 17 +-
.../apache/hadoop/fs/s3a/s3guard/S3GuardTool.java | 9 +-
.../site/markdown/tools/hadoop-aws/performance.md | 110 --
.../fs/contract/s3a/ITestS3AContractCreate.java| 4 +-
.../hadoop/fs/s3a/ITestS3AFileOperationCost.java | 5 +-
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 5 +
.../fs/s3a/impl/ITestConnectionTimeouts.java | 4 +-
.../fs/s3a/performance/AbstractS3ACostTest.java| 3 +-
.../fs/s3a/performance/ITestCreateFileCost.java| 4 +-
.../performance/ITestDirectoryMarkerListing.java | 4 +-
.../fs/s3a/performance/ITestS3ADeleteCost.java | 5 +-
.../fs/s3a/tools/AbstractMarkerToolTest.java | 3 +-
21 files changed, 1350 insertions(+), 60 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index ea3d6dc74e4..44579b90337 100755
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -49,6 +49,7 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
+import java.util.EnumSet;
import java.util.Enumeration;
import java.util.HashMap;
import java.util.HashSet;
@@ -99,6 +100,7 @@ import org.apache.hadoop.security.alias.CredentialProvider;
import org.apache.hadoop.security.alias.CredentialProvider.CredentialEntry;
import org.apache.hadoop.security.alias.CredentialProviderFactory;
import org.apache.hadoop.thirdparty.com.google.common.base.Strings;
+import org.apache.hadoop.util.ConfigurationHelper;
import org.apache.hadoop.util.Preconditions;
import org.apache.hadoop.util.ReflectionUtils;
import org.apache.hadoop.util.StringInterner;
@@ -1786,6 +1788,26 @@ public class Configuration implements
Iterable>,
: Enum.valueOf(defaultValue.getDeclaringClass(), val);
}
+ /**
+ * Build an enumset from a comma separated list of values.
+ * Case independent.
+ * Special handling of "*" meaning: all values.
+ * @param key key to look for
+ * @param enumClass class of enum
+ * @param ign
(hadoop) branch trunk updated: HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags (#6789)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new a5806a9e7bc HADOOP-19161. S3A: option "fs.s3a.performance.flags" to
take list of performance flags (#6789)
a5806a9e7bc is described below
commit a5806a9e7bc6d018de84e6511f10c359f110f78c
Author: Steve Loughran
AuthorDate: Mon Jul 29 11:33:51 2024 +0100
HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of
performance flags (#6789)
1. Configuration adds new method `getEnumSet()` to get a set of enum values
from
a configuration string.
> EnumSet getEnumSet(String key, Class
enumClass, boolean ignoreUnknown)
Whitespace is ignored, case is ignored and the value "*" is mapped to
"all values of the enum".
If "ignoreUnknown" is true then when parsing, unknown values are ignored.
This is recommended for forward compatiblity with later versions.
2. This support is implemented in
org.apache.hadoop.fs.s3a.impl.ConfigurationHelper -it can be used
elsewhere in the hadoop codebase.
3. A new private FlagSet class in hadoop common manages a set of enum flags.
It implements StreamCapabilities and can be probed for a specific
option being set
(with a prefix)
S3A adds an option fs.s3a.performance.flags which builds a FlagSet with enum
type PerformanceFlagEnum
* which initially contains {Create, Delete, Mkdir, Open}
* the existing fs.s3a.create.performance option sets the flag "Create".
* tests which configure fs.s3a.create.performance MUST clear
fs.s3a.performance.flags in test setup.
Future performance flags are planned, with different levels of safety
and/or backwards compatibility.
Contributed by Steve Loughran
---
.../java/org/apache/hadoop/conf/Configuration.java | 22 ++
.../java/org/apache/hadoop/fs/impl/FlagSet.java| 327
.../apache/hadoop/util/ConfigurationHelper.java| 126 ++
.../org/apache/hadoop/fs/impl/TestFlagSet.java | 431 +
.../hadoop/util/TestConfigurationHelper.java | 174 +
.../java/org/apache/hadoop/fs/s3a/Constants.java | 5 +
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java| 82 +++-
.../hadoop/fs/s3a/api/PerformanceFlagEnum.java | 51 +++
.../apache/hadoop/fs/s3a/impl/StoreContext.java| 19 +-
.../hadoop/fs/s3a/impl/StoreContextBuilder.java| 17 +-
.../apache/hadoop/fs/s3a/s3guard/S3GuardTool.java | 9 +-
.../site/markdown/tools/hadoop-aws/performance.md | 110 --
.../fs/contract/s3a/ITestS3AContractCreate.java| 4 +-
.../hadoop/fs/s3a/ITestS3AFileOperationCost.java | 5 +-
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 5 +
.../fs/s3a/impl/ITestConnectionTimeouts.java | 4 +-
.../fs/s3a/performance/AbstractS3ACostTest.java| 3 +-
.../fs/s3a/performance/ITestCreateFileCost.java| 4 +-
.../performance/ITestDirectoryMarkerListing.java | 4 +-
.../fs/s3a/performance/ITestS3ADeleteCost.java | 5 +-
.../fs/s3a/tools/AbstractMarkerToolTest.java | 3 +-
21 files changed, 1350 insertions(+), 60 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 8fc3a696c4a..94285a4dfb7 100755
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -49,6 +49,7 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
+import java.util.EnumSet;
import java.util.Enumeration;
import java.util.HashMap;
import java.util.HashSet;
@@ -99,6 +100,7 @@ import org.apache.hadoop.security.alias.CredentialProvider;
import org.apache.hadoop.security.alias.CredentialProvider.CredentialEntry;
import org.apache.hadoop.security.alias.CredentialProviderFactory;
import org.apache.hadoop.thirdparty.com.google.common.base.Strings;
+import org.apache.hadoop.util.ConfigurationHelper;
import org.apache.hadoop.util.Preconditions;
import org.apache.hadoop.util.ReflectionUtils;
import org.apache.hadoop.util.StringInterner;
@@ -1786,6 +1788,26 @@ public class Configuration implements
Iterable>,
: Enum.valueOf(defaultValue.getDeclaringClass(), val);
}
+ /**
+ * Build an enumset from a comma separated list of values.
+ * Case independent.
+ * Special handling of "*" meaning: all values.
+ * @param key key to look for
+ * @param enumClass class of enum
+ * @param ignoreUnknown should unkn
(hadoop) branch trunk updated (e2a0dca43b5 -> 4525c7e35ea)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from e2a0dca43b5 HDFS-16690. Automatically format unformatted JNs with JournalNodeSyncer (#6925). Contributed by Aswin M Prabhu. add 4525c7e35ea HADOOP-19197. S3A: Support AWS KMS Encryption Context (#6874) No new revisions were added by this update. Summary of changes: .../hadoop/fs/CommonConfigurationKeysPublic.java | 1 + .../src/main/resources/core-default.xml| 10 ++ .../java/org/apache/hadoop/fs/s3a/Constants.java | 10 ++ .../java/org/apache/hadoop/fs/s3a/S3AUtils.java| 22 - .../delegation/EncryptionSecretOperations.java | 16 .../fs/s3a/auth/delegation/EncryptionSecrets.java | 35 ++- .../hadoop/fs/s3a/impl/RequestFactoryImpl.java | 14 +++ .../apache/hadoop/fs/s3a/impl/S3AEncryption.java | 106 + .../site/markdown/tools/hadoop-aws/encryption.md | 30 ++ .../src/site/markdown/tools/hadoop-aws/index.md| 14 +++ .../hadoop/fs/s3a/AbstractTestS3AEncryption.java | 2 + ...stS3AEncryptionSSEKMSWithEncryptionContext.java | 101 .../apache/hadoop/fs/s3a/TestSSEConfiguration.java | 69 +++--- .../fs/s3a/auth/TestMarshalledCredentials.java | 3 +- .../delegation/ITestSessionDelegationTokens.java | 6 +- .../delegation/TestS3ADelegationTokenSupport.java | 24 - .../hadoop/fs/s3a/impl/TestRequestFactory.java | 2 +- .../hadoop/fs/s3a/impl/TestS3AEncryption.java | 77 +++ 18 files changed, 513 insertions(+), 29 deletions(-) create mode 100644 hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AEncryption.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEKMSWithEncryptionContext.java create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestS3AEncryption.java - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19195. S3A: Upgrade aws sdk v2 to 2.25.53 (#6900)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new fc86a52c884 HADOOP-19195. S3A: Upgrade aws sdk v2 to 2.25.53 (#6900) fc86a52c884 is described below commit fc86a52c884f15b2f2fb401bbf0baaa36a057651 Author: HarshitGupta11 <50410275+harshitgupt...@users.noreply.github.com> AuthorDate: Mon Jul 8 14:48:53 2024 +0530 HADOOP-19195. S3A: Upgrade aws sdk v2 to 2.25.53 (#6900) Contributed by Harshit Gupta --- LICENSE-binary | 2 +- hadoop-project/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 3ab3ef5d5e2..45616936bb6 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -361,7 +361,7 @@ org.objenesis:objenesis:2.6 org.xerial.snappy:snappy-java:1.1.10.4 org.yaml:snakeyaml:2.0 org.wildfly.openssl:wildfly-openssl:1.1.3.Final -software.amazon.awssdk:bundle:jar:2.24.6 +software.amazon.awssdk:bundle:jar:2.25.53 diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 4e42e3c895e..9b9176a029f 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -187,7 +187,7 @@ 1.0-beta-1 900 1.12.720 -2.24.6 +2.25.53 1.0.1 2.7.1 1.11.2 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (8ca4627a0da -> b1d96f6101c)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 8ca4627a0da HDFS-17557. Fix bug for TestRedundancyMonitor#testChooseTargetWhenAllDataNodesStop (#6897). Contributed by Haiyang Hu. add b1d96f6101c HADOOP-19195. S3A: Upgrade aws sdk v2 to 2.25.53 (#6900) No new revisions were added by this update. Summary of changes: LICENSE-binary | 2 +- hadoop-project/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19205. S3A: initialization/close slower than with v1 SDK (#6892)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new b7630e2b36b HADOOP-19205. S3A: initialization/close slower than with v1 SDK (#6892) b7630e2b36b is described below commit b7630e2b36b95cc4ea118f0961ad61e95018a05c Author: Steve Loughran AuthorDate: Fri Jul 5 16:38:37 2024 +0100 HADOOP-19205. S3A: initialization/close slower than with v1 SDK (#6892) Adds new ClientManager interface/implementation which provides on-demand creation of synchronous and asynchronous s3 clients, s3 transfer manager, and in close() terminates these. S3A FS is modified to * Create a ClientManagerImpl instance and pass down to its S3Store. * Use the same ClientManager interface against S3Store to demand-create the services. * Only create the async client as part of the transfer manager creation, which will take place during the first rename() operation. * Statistics on client creation count and duration are recorded. + Statistics on the time to initialize and shutdown the S3A FS are collected in IOStatistics for reporting. Adds to hadoop common class LazyAtomicReference implements CallableRaisingIOE, Supplier and subclass LazyAutoCloseableReference extends LazyAtomicReference implements AutoCloseable These evaluate the Supplier/CallableRaisingIOE they were constructed with on the first (successful) read of the the value. Any exception raised during this operation will be rethrown, and on future evaluations the same operation retried. These classes implement the Supplier and CallableRaisingIOE interfaces so can actually be used for to implement lazy function evaluation as Haskell and some other functional languages do. LazyAutoCloseableReference is AutoCloseable; its close() method will close the inner reference if it is set This class is used in ClientManagerImpl for the lazy S3 Cliehnt creation and closure. Contributed by Steve Loughran. --- .../fs/statistics/FileSystemStatisticNames.java| 45 +++ .../hadoop/fs/statistics/StoreStatisticNames.java | 6 + .../hadoop/util/functional/FunctionalIO.java | 23 +- .../apache/hadoop/util/functional/FutureIO.java| 50 +-- .../util/functional/LazyAtomicReference.java | 152 + .../functional/LazyAutoCloseableReference.java | 102 ++ .../hadoop/util/functional/TestLazyReferences.java | 263 ++ .../hadoop-aws/dev-support/findbugs-exclude.xml| 5 - .../org/apache/hadoop/fs/s3a/S3AFileSystem.java| 259 -- .../java/org/apache/hadoop/fs/s3a/S3AStore.java| 9 +- .../org/apache/hadoop/fs/s3a/S3ClientFactory.java | 2 - .../java/org/apache/hadoop/fs/s3a/Statistic.java | 16 + .../apache/hadoop/fs/s3a/impl/ClientManager.java | 50 +++ .../hadoop/fs/s3a/impl/ClientManagerImpl.java | 238 + .../apache/hadoop/fs/s3a/impl/S3AStoreBuilder.java | 21 +- .../apache/hadoop/fs/s3a/impl/S3AStoreImpl.java| 121 --- .../apache/hadoop/fs/s3a/MockS3AFileSystem.java| 7 + .../fs/s3a/commit/staging/StagingTestBase.java | 11 +- .../hadoop/fs/s3a/impl/TestClientManager.java | 379 + .../hadoop/fs/s3a/test/StubS3ClientFactory.java| 122 +++ 20 files changed, 1664 insertions(+), 217 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/FileSystemStatisticNames.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/FileSystemStatisticNames.java new file mode 100644 index 000..cd8df2f8536 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/FileSystemStatisticNames.java @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.statistics; + +import org.apache.hadoop.classification.Interf
(hadoop) branch trunk updated: HADOOP-19205. S3A: initialization/close slower than with v1 SDK (#6892)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 4c55adbb6bc HADOOP-19205. S3A: initialization/close slower than with v1 SDK (#6892) 4c55adbb6bc is described below commit 4c55adbb6bc25fe76943535fd97cbd2b6d350e33 Author: Steve Loughran AuthorDate: Fri Jul 5 16:38:37 2024 +0100 HADOOP-19205. S3A: initialization/close slower than with v1 SDK (#6892) Adds new ClientManager interface/implementation which provides on-demand creation of synchronous and asynchronous s3 clients, s3 transfer manager, and in close() terminates these. S3A FS is modified to * Create a ClientManagerImpl instance and pass down to its S3Store. * Use the same ClientManager interface against S3Store to demand-create the services. * Only create the async client as part of the transfer manager creation, which will take place during the first rename() operation. * Statistics on client creation count and duration are recorded. + Statistics on the time to initialize and shutdown the S3A FS are collected in IOStatistics for reporting. Adds to hadoop common class LazyAtomicReference implements CallableRaisingIOE, Supplier and subclass LazyAutoCloseableReference extends LazyAtomicReference implements AutoCloseable These evaluate the Supplier/CallableRaisingIOE they were constructed with on the first (successful) read of the the value. Any exception raised during this operation will be rethrown, and on future evaluations the same operation retried. These classes implement the Supplier and CallableRaisingIOE interfaces so can actually be used for to implement lazy function evaluation as Haskell and some other functional languages do. LazyAutoCloseableReference is AutoCloseable; its close() method will close the inner reference if it is set This class is used in ClientManagerImpl for the lazy S3 Cliehnt creation and closure. Contributed by Steve Loughran. --- .../fs/statistics/FileSystemStatisticNames.java| 45 +++ .../hadoop/fs/statistics/StoreStatisticNames.java | 6 + .../hadoop/util/functional/FunctionalIO.java | 23 +- .../apache/hadoop/util/functional/FutureIO.java| 50 +-- .../util/functional/LazyAtomicReference.java | 152 + .../functional/LazyAutoCloseableReference.java | 102 ++ .../hadoop/util/functional/TestLazyReferences.java | 263 ++ .../hadoop-aws/dev-support/findbugs-exclude.xml| 5 - .../org/apache/hadoop/fs/s3a/S3AFileSystem.java| 259 -- .../java/org/apache/hadoop/fs/s3a/S3AStore.java| 9 +- .../org/apache/hadoop/fs/s3a/S3ClientFactory.java | 2 - .../java/org/apache/hadoop/fs/s3a/Statistic.java | 16 + .../apache/hadoop/fs/s3a/impl/ClientManager.java | 50 +++ .../hadoop/fs/s3a/impl/ClientManagerImpl.java | 238 + .../apache/hadoop/fs/s3a/impl/S3AStoreBuilder.java | 21 +- .../apache/hadoop/fs/s3a/impl/S3AStoreImpl.java| 121 --- .../apache/hadoop/fs/s3a/MockS3AFileSystem.java| 7 + .../fs/s3a/commit/staging/StagingTestBase.java | 11 +- .../hadoop/fs/s3a/impl/TestClientManager.java | 379 + .../hadoop/fs/s3a/test/StubS3ClientFactory.java| 122 +++ 20 files changed, 1664 insertions(+), 217 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/FileSystemStatisticNames.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/FileSystemStatisticNames.java new file mode 100644 index 000..cd8df2f8536 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/FileSystemStatisticNames.java @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.statistics; + +import org.apache.hadoop.classification.Interf
(hadoop) branch branch-3.4 updated: HADOOP-19210. S3A: Speed up some slow unit tests (#6907)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new b5e21f94950 HADOOP-19210. S3A: Speed up some slow unit tests (#6907)
b5e21f94950 is described below
commit b5e21f949508f6837502c6613d8a914b59cf6cde
Author: Steve Loughran
AuthorDate: Tue Jul 2 11:34:45 2024 +0100
HADOOP-19210. S3A: Speed up some slow unit tests (#6907)
Speed up slow tests
* TestS3AAWSCredentialsProvider: decrease thread pool shutdown time
* TestS3AInputStreamRetry: reduce retry limit and intervals
Contributed by Steve Loughran
---
.../test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java | 9 +
.../org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java | 8 +---
2 files changed, 14 insertions(+), 3 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java
index f43710cf25e..e76b3046048 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java
@@ -80,6 +80,15 @@ public abstract class AbstractS3AMockTest {
conf.setInt(ASYNC_DRAIN_THRESHOLD, Integer.MAX_VALUE);
// set the region to avoid the getBucketLocation on FS init.
conf.set(AWS_REGION, "eu-west-1");
+
+// tight retry logic as all failures are simulated
+final String interval = "1ms";
+final int limit = 3;
+conf.set(RETRY_THROTTLE_INTERVAL, interval);
+conf.setInt(RETRY_THROTTLE_LIMIT, limit);
+conf.set(RETRY_INTERVAL, interval);
+conf.setInt(RETRY_LIMIT, limit);
+
return conf;
}
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
index 0ffd7e75b18..d51bc954a63 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
@@ -86,6 +86,8 @@ public class TestS3AAWSCredentialsProvider extends
AbstractS3ATestBase {
private static final Logger LOG =
LoggerFactory.getLogger(TestS3AAWSCredentialsProvider.class);
+ public static final int TERMINATION_TIMEOUT = 3;
+
@Test
public void testProviderWrongClass() throws Exception {
expectProviderInstantiationFailure(this.getClass(),
@@ -579,7 +581,7 @@ public class TestS3AAWSCredentialsProvider extends
AbstractS3ATestBase {
}
}
- private static final int CONCURRENT_THREADS = 10;
+ private static final int CONCURRENT_THREADS = 4;
@Test
public void testConcurrentAuthentication() throws Throwable {
@@ -619,7 +621,7 @@ public class TestS3AAWSCredentialsProvider extends
AbstractS3ATestBase {
"expectedSecret", credentials.secretAccessKey());
}
} finally {
- pool.awaitTermination(10, TimeUnit.SECONDS);
+ pool.awaitTermination(TERMINATION_TIMEOUT, TimeUnit.SECONDS);
pool.shutdown();
}
@@ -685,7 +687,7 @@ public class TestS3AAWSCredentialsProvider extends
AbstractS3ATestBase {
);
}
} finally {
- pool.awaitTermination(10, TimeUnit.SECONDS);
+ pool.awaitTermination(TERMINATION_TIMEOUT, TimeUnit.SECONDS);
pool.shutdown();
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19210. S3A: Speed up some slow unit tests (#6907)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new c33d8686060 HADOOP-19210. S3A: Speed up some slow unit tests (#6907)
c33d8686060 is described below
commit c33d86860606f972f8b743b02f629b14f83d14f2
Author: Steve Loughran
AuthorDate: Tue Jul 2 11:34:45 2024 +0100
HADOOP-19210. S3A: Speed up some slow unit tests (#6907)
Speed up slow tests
* TestS3AAWSCredentialsProvider: decrease thread pool shutdown time
* TestS3AInputStreamRetry: reduce retry limit and intervals
Contributed by Steve Loughran
---
.../test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java | 9 +
.../org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java | 8 +---
2 files changed, 14 insertions(+), 3 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java
index f43710cf25e..e76b3046048 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3AMockTest.java
@@ -80,6 +80,15 @@ public abstract class AbstractS3AMockTest {
conf.setInt(ASYNC_DRAIN_THRESHOLD, Integer.MAX_VALUE);
// set the region to avoid the getBucketLocation on FS init.
conf.set(AWS_REGION, "eu-west-1");
+
+// tight retry logic as all failures are simulated
+final String interval = "1ms";
+final int limit = 3;
+conf.set(RETRY_THROTTLE_INTERVAL, interval);
+conf.setInt(RETRY_THROTTLE_LIMIT, limit);
+conf.set(RETRY_INTERVAL, interval);
+conf.setInt(RETRY_LIMIT, limit);
+
return conf;
}
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
index 0ffd7e75b18..d51bc954a63 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
@@ -86,6 +86,8 @@ public class TestS3AAWSCredentialsProvider extends
AbstractS3ATestBase {
private static final Logger LOG =
LoggerFactory.getLogger(TestS3AAWSCredentialsProvider.class);
+ public static final int TERMINATION_TIMEOUT = 3;
+
@Test
public void testProviderWrongClass() throws Exception {
expectProviderInstantiationFailure(this.getClass(),
@@ -579,7 +581,7 @@ public class TestS3AAWSCredentialsProvider extends
AbstractS3ATestBase {
}
}
- private static final int CONCURRENT_THREADS = 10;
+ private static final int CONCURRENT_THREADS = 4;
@Test
public void testConcurrentAuthentication() throws Throwable {
@@ -619,7 +621,7 @@ public class TestS3AAWSCredentialsProvider extends
AbstractS3ATestBase {
"expectedSecret", credentials.secretAccessKey());
}
} finally {
- pool.awaitTermination(10, TimeUnit.SECONDS);
+ pool.awaitTermination(TERMINATION_TIMEOUT, TimeUnit.SECONDS);
pool.shutdown();
}
@@ -685,7 +687,7 @@ public class TestS3AAWSCredentialsProvider extends
AbstractS3ATestBase {
);
}
} finally {
- pool.awaitTermination(10, TimeUnit.SECONDS);
+ pool.awaitTermination(TERMINATION_TIMEOUT, TimeUnit.SECONDS);
pool.shutdown();
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19194:Add test to find unshaded dependencies in the aws sdk (#6865)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 2b46ca4ba59 HADOOP-19194:Add test to find unshaded dependencies in the
aws sdk (#6865)
2b46ca4ba59 is described below
commit 2b46ca4ba59e4a8d62a43ca23b104c30b615141f
Author: HarshitGupta11 <50410275+harshitgupt...@users.noreply.github.com>
AuthorDate: Mon Jun 24 15:11:11 2024 +0530
HADOOP-19194:Add test to find unshaded dependencies in the aws sdk (#6865)
The new test TestAWSV2SDK scans the aws sdk bundle.jar and prints out all
classes
which are unshaded, so at risk of creating classpath problems
It does not fail the test if this holds, because the current SDKs
do ship with unshaded classes; the test would always fail.
The SDK upgrade process should include inspecting the output
of this test to see if it has got worse (do a before/after check).
Once the AWS SDK does shade everything, we can have this
test fail on any regression
Contributed by Harshit Gupta
---
.../src/site/markdown/tools/hadoop-aws/testing.md | 1 +
.../org/apache/hadoop/fs/sdk/TestAWSV2SDK.java | 94 ++
2 files changed, 95 insertions(+)
diff --git
a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
index 45d1c847657..7222eee98ba 100644
--- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
+++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
@@ -1184,6 +1184,7 @@ your IDE or via maven.
1. Run a full AWS-test suite with S3 client-side encryption enabled by
setting `fs.s3a.encryption.algorithm` to 'CSE-KMS' and setting up AWS-KMS
Key ID in `fs.s3a.encryption.key`.
+2. Verify that the output of test `TestAWSV2SDK` doesn't contain any unshaded
classes.
The dependency chain of the `hadoop-aws` module should be similar to this,
albeit
with different version numbers:
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/sdk/TestAWSV2SDK.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/sdk/TestAWSV2SDK.java
new file mode 100644
index 000..fca9fcc300c
--- /dev/null
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/sdk/TestAWSV2SDK.java
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.sdk;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.jar.JarEntry;
+import java.util.jar.JarFile;
+
+import org.junit.Test;
+import org.apache.hadoop.test.AbstractHadoopTestBase;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Tests to verify AWS SDK based issues like duplicated shaded classes and
others.
+ */
+public class TestAWSV2SDK extends AbstractHadoopTestBase {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(TestAWSV2SDK.class.getName());
+
+ @Test
+ public void testShadedClasses() throws IOException {
+String allClassPath = System.getProperty("java.class.path");
+LOG.debug("Current classpath:{}", allClassPath);
+String[] classPaths = allClassPath.split(File.pathSeparator);
+String v2ClassPath = null;
+for (String classPath : classPaths) {
+ //Checking for only version 2.x sdk here
+ if (classPath.contains("awssdk/bundle/2")) {
+v2ClassPath = classPath;
+break;
+ }
+}
+LOG.debug("AWS SDK V2 Classpath:{}", v2ClassPath);
+assertThat(v2ClassPath)
+.as("AWS V2 SDK should be present on the classpath").isNotNull();
+List listOfV2SdkClasses = getClassNamesFromJarFile(v2ClassPath);
+String awsSdkPrefix = "software/amazon/awssdk";
+List unshadedClasses = new ArrayList<&
(hadoop) branch trunk updated: HADOOP-19194:Add test to find unshaded dependencies in the aws sdk (#6865)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new d3b98cb1b23 HADOOP-19194:Add test to find unshaded dependencies in the
aws sdk (#6865)
d3b98cb1b23 is described below
commit d3b98cb1b23841a57b966c5cedab312687f098cb
Author: HarshitGupta11 <50410275+harshitgupt...@users.noreply.github.com>
AuthorDate: Mon Jun 24 15:11:11 2024 +0530
HADOOP-19194:Add test to find unshaded dependencies in the aws sdk (#6865)
The new test TestAWSV2SDK scans the aws sdk bundle.jar and prints out all
classes
which are unshaded, so at risk of creating classpath problems
It does not fail the test if this holds, because the current SDKs
do ship with unshaded classes; the test would always fail.
The SDK upgrade process should include inspecting the output
of this test to see if it has got worse (do a before/after check).
Once the AWS SDK does shade everything, we can have this
test fail on any regression
Contributed by Harshit Gupta
---
.../src/site/markdown/tools/hadoop-aws/testing.md | 1 +
.../org/apache/hadoop/fs/sdk/TestAWSV2SDK.java | 94 ++
2 files changed, 95 insertions(+)
diff --git
a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
index 45d1c847657..7222eee98ba 100644
--- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
+++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
@@ -1184,6 +1184,7 @@ your IDE or via maven.
1. Run a full AWS-test suite with S3 client-side encryption enabled by
setting `fs.s3a.encryption.algorithm` to 'CSE-KMS' and setting up AWS-KMS
Key ID in `fs.s3a.encryption.key`.
+2. Verify that the output of test `TestAWSV2SDK` doesn't contain any unshaded
classes.
The dependency chain of the `hadoop-aws` module should be similar to this,
albeit
with different version numbers:
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/sdk/TestAWSV2SDK.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/sdk/TestAWSV2SDK.java
new file mode 100644
index 000..fca9fcc300c
--- /dev/null
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/sdk/TestAWSV2SDK.java
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.sdk;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.jar.JarEntry;
+import java.util.jar.JarFile;
+
+import org.junit.Test;
+import org.apache.hadoop.test.AbstractHadoopTestBase;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Tests to verify AWS SDK based issues like duplicated shaded classes and
others.
+ */
+public class TestAWSV2SDK extends AbstractHadoopTestBase {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(TestAWSV2SDK.class.getName());
+
+ @Test
+ public void testShadedClasses() throws IOException {
+String allClassPath = System.getProperty("java.class.path");
+LOG.debug("Current classpath:{}", allClassPath);
+String[] classPaths = allClassPath.split(File.pathSeparator);
+String v2ClassPath = null;
+for (String classPath : classPaths) {
+ //Checking for only version 2.x sdk here
+ if (classPath.contains("awssdk/bundle/2")) {
+v2ClassPath = classPath;
+break;
+ }
+}
+LOG.debug("AWS SDK V2 Classpath:{}", v2ClassPath);
+assertThat(v2ClassPath)
+.as("AWS V2 SDK should be present on the classpath").isNotNull();
+List listOfV2SdkClasses = getClassNamesFromJarFile(v2ClassPath);
+String awsSdkPrefix = "software/amazon/awssdk";
+List unshadedClasses = new ArrayList<>();
+for (String awsSdkCla
(hadoop) branch branch-3.4 updated: HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as UncheckedIOExceptions (#6885)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new b8a390122fd HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as
UncheckedIOExceptions (#6885)
b8a390122fd is described below
commit b8a390122fd7a1c2d5baebeb60d7509bc22a1790
Author: Steve Loughran
AuthorDate: Wed Jun 19 18:47:29 2024 +0100
HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as
UncheckedIOExceptions (#6885)
* WrappedIO methods raise UncheckedIOExceptions
*New class org.apache.hadoop.util.functional.FunctionalIO
with wrap/unwrap and the ability to generate a
java.util.function.Supplier around a CallableRaisingIOE.
Contributed by Steve Loughran
---
.../org/apache/hadoop/io/wrappedio/WrappedIO.java | 37
.../util/functional/CommonCallableSupplier.java| 5 +-
.../hadoop/util/functional/FunctionalIO.java | 99 ++
.../hadoop/util/functional/TestFunctionalIO.java | 97 +
4 files changed, 221 insertions(+), 17 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/wrappedio/WrappedIO.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/wrappedio/WrappedIO.java
index 286557c2c37..d6fe311fba8 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/wrappedio/WrappedIO.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/wrappedio/WrappedIO.java
@@ -18,7 +18,7 @@
package org.apache.hadoop.io.wrappedio;
-import java.io.IOException;
+import java.io.UncheckedIOException;
import java.util.Collection;
import java.util.List;
import java.util.Map;
@@ -29,17 +29,19 @@ import org.apache.hadoop.fs.BulkDelete;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import static
org.apache.hadoop.util.functional.FunctionalIO.uncheckIOExceptions;
+
/**
* Reflection-friendly access to APIs which are not available in
* some of the older Hadoop versions which libraries still
* compile against.
*
* The intent is to avoid the need for complex reflection operations
- * including wrapping of parameter classes, direct instatiation of
+ * including wrapping of parameter classes, direct instantiation of
* new classes etc.
*/
@InterfaceAudience.Public
-@InterfaceStability.Evolving
+@InterfaceStability.Unstable
public final class WrappedIO {
private WrappedIO() {
@@ -52,12 +54,15 @@ public final class WrappedIO {
* @return a number greater than or equal to zero.
* @throws UnsupportedOperationException bulk delete under that path is not
supported.
* @throws IllegalArgumentException path not valid.
- * @throws IOException problems resolving paths
+ * @throws UncheckedIOException if an IOE was raised.
*/
- public static int bulkDelete_pageSize(FileSystem fs, Path path) throws
IOException {
-try (BulkDelete bulk = fs.createBulkDelete(path)) {
- return bulk.pageSize();
-}
+ public static int bulkDelete_pageSize(FileSystem fs, Path path) {
+
+return uncheckIOExceptions(() -> {
+ try (BulkDelete bulk = fs.createBulkDelete(path)) {
+return bulk.pageSize();
+ }
+});
}
/**
@@ -79,15 +84,17 @@ public final class WrappedIO {
* @param paths list of paths which must be absolute and under the base path.
* @return a list of all the paths which couldn't be deleted for a reason
other than "not found" and any associated error message.
* @throws UnsupportedOperationException bulk delete under that path is not
supported.
- * @throws IOException IO problems including networking, authentication and
more.
+ * @throws UncheckedIOException if an IOE was raised.
* @throws IllegalArgumentException if a path argument is invalid.
*/
public static List> bulkDelete_delete(FileSystem fs,
-Path base,
-
Collection paths)
-throws IOException {
-try (BulkDelete bulk = fs.createBulkDelete(base)) {
- return bulk.bulkDelete(paths);
-}
+ Path base,
+ Collection paths) {
+
+return uncheckIOExceptions(() -> {
+ try (BulkDelete bulk = fs.createBulkDelete(base)) {
+return bulk.bulkDelete(paths);
+ }
+});
}
}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/CommonCallableSupplier.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/CommonCallableSupplier.java
index 67299ef96ae..7a3193efbf0 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/CommonCallableSupplier.java
+++
b/hadoop-comm
(hadoop) branch branch-3.4 updated: HADOOP-19204. VectorIO regression: empty ranges are now rejected (#6887)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 30aaad808ac HADOOP-19204. VectorIO regression: empty ranges are now
rejected (#6887)
30aaad808ac is described below
commit 30aaad808acad193733293a6d39d1cb47e9a8e0b
Author: Steve Loughran
AuthorDate: Wed Jun 19 12:05:24 2024 +0100
HADOOP-19204. VectorIO regression: empty ranges are now rejected (#6887)
- restore old outcome: no-op
- test this
- update spec
This is a critical fix for vector IO and MUST be cherrypicked to all
branches with
that feature
Contributed by Steve Loughran
---
.../src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java | 9 -
.../src/site/markdown/filesystem/fsdatainputstream.md | 1 -
.../hadoop/fs/contract/AbstractContractVectoredReadTest.java | 11 +++
.../java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java | 7 +++
4 files changed, 22 insertions(+), 6 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
index 493b8c3a33d..fa0440620a4 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
@@ -294,7 +294,14 @@ public final class VectoredReadUtils {
final Optional fileLength) throws EOFException {
requireNonNull(input, "Null input list");
-checkArgument(!input.isEmpty(), "Empty input list");
+
+if (input.isEmpty()) {
+ // this may seem a pathological case, but it was valid
+ // before and somehow Spark can call it through parquet.
+ LOG.debug("Empty input list");
+ return input;
+}
+
final List sortedRanges;
if (input.size() == 1) {
diff --git
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
index 6cbb54ea701..db844a94e39 100644
---
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
+++
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
@@ -474,7 +474,6 @@ No empty lists.
```python
if ranges = null raise NullPointerException
-if ranges.len() = 0 raise IllegalArgumentException
if allocate = null raise NullPointerException
```
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractVectoredReadTest.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractVectoredReadTest.java
index d6a1fb1f0b7..aa478f3af63 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractVectoredReadTest.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractVectoredReadTest.java
@@ -340,6 +340,17 @@ public abstract class AbstractContractVectoredReadTest
extends AbstractFSContrac
}
}
+ @Test
+ public void testEmptyRanges() throws Exception {
+List fileRanges = new ArrayList<>();
+try (FSDataInputStream in = openVectorFile()) {
+ in.readVectored(fileRanges, allocate);
+ Assertions.assertThat(fileRanges)
+ .describedAs("Empty ranges must stay empty")
+ .isEmpty();
+}
+ }
+
/**
* Test to validate EOF ranges.
*
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
index 2a290058cae..3fd3fe4d1f4 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
@@ -702,12 +702,11 @@ public class TestVectoredReadUtils extends HadoopTestBase
{
}
/**
- * Empty ranges cannot be sorted.
+ * Empty ranges are allowed.
*/
@Test
- public void testEmptyRangesRaisesIllegalArgument() throws Throwable {
-intercept(IllegalArgumentException.class,
-() -> validateAndSortRanges(Collections.emptyList(),
Optional.empty()));
+ public void testEmptyRangesAllowed() throws Throwable {
+validateAndSortRanges(Collections.emptyList(), Optional.empty());
}
/**
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as UncheckedIOExceptions (#6885)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 8ac9c1839ac HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as
UncheckedIOExceptions (#6885)
8ac9c1839ac is described below
commit 8ac9c1839acba61d73fb2c9109e3c5b8cefb33c0
Author: Steve Loughran
AuthorDate: Wed Jun 19 18:47:29 2024 +0100
HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as
UncheckedIOExceptions (#6885)
* WrappedIO methods raise UncheckedIOExceptions
*New class org.apache.hadoop.util.functional.FunctionalIO
with wrap/unwrap and the ability to generate a
java.util.function.Supplier around a CallableRaisingIOE.
Contributed by Steve Loughran
---
.../org/apache/hadoop/io/wrappedio/WrappedIO.java | 37
.../util/functional/CommonCallableSupplier.java| 5 +-
.../hadoop/util/functional/FunctionalIO.java | 99 ++
.../hadoop/util/functional/TestFunctionalIO.java | 97 +
4 files changed, 221 insertions(+), 17 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/wrappedio/WrappedIO.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/wrappedio/WrappedIO.java
index 286557c2c37..d6fe311fba8 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/wrappedio/WrappedIO.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/wrappedio/WrappedIO.java
@@ -18,7 +18,7 @@
package org.apache.hadoop.io.wrappedio;
-import java.io.IOException;
+import java.io.UncheckedIOException;
import java.util.Collection;
import java.util.List;
import java.util.Map;
@@ -29,17 +29,19 @@ import org.apache.hadoop.fs.BulkDelete;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import static
org.apache.hadoop.util.functional.FunctionalIO.uncheckIOExceptions;
+
/**
* Reflection-friendly access to APIs which are not available in
* some of the older Hadoop versions which libraries still
* compile against.
*
* The intent is to avoid the need for complex reflection operations
- * including wrapping of parameter classes, direct instatiation of
+ * including wrapping of parameter classes, direct instantiation of
* new classes etc.
*/
@InterfaceAudience.Public
-@InterfaceStability.Evolving
+@InterfaceStability.Unstable
public final class WrappedIO {
private WrappedIO() {
@@ -52,12 +54,15 @@ public final class WrappedIO {
* @return a number greater than or equal to zero.
* @throws UnsupportedOperationException bulk delete under that path is not
supported.
* @throws IllegalArgumentException path not valid.
- * @throws IOException problems resolving paths
+ * @throws UncheckedIOException if an IOE was raised.
*/
- public static int bulkDelete_pageSize(FileSystem fs, Path path) throws
IOException {
-try (BulkDelete bulk = fs.createBulkDelete(path)) {
- return bulk.pageSize();
-}
+ public static int bulkDelete_pageSize(FileSystem fs, Path path) {
+
+return uncheckIOExceptions(() -> {
+ try (BulkDelete bulk = fs.createBulkDelete(path)) {
+return bulk.pageSize();
+ }
+});
}
/**
@@ -79,15 +84,17 @@ public final class WrappedIO {
* @param paths list of paths which must be absolute and under the base path.
* @return a list of all the paths which couldn't be deleted for a reason
other than "not found" and any associated error message.
* @throws UnsupportedOperationException bulk delete under that path is not
supported.
- * @throws IOException IO problems including networking, authentication and
more.
+ * @throws UncheckedIOException if an IOE was raised.
* @throws IllegalArgumentException if a path argument is invalid.
*/
public static List> bulkDelete_delete(FileSystem fs,
-Path base,
-
Collection paths)
-throws IOException {
-try (BulkDelete bulk = fs.createBulkDelete(base)) {
- return bulk.bulkDelete(paths);
-}
+ Path base,
+ Collection paths) {
+
+return uncheckIOExceptions(() -> {
+ try (BulkDelete bulk = fs.createBulkDelete(base)) {
+return bulk.bulkDelete(paths);
+ }
+});
}
}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/CommonCallableSupplier.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/CommonCallableSupplier.java
index 67299ef96ae..7a3193efbf0 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/CommonCallableSupplier.java
+++
b/hadoop-comm
(hadoop) branch trunk updated: HADOOP-19204. VectorIO regression: empty ranges are now rejected (#6887)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 56c8aa5f1c4 HADOOP-19204. VectorIO regression: empty ranges are now
rejected (#6887)
56c8aa5f1c4 is described below
commit 56c8aa5f1c4a0336f69083c742e2504ccc828d7d
Author: Steve Loughran
AuthorDate: Wed Jun 19 12:05:24 2024 +0100
HADOOP-19204. VectorIO regression: empty ranges are now rejected (#6887)
- restore old outcome: no-op
- test this
- update spec
This is a critical fix for vector IO and MUST be cherrypicked to all
branches with
that feature
Contributed by Steve Loughran
---
.../src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java | 9 -
.../src/site/markdown/filesystem/fsdatainputstream.md | 1 -
.../hadoop/fs/contract/AbstractContractVectoredReadTest.java | 11 +++
.../java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java | 7 +++
4 files changed, 22 insertions(+), 6 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
index 493b8c3a33d..fa0440620a4 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/VectoredReadUtils.java
@@ -294,7 +294,14 @@ public final class VectoredReadUtils {
final Optional fileLength) throws EOFException {
requireNonNull(input, "Null input list");
-checkArgument(!input.isEmpty(), "Empty input list");
+
+if (input.isEmpty()) {
+ // this may seem a pathological case, but it was valid
+ // before and somehow Spark can call it through parquet.
+ LOG.debug("Empty input list");
+ return input;
+}
+
final List sortedRanges;
if (input.size() == 1) {
diff --git
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
index 6cbb54ea701..db844a94e39 100644
---
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
+++
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
@@ -474,7 +474,6 @@ No empty lists.
```python
if ranges = null raise NullPointerException
-if ranges.len() = 0 raise IllegalArgumentException
if allocate = null raise NullPointerException
```
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractVectoredReadTest.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractVectoredReadTest.java
index d6a1fb1f0b7..aa478f3af63 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractVectoredReadTest.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractVectoredReadTest.java
@@ -340,6 +340,17 @@ public abstract class AbstractContractVectoredReadTest
extends AbstractFSContrac
}
}
+ @Test
+ public void testEmptyRanges() throws Exception {
+List fileRanges = new ArrayList<>();
+try (FSDataInputStream in = openVectorFile()) {
+ in.readVectored(fileRanges, allocate);
+ Assertions.assertThat(fileRanges)
+ .describedAs("Empty ranges must stay empty")
+ .isEmpty();
+}
+ }
+
/**
* Test to validate EOF ranges.
*
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
index 2a290058cae..3fd3fe4d1f4 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestVectoredReadUtils.java
@@ -702,12 +702,11 @@ public class TestVectoredReadUtils extends HadoopTestBase
{
}
/**
- * Empty ranges cannot be sorted.
+ * Empty ranges are allowed.
*/
@Test
- public void testEmptyRangesRaisesIllegalArgument() throws Throwable {
-intercept(IllegalArgumentException.class,
-() -> validateAndSortRanges(Collections.emptyList(),
Optional.empty()));
+ public void testEmptyRangesAllowed() throws Throwable {
+validateAndSortRanges(Collections.emptyList(), Optional.empty());
}
/**
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-18508. S3A: Support parallel integration test runs on same bucket (#5081)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 3173b72aeaa HADOOP-18508. S3A: Support parallel integration test runs
on same bucket (#5081)
3173b72aeaa is described below
commit 3173b72aeaa13c074f16667af7391f8c1d14bb4f
Author: Steve Loughran
AuthorDate: Fri Jun 14 19:34:52 2024 +0100
HADOOP-18508. S3A: Support parallel integration test runs on same bucket
(#5081)
It is now possible to provide a job ID in the maven "job.id" property
hadoop-aws test runs to isolate paths under a the test bucket
under which all tests will be executed.
This will allow independent builds *in different source trees*
to test against the same bucket in parallel, and is designed for
CI testing.
Example:
mvn verify -Dparallel-tests -Droot.tests.enabled=false -Djob.id=1
mvn verify -Droot.tests.enabled=false -Djob.id=2
- Root tests must be be disabled to stop them cleaning up
the test paths of other test runs.
- Do still regularly run the root tests just to force cleanup
of the output of any interrupted test suites.
Contributed by Steve Loughran
---
.../apache/hadoop/fs/FSMainOperationsBaseTest.java | 6 +-
.../fs/FileContextMainOperationsBaseTest.java | 8 +-
.../fs/TestFSMainOperationsLocalFileSystem.java| 26 +-
.../TestFSMainOperationsLocalFileSystem.java | 9 ---
hadoop-tools/hadoop-aws/pom.xml| 27 ---
.../src/site/markdown/tools/hadoop-aws/testing.md | 49 +++-
.../fs/contract/s3a/ITestS3AContractRootDir.java | 8 ++
.../hadoop/fs/s3a/ITestS3AConfiguration.java | 6 +-
.../hadoop/fs/s3a/ITestS3AEncryptionSSEC.java | 92 +-
.../org/apache/hadoop/fs/s3a/S3ATestConstants.java | 12 +++
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 12 ++-
.../fs/s3a/commit/terasort/ITestTerasortOnS3A.java | 22 --
.../ITestS3AFileContextMainOperations.java | 30 +--
.../fs/s3a/scale/AbstractSTestS3AHugeFiles.java| 10 +++
.../hadoop/fs/s3a/scale/ITestS3AConcurrentOps.java | 5 +-
.../hadoop/fs/s3a/scale/S3AScaleTestBase.java | 2 +-
.../s3a/tools/ITestMarkerToolRootOperations.java | 2 +
.../org/apache/hadoop/fs/s3a/yarn/ITestS3A.java| 41 +++---
18 files changed, 197 insertions(+), 170 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java
index f0c00c4cdee..07f0e816193 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java
@@ -102,7 +102,9 @@ public abstract class FSMainOperationsBaseTest extends
FileSystemTestHelper {
@After
public void tearDown() throws Exception {
-fSys.delete(new Path(getAbsoluteTestRootPath(fSys), new Path("test")),
true);
+if (fSys != null) {
+ fSys.delete(new Path(getAbsoluteTestRootPath(fSys), new Path("test")),
true);
+}
}
@@ -192,7 +194,7 @@ public abstract class FSMainOperationsBaseTest extends
FileSystemTestHelper {
@Test
public void testWDAbsolute() throws IOException {
-Path absoluteDir = new Path(fSys.getUri() + "/test/existingDir");
+Path absoluteDir = getTestRootPath(fSys, "test/existingDir");
fSys.mkdirs(absoluteDir);
fSys.setWorkingDirectory(absoluteDir);
Assert.assertEquals(absoluteDir, fSys.getWorkingDirectory());
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java
index 4c90490b090..6897a0d1943 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java
@@ -81,6 +81,12 @@ public abstract class FileContextMainOperationsBaseTest {
protected final FileContextTestHelper fileContextTestHelper =
createFileContextHelper();
+ /**
+ * Create the test helper.
+ * Important: this is invoked during the construction of the base class,
+ * so is very brittle.
+ * @return a test helper.
+ */
protected FileContextTestHelper createFileContextHelper() {
return new FileContextTestHelper();
}
@@ -107,7 +113,7 @@ public abstract class FileContextMainOperationsBaseTest {
private static final byte[] data = getFileData(numBlock
(hadoop) branch trunk updated: HADOOP-18508. S3A: Support parallel integration test runs on same bucket (#5081)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 2d5fa9e016d HADOOP-18508. S3A: Support parallel integration test runs
on same bucket (#5081)
2d5fa9e016d is described below
commit 2d5fa9e016d0c16dcd472c6bac18420c6d97118a
Author: Steve Loughran
AuthorDate: Fri Jun 14 19:34:52 2024 +0100
HADOOP-18508. S3A: Support parallel integration test runs on same bucket
(#5081)
It is now possible to provide a job ID in the maven "job.id" property
hadoop-aws test runs to isolate paths under a the test bucket
under which all tests will be executed.
This will allow independent builds *in different source trees*
to test against the same bucket in parallel, and is designed for
CI testing.
Example:
mvn verify -Dparallel-tests -Droot.tests.enabled=false -Djob.id=1
mvn verify -Droot.tests.enabled=false -Djob.id=2
- Root tests must be be disabled to stop them cleaning up
the test paths of other test runs.
- Do still regularly run the root tests just to force cleanup
of the output of any interrupted test suites.
Contributed by Steve Loughran
---
.../apache/hadoop/fs/FSMainOperationsBaseTest.java | 6 +-
.../fs/FileContextMainOperationsBaseTest.java | 8 +-
.../fs/TestFSMainOperationsLocalFileSystem.java| 26 +-
.../TestFSMainOperationsLocalFileSystem.java | 9 ---
hadoop-tools/hadoop-aws/pom.xml| 27 ---
.../src/site/markdown/tools/hadoop-aws/testing.md | 49 +++-
.../fs/contract/s3a/ITestS3AContractRootDir.java | 8 ++
.../hadoop/fs/s3a/ITestS3AConfiguration.java | 6 +-
.../hadoop/fs/s3a/ITestS3AEncryptionSSEC.java | 92 +-
.../org/apache/hadoop/fs/s3a/S3ATestConstants.java | 12 +++
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 12 ++-
.../fs/s3a/commit/terasort/ITestTerasortOnS3A.java | 22 --
.../ITestS3AFileContextMainOperations.java | 30 +--
.../fs/s3a/scale/AbstractSTestS3AHugeFiles.java| 10 +++
.../hadoop/fs/s3a/scale/ITestS3AConcurrentOps.java | 5 +-
.../hadoop/fs/s3a/scale/S3AScaleTestBase.java | 2 +-
.../s3a/tools/ITestMarkerToolRootOperations.java | 2 +
.../org/apache/hadoop/fs/s3a/yarn/ITestS3A.java| 41 +++---
18 files changed, 197 insertions(+), 170 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java
index f0c00c4cdee..07f0e816193 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java
@@ -102,7 +102,9 @@ public abstract class FSMainOperationsBaseTest extends
FileSystemTestHelper {
@After
public void tearDown() throws Exception {
-fSys.delete(new Path(getAbsoluteTestRootPath(fSys), new Path("test")),
true);
+if (fSys != null) {
+ fSys.delete(new Path(getAbsoluteTestRootPath(fSys), new Path("test")),
true);
+}
}
@@ -192,7 +194,7 @@ public abstract class FSMainOperationsBaseTest extends
FileSystemTestHelper {
@Test
public void testWDAbsolute() throws IOException {
-Path absoluteDir = new Path(fSys.getUri() + "/test/existingDir");
+Path absoluteDir = getTestRootPath(fSys, "test/existingDir");
fSys.mkdirs(absoluteDir);
fSys.setWorkingDirectory(absoluteDir);
Assert.assertEquals(absoluteDir, fSys.getWorkingDirectory());
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java
index 4c90490b090..6897a0d1943 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java
@@ -81,6 +81,12 @@ public abstract class FileContextMainOperationsBaseTest {
protected final FileContextTestHelper fileContextTestHelper =
createFileContextHelper();
+ /**
+ * Create the test helper.
+ * Important: this is invoked during the construction of the base class,
+ * so is very brittle.
+ * @return a test helper.
+ */
protected FileContextTestHelper createFileContextHelper() {
return new FileContextTestHelper();
}
@@ -107,7 +113,7 @@ public abstract class FileContextMainOperationsBaseTest {
private static final byte[] data = getFileData(numBlock
(hadoop) branch branch-3.4 updated: HADOOP-18931. FileSystem.getFileSystemClass() to log the jar the .class came from (#6197)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new acec539ab33 HADOOP-18931. FileSystem.getFileSystemClass() to log the
jar the .class came from (#6197)
acec539ab33 is described below
commit acec539ab3337129ad3e57ee562330720ba870b8
Author: Viraj Jasani
AuthorDate: Fri Jun 14 10:14:54 2024 -0800
HADOOP-18931. FileSystem.getFileSystemClass() to log the jar the .class
came from (#6197)
Set the log level of logger org.apache.hadoop.fs.FileSystem to DEBUG to see
this.
Contributed by Viraj Jasani
---
.../main/java/org/apache/hadoop/fs/FileSystem.java | 10 -
.../java/org/apache/hadoop/util/ClassUtil.java | 22 ---
.../java/org/apache/hadoop/util/TestClassUtil.java | 44 +-
3 files changed, 61 insertions(+), 15 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
index 2155e17328a..38ec6114517 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
@@ -3581,7 +3581,15 @@ public abstract class FileSystem extends Configured
throw new UnsupportedFileSystemException("No FileSystem for scheme "
+ "\"" + scheme + "\"");
}
-LOGGER.debug("FS for {} is {}", scheme, clazz);
+if (LOGGER.isDebugEnabled()) {
+ LOGGER.debug("FS for {} is {}", scheme, clazz);
+ final String jarLocation = ClassUtil.findContainingJar(clazz);
+ if (jarLocation != null) {
+LOGGER.debug("Jar location for {} : {}", clazz, jarLocation);
+ } else {
+LOGGER.debug("Class location for {} : {}", clazz,
ClassUtil.findClassLocation(clazz));
+ }
+}
return clazz;
}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ClassUtil.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ClassUtil.java
index 44c94669f51..c17445c57ce 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ClassUtil.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ClassUtil.java
@@ -36,13 +36,25 @@ public class ClassUtil {
* @return a jar file that contains the class, or null.
*/
public static String findContainingJar(Class clazz) {
-ClassLoader loader = clazz.getClassLoader();
-String classFile = clazz.getName().replaceAll("\\.", "/") + ".class";
+return findContainingResource(clazz.getClassLoader(), clazz.getName(),
"jar");
+ }
+
+ /**
+ * Find the absolute location of the class.
+ *
+ * @param clazz the class to find.
+ * @return the class file with absolute location, or null.
+ */
+ public static String findClassLocation(Class clazz) {
+return findContainingResource(clazz.getClassLoader(), clazz.getName(),
"file");
+ }
+
+ private static String findContainingResource(ClassLoader loader, String
clazz, String resource) {
+String classFile = clazz.replaceAll("\\.", "/") + ".class";
try {
- for(final Enumeration itr = loader.getResources(classFile);
- itr.hasMoreElements();) {
+ for (final Enumeration itr = loader.getResources(classFile);
itr.hasMoreElements();) {
final URL url = itr.nextElement();
-if ("jar".equals(url.getProtocol())) {
+if (resource.equals(url.getProtocol())) {
String toReturn = url.getPath();
if (toReturn.startsWith("file:")) {
toReturn = toReturn.substring("file:".length());
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestClassUtil.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestClassUtil.java
index 04337929abd..3a7e12e8f03 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestClassUtil.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestClassUtil.java
@@ -20,21 +20,47 @@ package org.apache.hadoop.util;
import java.io.File;
-import org.junit.Assert;
+import org.apache.hadoop.fs.viewfs.ViewFileSystem;
-import org.apache.log4j.Logger;
+import org.assertj.core.api.Assertions;
import org.junit.Test;
public class TestClassUtil {
+
@Test(timeout=1)
public void testFindContainingJar() {
-String containingJar = ClassUtil.findContainingJar(Logger.class);
-Assert.assertNotNull("Containing jar no
(hadoop) branch trunk updated (2bde5ccb813 -> 240fddcf17f)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 2bde5ccb813 HADOOP-19192. Log level is WARN when fail to load native hadoop libs (#6863) add 240fddcf17f HADOOP-18931. FileSystem.getFileSystemClass() to log the jar the .class came from (#6197) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/hadoop/fs/FileSystem.java | 10 - .../java/org/apache/hadoop/util/ClassUtil.java | 22 --- .../java/org/apache/hadoop/util/TestClassUtil.java | 44 +- 3 files changed, 61 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19192. Log level is WARN when fail to load native hadoop libs (#6863)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new 4eeb10318f0 HADOOP-19192. Log level is WARN when fail to load native hadoop libs (#6863) 4eeb10318f0 is described below commit 4eeb10318f0ee166446bb2bc9f311f3012a8e9d4 Author: Cheng Pan AuthorDate: Sat Jun 15 02:05:27 2024 +0800 HADOOP-19192. Log level is WARN when fail to load native hadoop libs (#6863) Updates the documentation to be consistent with the logging. Contributed by Cheng Pan --- .../hadoop-common/src/site/markdown/NativeLibraries.md.vm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/NativeLibraries.md.vm b/hadoop-common-project/hadoop-common/src/site/markdown/NativeLibraries.md.vm index 9756c42340d..a5d93a60e07 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/NativeLibraries.md.vm +++ b/hadoop-common-project/hadoop-common/src/site/markdown/NativeLibraries.md.vm @@ -104,7 +104,7 @@ The bin/hadoop script ensures that the native hadoop library is on the library p During runtime, check the hadoop log files for your MapReduce tasks. * If everything is all right, then: `DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...` `INFO util.NativeCodeLoader - Loaded the native-hadoop library` -* If something goes wrong, then: `INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable` +* If something goes wrong, then: `WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable` Check - - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19192. Log level is WARN when fail to load native hadoop libs (#6863)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 2bde5ccb813 HADOOP-19192. Log level is WARN when fail to load native hadoop libs (#6863) 2bde5ccb813 is described below commit 2bde5ccb8139aa798eedf554bf52007895777595 Author: Cheng Pan AuthorDate: Sat Jun 15 02:05:27 2024 +0800 HADOOP-19192. Log level is WARN when fail to load native hadoop libs (#6863) Updates the documentation to be consistent with the logging. Contributed by Cheng Pan --- .../hadoop-common/src/site/markdown/NativeLibraries.md.vm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/NativeLibraries.md.vm b/hadoop-common-project/hadoop-common/src/site/markdown/NativeLibraries.md.vm index 9756c42340d..a5d93a60e07 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/NativeLibraries.md.vm +++ b/hadoop-common-project/hadoop-common/src/site/markdown/NativeLibraries.md.vm @@ -104,7 +104,7 @@ The bin/hadoop script ensures that the native hadoop library is on the library p During runtime, check the hadoop log files for your MapReduce tasks. * If everything is all right, then: `DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...` `INFO util.NativeCodeLoader - Loaded the native-hadoop library` -* If something goes wrong, then: `INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable` +* If something goes wrong, then: `WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable` Check - - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19163. Use hadoop-shaded-protobuf_3_25 (#6858)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new bb30545583c HADOOP-19163. Use hadoop-shaded-protobuf_3_25 (#6858)
bb30545583c is described below
commit bb30545583c5c78199143d9cb9dd84cd3dfa8068
Author: PJ Fanning
AuthorDate: Tue Jun 11 17:10:00 2024 +0100
HADOOP-19163. Use hadoop-shaded-protobuf_3_25 (#6858)
Contributed by PJ Fanning
---
hadoop-common-project/hadoop-common/pom.xml | 2 +-
hadoop-project/pom.xml | 2 +-
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml | 6 +++---
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index a7dcbb24a9b..7521cec6a1d 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -40,7 +40,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_23
+ hadoop-shaded-protobuf_3_25
org.apache.hadoop
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 5b63dd129d6..be0f58aef63 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -250,7 +250,7 @@
org.apache.hadoop.thirdparty
-hadoop-shaded-protobuf_3_23
+hadoop-shaded-protobuf_3_25
${hadoop-thirdparty-protobuf.version}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
index 217335323a7..7cef2ec4db3 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
@@ -51,7 +51,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_23
+ hadoop-shaded-protobuf_3_25
@@ -64,7 +64,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_23
+ hadoop-shaded-protobuf_3_25
@@ -75,7 +75,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_23
+ hadoop-shaded-protobuf_3_25
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.3 updated: HADOOP-19116. Update to zookeeper client 3.8.4 due to CVE-2024-23944. (#6638)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new bd63358c0bb HADOOP-19116. Update to zookeeper client 3.8.4 due to
CVE-2024-23944. (#6638)
bd63358c0bb is described below
commit bd63358c0bb53bb1097f38ebf6c125547fe5e547
Author: PJ Fanning
AuthorDate: Tue Jun 11 13:09:23 2024 +0100
HADOOP-19116. Update to zookeeper client 3.8.4 due to CVE-2024-23944.
(#6638)
Updated ZK client dependency to 3.8.4 to address CVE-2024-23944.
Contributed by PJ Fanning
---
LICENSE-binary | 2 +-
hadoop-project/pom.xml | 18 +-
2 files changed, 18 insertions(+), 2 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index 890f1a75f38..30fe3f701c4 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -341,7 +341,7 @@ org.apache.kerby:kerby-util:1.0.1
org.apache.kerby:kerby-xdr:1.0.1
org.apache.kerby:token-provider:1.0.1
org.apache.yetus:audience-annotations:0.5.0
-org.apache.zookeeper:zookeeper:3.7.2
+org.apache.zookeeper:zookeeper:3.8.4
org.codehaus.jettison:jettison:1.5.4
org.eclipse.jetty:jetty-annotations:9.4.53.v20231009
org.eclipse.jetty:jetty-http:9.4.53.v20231009
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 13318c07cb8..1c08648eeb1 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -105,7 +105,7 @@
${hadoop-thirdparty-shaded-prefix}.protobuf
${hadoop-thirdparty-shaded-prefix}.com.google.common
-3.7.2
+3.8.4
5.2.0
3.0.5
2.1.7
@@ -1415,6 +1415,14 @@
log4j
log4j
+
+ch.qos.logback
+logback-core
+
+
+ch.qos.logback
+logback-classic
+
org.slf4j
slf4j-api
@@ -1463,6 +1471,14 @@
log4j
log4j
+
+ch.qos.logback
+logback-core
+
+
+ch.qos.logback
+logback-classic
+
org.slf4j
slf4j-log4j12
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19189. ITestS3ACommitterFactory failing (#6857)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 5826b15fe6a8 HADOOP-19189. ITestS3ACommitterFactory failing (#6857)
5826b15fe6a8 is described below
commit 5826b15fe6a8b2723b084e042b098c99cfca8edb
Author: Steve Loughran
AuthorDate: Fri Jun 7 17:34:01 2024 +0100
HADOOP-19189. ITestS3ACommitterFactory failing (#6857)
* parameterize the test run rather than do it from within the test suite.
* log what the committer factory is up to (and improve its logging)
* close all filesystems, then create the test filesystem with cache enabled.
The cache is critical, we want the fs from cache to be used when querying
filesystem properties, rather than one created from the committer jobconf,
which will have the same options as the task committer, so not actually
validate the override logic.
Contributed by Steve Loughran
---
.../fs/s3a/commit/AbstractS3ACommitterFactory.java | 5 +-
.../hadoop/fs/s3a/commit/S3ACommitterFactory.java | 7 +-
.../fs/s3a/commit/ITestS3ACommitterFactory.java| 234 +
.../hadoop-aws/src/test/resources/log4j.properties | 2 +
4 files changed, 151 insertions(+), 97 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitterFactory.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitterFactory.java
index 6e7a99f50ef9..cbbe5fdc602d 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitterFactory.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitterFactory.java
@@ -51,9 +51,10 @@ public abstract class AbstractS3ACommitterFactory
throw new PathCommitException(outputPath,
"Filesystem not supported by this committer");
}
-LOG.info("Using Committer {} for {}",
+LOG.info("Using Committer {} for {} created by {}",
outputCommitter,
-outputPath);
+outputPath,
+this);
return outputCommitter;
}
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/S3ACommitterFactory.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/S3ACommitterFactory.java
index 36d0af187d3c..7f5455b6098d 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/S3ACommitterFactory.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/S3ACommitterFactory.java
@@ -113,11 +113,14 @@ public class S3ACommitterFactory extends
AbstractS3ACommitterFactory {
// job/task configurations.
Configuration fsConf = fileSystem.getConf();
-String name = fsConf.getTrimmed(FS_S3A_COMMITTER_NAME,
COMMITTER_NAME_FILE);
+String name = fsConf.getTrimmed(FS_S3A_COMMITTER_NAME, "");
+LOG.debug("Committer from filesystems \"{}\"", name);
+
name = taskConf.getTrimmed(FS_S3A_COMMITTER_NAME, name);
-LOG.debug("Committer option is {}", name);
+LOG.debug("Committer option is \"{}\"", name);
switch (name) {
case COMMITTER_NAME_FILE:
+case "":
factory = null;
break;
case COMMITTER_NAME_DIRECTORY:
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestS3ACommitterFactory.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestS3ACommitterFactory.java
index 2ad2568d5cc2..2561a69f60b5 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestS3ACommitterFactory.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestS3ACommitterFactory.java
@@ -19,15 +19,24 @@
package org.apache.hadoop.fs.s3a.commit;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
import org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter;
import org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter;
import org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitter;
import org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter;
+import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapreduce.MRJobConfig;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.mapreduce.TaskAttemp
(hadoop) branch branch-3.3 updated: HADOOP-19178: [WASB Deprecation] Updating Documentation on Upcoming Plans for Hadoop-Azure (#6862)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.3 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.3 by this push: new 0c3c9d199702 HADOOP-19178: [WASB Deprecation] Updating Documentation on Upcoming Plans for Hadoop-Azure (#6862) 0c3c9d199702 is described below commit 0c3c9d199702488e9dc971107cb1e6796fe87a8c Author: Anuj Modi <128447756+anujmodi2...@users.noreply.github.com> AuthorDate: Fri Jun 7 18:58:24 2024 +0530 HADOOP-19178: [WASB Deprecation] Updating Documentation on Upcoming Plans for Hadoop-Azure (#6862) Contributed by Anuj Modi --- .../hadoop-azure/src/site/markdown/index.md| 1 + .../hadoop-azure/src/site/markdown/wasb.md | 97 ++ 2 files changed, 98 insertions(+) diff --git a/hadoop-tools/hadoop-azure/src/site/markdown/index.md b/hadoop-tools/hadoop-azure/src/site/markdown/index.md index 2af6b498a274..eba49967cbc1 100644 --- a/hadoop-tools/hadoop-azure/src/site/markdown/index.md +++ b/hadoop-tools/hadoop-azure/src/site/markdown/index.md @@ -18,6 +18,7 @@ See also: +* [WASB](./wasb.html) * [ABFS](./abfs.html) * [Testing](./testing_azure.html) diff --git a/hadoop-tools/hadoop-azure/src/site/markdown/wasb.md b/hadoop-tools/hadoop-azure/src/site/markdown/wasb.md new file mode 100644 index ..270fd14da4c4 --- /dev/null +++ b/hadoop-tools/hadoop-azure/src/site/markdown/wasb.md @@ -0,0 +1,97 @@ + + +# Hadoop Azure Support: WASB Driver + +## Introduction +WASB Driver is a legacy Hadoop File System driver that was developed to support +[FNS(FlatNameSpace) Azure Storage accounts](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) +that do not honor File-Folder syntax. +HDFS Folder operations hence are mimicked at client side by WASB driver and +certain folder operations like Rename and Delete can lead to a lot of IOPs with +client-side enumeration and orchestration of rename/delete operation blob by blob. +It was not ideal for other APIs too as initial checks for path is a file or folder +needs to be done over multiple metadata calls. These led to a degraded performance. + +To provide better service to Analytics users, Microsoft released [ADLS Gen2](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) +which are HNS (Hierarchical Namespace) enabled, i.e. File-Folder aware storage accounts. +ABFS driver was designed to overcome the inherent deficiencies of WASB and users +were informed to migrate to ABFS driver. + +### Challenges and limitations of WASB Driver +Users of the legacy WASB driver face a number of challenges and limitations: +1. They cannot leverage the optimizations and benefits of the latest ABFS driver. +2. They need to deal with the compatibility issues should the files and folders were +modified with the legacy WASB driver and the ABFS driver concurrently in a phased +transition situation. +3. There are differences for supported features for FNS and HNS over ABFS Driver. +4. In certain cases, they must perform a significant amount of re-work on their +workloads to migrate to the ABFS driver, which is available only on HNS enabled +accounts in a fully tested and supported scenario. + +## Deprecation plans for WASB Driver +We are introducing a new feature that will enable the ABFS driver to support +FNS accounts (over BlobEndpoint that WASB Driver uses) using the ABFS scheme. +This feature will enable us to use the ABFS driver to interact with data stored in GPv2 +(General Purpose v2) storage accounts. + +With this feature, the users who still use the legacy WASB driver will be able +to migrate to the ABFS driver without much re-work on their workloads. They will +however need to change the URIs from the WASB scheme to the ABFS scheme. + +Once ABFS driver has built FNS support capability to migrate WASB users, WASB +driver will be marked for removal in next major release. This will remove any ambiguity +for new users onboards as there will be only one Microsoft driver for Azure Storage +and migrating users will get SLA bound support for driver and service, +which was not guaranteed over WASB. + +We anticipate that this feature will serve as a stepping stone for users to +move to HNS enabled accounts with the ABFS driver, which is our recommended stack +for big data analytics on ADLS Gen2. + +### Impact for existing ABFS users using ADLS Gen2 (HNS enabled account) +This feature does not impact the existing users who are using ADLS Gen2 Accounts +(HNS enabled account) with ABFS driver. + +They do not need to make any changes to their workloads or configurations. They +will still enjoy the benefits of HNS, such as atomic operations, fine-grained +access control, scalability, and performance. + +### Official recommendation +Microsoft continues to recommend all Big Data and Analytics users
(hadoop) branch trunk updated: HADOOP-19189. ITestS3ACommitterFactory failing (#6857)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 01d257d5aa94 HADOOP-19189. ITestS3ACommitterFactory failing (#6857)
01d257d5aa94 is described below
commit 01d257d5aa94163244cd3f1149d5ba2cb9f1e6ff
Author: Steve Loughran
AuthorDate: Fri Jun 7 17:34:01 2024 +0100
HADOOP-19189. ITestS3ACommitterFactory failing (#6857)
* parameterize the test run rather than do it from within the test suite.
* log what the committer factory is up to (and improve its logging)
* close all filesystems, then create the test filesystem with cache enabled.
The cache is critical, we want the fs from cache to be used when querying
filesystem properties, rather than one created from the committer jobconf,
which will have the same options as the task committer, so not actually
validate the override logic.
Contributed by Steve Loughran
---
.../fs/s3a/commit/AbstractS3ACommitterFactory.java | 5 +-
.../hadoop/fs/s3a/commit/S3ACommitterFactory.java | 7 +-
.../fs/s3a/commit/ITestS3ACommitterFactory.java| 234 +
.../hadoop-aws/src/test/resources/log4j.properties | 2 +
4 files changed, 151 insertions(+), 97 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitterFactory.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitterFactory.java
index 6e7a99f50ef9..cbbe5fdc602d 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitterFactory.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/AbstractS3ACommitterFactory.java
@@ -51,9 +51,10 @@ public abstract class AbstractS3ACommitterFactory
throw new PathCommitException(outputPath,
"Filesystem not supported by this committer");
}
-LOG.info("Using Committer {} for {}",
+LOG.info("Using Committer {} for {} created by {}",
outputCommitter,
-outputPath);
+outputPath,
+this);
return outputCommitter;
}
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/S3ACommitterFactory.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/S3ACommitterFactory.java
index 36d0af187d3c..7f5455b6098d 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/S3ACommitterFactory.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/S3ACommitterFactory.java
@@ -113,11 +113,14 @@ public class S3ACommitterFactory extends
AbstractS3ACommitterFactory {
// job/task configurations.
Configuration fsConf = fileSystem.getConf();
-String name = fsConf.getTrimmed(FS_S3A_COMMITTER_NAME,
COMMITTER_NAME_FILE);
+String name = fsConf.getTrimmed(FS_S3A_COMMITTER_NAME, "");
+LOG.debug("Committer from filesystems \"{}\"", name);
+
name = taskConf.getTrimmed(FS_S3A_COMMITTER_NAME, name);
-LOG.debug("Committer option is {}", name);
+LOG.debug("Committer option is \"{}\"", name);
switch (name) {
case COMMITTER_NAME_FILE:
+case "":
factory = null;
break;
case COMMITTER_NAME_DIRECTORY:
diff --git
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestS3ACommitterFactory.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestS3ACommitterFactory.java
index 2ad2568d5cc2..2561a69f60b5 100644
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestS3ACommitterFactory.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestS3ACommitterFactory.java
@@ -19,15 +19,24 @@
package org.apache.hadoop.fs.s3a.commit;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
import org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter;
import org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter;
import org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitter;
import org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter;
+import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapreduce.MRJobConfig;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.mapreduce.TaskAttemp
(hadoop) branch branch-3.4 updated: HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS Authentication (#6552)
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 93c787be00de HADOOP-18516: [ABFS][Authentication] Support Fixed SAS
Token for ABFS Authentication (#6552)
93c787be00de is described below
commit 93c787be00de67d085d5731450c9f075ebcbadf5
Author: Anuj Modi <128447756+anujmodi2...@users.noreply.github.com>
AuthorDate: Fri Jun 7 19:03:23 2024 +0530
HADOOP-18516: [ABFS][Authentication] Support Fixed SAS Token for ABFS
Authentication (#6552)
Contributed by Anuj Modi
---
.../hadoop/fs/azurebfs/AbfsConfiguration.java | 75 ++---
.../hadoop/fs/azurebfs/AzureBlobFileSystem.java| 3 +-
.../fs/azurebfs/AzureBlobFileSystemStore.java | 2 +-
.../fs/azurebfs/constants/ConfigurationKeys.java | 5 +-
.../hadoop/fs/azurebfs/services/AbfsClient.java| 9 +-
.../azurebfs/services/FixedSASTokenProvider.java | 65
.../hadoop-azure/src/site/markdown/abfs.md | 149 ++---
.../fs/azurebfs/AbstractAbfsIntegrationTest.java | 23 ++-
.../ITestAzureBlobFileSystemChooseSAS.java | 182 +
.../extensions/MockDelegationSASTokenProvider.java | 2 +-
.../azurebfs/extensions/MockSASTokenProvider.java | 16 +-
.../fs/azurebfs/utils/AccountSASGenerator.java | 103
.../hadoop/fs/azurebfs/utils/SASGenerator.java | 34 +++-
.../fs/azurebfs/utils/ServiceSASGenerator.java | 15 +-
14 files changed, 611 insertions(+), 72 deletions(-)
diff --git
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
index a1b6fc12a5ce..1bca79628702 100644
---
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
+++
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
@@ -22,6 +22,7 @@ import java.io.IOException;
import java.lang.reflect.Field;
import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.azurebfs.services.FixedSASTokenProvider;
import org.apache.hadoop.util.Preconditions;
import org.apache.commons.lang3.StringUtils;
@@ -980,33 +981,63 @@ public class AbfsConfiguration{
}
}
+ /**
+ * Returns the SASTokenProvider implementation to be used to generate SAS
token.
+ * Users can choose between a custom implementation of {@link
SASTokenProvider}
+ * or an in house implementation {@link FixedSASTokenProvider}.
+ * For Custom implementation "fs.azure.sas.token.provider.type" needs to be
provided.
+ * For Fixed SAS Token use "fs.azure.sas.fixed.token" needs to be
provided.
+ * In case both are provided, Preference will be given to Custom
implementation.
+ * Avoid using a custom tokenProvider implementation just to read the
configured
+ * fixed token, as this could create confusion. Also,implementing the
SASTokenProvider
+ * requires relying on the raw configurations. It is more stable to depend on
+ * the AbfsConfiguration with which a filesystem is initialized, and
eliminate
+ * chances of dynamic modifications and spurious situations.
+ * @return sasTokenProvider object based on configurations provided
+ * @throws AzureBlobFileSystemException
+ */
public SASTokenProvider getSASTokenProvider() throws
AzureBlobFileSystemException {
AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME,
AuthType.SharedKey);
if (authType != AuthType.SAS) {
throw new SASTokenProviderException(String.format(
-"Invalid auth type: %s is being used, expecting SAS", authType));
+ "Invalid auth type: %s is being used, expecting SAS.", authType));
}
try {
- String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
- Class sasTokenProviderClass =
- getTokenProviderClass(authType, configKey, null,
- SASTokenProvider.class);
-
- Preconditions.checkArgument(sasTokenProviderClass != null,
- String.format("The configuration value for \"%s\" is invalid.",
configKey));
-
- SASTokenProvider sasTokenProvider = ReflectionUtils
- .newInstance(sasTokenProviderClass, rawConfig);
- Preconditions.checkArgument(sasTokenProvider != null,
- String.format("Failed to initialize %s", sasTokenProviderClass));
-
- LOG.trace("Initializing {}", sasTokenProviderClass.getName());
- sasTokenProvider.initialize(rawConfig, accountName);
- LOG.trace("{} init complete", sasTokenProviderClass.getName());
- return sasTokenProvider;
+ Class customSasTokenProviderImplementation =
+ getTokenProviderClass(authTyp
(hadoop) branch branch-3.4 updated: HADOOP-19154. Upgrade bouncycastle to 1.78.1 due to CVEs (#6755) (#6866)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new 59b29800bdce HADOOP-19154. Upgrade bouncycastle to 1.78.1 due to CVEs (#6755) (#6866) 59b29800bdce is described below commit 59b29800bdce4bfe9b52ccf2d8bd6eeb0f35176e Author: PJ Fanning AuthorDate: Fri Jun 7 14:32:27 2024 +0100 HADOOP-19154. Upgrade bouncycastle to 1.78.1 due to CVEs (#6755) (#6866) Addresses * CVE-2024-29857 - Importing an EC certificate with specially crafted F2m parameters can cause high CPU usage during parameter evaluation. * CVE-2024-30171 - Possible timing based leakage in RSA based handshakes due to exception processing eliminated. * CVE-2024-30172 - Crafted signature and public key can be used to trigger an infinite loop in the Ed25519 verification code. * CVE-2024-301XX - When endpoint identification is enabled and an SSL socket is not created with an explicit hostname (as happens with HttpsURLConnection), hostname verification could be performed against a DNS-resolved IP address. Contributed by PJ Fanning --- LICENSE-binary | 6 +++--- .../hadoop-cos/src/site/markdown/cloud-storage/index.md | 2 +- hadoop-project/pom.xml | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 92d20725b813..3ab3ef5d5e28 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -481,9 +481,9 @@ com.microsoft.azure:azure-cosmosdb-gateway:2.4.5 com.microsoft.azure:azure-data-lake-store-sdk:2.3.3 com.microsoft.azure:azure-keyvault-core:1.0.0 com.microsoft.sqlserver:mssql-jdbc:6.2.1.jre7 -org.bouncycastle:bcpkix-jdk18on:1.77 -org.bouncycastle:bcprov-jdk18on:1.77 -org.bouncycastle:bcutil-jdk18on:1.77 +org.bouncycastle:bcpkix-jdk18on:1.78.1 +org.bouncycastle:bcprov-jdk18on:1.78.1 +org.bouncycastle:bcutil-jdk18on:1.78.1 org.checkerframework:checker-qual:2.5.2 org.codehaus.mojo:animal-sniffer-annotations:1.21 org.jruby.jcodings:jcodings:1.0.13 diff --git a/hadoop-cloud-storage-project/hadoop-cos/src/site/markdown/cloud-storage/index.md b/hadoop-cloud-storage-project/hadoop-cos/src/site/markdown/cloud-storage/index.md index 64647b03e9ba..60c9c9065946 100644 --- a/hadoop-cloud-storage-project/hadoop-cos/src/site/markdown/cloud-storage/index.md +++ b/hadoop-cloud-storage-project/hadoop-cos/src/site/markdown/cloud-storage/index.md @@ -86,7 +86,7 @@ Linux kernel 2.6+ - joda-time (version 2.9.9 recommended) - httpClient (version 4.5.1 or later recommended) - Jackson: jackson-core, jackson-databind, jackson-annotations (version 2.9.8 or later) -- bcprov-jdk18on (version 1.77 recommended) +- bcprov-jdk18on (version 1.78.1 recommended) Configure Properties diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index f7b13344ea6c..4e42e3c895e9 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -111,7 +111,7 @@ 27.0-jre 4.2.3 -1.77 +1.78.1 2.0.0.AM26 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19178: [WASB Deprecation] Updating Documentation on Upcoming Plans for Hadoop-Azure (#6862)
This is an automated email from the ASF dual-hosted git repository. stevel pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new f2e16c586958 HADOOP-19178: [WASB Deprecation] Updating Documentation on Upcoming Plans for Hadoop-Azure (#6862) f2e16c586958 is described below commit f2e16c58695800a8444cceed6d6cea9ac5ca1599 Author: Anuj Modi <128447756+anujmodi2...@users.noreply.github.com> AuthorDate: Fri Jun 7 18:58:24 2024 +0530 HADOOP-19178: [WASB Deprecation] Updating Documentation on Upcoming Plans for Hadoop-Azure (#6862) Contributed by Anuj Modi --- .../hadoop-azure/src/site/markdown/index.md| 1 + .../hadoop-azure/src/site/markdown/wasb.md | 97 ++ 2 files changed, 98 insertions(+) diff --git a/hadoop-tools/hadoop-azure/src/site/markdown/index.md b/hadoop-tools/hadoop-azure/src/site/markdown/index.md index 595353896d12..177ab282c112 100644 --- a/hadoop-tools/hadoop-azure/src/site/markdown/index.md +++ b/hadoop-tools/hadoop-azure/src/site/markdown/index.md @@ -18,6 +18,7 @@ See also: +* [WASB](./wasb.html) * [ABFS](./abfs.html) * [Testing](./testing_azure.html) diff --git a/hadoop-tools/hadoop-azure/src/site/markdown/wasb.md b/hadoop-tools/hadoop-azure/src/site/markdown/wasb.md new file mode 100644 index ..270fd14da4c4 --- /dev/null +++ b/hadoop-tools/hadoop-azure/src/site/markdown/wasb.md @@ -0,0 +1,97 @@ + + +# Hadoop Azure Support: WASB Driver + +## Introduction +WASB Driver is a legacy Hadoop File System driver that was developed to support +[FNS(FlatNameSpace) Azure Storage accounts](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) +that do not honor File-Folder syntax. +HDFS Folder operations hence are mimicked at client side by WASB driver and +certain folder operations like Rename and Delete can lead to a lot of IOPs with +client-side enumeration and orchestration of rename/delete operation blob by blob. +It was not ideal for other APIs too as initial checks for path is a file or folder +needs to be done over multiple metadata calls. These led to a degraded performance. + +To provide better service to Analytics users, Microsoft released [ADLS Gen2](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) +which are HNS (Hierarchical Namespace) enabled, i.e. File-Folder aware storage accounts. +ABFS driver was designed to overcome the inherent deficiencies of WASB and users +were informed to migrate to ABFS driver. + +### Challenges and limitations of WASB Driver +Users of the legacy WASB driver face a number of challenges and limitations: +1. They cannot leverage the optimizations and benefits of the latest ABFS driver. +2. They need to deal with the compatibility issues should the files and folders were +modified with the legacy WASB driver and the ABFS driver concurrently in a phased +transition situation. +3. There are differences for supported features for FNS and HNS over ABFS Driver. +4. In certain cases, they must perform a significant amount of re-work on their +workloads to migrate to the ABFS driver, which is available only on HNS enabled +accounts in a fully tested and supported scenario. + +## Deprecation plans for WASB Driver +We are introducing a new feature that will enable the ABFS driver to support +FNS accounts (over BlobEndpoint that WASB Driver uses) using the ABFS scheme. +This feature will enable us to use the ABFS driver to interact with data stored in GPv2 +(General Purpose v2) storage accounts. + +With this feature, the users who still use the legacy WASB driver will be able +to migrate to the ABFS driver without much re-work on their workloads. They will +however need to change the URIs from the WASB scheme to the ABFS scheme. + +Once ABFS driver has built FNS support capability to migrate WASB users, WASB +driver will be marked for removal in next major release. This will remove any ambiguity +for new users onboards as there will be only one Microsoft driver for Azure Storage +and migrating users will get SLA bound support for driver and service, +which was not guaranteed over WASB. + +We anticipate that this feature will serve as a stepping stone for users to +move to HNS enabled accounts with the ABFS driver, which is our recommended stack +for big data analytics on ADLS Gen2. + +### Impact for existing ABFS users using ADLS Gen2 (HNS enabled account) +This feature does not impact the existing users who are using ADLS Gen2 Accounts +(HNS enabled account) with ABFS driver. + +They do not need to make any changes to their workloads or configurations. They +will still enjoy the benefits of HNS, such as atomic operations, fine-grained +access control, scalability, and performance. + +### Official recommendation +Microsoft continues to recommend all Big Data and Analytics users
