This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 113891130ce HADOOP-19696. hadoop binary distribution to move cloud
connectors to hadoop common/lib (#7980) (#8094)
113891130ce is described below
commit 113891130ce0a1429770a54d8fad0fa19019ea33
Author: Steve Loughran <[email protected]>
AuthorDate: Sun Nov 30 15:04:42 2025 +0000
HADOOP-19696. hadoop binary distribution to move cloud connectors to hadoop
common/lib (#7980) (#8094)
This moves all the cloud connector libraries to common/lib
There are specific build options to control which libraries to include
The hadoop-* JARs of the modules are includes, but dependencies
are only included when the build-time options specify it.
Available package profiles:
hadoop-aliyun-package
hadoop-aws-package
hadoop-azure-datalake-package
hadoop-cos-package
hadoop-huaweicloud-package
This means that by default AWS bundle.jar is no longer included
in the distribution: to add it users must drop their chosen version
of the SDK into share/hadoop/common/lib
Anyone building their own release now has a choice of which connectors
to bundle. The ASF ones will stay fairly lean to reduce the CVE
attack surface as well as keep package size under control.
+ followups
* HADOOP-19696. restore hadoop-aliyun-package to docs
* HADOOP-19696. Followup: Li2cense spelling in assembly xml files
Contributed by Steve Loughran
---
BUILDING.txt | 44 ++++
LICENSE-binary | 22 +-
dev-support/bin/dist-layout-stitching | 4 +
.../resources/assemblies/hadoop-cloud-storage.xml | 59 +++++
.../resources/assemblies/hadoop-dynamometer.xml | 2 +-
.../src/main/resources/assemblies/hadoop-src.xml | 1 +
.../src/main/resources/assemblies/hadoop-tools.xml | 2 +-
.../hadoop-cloud-storage-dist/pom.xml | 282 +++++++++++++++++++++
.../hadoop-huaweicloud/pom.xml | 4 +
.../services/org.apache.hadoop.fs.FileSystem | 16 --
hadoop-cloud-storage-project/pom.xml | 16 +-
.../src/main/resources/core-default.xml | 6 +
.../hadoop/conf/TestCommonConfigurationFields.java | 2 +
hadoop-project/pom.xml | 24 ++
.../services/org.apache.hadoop.fs.FileSystem | 16 --
hadoop-tools/hadoop-tools-dist/pom.xml | 24 --
16 files changed, 462 insertions(+), 62 deletions(-)
diff --git a/BUILDING.txt b/BUILDING.txt
index 1e2a1fef1a0..173cc0bcad4 100644
--- a/BUILDING.txt
+++ b/BUILDING.txt
@@ -385,6 +385,50 @@ Create a local staging version of the website (in
/tmp/hadoop-site)
Note that the site needs to be built in a second pass after other artifacts.
+----------------------------------------------------------------------------------
+Including Cloud Connector Dependencies in Distributions:
+
+Hadoop distributions include the hadoop modules needed to work with data and
services
+on cloud infrastructure
+
+However, dependencies are omitted for all cloud connectors except hadoop-azure
+(abfs:// and wasb://) and possibly hadoop-gcp (gs://) and hadoop-tos (tos://).
+For the latter two modules, it depends on shading options.
+
+For hadoop-aws the AWS SDK bundle.jar is omitted, but everything else is
included.
+
+Excluding the extra binaries:
+* Keeps release artifact size below the limit of the ASF distribution network.
+* Reduces download and size overhead in docker usage.
+* Reduces the CVE attack surface and audit-related complaints about those same
CVEs.
+* Reduces the risk of classpath conflict.
+
+To produce a build with the specific desired dependencies, the build must be
executed
+with the relevant profile of ${module}-package alongside the -Pdist profile.
+
+For example, a build with the hadoop-aws and hadoop-azure-datalake
dependencies,
+run with
+
+ mvn package -Pdist -DskipTests -Dhadoop-aws-package
-Dhadoop-azure-datalake-package
+
+Available package profiles:
+ hadoop-aliyun-package
+ hadoop-aws-package
+ hadoop-azure-datalake-package
+ hadoop-cos-package
+ hadoop-huaweicloud-package
+
+To build a complete distribution then with all cloud dependencies included:
+
+mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true \
+ -Dhadoop-aliyun-package \
+ -Dhadoop-aws-package \
+ -Dhadoop-azure-datalake-package \
+ -Dhadoop-cos-package \
+ -Dhadoop-huaweicloud-package
+
+The resulting tar file will be too large to be distributable through ASF
infrastructure.
+
----------------------------------------------------------------------------------
Installing Hadoop
diff --git a/LICENSE-binary b/LICENSE-binary
index a1bc31f23b8..12842b6961b 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -203,7 +203,10 @@
--------------------------------------------------------------------------------
This project bundles some components that are also licensed under the Apache
-License Version 2.0:
+License Version 2.0.
+Note: some of the listed artifacts may not be included in a given build of the
binary
+distribution; it depends on the build options. This list intends
+to be inclusive of all which may be included:
hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/nvd3-1.8.5.* (css and
js files)
@@ -211,10 +214,12 @@
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/data
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/TimeoutFuture.java
ch.qos.reload4j:reload4j:1.2.22
+com.aliyun:aliyun-java-core:0.2.11-beta
com.aliyun:aliyun-java-sdk-core:4.5.10
com.aliyun:aliyun-java-sdk-kms:2.11.0
com.aliyun:aliyun-java-sdk-ram:3.1.0
com.aliyun:aliyun-java-sdk-sts:3.0.0
+com.aliyun:java-trace-api:0.2.11-beta
com.aliyun.oss:aliyun-sdk-oss:3.13.2
com.cedarsoftware:java-util:1.9.0
com.cedarsoftware:json-io:2.5.1
@@ -241,6 +246,8 @@ com.google.guava:guava:33.4.8-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.microsoft.azure:azure-storage:7.0.0
com.nimbusds:nimbus-jose-jwt:10.4
+com.squareup.okhttp3:okhttp:jar:3.14.2
+com.squareup.okio:okio:jar:1.17.2
com.zaxxer:HikariCP:4.0.3
commons-beanutils:commons-beanutils:1.9.4
commons-cli:commons-cli:1.9.0
@@ -289,6 +296,9 @@ io.netty:netty-transport-native-kqueue:4.1.127.Final
io.netty:netty-resolver-dns-native-macos:4.1.127.Final
io.opencensus:opencensus-api:0.12.3
io.opencensus:opencensus-contrib-grpc-metrics:0.12.3
+io.opentracing:opentracing-api:0.33.0.jar
+io.opentracing:opentracing-noop:0.33.0.jar
+io.opentracing:opentracing-util:0.33.0.jar
io.reactivex:rxjava:1.3.8
io.reactivex:rxjava-string:1.1.1
io.reactivex:rxnetty:0.4.20
@@ -316,6 +326,8 @@ org.apache.htrace:htrace-core:3.1.0-incubating
org.apache.htrace:htrace-core4:4.1.0-incubating
org.apache.httpcomponents:httpclient:4.5.13
org.apache.httpcomponents:httpcore:4.4.13
+org.apache.httpcomponents.client5:httpclient5:5.5
+org.apache.httpcomponents.core5:httpcore5:5.5
org.apache.kafka:kafka-clients:3.9.0
org.apache.kerby:kerb-admin:2.0.3
org.apache.kerby:kerb-client:2.0.3
@@ -432,6 +444,7 @@
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanage
bootstrap v3.3.6
broccoli-asset-rev v2.4.2
broccoli-funnel v1.0.1
+cos_api-bundle-5.6.19.jar
datatables v1.11.5
em-helpers v0.5.13
em-table v0.1.6
@@ -477,7 +490,7 @@ com.microsoft.azure:azure-cosmosdb:2.4.5
com.microsoft.azure:azure-cosmosdb-commons:2.4.5
com.microsoft.azure:azure-cosmosdb-direct:2.4.5
com.microsoft.azure:azure-cosmosdb-gateway:2.4.5
-com.microsoft.azure:azure-data-lake-store-sdk:2.3.3
+com.microsoft.azure:azure-data-lake-store-sdk:2.3.9
com.microsoft.azure:azure-keyvault-core:1.0.0
com.microsoft.sqlserver:mssql-jdbc:6.2.1.jre7
org.bouncycastle:bcpkix-jdk18on:1.82
@@ -536,3 +549,8 @@ Public Domain
-------------
aopalliance:aopalliance:1.0
+
+Dom4J license
+-------------
+
+org.dom4j:dom4j:2.1.4.jar
\ No newline at end of file
diff --git a/dev-support/bin/dist-layout-stitching
b/dev-support/bin/dist-layout-stitching
index d4bfd8aaada..e19b2e4f1d6 100755
--- a/dev-support/bin/dist-layout-stitching
+++ b/dev-support/bin/dist-layout-stitching
@@ -130,6 +130,10 @@ run cp -p "${ROOT}/README.txt" .
run copy
"${ROOT}/hadoop-common-project/hadoop-common/target/hadoop-common-${VERSION}" .
run copy
"${ROOT}/hadoop-common-project/hadoop-nfs/target/hadoop-nfs-${VERSION}" .
run copy
"${ROOT}/hadoop-common-project/hadoop-registry/target/hadoop-registry-${VERSION}"
.
+
+# cloud connectors go into common
+run copy
"${ROOT}/hadoop-cloud-storage-project/hadoop-cloud-storage-dist/target/hadoop-cloud-storage-dist-${VERSION}"
.
+
run copy
"${ROOT}/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-${VERSION}" .
run copy
"${ROOT}/hadoop-hdfs-project/hadoop-hdfs-nfs/target/hadoop-hdfs-nfs-${VERSION}"
.
run copy
"${ROOT}/hadoop-hdfs-project/hadoop-hdfs-client/target/hadoop-hdfs-client-${VERSION}"
.
diff --git
a/hadoop-assemblies/src/main/resources/assemblies/hadoop-cloud-storage.xml
b/hadoop-assemblies/src/main/resources/assemblies/hadoop-cloud-storage.xml
new file mode 100644
index 00000000000..03f62dcb4b8
--- /dev/null
+++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-cloud-storage.xml
@@ -0,0 +1,59 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3
https://maven.apache.org/xsd/assembly-1.1.3.xsd";>
+ <id>hadoop-cloud-storage</id>
+ <formats>
+ <format>dir</format>
+ </formats>
+ <includeBaseDirectory>false</includeBaseDirectory>
+
+ <!--
+ This is executed in directory
hadoop-cloud-storage-project/hadoop-cloud-storage-dist
+ All paths must be relative to that.
+ -->
+ <fileSets>
+ <fileSet>
+ <directory>../../hadoop-tools/hadoop-aws/src/main/bin</directory>
+ <outputDirectory>/bin</outputDirectory>
+ <fileMode>0755</fileMode>
+ </fileSet>
+ <fileSet>
+
<directory>./../hadoop-tools/hadoop-aws/src/main/shellprofile.d</directory>
+ <includes>
+ <include>*</include>
+ </includes>
+ <outputDirectory>/libexec/shellprofile.d</outputDirectory>
+ <fileMode>0755</fileMode>
+ </fileSet>
+ </fileSets>
+
+ <dependencySets>
+ <dependencySet>
+ <outputDirectory>/share/hadoop/common/lib</outputDirectory>
+ <unpack>false</unpack>
+ <scope>runtime</scope>
+ <useProjectArtifact>false</useProjectArtifact>
+ <!-- Stop some needless artifact propagation -->
+ <excludes>
+ <exclude>org.apache.hadoop:hadoop-annotations</exclude>
+ <exclude>org.apache.hadoop.thirdparty:hadoop-shaded-guava</exclude>
+ </excludes>
+ </dependencySet>
+ </dependencySets>
+</assembly>
diff --git
a/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer.xml
b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer.xml
index b2ce562231c..508faa9c85e 100644
--- a/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer.xml
+++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer.xml
@@ -9,7 +9,7 @@
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
- distributed under the Li2cense is distributed on an "AS IS" BASIS,
+ distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-src.xml
b/hadoop-assemblies/src/main/resources/assemblies/hadoop-src.xml
index 87169420939..7895f4e5714 100644
--- a/hadoop-assemblies/src/main/resources/assemblies/hadoop-src.xml
+++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-src.xml
@@ -57,6 +57,7 @@
<exclude>**/file:/**</exclude>
<exclude>**/SecurityAuth.audit*</exclude>
<exclude>patchprocess/**</exclude>
+ <exclude>**/auth-keys.xml</exclude>
</excludes>
</fileSet>
</fileSets>
diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
b/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
index c01d9c42820..4276820a7b8 100644
--- a/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
+++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
@@ -9,7 +9,7 @@
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
- distributed under the Li2cense is distributed on an "AS IS" BASIS,
+ distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
diff --git a/hadoop-cloud-storage-project/hadoop-cloud-storage-dist/pom.xml
b/hadoop-cloud-storage-project/hadoop-cloud-storage-dist/pom.xml
new file mode 100644
index 00000000000..10bfdeb6f0b
--- /dev/null
+++ b/hadoop-cloud-storage-project/hadoop-cloud-storage-dist/pom.xml
@@ -0,0 +1,282 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
https://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-project</artifactId>
+ <version>3.4.3-SNAPSHOT</version>
+ <relativePath>../../hadoop-project</relativePath>
+ </parent>
+ <artifactId>hadoop-cloud-storage-dist</artifactId>
+ <version>3.4.3-SNAPSHOT</version>
+ <packaging>jar</packaging>
+
+ <description>Apache Hadoop Cloud Storage Distribution</description>
+ <name>Apache Hadoop Cloud Storage Distribution</name>
+
+ <!--
+ This pulls in all the artifacts to copy into common/lib and so put into
+ the Hadoop distro and onto the classpath.
+
+ The assembly file
/hadoop-assemblies/src/main/resources/assemblies/hadoop-cloud-storage.xml
+ is processed to define the layout and to add extra files alongside
+ the Jars.
+
+ By default, while hadoop-* artifacts are all included, dependencies
+ are omitted for all cloud connectors except hadoop-azure and
+ possibly hadoop-gcp and hadoop-tos modules.
+ For hadoop-aws the AWS SDK bundle.jar omitted, but everything else is
included.
+
+ * This keeps binary release size below the limit of apache distributions
+ * Reduces download and size overhead in docker usage.
+ * Reduces the CVE attack surface
+ * Reduces the risk of classpath conflict.
+
+ To produce a build with the specific desired dependencies, the build must be
executed
+ with the relevant profile of ${module}-package.
+
+ For example, a build with the hadoop-aws and hadoop-azure-datalake
dependencies,
+ build with -Dhadoop-aws-package -Dhadoop-azure-datalake-package
+
+ Available package profiles:
+ hadoop-aliyun-package
+ hadoop-aws-package
+ hadoop-azure-datalake-package
+ hadoop-cos-package
+ hadoop-huaweicloud-package
+
+ To build a complete distribution then:
+
+mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true \
+ -Dhadoop-aws-package \
+ -Dhadoop-azure-datalake-package \
+ -Dhadoop-cos-package \
+ -Dhadoop-huaweicloud-package
+
+ -->
+ <properties>
+ <hadoop.component>cloud-storage</hadoop.component>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-annotations</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <scope>provided</scope>
+ </dependency>
+
+ <!-- By default the AWS SDK is omitted from releases, but everything else
is
+ included -->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-aws</artifactId>
+ <scope>compile</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-azure</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-azure-datalake</artifactId>
+ <scope>compile</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-cos</artifactId>
+ <scope>compile</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-huaweicloud</artifactId>
+ <scope>compile</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <artifactId>maven-deploy-plugin</artifactId>
+ <configuration>
+ <skip>true</skip>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ <configuration>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+
+ <profiles>
+
+ <!-- distribution profile builds the artifacts. -->
+ <profile>
+ <id>dist</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-assemblies</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+ <executions>
+ <execution>
+ <id>dist</id>
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ <configuration>
+ <appendAssemblyId>false</appendAssemblyId>
+ <attach>false</attach>
+
<finalName>${project.artifactId}-${project.version}</finalName>
+ <descriptorRefs>
+ <descriptorRef>hadoop-cloud-storage</descriptorRef>
+ </descriptorRefs>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+
+ <!-- Pull in aliyun -->
+ <profile>
+ <id>hadoop-aliyun-package</id>
+ <activation>
+ <property><name>hadoop-aliyun-package</name></property>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-aliyun</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+
+ <!-- Pull in the AWS SDK -->
+ <profile>
+ <id>hadoop-aws-package</id>
+ <activation>
+ <property><name>hadoop-aws-package</name></property>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-aws</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+
+ <!-- Pull in ADLS gen1 support -->
+ <profile>
+ <id>hadoop-azure-datalake-package</id>
+ <activation>
+ <property><name>hadoop-azure-datalake-package</name></property>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-azure-datalake</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+
+ <!-- Pull in all the hadoop-cos dependencies -->
+ <profile>
+ <id>hadoop-cos-package</id>
+ <activation>
+ <property><name>hadoop-cos-package</name></property>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-cos</artifactId>
+ <scope>compile</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+
+ <!-- Pull in the huaweicloud dependencies -->
+ <profile>
+ <id>hadoop-huaweicloud-package</id>
+ <activation>
+ <property><name>hadoop-huaweicloud-package</name></property>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-huaweicloud</artifactId>
+ <scope>compile</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.fasterxml.jackson</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+ </profile>
+
+ </profiles>
+</project>
diff --git a/hadoop-cloud-storage-project/hadoop-huaweicloud/pom.xml
b/hadoop-cloud-storage-project/hadoop-huaweicloud/pom.xml
index 3b7c81404a3..f7b648ae453 100755
--- a/hadoop-cloud-storage-project/hadoop-huaweicloud/pom.xml
+++ b/hadoop-cloud-storage-project/hadoop-huaweicloud/pom.xml
@@ -169,6 +169,10 @@
<artifactId>log4j-api</artifactId>
<groupId>org.apache.logging.log4j</groupId>
</exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
</exclusions>
</dependency>
<dependency>
diff --git
a/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
b/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
deleted file mode 100644
index e77425ab529..00000000000
---
a/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
+++ /dev/null
@@ -1,16 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-org.apache.hadoop.fs.obs.OBSFileSystem
diff --git a/hadoop-cloud-storage-project/pom.xml
b/hadoop-cloud-storage-project/pom.xml
index 6e32bc22188..70904617036 100644
--- a/hadoop-cloud-storage-project/pom.xml
+++ b/hadoop-cloud-storage-project/pom.xml
@@ -34,7 +34,6 @@
<module>hadoop-cos</module>
<module>hadoop-huaweicloud</module>
</modules>
-
<build>
<plugins>
<plugin>
@@ -51,5 +50,18 @@
</plugin>
</plugins>
</build>
-
+ <profiles>
+ <profile>
+ <id>dist</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ <property>
+ <name>tar</name>
+ </property>
+ </activation>
+ <modules>
+ <module>hadoop-cloud-storage-dist</module>
+ </modules>
+ </profile>
+ </profiles>
</project>
diff --git
a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
index 3e1351c3c88..db548e1e692 100644
--- a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
+++ b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
@@ -4438,4 +4438,10 @@ The switch to turn S3A auditing on or off.
<description>The AbstractFileSystem for Ozone FileSystem o3fs
uri</description>
</property>
+ <property>
+ <name>fs.obs.impl</name>
+ <value>org.apache.hadoop.fs.obs.OBSFileSystem</value>
+ <description>The FileSystem for obs: uris.</description>
+ </property>
+
</configuration>
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
index 210b36f2dbd..c0e8b4690cb 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
@@ -250,5 +250,7 @@ public void initializeMemberVariables() {
xmlPropsToSkipCompare.add("io.seqfile.local.dir");
xmlPropsToSkipCompare.add("hadoop.http.sni.host.check.enabled");
+
+ xmlPrefixToSkipCompare.add("fs.obs.");
}
}
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 3e79fe1b758..251d86965e3 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -75,6 +75,8 @@
<!-- httpcomponents versions -->
<httpclient.version>4.5.13</httpclient.version>
<httpcore.version>4.4.13</httpcore.version>
+ <httpclient5.version>5.5</httpclient5.version>
+ <httpcore5.version>5.3.6</httpcore5.version>
<!-- SLF4J/LOG4J version -->
<slf4j.version>1.7.36</slf4j.version>
@@ -840,6 +842,28 @@
<artifactId>httpcore</artifactId>
<version>${httpcore.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents.client5</groupId>
+ <artifactId>httpclient5</artifactId>
+ <version>${httpclient5.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents.core5</groupId>
+ <artifactId>httpcore5</artifactId>
+ <version>${httpcore5.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
diff --git
a/hadoop-tools/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
b/hadoop-tools/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
deleted file mode 100644
index 1938bdefa41..00000000000
---
a/hadoop-tools/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
+++ /dev/null
@@ -1,16 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-org.apache.hadoop.fs.s3native.NativeS3FileSystem
diff --git a/hadoop-tools/hadoop-tools-dist/pom.xml
b/hadoop-tools/hadoop-tools-dist/pom.xml
index 2f064d82401..1da0df465ca 100644
--- a/hadoop-tools/hadoop-tools-dist/pom.xml
+++ b/hadoop-tools/hadoop-tools-dist/pom.xml
@@ -92,30 +92,12 @@
<type>pom</type>
<version>${project.version}</version>
</dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-aws</artifactId>
- <scope>compile</scope>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-kafka</artifactId>
<scope>compile</scope>
<version>${project.version}</version>
</dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-azure</artifactId>
- <scope>compile</scope>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-aliyun</artifactId>
- <scope>compile</scope>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-sls</artifactId>
@@ -127,12 +109,6 @@
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-azure-datalake</artifactId>
- <scope>compile</scope>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-fs2img</artifactId>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
