(hadoop) branch branch-3.4 updated: HADOOP-19222. Switch yum repo baseurl due to CentOS 7 sunset (#6932) Contributed by Cheng Pan.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new 69806dee9da HADOOP-19222. Switch yum repo baseurl due to CentOS 7 sunset (#6932) Contributed by Cheng Pan. 69806dee9da is described below commit 69806dee9da969d4a0050b07e60c5f62a0903844 Author: Cheng Pan AuthorDate: Fri Jul 12 06:16:44 2024 +0800 HADOOP-19222. Switch yum repo baseurl due to CentOS 7 sunset (#6932) Contributed by Cheng Pan. Reviewed-by: Gautham Banasandra Signed-off-by: Shilun Fan --- dev-support/docker/Dockerfile_centos_7 | 16 ++-- .../docker/pkg-resolver/set-vault-as-baseurl-centos.sh | 2 +- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/dev-support/docker/Dockerfile_centos_7 b/dev-support/docker/Dockerfile_centos_7 index ff947637a2c..b97e59969a7 100644 --- a/dev-support/docker/Dockerfile_centos_7 +++ b/dev-support/docker/Dockerfile_centos_7 @@ -30,6 +30,13 @@ COPY pkg-resolver pkg-resolver RUN chmod a+x pkg-resolver/*.sh pkg-resolver/*.py \ && chmod a+r pkg-resolver/*.json +## +# Centos 7 has reached its EOL and the packages +# are no longer available on mirror.centos.org site. +# Please see https://www.centos.org/centos-linux-eol/ +## +RUN pkg-resolver/set-vault-as-baseurl-centos.sh centos:7 + ## # Install packages from yum ## @@ -38,8 +45,13 @@ RUN yum update -y \ && yum groupinstall -y "Development Tools" \ && yum install -y \ centos-release-scl \ -python3 \ -&& yum install -y $(pkg-resolver/resolve.py centos:7) +python3 + +# Apply the script again because centos-release-scl creates new YUM repo files +RUN pkg-resolver/set-vault-as-baseurl-centos.sh centos:7 + +# hadolint ignore=DL3008,SC2046 +RUN yum install -y $(pkg-resolver/resolve.py centos:7) # Set GCC 9 as the default C/C++ compiler RUN echo "source /opt/rh/devtoolset-9/enable" >> /etc/bashrc diff --git a/dev-support/docker/pkg-resolver/set-vault-as-baseurl-centos.sh b/dev-support/docker/pkg-resolver/set-vault-as-baseurl-centos.sh index 4be4cd956b1..905ac5077de 100644 --- a/dev-support/docker/pkg-resolver/set-vault-as-baseurl-centos.sh +++ b/dev-support/docker/pkg-resolver/set-vault-as-baseurl-centos.sh @@ -24,7 +24,7 @@ fi if [ "$1" == "centos:7" ] || [ "$1" == "centos:8" ]; then cd /etc/yum.repos.d/ || exit && sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-* && -sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-* && +sed -i 's|# *baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-* && yum update -y && cd /root || exit else - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19222. Switch yum repo baseurl due to CentOS 7 sunset (#6932) Contributed by Cheng Pan.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 4f0ee9d67d9 HADOOP-19222. Switch yum repo baseurl due to CentOS 7 sunset (#6932) Contributed by Cheng Pan. 4f0ee9d67d9 is described below commit 4f0ee9d67d9e7ac4195fe2ed8227760a63064091 Author: Cheng Pan AuthorDate: Fri Jul 12 06:16:44 2024 +0800 HADOOP-19222. Switch yum repo baseurl due to CentOS 7 sunset (#6932) Contributed by Cheng Pan. Reviewed-by: Gautham Banasandra Signed-off-by: Shilun Fan --- dev-support/docker/Dockerfile_centos_7 | 16 ++-- .../docker/pkg-resolver/set-vault-as-baseurl-centos.sh | 2 +- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/dev-support/docker/Dockerfile_centos_7 b/dev-support/docker/Dockerfile_centos_7 index ff947637a2c..b97e59969a7 100644 --- a/dev-support/docker/Dockerfile_centos_7 +++ b/dev-support/docker/Dockerfile_centos_7 @@ -30,6 +30,13 @@ COPY pkg-resolver pkg-resolver RUN chmod a+x pkg-resolver/*.sh pkg-resolver/*.py \ && chmod a+r pkg-resolver/*.json +## +# Centos 7 has reached its EOL and the packages +# are no longer available on mirror.centos.org site. +# Please see https://www.centos.org/centos-linux-eol/ +## +RUN pkg-resolver/set-vault-as-baseurl-centos.sh centos:7 + ## # Install packages from yum ## @@ -38,8 +45,13 @@ RUN yum update -y \ && yum groupinstall -y "Development Tools" \ && yum install -y \ centos-release-scl \ -python3 \ -&& yum install -y $(pkg-resolver/resolve.py centos:7) +python3 + +# Apply the script again because centos-release-scl creates new YUM repo files +RUN pkg-resolver/set-vault-as-baseurl-centos.sh centos:7 + +# hadolint ignore=DL3008,SC2046 +RUN yum install -y $(pkg-resolver/resolve.py centos:7) # Set GCC 9 as the default C/C++ compiler RUN echo "source /opt/rh/devtoolset-9/enable" >> /etc/bashrc diff --git a/dev-support/docker/pkg-resolver/set-vault-as-baseurl-centos.sh b/dev-support/docker/pkg-resolver/set-vault-as-baseurl-centos.sh index 4be4cd956b1..905ac5077de 100644 --- a/dev-support/docker/pkg-resolver/set-vault-as-baseurl-centos.sh +++ b/dev-support/docker/pkg-resolver/set-vault-as-baseurl-centos.sh @@ -24,7 +24,7 @@ fi if [ "$1" == "centos:7" ] || [ "$1" == "centos:8" ]; then cd /etc/yum.repos.d/ || exit && sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-* && -sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-* && +sed -i 's|# *baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-* && yum update -y && cd /root || exit else - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: YARN-11701. [Federation] Enhance Federation Cache Clean Conditions. (#6889) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 9710a8d52f9 YARN-11701. [Federation] Enhance Federation Cache Clean
Conditions. (#6889) Contributed by Shilun Fan.
9710a8d52f9 is described below
commit 9710a8d52f9449ac8f74ef82f512f12018503c19
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Wed Jun 19 08:34:19 2024 +0800
YARN-11701. [Federation] Enhance Federation Cache Clean Conditions. (#6889)
Contributed by Shilun Fan.
Reviewed-by: Ayush Saxena
---
.../hadoop/yarn/server/federation/cache/FederationCaffeineCache.java | 4 +++-
.../hadoop/yarn/server/federation/cache/FederationGuavaCache.java | 4 +++-
.../apache/hadoop/yarn/server/federation/cache/FederationJCache.java | 4 +++-
3 files changed, 9 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationCaffeineCache.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationCaffeineCache.java
index cbf3e9db3db..96a630bae38 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationCaffeineCache.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationCaffeineCache.java
@@ -79,7 +79,9 @@ public class FederationCaffeineCache extends FederationCache {
@Override
public void clearCache() {
-this.cache.cleanUp();
+if (this.cache != null) {
+ this.cache.cleanUp();
+}
this.cache = null;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationGuavaCache.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationGuavaCache.java
index 2ba9e2869fe..01f38343fba 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationGuavaCache.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationGuavaCache.java
@@ -76,7 +76,9 @@ public class FederationGuavaCache extends FederationCache {
@Override
public void clearCache() {
-cache.invalidateAll();
+if (this.cache != null) {
+ cache.invalidateAll();
+}
cache = null;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationJCache.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationJCache.java
index 07f300e65f6..b31a24d2dda 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationJCache.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationJCache.java
@@ -91,7 +91,9 @@ public class FederationJCache extends FederationCache {
@Override
public void clearCache() {
-this.cache.clear();
+if (this.cache != null) {
+ this.cache.clear();
+}
this.cache = null;
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (01d257d5aa94 -> 10df59e42102)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 01d257d5aa94 HADOOP-19189. ITestS3ACommitterFactory failing (#6857) add 10df59e42102 Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#6664)" (#6875) No new revisions were added by this update. Summary of changes: hadoop-project/pom.xml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) 01/01: Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#666…"
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch revert-6664-HADOOP-19071-V3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 83cefeeb5e9db4e79be4d883a5a968e96712e148
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Sat Jun 8 14:47:36 2024 +0800
Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5.
(#666…"
This reverts commit 88ad7db80de6f5e2d6185d5c397b32218d6c20ea.
---
hadoop-project/pom.xml | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index a8ef068bf8da..5b63dd129d6e 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -170,7 +170,7 @@
-Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError
-3.2.5
+3.0.0-M1
${maven-surefire-plugin.version}
${maven-surefire-plugin.version}
@@ -2450,7 +2450,6 @@
${env.DYLD_LIBRARY_PATH}:${project.build.directory}/native/target/usr/local/lib:${hadoop.common.build.dir}/native/target/usr/local/lib
4
- false
false
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch revert-6664-HADOOP-19071-V3 created (now 83cefeeb5e9d)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to branch revert-6664-HADOOP-19071-V3 in repository https://gitbox.apache.org/repos/asf/hadoop.git at 83cefeeb5e9d Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#666…" This branch includes the following new commits: new 83cefeeb5e9d Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#666…" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (9f6c997662c5 -> 167d4c8447f9)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 9f6c997662c5 YARN-11471. [Federation] FederationStateStoreFacade Cache Support Caffeine. (#6795) Contributed by Shilun Fan. add 167d4c8447f9 YARN-11699. Diagnostics lacks userlimit info when user capacity has reached its maximum limit (#6849) Contributed by Jiandan Yang. No new revisions were added by this update. Summary of changes: .../resourcemanager/scheduler/capacity/AbstractLeafQueue.java | 8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: YARN-11471. [Federation] FederationStateStoreFacade Cache Support Caffeine. (#6795) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 9f6c997662c5 YARN-11471. [Federation] FederationStateStoreFacade Cache
Support Caffeine. (#6795) Contributed by Shilun Fan.
9f6c997662c5 is described below
commit 9f6c997662c5212bd7b542c1e5188ad4ede3f840
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Sat Jun 1 06:15:20 2024 +0800
YARN-11471. [Federation] FederationStateStoreFacade Cache Support Caffeine.
(#6795) Contributed by Shilun Fan.
Reviewed-by: Inigo Goiri
Signed-off-by: Shilun Fan
---
LICENSE-binary | 1 +
hadoop-project/pom.xml | 6 ++
.../hadoop-yarn-server-common/pom.xml | 14
...ionJCache.java => FederationCaffeineCache.java} | 86 --
.../server/federation/cache/FederationJCache.java | 2 +-
.../federation/cache/TestFederationCache.java | 3 +-
.../pom.xml| 4 +
.../src/site/markdown/Federation.md| 8 +-
8 files changed, 66 insertions(+), 58 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index c0258e9311b1..32f9f06ae15d 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -226,6 +226,7 @@
com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.12.7
com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.12.7
com.fasterxml.uuid:java-uuid-generator:3.1.4
com.fasterxml.woodstox:woodstox-core:5.4.0
+com.github.ben-manes.caffeine:caffeine:2.9.3
com.github.davidmoten:rxjava-extras:0.8.0.17
com.github.stephenc.jcip:jcip-annotations:1.0-1
com.google:guice:4.0
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index ba7631189a1a..0345925e9994 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -134,6 +134,7 @@
2.0.3
3.8.2
1.1.1
+2.9.3
4.0.3
10.14.2.0
6.2.1.jre7
@@ -1975,6 +1976,11 @@
+
+ com.github.ben-manes.caffeine
+ caffeine
+ ${caffeine.version}
+
com.zaxxer
HikariCP
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
index 1f762d31800d..e768ad5e4845 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
@@ -131,6 +131,20 @@
org.ehcache
ehcache
+
+ com.github.ben-manes.caffeine
+ caffeine
+
+
+ org.checkerframework
+ checker-qual
+
+
+ com.google.errorprone
+ error_prone_annotations
+
+
+
com.zaxxer
HikariCP
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationJCache.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationCaffeineCache.java
similarity index 60%
copy from
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationJCache.java
copy to
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationCaffeineCache.java
index b4dbefe1278a..cbf3e9db3db3 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationJCache.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationCaffeineCache.java
@@ -14,10 +14,10 @@
* License for the specific language governing permissions and limitations
under
* the License.
*/
-
package org.apache.hadoop.yarn.server.federation.cache;
-import org.apache.hadoop.classification.VisibleForTesting;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
@@ -26,31 +26,28 @@ import
org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo
(hadoop) branch trunk updated: Revert "Bump org.apache.derby:derby in /hadoop-project (#6816)" (#6841)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new be28467374bf Revert "Bump org.apache.derby:derby in /hadoop-project (#6816)" (#6841) be28467374bf is described below commit be28467374bfc97b731eda59cccefcaac0fa79b8 Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Tue May 21 08:46:14 2024 +0800 Revert "Bump org.apache.derby:derby in /hadoop-project (#6816)" (#6841) This reverts commit b5a90d9500f11aabd1f01c7e905b994962ea0424. --- hadoop-project/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 58576b428799..c795b41340f6 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -135,7 +135,7 @@ 3.8.2 1.1.1 4.0.3 -10.17.1.0 +10.14.2.0 6.2.1.jre7 4.11.0 1.6.20 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch revert-6816-dependabot/maven/hadoop-project/org.apache.derby-derby-10.17.1.0 created (now 78c4077a7d56)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to branch revert-6816-dependabot/maven/hadoop-project/org.apache.derby-derby-10.17.1.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git at 78c4077a7d56 Revert "Bump org.apache.derby:derby in /hadoop-project (#6816)" This branch includes the following new commits: new 78c4077a7d56 Revert "Bump org.apache.derby:derby in /hadoop-project (#6816)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) 01/01: Revert "Bump org.apache.derby:derby in /hadoop-project (#6816)"
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch revert-6816-dependabot/maven/hadoop-project/org.apache.derby-derby-10.17.1.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git commit 78c4077a7d56780e91f3b0510abca4c57a6cfc7b Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Tue May 21 08:44:06 2024 +0800 Revert "Bump org.apache.derby:derby in /hadoop-project (#6816)" This reverts commit b5a90d9500f11aabd1f01c7e905b994962ea0424. --- hadoop-project/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 58576b428799..c795b41340f6 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -135,7 +135,7 @@ 3.8.2 1.1.1 4.0.3 -10.17.1.0 +10.14.2.0 6.2.1.jre7 4.11.0 1.6.20 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (2e77b7b02ca9 -> cab0f4c9ec04)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 2e77b7b02ca9 [HADOOP-18786] Use CDN instead of ASF archive (#5789) add cab0f4c9ec04 HDFS-17520. [BugFix] TestDFSAdmin.testAllDatanodesReconfig and TestDFSAdmin.testDecommissionDataNodesReconfig failed (#6812) Contributed by Zengqiang Xu. No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java | 11 +-- 1 file changed, 5 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: Bump org.apache.derby:derby in /hadoop-project (#6816)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new b5a90d9500f1 Bump org.apache.derby:derby in /hadoop-project (#6816) b5a90d9500f1 is described below commit b5a90d9500f11aabd1f01c7e905b994962ea0424 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon May 13 12:47:31 2024 +0800 Bump org.apache.derby:derby in /hadoop-project (#6816) Bumps org.apache.derby:derby from 10.14.2.0 to 10.17.1.0. --- updated-dependencies: - dependency-name: org.apache.derby:derby dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- hadoop-project/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 35010750ce83..945360a9446e 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -135,7 +135,7 @@ 3.8.2 1.1.1 4.0.3 -10.14.2.0 +10.17.1.0 6.2.1.jre7 4.11.0 1.6.20 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: YARN-11191. Fix potentional deadlock in GlobalScheduler refreshQueues. (#6768) Contributed by Tamas Domok.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 89ebb97b0a22 YARN-11191. Fix potentional deadlock in GlobalScheduler
refreshQueues. (#6768) Contributed by Tamas Domok.
89ebb97b0a22 is described below
commit 89ebb97b0a224fc967f5fbf3c4d61f8415635008
Author: Tamas Domok
AuthorDate: Sun Apr 28 02:07:24 2024 +0200
YARN-11191. Fix potentional deadlock in GlobalScheduler refreshQueues.
(#6768) Contributed by Tamas Domok.
Signed-off-by: Shilun Fan
---
.../scheduler/capacity/AbstractLeafQueue.java | 5 ++
.../scheduler/capacity/AbstractParentQueue.java| 13
.../scheduler/capacity/CSQueue.java| 6 ++
.../capacity/preemption/PreemptionManager.java | 6 +-
.../scheduler/capacity/TestCapacityScheduler.java | 76 ++
5 files changed, 104 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractLeafQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractLeafQueue.java
index 280d3d182fb2..4388489a3b94 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractLeafQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractLeafQueue.java
@@ -369,6 +369,11 @@ public class AbstractLeafQueue extends AbstractCSQueue {
return null;
}
+ @Override
+ public List getChildQueuesByTryLock() {
+return null;
+ }
+
/**
* Set user limit.
* @param userLimit new user limit
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractParentQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractParentQueue.java
index 50516dd2bc5f..4ab8cdf8b36e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractParentQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractParentQueue.java
@@ -27,6 +27,7 @@ import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.locks.LockSupport;
import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableList;
import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableMap;
@@ -1348,6 +1349,18 @@ public abstract class AbstractParentQueue extends
AbstractCSQueue {
}
+ @Override
+ public List getChildQueuesByTryLock() {
+try {
+ while (!readLock.tryLock()){
+LockSupport.parkNanos(1);
+ }
+ return new ArrayList<>(childQueues);
+} finally {
+ readLock.unlock();
+}
+ }
+
@Override
public void recoverContainer(Resource clusterResource,
SchedulerApplicationAttempt attempt, RMContainer rmContainer) {
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
index 72f3cd16fe82..a8ee15303f8a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
@@ -175,6 +175,12 @@ public interface CSQueue extends SchedulerQueue {
* @return child queues
*/
public List getChildQueues();
+
+ /**
+ * Get child queues By tryLock.
+ * @return child queues
+ */
+ List getChildQueuesByTryLock();
/**
* Check if the user has permission to perform the operation
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoo
(hadoop) branch trunk updated: HDFS-17367. Add PercentUsed for Different StorageTypes in JMX (#6735) Contributed by Hualong Zhang.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new daafc8a0b849 HDFS-17367. Add PercentUsed for Different StorageTypes in
JMX (#6735) Contributed by Hualong Zhang.
daafc8a0b849 is described below
commit daafc8a0b849ffdf851c6a618684656925f1df76
Author: zhtttylz
AuthorDate: Sat Apr 27 20:36:11 2024 +0800
HDFS-17367. Add PercentUsed for Different StorageTypes in JMX (#6735)
Contributed by Hualong Zhang.
Signed-off-by: Shilun Fan
---
.../hadoop-common/src/site/markdown/Metrics.md | 9
.../server/blockmanagement/StorageTypeStats.java | 19
.../blockmanagement/TestBlockStatsMXBean.java | 26 ++
3 files changed, 54 insertions(+)
diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
b/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
index b75523b08a0e..aaead837102e 100644
--- a/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
+++ b/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
@@ -326,6 +326,15 @@ Each metrics record contains tags such as HAState and
Hostname as additional inf
| `FSN(Read/Write)LockOverallNanosAvgTime` | Average time of holding the lock
by all operations in nanoseconds |
| `PendingSPSPaths` | The number of paths to be processed by storage policy
satisfier |
+BlockManager
+-
+
+The metrics present statistics from the BlockManager's perspective.
+
+| Name | Description
|
+|:
|:|
+| `StorageTypeStats` | key represents different StorageTypes, and value
represents the detailed storage information corresponding to each StorageType. |
+
JournalNode
---
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/StorageTypeStats.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/StorageTypeStats.java
index fecbe6a81240..83f18bf7804d 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/StorageTypeStats.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/StorageTypeStats.java
@@ -24,6 +24,7 @@ import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.hdfs.DFSUtilClient;
/**
* Statistics per StorageType.
@@ -107,6 +108,24 @@ public class StorageTypeStats {
return blockPoolUsed;
}
+ public float getPercentUsed() {
+long used = getCapacityUsed();
+long total = getCapacityTotal();
+return DFSUtilClient.getPercentUsed(used, total);
+ }
+
+ public float getPercentBlockPoolUsed() {
+long poolUsed = getBlockPoolUsed();
+long total = getCapacityTotal();
+return DFSUtilClient.getPercentUsed(poolUsed, total);
+ }
+
+ public float getPercentRemaining() {
+long remaining = getCapacityRemaining();
+long total = getCapacityTotal();
+return DFSUtilClient.getPercentUsed(remaining, total);
+ }
+
public int getNodesInService() {
return nodesInService;
}
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
index 076a64080860..5753de7c83fd 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
@@ -291,4 +291,30 @@ public class TestBlockStatsMXBean {
5000);
IOUtils.closeStreams(hotSpFileStream, coldSpFileStream);
}
+
+ @Test
+ public void testStorageTypePercentJMX() throws Exception {
+URL baseUrl = new URL(cluster.getHttpUri(0));
+String result = readOutput(new URL(baseUrl, "/jmx"));
+
+Map stat = (Map) JSON.parse(result);
+Object[] beans = (Object[]) stat.get("beans");
+Map blockStats = null;
+for (Object bean : beans) {
+ Map map = (Map) bean;
+ if (map.get("name").equals("Hadoop:service=NameNode,name=BlockStats")) {
+blockStats = map;
+ }
+}
+assertNotNull(blockStats);
+Object[] storageTypeStatsList =
+(Ob
(hadoop) branch branch-3.3 updated: YARN-11191. Fix potentional deadlock in GlobalScheduler refreshQueues (#6732) (#6769) Contributed by Tamas Domok
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 75ad60ba99bd YARN-11191. Fix potentional deadlock in GlobalScheduler
refreshQueues (#6732) (#6769) Contributed by Tamas Domok
75ad60ba99bd is described below
commit 75ad60ba99bdb4bd044879af20cab4f368866613
Author: Tamas Domok
AuthorDate: Sat Apr 27 14:32:28 2024 +0200
YARN-11191. Fix potentional deadlock in GlobalScheduler refreshQueues
(#6732) (#6769) Contributed by Tamas Domok
Signed-off-by: Shilun Fan
---
.../scheduler/capacity/CSQueue.java| 6 ++
.../scheduler/capacity/LeafQueue.java | 7 +-
.../scheduler/capacity/ParentQueue.java| 13
.../capacity/preemption/PreemptionManager.java | 6 +-
.../scheduler/capacity/TestCapacityScheduler.java | 76 ++
5 files changed, 105 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
index 77d8fa9c7ce6..0f9abef7663b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
@@ -157,6 +157,12 @@ public interface CSQueue extends SchedulerQueue {
* @return child queues
*/
public List getChildQueues();
+
+ /**
+ * Get child queues By tryLock.
+ * @return child queues
+ */
+ List getChildQueuesByTryLock();
/**
* Check if the user has permission to perform the operation
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
index ccf33632efd7..c0423a015c2d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
@@ -378,7 +378,12 @@ public class LeafQueue extends AbstractCSQueue {
public List getChildQueues() {
return null;
}
-
+
+ @Override
+ public List getChildQueuesByTryLock() {
+return null;
+ }
+
/**
* Set user limit - used only for testing.
* @param userLimit new user limit
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
index 1f31d8b0a0da..7c7ce1de0bab 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
@@ -26,6 +26,7 @@ import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.concurrent.locks.LockSupport;
import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
@@ -1160,6 +1161,18 @@ public class ParentQueue extends AbstractCSQueue {
}
}
+
+ @Override
+ public List getChildQueuesByTryLock() {
+try {
+ while (!readLock.tryLock()) {
+LockSupport.parkNanos(1);
+ }
+ return new ArrayList<>(childQueues);
+} finally {
+ readLock.unlock();
+}
+ }
@Override
public void recoverContainer(Resource clusterResource,
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/preemption/PreemptionManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-
(hadoop) branch trunk updated: HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#6664) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 88ad7db80de6 HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to
3.2.5. (#6664) Contributed by Shilun Fan.
88ad7db80de6 is described below
commit 88ad7db80de6f5e2d6185d5c397b32218d6c20ea
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Sat Apr 27 20:30:21 2024 +0800
HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#6664)
Contributed by Shilun Fan.
Reviewed-by: Steve Loughran
Reviewed-by: Ayush Saxena
Signed-off-by: Shilun Fan
---
hadoop-project/pom.xml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index c2b5fbe9d60d..9c812554c4d0 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -169,7 +169,7 @@
-Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError
-3.0.0-M1
+3.2.5
${maven-surefire-plugin.version}
${maven-surefire-plugin.version}
@@ -2439,6 +2439,7 @@
${env.DYLD_LIBRARY_PATH}:${project.build.directory}/native/target/usr/local/lib:${hadoop.common.build.dir}/native/target/usr/local/lib
4
+ false
false
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: YARN-11684. Fix general contract violation in PriorityQueueComparator. (#6753) Contributed by Tamas Domok.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 0108a94c5b2f YARN-11684. Fix general contract violation in
PriorityQueueComparator. (#6753) Contributed by Tamas Domok.
0108a94c5b2f is described below
commit 0108a94c5b2f7015b3de539d55bb8784e77d1df7
Author: Tamas Domok
AuthorDate: Thu Apr 25 17:52:12 2024 +0200
YARN-11684. Fix general contract violation in PriorityQueueComparator.
(#6753) Contributed by Tamas Domok.
Signed-off-by: Shilun Fan
---
.../PriorityUtilizationQueueOrderingPolicy.java| 78 +--
...TestPriorityUtilizationQueueOrderingPolicy.java | 91 ++
2 files changed, 128 insertions(+), 41 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
index f60208e04849..5da54e1ec6c6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
@@ -20,6 +20,7 @@ package
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy;
import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.yarn.api.records.Priority;
import org.apache.hadoop.yarn.api.records.Resource;
import org.apache.hadoop.yarn.server.resourcemanager.nodelabels
.RMNodeLabelsManager;
@@ -32,7 +33,6 @@ import java.util.ArrayList;
import java.util.Comparator;
import java.util.Iterator;
import java.util.List;
-import java.util.function.Supplier;
import java.util.stream.Collectors;
/**
@@ -54,17 +54,7 @@ import java.util.stream.Collectors;
public class PriorityUtilizationQueueOrderingPolicy
implements QueueOrderingPolicy {
private List queues;
- private boolean respectPriority;
-
- // This makes multiple threads can sort queues at the same time
- // For different partitions.
- private static ThreadLocal partitionToLookAt =
- ThreadLocal.withInitial(new Supplier() {
-@Override
-public String get() {
- return RMNodeLabelsManager.NO_LABEL;
-}
- });
+ private final boolean respectPriority;
/**
* Compare two queues with possibly different priority and assigned capacity,
@@ -101,15 +91,21 @@ public class PriorityUtilizationQueueOrderingPolicy
/**
* Comparator that both looks at priority and utilization
*/
- private class PriorityQueueComparator
+ final private class PriorityQueueComparator
implements Comparator {
+final private String partition;
+
+private PriorityQueueComparator(String partition) {
+ this.partition = partition;
+}
+
@Override
public int compare(PriorityQueueResourcesForSorting q1Sort,
PriorityQueueResourcesForSorting q2Sort) {
- String p = partitionToLookAt.get();
-
- int rc = compareQueueAccessToPartition(q1Sort.queue, q2Sort.queue, p);
+ int rc = compareQueueAccessToPartition(
+ q1Sort.nodeLabelAccessible,
+ q2Sort.nodeLabelAccessible);
if (0 != rc) {
return rc;
}
@@ -133,8 +129,8 @@ public class PriorityUtilizationQueueOrderingPolicy
float used2 = q2Sort.absoluteUsedCapacity;
return compare(q1Sort, q2Sort, used1, used2,
-q1Sort.queue.getPriority().
-getPriority(), q2Sort.queue.getPriority().getPriority());
+q1Sort.priority.
+getPriority(), q2Sort.priority.getPriority());
} else{
// both q1 has positive abs capacity and q2 has positive abs
// capacity
@@ -142,8 +138,8 @@ public class PriorityUtilizationQueueOrderingPolicy
float used2 = q2Sort.usedCapacity;
return compare(q1Sort, q2Sort, used1, used2,
-q1Sort.queue.getPriority().getPriority(),
-q2Sort.queue.getPriority().getPriority());
+q1Sort.priority.getPriority(),
+q2Sort.priority.getPriority());
}
}
@@ -181,8 +177,7 @@ public class PriorityUtilizationQueueOrderingPolicy
return rc;
}
-private int compareQueueAccessToPartition(CSQueue q1, CSQueue q2
(hadoop) branch branch-3.3 updated: YARN-11684. Fix general contract violation in PriorityQueueComparator. (#6754) Contributed by Tamas Domok.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new efc53d87252a YARN-11684. Fix general contract violation in
PriorityQueueComparator. (#6754) Contributed by Tamas Domok.
efc53d87252a is described below
commit efc53d87252af3b2455d4c38df08d8d850d1879f
Author: Tamas Domok
AuthorDate: Tue Apr 23 02:14:03 2024 +0200
YARN-11684. Fix general contract violation in PriorityQueueComparator.
(#6754) Contributed by Tamas Domok.
Signed-off-by: Shilun Fan
---
.../PriorityUtilizationQueueOrderingPolicy.java| 78 +--
...TestPriorityUtilizationQueueOrderingPolicy.java | 91 ++
2 files changed, 128 insertions(+), 41 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
index 90ea082426d2..5b4c103071d3 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
@@ -20,6 +20,7 @@ package
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy;
import
org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.yarn.api.records.Priority;
import org.apache.hadoop.yarn.api.records.Resource;
import org.apache.hadoop.yarn.server.resourcemanager.nodelabels
.RMNodeLabelsManager;
@@ -32,7 +33,6 @@ import java.util.ArrayList;
import java.util.Comparator;
import java.util.Iterator;
import java.util.List;
-import java.util.function.Supplier;
import java.util.stream.Collectors;
/**
@@ -54,17 +54,7 @@ import java.util.stream.Collectors;
public class PriorityUtilizationQueueOrderingPolicy
implements QueueOrderingPolicy {
private List queues;
- private boolean respectPriority;
-
- // This makes multiple threads can sort queues at the same time
- // For different partitions.
- private static ThreadLocal partitionToLookAt =
- ThreadLocal.withInitial(new Supplier() {
-@Override
-public String get() {
- return RMNodeLabelsManager.NO_LABEL;
-}
- });
+ private final boolean respectPriority;
/**
* Compare two queues with possibly different priority and assigned capacity,
@@ -101,15 +91,21 @@ public class PriorityUtilizationQueueOrderingPolicy
/**
* Comparator that both looks at priority and utilization
*/
- private class PriorityQueueComparator
+ final private class PriorityQueueComparator
implements Comparator {
+final private String partition;
+
+private PriorityQueueComparator(String partition) {
+ this.partition = partition;
+}
+
@Override
public int compare(PriorityQueueResourcesForSorting q1Sort,
PriorityQueueResourcesForSorting q2Sort) {
- String p = partitionToLookAt.get();
-
- int rc = compareQueueAccessToPartition(q1Sort.queue, q2Sort.queue, p);
+ int rc = compareQueueAccessToPartition(
+ q1Sort.nodeLabelAccessible,
+ q2Sort.nodeLabelAccessible);
if (0 != rc) {
return rc;
}
@@ -133,8 +129,8 @@ public class PriorityUtilizationQueueOrderingPolicy
float used2 = q2Sort.absoluteUsedCapacity;
return compare(q1Sort, q2Sort, used1, used2,
-q1Sort.queue.getPriority().
-getPriority(), q2Sort.queue.getPriority().getPriority());
+q1Sort.priority.
+getPriority(), q2Sort.priority.getPriority());
} else{
// both q1 has positive abs capacity and q2 has positive abs
// capacity
@@ -142,8 +138,8 @@ public class PriorityUtilizationQueueOrderingPolicy
float used2 = q2Sort.usedCapacity;
return compare(q1Sort, q2Sort, used1, used2,
-q1Sort.queue.getPriority().getPriority(),
-q2Sort.queue.getPriority().getPriority());
+q1Sort.priority.getPriority(),
+q2Sort.priority.getPriority());
}
}
@@ -181,8 +177,7 @@ public class PriorityUtilizationQueueOrderingPolicy
return rc;
}
-private int
(hadoop) branch trunk updated: YARN-11684. Fix general contract violation in PriorityQueueComparator. (#6725) Contributed by Tamas Domok.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new a386ac1f5632 YARN-11684. Fix general contract violation in
PriorityQueueComparator. (#6725) Contributed by Tamas Domok.
a386ac1f5632 is described below
commit a386ac1f5632949bc49a9d09195073ba5f980b5c
Author: Tamas Domok
AuthorDate: Fri Apr 19 02:37:05 2024 +0200
YARN-11684. Fix general contract violation in PriorityQueueComparator.
(#6725) Contributed by Tamas Domok.
Signed-off-by: Shilun Fan
---
.../PriorityUtilizationQueueOrderingPolicy.java| 78 +--
...TestPriorityUtilizationQueueOrderingPolicy.java | 91 ++
2 files changed, 128 insertions(+), 41 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
index f60208e04849..5da54e1ec6c6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java
@@ -20,6 +20,7 @@ package
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy;
import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.yarn.api.records.Priority;
import org.apache.hadoop.yarn.api.records.Resource;
import org.apache.hadoop.yarn.server.resourcemanager.nodelabels
.RMNodeLabelsManager;
@@ -32,7 +33,6 @@ import java.util.ArrayList;
import java.util.Comparator;
import java.util.Iterator;
import java.util.List;
-import java.util.function.Supplier;
import java.util.stream.Collectors;
/**
@@ -54,17 +54,7 @@ import java.util.stream.Collectors;
public class PriorityUtilizationQueueOrderingPolicy
implements QueueOrderingPolicy {
private List queues;
- private boolean respectPriority;
-
- // This makes multiple threads can sort queues at the same time
- // For different partitions.
- private static ThreadLocal partitionToLookAt =
- ThreadLocal.withInitial(new Supplier() {
-@Override
-public String get() {
- return RMNodeLabelsManager.NO_LABEL;
-}
- });
+ private final boolean respectPriority;
/**
* Compare two queues with possibly different priority and assigned capacity,
@@ -101,15 +91,21 @@ public class PriorityUtilizationQueueOrderingPolicy
/**
* Comparator that both looks at priority and utilization
*/
- private class PriorityQueueComparator
+ final private class PriorityQueueComparator
implements Comparator {
+final private String partition;
+
+private PriorityQueueComparator(String partition) {
+ this.partition = partition;
+}
+
@Override
public int compare(PriorityQueueResourcesForSorting q1Sort,
PriorityQueueResourcesForSorting q2Sort) {
- String p = partitionToLookAt.get();
-
- int rc = compareQueueAccessToPartition(q1Sort.queue, q2Sort.queue, p);
+ int rc = compareQueueAccessToPartition(
+ q1Sort.nodeLabelAccessible,
+ q2Sort.nodeLabelAccessible);
if (0 != rc) {
return rc;
}
@@ -133,8 +129,8 @@ public class PriorityUtilizationQueueOrderingPolicy
float used2 = q2Sort.absoluteUsedCapacity;
return compare(q1Sort, q2Sort, used1, used2,
-q1Sort.queue.getPriority().
-getPriority(), q2Sort.queue.getPriority().getPriority());
+q1Sort.priority.
+getPriority(), q2Sort.priority.getPriority());
} else{
// both q1 has positive abs capacity and q2 has positive abs
// capacity
@@ -142,8 +138,8 @@ public class PriorityUtilizationQueueOrderingPolicy
float used2 = q2Sort.usedCapacity;
return compare(q1Sort, q2Sort, used1, used2,
-q1Sort.queue.getPriority().getPriority(),
-q2Sort.queue.getPriority().getPriority());
+q1Sort.priority.getPriority(),
+q2Sort.priority.getPriority());
}
}
@@ -181,8 +177,7 @@ public class PriorityUtilizationQueueOrderingPolicy
return rc;
}
-private int compareQueueAccessToPartition(CSQueue q1, CSQueue q2
(hadoop) branch trunk updated: YARN-11444. Improve YARN md documentation format. (#6711) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 8c378d1ea1d6 YARN-11444. Improve YARN md documentation format. (#6711) Contributed by Shilun Fan. 8c378d1ea1d6 is described below commit 8c378d1ea1d6026511981d6bda6c2b22bed03986 Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Sun Apr 7 20:50:46 2024 +0800 YARN-11444. Improve YARN md documentation format. (#6711) Contributed by Shilun Fan. Reviewed-by: Ayush Saxena Signed-off-by: Shilun Fan --- .../hadoop-yarn-site/src/site/markdown/CapacityScheduler.md | 4 ++-- .../src/site/markdown/DevelopYourOwnDevicePlugin.md | 2 +- .../hadoop-yarn-site/src/site/markdown/DockerContainers.md | 6 +++--- .../hadoop-yarn-site/src/site/markdown/GracefulDecommission.md | 4 ++-- .../hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManager.md | 2 +- .../hadoop-yarn-site/src/site/markdown/NodeManagerCGroupsMemory.md | 2 +- .../hadoop-yarn-site/src/site/markdown/PluggableDeviceFramework.md | 2 +- .../hadoop-yarn-site/src/site/markdown/ReservationSystem.md | 2 +- .../hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceModel.md | 2 +- .../hadoop-yarn-site/src/site/markdown/RuncContainers.md| 2 +- .../hadoop-yarn-site/src/site/markdown/TimelineServer.md| 6 +++--- .../hadoop-yarn/hadoop-yarn-site/src/site/markdown/UsingGpus.md | 2 +- .../hadoop-yarn-site/src/site/markdown/WebServicesIntro.md | 6 +++--- .../hadoop-yarn-site/src/site/markdown/YarnApplicationSecurity.md | 2 +- 14 files changed, 22 insertions(+), 22 deletions(-) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md index 1b94d6ff33e3..2b46c68bd7f3 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md @@ -633,7 +633,7 @@ The following configuration parameters can be configured in yarn-site.xml to con | `yarn.resourcemanager.reservation-system.planfollower.time-step` | *Optional* parameter: the frequency in milliseconds of the `PlanFollower` timer. Long value expected. The default value is *1000*. | -The `ReservationSystem` is integrated with the `CapacityScheduler` queue hierachy and can be configured for any **LeafQueue** currently. The `CapacityScheduler` supports the following parameters to tune the `ReservationSystem`: +The `ReservationSystem` is integrated with the `CapacityScheduler` queue hierarchy and can be configured for any **LeafQueue** currently. The `CapacityScheduler` supports the following parameters to tune the `ReservationSystem`: | Property | Description | |: |: | @@ -879,7 +879,7 @@ Changing queue/scheduler properties and adding/removing queues can be done in tw Remove the queue configurations from the file and run refresh as described above ### Enabling periodic configuration refresh -Enabling queue configuration periodic refresh allows reloading and applying the configuration by editing the *conf/capacity-scheduler.xml* without the necessicity of calling yarn rmadmin -refreshQueues. +Enabling queue configuration periodic refresh allows reloading and applying the configuration by editing the *conf/capacity-scheduler.xml* without the necessity of calling yarn rmadmin -refreshQueues. | Property | Description | |: |: | diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DevelopYourOwnDevicePlugin.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DevelopYourOwnDevicePlugin.md index 0331f7261531..e129c6c2e0bd 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DevelopYourOwnDevicePlugin.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DevelopYourOwnDevicePlugin.md @@ -173,5 +173,5 @@ class and want to give it a try in your Hadoop cluster. Firstly, put the jar file under a directory in Hadooop classpath. -(recommend $HADOOP_COMMOND_HOME/share/hadoop/yarn). Secondly, +(recommend $HADOOP_COMMAND_HOME/share/hadoop/yarn). Secondly, follow the configurations described in [Pluggable Device Framework](./PluggableDeviceFramework.html) and restart YARN. \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md index fe7d1e1ad39e..e512363d022b 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md +++ b/hadoo
(hadoop) branch branch-3.4 updated: HADOOP-19135. Remove Jcache 1.0-alpha. (#6695) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new d02ca966ea31 HADOOP-19135. Remove Jcache 1.0-alpha. (#6695)
Contributed by Shilun Fan.
d02ca966ea31 is described below
commit d02ca966ea3161f8fa80ef0cab8439b6cbc2be6f
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Fri Apr 5 22:09:15 2024 +0800
HADOOP-19135. Remove Jcache 1.0-alpha. (#6695) Contributed by Shilun Fan.
Reviewed-by: Steve Loughran
Signed-off-by: Shilun Fan
---
LICENSE-binary | 2 +-
hadoop-project/pom.xml | 12 ++--
.../hadoop-yarn-server/hadoop-yarn-server-common/pom.xml | 10 ++
3 files changed, 9 insertions(+), 15 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index 182938cfb933..fb910908c0de 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -310,7 +310,6 @@ org.apache.commons:commons-validator:1.6
org.apache.curator:curator-client:5.2.0
org.apache.curator:curator-framework:5.2.0
org.apache.curator:curator-recipes:5.2.0
-org.apache.geronimo.specs:geronimo-jcache_1.0_spec:1.0-alpha-1
org.apache.hbase:hbase-annotations:1.7.1
org.apache.hbase:hbase-client:1.7.1
org.apache.hbase:hbase-common:1.7.1
@@ -507,6 +506,7 @@ com.sun.jersey:jersey-server:1.19.4
com.sun.jersey:jersey-servlet:1.19.4
com.sun.xml.bind:jaxb-impl:2.2.3-1
javax.annotation:javax.annotation-api:1.3.2
+javax.cache:cache-api:1.1.1
javax.servlet:javax.servlet-api:3.1.0
javax.servlet.jsp:jsp-api:2.1
javax.websocket:javax.websocket-api:1.0
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 8ca757650579..702b9d9a1167 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -132,8 +132,8 @@
1.10.0
2.0.3
-1.0-alpha-1
3.8.2
+1.1.1
4.0.3
10.14.2.0
6.2.1.jre7
@@ -1943,11 +1943,6 @@
kerb-core
${kerby.version}
-
- org.apache.geronimo.specs
- geronimo-jcache_1.0_spec
- ${jcache.version}
-
org.ehcache
ehcache
@@ -2078,6 +2073,11 @@
log4j-web
${log4j2.version}
+
+javax.cache
+cache-api
+${cache.api.version}
+
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
index 51f0b0960388..a6f9fd79dd6c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
@@ -124,14 +124,8 @@
leveldbjni-all
- org.apache.geronimo.specs
- geronimo-jcache_1.0_spec
-
-
- org.osgi
- org.osgi.core
-
-
+ javax.cache
+ cache-api
org.ehcache
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: YARN-11663. [Federation] Add Cache Entity Nums Limit. (#6662) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 80662741870a YARN-11663. [Federation] Add Cache Entity Nums Limit.
(#6662) Contributed by Shilun Fan.
80662741870a is described below
commit 80662741870ae93493e0da53651dcd486cf13dd8
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Tue Apr 2 07:47:59 2024 +0800
YARN-11663. [Federation] Add Cache Entity Nums Limit. (#6662) Contributed
by Shilun Fan.
Reviewed-by: Dinesh Chitlangia
Signed-off-by: Shilun Fan
---
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 4 ++
.../src/main/resources/yarn-default.xml| 9
.../federation/cache/FederationGuavaCache.java | 12 -
.../server/federation/cache/FederationJCache.java | 60 +++---
.../utils/TestFederationStateStoreFacade.java | 4 +-
5 files changed, 56 insertions(+), 33 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 0ab4107c1320..650e82d67381 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -4031,6 +4031,10 @@ public class YarnConfiguration extends Configuration {
// 5 minutes
public static final int DEFAULT_FEDERATION_CACHE_TIME_TO_LIVE_SECS = 5 * 60;
+ public static final String FEDERATION_CACHE_ENTITY_NUMS =
+ FEDERATION_PREFIX + "cache-entity.nums";
+ public static final int DEFAULT_FEDERATION_CACHE_ENTITY_NUMS = 1000;
+
public static final String FEDERATION_FLUSH_CACHE_FOR_RM_ADDR =
FEDERATION_PREFIX + "flush-cache-for-rm-addr";
public static final boolean DEFAULT_FEDERATION_FLUSH_CACHE_FOR_RM_ADDR =
true;
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index 41e38f601cbd..6b2d2cd817c6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -3787,6 +3787,15 @@
300
+
+
+ The number of entries in the Federation cache.
+ default is 1000.
+
+yarn.federation.cache-entity.nums
+1000
+
+
The registry base directory for federation.
yarn.federation.registry.base-dir
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationGuavaCache.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationGuavaCache.java
index 5ab0ef77218d..2ba9e2869fe8 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationGuavaCache.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationGuavaCache.java
@@ -27,15 +27,20 @@ import
org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterPolicyConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.util.Map;
import java.util.concurrent.TimeUnit;
public class FederationGuavaCache extends FederationCache {
+ private static final Logger LOG =
LoggerFactory.getLogger(FederationCache.class);
+
private Cache> cache;
private int cacheTimeToLive;
+ private long cacheEntityNums;
private String className = this.getClass().getSimpleName();
@@ -52,6 +57,8 @@ public class FederationGuavaCache extends FederationCache {
// no conflict or pick up a specific one in the future.
cacheTimeToLive =
pConf.getInt(YarnConfiguration.FEDERATION_CACHE_TIME_TO_LIVE_SECS,
YarnConfiguration.DEFAULT_FEDERATION_CACHE_TIME_TO_LIVE_SECS);
+cacheEntityNums =
pConf.getLong(YarnConfiguration.FEDERATION_CACHE_ENTITY_NUMS,
+YarnConfiguration.DEFAULT_FEDERATION_CACHE_ENTITY_NUMS);
if (cacheTimeToLive <= 0) {
isCachingEnabled = false;
return;
@@ -59,8 +66,11 @@ public cl
(hadoop) branch trunk updated: HADOOP-19135. Remove Jcache 1.0-alpha. (#6695) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new a1ae35e69164 HADOOP-19135. Remove Jcache 1.0-alpha. (#6695)
Contributed by Shilun Fan.
a1ae35e69164 is described below
commit a1ae35e69164e42af68bf1d940744ae326ce130a
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Fri Apr 5 22:09:15 2024 +0800
HADOOP-19135. Remove Jcache 1.0-alpha. (#6695) Contributed by Shilun Fan.
Reviewed-by: Steve Loughran
Signed-off-by: Shilun Fan
---
LICENSE-binary | 2 +-
hadoop-project/pom.xml | 12 ++--
.../hadoop-yarn-server/hadoop-yarn-server-common/pom.xml | 10 ++
3 files changed, 9 insertions(+), 15 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index 02869088e1d9..c9577c8737a8 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -310,7 +310,6 @@ org.apache.commons:commons-validator:1.6
org.apache.curator:curator-client:5.2.0
org.apache.curator:curator-framework:5.2.0
org.apache.curator:curator-recipes:5.2.0
-org.apache.geronimo.specs:geronimo-jcache_1.0_spec:1.0-alpha-1
org.apache.hbase:hbase-annotations:1.7.1
org.apache.hbase:hbase-client:1.7.1
org.apache.hbase:hbase-common:1.7.1
@@ -507,6 +506,7 @@ com.sun.jersey:jersey-server:1.19.4
com.sun.jersey:jersey-servlet:1.19.4
com.sun.xml.bind:jaxb-impl:2.2.3-1
javax.annotation:javax.annotation-api:1.3.2
+javax.cache:cache-api:1.1.1
javax.servlet:javax.servlet-api:3.1.0
javax.servlet.jsp:jsp-api:2.1
javax.websocket:javax.websocket-api:1.0
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index a01a31289d4e..67cd8fe58432 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -132,8 +132,8 @@
1.10.0
2.0.3
-1.0-alpha-1
3.8.2
+1.1.1
4.0.3
10.14.2.0
6.2.1.jre7
@@ -1943,11 +1943,6 @@
kerb-core
${kerby.version}
-
- org.apache.geronimo.specs
- geronimo-jcache_1.0_spec
- ${jcache.version}
-
org.ehcache
ehcache
@@ -2078,6 +2073,11 @@
log4j-web
${log4j2.version}
+
+javax.cache
+cache-api
+${cache.api.version}
+
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
index e67f53e21376..1f762d31800d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
@@ -124,14 +124,8 @@
leveldbjni-all
- org.apache.geronimo.specs
- geronimo-jcache_1.0_spec
-
-
- org.osgi
- org.osgi.core
-
-
+ javax.cache
+ cache-api
org.ehcache
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19124. Update org.ehcache from 3.3.1 to 3.8.2. (#6665) (#6705) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new e0fcb173964c HADOOP-19124. Update org.ehcache from 3.3.1 to 3.8.2. (#6665) (#6705) Contributed by Shilun Fan. e0fcb173964c is described below commit e0fcb173964c85c1ab79e6b3a32eca02c3cb4221 Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Fri Apr 5 22:07:28 2024 +0800 HADOOP-19124. Update org.ehcache from 3.3.1 to 3.8.2. (#6665) (#6705) Contributed by Shilun Fan. Reviewed-by: Steve Loughran Signed-off-by: Shilun Fan --- LICENSE-binary | 2 +- hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml | 1 + .../src/test/resources/ensure-jars-have-correct-contents.sh | 6 ++ hadoop-project/pom.xml | 2 +- 4 files changed, 9 insertions(+), 2 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index b2a635a18757..182938cfb933 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -356,7 +356,7 @@ org.eclipse.jetty:jetty-webapp:9.4.53.v20231009 org.eclipse.jetty:jetty-xml:9.4.53.v20231009 org.eclipse.jetty.websocket:javax-websocket-client-impl:9.4.53.v20231009 org.eclipse.jetty.websocket:javax-websocket-server-impl:9.4.53.v20231009 -org.ehcache:ehcache:3.3.1 +org.ehcache:ehcache:3.8.2 org.ini4j:ini4j:0.5.4 org.lz4:lz4-java:1.7.1 org.objenesis:objenesis:2.6 diff --git a/hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml b/hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml index a4164ec0b4bc..ebacc249badb 100644 --- a/hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml +++ b/hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml @@ -100,6 +100,7 @@ org.bouncycastle:* org.xerial.snappy:* +org.ehcache:* diff --git a/hadoop-client-modules/hadoop-client-check-test-invariants/src/test/resources/ensure-jars-have-correct-contents.sh b/hadoop-client-modules/hadoop-client-check-test-invariants/src/test/resources/ensure-jars-have-correct-contents.sh index 0dbfefbf4f16..ca68608fd6a4 100644 --- a/hadoop-client-modules/hadoop-client-check-test-invariants/src/test/resources/ensure-jars-have-correct-contents.sh +++ b/hadoop-client-modules/hadoop-client-check-test-invariants/src/test/resources/ensure-jars-have-correct-contents.sh @@ -58,6 +58,12 @@ allowed_expr+="|^org.apache.hadoop.application-classloader.properties$" allowed_expr+="|^java.policy$" # * Used by javax.annotation allowed_expr+="|^jndi.properties$" +# * Used by ehcache +allowed_expr+="|^ehcache-107-ext.xsd$" +allowed_expr+="|^ehcache-multi.xsd$" +allowed_expr+="|^.gitkeep$" +allowed_expr+="|^OSGI-INF.*$" +allowed_expr+="|^javax.*$" allowed_expr+=")" declare -i bad_artifacts=0 diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index aefeaa91baf1..8ca757650579 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -133,7 +133,7 @@ 2.0.3 1.0-alpha-1 -3.3.1 +3.8.2 4.0.3 10.14.2.0 6.2.1.jre7 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.3 updated: HADOOP-19115. Upgrade to nimbus-jose-jwt 9.37.2 due to CVE-2023-52428. (#6637) (#6689) Contributed by PJ Fanning.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.3 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.3 by this push: new 7cc64ce7a7c9 HADOOP-19115. Upgrade to nimbus-jose-jwt 9.37.2 due to CVE-2023-52428. (#6637) (#6689) Contributed by PJ Fanning. 7cc64ce7a7c9 is described below commit 7cc64ce7a7c9c37cd9490fc9060e64d73e72b42b Author: PJ Fanning AuthorDate: Tue Apr 2 01:50:33 2024 +0200 HADOOP-19115. Upgrade to nimbus-jose-jwt 9.37.2 due to CVE-2023-52428. (#6637) (#6689) Contributed by PJ Fanning. Signed-off-by: Shilun Fan --- LICENSE-binary | 2 +- hadoop-project/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 31d744b19d5f..43866bce657f 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -242,7 +242,7 @@ com.google.guava:guava:jar:30.1.1-jre com.google.guava:listenablefuture:.0-empty-to-avoid-conflict-with-guava com.google.j2objc:j2objc-annotations:1.3 com.microsoft.azure:azure-storage:7.0.1 -com.nimbusds:nimbus-jose-jwt:9.31 +com.nimbusds:nimbus-jose-jwt:9.37.2 com.yammer.metrics:metrics-core:2.2.0 com.zaxxer:HikariCP-java7:2.4.12 commons-beanutils:commons-beanutils:1.9.4 diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 5b910adfecb4..f9158e833fc3 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -217,7 +217,7 @@ 8.8.2 1.1.3.Final 5.4.0 -9.31 +9.37.2 v12.22.1 v1.22.5 1.10.13 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: YARN-11663. [Federation] Add Cache Entity Nums Limit. (#6662) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 5f3eb446f768 YARN-11663. [Federation] Add Cache Entity Nums Limit.
(#6662) Contributed by Shilun Fan.
5f3eb446f768 is described below
commit 5f3eb446f76803ed654597bdf88c70268d8d9623
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Tue Apr 2 07:47:59 2024 +0800
YARN-11663. [Federation] Add Cache Entity Nums Limit. (#6662) Contributed
by Shilun Fan.
Reviewed-by: Dinesh Chitlangia
Signed-off-by: Shilun Fan
---
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 4 ++
.../src/main/resources/yarn-default.xml| 9
.../federation/cache/FederationGuavaCache.java | 12 -
.../server/federation/cache/FederationJCache.java | 60 +++---
.../utils/TestFederationStateStoreFacade.java | 4 +-
5 files changed, 56 insertions(+), 33 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 0ab4107c1320..650e82d67381 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -4031,6 +4031,10 @@ public class YarnConfiguration extends Configuration {
// 5 minutes
public static final int DEFAULT_FEDERATION_CACHE_TIME_TO_LIVE_SECS = 5 * 60;
+ public static final String FEDERATION_CACHE_ENTITY_NUMS =
+ FEDERATION_PREFIX + "cache-entity.nums";
+ public static final int DEFAULT_FEDERATION_CACHE_ENTITY_NUMS = 1000;
+
public static final String FEDERATION_FLUSH_CACHE_FOR_RM_ADDR =
FEDERATION_PREFIX + "flush-cache-for-rm-addr";
public static final boolean DEFAULT_FEDERATION_FLUSH_CACHE_FOR_RM_ADDR =
true;
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index 41e38f601cbd..6b2d2cd817c6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -3787,6 +3787,15 @@
300
+
+
+ The number of entries in the Federation cache.
+ default is 1000.
+
+yarn.federation.cache-entity.nums
+1000
+
+
The registry base directory for federation.
yarn.federation.registry.base-dir
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationGuavaCache.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationGuavaCache.java
index 5ab0ef77218d..2ba9e2869fe8 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationGuavaCache.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationGuavaCache.java
@@ -27,15 +27,20 @@ import
org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterPolicyConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.util.Map;
import java.util.concurrent.TimeUnit;
public class FederationGuavaCache extends FederationCache {
+ private static final Logger LOG =
LoggerFactory.getLogger(FederationCache.class);
+
private Cache> cache;
private int cacheTimeToLive;
+ private long cacheEntityNums;
private String className = this.getClass().getSimpleName();
@@ -52,6 +57,8 @@ public class FederationGuavaCache extends FederationCache {
// no conflict or pick up a specific one in the future.
cacheTimeToLive =
pConf.getInt(YarnConfiguration.FEDERATION_CACHE_TIME_TO_LIVE_SECS,
YarnConfiguration.DEFAULT_FEDERATION_CACHE_TIME_TO_LIVE_SECS);
+cacheEntityNums =
pConf.getLong(YarnConfiguration.FEDERATION_CACHE_ENTITY_NUMS,
+YarnConfiguration.DEFAULT_FEDERATION_CACHE_ENTITY_NUMS);
if (cacheTimeToLive <= 0) {
isCachingEnabled = false;
return;
@@ -59,8 +66,11 @@ public class FederationGu
(hadoop) branch trunk updated: YARN-11668. Fix RM crash for potential concurrent modification exception when updating node attributes (#6681) Contributed by Junfan Zhang.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new f1f2abe64183 YARN-11668. Fix RM crash for potential concurrent
modification exception when updating node attributes (#6681) Contributed by
Junfan Zhang.
f1f2abe64183 is described below
commit f1f2abe6418308a8124cdb12aa98bd35168ba379
Author: Junfan Zhang
AuthorDate: Fri Mar 29 09:45:16 2024 +0800
YARN-11668. Fix RM crash for potential concurrent modification exception
when updating node attributes (#6681) Contributed by Junfan Zhang.
Reviewed-by: Dinesh Chitlangia
Signed-off-by: Shilun Fan
---
.../server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java
index 161e104e75b0..2b7561367033 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeAttributesManagerImpl.java
@@ -613,7 +613,7 @@ public class NodeAttributesManagerImpl extends
NodeAttributesManager {
}
public Host(String hostName) {
- this(hostName, new HashMap());
+ this(hostName, new ConcurrentHashMap());
}
public Host(String hostName,
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19115. Upgrade to nimbus-jose-jwt 9.37.2 due to CVE-2023-52428. (#6637) (#6682) Contributed by PJ Fanning.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new 193c72c1e410 HADOOP-19115. Upgrade to nimbus-jose-jwt 9.37.2 due to CVE-2023-52428. (#6637) (#6682) Contributed by PJ Fanning. 193c72c1e410 is described below commit 193c72c1e4102e7a3f09b5003ef0373f11266a83 Author: PJ Fanning AuthorDate: Fri Mar 29 02:43:32 2024 +0100 HADOOP-19115. Upgrade to nimbus-jose-jwt 9.37.2 due to CVE-2023-52428. (#6637) (#6682) Contributed by PJ Fanning. Reviewed-by: Dinesh Chitlangia Signed-off-by: Shilun Fan --- LICENSE-binary | 2 +- hadoop-project/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index fcbff70a19fe..3c02de55effc 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -241,7 +241,7 @@ com.google.guava:guava:20.0 com.google.guava:guava:27.0-jre com.google.guava:listenablefuture:.0-empty-to-avoid-conflict-with-guava com.microsoft.azure:azure-storage:7.0.0 -com.nimbusds:nimbus-jose-jwt:9.31 +com.nimbusds:nimbus-jose-jwt:9.37.2 com.zaxxer:HikariCP:4.0.3 commons-beanutils:commons-beanutils:1.9.4 commons-cli:commons-cli:1.5.0 diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 7f91be82e38b..77b71a884225 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -216,7 +216,7 @@ 1.1.3.Final 1.0.2 5.4.0 -9.31 +9.37.2 v12.22.1 v1.22.5 1.10.13 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HDFS-17429. Fixing wrong log file name in datatransfer Sender.java (#6670) Contributed by Zhongkun Wu.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 50370769cd56 HDFS-17429. Fixing wrong log file name in datatransfer
Sender.java (#6670) Contributed by Zhongkun Wu.
50370769cd56 is described below
commit 50370769cd56f37d3bf7698d65e9044c264e7d06
Author: Alex
AuthorDate: Wed Mar 27 06:59:15 2024 +0800
HDFS-17429. Fixing wrong log file name in datatransfer Sender.java (#6670)
Contributed by Zhongkun Wu.
Reviewed-by: Ayush Saxena
Signed-off-by: Shilun Fan
---
.../java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java| 5 +
1 file changed, 5 insertions(+)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
index 3d81a62993ef..86c6513745ea 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java
@@ -57,11 +57,16 @@ import org.apache.hadoop.tracing.Tracer;
import org.apache.hadoop.tracing.TraceUtils;
import org.apache.hadoop.thirdparty.protobuf.Message;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
/** Sender */
@InterfaceAudience.Private
@InterfaceStability.Evolving
public class Sender implements DataTransferProtocol {
+ private static final Logger LOG = LoggerFactory.getLogger(Sender.class);
+
private final DataOutputStream out;
/** Create a sender for DataTransferProtocol with a output stream. */
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (a60b5e2de307 -> 55dca911cc67)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from a60b5e2de307 MAPREDUCE-7469. NNBench createControlFiles should use thread pool to improve performance. (#6463) Contributed by liuguanghua. add 55dca911cc67 HADOOP-19052.Hadoop use Shell command to get the count of the hard link which takes a lot of time (#6587) Contributed by liangyu. No new revisions were added by this update. Summary of changes: .../main/java/org/apache/hadoop/fs/HardLink.java | 28 ++ .../java/org/apache/hadoop/fs/TestHardLink.java| 10 2 files changed, 38 insertions(+) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch HDFS-17384 updated: HADOOP-19090. Use protobuf-java 3.23.4. (#6593). Contributed by PJ Fanning. (#6663)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch HDFS-17384
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/HDFS-17384 by this push:
new d32bc502fca2 HADOOP-19090. Use protobuf-java 3.23.4. (#6593).
Contributed by PJ Fanning. (#6663)
d32bc502fca2 is described below
commit d32bc502fca2010bcadbcc295ab3cfc7cce01ea1
Author: ZanderXu
AuthorDate: Sun Mar 24 08:07:21 2024 +0800
HADOOP-19090. Use protobuf-java 3.23.4. (#6593). Contributed by PJ Fanning.
(#6663)
Signed-off-by: Shilun Fan
---
hadoop-common-project/hadoop-common/pom.xml | 2 +-
hadoop-project/pom.xml | 6 +++---
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml | 6 +++---
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index 392a26c2ff9a..3b03fc6aadb7 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -40,7 +40,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_21
+ hadoop-shaded-protobuf_3_23
org.apache.hadoop
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 3086e7496998..4aecf3d733dc 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -93,11 +93,11 @@
${common.protobuf2.scope}
-3.21.12
+3.23.4
${env.HADOOP_PROTOC_PATH}
1.2.0
-
${hadoop-thirdparty.version}
+
1.3.0-SNAPSHOT
${hadoop-thirdparty.version}
org.apache.hadoop.thirdparty
${hadoop-thirdparty-shaded-prefix}.protobuf
@@ -250,7 +250,7 @@
org.apache.hadoop.thirdparty
-hadoop-shaded-protobuf_3_21
+hadoop-shaded-protobuf_3_23
${hadoop-thirdparty-protobuf.version}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
index 8dbec9ed4e9d..102f90087e1a 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
@@ -51,7 +51,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_21
+ hadoop-shaded-protobuf_3_23
@@ -69,7 +69,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_21
+ hadoop-shaded-protobuf_3_23
@@ -80,7 +80,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_21
+ hadoop-shaded-protobuf_3_23
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch HDFS-17384 updated: HDFS-17422. Enhance the stability of the unit test TestDFSAdmin (#6621). Contributed by lei w and Hualong Zhang. (#6657)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch HDFS-17384
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/HDFS-17384 by this push:
new 8a3eeca112f9 HDFS-17422. Enhance the stability of the unit test
TestDFSAdmin (#6621). Contributed by lei w and Hualong Zhang. (#6657)
8a3eeca112f9 is described below
commit 8a3eeca112f9286c0dad40de07ae54ee0b632c95
Author: ZanderXu
AuthorDate: Sat Mar 23 07:32:21 2024 +0800
HDFS-17422. Enhance the stability of the unit test TestDFSAdmin (#6621).
Contributed by lei w and Hualong Zhang. (#6657)
Reviewed-by: Dinesh Chitlangia
Signed-off-by: Shilun Fan
---
.../test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
index 451a419aa140..392866e86ea2 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
@@ -1353,8 +1353,14 @@ public class TestDFSAdmin {
decommissioningNode1.getIpcPort();
String node2Addr = decommissioningNode2.getIpAddr() + ":" +
decommissioningNode2.getIpcPort();
- assertTrue(outsForFinishReconf.get(0).contains(node1Addr)
- && outsForFinishReconf.get(0).contains(node2Addr));
+ int finishedReconfCount = 0;
+ for (String outMessage : outsForFinishReconf) {
+finishedReconfCount = outMessage.contains(node1Addr) ?
+finishedReconfCount + 1 : finishedReconfCount + 0;
+finishedReconfCount = outMessage.contains(node2Addr) ?
+finishedReconfCount + 1 : finishedReconfCount + 0;
+ }
+ assertTrue(finishedReconfCount == 2);
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (8cd4704e0a1a -> a60b5e2de307)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 8cd4704e0a1a HDFS-17430. RecoveringBlock will skip no live replicas when get block recovery command. (#6635) add a60b5e2de307 MAPREDUCE-7469. NNBench createControlFiles should use thread pool to improve performance. (#6463) Contributed by liuguanghua. No new revisions were added by this update. Summary of changes: .../test/java/org/apache/hadoop/hdfs/NNBench.java | 45 +- .../java/org/apache/hadoop/hdfs/TestNNBench.java | 20 ++ 2 files changed, 63 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-site) branch asf-site updated: Add Menu List 3.3.6. (#56) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hadoop-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 0f91b026b6 Add Menu List 3.3.6. (#56) Contributed by Shilun Fan. 0f91b026b6 is described below commit 0f91b026b6016a8aab95a7584214173f71d4b5cb Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Sat Mar 23 05:42:11 2024 +0800 Add Menu List 3.3.6. (#56) Contributed by Shilun Fan. Reviewed-by: Dinesh Chitlangia Signed-off-by: Shilun Fan --- content/bylaws.html| 2 ++ content/categories.html| 2 ++ content/committer_criteria.html| 2 ++ content/cve_list.html | 2 ++ content/description.html | 2 ++ content/index.html | 2 ++ content/issue_tracking.html| 2 ++ content/mailing_lists.html | 2 ++ content/modules.html | 2 ++ content/news.html | 2 ++ content/news/2008-07-xx-terabyte-sort.html | 2 ++ content/news/2008-11-xx-apachecon.html | 2 ++ content/news/2009-03-xx-apachecon-eu.html | 2 ++ content/news/2009-07-xx-subprojects.html | 2 ++ content/news/2010-05-xx-avro-hbase.html| 2 ++ content/news/2010-10-xx-hive-pig.html | 2 ++ content/news/2011-01-xx-zookeeper.html | 2 ++ content/news/2011-03-xx-award.html | 2 ++ content/news/2018-10-01-ozone-0.2.1-alpha.html | 2 ++ content/news/2018-11-22-ozone-0.3.0-alpha.html | 2 ++ content/news/2019-05-07-ozone-0.4.0-alpha.html | 2 ++ content/news/2019-10-13-ozone-0.4.1-alpha.html | 2 ++ content/news/2020-03-24-ozone-0.5.0-beta.html | 2 ++ content/news/2020-09-02-ozone-1.0.0.html | 2 ++ content/news/2021-04-17-ozone-1.1.0.html | 2 ++ content/news/2021-12-17-log4shell.html | 2 ++ content/news/page/2.html | 2 ++ content/privacy_policy.html| 2 ++ content/related.html | 2 ++ content/release.html | 2 ++ content/release/0.14.1.html| 2 ++ content/release/0.14.3.html| 2 ++ content/release/0.14.4.html| 2 ++ content/release/0.15.0.html| 2 ++ content/release/0.15.1.html| 2 ++ content/release/0.15.2.html| 2 ++ content/release/0.15.3.html| 2 ++ content/release/0.16.0.html| 2 ++ content/release/0.16.1.html| 2 ++ content/release/0.16.2.html| 2 ++ content/release/0.16.3.html| 2 ++ content/release/0.16.4.html| 2 ++ content/release/0.17.0.html| 2 ++ content/release/0.17.1.html| 2 ++ content/release/0.17.2.html| 2 ++ content/release/0.18.0.html| 2 ++ content/release/0.18.1.html| 2 ++ content/release/0.18.2.html| 2 ++ content/release/0.18.3.html| 2 ++ content/release/0.19.0.html| 2 ++ content/release/0.19.1.html| 2 ++ content/release/0.19.2.html| 2 ++ content/release/0.20.0.html| 2 ++ content/release/0.20.1.html| 2 ++ content/release/0.20.2.html| 2 ++ content/release/0.20.203.0.html| 2 ++ content/release/0.20.204.0.html| 2 ++ content/release/0.20.205.0.html| 2 ++ content/release/0.21.0.html| 2 ++ content/release/0.22.0.html| 2 ++ content/release/0.23.0.html| 2 ++ content/release/0.23.1.html| 2 ++ content/release/0.23.10.html | 2 ++ content/release/0.23.11.html | 2 ++ content/release/0.23.3.html| 2 ++ content/release/0.23.4.html| 2 ++ content/release/0.23.5.html| 2 ++ content/release/0.23.6.html| 2 ++ content/release/0.23.7.html| 2 ++ content/release/0.23.8.html| 2 ++ content/release/0.23.9.html| 2 ++ content/release/1.0.0.html | 2 ++ content/release/1.0.1.html | 2 ++ content/release/1.0.2.html | 2 ++ content/release/1.0.3.html | 2 ++ content/release/1.0.4.html | 2 ++ content/release/1.1.0.html | 2 ++ content/release/1.1.
(hadoop-site) branch asf-site updated: Change Stable to hadoop 3.3.6. (#55) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hadoop-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 25658da254 Change Stable to hadoop 3.3.6. (#55) Contributed by Shilun Fan. 25658da254 is described below commit 25658da2544700b00aa2c1bb45c5299761fa62ee Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Fri Mar 22 19:53:40 2024 +0800 Change Stable to hadoop 3.3.6. (#55) Contributed by Shilun Fan. Reviewed-by: He Xiaoqiao Signed-off-by: Shilun Fan --- content/docs/stable3 | 2 +- src/release/3.3.6.md | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/content/docs/stable3 b/content/docs/stable3 index 902332bd22..b7086a22f6 12 --- a/content/docs/stable3 +++ b/content/docs/stable3 @@ -1 +1 @@ -r3.4.0 \ No newline at end of file +r3.3.6 \ No newline at end of file diff --git a/src/release/3.3.6.md b/src/release/3.3.6.md index 346e2bffbd..76756cb82e 100644 --- a/src/release/3.3.6.md +++ b/src/release/3.3.6.md @@ -1,6 +1,7 @@ --- title: Release 3.3.6 available date: 2023-06-23 +linked: true ---
(hadoop-site) branch asf-site updated: Update Doap Release 3.4.0. (#54) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hadoop-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 9cf727338b Update Doap Release 3.4.0. (#54) Contributed by Shilun Fan. 9cf727338b is described below commit 9cf727338b065af782f4edf7b877f60607268bee Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Fri Mar 22 19:51:58 2024 +0800 Update Doap Release 3.4.0. (#54) Contributed by Shilun Fan. Reviewed-by: He Xiaoqiao Signed-off-by: Shilun Fan --- content/doap_Hadoop.rdf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/doap_Hadoop.rdf b/content/doap_Hadoop.rdf index 2b25fa687a..96e6644f49 100644 --- a/content/doap_Hadoop.rdf +++ b/content/doap_Hadoop.rdf @@ -37,8 +37,8 @@ Stable Release -2023-06-18 -3.3.6 +2024-03-17 +3.4.0 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) annotated tag rel/release-3.4.0 updated (bd8b77f398f6 -> 3ed4d8c864ff)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to annotated tag rel/release-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git *** WARNING: tag rel/release-3.4.0 was modified! *** from bd8b77f398f6 (commit) to 3ed4d8c864ff (tag) tagging bd8b77f398f626bb7791783192ee7a5dfaeec760 (commit) replaces release-3.4.0-RC2 by Shilun Fan on Sun Mar 17 15:26:43 2024 +0800 - Log - Hadoop 3.4.0 release -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEECm40PGXwzjlsDKJKl4RW45ZNtwEFAmX2mzMACgkQl4RW45ZN twG6Xg/+KUYv69ThqVeeR9f96NmxJuNgxpsfWpmQ628LThAxMBDM+f5I5ZUORF05 Li3HuzpLRnlRc5WEyxPXFiN4BSfp8OFu1nKa5epMN3QE88LSDZRyiG24REtkNna/ SFrtkgucZoEizeRIWCqb6M16EJerk4OV1dRyObZr9SACs4wWsTeI94q8dIQwAjJ8 e6fzokr/0oWV/7E3u4w5y6j73mp025iZnCkbizuCegDYveDJEZrdH/XzrsSLnB1x C0DtgnZXvDr84eDh9WYj/PDVY9M3tZFvDwa0MdfIK7XBqGzcJtAtNSueae0Unc3b sd75b5BTSqxZb4WyeszoW5F1KEyrvfkYCRdS1Fy6uvvxjIm8ot0M0wkPux7ufD+k CYi3D2rxeeCJlCeAzK/44se+CQrib8FoCubRW6l7bnhf0kW0yDXe2aehOJVQ3h9m 1UHnPGWiZbcaTnDIvAfLLzy4sy9+ittR1vr91BY/Sj9A6zYXmcmloY8m3rGIrq03 iZCjbmpb9tQcUaqAboI6prLJKI33Kn6F5+RRm78UA6gQLl1fOOhjpkqXuUdKH8Ti Pi7F41hnhsEVIx7MP4aYoT8ZyywQj4LujoSU+rtzvWZ5s9IMfHxmU3EuyEdhQTDK K4FoKDqOuwG9TQ4Cc8LeO07amcMXvT04gzobCoUm5/yG4863Pgw= =wZ0b -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
svn commit: r67976 - /release/hadoop/common/current
Author: slfan1989 Date: Sun Mar 17 07:07:03 2024 New Revision: 67976 Log: Update current link hadoop-3.4.0 Modified: release/hadoop/common/current Modified: release/hadoop/common/current == --- release/hadoop/common/current (original) +++ release/hadoop/common/current Sun Mar 17 07:07:03 2024 @@ -1 +1 @@ -link hadoop-3.3.6 \ No newline at end of file +link hadoop-3.4.0 \ No newline at end of file - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
svn commit: r67975 - /release/hadoop/common/hadoop-3.4.0/
Author: slfan1989 Date: Sun Mar 17 06:50:01 2024 New Revision: 67975 Log: Publishing the bits for release 3.4.0 Added: release/hadoop/common/hadoop-3.4.0/ - copied from r67974, dev/hadoop/hadoop-3.4.0-RC3/ - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (a325876feca2 -> 783cc3eda0a9)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from a325876feca2 HADOOP-19066. Run FIPS test for valid bucket locations (ADDENDUM) (#6624) add 783cc3eda0a9 YARN-11660. Fix performance regression for SingleConstraintAppPlacementAllocator (#6623) Contributed by Junfan Zhang. No new revisions were added by this update. Summary of changes: .../scheduler/placement/SingleConstraintAppPlacementAllocator.java | 2 ++ 1 file changed, 2 insertions(+) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
svn commit: r67696 [3/5] - /dev/hadoop/hadoop-3.4.0-RC3/
Added: dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md.asc
==
Binary file - no diff available.
Propchange: dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md.asc
--
svn:mime-type = application/pgp-signature
Added: dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md.sha512
==
--- dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md.sha512 (added)
+++ dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md.sha512 Mon Mar 4 09:36:35 2024
@@ -0,0 +1 @@
+SHA512 (CHANGELOG.md) =
1af8f89788be4e1fad827d87aa4056a9c3b1ede1c57d45fd36db1e8c0e838bb244dc1b7e15485d4c5825329c81fb54604c1fc2cb7893efcf63b507a4ab2dfdb1
Added: dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
==
--- dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md (added)
+++ dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md Mon Mar 4 09:36:35 2024
@@ -0,0 +1,652 @@
+
+
+# Apache Hadoop 3.4.0 Release Notes
+
+These release notes cover new developer and user-facing incompatibilities,
important issues, features, and major improvements.
+
+
+---
+
+* [HADOOP-16054](https://issues.apache.org/jira/browse/HADOOP-16054) | *Major*
| **Update Dockerfile to use Bionic**
+
+The build image has been upgraded to Bionic.
+
+
+---
+
+* [HDFS-15281](https://issues.apache.org/jira/browse/HDFS-15281) | *Major* |
**ZKFC ignores dfs.namenode.rpc-bind-host and uses dfs.namenode.rpc-address to
bind to host address**
+
+ZKFC binds host address to "dfs.namenode.servicerpc-bind-host", if configured.
Otherwise, it binds to "dfs.namenode.rpc-bind-host". If neither of those is
configured, ZKFC binds itself to NameNode RPC server address (effectively
"dfs.namenode.rpc-address").
+
+
+---
+
+* [HADOOP-17010](https://issues.apache.org/jira/browse/HADOOP-17010) | *Major*
| **Add queue capacity weights support in FairCallQueue**
+
+When FairCallQueue is enabled, user can specify capacity allocation among all
sub-queues via configuration âipc.\.callqueue.capacity.weightsâ. The
value of this config is a comma-separated list of positive integers, each of
which specifies the weight associated with the sub-queue at that index. This
list length should be IPC scheduler priority levels, defined by
"scheduler.priority.levels".
+
+By default, each sub-queue is associated with weight 1, i.e., all sub-queues
are allocated with the same capacity.
+
+
+---
+
+* [HADOOP-17024](https://issues.apache.org/jira/browse/HADOOP-17024) | *Major*
| **ListStatus on ViewFS root (ls "/") should list the linkFallBack root
(configured target root).**
+
+ViewFS#listStatus on root("/") considers listing from fallbackLink if
available. If the same directory name is present in configured mount path as
well as in fallback link, then only the configured mount path will be listed in
the returned result.
+
+
+---
+
+* [HDFS-15288](https://issues.apache.org/jira/browse/HDFS-15288) | *Major* |
**Add Available Space Rack Fault Tolerant BPP**
+
+Added a new BlockPlacementPolicy:
"AvailableSpaceRackFaultTolerantBlockPlacementPolicy" which uses the same
optimization logic as the AvailableSpaceBlockPlacementPolicy along with
spreading the replicas across maximum number of racks, similar to
BlockPlacementPolicyRackFaultTolerant.
+The BPP can be configured by setting the blockplacement policy class as
org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceRackFaultTolerantBlockPlacementPolicy
+
+
+---
+
+* [HDFS-13183](https://issues.apache.org/jira/browse/HDFS-13183) | *Major* |
**Standby NameNode process getBlocks request to reduce Active load**
+
+Enable balancer to redirect getBlocks request to a Standby Namenode, thus
reducing the performance impact of balancer to the Active NameNode.
+
+The feature is disabled by default. To enable it, configure the hdfs-site.xml
of balancer:
+dfs.ha.allow.stale.reads = true.
+
+
+---
+
+* [HADOOP-17079](https://issues.apache.org/jira/browse/HADOOP-17079) | *Major*
| **Optimize UGI#getGroups by adding UGI#getGroupsSet**
+
+Added a UserGroupMapping#getGroupsSet() API and deprecate
UserGroupMapping#getGroups.
+
+The UserGroupMapping#getGroups() can be expensive as it involves Set-\>List
conversion. For user with large group membership (i.e., \> 1000 groups), we
recommend using getGroupSet to avoid the conversion and fast membership look up.
+
+
+---
+
+* [HDFS-15385](https://issues.apache.org/jira/browse/HDFS-15385) | *Critical*
| **Upgrade boost library to 1.72**
+
+Boost 1.72 is required when building native code.
+
+
+---
+
+* [HADOOP-17091](https://issues.apache.org/jira/browse/HADOOP-17091) | *Major*
| **[JDK11] Fix Javadoc errors**
+
+
+* Upgraded to Yetus master to change javadoc goals.
+* Changed javadoc goals from `javadoc:javadoc` to `process-sources
javadoc:javadoc-no-fork`.
+* Javadoc with JDK11 fails in
svn commit: r67696 [2/5] - /dev/hadoop/hadoop-3.4.0-RC3/
Added: dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
==
--- dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md (added)
+++ dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md Mon Mar 4 09:36:35 2024
@@ -0,0 +1,2954 @@
+
+
+# Apache Hadoop Changelog
+
+## Release 3.4.0 - 2024-03-04
+
+
+
+### IMPORTANT ISSUES:
+
+| JIRA | Summary | Priority | Component | Reporter | Contributor |
+|: |: | :--- |: |: |: |
+| [HDFS-15380](https://issues.apache.org/jira/browse/HDFS-15380) | RBF: Could
not fetch real remote IP in RouterWebHdfsMethods | Major | webhdfs | Tao Li |
Tao Li |
+| [HDFS-15814](https://issues.apache.org/jira/browse/HDFS-15814) | Make some
parameters configurable for DataNodeDiskMetrics | Major | hdfs | Tao Li | Tao
Li |
+| [HDFS-16265](https://issues.apache.org/jira/browse/HDFS-16265) | Refactor
HDFS tool tests for better reuse | Blocker | hdfs-client, libhdfs++, tools |
Gautham Banasandra | Gautham Banasandra |
+| [HADOOP-17956](https://issues.apache.org/jira/browse/HADOOP-17956) | Replace
all default Charset usage with UTF-8 | Major | common | Viraj Jasani | Viraj
Jasani |
+| [HDFS-16278](https://issues.apache.org/jira/browse/HDFS-16278) | Make HDFS
snapshot tools cross platform | Major | hdfs-client, libhdfs++, tools |
Gautham Banasandra | Gautham Banasandra |
+| [HDFS-16285](https://issues.apache.org/jira/browse/HDFS-16285) | Make HDFS
ownership tools cross platform | Major | hdfs-client, libhdfs++, tools |
Gautham Banasandra | Gautham Banasandra |
+| [HDFS-16419](https://issues.apache.org/jira/browse/HDFS-16419) | Make HDFS
data transfer tools cross platform | Major | hdfs-client, libhdfs++, tools |
Gautham Banasandra | Gautham Banasandra |
+| [HDFS-16511](https://issues.apache.org/jira/browse/HDFS-16511) | Improve
lock type for ReplicaMap under fine-grain lock mode. | Major | hdfs |
Mingxiang Li | Mingxiang Li |
+| [HDFS-16534](https://issues.apache.org/jira/browse/HDFS-16534) | Split
datanode block pool locks to volume grain. | Major | datanode | Mingxiang Li |
Mingxiang Li |
+| [HADOOP-18219](https://issues.apache.org/jira/browse/HADOOP-18219) | Fix
shadedclient test failure | Blocker | test | Gautham Banasandra | Akira
Ajisaka |
+| [HADOOP-18621](https://issues.apache.org/jira/browse/HADOOP-18621) |
CryptoOutputStream::close leak when encrypted zones + quota exceptions |
Critical | fs | Colm Dougan | Colm Dougan |
+| [YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) | YARN
Federation improvements | Major | federation | Subramaniam Krishnan |
Subramaniam Krishnan |
+
+
+### NEW FEATURES:
+
+| JIRA | Summary | Priority | Component | Reporter | Contributor |
+|: |: | :--- |: |: |: |
+| [HADOOP-17010](https://issues.apache.org/jira/browse/HADOOP-17010) | Add
queue capacity weights support in FairCallQueue | Major | ipc | Fengnan Li |
Fengnan Li |
+| [HDFS-15288](https://issues.apache.org/jira/browse/HDFS-15288) | Add
Available Space Rack Fault Tolerant BPP | Major | block placement | Ayush
Saxena | Ayush Saxena |
+| [HDFS-13183](https://issues.apache.org/jira/browse/HDFS-13183) | Standby
NameNode process getBlocks request to reduce Active load | Major | balancer &
mover, namenode | Xiaoqiao He | Xiaoqiao He |
+| [HDFS-15463](https://issues.apache.org/jira/browse/HDFS-15463) | Add a tool
to validate FsImage | Major | namenode | Tsz-wo Sze | Tsz-wo Sze |
+| [HADOOP-17165](https://issues.apache.org/jira/browse/HADOOP-17165) |
Implement service-user feature in DecayRPCScheduler | Major | rpc-server |
Takanobu Asanuma | Takanobu Asanuma |
+| [HADOOP-15891](https://issues.apache.org/jira/browse/HADOOP-15891) | Provide
Regex Based Mount Point In Inode Tree | Major | viewfs | zhenzhao wang |
zhenzhao wang |
+| [HDFS-15025](https://issues.apache.org/jira/browse/HDFS-15025) | Applying
NVDIMM storage media to HDFS | Major | datanode, hdfs | YaYun Wang | YaYun
Wang |
+| [HDFS-15098](https://issues.apache.org/jira/browse/HDFS-15098) | Add SM4
encryption method for HDFS | Major | hdfs | liusheng | liusheng |
+| [HADOOP-17125](https://issues.apache.org/jira/browse/HADOOP-17125) | Using
snappy-java in SnappyCodec | Major | common | DB Tsai | L. C. Hsieh |
+| [HDFS-15294](https://issues.apache.org/jira/browse/HDFS-15294) | Federation
balance tool | Major | rbf, tools | Jinglun | Jinglun |
+| [HADOOP-17292](https://issues.apache.org/jira/browse/HADOOP-17292) | Using
lz4-java in Lz4Codec | Major | common | L. C. Hsieh | L. C. Hsieh |
+| [HDFS-14090](https://issues.apache.org/jira/browse/HDFS-14090) | RBF:
Improved isolation for downstream name nodes. {Static} | Major | rbf | CR Hota
| Fengnan Li |
+| [HDFS-15711](https://issues.apache.org/jira/browse/HDFS-15711) | Add Metrics
to HttpFS Server | Major | httpfs | Ahmed Hussein | Ahmed Hussein |
+| [HADOOP-16492](https://issues.apache.org/jira/browse/HADOOP-16492) | Support
HuaweiCloud Object
svn commit: r67696 [4/5] - /dev/hadoop/hadoop-3.4.0-RC3/
Added: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-rat.txt == --- dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-rat.txt (added) +++ dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-rat.txt Mon Mar 4 09:36:35 2024 @@ -0,0 +1,17723 @@ + +* +Summary +--- +Generated at: 2024-03-04T06:27:02+00:00 + +Notes: 0 +Binaries: 0 +Archives: 0 +Standards: 11 + +Apache Licensed: 11 +Generated Documents: 0 + +JavaDocs are generated, thus a license header is optional. +Generated files do not require license headers. + +0 Unknown Licenses + +* + Files with Apache License headers will be marked AL + Binary files (which do not require any license headers) will be marked B + Compressed archives will be marked A + Notices, licenses etc. will be marked N + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/VisibleForTesting.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceAudience.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/package-info.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/ExcludePrivateAnnotationsJDiffDoclet.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/StabilityOptions.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/IncludePublicAnnotationsJDiffDoclet.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/IncludePublicAnnotationsStandardDoclet.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/ExcludePrivateAnnotationsStandardDoclet.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceStability.java + AL/build/source/hadoop-common-project/hadoop-annotations/pom.xml + +* + +* +Summary +--- +Generated at: 2024-03-04T06:27:02+00:00 + +Notes: 2 +Binaries: 0 +Archives: 0 +Standards: 69 + +Apache Licensed: 69 +Generated Documents: 0 + +JavaDocs are generated, thus a license header is optional. +Generated files do not require license headers. + +0 Unknown Licenses + +* + Files with Apache License headers will be marked AL + Binary files (which do not require any license headers) will be marked B + Compressed archives will be marked A + Notices, licenses etc. will be marked N + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticationException.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/ConnectionConfigurator.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/PseudoAuthenticator.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/Authenticator.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/package-info.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationHandlerUtil.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/CompositeAuthenticationHandler.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/MultiSchemeAuthenticationHandler.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationHandler.java + AL
svn commit: r67696 [5/5] - /dev/hadoop/hadoop-3.4.0-RC3/
Added: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-rat.txt.asc == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-rat.txt.asc -- svn:mime-type = application/pgp-signature Added: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-rat.txt.sha512 == --- dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-rat.txt.sha512 (added) +++ dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-rat.txt.sha512 Mon Mar 4 09:36:35 2024 @@ -0,0 +1 @@ +SHA512 (hadoop-3.4.0-rat.txt) = 08b71b01013571a4a3566a66b02e9bf5ef904dd97960afa1d6857ae526ad12799b7f956a9fbe10b62fd8c0f1ecb88e079b8111fe63c3a66e39cee397a284385e Added: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-site.tar.gz == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-site.tar.gz -- svn:mime-type = application/x-gzip Added: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-site.tar.gz.asc == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-site.tar.gz.asc -- svn:mime-type = application/pgp-signature Added: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-site.tar.gz.sha512 == --- dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-site.tar.gz.sha512 (added) +++ dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-site.tar.gz.sha512 Mon Mar 4 09:36:35 2024 @@ -0,0 +1 @@ +SHA512 (hadoop-3.4.0-site.tar.gz) = d393743ed772c0a3afdfad8048e6d9f6f7f5abbc50c3b62c5535a47abd78206c357194166c85596b0faf284b960e24d7526978e66a474f4b932b0d54644522e5 Added: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-src.tar.gz == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-src.tar.gz -- svn:mime-type = application/x-gzip Added: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-src.tar.gz.asc == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-src.tar.gz.asc -- svn:mime-type = application/pgp-signature Added: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-src.tar.gz.sha512 == --- dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-src.tar.gz.sha512 (added) +++ dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-src.tar.gz.sha512 Mon Mar 4 09:36:35 2024 @@ -0,0 +1 @@ +SHA512 (hadoop-3.4.0-src.tar.gz) = c10c730017debaa468352458397e624e059586577797ee5821ebc238a3543139eff674a45cbe66041e7fc1fde6b35f352b6fad828f6503cc935360f1893cd5b9 Added: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0.tar.gz == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0.tar.gz -- svn:mime-type = application/x-gzip Added: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0.tar.gz.asc == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0.tar.gz.asc -- svn:mime-type = application/pgp-signature Added: dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0.tar.gz.sha512 == --- dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0.tar.gz.sha512 (added) +++ dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0.tar.gz.sha512 Mon Mar 4 09:36:35 2024 @@ -0,0 +1 @@ +SHA512 (hadoop-3.4.0.tar.gz) = 6f653c0109f97430047bd3677c50da7c8a2809d153b231794cf980b3208a6b4beff8ff1a03a01094299d459a3a37a3fe16731629987165d71f328657dbf2f24c - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
svn commit: r67696 [1/5] - /dev/hadoop/hadoop-3.4.0-RC3/
Author: slfan1989 Date: Mon Mar 4 09:36:35 2024 New Revision: 67696 Log: HADOOP-19018. Hadoop-3.4.0-RC3 built from bd8b77f398f Added: dev/hadoop/hadoop-3.4.0-RC3/ dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md.asc (with props) dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md.sha512 dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md.asc (with props) dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md.sha512 dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-aarch64.tar.gz (with props) dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-aarch64.tar.gz.asc (with props) dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-aarch64.tar.gz.sha512 dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-rat.txt dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-rat.txt.asc (with props) dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-rat.txt.sha512 dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-site.tar.gz (with props) dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-site.tar.gz.asc (with props) dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-site.tar.gz.sha512 dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-src.tar.gz (with props) dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-src.tar.gz.asc (with props) dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0-src.tar.gz.sha512 dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0.tar.gz (with props) dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0.tar.gz.asc (with props) dev/hadoop/hadoop-3.4.0-RC3/hadoop-3.4.0.tar.gz.sha512 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
svn commit: r67694 - /dev/hadoop/hadoop-3.4.0-RC2/
Author: slfan1989 Date: Mon Mar 4 09:29:27 2024 New Revision: 67694 Log: delete staged hadoop/3.4.0-RC2 Removed: dev/hadoop/hadoop-3.4.0-RC2/ - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) annotated tag release-3.4.0-RC3 updated (bd8b77f398f6 -> 6cb7b4b4d92d)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to annotated tag release-3.4.0-RC3 in repository https://gitbox.apache.org/repos/asf/hadoop.git *** WARNING: tag release-3.4.0-RC3 was modified! *** from bd8b77f398f6 (commit) to 6cb7b4b4d92d (tag) tagging bd8b77f398f626bb7791783192ee7a5dfaeec760 (commit) replaces release-3.4.0-RC2 by Shilun Fan on Mon Mar 4 13:28:30 2024 +0800 - Log - Release candidate - 3.4.0-RC3 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEECm40PGXwzjlsDKJKl4RW45ZNtwEFAmXlW/4ACgkQl4RW45ZN twGoeQ/9GeULZArBzHKuw9rUFZtk8u3l3VziFrdB42CYpW8uvYQtPSG58DcSPnc+ 8wItOS7Pkj8GPN5SmMVAd88B7U2ATGt75dai837h+gjureI+zLSsR4E9bANyakJz DAk1uXDi4mGenLL0ipcmDPahoJkYfiQUYgUZW+m5o04w38MGod+NRoQ38otq77Hx qdbQEc+MvsTwXVm1/cYgBhi7/kVMadHEbvj2BVWk7WZ7nivVo7JtTat9+1r8i8V8 nRc2XkJMMAo2pp3MS6REKMUGXJFywKBdIVsbWc6okQI67AsYY3hd/o0sHgQdImRt 0n46ZBKTg1Qqr9Qy94GLdke473pbWRzQ93Y0Wmtx57++fPa4dnKSd3MHNawpFmkA F8h7xy9frBgM5/LSTwHOHEc0ksSvXuEmEu1vhmx1yE/GIEMMag1kZdVIUzWQj4UF /2JiDWFtiCRw/9mwQ9oG1ArN/tlGHKa0dsKEP3cwCreD8/oCXbVEutgnjrfkqlI5 5aHHFGh94c/EfLy+UEvg1uT2UeKc4NADE3AJVxBqdCK3l8pF1gcZrzGLXyCZT7R9 JDQWWn9hPsIQsiU0k4FCVhQDEzlsd8dc/BWt/0T+7rM2sMKxOzafzFbDNmvqgUDs q+pAkexeZUYvyBVCiruKriAoACP6aAiEWT0TWz2L4TtXdF3xR6g= =NlTM -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) annotated tag release-3.4.0-RC3 deleted (was 70a9be135011)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to annotated tag release-3.4.0-RC3 in repository https://gitbox.apache.org/repos/asf/hadoop.git *** WARNING: tag release-3.4.0-RC3 was deleted! *** tag was 70a9be135011 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-19099. Add Protobuf Compatibility Notes (#6607) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.0 by this push: new bd8b77f398f6 HADOOP-19099. Add Protobuf Compatibility Notes (#6607) Contributed by Shilun Fan. bd8b77f398f6 is described below commit bd8b77f398f626bb7791783192ee7a5dfaeec760 Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Mon Mar 4 13:19:28 2024 +0800 HADOOP-19099. Add Protobuf Compatibility Notes (#6607) Contributed by Shilun Fan. Reviewed-by: He Xiaoqiao Signed-off-by: Shilun Fan --- hadoop-project/src/site/markdown/index.md.vm | 10 ++ 1 file changed, 10 insertions(+) diff --git a/hadoop-project/src/site/markdown/index.md.vm b/hadoop-project/src/site/markdown/index.md.vm index 54e8055e633d..fa728f74463a 100644 --- a/hadoop-project/src/site/markdown/index.md.vm +++ b/hadoop-project/src/site/markdown/index.md.vm @@ -242,6 +242,16 @@ you want to remain exclusively *your cluster*. Finally, if you are using Hadoop as a service deployed/managed by someone else, do determine what security their products offer and make sure it meets your requirements. +Protobuf Compatibility +=== + +In HADOOP-18197, we upgraded the Protobuf in hadoop-thirdparty to version 3.21.12. +This version may have compatibility issues with certain versions of JDK8, +and you may encounter some errors (please refer to the discussion in HADOOP-18197 for specific details). + +To address this issue, we recommend upgrading the JDK version in your production environment to a higher version (> JDK8). +We will resolve this issue by upgrading hadoop-thirdparty's Protobuf to a higher version in a future release of 3.4.x. +Please note that we will discontinue support for JDK8 in future releases of 3.4.x. Getting Started === - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19099. Add Protobuf Compatibility Notes (#6607) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new 65a2bdeeead0 HADOOP-19099. Add Protobuf Compatibility Notes (#6607) Contributed by Shilun Fan. 65a2bdeeead0 is described below commit 65a2bdeeead0c4826588f2bb05b10bb307a2727f Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Mon Mar 4 13:19:28 2024 +0800 HADOOP-19099. Add Protobuf Compatibility Notes (#6607) Contributed by Shilun Fan. Reviewed-by: He Xiaoqiao Signed-off-by: Shilun Fan --- hadoop-project/src/site/markdown/index.md.vm | 10 ++ 1 file changed, 10 insertions(+) diff --git a/hadoop-project/src/site/markdown/index.md.vm b/hadoop-project/src/site/markdown/index.md.vm index 54e8055e633d..fa728f74463a 100644 --- a/hadoop-project/src/site/markdown/index.md.vm +++ b/hadoop-project/src/site/markdown/index.md.vm @@ -242,6 +242,16 @@ you want to remain exclusively *your cluster*. Finally, if you are using Hadoop as a service deployed/managed by someone else, do determine what security their products offer and make sure it meets your requirements. +Protobuf Compatibility +=== + +In HADOOP-18197, we upgraded the Protobuf in hadoop-thirdparty to version 3.21.12. +This version may have compatibility issues with certain versions of JDK8, +and you may encounter some errors (please refer to the discussion in HADOOP-18197 for specific details). + +To address this issue, we recommend upgrading the JDK version in your production environment to a higher version (> JDK8). +We will resolve this issue by upgrading hadoop-thirdparty's Protobuf to a higher version in a future release of 3.4.x. +Please note that we will discontinue support for JDK8 in future releases of 3.4.x. Getting Started === - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) annotated tag release-3.4.0-RC3 updated (5ed3a27df041 -> 70a9be135011)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to annotated tag release-3.4.0-RC3 in repository https://gitbox.apache.org/repos/asf/hadoop.git *** WARNING: tag release-3.4.0-RC3 was modified! *** from 5ed3a27df041 (commit) to 70a9be135011 (tag) tagging 5ed3a27df041ebb57e4983792e28b13320cb96df (commit) replaces release-3.4.0-RC2 by Shilun Fan on Sun Mar 3 18:16:05 2024 +0800 - Log - Release candidate - 3.4.0-RC3 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEECm40PGXwzjlsDKJKl4RW45ZNtwEFAmXkTeUACgkQl4RW45ZN twFz9w//c1yFtQ5bmCKbO0hDovBnwBFkZAR80d1MkLwBlTLolR3cLKlt+ZzM9j7h X538zoEwztJ/k9oRrhUgmjizaRq9T4DZzVe9a5g+8eavQybEBy23J53/aPT94iVb YSYyhI7I+35HhV3IllQwqIaFy84B4r8sLjTlmmPDjyChdEpeUJdbNFHVSdAkda71 8e5cxrB07mRBDGy79v+zCbzjl/amfWrfht3A91MZZi/FOJL2KC14zfNHOSZTpo0p UsjAb3nStrTDoXaRwJlAI4LJ2VD8iI9Qwx1jBGggLJmAFN2ixIrueGKCiefbDQZ9 Qt/A+KzCi7inCFdB98xBkbVaQGlymLZLFYqau51K/CttjswC1vsggNiIfiVS+JYg UwrNaY8FyK26E+JZ8lMK4Je1cmksZi79qEzGn5Cm/KtzAfl6lu4+CP4IyXlgkhsW ETu5aXCmrYAPJiy7beFiyBq3En+K0772WG6A/f2I7N7CBBepSP4I1PtIBLzgpkZU cCDONqVK+4BT5kw9UlvkVeNAfT4gdb7gVvXixVnxwDbtz4/eKbRVlOmURPOWY4/C /0z5cv3rYTOrEV8kL0Bd8Pb/J2Am4sPOKQPX100HwknOxcWAti4IyB84Tj9UKpqi JX9kXkztzbXMQR/UOeHDHFGp9ZfIExD61617QRvz5MAX4IAt3qM= =3Z4t -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-19084. Prune hadoop-common transitive dependencies (#6574)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new 5ed3a27df041 HADOOP-19084. Prune hadoop-common transitive dependencies
(#6574)
5ed3a27df041 is described below
commit 5ed3a27df041ebb57e4983792e28b13320cb96df
Author: Steve Loughran
AuthorDate: Fri Feb 23 17:56:46 2024 +
HADOOP-19084. Prune hadoop-common transitive dependencies (#6574)
Exclude more artifacts which are dependencies of hadoop-* modules,
with the goal of keeping conflict out of downstream applications.
In particular we have pruned the dependencies of of:
-zookeeper
-other libraries referencing logging
This keeps slf4j-log4j12 and log4j12 off the classpath
of applications importing hadoop-common.
Somehow logback references do still surface; applications
pulling in hadoop-common directly or indirectly should
review their imports carefully.
Contributed by Steve Loughran
---
hadoop-project/pom.xml | 46 ++
.../hadoop-yarn-applications-mawo-core/pom.xml | 19 -
2 files changed, 29 insertions(+), 36 deletions(-)
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index bd93a7e392c7..0ed96d087bc6 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -126,7 +126,7 @@
1.9.0
2.14.0
3.12.0
-1.1.3
+1.2
3.6.1
3.9.0
1.10.0
@@ -1435,15 +1435,11 @@
io.netty
-netty-all
-
-
-io.netty
-netty-handler
+*
-io.netty
-netty-transport-native-epoll
+commons-io
+commons-io
commons-collections
@@ -1507,15 +1503,7 @@
io.netty
-netty-all
-
-
-io.netty
-netty-handler
-
-
-io.netty
-netty-transport-native-epoll
+*
org.eclipse.jetty
@@ -1596,6 +1584,12 @@
${leveldbjni.group}
leveldbjni-all
1.8
+
+
+com.fasterxml.jackson.core
+jackson-core
+
+
org.fusesource.hawtjni
@@ -1958,11 +1952,23 @@
org.ehcache
ehcache
${ehcache.version}
+
+
+ org.slf4j
+ slf4j-api
+
+
com.zaxxer
HikariCP
${hikari.version}
+
+
+ org.slf4j
+ slf4j-api
+
+
org.apache.derby
@@ -2024,6 +2030,12 @@
org.jsonschema2pojo
jsonschema2pojo-core
${jsonschema2pojo.version}
+
+
+commons-io
+commons-io
+
+
org.xerial.snappy
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/pom.xml
index ecdda6280f1c..f53cb2fff8ad 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/pom.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core/pom.xml
@@ -93,25 +93,6 @@
org.apache.zookeeper
zookeeper
-
-
-
- junit
- junit
-
-
- com.sun.jdmk
- jmxtools
-
-
- com.sun.jmx
- jmxri
-
-
- org.jboss.netty
- netty
-
-
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-18088. Replace log4j 1.x with reload4j. (#4052)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.0 by this push: new 253afde7b2d0 HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 253afde7b2d0 is described below commit 253afde7b2d00444145c242832ea1f6da73528b2 Author: Steve Loughran AuthorDate: Tue Feb 13 16:33:51 2024 + HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) Co-authored-by: Wei-Chiu Chuang Includes HADOOP-18354. Upgrade reload4j to 1.22.2 due to XXE vulnerability (#4607). Log4j 1.2.17 has been replaced by reloadj 1.22.2 SLF4J is at 1.7.36 (cherry picked from commit 095dfcca306289e8f676de89d6a054a193593d5d) --- LICENSE-binary | 11 +-- .../resources/assemblies/hadoop-dynamometer.xml| 2 +- .../resources/assemblies/hadoop-hdfs-nfs-dist.xml | 2 +- .../resources/assemblies/hadoop-httpfs-dist.xml| 2 +- .../main/resources/assemblies/hadoop-kms-dist.xml | 2 +- .../resources/assemblies/hadoop-mapreduce-dist.xml | 2 +- .../main/resources/assemblies/hadoop-nfs-dist.xml | 2 +- .../src/main/resources/assemblies/hadoop-tools.xml | 2 +- .../main/resources/assemblies/hadoop-yarn-dist.xml | 2 +- .../hadoop-client-check-invariants/pom.xml | 4 +- .../hadoop-client-check-test-invariants/pom.xml| 4 +- .../hadoop-client-integration-tests/pom.xml| 9 +- .../hadoop-client-minicluster/pom.xml | 10 +- .../hadoop-client-runtime/pom.xml | 8 +- hadoop-client-modules/hadoop-client/pom.xml| 14 +-- hadoop-common-project/hadoop-auth-examples/pom.xml | 6 +- hadoop-common-project/hadoop-auth/pom.xml | 12 ++- hadoop-common-project/hadoop-common/pom.xml| 6 +- .../java/org/apache/hadoop/util/GenericsUtil.java | 2 +- .../java/org/apache/hadoop/util/TestClassUtil.java | 2 +- hadoop-common-project/hadoop-kms/pom.xml | 6 +- hadoop-common-project/hadoop-minikdc/pom.xml | 2 +- hadoop-common-project/hadoop-nfs/pom.xml | 6 +- hadoop-hdfs-project/hadoop-hdfs-client/pom.xml | 4 +- hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml | 6 +- hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml| 6 +- hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml| 6 +- hadoop-hdfs-project/hadoop-hdfs/pom.xml| 6 +- .../hadoop-mapreduce-client/pom.xml| 2 +- hadoop-project/pom.xml | 105 +++-- hadoop-tools/hadoop-azure/pom.xml | 4 +- .../pom.xml| 4 +- .../hadoop-yarn-services-core/pom.xml | 4 +- .../hadoop-yarn/hadoop-yarn-client/pom.xml | 4 +- .../hadoop-yarn/hadoop-yarn-common/pom.xml | 4 +- .../hadoop-yarn-server-resourcemanager/pom.xml | 4 +- .../pom.xml| 4 + 37 files changed, 186 insertions(+), 95 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index c5ba9f938aef..c2cc745b4da3 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -210,6 +210,7 @@ hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/nvd3-1.8.5.* (css and js hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/TimeoutFuture.java +ch.qos.reload4j:reload4j:1.2.22 com.aliyun:aliyun-java-sdk-core:4.5.10 com.aliyun:aliyun-java-sdk-kms:2.11.0 com.aliyun:aliyun-java-sdk-ram:3.1.0 @@ -294,7 +295,6 @@ io.reactivex:rxjava-string:1.1.1 io.reactivex:rxnetty:0.4.20 io.swagger:swagger-annotations:1.5.4 javax.inject:javax.inject:1 -log4j:log4j:1.2.17 net.java.dev.jna:jna:5.2.0 net.minidev:accessors-smart:1.2 org.apache.avro:avro:1.9.2 @@ -489,11 +489,10 @@ org.checkerframework:checker-qual:2.5.2 org.codehaus.mojo:animal-sniffer-annotations:1.21 org.jruby.jcodings:jcodings:1.0.13 org.jruby.joni:joni:2.1.2 -org.slf4j:jul-to-slf4j:jar:1.7.25 -org.ojalgo:ojalgo:43.0:compile -org.slf4j:jul-to-slf4j:1.7.25 -org.slf4j:slf4j-api:1.7.25 -org.slf4j:slf4j-log4j12:1.7.25 +org.ojalgo:ojalgo:43.0 +org.slf4j:jul-to-slf4j:1.7.36 +org.slf4j:slf4j-api:1.7.36 +org.slf4j:slf4j-reload4j:1.7.36 CDDL 1.1 + GPLv2 with classpath exception diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer.xml b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer.xml index 448035262e12..b2ce562231c5 100644 --- a/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer.xml +++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-dynamometer.xml @@ -66,7 +66,7 @@ org.slf4j:slf4j-api -org.slf4j:slf4j-log4j12
(hadoop) branch HDFS-17384 updated: Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#6537)" (#6578)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch HDFS-17384
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/HDFS-17384 by this push:
new f42e4ca3483e Revert "HADOOP-19071. Update maven-surefire-plugin from
3.0.0 to 3.2.5. (#6537)" (#6578)
f42e4ca3483e is described below
commit f42e4ca3483e13fa42963ec12eb0a037288b7d84
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Fri Feb 23 14:25:15 2024 +0800
Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5.
(#6537)" (#6578)
This reverts commit 555faf28ce365443c17767adf0d6d570cef32ee3.
---
dev-support/bin/hadoop.sh | 3 +--
hadoop-project/pom.xml| 2 +-
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/dev-support/bin/hadoop.sh b/dev-support/bin/hadoop.sh
index c60603334feb..fd9363e66bbd 100755
--- a/dev-support/bin/hadoop.sh
+++ b/dev-support/bin/hadoop.sh
@@ -577,7 +577,6 @@ function shadedclient_rebuild
extra=(
"-Dtest=NoUnitTests"
-"-Dsurefire.failIfNoSpecifiedTests=false"
"-Dmaven.javadoc.skip=true"
"-Dcheckstyle.skip=true"
"-Dspotbugs.skip=true"
@@ -615,7 +614,7 @@ function shadedclient_rebuild
echo_and_redirect "${logfile}" \
"${MAVEN}" "${MAVEN_ARGS[@]}" verify -fae --batch-mode -am \
"${modules[@]}" \
- -DskipShade -Dtest=NoUnitTests -Dsurefire.failIfNoSpecifiedTests=false
-Dmaven.javadoc.skip=true -Dcheckstyle.skip=true \
+ -DskipShade -Dtest=NoUnitTests -Dmaven.javadoc.skip=true
-Dcheckstyle.skip=true \
-Dspotbugs.skip=true ${extra[*]}
count=$("${GREP}" -c '\[ERROR\]' "${logfile}")
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index a344b562b951..3086e7496998 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -169,7 +169,7 @@
-Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError
-3.2.5
+3.0.0-M1
${maven-surefire-plugin.version}
${maven-surefire-plugin.version}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19065. Update Protocol Buffers installation to 3.21.12 (#6576) Contributed by Zhaobo Huang.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 175f245107f8 HADOOP-19065. Update Protocol Buffers installation to
3.21.12 (#6576) Contributed by Zhaobo Huang.
175f245107f8 is described below
commit 175f245107f8547d6af7c01dc5c6c0f8cd30dd66
Author: huangzhaobo
AuthorDate: Mon Feb 26 22:51:20 2024 +0800
HADOOP-19065. Update Protocol Buffers installation to 3.21.12 (#6576)
Contributed by Zhaobo Huang.
Reviewed-by: Steve Loughran
Signed-off-by: Shilun Fan
---
BUILDING.txt | 29 +++---
dev-support/docker/Dockerfile | 2 +-
dev-support/docker/Dockerfile_aarch64 | 2 +-
dev-support/docker/Dockerfile_centos_7 | 2 +-
dev-support/docker/Dockerfile_centos_8 | 2 +-
dev-support/docker/Dockerfile_debian_10| 2 +-
.../docker/pkg-resolver/install-protobuf.sh| 9 ---
7 files changed, 24 insertions(+), 24 deletions(-)
diff --git a/BUILDING.txt b/BUILDING.txt
index a98d7049a2b0..3d35007c9cbe 100644
--- a/BUILDING.txt
+++ b/BUILDING.txt
@@ -7,7 +7,7 @@ Requirements:
* JDK 1.8
* Maven 3.3 or later
* Boost 1.72 (if compiling native code)
-* Protocol Buffers 3.7.1 (if compiling native code)
+* Protocol Buffers 3.21.12 (if compiling native code)
* CMake 3.19 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* Cyrus SASL devel (if compiling native code)
@@ -74,10 +74,10 @@ Refer to dev-support/docker/Dockerfile):
$ ./bootstrap
$ make -j$(nproc)
$ sudo make install
-* Protocol Buffers 3.7.1 (required to build native code)
- $ curl -L -s -S
https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz
-o protobuf-3.7.1.tar.gz
- $ mkdir protobuf-3.7-src
- $ tar xzf protobuf-3.7.1.tar.gz --strip-components 1 -C protobuf-3.7-src &&
cd protobuf-3.7-src
+* Protocol Buffers 3.21.12 (required to build native code)
+ $ curl -L
https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.21.12.tar.gz >
protobuf-3.21.12.tar.gz
+ $ tar -zxvf protobuf-3.21.12.tar.gz && cd protobuf-3.21.12
+ $ ./autogen.sh
$ ./configure
$ make -j$(nproc)
$ sudo make install
@@ -437,10 +437,10 @@ Installing required dependencies for clean install of
macOS 10.14:
* Install native libraries, only openssl is required to compile native code,
you may optionally install zlib, lz4, etc.
$ brew install openssl
-* Protocol Buffers 3.7.1 (required to compile native code)
- $ wget
https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz
- $ mkdir -p protobuf-3.7 && tar zxvf protobuf-java-3.7.1.tar.gz
--strip-components 1 -C protobuf-3.7
- $ cd protobuf-3.7
+* Protocol Buffers 3.21.12 (required to compile native code)
+ $ curl -L
https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.21.12.tar.gz >
protobuf-3.21.12.tar.gz
+ $ tar -zxvf protobuf-3.21.12.tar.gz && cd protobuf-3.21.12
+ $ ./autogen.sh
$ ./configure
$ make
$ make check
@@ -476,11 +476,10 @@ Building on CentOS 8
* Install python2 for building documentation.
$ sudo dnf install python2
-* Install Protocol Buffers v3.7.1.
- $ git clone https://github.com/protocolbuffers/protobuf
- $ cd protobuf
- $ git checkout v3.7.1
- $ autoreconf -i
+* Install Protocol Buffers v3.21.12.
+ $ curl -L
https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.21.12.tar.gz >
protobuf-3.21.12.tar.gz
+ $ tar -zxvf protobuf-3.21.12.tar.gz && cd protobuf-3.21.12
+ $ ./autogen.sh
$ ./configure --prefix=/usr/local
$ make
$ sudo make install
@@ -535,7 +534,7 @@ Requirements:
* JDK 1.8
* Maven 3.0 or later (maven.apache.org)
* Boost 1.72 (boost.org)
-* Protocol Buffers 3.7.1 (https://github.com/protocolbuffers/protobuf/releases)
+* Protocol Buffers 3.21.12 (https://github.com/protocolbuffers/protobuf/tags)
* CMake 3.19 or newer (cmake.org)
* Visual Studio 2019 (visualstudio.com)
* Windows SDK 8.1 (optional, if building CPU rate control for the container
executor. Get this from
diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile
index fac364bbd436..3b71e622a575 100644
--- a/dev-support/docker/Dockerfile
+++ b/dev-support/docker/Dockerfile
@@ -66,7 +66,7 @@ ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
ENV SPOTBUGS_HOME /opt/spotbugs
###
-# Set env vars for Google Protobuf 3.7.1
+# Set env vars for Google Protobuf 3.21.12
###
ENV PROTOBUF_HOME /opt/protobuf
ENV PATH "${PATH}:/opt/protobuf/bin"
diff --git a/dev-support/docker/Dockerfile_aarch64
b/dev-support/docker/Dockerfile_aarch64
index 14a537801270..9941c7dd619f 100644
--- a/dev-support/docker/Dockerfile_a
(hadoop) branch trunk updated: Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#6537)" (#6578)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 10ab8abccd7e Revert "HADOOP-19071. Update maven-surefire-plugin from
3.0.0 to 3.2.5. (#6537)" (#6578)
10ab8abccd7e is described below
commit 10ab8abccd7e086d1d47eb7d706c8d5c42052172
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Fri Feb 23 14:25:15 2024 +0800
Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5.
(#6537)" (#6578)
This reverts commit 555faf28ce365443c17767adf0d6d570cef32ee3.
---
dev-support/bin/hadoop.sh | 3 +--
hadoop-project/pom.xml| 2 +-
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/dev-support/bin/hadoop.sh b/dev-support/bin/hadoop.sh
index c60603334feb..fd9363e66bbd 100755
--- a/dev-support/bin/hadoop.sh
+++ b/dev-support/bin/hadoop.sh
@@ -577,7 +577,6 @@ function shadedclient_rebuild
extra=(
"-Dtest=NoUnitTests"
-"-Dsurefire.failIfNoSpecifiedTests=false"
"-Dmaven.javadoc.skip=true"
"-Dcheckstyle.skip=true"
"-Dspotbugs.skip=true"
@@ -615,7 +614,7 @@ function shadedclient_rebuild
echo_and_redirect "${logfile}" \
"${MAVEN}" "${MAVEN_ARGS[@]}" verify -fae --batch-mode -am \
"${modules[@]}" \
- -DskipShade -Dtest=NoUnitTests -Dsurefire.failIfNoSpecifiedTests=false
-Dmaven.javadoc.skip=true -Dcheckstyle.skip=true \
+ -DskipShade -Dtest=NoUnitTests -Dmaven.javadoc.skip=true
-Dcheckstyle.skip=true \
-Dspotbugs.skip=true ${extra[*]}
count=$("${GREP}" -c '\[ERROR\]' "${logfile}")
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index a344b562b951..3086e7496998 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -169,7 +169,7 @@
-Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError
-3.2.5
+3.0.0-M1
${maven-surefire-plugin.version}
${maven-surefire-plugin.version}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) 01/01: Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#653…"
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch revert-6537-HADOOP-19071
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit c8412c82fe249d9ef556a57a2235f3d733b569b7
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Fri Feb 23 14:18:56 2024 +0800
Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5.
(#653…"
This reverts commit 555faf28ce365443c17767adf0d6d570cef32ee3.
---
dev-support/bin/hadoop.sh | 3 +--
hadoop-project/pom.xml| 2 +-
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/dev-support/bin/hadoop.sh b/dev-support/bin/hadoop.sh
index c60603334feb..fd9363e66bbd 100755
--- a/dev-support/bin/hadoop.sh
+++ b/dev-support/bin/hadoop.sh
@@ -577,7 +577,6 @@ function shadedclient_rebuild
extra=(
"-Dtest=NoUnitTests"
-"-Dsurefire.failIfNoSpecifiedTests=false"
"-Dmaven.javadoc.skip=true"
"-Dcheckstyle.skip=true"
"-Dspotbugs.skip=true"
@@ -615,7 +614,7 @@ function shadedclient_rebuild
echo_and_redirect "${logfile}" \
"${MAVEN}" "${MAVEN_ARGS[@]}" verify -fae --batch-mode -am \
"${modules[@]}" \
- -DskipShade -Dtest=NoUnitTests -Dsurefire.failIfNoSpecifiedTests=false
-Dmaven.javadoc.skip=true -Dcheckstyle.skip=true \
+ -DskipShade -Dtest=NoUnitTests -Dmaven.javadoc.skip=true
-Dcheckstyle.skip=true \
-Dspotbugs.skip=true ${extra[*]}
count=$("${GREP}" -c '\[ERROR\]' "${logfile}")
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index a344b562b951..3086e7496998 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -169,7 +169,7 @@
-Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError
-3.2.5
+3.0.0-M1
${maven-surefire-plugin.version}
${maven-surefire-plugin.version}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch revert-6537-HADOOP-19071 created (now c8412c82fe24)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to branch revert-6537-HADOOP-19071 in repository https://gitbox.apache.org/repos/asf/hadoop.git at c8412c82fe24 Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#653…" This branch includes the following new commits: new c8412c82fe24 Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#653…" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HDFS-17146.Use the dfsadmin -reconfig command to initiate reconfiguration on all decommissioning datanodes. (#6504) Contributed by Hualong Zhang.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 9751b6e41a71 HDFS-17146.Use the dfsadmin -reconfig command to initiate
reconfiguration on all decommissioning datanodes. (#6504) Contributed by
Hualong Zhang.
9751b6e41a71 is described below
commit 9751b6e41a719c39c90b776dfe23d7d61cd5b05d
Author: zhtttylz
AuthorDate: Sun Feb 18 09:15:40 2024 +0800
HDFS-17146.Use the dfsadmin -reconfig command to initiate reconfiguration
on all decommissioning datanodes. (#6504) Contributed by Hualong Zhang.
Reviewed-by: Shilun Fan
Signed-off-by: Shilun Fan
---
.../org/apache/hadoop/hdfs/tools/DFSAdmin.java | 67 +--
.../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 4 +-
.../hadoop-hdfs/src/site/markdown/HdfsUserGuide.md | 12 +--
.../org/apache/hadoop/hdfs/tools/TestDFSAdmin.java | 98 ++
4 files changed, 147 insertions(+), 34 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
index ad4292a6b79f..b57c146023c6 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
@@ -35,6 +35,7 @@ import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
@@ -453,7 +454,8 @@ public class DFSAdmin extends FsShell {
"\t[-refreshSuperUserGroupsConfiguration]\n" +
"\t[-refreshCallQueue]\n" +
"\t[-refresh [arg1..argn]\n" +
- "\t[-reconfig
]\n" +
+ "\t[-reconfig
\n" +
+"\t]\n" +
"\t[-printTopology]\n" +
"\t[-refreshNamenodes datanode_host:ipc_port]\n" +
"\t[-getVolumeReport datanode_host:ipc_port]\n" +
@@ -1251,14 +1253,15 @@ public class DFSAdmin extends FsShell {
String refreshCallQueue = "-refreshCallQueue: Reload the call queue from
config\n";
-String reconfig = "-reconfig
" +
+String reconfig = "-reconfig
" +
":\n" +
"\tStarts or gets the status of a reconfiguration operation, \n" +
"\tor gets a list of reconfigurable properties.\n" +
"\tThe second parameter specifies the node type\n" +
"\tThe third parameter specifies host address. For start or status,
\n" +
-"\tdatanode supports livenodes as third parameter, which will start
\n" +
-"\tor retrieve reconfiguration on all live datanodes.";
+"\tdatanode supports livenodes and decomnodes as the third parameter,
\n" +
+"\twhich will start or retrieve reconfiguration on all live " +
+"\tor decommissioning datanodes. \n";
String genericRefresh = "-refresh: Arguments are " +
" [arg1..argn]\n" +
"\tTriggers a runtime-refresh of the resource specified by " +
@@ -1928,14 +1931,14 @@ public class DFSAdmin extends FsShell {
return -1;
}
- int startReconfiguration(final String nodeThpe, final String address)
+ int startReconfiguration(final String nodeType, final String address)
throws IOException, InterruptedException {
-return startReconfigurationUtil(nodeThpe, address, System.out, System.err);
+return startReconfigurationUtil(nodeType, address, System.out, System.err);
}
int startReconfigurationUtil(final String nodeType, final String address,
final PrintStream out,
final PrintStream err) throws IOException, InterruptedException {
-if (!"livenodes".equals(address)) {
+if (!"livenodes".equals(address) && !"decomnodes".equals(address)) {
return startReconfiguration(nodeType, address, out, err);
}
if (!"datanode".equals(nodeType)) {
@@ -1944,23 +1947,28 @@ public class DFSAdmin extends FsShell {
}
ExecutorService executorService = Executors.newFixedThreadPool(5);
DistributedFileSystem dfs = getDFS();
-DatanodeInfo[] nodes = dfs.getDataNodeStats(DatanodeReportType.LIVE);
+final DatanodeInfo[] nodes = "livenodes".equals(address) ?
+dfs.getDataNodeStats(DatanodeReportType.LIVE) :
+dfs.getDataNodeStats(DatanodeReportType.DECOMMISSIONING);
AtomicInteger successCount = new AtomicInteger();
AtomicInteger failCount = ne
svn commit: r67299 [3/5] - /dev/hadoop/hadoop-3.4.0-RC2/
Added: dev/hadoop/hadoop-3.4.0-RC2/CHANGELOG.md.asc
==
Binary file - no diff available.
Propchange: dev/hadoop/hadoop-3.4.0-RC2/CHANGELOG.md.asc
--
svn:mime-type = application/pgp-signature
Added: dev/hadoop/hadoop-3.4.0-RC2/CHANGELOG.md.sha512
==
--- dev/hadoop/hadoop-3.4.0-RC2/CHANGELOG.md.sha512 (added)
+++ dev/hadoop/hadoop-3.4.0-RC2/CHANGELOG.md.sha512 Mon Feb 12 12:59:33 2024
@@ -0,0 +1 @@
+SHA512 (CHANGELOG.md) =
f824ba0797a994405e82d3dda4ec157ab13a18077a55579aff5076a272b79d3bd32093f699ee131d488773adb45aa40949af07dae78061082bd840946d22c24a
Added: dev/hadoop/hadoop-3.4.0-RC2/RELEASENOTES.md
==
--- dev/hadoop/hadoop-3.4.0-RC2/RELEASENOTES.md (added)
+++ dev/hadoop/hadoop-3.4.0-RC2/RELEASENOTES.md Mon Feb 12 12:59:33 2024
@@ -0,0 +1,637 @@
+
+
+# Apache Hadoop 3.4.0 Release Notes
+
+These release notes cover new developer and user-facing incompatibilities,
important issues, features, and major improvements.
+
+
+---
+
+* [HADOOP-16054](https://issues.apache.org/jira/browse/HADOOP-16054) | *Major*
| **Update Dockerfile to use Bionic**
+
+The build image has been upgraded to Bionic.
+
+
+---
+
+* [HDFS-15281](https://issues.apache.org/jira/browse/HDFS-15281) | *Major* |
**ZKFC ignores dfs.namenode.rpc-bind-host and uses dfs.namenode.rpc-address to
bind to host address**
+
+ZKFC binds host address to "dfs.namenode.servicerpc-bind-host", if configured.
Otherwise, it binds to "dfs.namenode.rpc-bind-host". If neither of those is
configured, ZKFC binds itself to NameNode RPC server address (effectively
"dfs.namenode.rpc-address").
+
+
+---
+
+* [HADOOP-17010](https://issues.apache.org/jira/browse/HADOOP-17010) | *Major*
| **Add queue capacity weights support in FairCallQueue**
+
+When FairCallQueue is enabled, user can specify capacity allocation among all
sub-queues via configuration âipc.\.callqueue.capacity.weightsâ. The
value of this config is a comma-separated list of positive integers, each of
which specifies the weight associated with the sub-queue at that index. This
list length should be IPC scheduler priority levels, defined by
"scheduler.priority.levels".
+
+By default, each sub-queue is associated with weight 1, i.e., all sub-queues
are allocated with the same capacity.
+
+
+---
+
+* [HADOOP-17024](https://issues.apache.org/jira/browse/HADOOP-17024) | *Major*
| **ListStatus on ViewFS root (ls "/") should list the linkFallBack root
(configured target root).**
+
+ViewFS#listStatus on root("/") considers listing from fallbackLink if
available. If the same directory name is present in configured mount path as
well as in fallback link, then only the configured mount path will be listed in
the returned result.
+
+
+---
+
+* [HDFS-15288](https://issues.apache.org/jira/browse/HDFS-15288) | *Major* |
**Add Available Space Rack Fault Tolerant BPP**
+
+Added a new BlockPlacementPolicy:
"AvailableSpaceRackFaultTolerantBlockPlacementPolicy" which uses the same
optimization logic as the AvailableSpaceBlockPlacementPolicy along with
spreading the replicas across maximum number of racks, similar to
BlockPlacementPolicyRackFaultTolerant.
+The BPP can be configured by setting the blockplacement policy class as
org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceRackFaultTolerantBlockPlacementPolicy
+
+
+---
+
+* [HDFS-13183](https://issues.apache.org/jira/browse/HDFS-13183) | *Major* |
**Standby NameNode process getBlocks request to reduce Active load**
+
+Enable balancer to redirect getBlocks request to a Standby Namenode, thus
reducing the performance impact of balancer to the Active NameNode.
+
+The feature is disabled by default. To enable it, configure the hdfs-site.xml
of balancer:
+dfs.ha.allow.stale.reads = true.
+
+
+---
+
+* [HADOOP-17079](https://issues.apache.org/jira/browse/HADOOP-17079) | *Major*
| **Optimize UGI#getGroups by adding UGI#getGroupsSet**
+
+Added a UserGroupMapping#getGroupsSet() API and deprecate
UserGroupMapping#getGroups.
+
+The UserGroupMapping#getGroups() can be expensive as it involves Set-\>List
conversion. For user with large group membership (i.e., \> 1000 groups), we
recommend using getGroupSet to avoid the conversion and fast membership look up.
+
+
+---
+
+* [HDFS-15385](https://issues.apache.org/jira/browse/HDFS-15385) | *Critical*
| **Upgrade boost library to 1.72**
+
+Boost 1.72 is required when building native code.
+
+
+---
+
+* [HADOOP-17091](https://issues.apache.org/jira/browse/HADOOP-17091) | *Major*
| **[JDK11] Fix Javadoc errors**
+
+
+* Upgraded to Yetus master to change javadoc goals.
+* Changed javadoc goals from `javadoc:javadoc` to `process-sources
javadoc:javadoc-no-fork`.
+* Javadoc with JDK11 fails in
svn commit: r67299 [4/5] - /dev/hadoop/hadoop-3.4.0-RC2/
Added: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-rat.txt == --- dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-rat.txt (added) +++ dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-rat.txt Mon Feb 12 12:59:33 2024 @@ -0,0 +1,17723 @@ + +* +Summary +--- +Generated at: 2024-02-12T09:26:33+00:00 + +Notes: 0 +Binaries: 0 +Archives: 0 +Standards: 11 + +Apache Licensed: 11 +Generated Documents: 0 + +JavaDocs are generated, thus a license header is optional. +Generated files do not require license headers. + +0 Unknown Licenses + +* + Files with Apache License headers will be marked AL + Binary files (which do not require any license headers) will be marked B + Compressed archives will be marked A + Notices, licenses etc. will be marked N + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/VisibleForTesting.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceAudience.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/package-info.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/ExcludePrivateAnnotationsJDiffDoclet.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/StabilityOptions.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/IncludePublicAnnotationsJDiffDoclet.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/IncludePublicAnnotationsStandardDoclet.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/ExcludePrivateAnnotationsStandardDoclet.java + AL /build/source/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceStability.java + AL/build/source/hadoop-common-project/hadoop-annotations/pom.xml + +* + +* +Summary +--- +Generated at: 2024-02-12T09:26:33+00:00 + +Notes: 2 +Binaries: 0 +Archives: 0 +Standards: 69 + +Apache Licensed: 69 +Generated Documents: 0 + +JavaDocs are generated, thus a license header is optional. +Generated files do not require license headers. + +0 Unknown Licenses + +* + Files with Apache License headers will be marked AL + Binary files (which do not require any license headers) will be marked B + Compressed archives will be marked A + Notices, licenses etc. will be marked N + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticationException.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/ConnectionConfigurator.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/PseudoAuthenticator.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/AuthenticatedURL.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/Authenticator.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AltKerberosAuthenticationHandler.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/package-info.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationHandlerUtil.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/CompositeAuthenticationHandler.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/MultiSchemeAuthenticationHandler.java + AL /build/source/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationHandler.java + AL
svn commit: r67299 [5/5] - /dev/hadoop/hadoop-3.4.0-RC2/
Added: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-rat.txt.asc == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-rat.txt.asc -- svn:mime-type = application/pgp-signature Added: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-rat.txt.sha512 == --- dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-rat.txt.sha512 (added) +++ dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-rat.txt.sha512 Mon Feb 12 12:59:33 2024 @@ -0,0 +1 @@ +SHA512 (hadoop-3.4.0-rat.txt) = 65114ca87697dbabaffb27a7ea1ba8b91b60f578dec323ab802cb5b839bd0b1438a64d7fbfaba2396adb04001c36939f494fa7760426f0b8b717740612e94c7f Added: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-site.tar.gz == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-site.tar.gz -- svn:mime-type = application/x-gzip Added: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-site.tar.gz.asc == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-site.tar.gz.asc -- svn:mime-type = application/pgp-signature Added: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-site.tar.gz.sha512 == --- dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-site.tar.gz.sha512 (added) +++ dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-site.tar.gz.sha512 Mon Feb 12 12:59:33 2024 @@ -0,0 +1 @@ +SHA512 (hadoop-3.4.0-site.tar.gz) = 72c1f0d20e696711c838b555a00772bc84436f314838fbbb78becebba510e3e949a4b7edb268dbbd3089d5b62dc883cbdbed0151b15d68cc262a55697b524ec1 Added: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-src.tar.gz == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-src.tar.gz -- svn:mime-type = application/x-gzip Added: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-src.tar.gz.asc == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-src.tar.gz.asc -- svn:mime-type = application/pgp-signature Added: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-src.tar.gz.sha512 == --- dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-src.tar.gz.sha512 (added) +++ dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-src.tar.gz.sha512 Mon Feb 12 12:59:33 2024 @@ -0,0 +1 @@ +SHA512 (hadoop-3.4.0-src.tar.gz) = b5a4c93caa6ded0360427ad9fd87d849d28fa4333f935259a0374b8e3feff5ce990c7a7b493ba87674f088a77a4f3d8d722d2d542c867789de6650c6ca7bee3a Added: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0.tar.gz == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0.tar.gz -- svn:mime-type = application/x-gzip Added: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0.tar.gz.asc == Binary file - no diff available. Propchange: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0.tar.gz.asc -- svn:mime-type = application/pgp-signature Added: dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0.tar.gz.sha512 == --- dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0.tar.gz.sha512 (added) +++ dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0.tar.gz.sha512 Mon Feb 12 12:59:33 2024 @@ -0,0 +1 @@ +SHA512 (hadoop-3.4.0.tar.gz) = ebc6bdf8a9cbe487ce06c4a81ed0cd709254de10cec1b47f968eb89baf6e2dcd49425f7d261698450f03382d45d0e96b6c675fa60eb9b2df78e71bbd695146f0 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
svn commit: r67299 [2/5] - /dev/hadoop/hadoop-3.4.0-RC2/
Added: dev/hadoop/hadoop-3.4.0-RC2/CHANGELOG.md
==
--- dev/hadoop/hadoop-3.4.0-RC2/CHANGELOG.md (added)
+++ dev/hadoop/hadoop-3.4.0-RC2/CHANGELOG.md Mon Feb 12 12:59:33 2024
@@ -0,0 +1,2950 @@
+
+
+# Apache Hadoop Changelog
+
+## Release 3.4.0 - 2024-02-12
+
+
+
+### IMPORTANT ISSUES:
+
+| JIRA | Summary | Priority | Component | Reporter | Contributor |
+|: |: | :--- |: |: |: |
+| [HDFS-15380](https://issues.apache.org/jira/browse/HDFS-15380) | RBF: Could
not fetch real remote IP in RouterWebHdfsMethods | Major | webhdfs | Tao Li |
Tao Li |
+| [HDFS-15814](https://issues.apache.org/jira/browse/HDFS-15814) | Make some
parameters configurable for DataNodeDiskMetrics | Major | hdfs | Tao Li | Tao
Li |
+| [HDFS-16265](https://issues.apache.org/jira/browse/HDFS-16265) | Refactor
HDFS tool tests for better reuse | Blocker | hdfs-client, libhdfs++, tools |
Gautham Banasandra | Gautham Banasandra |
+| [HADOOP-17956](https://issues.apache.org/jira/browse/HADOOP-17956) | Replace
all default Charset usage with UTF-8 | Major | common | Viraj Jasani | Viraj
Jasani |
+| [HDFS-16278](https://issues.apache.org/jira/browse/HDFS-16278) | Make HDFS
snapshot tools cross platform | Major | hdfs-client, libhdfs++, tools |
Gautham Banasandra | Gautham Banasandra |
+| [HDFS-16285](https://issues.apache.org/jira/browse/HDFS-16285) | Make HDFS
ownership tools cross platform | Major | hdfs-client, libhdfs++, tools |
Gautham Banasandra | Gautham Banasandra |
+| [HDFS-16419](https://issues.apache.org/jira/browse/HDFS-16419) | Make HDFS
data transfer tools cross platform | Major | hdfs-client, libhdfs++, tools |
Gautham Banasandra | Gautham Banasandra |
+| [HDFS-16511](https://issues.apache.org/jira/browse/HDFS-16511) | Improve
lock type for ReplicaMap under fine-grain lock mode. | Major | hdfs |
Mingxiang Li | Mingxiang Li |
+| [HDFS-16534](https://issues.apache.org/jira/browse/HDFS-16534) | Split
datanode block pool locks to volume grain. | Major | datanode | Mingxiang Li |
Mingxiang Li |
+| [HADOOP-18219](https://issues.apache.org/jira/browse/HADOOP-18219) | Fix
shadedclient test failure | Blocker | test | Gautham Banasandra | Akira
Ajisaka |
+| [HADOOP-18621](https://issues.apache.org/jira/browse/HADOOP-18621) |
CryptoOutputStream::close leak when encrypted zones + quota exceptions |
Critical | fs | Colm Dougan | Colm Dougan |
+| [YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) | YARN
Federation improvements | Major | federation | Subramaniam Krishnan |
Subramaniam Krishnan |
+
+
+### NEW FEATURES:
+
+| JIRA | Summary | Priority | Component | Reporter | Contributor |
+|: |: | :--- |: |: |: |
+| [HADOOP-17010](https://issues.apache.org/jira/browse/HADOOP-17010) | Add
queue capacity weights support in FairCallQueue | Major | ipc | Fengnan Li |
Fengnan Li |
+| [HDFS-15288](https://issues.apache.org/jira/browse/HDFS-15288) | Add
Available Space Rack Fault Tolerant BPP | Major | block placement | Ayush
Saxena | Ayush Saxena |
+| [HDFS-13183](https://issues.apache.org/jira/browse/HDFS-13183) | Standby
NameNode process getBlocks request to reduce Active load | Major | balancer &
mover, namenode | Xiaoqiao He | Xiaoqiao He |
+| [HDFS-15463](https://issues.apache.org/jira/browse/HDFS-15463) | Add a tool
to validate FsImage | Major | namenode | Tsz-wo Sze | Tsz-wo Sze |
+| [HADOOP-17165](https://issues.apache.org/jira/browse/HADOOP-17165) |
Implement service-user feature in DecayRPCScheduler | Major | rpc-server |
Takanobu Asanuma | Takanobu Asanuma |
+| [HADOOP-15891](https://issues.apache.org/jira/browse/HADOOP-15891) | Provide
Regex Based Mount Point In Inode Tree | Major | viewfs | zhenzhao wang |
zhenzhao wang |
+| [HDFS-15025](https://issues.apache.org/jira/browse/HDFS-15025) | Applying
NVDIMM storage media to HDFS | Major | datanode, hdfs | YaYun Wang | YaYun
Wang |
+| [HDFS-15098](https://issues.apache.org/jira/browse/HDFS-15098) | Add SM4
encryption method for HDFS | Major | hdfs | liusheng | liusheng |
+| [HADOOP-17125](https://issues.apache.org/jira/browse/HADOOP-17125) | Using
snappy-java in SnappyCodec | Major | common | DB Tsai | L. C. Hsieh |
+| [HDFS-15294](https://issues.apache.org/jira/browse/HDFS-15294) | Federation
balance tool | Major | rbf, tools | Jinglun | Jinglun |
+| [HADOOP-17292](https://issues.apache.org/jira/browse/HADOOP-17292) | Using
lz4-java in Lz4Codec | Major | common | L. C. Hsieh | L. C. Hsieh |
+| [HDFS-14090](https://issues.apache.org/jira/browse/HDFS-14090) | RBF:
Improved isolation for downstream name nodes. {Static} | Major | rbf | CR Hota
| Fengnan Li |
+| [HDFS-15711](https://issues.apache.org/jira/browse/HDFS-15711) | Add Metrics
to HttpFS Server | Major | httpfs | Ahmed Hussein | Ahmed Hussein |
+| [HADOOP-16492](https://issues.apache.org/jira/browse/HADOOP-16492) | Support
HuaweiCloud Object
svn commit: r67299 [1/5] - /dev/hadoop/hadoop-3.4.0-RC2/
Author: slfan1989 Date: Mon Feb 12 12:59:33 2024 New Revision: 67299 Log: HADOOP-19018. Hadoop-3.4.0-RC2 built from 88fbe62f27e Added: dev/hadoop/hadoop-3.4.0-RC2/ dev/hadoop/hadoop-3.4.0-RC2/CHANGELOG.md dev/hadoop/hadoop-3.4.0-RC2/CHANGELOG.md.asc (with props) dev/hadoop/hadoop-3.4.0-RC2/CHANGELOG.md.sha512 dev/hadoop/hadoop-3.4.0-RC2/RELEASENOTES.md dev/hadoop/hadoop-3.4.0-RC2/RELEASENOTES.md.asc (with props) dev/hadoop/hadoop-3.4.0-RC2/RELEASENOTES.md.sha512 dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-aarch64.tar.gz (with props) dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-aarch64.tar.gz.asc (with props) dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-aarch64.tar.gz.sha512 dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-rat.txt dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-rat.txt.asc (with props) dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-rat.txt.sha512 dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-site.tar.gz (with props) dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-site.tar.gz.asc (with props) dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-site.tar.gz.sha512 dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-src.tar.gz (with props) dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-src.tar.gz.asc (with props) dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0-src.tar.gz.sha512 dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0.tar.gz (with props) dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0.tar.gz.asc (with props) dev/hadoop/hadoop-3.4.0-RC2/hadoop-3.4.0.tar.gz.sha512 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
svn commit: r67298 - /dev/hadoop/hadoop-3.4.0-RC1/
Author: slfan1989 Date: Mon Feb 12 12:51:04 2024 New Revision: 67298 Log: delete staged hadoop/3.4.0-RC1 Removed: dev/hadoop/hadoop-3.4.0-RC1/ - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) annotated tag release-3.4.0-RC2 updated (88fbe62f27e8 -> 3e0acbd372c9)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to annotated tag release-3.4.0-RC2 in repository https://gitbox.apache.org/repos/asf/hadoop.git *** WARNING: tag release-3.4.0-RC2 was modified! *** from 88fbe62f27e8 (commit) to 3e0acbd372c9 (tag) tagging 88fbe62f27e859feda7f8d876118707993e85641 (commit) replaces release-3.4.0-RC1 by Shilun Fan on Mon Feb 12 16:36:05 2024 +0800 - Log - Release candidate - 3.4.0-RC2 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEECm40PGXwzjlsDKJKl4RW45ZNtwEFAmXJ2HUACgkQl4RW45ZN twGpLw//X89CYUA1M1DbY++J2haR2q2PC9lLN8PXsaNxafx+Hq8cu6K0Njm0x/so Jgk/R1nLWom0+xx5BG0M0VzrjNOrnEFdWfevCLahtdbqVsxn8G4dm0ze2SCF9zOO YvLeL2v8ALnw4W7NGmZOG9ZwsvTqslRrNLNZjcf6nTg4nzu066gndTktFVIxOXsi KEmz4PmIdXIIGQ9s1S2x0dMQmAX6N5d3YQpomcukrag5FHQhk9yLVv/C2nX4qH+r wDcaJBJd5Gz5jAUk8MYLWxcZhTC+JCQDUWBc6jGDQScVFN4tgM6aouhOrDDb96DC jTeJbQmqFaCi3Lq+G9vcsR0J+NWWJNIAKCc4Ohyu/xqBjo6MlUYx1CX2v6koNrhe LhFFXLq6ebdu3ISaoffAgbmzjNbpF9lg3QbM9Lz+nEIsPW3781ijIkhymsamgK3+ 36GscDFHBbHgJY/thkbHVHFqKUUc6/c7JP0S8kaHS/ykoSn6h5jlUwEZ4yMEI2On 5kiCV7w+J67eHnT1blQ2zjGWDjd/GVR/Jn5bqs43oPCpd+kgPNo0nvotRjxAgtnu bUGP2e7/3PoU1Ur/WvnknV4HxVFJQDiC6M/QZArhqIhMY5aaPqqcKb7h45n+L/MB vm394Dc05xVLMQqJImCIbt1rvgZ0seXGOI1lB4y/Eb9YIWFBnAc= =gyyS -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#6537) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 555faf28ce36 HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to
3.2.5. (#6537) Contributed by Shilun Fan
555faf28ce36 is described below
commit 555faf28ce365443c17767adf0d6d570cef32ee3
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Sun Feb 11 07:41:46 2024 +0800
HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#6537)
Contributed by Shilun Fan
Reviewed-by: Steve Loughran
Signed-off-by: Shilun Fan
---
dev-support/bin/hadoop.sh | 3 ++-
hadoop-project/pom.xml| 2 +-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/dev-support/bin/hadoop.sh b/dev-support/bin/hadoop.sh
index fd9363e66bbd..c60603334feb 100755
--- a/dev-support/bin/hadoop.sh
+++ b/dev-support/bin/hadoop.sh
@@ -577,6 +577,7 @@ function shadedclient_rebuild
extra=(
"-Dtest=NoUnitTests"
+"-Dsurefire.failIfNoSpecifiedTests=false"
"-Dmaven.javadoc.skip=true"
"-Dcheckstyle.skip=true"
"-Dspotbugs.skip=true"
@@ -614,7 +615,7 @@ function shadedclient_rebuild
echo_and_redirect "${logfile}" \
"${MAVEN}" "${MAVEN_ARGS[@]}" verify -fae --batch-mode -am \
"${modules[@]}" \
- -DskipShade -Dtest=NoUnitTests -Dmaven.javadoc.skip=true
-Dcheckstyle.skip=true \
+ -DskipShade -Dtest=NoUnitTests -Dsurefire.failIfNoSpecifiedTests=false
-Dmaven.javadoc.skip=true -Dcheckstyle.skip=true \
-Dspotbugs.skip=true ${extra[*]}
count=$("${GREP}" -c '\[ERROR\]' "${logfile}")
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index e457797e3ad8..c732837dd17f 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -169,7 +169,7 @@
-Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError
-3.0.0-M1
+3.2.5
${maven-surefire-plugin.version}
${maven-surefire-plugin.version}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new 88fbe62f27e8 HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533)
Contributed by Shilun Fan
88fbe62f27e8 is described below
commit 88fbe62f27e859feda7f8d876118707993e85641
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Thu Feb 8 19:18:04 2024 +0800
HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533) Contributed by Shilun Fan
Reviewed-by: He Xiaoqiao
Signed-off-by: Shilun Fan
---
hadoop-common-project/hadoop-common/pom.xml | 2 +-
hadoop-project/pom.xml | 6 +++---
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml | 6 +++---
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index a18fdbf37f8f..d3b91de7a826 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -40,7 +40,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
org.apache.hadoop
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 2ef800e2d348..accaa2efc8f3 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -93,10 +93,10 @@
${common.protobuf2.scope}
-3.7.1
+3.21.12
${env.HADOOP_PROTOC_PATH}
-1.1.1
+1.2.0
${hadoop-thirdparty.version}
${hadoop-thirdparty.version}
org.apache.hadoop.thirdparty
@@ -250,7 +250,7 @@
org.apache.hadoop.thirdparty
-hadoop-shaded-protobuf_3_7
+hadoop-shaded-protobuf_3_21
${hadoop-thirdparty-protobuf.version}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
index 86e9b2da8e3b..ac85bead0f05 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
@@ -51,7 +51,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
@@ -69,7 +69,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
@@ -80,7 +80,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new e6eeaa05abdd HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533)
Contributed by Shilun Fan
e6eeaa05abdd is described below
commit e6eeaa05abddfcd4a55077c4ce9f71ea68cc3248
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Thu Feb 8 19:18:04 2024 +0800
HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533) Contributed by Shilun Fan
Reviewed-by: He Xiaoqiao
Signed-off-by: Shilun Fan
---
hadoop-common-project/hadoop-common/pom.xml | 2 +-
hadoop-project/pom.xml | 6 +++---
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml | 6 +++---
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index e1d1683d7278..86ae00ead72a 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -40,7 +40,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
org.apache.hadoop
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 3fc37667e1cc..a189514b9fb8 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -93,10 +93,10 @@
${common.protobuf2.scope}
-3.7.1
+3.21.12
${env.HADOOP_PROTOC_PATH}
-1.1.1
+1.2.0
${hadoop-thirdparty.version}
${hadoop-thirdparty.version}
org.apache.hadoop.thirdparty
@@ -250,7 +250,7 @@
org.apache.hadoop.thirdparty
-hadoop-shaded-protobuf_3_7
+hadoop-shaded-protobuf_3_21
${hadoop-thirdparty-protobuf.version}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
index 9bc1f1737a5a..ea01ec9cce7f 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
@@ -51,7 +51,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
@@ -69,7 +69,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
@@ -80,7 +80,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-18980. S3A credential provider remapping: make extensible (#6406)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new f4a44d2b3562 HADOOP-18980. S3A credential provider remapping: make
extensible (#6406)
f4a44d2b3562 is described below
commit f4a44d2b35622d0432c763633b4c957e87143e6f
Author: Viraj Jasani
AuthorDate: Fri Feb 9 05:29:40 2024 -0900
HADOOP-18980. S3A credential provider remapping: make extensible (#6406)
Contributed by Viraj Jasani
---
.../java/org/apache/hadoop/conf/Configuration.java | 4 +-
.../java/org/apache/hadoop/util/StringUtils.java | 40 +-
.../org/apache/hadoop/util/TestStringUtils.java| 56 +
.../java/org/apache/hadoop/fs/s3a/Constants.java | 7 ++
.../java/org/apache/hadoop/fs/s3a/S3AUtils.java| 23
.../fs/s3a/auth/CredentialProviderListFactory.java | 8 ++
.../markdown/tools/hadoop-aws/aws_sdk_upgrade.md | 49
.../src/site/markdown/tools/hadoop-aws/index.md| 22
.../fs/s3a/ITestS3AAWSCredentialsProvider.java | 31 +
.../fs/s3a/TestS3AAWSCredentialsProvider.java | 139 +
10 files changed, 376 insertions(+), 3 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 7c4f617b179e..ea3d6dc74e4a 100755
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -2339,8 +2339,8 @@ public class Configuration implements
Iterable>,
}
return StringUtils.getTrimmedStringCollection(valueString);
}
-
- /**
+
+ /**
* Get the comma delimited values of the name property as
* an array of Strings, trimmed of the leading and trailing
whitespace.
* If no such property is specified then an empty array is returned.
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
index 3debd36da78d..b8d999162d38 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
@@ -25,6 +25,7 @@ import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
+import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedHashSet;
import java.util.List;
@@ -479,7 +480,28 @@ public class StringUtils {
set.remove("");
return set;
}
-
+
+ /**
+ * Splits an "=" separated value String, trimming leading and
+ * trailing whitespace on each value after splitting by comma and new line
separator.
+ *
+ * @param str a comma separated String with values, may be null
+ * @return a Map of String keys and values, empty
+ * Collection if null String input.
+ */
+ public static Map getTrimmedStringCollectionSplitByEquals(
+ String str) {
+String[] trimmedList = getTrimmedStrings(str);
+Map pairs = new HashMap<>();
+for (String s : trimmedList) {
+ String[] splitByKeyVal = getTrimmedStringsSplitByEquals(s);
+ if (splitByKeyVal.length == 2) {
+pairs.put(splitByKeyVal[0], splitByKeyVal[1]);
+ }
+}
+return pairs;
+ }
+
/**
* Splits a comma or newline separated value String, trimming
* leading and trailing whitespace on each value.
@@ -497,6 +519,22 @@ public class StringUtils {
return str.trim().split("\\s*[,\n]\\s*");
}
+ /**
+ * Splits "=" separated value String, trimming
+ * leading and trailing whitespace on each value.
+ *
+ * @param str an "=" separated String with values,
+ *may be null
+ * @return an array of String values, empty array if null String
+ * input
+ */
+ public static String[] getTrimmedStringsSplitByEquals(String str){
+if (null == str || str.trim().isEmpty()) {
+ return emptyStringArray;
+}
+return str.trim().split("\\s*=\\s*");
+ }
+
final public static String[] emptyStringArray = {};
final public static char COMMA = ',';
final public static String COMMA_STR = ",";
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
index f05b58956760..d9bcf5842689 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
+++
b/hadoop-common-
(hadoop) branch branch-3.4.0 updated: HADOOP-19059. S3A: Update AWS Java SDK to 2.23.19 (#6538)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.0 by this push: new 893b9efb5398 HADOOP-19059. S3A: Update AWS Java SDK to 2.23.19 (#6538) 893b9efb5398 is described below commit 893b9efb5398d841af0b17b6835245e37fa5b2c5 Author: Adnan Hemani AuthorDate: Thu Feb 8 12:38:37 2024 -0800 HADOOP-19059. S3A: Update AWS Java SDK to 2.23.19 (#6538) Contributed by Adnan Hemani --- LICENSE-binary | 2 +- hadoop-project/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 93e38cc34ee4..c5ba9f938aef 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -363,7 +363,7 @@ org.objenesis:objenesis:2.6 org.xerial.snappy:snappy-java:1.1.10.4 org.yaml:snakeyaml:2.0 org.wildfly.openssl:wildfly-openssl:1.1.3.Final -software.amazon.awssdk:bundle:jar:2.23.5 +software.amazon.awssdk:bundle:jar:2.23.19 diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 5fbcf409f352..2ef800e2d348 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -187,7 +187,7 @@ 1.0-beta-1 900 1.12.599 -2.23.5 +2.23.19 1.0.1 2.7.1 1.11.2 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-18993. Add option fs.s3a.classloader.isolation (#6301)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new 836f0aeadc2c HADOOP-18993. Add option fs.s3a.classloader.isolation
(#6301)
836f0aeadc2c is described below
commit 836f0aeadc2cd12baf54c50cb34c9b1afa2303d4
Author: Antonio Murgia
AuthorDate: Wed Feb 7 15:30:19 2024 +0100
HADOOP-18993. Add option fs.s3a.classloader.isolation (#6301)
The option fs.s3a.classloader.isolation (default: true) can be set to false
to disable s3a classloader isolation;
This can assist in using custom credential providers and other extension
points.
Contributed by Antonio Murgia
---
.../java/org/apache/hadoop/fs/s3a/Constants.java | 17 +++
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java| 4 +-
.../java/org/apache/hadoop/fs/s3a/S3AUtils.java| 20 +++
.../src/site/markdown/tools/hadoop-aws/index.md| 24
.../s3a/ITestS3AFileSystemIsolatedClassloader.java | 134 +
5 files changed, 197 insertions(+), 2 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
index 744146ccf4f3..9ed6594244b9 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
@@ -1600,4 +1600,21 @@ public final class Constants {
*/
public static final boolean CHECKSUM_VALIDATION_DEFAULT = false;
+ /**
+ * Are extensions classes, such as {@code fs.s3a.aws.credentials.provider},
+ * going to be loaded from the same classloader that loaded
+ * the {@link S3AFileSystem}?
+ * It is useful to turn classloader isolation off for Apache Spark
applications
+ * that might load {@link S3AFileSystem} from the Spark distribution
(Launcher classloader)
+ * while users might want to provide custom extensions (loaded by Spark
MutableClassloader).
+ * Value: {@value}.
+ */
+ public static final String AWS_S3_CLASSLOADER_ISOLATION =
+"fs.s3a.classloader.isolation";
+
+ /**
+ * Default value for {@link #AWS_S3_CLASSLOADER_ISOLATION}.
+ * Value: {@value}.
+ */
+ public static final boolean DEFAULT_AWS_S3_CLASSLOADER_ISOLATION = true;
}
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
index de48c2df1569..3aec03766dac 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
@@ -559,8 +559,8 @@ public class S3AFileSystem extends FileSystem implements
StreamCapabilities,
// fix up the classloader of the configuration to be whatever
// classloader loaded this filesystem.
- // See: HADOOP-17372
- conf.setClassLoader(this.getClass().getClassLoader());
+ // See: HADOOP-17372 and follow-up on HADOOP-18993
+ S3AUtils.maybeIsolateClassloader(conf, this.getClass().getClassLoader());
// patch the Hadoop security providers
patchSecurityCredentialProviders(conf);
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
index 6a719739e720..cceaea6159d7 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
@@ -1670,4 +1670,24 @@ public final class S3AUtils {
return String.format("bytes=%d-%d", rangeStart, rangeEnd);
}
+ /**
+ * If classloader isolation is {@code true}
+ * (through {@link Constants#AWS_S3_CLASSLOADER_ISOLATION}) or not
+ * explicitly set, then the classLoader of the input configuration object
+ * will be set to the input classloader, otherwise nothing will happen.
+ * @param conf configuration object.
+ * @param classLoader isolated classLoader.
+ */
+ static void maybeIsolateClassloader(Configuration conf, ClassLoader
classLoader) {
+if (conf.getBoolean(Constants.AWS_S3_CLASSLOADER_ISOLATION,
+Constants.DEFAULT_AWS_S3_CLASSLOADER_ISOLATION)) {
+ LOG.debug("Configuration classloader set to S3AFileSystem classloader:
{}", classLoader);
+ conf.setClassLoader(classLoader);
+} else {
+ LOG.debug("Configuration classloader not changed, support classes needed
will be loaded " +
+ "from the classloader that instantiated the
Configuration object: {}",
+ conf.getClassLoader());
+}
+ }
+
}
diff --git
a/hadoop-t
(hadoop) branch branch-3.4.0 updated: HADOOP-19045. CreateSession Timeout - followup (#6532)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.0 by this push: new c8ee14a2eb2f HADOOP-19045. CreateSession Timeout - followup (#6532) c8ee14a2eb2f is described below commit c8ee14a2eb2fd1ea435945b6740b77c73c132f8a Author: Steve Loughran AuthorDate: Wed Feb 7 12:07:54 2024 + HADOOP-19045. CreateSession Timeout - followup (#6532) This is a followup to PR: HADOOP-19045. S3A: Validate CreateSession Timeout Propagation (#6470) Remove all declarations of fs.s3a.connection.request.timeout in - hadoop-common/src/main/resources/core-default.xml - hadoop-aws/src/test/resources/core-site.xml New test in TestAwsClientConfig to verify that the value defined in fs.s3a.Constants class is used. This is brittle to someone overriding it in their test setups, but as this test is intended to verify that the option is not explicitly set, there's no workaround. Contributed by Steve Loughran --- .../hadoop-common/src/main/resources/core-default.xml| 14 -- .../src/site/markdown/tools/hadoop-aws/performance.md| 2 +- .../markdown/tools/hadoop-aws/troubleshooting_s3a.md | 2 +- .../apache/hadoop/fs/s3a/impl/TestAwsClientConfig.java | 16 hadoop-tools/hadoop-aws/src/test/resources/core-site.xml | 5 - 5 files changed, 18 insertions(+), 21 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml index 29ec06db6598..00533e362b99 100644 --- a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml +++ b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml @@ -2097,20 +2097,6 @@ - - fs.s3a.connection.request.timeout - 0s - -Time out on HTTP requests to the AWS service; 0 means no timeout. - -Important: this is the maximum duration of any AWS service call, -including upload and copy operations. If non-zero, it must be larger -than the time to upload multi-megabyte blocks to S3 from the client, -and to rename many-GB files. Use with care. - - - - fs.s3a.etag.checksum.enabled false diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md index 4c03cca17161..44c8086d4f02 100644 --- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md +++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md @@ -231,7 +231,7 @@ The choice is yours. Generally recovery is better, but sometimes fail-fast is mo | `fs.s3a.connection.acquisition.timeout` | `60s` | `*` | Timeout for waiting for a connection from the pool. | | `fs.s3a.connection.establish.timeout` | `30s` | | Time to establish the TCP/TLS connection | | `fs.s3a.connection.idle.time` | `60s` | `*` | Maximum time for idle HTTP connections in the pool| -| `fs.s3a.connection.request.timeout` | `0` | | If greater than zero, maximum duration of any request | +| `fs.s3a.connection.request.timeout` | `60s` | | If greater than zero, maximum time for a response | | `fs.s3a.connection.timeout` | `200s` | | Timeout for socket problems on a TCP channel | | `fs.s3a.connection.ttl` | `5m`| | Lifetime of HTTP connections from the pool| diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md index c1b499e3da95..e53e4a002265 100644 --- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md +++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md @@ -1510,7 +1510,7 @@ It is possible to configure a global timeout for AWS service calls using followi ```xml fs.s3a.connection.request.timeout - 0 + 5m Time out on HTTP requests to the AWS service; 0 means no timeout. Measured in seconds; the usual time suffixes are all supported diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestAwsClientConfig.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestAwsClientConfig.java index 2a9e05755ce6..eacff90ea4c8 100644 --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestAwsClientConfig.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestAwsClientConfig.java @@ -40,6 +40,7 @@ import static org.apache.hadoop.fs.s3a.Constants.DEFAULT_CONNECTION_KEEPALIVE; import static
(hadoop) branch branch-3.4.0 updated: HDFS-17370. Fix junit dependency for running parameterized tests in hadoop-hdfs-rbf (#6522)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.0 by this push: new 65674b977e7e HDFS-17370. Fix junit dependency for running parameterized tests in hadoop-hdfs-rbf (#6522) 65674b977e7e is described below commit 65674b977e7e9ed118f90e73982e399aabc08790 Author: Takanobu Asanuma AuthorDate: Tue Feb 6 11:54:14 2024 +0900 HDFS-17370. Fix junit dependency for running parameterized tests in hadoop-hdfs-rbf (#6522) (cherry picked from commit 9a7eeadaac818258b319cdb0dc19e9bb1e4fa11a) --- hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml | 5 + 1 file changed, 5 insertions(+) diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml b/hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml index 9ed29346b920..7c3da647f96c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml @@ -177,6 +177,11 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd;> junit-jupiter-params test + + org.junit.jupiter + junit-jupiter-engine + test + - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-19049. Fix StatisticsDataReferenceCleaner classloader leak (#6488)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new d6c66f76aeb1 HADOOP-19049. Fix StatisticsDataReferenceCleaner
classloader leak (#6488)
d6c66f76aeb1 is described below
commit d6c66f76aeb17a9033197b56b278f3dd5d33b67c
Author: Jia Fan
AuthorDate: Sat Feb 3 22:48:52 2024 +0800
HADOOP-19049. Fix StatisticsDataReferenceCleaner classloader leak (#6488)
Contributed by Jia Fan
---
.../src/main/java/org/apache/hadoop/fs/FileSystem.java | 1 +
.../org/apache/hadoop/fs/TestFileSystemStorageStatistics.java| 9 +
2 files changed, 10 insertions(+)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
index 0213772ab6a5..768fd5b5e1ca 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
@@ -4077,6 +4077,7 @@ public abstract class FileSystem extends Configured
STATS_DATA_CLEANER.
setName(StatisticsDataReferenceCleaner.class.getName());
STATS_DATA_CLEANER.setDaemon(true);
+ STATS_DATA_CLEANER.setContextClassLoader(null);
STATS_DATA_CLEANER.start();
}
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemStorageStatistics.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemStorageStatistics.java
index e99f0f2348b3..5710049afb10 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemStorageStatistics.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemStorageStatistics.java
@@ -34,6 +34,7 @@ import java.util.concurrent.TimeUnit;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
/**
* This tests basic operations of {@link FileSystemStorageStatistics} class.
@@ -102,6 +103,14 @@ public class TestFileSystemStorageStatistics {
}
}
+ @Test
+ public void testStatisticsDataReferenceCleanerClassLoader() {
+Thread thread = Thread.getAllStackTraces().keySet().stream()
+.filter(t ->
t.getName().contains("StatisticsDataReferenceCleaner")).findFirst().get();
+ClassLoader classLoader = thread.getContextClassLoader();
+assertNull(classLoader);
+ }
+
/**
* Helper method to retrieve the specific FileSystem.Statistics value by
name.
*
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-19044. S3A: AWS SDK V2 - Update region logic (#6479)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new fd0d0c90d964 HADOOP-19044. S3A: AWS SDK V2 - Update region logic
(#6479)
fd0d0c90d964 is described below
commit fd0d0c90d9646becdc4ca6e8098f97b89250f7ee
Author: Viraj Jasani
AuthorDate: Sat Feb 3 05:44:23 2024 -0900
HADOOP-19044. S3A: AWS SDK V2 - Update region logic (#6479)
Improves region handling in the S3A connector, including enabling
cross-region support
when that is considered necessary.
Consult the documentation in connecting.md/connecting.html for the current
resolution process.
Contributed by Viraj Jasani
---
.../hadoop/fs/s3a/DefaultS3ClientFactory.java | 53 +--
.../tools/hadoop-aws/aws_sdk_v2_changelog.md | 4 +-
.../site/markdown/tools/hadoop-aws/connecting.md | 36 +
.../src/site/markdown/tools/hadoop-aws/index.md| 4 +-
.../hadoop/fs/s3a/ITestS3AEndpointRegion.java | 163 -
.../hadoop/fs/s3a/test/PublicDatasetTestUtils.java | 7 +
6 files changed, 251 insertions(+), 16 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
index 0a3267a9fe51..284ba8e6ae5c 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
@@ -267,9 +267,10 @@ public class DefaultS3ClientFactory extends Configured
*/
private , ClientT>
void configureEndpointAndRegion(
BuilderT builder, S3ClientCreationParameters parameters, Configuration
conf) {
-URI endpoint = getS3Endpoint(parameters.getEndpoint(), conf);
+final String endpointStr = parameters.getEndpoint();
+final URI endpoint = getS3Endpoint(endpointStr, conf);
-String configuredRegion = parameters.getRegion();
+final String configuredRegion = parameters.getRegion();
Region region = null;
String origin = "";
@@ -291,15 +292,33 @@ public class DefaultS3ClientFactory extends Configured
if (endpoint != null) {
checkArgument(!fipsEnabled,
"%s : %s", ERROR_ENDPOINT_WITH_FIPS, endpoint);
- builder.endpointOverride(endpoint);
- // No region was configured, try to determine it from the endpoint.
+ boolean endpointEndsWithCentral =
+ endpointStr.endsWith(CENTRAL_ENDPOINT);
+
+ // No region was configured,
+ // determine the region from the endpoint.
if (region == null) {
-region = getS3RegionFromEndpoint(parameters.getEndpoint());
+region = getS3RegionFromEndpoint(endpointStr,
+endpointEndsWithCentral);
if (region != null) {
origin = "endpoint";
}
}
- LOG.debug("Setting endpoint to {}", endpoint);
+
+ // No need to override endpoint with "s3.amazonaws.com".
+ // Let the client take care of endpoint resolution. Overriding
+ // the endpoint with "s3.amazonaws.com" causes 400 Bad Request
+ // errors for non-existent buckets and objects.
+ // ref: https://github.com/aws/aws-sdk-java-v2/issues/4846
+ if (!endpointEndsWithCentral) {
+builder.endpointOverride(endpoint);
+LOG.debug("Setting endpoint to {}", endpoint);
+ } else {
+builder.crossRegionAccessEnabled(true);
+origin = "central endpoint with cross region access";
+LOG.debug("Enabling cross region access for endpoint {}",
+endpointStr);
+ }
}
if (region != null) {
@@ -354,20 +373,32 @@ public class DefaultS3ClientFactory extends Configured
/**
* Parses the endpoint to get the region.
- * If endpoint is the central one, use US_EAST_1.
+ * If endpoint is the central one, use US_EAST_2.
*
* @param endpoint the configure endpoint.
+ * @param endpointEndsWithCentral true if the endpoint is configured as
central.
* @return the S3 region, null if unable to resolve from endpoint.
*/
- private static Region getS3RegionFromEndpoint(String endpoint) {
+ private static Region getS3RegionFromEndpoint(final String endpoint,
+ final boolean endpointEndsWithCentral) {
-if(!endpoint.endsWith(CENTRAL_ENDPOINT)) {
+if (!endpointEndsWithCentral) {
LOG.debug("Endpoint {} is not the default; parsing", endpoint);
return AwsHostNameUtils.parseSigningRegion(endpoint,
S3_SERVICE_NAME).orElse(null);
}
-// endpoint is for US_EAST_1;
-return Region.US_EAST_1;
+// Select default region here to enable cross-regi
(hadoop) branch branch-3.4.0 updated: HADOOP-18987. Various fixes to FileSystem API docs (#6292)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new fdbc67b9a8f5 HADOOP-18987. Various fixes to FileSystem API docs (#6292)
fdbc67b9a8f5 is described below
commit fdbc67b9a8f52237acdeed7b46f442e6e597588a
Author: DieterDP <90392398+dieterdp...@users.noreply.github.com>
AuthorDate: Fri Feb 2 12:49:31 2024 +0100
HADOOP-18987. Various fixes to FileSystem API docs (#6292)
Contributed by Dieter De Paepe
---
.../src/site/markdown/filesystem/abortable.md | 3 +-
.../src/site/markdown/filesystem/filesystem.md | 97 +++---
.../site/markdown/filesystem/fsdatainputstream.md | 8 +-
.../src/site/markdown/filesystem/model.md | 28 ---
.../src/site/markdown/filesystem/notation.md | 8 +-
5 files changed, 75 insertions(+), 69 deletions(-)
diff --git
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/abortable.md
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/abortable.md
index 7e6ea01a8fe9..dc7677bd9a54 100644
---
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/abortable.md
+++
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/abortable.md
@@ -88,14 +88,13 @@ for example. output streams returned by the S3A FileSystem.
The stream MUST implement `Abortable` and `StreamCapabilities`.
```python
- if unsupported:
+if unsupported:
throw UnsupportedException
if not isOpen(stream):
no-op
StreamCapabilities.hasCapability("fs.capability.outputstream.abortable") ==
True
-
```
diff --git
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
index 5fba8a2515bb..518026876ba0 100644
---
a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
+++
b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
@@ -64,13 +64,13 @@ a protected directory, result in such an exception being
raised.
### `boolean isDirectory(Path p)`
-def isDirectory(FS, p)= p in directories(FS)
+def isDir(FS, p) = p in directories(FS)
### `boolean isFile(Path p)`
-def isFile(FS, p) = p in files(FS)
+def isFile(FS, p) = p in filenames(FS)
### `FileStatus getFileStatus(Path p)`
@@ -250,7 +250,7 @@ process.
changes are made to the filesystem, the result of `listStatus(parent(P))`
SHOULD
include the value of `getFileStatus(P)`.
-* After an entry at path `P` is created, and before any other
+* After an entry at path `P` is deleted, and before any other
changes are made to the filesystem, the result of `listStatus(parent(P))`
SHOULD
NOT include the value of `getFileStatus(P)`.
@@ -305,7 +305,7 @@ that they must all be listed, and, at the time of listing,
exist.
All paths must exist. There is no requirement for uniqueness.
forall p in paths :
- exists(fs, p) else raise FileNotFoundException
+ exists(FS, p) else raise FileNotFoundException
Postconditions
@@ -381,7 +381,7 @@ being completely performed.
Path `path` must exist:
-exists(FS, path) : raise FileNotFoundException
+if not exists(FS, path) : raise FileNotFoundException
Postconditions
@@ -432,7 +432,7 @@ of data which must be collected in a single RPC call.
Preconditions
-exists(FS, path) else raise FileNotFoundException
+if not exists(FS, path) : raise FileNotFoundException
### Postconditions
@@ -463,7 +463,7 @@ and 1 for file count.
Preconditions
-exists(FS, path) else raise FileNotFoundException
+if not exists(FS, path) : raise FileNotFoundException
Postconditions
@@ -567,7 +567,7 @@ when writing objects to a path in the filesystem.
Postconditions
-result = integer >= 0
+result = integer >= 0
The outcome of this operation is usually identical to `getDefaultBlockSize()`,
with no checks for the existence of the given path.
@@ -591,12 +591,12 @@ on the filesystem.
Preconditions
-if not exists(FS, p) : raise FileNotFoundException
+if not exists(FS, p) : raise FileNotFoundException
Postconditions
-if len(FS, P) > 0: getFileStatus(P).getBlockSize() > 0
+if len(FS, P) > 0 : getFileStatus(P).getBlockSize() > 0
result == getFileStatus(P).getBlockSize()
1. The outcome of this operation MUST be identical to the value of
@@ -654,12 +654,12 @@ No ancestor may be a file
forall d = ancestors(FS, p) :
if exists(FS, d) and not isDir(FS, d) :
-raise [ParentNotDirectoryException, FileAlreadyExistsException,
IOException]
+raise {ParentNotDirectoryException, FileAlreadyExistsException,
IOException}
Po
(hadoop) branch branch-3.4.0 updated: HDFS-17359. EC: recheck failed streamers should only after flushing all packets. (#6503). Contributed by farmmamba.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new c8a17b09c33d HDFS-17359. EC: recheck failed streamers should only
after flushing all packets. (#6503). Contributed by farmmamba.
c8a17b09c33d is described below
commit c8a17b09c33db0f518984977ab5cf643a7ef34b2
Author: hfutatzhanghb
AuthorDate: Thu Feb 1 22:53:37 2024 +0800
HDFS-17359. EC: recheck failed streamers should only after flushing all
packets. (#6503). Contributed by farmmamba.
Signed-off-by: Takanobu Asanuma
(cherry picked from commit 4f4b84698658c42efd5f7af0c6061546f6fb7ce8)
---
.../src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
index a58c7bbb204f..8320cc9a4086 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
@@ -671,9 +671,9 @@ public class DFSStripedOutputStream extends DFSOutputStream
// for healthy streamers, wait till all of them have fetched the new
block
// and flushed out all the enqueued packets.
flushAllInternals();
+ // recheck failed streamers again after the flush
+ newFailed = checkStreamers();
}
-// recheck failed streamers again after the flush
-newFailed = checkStreamers();
while (newFailed.size() > 0) {
failedStreamers.addAll(newFailed);
coordinator.clearFailureStates();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-19046. S3A: update AWS V2 SDK to 2.23.5; v1 to 1.12.599 (#6467)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.0 by this push: new f01d33d4539f HADOOP-19046. S3A: update AWS V2 SDK to 2.23.5; v1 to 1.12.599 (#6467) f01d33d4539f is described below commit f01d33d4539f971bd7b1c04594c2294f148b4249 Author: Steve Loughran AuthorDate: Sun Jan 21 19:00:34 2024 + HADOOP-19046. S3A: update AWS V2 SDK to 2.23.5; v1 to 1.12.599 (#6467) This update ensures that the timeout set in fs.s3a.connection.request.timeout is passed down to calls to CreateSession made in the AWS SDK to get S3 Express session tokens. Contributed by Steve Loughran --- LICENSE-binary | 4 ++-- NOTICE-binary | 2 +- hadoop-project/pom.xml | 4 ++-- .../src/site/markdown/tools/hadoop-aws/testing.md | 13 ++--- 4 files changed, 15 insertions(+), 8 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 3720a7809563..93e38cc34ee4 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -215,7 +215,7 @@ com.aliyun:aliyun-java-sdk-kms:2.11.0 com.aliyun:aliyun-java-sdk-ram:3.1.0 com.aliyun:aliyun-java-sdk-sts:3.0.0 com.aliyun.oss:aliyun-sdk-oss:3.13.2 -com.amazonaws:aws-java-sdk-bundle:1.12.565 +com.amazonaws:aws-java-sdk-bundle:1.12.599 com.cedarsoftware:java-util:1.9.0 com.cedarsoftware:json-io:2.5.1 com.fasterxml.jackson.core:jackson-annotations:2.12.7 @@ -363,7 +363,7 @@ org.objenesis:objenesis:2.6 org.xerial.snappy:snappy-java:1.1.10.4 org.yaml:snakeyaml:2.0 org.wildfly.openssl:wildfly-openssl:1.1.3.Final -software.amazon.awssdk:bundle:jar:2.21.41 +software.amazon.awssdk:bundle:jar:2.23.5 diff --git a/NOTICE-binary b/NOTICE-binary index 6db51d08b42f..7389a31fd5a1 100644 --- a/NOTICE-binary +++ b/NOTICE-binary @@ -66,7 +66,7 @@ available from http://www.digip.org/jansson/. AWS SDK for Java -Copyright 2010-2022 Amazon.com, Inc. or its affiliates. All Rights Reserved. +Copyright 2010-2024 Amazon.com, Inc. or its affiliates. All Rights Reserved. This product includes software developed by Amazon Technologies, Inc (http://www.amazon.com/). diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 2c0cc3edcb6c..b79fdd576109 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -186,8 +186,8 @@ 1.3.1 1.0-beta-1 900 -1.12.565 -2.21.41 +1.12.599 +2.23.5 1.0.1 2.7.1 1.11.2 diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md index c2eafbcb8de2..62d449daeea5 100644 --- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md +++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md @@ -1142,7 +1142,7 @@ as it may take a couple of SDK updates before it is ready. 1. Identify the latest AWS SDK [available for download](https://aws.amazon.com/sdk-for-java/). 1. Create a private git branch of trunk for JIRA, and in `hadoop-project/pom.xml` update the `aws-java-sdk.version` to the new SDK version. -1. Update AWS SDK versions in NOTICE.txt. +1. Update AWS SDK versions in NOTICE.txt and LICENSE.binary 1. Do a clean build and rerun all the `hadoop-aws` tests. This includes the `-Pscale` set, with a role defined for the assumed role tests. in `fs.s3a.assumed.role.arn` for testing assumed roles, @@ -1164,11 +1164,18 @@ your IDE or via maven. `mvn dependency:tree -Dverbose > target/dependencies.txt`. Examine the `target/dependencies.txt` file to verify that no new artifacts have unintentionally been declared as dependencies - of the shaded `aws-java-sdk-bundle` artifact. + of the shaded `software.amazon.awssdk:bundle:jar` artifact. 1. Run a full AWS-test suite with S3 client-side encryption enabled by setting `fs.s3a.encryption.algorithm` to 'CSE-KMS' and setting up AWS-KMS Key ID in `fs.s3a.encryption.key`. +The dependency chain of the `hadoop-aws` module should be similar to this, albeit +with different version numbers: +``` +[INFO] +- org.apache.hadoop:hadoop-aws:jar:3.4.0-SNAPSHOT:compile +[INFO] | +- software.amazon.awssdk:bundle:jar:2.23.5:compile +[INFO] | \- org.wildfly.openssl:wildfly-openssl:jar:1.1.3.Final:compile +``` ### Basic command line regression testing We need a run through of the CLI to see if there have been changes there @@ -1365,5 +1372,5 @@ Don't be surprised if this happens, don't worry too much, and, while that rollback option is there to be used, ideally try to work forwards. If the problem is with the SDK, file issues with the - [AWS SDK Bug tracker](https://github.
(hadoop) branch branch-3.4.0 updated: HADOOP-19045. S3A: Validate CreateSession Timeout Propagation (#6470)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new 4344537aa2d1 HADOOP-19045. S3A: Validate CreateSession Timeout
Propagation (#6470)
4344537aa2d1 is described below
commit 4344537aa2d118eb751868c6fa10594df5a842df
Author: Steve Loughran
AuthorDate: Tue Jan 30 15:32:24 2024 +
HADOOP-19045. S3A: Validate CreateSession Timeout Propagation (#6470)
New test ITestCreateSessionTimeout to verify that the duration set
in fs.s3a.connection.request.timeout is passed all the way down.
This is done by adding a sleep() in a custom signer and verifying
that it is interrupted and that an AWSApiCallTimeoutException is
raised.
+ Fix testRequestTimeout()
* doesn't skip if considered cross-region
* sets a minimum duration of 0 before invocation
* resets the minimum afterwards
Contributed by Steve Loughran
---
.../java/org/apache/hadoop/fs/s3a/Constants.java | 25 ++-
.../hadoop/fs/s3a/auth/CustomHttpSigner.java | 2 +-
.../apache/hadoop/fs/s3a/impl/AWSClientConfig.java | 2 +-
.../hadoop/fs/s3a/ITestS3AConfiguration.java | 28 ++-
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 10 +
.../s3a/performance/ITestCreateSessionTimeout.java | 211 +
6 files changed, 262 insertions(+), 16 deletions(-)
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
index 4408cf68a451..744146ccf4f3 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
@@ -337,16 +337,33 @@ public final class Constants {
public static final int DEFAULT_SOCKET_TIMEOUT =
(int)DEFAULT_SOCKET_TIMEOUT_DURATION.toMillis();
/**
- * Time until a request is timed-out: {@value}.
- * If zero, there is no timeout.
+ * How long should the SDK retry/wait on a response from an S3 store:
{@value}
+ * including the time needed to sign the request.
+ *
+ * This is time to response, so for a GET request it is "time to 200
response"
+ * not the time limit to download the requested data.
+ * This makes it different from {@link #REQUEST_TIMEOUT}, which is for total
+ * HTTP request.
+ *
+ * Default unit is milliseconds.
+ *
+ * There is a minimum duration set in {@link
#MINIMUM_NETWORK_OPERATION_DURATION};
+ * it is impossible to set a delay less than this, even for testing.
+ * Why so? Too many deployments where the configuration assumed the timeout
was in seconds
+ * and that "120" was a reasonable value rather than "too short to work
reliably"
+ *
+ * Note for anyone writing tests which need to set a low value for this:
+ * to avoid the minimum duration overrides, call
+ * {@code AWSClientConfig.setMinimumOperationDuration()} and set a low value
+ * before creating the filesystem.
*/
public static final String REQUEST_TIMEOUT =
"fs.s3a.connection.request.timeout";
/**
- * Default duration of a request before it is timed out: Zero.
+ * Default duration of a request before it is timed out: 60s.
*/
- public static final Duration DEFAULT_REQUEST_TIMEOUT_DURATION =
Duration.ZERO;
+ public static final Duration DEFAULT_REQUEST_TIMEOUT_DURATION =
Duration.ofSeconds(60);
/**
* Default duration of a request before it is timed out: Zero.
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CustomHttpSigner.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CustomHttpSigner.java
index ba1169a5e598..528414b63e32 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CustomHttpSigner.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CustomHttpSigner.java
@@ -40,7 +40,7 @@ import
software.amazon.awssdk.identity.spi.AwsCredentialsIdentity;
* fs.s3a.http.signer.class = org.apache.hadoop.fs.s3a.auth.CustomHttpSigner
*
*/
-public final class CustomHttpSigner implements
HttpSigner {
+public class CustomHttpSigner implements HttpSigner {
private static final Logger LOG = LoggerFactory
.getLogger(CustomHttpSigner.class);
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java
index f6da9d84e0a7..60729ac30866 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java
@@ -577,7 +577
(hadoop) branch branch-3.4.0 updated: HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple server calls (#6022)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new 92e59743c104 HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution:
prevent multiple server calls (#6022)
92e59743c104 is described below
commit 92e59743c104970fe8878f925f65dffcf2eba092
Author: Pranav Saxena <108325433+saxenapra...@users.noreply.github.com>
AuthorDate: Tue Jan 30 05:17:04 2024 -0800
HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple
server calls (#6022)
Address JDK bug JDK-8314978 related to handling of HTTP 100
responses.
https://bugs.openjdk.org/browse/JDK-8314978
In the AbfsHttpOperation, after sendRequest() we call processResponse()
method from AbfsRestOperation.
Even if the conn.getOutputStream() fails due to expect-100 error,
we consume the exception and let the code go ahead.
This may call getHeaderField() / getHeaderFields() / getHeaderFieldLong()
after
getOutputStream() has failed. These invocation all lead to server calls.
This commit aims to prevent this.
If connection.getOutputStream() fails due to an Expect-100 error,
the ABFS client does not invoke getHeaderField(), getHeaderFields(),
getHeaderFieldLong() or getInputStream().
getResponseCode() is safe as on the failure it sets the
responseCode variable in HttpUrlConnection object.
Contributed by Pranav Saxena
---
.../fs/azurebfs/constants/AbfsHttpConstants.java | 1 +
.../fs/azurebfs/services/AbfsHttpOperation.java| 41 +--
.../fs/azurebfs/services/AbfsOutputStream.java | 9 +++-
.../fs/azurebfs/services/ITestAbfsClient.java | 3 +-
.../azurebfs/services/ITestAbfsOutputStream.java | 61 ++
.../azurebfs/services/ITestAbfsRestOperation.java | 3 +-
6 files changed, 109 insertions(+), 9 deletions(-)
diff --git
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java
index 91f6bddcc1d4..63de71eb178d 100644
---
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java
+++
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java
@@ -69,6 +69,7 @@ public final class AbfsHttpConstants {
* and should qualify for retry.
*/
public static final int HTTP_CONTINUE = 100;
+ public static final String EXPECT_100_JDK_ERROR = "Server rejected
operation";
// Abfs generic constants
public static final String SINGLE_WHITE_SPACE = " ";
diff --git
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java
index 7f5df6066f1b..c0b554f60702 100644
---
a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java
+++
b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java
@@ -22,12 +22,14 @@ import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.HttpURLConnection;
+import java.net.ProtocolException;
import java.net.URL;
import java.util.List;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLSocketFactory;
+import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.fs.azurebfs.utils.UriUtils;
import org.apache.hadoop.security.ssl.DelegatingSSLSocketFactory;
@@ -43,6 +45,7 @@ import
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations;
import org.apache.hadoop.fs.azurebfs.contracts.services.AbfsPerfLoggable;
import org.apache.hadoop.fs.azurebfs.contracts.services.ListResultSchema;
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.EXPECT_100_JDK_ERROR;
import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.HUNDRED_CONTINUE;
import static
org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.EXPECT;
@@ -83,6 +86,7 @@ public class AbfsHttpOperation implements AbfsPerfLoggable {
private long sendRequestTimeMs;
private long recvResponseTimeMs;
private boolean shouldMask = false;
+ private boolean connectionDisconnectedOnError = false;
public static AbfsHttpOperation getAbfsHttpOperationWithFixedResult(
final URL url,
@@ -324,14 +328,26 @@ public class AbfsHttpOperation implements
AbfsPerfLoggable {
*/
outputStream = getConnOutputStream();
} catch (IOException e) {
-/* If getOutputStream fails with an exception and expect header
- is enabled, we re
(hadoop) branch branch-3.4.0 updated: YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4.0 by this push: new 2b68496fad2b YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455) 2b68496fad2b is described below commit 2b68496fad2b9bfb2c4c11ebeecb19a62eeb49d2 Author: Ferenc Erdelyi <55103964+ferde...@users.noreply.github.com> AuthorDate: Mon Jan 22 15:41:48 2024 +0100 YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455) --- .../capacity/policy/PriorityUtilizationQueueOrderingPolicy.java| 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java index d12a5e74b8d0..f60208e04849 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java @@ -28,6 +28,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity .CapacitySchedulerConfiguration; import org.apache.hadoop.yarn.util.resource.Resources; +import java.util.ArrayList; import java.util.Comparator; import java.util.Iterator; import java.util.List; @@ -254,9 +255,9 @@ public class PriorityUtilizationQueueOrderingPolicy // partitionToLookAt is a thread local variable, therefore it is safe to mutate it. PriorityUtilizationQueueOrderingPolicy.partitionToLookAt.set(partition); -// Sort the snapshot of the queues in order to avoid breaking the prerequisites of TimSort. -// See YARN-10178 for details. -return queues.stream().map(PriorityQueueResourcesForSorting::new).sorted( +// Copy (for thread safety) and sort the snapshot of the queues in order to avoid breaking +// the prerequisites of TimSort. See YARN-10178 for details. +return new ArrayList<>(queues).stream().map(PriorityQueueResourcesForSorting::new).sorted( new PriorityQueueComparator()).map(PriorityQueueResourcesForSorting::getQueue).collect( Collectors.toList()).iterator(); } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new 6190423505f2 YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455) 6190423505f2 is described below commit 6190423505f2fc27a53e03ddbc027a53b20f5104 Author: Ferenc Erdelyi <55103964+ferde...@users.noreply.github.com> AuthorDate: Mon Jan 22 15:41:48 2024 +0100 YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455) --- .../capacity/policy/PriorityUtilizationQueueOrderingPolicy.java| 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java index d12a5e74b8d0..f60208e04849 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/policy/PriorityUtilizationQueueOrderingPolicy.java @@ -28,6 +28,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity .CapacitySchedulerConfiguration; import org.apache.hadoop.yarn.util.resource.Resources; +import java.util.ArrayList; import java.util.Comparator; import java.util.Iterator; import java.util.List; @@ -254,9 +255,9 @@ public class PriorityUtilizationQueueOrderingPolicy // partitionToLookAt is a thread local variable, therefore it is safe to mutate it. PriorityUtilizationQueueOrderingPolicy.partitionToLookAt.set(partition); -// Sort the snapshot of the queues in order to avoid breaking the prerequisites of TimSort. -// See YARN-10178 for details. -return queues.stream().map(PriorityQueueResourcesForSorting::new).sorted( +// Copy (for thread safety) and sort the snapshot of the queues in order to avoid breaking +// the prerequisites of TimSort. See YARN-10178 for details. +return new ArrayList<>(queues).stream().map(PriorityQueueResourcesForSorting::new).sorted( new PriorityQueueComparator()).map(PriorityQueueResourcesForSorting::getQueue).collect( Collectors.toList()).iterator(); } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4.0 updated: HADOOP-19056. Highlight RBF features and improvements targeting version 3.4. (#6512) Contributed by Takanobu Asanuma.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new 95951e928ee5 HADOOP-19056. Highlight RBF features and improvements
targeting version 3.4. (#6512) Contributed by Takanobu Asanuma.
95951e928ee5 is described below
commit 95951e928ee5d725704e61132f4ea850c8716469
Author: Takanobu Asanuma
AuthorDate: Wed Jan 31 14:30:35 2024 +0900
HADOOP-19056. Highlight RBF features and improvements targeting version
3.4. (#6512) Contributed by Takanobu Asanuma.
Signed-off-by: Shilun Fan
---
hadoop-project/src/site/markdown/index.md.vm | 12
1 file changed, 12 insertions(+)
diff --git a/hadoop-project/src/site/markdown/index.md.vm
b/hadoop-project/src/site/markdown/index.md.vm
index 7c1b3ef5eb40..54e8055e633d 100644
--- a/hadoop-project/src/site/markdown/index.md.vm
+++ b/hadoop-project/src/site/markdown/index.md.vm
@@ -89,8 +89,20 @@ Important features and improvements are as follows:
[HDFS-15294](https://issues.apache.org/jira/browse/HDFS-15294) HDFS Federation
balance tool introduces one tool to balance data across different namespace.
+[HDFS-13522](https://issues.apache.org/jira/browse/HDFS-13522),
[HDFS-16767](https://issues.apache.org/jira/browse/HDFS-16767) Support observer
node from Router-Based Federation.
+
**Improvement**
+[HADOOP-13144](https://issues.apache.org/jira/browse/HADOOP-13144),
[HDFS-13274](https://issues.apache.org/jira/browse/HDFS-13274),
[HDFS-15757](https://issues.apache.org/jira/browse/HDFS-15757)
+
+These tickets have enhanced IPC throughput between Router and NameNode via
multiple connections per user, and optimized connection management.
+
+[HDFS-14090](https://issues.apache.org/jira/browse/HDFS-14090) RBF: Improved
isolation for downstream name nodes. {Static}
+
+Router supports assignment of the dedicated number of RPC handlers to achieve
isolation for all downstream nameservices
+it is configured to proxy. Since large or busy clusters may have relatively
higher RPC traffic to the namenode compared to other clusters namenodes,
+this feature if enabled allows admins to configure higher number of RPC
handlers for busy clusters.
+
[HDFS-17128](https://issues.apache.org/jira/browse/HDFS-17128) RBF:
SQLDelegationTokenSecretManager should use version of tokens updated by other
routers.
The SQLDelegationTokenSecretManager enhances performance by maintaining
processed tokens in memory. However, there is
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch branch-3.4 updated: HADOOP-19056. Highlight RBF features and improvements targeting version 3.4. (#6512) Contributed by Takanobu Asanuma.
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 02c1273ceb9b HADOOP-19056. Highlight RBF features and improvements
targeting version 3.4. (#6512) Contributed by Takanobu Asanuma.
02c1273ceb9b is described below
commit 02c1273ceb9b2104c6a891dbedc43e0f0149f5ab
Author: Takanobu Asanuma
AuthorDate: Wed Jan 31 14:30:35 2024 +0900
HADOOP-19056. Highlight RBF features and improvements targeting version
3.4. (#6512) Contributed by Takanobu Asanuma.
Signed-off-by: Shilun Fan
---
hadoop-project/src/site/markdown/index.md.vm | 12
1 file changed, 12 insertions(+)
diff --git a/hadoop-project/src/site/markdown/index.md.vm
b/hadoop-project/src/site/markdown/index.md.vm
index 7c1b3ef5eb40..54e8055e633d 100644
--- a/hadoop-project/src/site/markdown/index.md.vm
+++ b/hadoop-project/src/site/markdown/index.md.vm
@@ -89,8 +89,20 @@ Important features and improvements are as follows:
[HDFS-15294](https://issues.apache.org/jira/browse/HDFS-15294) HDFS Federation
balance tool introduces one tool to balance data across different namespace.
+[HDFS-13522](https://issues.apache.org/jira/browse/HDFS-13522),
[HDFS-16767](https://issues.apache.org/jira/browse/HDFS-16767) Support observer
node from Router-Based Federation.
+
**Improvement**
+[HADOOP-13144](https://issues.apache.org/jira/browse/HADOOP-13144),
[HDFS-13274](https://issues.apache.org/jira/browse/HDFS-13274),
[HDFS-15757](https://issues.apache.org/jira/browse/HDFS-15757)
+
+These tickets have enhanced IPC throughput between Router and NameNode via
multiple connections per user, and optimized connection management.
+
+[HDFS-14090](https://issues.apache.org/jira/browse/HDFS-14090) RBF: Improved
isolation for downstream name nodes. {Static}
+
+Router supports assignment of the dedicated number of RPC handlers to achieve
isolation for all downstream nameservices
+it is configured to proxy. Since large or busy clusters may have relatively
higher RPC traffic to the namenode compared to other clusters namenodes,
+this feature if enabled allows admins to configure higher number of RPC
handlers for busy clusters.
+
[HDFS-17128](https://issues.apache.org/jira/browse/HDFS-17128) RBF:
SQLDelegationTokenSecretManager should use version of tokens updated by other
routers.
The SQLDelegationTokenSecretManager enhances performance by maintaining
processed tokens in memory. However, there is
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated: HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 8011b21c5277 HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533)
Contributed by Shilun Fan
8011b21c5277 is described below
commit 8011b21c527724104867d7c62b3ec20e1218f062
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Thu Feb 8 19:18:04 2024 +0800
HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533) Contributed by Shilun Fan
Reviewed-by: He Xiaoqiao
Signed-off-by: Shilun Fan
---
hadoop-common-project/hadoop-common/pom.xml | 2 +-
hadoop-project/pom.xml | 6 +++---
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml | 6 +++---
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index 3b3279ff4e25..8a08d21eec18 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -40,7 +40,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
org.apache.hadoop
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 1a135ded8809..a686bda13d0b 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -93,10 +93,10 @@
${common.protobuf2.scope}
-3.7.1
+3.21.12
${env.HADOOP_PROTOC_PATH}
-1.1.1
+1.2.0
${hadoop-thirdparty.version}
${hadoop-thirdparty.version}
org.apache.hadoop.thirdparty
@@ -250,7 +250,7 @@
org.apache.hadoop.thirdparty
-hadoop-shaded-protobuf_3_7
+hadoop-shaded-protobuf_3_21
${hadoop-thirdparty-protobuf.version}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
index 3f70972babda..8dbec9ed4e9d 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
@@ -51,7 +51,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
@@ -69,7 +69,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
@@ -80,7 +80,7 @@
org.apache.hadoop.thirdparty
- hadoop-shaded-protobuf_3_7
+ hadoop-shaded-protobuf_3_21
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-thirdparty) branch trunk updated: Make upstream aware of 1.2.0 release.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git The following commit(s) were added to refs/heads/trunk by this push: new 3885790 Make upstream aware of 1.2.0 release. 3885790 is described below commit 3885790c4b7d32b4e47ade26d17a579ae13883dc Author: Shilun Fan AuthorDate: Wed Feb 7 11:21:01 2024 +0800 Make upstream aware of 1.2.0 release. --- .../thirdparty-1.2.0/CHANGELOG.thirdparty-1.2.0.md | 58 ++ .../RELEASENOTES.thirdparty-1.2.0.md | 24 + 2 files changed, 82 insertions(+) diff --git a/src/site/markdown/release/thirdparty-1.2.0/CHANGELOG.thirdparty-1.2.0.md b/src/site/markdown/release/thirdparty-1.2.0/CHANGELOG.thirdparty-1.2.0.md new file mode 100644 index 000..a720589 --- /dev/null +++ b/src/site/markdown/release/thirdparty-1.2.0/CHANGELOG.thirdparty-1.2.0.md @@ -0,0 +1,58 @@ + + +# Apache Hadoop Third-party Libs Changelog + +## Release thirdparty-1.2.0 - 2024-02-04 + + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-18282](https://issues.apache.org/jira/browse/HADOOP-18282) | Add .asf.yaml to hadoop-thirdparty | Major | hadoop-thirdparty | Ayush Saxena | Ayush Saxena | +| [HADOOP-18721](https://issues.apache.org/jira/browse/HADOOP-18721) | Upgrade guava in hadoop-thirdparty to 31.1-jre | Major | hadoop-thirdparty | D M Murali Krishna Reddy | D M Murali Krishna Reddy | +| [HADOOP-18921](https://issues.apache.org/jira/browse/HADOOP-18921) | upgrade avro in hadoop-thirdparty to 1.11.3 | Major | hadoop-thirdparty | PJ Fanning | PJ Fanning | +| [HADOOP-18342](https://issues.apache.org/jira/browse/HADOOP-18342) | Upgrade to Avro 1.11.1 | Major | hadoop-thirdparty | PJ Fanning | PJ Fanning | +| [HADOOP-18197](https://issues.apache.org/jira/browse/HADOOP-18197) | Update protobuf 3.7.1 to a version without CVE-2021-22569 | Major | hadoop-thirdparty | Ivan Viaznikov | PJ Fanning | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-17707](https://issues.apache.org/jira/browse/HADOOP-17707) | Remove jaeger document from site index | Major | hadoop-thirdparty | Takanobu Asanuma | Takanobu Asanuma | +| [HADOOP-17730](https://issues.apache.org/jira/browse/HADOOP-17730) | [thirdparty] Do not exclude error\_prone | Major | hadoop-thirdparty | Wei-Chiu Chuang | Wei-Chiu Chuang | + + +### SUB-TASKS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-19055](https://issues.apache.org/jira/browse/HADOOP-19055) | Preparing for 1.3.0 development | Major | hadoop-thirdparty | Shilun Fan | Shilun Fan | +| [HADOOP-19054](https://issues.apache.org/jira/browse/HADOOP-19054) | Update hadoop-thirdparty index.md.vm | Major | hadoop-thirdparty | Shilun Fan | Shilun Fan | + + +### OTHER: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-18843](https://issues.apache.org/jira/browse/HADOOP-18843) | Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23) | Major | hadoop-thirdparty | Wei-Chiu Chuang | Wei-Chiu Chuang | + + diff --git a/src/site/markdown/release/thirdparty-1.2.0/RELEASENOTES.thirdparty-1.2.0.md b/src/site/markdown/release/thirdparty-1.2.0/RELEASENOTES.thirdparty-1.2.0.md new file mode 100644 index 000..5ea30bb --- /dev/null +++ b/src/site/markdown/release/thirdparty-1.2.0/RELEASENOTES.thirdparty-1.2.0.md @@ -0,0 +1,24 @@ + + +# Apache Hadoop Third-party Libs thirdparty-1.2.0 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + + - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
svn commit: r67223 - /release/hadoop/thirdparty/thirdparty-1.2.0/
Author: slfan1989 Date: Wed Feb 7 03:06:10 2024 New Revision: 67223 Log: Publishing the bits for Hadoop Thirdparty release 1.2.0 Added: release/hadoop/thirdparty/thirdparty-1.2.0/ release/hadoop/thirdparty/thirdparty-1.2.0/CHANGELOG.md release/hadoop/thirdparty/thirdparty-1.2.0/CHANGELOG.md.asc (with props) release/hadoop/thirdparty/thirdparty-1.2.0/CHANGELOG.md.sha512 release/hadoop/thirdparty/thirdparty-1.2.0/RELEASENOTES.md release/hadoop/thirdparty/thirdparty-1.2.0/RELEASENOTES.md.asc (with props) release/hadoop/thirdparty/thirdparty-1.2.0/RELEASENOTES.md.sha512 release/hadoop/thirdparty/thirdparty-1.2.0/hadoop-thirdparty-1.2.0-rat.txt release/hadoop/thirdparty/thirdparty-1.2.0/hadoop-thirdparty-1.2.0-rat.txt.asc (with props) release/hadoop/thirdparty/thirdparty-1.2.0/hadoop-thirdparty-1.2.0-rat.txt.sha512 release/hadoop/thirdparty/thirdparty-1.2.0/hadoop-thirdparty-1.2.0-site.tar.gz (with props) release/hadoop/thirdparty/thirdparty-1.2.0/hadoop-thirdparty-1.2.0-site.tar.gz.asc (with props) release/hadoop/thirdparty/thirdparty-1.2.0/hadoop-thirdparty-1.2.0-site.tar.gz.sha512 release/hadoop/thirdparty/thirdparty-1.2.0/hadoop-thirdparty-1.2.0-src.tar.gz (with props) release/hadoop/thirdparty/thirdparty-1.2.0/hadoop-thirdparty-1.2.0-src.tar.gz.asc (with props) release/hadoop/thirdparty/thirdparty-1.2.0/hadoop-thirdparty-1.2.0-src.tar.gz.sha512 Added: release/hadoop/thirdparty/thirdparty-1.2.0/CHANGELOG.md == --- release/hadoop/thirdparty/thirdparty-1.2.0/CHANGELOG.md (added) +++ release/hadoop/thirdparty/thirdparty-1.2.0/CHANGELOG.md Wed Feb 7 03:06:10 2024 @@ -0,0 +1,58 @@ + + +# Apache Hadoop Third-party Libs Changelog + +## Release thirdparty-1.2.0 - 2024-02-04 + + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-18282](https://issues.apache.org/jira/browse/HADOOP-18282) | Add .asf.yaml to hadoop-thirdparty | Major | hadoop-thirdparty | Ayush Saxena | Ayush Saxena | +| [HADOOP-18721](https://issues.apache.org/jira/browse/HADOOP-18721) | Upgrade guava in hadoop-thirdparty to 31.1-jre | Major | hadoop-thirdparty | D M Murali Krishna Reddy | D M Murali Krishna Reddy | +| [HADOOP-18921](https://issues.apache.org/jira/browse/HADOOP-18921) | upgrade avro in hadoop-thirdparty to 1.11.3 | Major | hadoop-thirdparty | PJ Fanning | PJ Fanning | +| [HADOOP-18342](https://issues.apache.org/jira/browse/HADOOP-18342) | Upgrade to Avro 1.11.1 | Major | hadoop-thirdparty | PJ Fanning | PJ Fanning | +| [HADOOP-18197](https://issues.apache.org/jira/browse/HADOOP-18197) | Update protobuf 3.7.1 to a version without CVE-2021-22569 | Major | hadoop-thirdparty | Ivan Viaznikov | PJ Fanning | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-17707](https://issues.apache.org/jira/browse/HADOOP-17707) | Remove jaeger document from site index | Major | hadoop-thirdparty | Takanobu Asanuma | Takanobu Asanuma | +| [HADOOP-17730](https://issues.apache.org/jira/browse/HADOOP-17730) | [thirdparty] Do not exclude error\_prone | Major | hadoop-thirdparty | Wei-Chiu Chuang | Wei-Chiu Chuang | + + +### SUB-TASKS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-19055](https://issues.apache.org/jira/browse/HADOOP-19055) | Preparing for 1.3.0 development | Major | hadoop-thirdparty | Shilun Fan | Shilun Fan | +| [HADOOP-19054](https://issues.apache.org/jira/browse/HADOOP-19054) | Update hadoop-thirdparty index.md.vm | Major | hadoop-thirdparty | Shilun Fan | Shilun Fan | + + +### OTHER: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-18843](https://issues.apache.org/jira/browse/HADOOP-18843) | Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23) | Major | hadoop-thirdparty | Wei-Chiu Chuang | Wei-Chiu Chuang | + + Added: release/hadoop/thirdparty/thirdparty-1.2.0/CHANGELOG.md.asc == Binary file - no diff available. Propchange: release/hadoop/thirdparty/thirdparty-1.2.0/CHANGELOG.md.asc -- svn:mime-type = application/pgp-signature Added: release/hadoop/thirdparty/thirdparty-1.2.0/CHANGELOG.md.sha512 == --- release/hadoop/thirdparty/thirdparty-1.2.0/CHANGELOG.md.sha512 (added) +++ release/hadoop/thirdparty/thirdparty-1.2.0/CHANGELOG.md.sha512 Wed Feb 7 03:06:10 2024 @@ -0,0 +1 @@ +SHA512 (CHANGELOG.md
(hadoop-thirdparty) annotated tag rel/release-1.2.0 updated (64b6721 -> 022c9c9)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to annotated tag rel/release-1.2.0 in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git *** WARNING: tag rel/release-1.2.0 was modified! *** from 64b6721 (commit) to 022c9c9 (tag) tagging 64b672142797b5acffa94cadfc2d2be6f4dbbb23 (commit) replaces release-1.2.0-RC0 by Shilun Fan on Wed Feb 7 10:57:02 2024 +0800 - Log - Hadoop Thirdparty 1.2.0 release -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEECm40PGXwzjlsDKJKl4RW45ZNtwEFAmXC8X4ACgkQl4RW45ZN twHdWw/+M9n5ajEw+DsLGO/g69FHUEEs13ZUg0uBm4c635fNPQoZPDvotjIMB/FJ H8gLnaHRU6uMEaECu2KYIEqyWsLZqm2afIa5BXpzeoQdT/n9GYbTi0rFT9vBE6dF YIZU6Ij2wQXXfpzS9LdSNFwA548RL8L0uhhBpWIysQQp77l6n8hMMMLATxGcf2Vz ULO0myI2EOGTSpwMoWk9e4hN9sZf8u67gnmUq2xYAHC3XNyhQzqMSvltGkGh9oQw diYGC99E/gRSRVFgmMVh2K0KcVPNEE6798evCNhefbrVt0vy1o53PdLHTMBk+dme yEjISUvB1Z6f10f7NwR7xl5w8vk0nj5jzG3U9trM1g81uS/RC4rc+0Z6wERjBsL7 m1lB0SzYJyKiMFmHHIJTM36wiY7GuKMhhZXbGjwqPEe5lD5KgP3CdonyxL1a5EvF FJpeugp/o/3j4X5J7+Zt/OMHPACeVu1ECVCaxQ5vF3RmED3eAWjrVnPP+AHc7X4V Z9NMhKyCil4EEAL3yM0wGNPoUjsj3MXJkMj40u6Be+LfNetQZwiyq6GOM/5MCL1t Ncie7GblhFtMbzL1d3GE41wqSPWTvqeRP7BXk71/hdyNRDZgl77+XFliearip9Bl bc1tgrkSTR3eIzYxsd+yocVssklPgzVnLCOSJBaVrIuIY+CJ+rE= =8BPA -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
svn commit: r67167 - /dev/hadoop/hadoop-thirdparty-1.2.0-RC1/
Author: slfan1989 Date: Sun Feb 4 08:27:38 2024 New Revision: 67167 Log: HADOOP-19053. Hadoop-thirdparty-1.2.0-RC1 built from 64b67214279 Added: dev/hadoop/hadoop-thirdparty-1.2.0-RC1/ dev/hadoop/hadoop-thirdparty-1.2.0-RC1/CHANGELOG.md dev/hadoop/hadoop-thirdparty-1.2.0-RC1/CHANGELOG.md.asc (with props) dev/hadoop/hadoop-thirdparty-1.2.0-RC1/CHANGELOG.md.sha512 dev/hadoop/hadoop-thirdparty-1.2.0-RC1/RELEASENOTES.md dev/hadoop/hadoop-thirdparty-1.2.0-RC1/RELEASENOTES.md.asc (with props) dev/hadoop/hadoop-thirdparty-1.2.0-RC1/RELEASENOTES.md.sha512 dev/hadoop/hadoop-thirdparty-1.2.0-RC1/hadoop-thirdparty-1.2.0-rat.txt dev/hadoop/hadoop-thirdparty-1.2.0-RC1/hadoop-thirdparty-1.2.0-rat.txt.asc (with props) dev/hadoop/hadoop-thirdparty-1.2.0-RC1/hadoop-thirdparty-1.2.0-rat.txt.sha512 dev/hadoop/hadoop-thirdparty-1.2.0-RC1/hadoop-thirdparty-1.2.0-site.tar.gz (with props) dev/hadoop/hadoop-thirdparty-1.2.0-RC1/hadoop-thirdparty-1.2.0-site.tar.gz.asc (with props) dev/hadoop/hadoop-thirdparty-1.2.0-RC1/hadoop-thirdparty-1.2.0-site.tar.gz.sha512 dev/hadoop/hadoop-thirdparty-1.2.0-RC1/hadoop-thirdparty-1.2.0-src.tar.gz (with props) dev/hadoop/hadoop-thirdparty-1.2.0-RC1/hadoop-thirdparty-1.2.0-src.tar.gz.asc (with props) dev/hadoop/hadoop-thirdparty-1.2.0-RC1/hadoop-thirdparty-1.2.0-src.tar.gz.sha512 Added: dev/hadoop/hadoop-thirdparty-1.2.0-RC1/CHANGELOG.md == --- dev/hadoop/hadoop-thirdparty-1.2.0-RC1/CHANGELOG.md (added) +++ dev/hadoop/hadoop-thirdparty-1.2.0-RC1/CHANGELOG.md Sun Feb 4 08:27:38 2024 @@ -0,0 +1,58 @@ + + +# Apache Hadoop Third-party Libs Changelog + +## Release thirdparty-1.2.0 - 2024-02-04 + + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-18282](https://issues.apache.org/jira/browse/HADOOP-18282) | Add .asf.yaml to hadoop-thirdparty | Major | hadoop-thirdparty | Ayush Saxena | Ayush Saxena | +| [HADOOP-18721](https://issues.apache.org/jira/browse/HADOOP-18721) | Upgrade guava in hadoop-thirdparty to 31.1-jre | Major | hadoop-thirdparty | D M Murali Krishna Reddy | D M Murali Krishna Reddy | +| [HADOOP-18921](https://issues.apache.org/jira/browse/HADOOP-18921) | upgrade avro in hadoop-thirdparty to 1.11.3 | Major | hadoop-thirdparty | PJ Fanning | PJ Fanning | +| [HADOOP-18342](https://issues.apache.org/jira/browse/HADOOP-18342) | Upgrade to Avro 1.11.1 | Major | hadoop-thirdparty | PJ Fanning | PJ Fanning | +| [HADOOP-18197](https://issues.apache.org/jira/browse/HADOOP-18197) | Update protobuf 3.7.1 to a version without CVE-2021-22569 | Major | hadoop-thirdparty | Ivan Viaznikov | PJ Fanning | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-17707](https://issues.apache.org/jira/browse/HADOOP-17707) | Remove jaeger document from site index | Major | hadoop-thirdparty | Takanobu Asanuma | Takanobu Asanuma | +| [HADOOP-17730](https://issues.apache.org/jira/browse/HADOOP-17730) | [thirdparty] Do not exclude error\_prone | Major | hadoop-thirdparty | Wei-Chiu Chuang | Wei-Chiu Chuang | + + +### SUB-TASKS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-19055](https://issues.apache.org/jira/browse/HADOOP-19055) | Preparing for 1.3.0 development | Major | hadoop-thirdparty | Shilun Fan | Shilun Fan | +| [HADOOP-19054](https://issues.apache.org/jira/browse/HADOOP-19054) | Update hadoop-thirdparty index.md.vm | Major | hadoop-thirdparty | Shilun Fan | Shilun Fan | + + +### OTHER: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-18843](https://issues.apache.org/jira/browse/HADOOP-18843) | Guava version 32.0.1 bump to fix CVE-2023-2976 (hadoop-thirdparty PR#23) | Major | hadoop-thirdparty | Wei-Chiu Chuang | Wei-Chiu Chuang | + + Added: dev/hadoop/hadoop-thirdparty-1.2.0-RC1/CHANGELOG.md.asc == Binary file - no diff available. Propchange: dev/hadoop/hadoop-thirdparty-1.2.0-RC1/CHANGELOG.md.asc -- svn:mime-type = application/pgp-signature Added: dev/hadoop/hadoop-thirdparty-1.2.0-RC1/CHANGELOG.md.sha512 == --- dev/hadoop/hadoop-thirdparty-1.2.0-RC1/CHANGELOG.md.sha512 (added) +++ dev/hadoop/hadoop-thirdparty-1.2.0-RC1/CHANGELOG.md.sha512 Sun Feb 4 08:27:38 2024 @@ -0,0 +1 @@ +SHA512 (CHANGELOG.md) = 51eb75afd69bcddb64e0ffa31ca1604e5ed11fbd3b099f02d8fb38ed18e3122545d7e177862fbfc8956b9088fe234e591820ec621419bc411716bf4c2ba81ccc Added
(hadoop-thirdparty) annotated tag release-1.2.0-RC1 updated (64b6721 -> b2022f0)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to annotated tag release-1.2.0-RC1 in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git *** WARNING: tag release-1.2.0-RC1 was modified! *** from 64b6721 (commit) to b2022f0 (tag) tagging 64b672142797b5acffa94cadfc2d2be6f4dbbb23 (commit) replaces release-1.2.0-RC0 by Shilun Fan on Sun Feb 4 16:07:20 2024 +0800 - Log - Release candidate - 1.2.0-RC1 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEECm40PGXwzjlsDKJKl4RW45ZNtwEFAmW/RbgACgkQl4RW45ZN twEYLw/+IeoiYCJs63nCGBWppkkIrZy7jjMO6XLr2VNGkwtrsxIX9gFVQJy5k0VI M/6Ch0j4PNHKp0JVUyf5X3upS0ZnnlH1Mabmh5LcTLPORXxSIWEUoCDt7ljddVSU a7SnhyJUhugocClxiUYh27GuXo1YsmUGMwikqzNu45EKQzwslstOS60I0kcfJuoT DBq5kDL+cJ6gsCETKjRyC0K7HJMd19LStcdfitIXJi9o1o+ern6D6gSk+it7WpaP +gUGmaRuFC69VlhfJFBtomZziPQvA9u3KMXZLbrd1/I9gbE5js5DUyejr+knyeaa s7ZD0pIpPAmkhdiL4g72dByq0299Lkar8RdlDtcrSVFzEUfBISObbhIV5kgE9kG1 Ljneiikg7srtFIwn5SMH/QsJWc6O36GV94jk1wG1G1drSw8VnEbvvHiJrIDUx7hv ifWz2vqr/bMRsfBDBLMaUTM4DqJpjPgdJRqea2JMpxMUaLxpxvqeeAbP0JjlJKak o6EdYk7NcSPLaHVgAHolCH27pVTwRJrA1yC4mdXWdr8QTZS8SU5GL9K6aWdEk6IU UjJHMRHGgFkpL06c9Z/ip6R7ihbLHM0EWw6Qs1azfvE3M7qxy2YpcL5d96huuO0i 6XZkWIJMvBZShull20SsxXpNPUK1AX3n40f9lWw0ilJcSKjk6rw= =VRGb -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-thirdparty) branch branch-1.2.0 updated: HADOOP-19063. [Addendum] [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy. (#32) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-1.2.0 in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git The following commit(s) were added to refs/heads/branch-1.2.0 by this push: new 64b6721 HADOOP-19063. [Addendum] [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy. (#32) Contributed by Shilun Fan 64b6721 is described below commit 64b672142797b5acffa94cadfc2d2be6f4dbbb23 Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Sun Feb 4 14:23:44 2024 +0800 HADOOP-19063. [Addendum] [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy. (#32) Contributed by Shilun Fan Reviewed-by: He Xiaoqiao Signed-off-by: Shilun Fan --- dev-support/docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile index f6926df..e330383 100644 --- a/dev-support/docker/Dockerfile +++ b/dev-support/docker/Dockerfile @@ -124,9 +124,9 @@ ENV FINDBUGS_HOME /usr # Instead, download the last isort 4.x version. RUN pip2 install isort==4.3.21 -RUN pip2 install pylint==1.9.2 RUN pip2 install setuptools-scm==5.0.2 RUN pip2 install lazy-object-proxy==1.5.0 +RUN pip2 install pylint==1.9.2 # Install dateutil.parser - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-thirdparty) branch branch-1.2 updated: HADOOP-19063. [Addendum] [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy. (#32) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-1.2 in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git The following commit(s) were added to refs/heads/branch-1.2 by this push: new 6396dc4 HADOOP-19063. [Addendum] [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy. (#32) Contributed by Shilun Fan 6396dc4 is described below commit 6396dc4bc2f86f31fe012c3832f0edc79f9a0dad Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Sun Feb 4 14:23:44 2024 +0800 HADOOP-19063. [Addendum] [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy. (#32) Contributed by Shilun Fan Reviewed-by: He Xiaoqiao Signed-off-by: Shilun Fan --- dev-support/docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile index f6926df..e330383 100644 --- a/dev-support/docker/Dockerfile +++ b/dev-support/docker/Dockerfile @@ -124,9 +124,9 @@ ENV FINDBUGS_HOME /usr # Instead, download the last isort 4.x version. RUN pip2 install isort==4.3.21 -RUN pip2 install pylint==1.9.2 RUN pip2 install setuptools-scm==5.0.2 RUN pip2 install lazy-object-proxy==1.5.0 +RUN pip2 install pylint==1.9.2 # Install dateutil.parser - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-thirdparty) branch trunk updated: HADOOP-19063. [Addendum] [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy. (#32) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git The following commit(s) were added to refs/heads/trunk by this push: new ef7bdaa HADOOP-19063. [Addendum] [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy. (#32) Contributed by Shilun Fan ef7bdaa is described below commit ef7bdaab5e6ef0bda023ea2cc628d3acbea2361f Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Sun Feb 4 14:23:44 2024 +0800 HADOOP-19063. [Addendum] [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy. (#32) Contributed by Shilun Fan Reviewed-by: He Xiaoqiao Signed-off-by: Shilun Fan --- dev-support/docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile index f6926df..e330383 100644 --- a/dev-support/docker/Dockerfile +++ b/dev-support/docker/Dockerfile @@ -124,9 +124,9 @@ ENV FINDBUGS_HOME /usr # Instead, download the last isort 4.x version. RUN pip2 install isort==4.3.21 -RUN pip2 install pylint==1.9.2 RUN pip2 install setuptools-scm==5.0.2 RUN pip2 install lazy-object-proxy==1.5.0 +RUN pip2 install pylint==1.9.2 # Install dateutil.parser - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-thirdparty) annotated tag release-1.2.0-RC1 deleted (was dcc0c7c)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to annotated tag release-1.2.0-RC1 in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git *** WARNING: tag release-1.2.0-RC1 was deleted! *** tag was dcc0c7c The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-thirdparty) annotated tag release-1.2.0-RC1 updated (b843720 -> dcc0c7c)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to annotated tag release-1.2.0-RC1 in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git *** WARNING: tag release-1.2.0-RC1 was modified! *** from b843720 (commit) to dcc0c7c (tag) tagging b84372042d490ed23b405d6e05e44cadf109a47d (commit) replaces release-1.2.0-RC0 by Shilun Fan on Sun Feb 4 10:23:44 2024 +0800 - Log - Release candidate - 1.2.0-RC1 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEECm40PGXwzjlsDKJKl4RW45ZNtwEFAmW+9TAACgkQl4RW45ZN twHaMw/9HB/MqYR0cSChpzba3SxyOuzJQ55FJGeuMx9f0/Wp63oebfA3TG2c64ep l+SnRa2qZYQRQRBp9UlmCdWjHF67GhyC+xU5iED1Zctvad89CZeqliJcKBdChbSp QhZFczUSGfrgXrupHx724fTlJ/i+ar6g35tYdz25VFI2Uz8pPAcTl5ANwSssvazP hy1k/ZMB1/v0AnxBHVmXCDqs2JqZ0K2WDu72foe+LArAkVbrcpQKI4PyU8vbyMZM 0uEpGplehFSvOUG8zPFzmz+7AANKC8Bt4bnClg4cl8Ze8YcaSyui0rRoZo5u+vjb wjJCcNElnDOC1d1qrkUxmAM4BJIJ/bcFgyhwSM6HG2gRrtOTGp+4x1OAjwA+Yt6I bICxBpb3jkaU9D6SXUiBCfSZssOXtTGp0bipd3rKs4in3FMGju/EWgKoSeNiCrHN qQY/yyKwbZNDmHFrP758IWeLMEAqUtiHgqJJl+KfiJg+QP6fnblbPhw7/AppRE/K IMOXu9dS9IY9qOen2wGyRDZMg6B/R+fTP791rtKKN4DDmG4N0nbNZey+ze9uvC+e 63Iqp/kSCMVkeL4zQokByZgMtBjU804zF6Wt4DZdInhVfsidVAyesnVC0FOPqfVE sZ16BSOdov2cIf5WTB0GdyccXGJvDjMMKMFuJM/6Dm03vm0BReQ= =16hJ -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-thirdparty) branch branch-1.2.0 updated: HADOOP-19063. [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy (#30) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch branch-1.2.0 in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git The following commit(s) were added to refs/heads/branch-1.2.0 by this push: new b843720 HADOOP-19063. [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy (#30) Contributed by Shilun Fan b843720 is described below commit b84372042d490ed23b405d6e05e44cadf109a47d Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Sun Feb 4 08:39:57 2024 +0800 HADOOP-19063. [thirdparty] Fix the docker image setuptools-scm && lazy-object-proxy (#30) Contributed by Shilun Fan Reviewed-by: Steve Loughran Reviewed-by: He Xiaoqiao Signed-off-by: Shilun Fan --- dev-support/docker/Dockerfile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile index 187fc64..f6926df 100644 --- a/dev-support/docker/Dockerfile +++ b/dev-support/docker/Dockerfile @@ -125,6 +125,8 @@ ENV FINDBUGS_HOME /usr RUN pip2 install isort==4.3.21 RUN pip2 install pylint==1.9.2 +RUN pip2 install setuptools-scm==5.0.2 +RUN pip2 install lazy-object-proxy==1.5.0 # Install dateutil.parser - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-thirdparty) branch branch-1.2.0 updated: HADOOP-19064. [thirdparty] add -mvnargs option to create-release command line. (#31) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-1.2.0
in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git
The following commit(s) were added to refs/heads/branch-1.2.0 by this push:
new ebba1b0 HADOOP-19064. [thirdparty] add -mvnargs option to
create-release command line. (#31) Contributed by Shilun Fan
ebba1b0 is described below
commit ebba1b00bb2d47caec64f45b27f360f2926242c8
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Sun Feb 4 08:39:10 2024 +0800
HADOOP-19064. [thirdparty] add -mvnargs option to create-release command
line. (#31) Contributed by Shilun Fan
Reviewed-by: Steve Loughran
Reviewed-by: He Xiaoqiao
Signed-off-by: Shilun Fan
---
dev-support/bin/create-release | 8
1 file changed, 8 insertions(+)
diff --git a/dev-support/bin/create-release b/dev-support/bin/create-release
index 30ced9b..2f296e6 100755
--- a/dev-support/bin/create-release
+++ b/dev-support/bin/create-release
@@ -241,6 +241,7 @@ function usage
echo "--security Emergency security release"
echo "--sign Use .gnupg dir to sign the artifacts and jars"
echo "--version=[version] Use an alternative version string"
+ echo "--mvnargs=[args]Extra Maven args to be provided when running
mvn commands"
}
function option_parse
@@ -291,6 +292,9 @@ function option_parse
--version=*)
HADOOP_THIRDPARTY_VERSION=${i#*=}
;;
+ --mvnargs=*)
+MVNEXTRAARGS=${i#*=}
+ ;;
esac
done
@@ -358,6 +362,10 @@ function option_parse
fi
fi
+ if [ -n "$MVNEXTRAARGS" ]; then
+MVN_ARGS+=("$MVNEXTRAARGS")
+ fi
+
if [[ "${SECURITYRELEASE}" = true ]]; then
if [[ ! -d
"${BASEDIR}/src/site/markdown/release/${HADOOP_THIRDPARTY_VERSION}" ]]; then
hadoop_error "ERROR:
${BASEDIR}/src/site/markdown/release/${HADOOP_THIRDPARTY_VERSION} does not
exist."
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-thirdparty) branch branch-1.2.0 updated: HADOOP-19062. Improve create-release RUN script. (#29) Contributed by Shilun Fan
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-1.2.0
in repository https://gitbox.apache.org/repos/asf/hadoop-thirdparty.git
The following commit(s) were added to refs/heads/branch-1.2.0 by this push:
new 80b93ee HADOOP-19062. Improve create-release RUN script. (#29)
Contributed by Shilun Fan
80b93ee is described below
commit 80b93ee9df07731a7bd04aa5e4b958c67a547efb
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Sun Feb 4 08:37:12 2024 +0800
HADOOP-19062. Improve create-release RUN script. (#29) Contributed by
Shilun Fan
Reviewed-by: He Xiaoqiao
Signed-off-by: Shilun Fan
---
dev-support/bin/create-release | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/dev-support/bin/create-release b/dev-support/bin/create-release
index 7859081..30ced9b 100755
--- a/dev-support/bin/create-release
+++ b/dev-support/bin/create-release
@@ -441,9 +441,9 @@ function dockermode
echo "LABEL org.apache.hadoop.create-release=\"cr-${RANDOM}\""
# setup ownerships, etc
-echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"
-echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}"
-echo "RUN chown -R ${user_name} /home/${user_name}"
+echo "RUN groupadd --non-unique -g ${group_id} ${user_name}; exit 0;"
+echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}; exit 0;"
+echo "RUN chown -R ${user_name} /home/${user_name}; exit 0;"
echo "ENV HOME /home/${user_name}"
echo "RUN mkdir -p /maven"
echo "RUN chown -R ${user_name} /maven"
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
