This is an automated email from the ASF dual-hosted git repository.
stevel pushed a change to branch feature-HADOOP-18073-s3a-sdk-upgrade-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
discard 1fe918cdf73 HADOOP-18073. Upgrade S3A in 3.3 branch to AWS SDK
V2.(#5707)
add 60b37bbdf71 HADOOP-18433. Fix main thread name for . (#4838) (#5692)
add a804f37ed52 HDFS-17003. Erasure Coding: invalidate wrong block after
reporting bad blocks from datanode (#5643). Contributed by hfutatzhanghb.
add afb863acf44 HADOOP-18740. S3A prefetch cache blocks should be accessed
by RW locks (#5675)
add 30dcd044c37 HADOOP-17386. Change default fs.s3a.buffer.dir to be under
Yarn container path on yarn applications (#3908)
add 03a548d4e56 HADOOP-18646. Upgrade Netty to 4.1.89.Final to fix
CVE-2022-41881 (#5435) (#5729)
add 0fd2b10bdc0 HDFS-16898. Remove write lock for processCommandFromActor
of DataNode to reduce impact on heartbeat. (#5408). Contributed by ZhangHB.
add 936e9e15d04 MAPREDUCE-7435. Manifest Committer OOM on abfs (#5519)
add 20d073cb2c5 HADOOP-18718. Fix several maven build warnings (#5592).
Contributed by Dongjoon Hyun.
add dea53dad679 HADOOP-18761. Remove mysql-connector-java (#5731)
add b8a76f6240b HADOOP-18622. Upgrade ant to 1.10.13 (#5360). Contributed
by Aleksandr Nikolaev.
add 50125e249f7 HADOOP-18538. Upgrade kafka to 2.8.2 (#5164)
add 4f6ebabdba8 HADOOP-18763. Upgrade aws-java-sdk to 1.12.367 (#5741)
add 51a7f7b024e HADOOP-18756. S3A prefetch - CachingBlockManager to use
AtomicBoolean for closed flag (#5718)
add cc121e2124a Post release update
add 384891ceeaf HADOOP-18782. Upgrade to snappy-java 1.1.10.1 due to CVEs
(#5773)
add 48f1a3c5483 MAPREDUCE-7432. Make manifest committer default on abfs
and gcs stores (#5378)
add 29362cdf104 HADOOP-18773. Upgrade maven-shade-plugin to 3.4.1 (#5750)
add b0377379af1 HADOOP-18764. fs.azure.buffer.dir to be under Yarn
container path on yarn applications (#5788)
add 6b9cfe46909 HDFS-17055 Export HAState as a metric from Namenode for
monitoring (#5790)
add f02fa6683d2 HADOOP-18793. S3A StagingCommitter does not clean up
staging-uploads directory (#5818)
add b85b7b55a5a YARN-11528. Lock triple-beam to the version compatible
with node.js 12 to avoid compilation error. (#5827). Contributed by Masatake
Iwasaki
add e4b39b9cb0a HADOOP-18781. ABFS backReference passed down to streams to
avoid GC closing the FS. (#5830)
add 2805189237f HADOOP-18291. S3A prefetch - Implement thread-safe LRU
cache for SingleFilePerBlockCache (#5754)
add adab0babc74 HADOOP-18809. S3A prefetch read/write file operations
should guard channel close (#5853)
add 850b8ab5a01 HADOOP-18808. LogExactlyOnce to add a debug() method
(#5850)
add 236b9aa124e HADOOP-18757. S3A Committer only finalizes the commits in
a single thread (#5706)
add 8ee580aabf8 HDFS-17042 Add rpcCallSuccesses and
OverallRpcProcessingTime to RpcMetrics for Namenode (#5730) (#5804)
add 708ee3b4b8c HDFS-17094. EC: Fix bug in block recovery when there are
stale datanodes. (#5854)
add caeed401b2e HADOOP-18795. S3A DelegationToken plugin to expand return
type of binding (#5821)
add d567a714095 HDFS-13916. Distcp SnapshotDiff to support WebHDFS.
(#5839)
add 210104e99e2 HADOOP-18752. Change fs.s3a.directory.marker.retention to
"keep" -everything but the switch (#5859)
add d4deb9bebae HADOOP-18073. Upgrade S3A in 3.3 branch to AWS SDK
V2.(#5707)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (1fe918cdf73)
\
N -- N -- N refs/heads/feature-HADOOP-18073-s3a-sdk-upgrade-3.3
(d4deb9bebae)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
LICENSE-binary | 14 +-
.../hadoop-client-check-invariants/pom.xml | 1 -
.../hadoop-client-check-test-invariants/pom.xml | 1 -
.../jdiff/Apache_Hadoop_Common_3.3.6.xml | 40994 +++++++++++++++++++
hadoop-common-project/hadoop-common/pom.xml | 1 -
.../BackReference.java} | 52 +-
.../fs/impl/prefetch/CachingBlockManager.java | 12 +-
.../hadoop/fs/impl/prefetch/PrefetchConstants.java | 44 +
.../fs/impl/prefetch/SingleFilePerBlockCache.java | 310 +-
.../hadoop/fs/statistics/IOStatisticsSetters.java | 75 +
.../hadoop/fs/statistics/IOStatisticsSnapshot.java | 30 +-
.../impl/ForwardingIOStatisticsStore.java | 3 +-
.../fs/statistics/impl/IOStatisticsStore.java | 48 +-
.../statistics/impl/IOStatisticsStoreBuilder.java | 11 +
.../impl/IOStatisticsStoreBuilderImpl.java | 12 +
.../org/apache/hadoop/fs/store/LogExactlyOnce.java | 11 +
.../org/apache/hadoop/ipc/ProcessingDetails.java | 15 +-
.../main/java/org/apache/hadoop/ipc/Server.java | 20 +-
.../hadoop/ipc/metrics/RpcDetailedMetrics.java | 25 +-
.../org/apache/hadoop/ipc/metrics/RpcMetrics.java | 10 +
.../metrics2/lib/MutableRatesWithAggregation.java | 4 +-
.../hadoop/util/functional/RemoteIterators.java | 124 +-
.../src/main/resources/core-default.xml | 13 +-
.../hadoop-common/src/site/markdown/Metrics.md | 7 +-
.../site/markdown/release/3.3.6/CHANGELOG.3.3.6.md | 176 +
.../markdown/release/3.3.6/RELEASENOTES.3.3.6.md | 31 +
.../hadoop/fs/impl/prefetch/TestBlockCache.java | 6 +-
.../fs/statistics/TestIOStatisticsSetters.java | 189 +
.../test/java/org/apache/hadoop/ipc/TestRPC.java | 80 +
.../hadoop/metrics2/lib/TestMutableMetrics.java | 15 +-
.../org/apache/hadoop/test/MetricsAsserts.java | 13 +
.../util/functional/TestRemoteIterators.java | 56 +-
hadoop-dist/pom.xml | 3 -
hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml | 18 +-
.../security/token/SQLConnectionFactory.java | 1 -
.../dev-support/jdiff/Apache_Hadoop_HDFS_3.3.6.xml | 835 +
hadoop-hdfs-project/hadoop-hdfs/pom.xml | 4 +-
.../hdfs/server/blockmanagement/BlockManager.java | 17 +-
.../BlockUnderConstructionFeature.java | 8 +
.../server/blockmanagement/DatanodeManager.java | 11 +-
.../hdfs/server/datanode/BPOfferService.java | 34 +-
.../hdfs/server/datanode/BPServiceActor.java | 2 +-
.../hadoop/hdfs/server/namenode/BackupNode.java | 2 +
.../hadoop/hdfs/server/namenode/NameNode.java | 25 +
.../hdfs/ReadStripedFileWithDecodingHelper.java | 22 +-
.../org/apache/hadoop/hdfs/TestDFSFinalize.java | 2 +-
.../org/apache/hadoop/hdfs/TestDFSRollback.java | 2 +-
.../org/apache/hadoop/hdfs/TestDFSUpgrade.java | 4 +-
.../hadoop/hdfs/TestLeaseRecoveryStriped.java | 60 +
.../hdfs/TestReadStripedFileWithDecoding.java | 127 +-
.../server/namenode/TestNameNodeMetricsLogger.java | 2 +
.../hdfs/server/namenode/ha/TestHAMetrics.java | 55 +
.../jdiff/Apache_Hadoop_MapReduce_Common_3.3.6.xml | 113 +
.../jdiff/Apache_Hadoop_MapReduce_Core_3.3.6.xml | 28994 +++++++++++++
.../Apache_Hadoop_MapReduce_JobClient_3.3.6.xml | 16 +
.../hadoop-mapreduce-client-core/pom.xml | 4 +-
.../committer/manifest/ManifestCommitter.java | 8 +-
.../manifest/ManifestCommitterConfig.java | 21 +-
.../manifest/ManifestCommitterConstants.java | 22 +-
.../committer/manifest/files/DiagnosticKeys.java | 3 +
.../output/committer/manifest/files/DirEntry.java | 26 +-
.../output/committer/manifest/files/FileEntry.java | 36 +-
.../committer/manifest/impl/EntryFileIO.java | 569 +
.../manifest/impl/LoadedManifestData.java | 99 +
.../manifest/impl/ManifestCommitterSupport.java | 24 +-
.../manifest/stages/AbstractJobOrTaskStage.java | 26 +-
.../committer/manifest/stages/CleanupJobStage.java | 2 +-
.../committer/manifest/stages/CommitJobStage.java | 201 +-
.../stages/CreateOutputDirectoriesStage.java | 88 +-
.../manifest/stages/LoadManifestsStage.java | 305 +-
.../manifest/stages/RenameFilesStage.java | 58 +-
.../committer/manifest/stages/StageConfig.java | 72 +
.../manifest/stages/ValidateRenamedFilesStage.java | 51 +-
.../src/main/resources/mapred-default.xml | 12 +-
.../src/site/markdown/manifest_committer.md | 58 +-
.../lib/output/TestFileOutputCommitter.java | 110 +-
.../manifest/AbstractManifestCommitterTest.java | 6 +-
.../manifest/ManifestCommitterTestSupport.java | 53 +-
.../manifest/TestCreateOutputDirectoriesStage.java | 39 +-
.../manifest/TestJobThroughManifestCommitter.java | 98 +-
.../committer/manifest/TestLoadManifestsStage.java | 97 +-
.../committer/manifest/TestRenameStageFailure.java | 32 +-
.../committer/manifest/impl/TestEntryFileIO.java | 382 +
hadoop-project-dist/pom.xml | 3 +-
hadoop-project/pom.xml | 18 +-
.../java/org/apache/hadoop/fs/s3a/Constants.java | 58 +-
.../apache/hadoop/fs/s3a/S3AInstrumentation.java | 2 +-
.../delegation/AbstractDelegationTokenBinding.java | 35 +-
.../s3a/auth/delegation/DelegationBindingInfo.java | 40 +-
.../s3a/auth/delegation/S3ADelegationTokens.java | 40 +-
.../hadoop/fs/s3a/commit/impl/CommitContext.java | 2 +-
.../apache/hadoop/fs/s3a/commit/staging/Paths.java | 13 +
.../fs/s3a/commit/staging/StagingCommitter.java | 32 +-
.../hadoop/fs/s3a/impl/DirectoryPolicyImpl.java | 4 +-
.../fs/s3a/prefetch/S3ACachingBlockManager.java | 11 +-
.../site/markdown/tools/hadoop-aws/committers.md | 4 +-
.../markdown/tools/hadoop-aws/directory_markers.md | 233 +-
.../src/site/markdown/tools/hadoop-aws/index.md | 23 +-
.../src/site/markdown/tools/hadoop-aws/s3guard.md | 10 +-
.../src/site/markdown/tools/hadoop-aws/testing.md | 36 +-
.../fs/s3a/ITestS3APrefetchingLruEviction.java | 243 +
.../fs/s3a/commit/AbstractITCommitProtocol.java | 24 +
.../integration/ITestStagingCommitProtocol.java | 70 +
.../ITestStagingCommitProtocolFailure.java | 2 +
.../hadoop/fs/s3a/prefetch/S3APrefetchFakes.java | 6 +-
.../s3a/prefetch/TestS3ACachingBlockManager.java | 20 +-
.../hadoop/fs/azurebfs/AzureBlobFileSystem.java | 9 +-
.../fs/azurebfs/AzureBlobFileSystemStore.java | 14 +
.../commit/AbfsManifestStoreOperations.java | 14 +-
.../fs/azurebfs/services/AbfsInputStream.java | 10 +
.../azurebfs/services/AbfsInputStreamContext.java | 14 +
.../fs/azurebfs/services/AbfsOutputStream.java | 21 +
.../azurebfs/services/AbfsOutputStreamContext.java | 14 +
.../fs/azurebfs/commit/AbfsCommitTestHelper.java | 11 +
.../commit/ITestAbfsLoadManifestsStage.java | 14 +
.../fs/azurebfs/services/ITestAbfsInputStream.java | 21 +
.../azurebfs/services/ITestAbfsOutputStream.java | 78 +
hadoop-tools/hadoop-benchmark/pom.xml | 7 -
hadoop-tools/hadoop-distcp/pom.xml | 24 +-
.../java/org/apache/hadoop/tools/DistCpSync.java | 74 +-
.../org/apache/hadoop/tools/TestDistCpSync.java | 161 +-
.../jdiff/Apache_Hadoop_YARN_API_3.3.6.xml | 26440 ++++++++++++
.../jdiff/Apache_Hadoop_YARN_Client_3.3.6.xml | 3067 ++
.../jdiff/Apache_Hadoop_YARN_Common_3.3.6.xml | 3987 ++
.../Apache_Hadoop_YARN_Server_Common_3.3.6.xml | 1456 +
.../package.json | 3 +
.../hadoop-yarn/hadoop-yarn-common/pom.xml | 4 +-
.../hadoop-yarn-server-common/pom.xml | 4 +-
pom.xml | 4 +
129 files changed, 110660 insertions(+), 982 deletions(-)
create mode 100644
hadoop-common-project/hadoop-common/dev-support/jdiff/Apache_Hadoop_Common_3.3.6.xml
copy
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/{store/LogExactlyOnce.java
=> impl/BackReference.java} (50%)
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/PrefetchConstants.java
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/IOStatisticsSetters.java
rename {hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a =>
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs}/statistics/impl/ForwardingIOStatisticsStore.java
(97%)
create mode 100644
hadoop-common-project/hadoop-common/src/site/markdown/release/3.3.6/CHANGELOG.3.3.6.md
create mode 100644
hadoop-common-project/hadoop-common/src/site/markdown/release/3.3.6/RELEASENOTES.3.3.6.md
create mode 100644
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/statistics/TestIOStatisticsSetters.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/dev-support/jdiff/Apache_Hadoop_HDFS_3.3.6.xml
create mode 100644
hadoop-mapreduce-project/dev-support/jdiff/Apache_Hadoop_MapReduce_Common_3.3.6.xml
create mode 100644
hadoop-mapreduce-project/dev-support/jdiff/Apache_Hadoop_MapReduce_Core_3.3.6.xml
create mode 100644
hadoop-mapreduce-project/dev-support/jdiff/Apache_Hadoop_MapReduce_JobClient_3.3.6.xml
create mode 100644
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/impl/EntryFileIO.java
create mode 100644
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/impl/LoadedManifestData.java
create mode 100644
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/impl/TestEntryFileIO.java
copy
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/files/DiagnosticKeys.java
=>
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/DelegationBindingInfo.java
(51%)
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingLruEviction.java
create mode 100644
hadoop-yarn-project/hadoop-yarn/dev-support/jdiff/Apache_Hadoop_YARN_API_3.3.6.xml
create mode 100644
hadoop-yarn-project/hadoop-yarn/dev-support/jdiff/Apache_Hadoop_YARN_Client_3.3.6.xml
create mode 100644
hadoop-yarn-project/hadoop-yarn/dev-support/jdiff/Apache_Hadoop_YARN_Common_3.3.6.xml
create mode 100644
hadoop-yarn-project/hadoop-yarn/dev-support/jdiff/Apache_Hadoop_YARN_Server_Common_3.3.6.xml
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
