This is an automated email from the ASF dual-hosted git repository.
keepromise pushed a change to branch HDFS-17531
in repository https://gitbox.apache.org/repos/asf/hadoop.git
omit df6a5b618dc HDFS-17596. [ARR] RouterStoragePolicy supports
asynchronous rpc. (#6988). Contributed by hfutatzhanghb.
omit 463de517dfc HDFS-17601. [ARR] RouterRpcServer supports asynchronous
rpc. (#7108). Contributed by hfutatzhanghb.
omit aba8944339f HDFS-17595. [ARR] ErasureCoding supports asynchronous rpc.
(#6983). Contributed by hfutatzhanghb.
omit 7d4a628b005 HDFS-17597. [ARR] RouterSnapshot supports asynchronous
rpc. (#6994). Contributed by Wenqi Li.
omit 0cd15000b85 HDFS-17594. [ARR] RouterCacheAdmin supports asynchronous
rpc. (#6986). Contributed by Archie73.
omit 8a61436d406 HDFS-17545. [ARR] router async rpc client. (#6871).
Contributed by Jian Zhang.
omit a5919024552 HDFS-17544. [ARR] The router client rpc protocol PB
supports asynchrony. (#6870). Contributed by Jian Zhang.
omit 1f91aefc0f0 HADOOP-19235. IPC client uses CompletableFuture to support
asynchronous operations. (#6888)
omit 4bb587ea2ba HDFS-17543. [ARR] AsyncUtil makes asynchronous code more
concise and easier. (#6868). Contributed by Jian Zhang.
add ea6e0f7cd58 HADOOP-19221. S3A: Unable to recover from failure of
multipart block upload attempt (#6938)
add 4d968add52d HADOOP-19271. NPE in
AbfsManagedApacheHttpConnection.toString() when not connected (#7040)
add 182feb11a0f HADOOP-19277. Files and directories mixed up in
TreeScanResults#dump (#7047)
add d1311e52f78 YARN-11709. NodeManager should be marked unhealthy on
localizer config issues (#7043)
add ee2e5ac4e41 HADOOP-19272. S3A: AWS SDK 2.25.53 warnings logged by
transfer manager (#7048)
add 6bcc2541235 HADOOP-19279. ABFS: Disabling Apache Http Client as
Default Http Client for ABFS Driver(#7055)
add 28538d628ec HADOOP-19164. Hadoop CLI MiniCluster is broken (#7050).
Contributed by Ayush Saxena.
add ff75aa52f3a HDFS-17611. Move all DistCp execution logic to execute()
(#7025)
add 81faae63437 Revert "HDFS-17611. Move all DistCp execution logic to
execute() (#7025)" (#7059)
add d8ca2dbe34b YARN-11730. Mark unreported nodes as LOST on RM Startup/HA
failover (#7049) Contributed by Arjun Mohnot.
add 50fcfd8b298 HDFS-17526. getMetadataInputStream should use
getShareDeleteFileInputStream for windows (#6826). Contributed by Danny Becker
add ae63d8edd54 HDFS-17621. Make PathIsNotEmptyDirectoryException terse
(#7036). Contributed by dzcxzl.
add f5345b881d9 YARN-11560. Fix NPE bug when multi-node enabled with
schedule asynchronously (#6021). Contributed by wangzhongwei.
add fccc268cdea HADOOP-19283. Move all DistCp execution logic to execute()
(#7060)
add 37a74f06921 HADOOP-19285. [ABFS] Restore ETAGS_AVAILABLE to abfs path
capabilities (#7064)
add 68315744f06 HDFS-17040. Namenode web UI should set content type to
application/octet-stream when uploading a file. (#5721)
add b078f86d69c YARN-11733. Fix the order of updating CPU controls with
cgroup v1 (#7069)
add f90a703e480 HADOOP-19165. Drop protobuf 2.5.0 from the distribution
(#7051). Contributed by Ayush Saxena.
add e602c601ddd HADOOP-15760. Upgrade commons-collections to
commons-collections4 (#7006)
add 21ec686be3a YARN-11702: Fix Yarn over allocating containers (#6990)
Contributed by Syed Shameerur Rahman.
add 49a495803a9 HDFS-17381. Distcp of EC files should not be limited to
DFS. (#6551)
add 21cdb450ef2 HADOOP-19284: [ABFS] Allow "fs.azure.account.hns.enabled"
to be set as Account Specific Config (#7062)
add 3d81dde28b6 HDFS-17624. Fix
DFSNetworkTopology#chooseRandomWithStorageType() availableCount when excluded
node is not in selected scope. (#7042). Contributed by fuchaohong.
add 01401d71eff HADOOP-19281. MetricsSystemImpl should not print INFO
message in CLI (#7071)
add 3fda2434196 HADOOP-19290. Operating on / in ChecksumFileSystem throws
NPE. (#7074). Contributed by Ayush Saxena.
add a9b7913d568 HDFS-17626. Reduce lock contention at datanode startup
(#7053). Contributed by Tao Li.
add 9aca73481e0 HADOOP-19280. [ABFS] Initialize client timer only if
metric collection is enabled (#7061)
add 6fd4fea748e HADOOP-19261. Support force close a DomainSocket for
server service (#7057)
add 45b1c86fe51 HADOOP-19294. NPE on maven enforcer with -Pnative on arm
mac (#7082)
add 4ff0dceebda HADOOP-19288. hadoop-client-runtime to exclude dnsjava
InetAddressResolverProvider (#7070)
add e9ed21c065f HADOOP-19286: S3A: Support cross region access when S3
region/endpoint is set (#7067)
add 3f637efaa2c HADOOP-19219. Add JPMS options required by hadoop-common
(#7084) Contributed by Cheng Pan.
add b781882020f YARN-11734. Fix spotbugs in ServiceScheduler#load (#7088)
Contributed by Hualong Zhang.
add 1b5a2a7f65f YARN-11708: Setting maximum-application-lifetime using
AQCv2 templates doesn't apply on the first submitted app (#7041)
add 4e6432a0abf HADOOP-19296. [JDK17] Upgrade maven-war-plugin to 3.4.0.
(#7086) Contributed by Shilun Fan.
add 5ea3a1bd0a5 HADOOP-19286: S3A: Support cross region access when S3
region/endpoint is set (ADDENDUM) (#7098)
add 1f0d9df8875 HDFS-17637. Fix spotbugs in HttpFSFileSystem#getXAttr
(#7099) Contributed by Hualong Zhang.
add 50e6b49e057 HADOOP-19299. HttpReferrerAuditHeader resilience (#7095)
add dc56fc385a4 HADOOP-19295. S3A: large uploads can timeout over slow
links (#7089)
add e4b070025b3 HADOOP-19291. RawLocalFileSystem to allow overlapping
ranges (#7101)
add 9321e322d2a HADOOP-19310. Add JPMS options required by Java 17+
(#7114) Contributed by Cheng Pan.
add 78a08b3b782 MAPREDUCE-7494. File stream leak when LineRecordReader is
interrupted (#7117)
add c63aafd7d1c YARN-11732. Fix potential NPE when calling
SchedulerNode#reservedContainer for CapacityScheduler (#7065). Contributed by
Tao Yang.
add 6589d9f6aae HDFS-17631. Fix RedundantEditLogInputStream.nextOp() state
error when EditLogInputStream.skipUntil() throw IOException (#7066).
Contributed by liuguanghua.
add f931ede86b6 HADOOP-19298. [JDK17] Add a JDK17 profile. (#7085)
Contributed by Shilun Fan.
add 09b348753f8 HDFS-17634. RBF: Fix web UI missing DN last block report
(#7080)
add d1ce9656459 HDFS-17636. Don't add declspec for Windows (#7096)
add 0b3755347c8 HADOOP-19309: S3A: CopyFromLocalFile operation fails when
the source file does not contain file scheme (#7113)
add eb1e30395b3 HDFS-17644:Add log when a node selection is rejected by
BPP UpgradeDomain (#7109)
add 7a7b346b0ab Revert "HADOOP-19298. [JDK17] Add a JDK17 profile. (#7085)
Contributed by Shilun Fan." (#7132)
add 00cddf5bea9 HDFS-17646. Add Option to limit Balancer overUtilized
nodes num in each iteration. (#7120). Contributed by Zhaobo Huang.
add 4f3abd2f482 HDFS-17654. Fix bugs in TestRouterMountTable (#7137).
Contributed by farmmamba.
add e4789a2fd3e HDFS-17607. Reduce the number of times conf is loaded when
DataNode startUp (#7012). Contributed by lei w.
add 66baf1eb51d HADOOP-18682. Move hadoop docker scripts under the main
source code (#6483). Contributed by Christos Bisias.
add df979e70de9 HADOOP-19297. [JDK17] Upgrade maven.plugin-tools.version
to 3.10.2 (#7125) Contributed by Min Yan.
add 9ae01bdbe83 HADOOP-19143. Upgrade commons-cli to 1.9.0 (#7126)
Contributed by Min Yan.
add 51ebc3c20e8 HADOOP-18583. Fix loading of OpenSSL 3.x symbols (#5256)
add f7651e2f63d HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968)
add 487727a5d10 HADOOP-18960: [ABFS] Making Contract tests run in
sequential and Other Test Fixes (#7104)
add 96572764921 HDFS-17642. Add target node list, exclude source node
list, and exclude target node list parameters to balancer (#7127)
add 9a743bd17f5 HADOOP-19315. Upgrade Apache Avro to 1.11.4 (#7128)
add 2273278d0b0 HADOOP-18708: S3A: Support S3 Client Side Encryption(CSE)
(#6884)
add 7999db55da4 HADOOP-19330. S3A: Add LeakReporter; use in S3AInputStream
(#7151)
add 317db31a9a5 HADOOP-19306. Support user defined auth Callback in
SaslRpcServer. (#7140)
add 7543f3aadab HADOOP-19317. S3A: fs.s3a.connection.expect.continue
controls 100 CONTINUE behavior (#7134)
add c2f13cb7621 HADOOP-18325: [ABFS] Fix metric related test failures due
to missing config (#6847)
add a3e5258a4ce HDFS-17543. [ARR] AsyncUtil makes asynchronous code more
concise and easier. (#6868). Contributed by Jian Zhang.
add ff2a574cd72 HADOOP-19235. IPC client uses CompletableFuture to support
asynchronous operations. (#6888)
add 92173176713 HDFS-17544. [ARR] The router client rpc protocol PB
supports asynchrony. (#6870). Contributed by Jian Zhang.
add 71c74660b61 HDFS-17545. [ARR] router async rpc client. (#6871).
Contributed by Jian Zhang.
add 724c7d3e1bb HDFS-17594. [ARR] RouterCacheAdmin supports asynchronous
rpc. (#6986). Contributed by Archie73.
add 1152442a98e HDFS-17597. [ARR] RouterSnapshot supports asynchronous
rpc. (#6994). Contributed by Wenqi Li.
add 2aaa1176563 HDFS-17595. [ARR] ErasureCoding supports asynchronous rpc.
(#6983). Contributed by hfutatzhanghb.
add 118061c4435 HDFS-17601. [ARR] RouterRpcServer supports asynchronous
rpc. (#7108). Contributed by hfutatzhanghb.
add ea3c4c84ba2 HDFS-17596. [ARR] RouterStoragePolicy supports
asynchronous rpc. (#6988). Contributed by hfutatzhanghb.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (df6a5b618dc)
\
N -- N -- N refs/heads/HDFS-17531 (ea3c4c84ba2)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
LICENSE-binary | 7 +-
.../hadoop-client-minicluster/pom.xml | 17 +-
.../hadoop-client-runtime/pom.xml | 7 +-
.../hadoop-huaweicloud/pom.xml | 18 +-
hadoop-common-project/hadoop-auth/pom.xml | 5 +
hadoop-common-project/hadoop-common/pom.xml | 12 +-
.../hadoop-common/src/main/bin/hadoop-functions.sh | 23 +
.../java/org/apache/hadoop/conf/Configuration.java | 6 +-
.../org/apache/hadoop/fs/ChecksumFileSystem.java | 2 +-
.../hadoop/fs/CommonConfigurationKeysPublic.java | 6 +
.../main/java/org/apache/hadoop/fs/FileUtil.java | 21 +-
.../org/apache/hadoop/fs/MultipartUploader.java | 4 +-
.../main/java/org/apache/hadoop/fs/Options.java | 5 +
.../org/apache/hadoop/fs/RawLocalFileSystem.java | 7 +-
.../org/apache/hadoop/fs/WithErasureCoding.java | 50 ++
.../hadoop/fs/impl/AbstractMultipartUploader.java | 2 +-
.../fs/impl/FileSystemMultipartUploader.java | 2 +-
.../org/apache/hadoop/fs/impl/LeakReporter.java | 143 ++++++
.../hadoop/fs/statistics/StoreStatisticNames.java | 41 ++
.../hadoop/fs/statistics/StreamStatisticNames.java | 8 +
.../hadoop/fs/store/ByteBufferInputStream.java | 199 +++++++
.../org/apache/hadoop/fs/store/DataBlocks.java | 155 +-----
.../fs/store/audit/HttpReferrerAuditHeader.java | 45 +-
.../main/java/org/apache/hadoop/ipc/Server.java | 8 +-
.../apache/hadoop/metrics2/impl/MetricsConfig.java | 14 +-
.../hadoop/metrics2/impl/MetricsSystemImpl.java | 19 +-
.../org/apache/hadoop/net/unix/DomainSocket.java | 71 ++-
.../security}/CustomizedCallbackHandler.java | 67 ++-
.../hadoop/security/JniBasedUnixGroupsMapping.java | 2 +-
.../org/apache/hadoop/security/SaslConstants.java | 45 --
.../hadoop/security/SaslMechanismFactory.java | 75 +++
.../org/apache/hadoop/security/SaslRpcServer.java | 59 ++-
.../org/apache/hadoop/security/package-info.java | 14 +-
.../apache/hadoop/util/functional/FutureIO.java | 91 +++-
.../hadoop/util/functional/RunnableRaisingIOE.java | 34 +-
.../src/org/apache/hadoop/crypto/OpensslCipher.c | 68 ++-
.../src/main/resources/core-default.xml | 17 +
.../src/site/markdown/CLIMiniCluster.md.vm | 6 +-
.../src/site/markdown/HadoopDocker.md | 68 +++
.../src/site/markdown/SingleCluster.md.vm | 6 +
.../site/markdown/filesystem/fsdatainputstream.md | 9 +-
.../apache/hadoop/constants/ConfigConstants.java | 24 +-
.../org/apache/hadoop/constants/package-info.java | 13 +-
.../apache/hadoop/fs/TestChecksumFileSystem.java | 7 +
.../test/java/org/apache/hadoop/fs/TestPath.java | 4 +
.../AbstractContractMultipartUploaderTest.java | 36 +-
.../contract/AbstractContractVectoredReadTest.java | 38 +-
.../apache/hadoop/fs/contract/ContractOptions.java | 2 +
.../hadoop/fs/contract/ContractTestUtils.java | 4 +-
.../apache/hadoop/fs/impl/TestLeakReporter.java | 165 ++++++
.../test/java/org/apache/hadoop/io/TestText.java | 4 +
.../io/serializer/avro/TestAvroSerialization.java | 2 +-
.../java/org/apache/hadoop/ipc/TestSaslRPC.java | 2 +-
.../java/org/apache/hadoop/ipc/TestServer.java | 2 +
.../hadoop/net/unix/TemporarySocketDirectory.java | 4 +-
.../apache/hadoop/net/unix/TestDomainSocket.java | 61 ++-
.../security/http/TestCrossOriginFilter.java | 10 +-
.../http/TestRestCsrfPreventionFilter.java | 12 +-
.../java/org/apache/hadoop/test/MockitoUtil.java | 9 +
.../src/test/resources/contract/rawlocal.xml | 5 +
hadoop-common-project/hadoop-kms/pom.xml | 2 +-
hadoop-common-project/hadoop-nfs/pom.xml | 2 +-
hadoop-common-project/hadoop-registry/pom.xml | 2 +-
hadoop-dist/pom.xml | 94 ++++
hadoop-dist/src/main/compose/hadoop/.env | 18 +
hadoop-dist/src/main/compose/hadoop/config | 50 ++
.../src/main/compose/hadoop/docker-compose.yaml | 46 ++
hadoop-dist/src/main/docker/Dockerfile | 26 +
hadoop-hdfs-project/hadoop-hdfs-client/pom.xml | 2 +-
.../java/org/apache/hadoop/hdfs/DFSUtilClient.java | 2 +-
.../apache/hadoop/hdfs/DistributedFileSystem.java | 16 +-
.../hadoop/hdfs/client/DfsPathCapabilities.java | 2 +
.../hadoop/hdfs/client/HdfsClientConfigKeys.java | 3 -
.../datatransfer/sasl/SaslParticipant.java | 6 +-
.../delegation/DelegationTokenIdentifier.java | 2 +-
.../hdfs/shortcircuit/ShortCircuitCache.java | 2 +-
hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml | 2 +-
.../hadoop/fs/http/client/HttpFSFileSystem.java | 2 +-
.../hadoop-hdfs-native-client/pom.xml | 2 +-
.../src/main/native/libhdfs/CMakeLists.txt | 12 +-
.../src/main/native/libhdfspp/tests/CMakeLists.txt | 13 +-
.../native/libhdfspp/tests/libhdfs_getjni_test.cc | 17 +-
hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml | 2 +-
hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml | 2 +-
.../federation/metrics/NamenodeBeanMetrics.java | 7 +
.../federation/resolver/order/RandomResolver.java | 2 +-
.../server/federation/router/RouterRpcServer.java | 4 +-
.../server/federation/metrics/TestRBFMetrics.java | 2 +-
.../server/federation/router/TestRouterAdmin.java | 21 +-
.../federation/router/TestRouterMountTable.java | 3 +-
.../server/federation/router/TestRouterRpc.java | 18 +
.../router/TestRouterRpcMultiDestination.java | 2 +-
hadoop-hdfs-project/hadoop-hdfs/pom.xml | 2 +-
.../apache/hadoop/hdfs/net/DFSNetworkTopology.java | 3 +-
.../datatransfer/sasl/SaslDataTransferServer.java | 20 +-
.../protocol/datatransfer/sasl/package-info.java | 11 +-
.../hadoop/hdfs/server/balancer/Balancer.java | 90 +++-
.../hdfs/server/balancer/BalancerParameters.java | 70 ++-
.../BlockPlacementPolicyDefault.java | 7 +-
.../BlockPlacementPolicyWithUpgradeDomain.java | 3 +
.../hdfs/server/datanode/BPServiceActor.java | 2 +-
.../hadoop/hdfs/server/datanode/DataStorage.java | 4 +-
.../hdfs/server/datanode/DirectoryScanner.java | 2 +-
.../hadoop/hdfs/server/datanode/LocalReplica.java | 6 +
.../fsdataset/impl/ReplicaCachingGetSpaceUsed.java | 2 +-
.../hadoop/hdfs/server/namenode/INodeFile.java | 5 +-
.../hdfs/server/namenode/NameNodeRpcServer.java | 4 +-
.../namenode/RedundantEditLogInputStream.java | 2 +
.../src/main/resources/hdfs-default.xml | 9 -
.../hadoop-hdfs/src/main/webapps/hdfs/explorer.js | 3 +-
.../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 2 +
.../hadoop/fs/TestEnhancedByteBufferAccess.java | 2 +-
.../org/apache/hadoop/hdfs/TestFileCreation.java | 5 +-
.../java/org/apache/hadoop/hdfs/TestSetTimes.java | 5 +-
.../hadoop/hdfs/net/TestDFSNetworkTopology.java | 19 +
.../sasl/TestCustomizedCallbackHandler.java | 39 +-
.../hadoop/hdfs/server/balancer/TestBalancer.java | 305 ++++++++++-
.../balancer/TestBalancerLongRunningTasks.java | 92 ++++
.../balancer/TestBalancerWithHANameNodes.java | 4 +-
.../blockmanagement/TestBlockManagerSafeMode.java | 1 +
.../hdfs/server/namenode/NameNodeAdapter.java | 103 ----
.../namenode/NameNodeAdapterMockitoUtil.java | 124 +++++
.../hdfs/server/namenode/TestCacheDirectives.java | 3 +-
.../namenode/TestCommitBlockSynchronization.java | 7 +-
.../hdfs/server/namenode/TestFSDirAttrOp.java | 6 +-
.../namenode/TestRedundantEditLogInputStream.java | 67 +++
.../server/namenode/TestSnapshotPathINodes.java | 4 +-
.../server/namenode/ha/TestFailureToReadEdits.java | 3 +-
.../server/namenode/ha/TestHAStateTransitions.java | 3 +-
.../hdfs/server/namenode/ha/TestObserverNode.java | 8 +-
.../server/namenode/ha/TestStandbyCheckpoints.java | 8 +-
.../snapshot/TestFileWithSnapshotFeature.java | 19 +-
.../namenode/snapshot/TestSnapshotDiffReport.java | 2 +-
.../hdfs/shortcircuit/TestShortCircuitCache.java | 2 +-
.../org/apache/hadoop/hdfs/web/TestWebHDFS.java | 4 +-
.../mapreduce/v2/app/rm/TestRMCommunicator.java | 2 +-
.../hadoop/mapred/.LineRecordReader.java.swp | Bin 0 -> 20480 bytes
.../java/org/apache/hadoop/mapred/Counters.java | 2 +-
.../org/apache/hadoop/mapred/LineRecordReader.java | 2 +
.../mapreduce/jobhistory/JobQueueChangeEvent.java | 12 +-
.../mapreduce/lib/input/LineRecordReader.java | 79 +--
.../v2/hs/webapp/TestHsWebServicesAcls.java | 8 +-
.../hadoop/mapreduce/MiniHadoopClusterManager.java | 2 +-
.../mapreduce/TestYarnClientProtocolProvider.java | 5 +-
.../hadoop/mapred/uploader/FrameworkUploader.java | 2 +-
.../hadoop-mapreduce-client/pom.xml | 6 +-
hadoop-maven-plugins/pom.xml | 2 +-
hadoop-project/pom.xml | 70 ++-
.../hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java | 2 +-
.../fs/aliyun/oss/AliyunOSSFileSystemStore.java | 2 +-
hadoop-tools/hadoop-archive-logs/pom.xml | 2 +-
hadoop-tools/hadoop-archives/pom.xml | 2 +-
hadoop-tools/hadoop-aws/pom.xml | 17 +-
.../hadoop/fs/s3a/AWSStatus500Exception.java | 21 +-
.../java/org/apache/hadoop/fs/s3a/Constants.java | 87 ++++
.../hadoop/fs/s3a/DefaultS3ClientFactory.java | 37 +-
.../java/org/apache/hadoop/fs/s3a/Listing.java | 13 +-
.../fs/s3a/ProgressableProgressListener.java | 16 +-
.../apache/hadoop/fs/s3a/S3ABlockOutputStream.java | 488 +++++++++++++-----
.../org/apache/hadoop/fs/s3a/S3ADataBlocks.java | 373 ++++++--------
.../org/apache/hadoop/fs/s3a/S3AFileSystem.java | 351 ++++++-------
.../org/apache/hadoop/fs/s3a/S3AInputStream.java | 78 ++-
.../apache/hadoop/fs/s3a/S3AInstrumentation.java | 11 +
.../org/apache/hadoop/fs/s3a/S3ARetryPolicy.java | 18 +-
.../java/org/apache/hadoop/fs/s3a/S3AStore.java | 169 ++++++
.../java/org/apache/hadoop/fs/s3a/S3AUtils.java | 28 +-
.../org/apache/hadoop/fs/s3a/S3ClientFactory.java | 76 +++
.../java/org/apache/hadoop/fs/s3a/Statistic.java | 36 ++
.../apache/hadoop/fs/s3a/WriteOperationHelper.java | 69 ++-
.../org/apache/hadoop/fs/s3a/WriteOperations.java | 13 +-
.../apache/hadoop/fs/s3a/api/RequestFactory.java | 2 +
.../s3a/audit/impl/AbstractOperationAuditor.java | 4 +-
.../fs/s3a/audit/impl/ActiveAuditManagerS3A.java | 12 +-
.../hadoop/fs/s3a/audit/impl/LoggingAuditor.java | 36 +-
.../fs/s3a/commit/impl/CommitOperations.java | 98 +++-
.../fs/s3a/commit/magic/S3MagicCommitTracker.java | 14 +-
.../apache/hadoop/fs/s3a/impl/AWSClientConfig.java | 43 +-
.../org/apache/hadoop/fs/s3a/impl/AWSHeaders.java | 5 +
.../hadoop/fs/s3a/impl/AwsSdkWorkarounds.java | 59 +++
.../fs/s3a/impl/BaseS3AFileSystemOperations.java | 132 +++++
.../apache/hadoop/fs/s3a/impl/CSEMaterials.java | 132 +++++
.../fs/s3a/impl/CSES3AFileSystemOperations.java | 135 +++++
.../org/apache/hadoop/fs/s3a/impl/CSEUtils.java | 198 +++++++
.../CSEV1CompatibleS3AFileSystemOperations.java | 101 ++++
.../apache/hadoop/fs/s3a/impl/ClientManager.java | 33 ++
.../hadoop/fs/s3a/impl/ClientManagerImpl.java | 69 +++
.../hadoop/fs/s3a/impl/CopyFromLocalOperation.java | 4 +-
.../fs/s3a/impl/EncryptionS3ClientFactory.java | 324 ++++++++++++
.../hadoop/fs/s3a/impl/ErrorTranslation.java | 55 ++
.../hadoop/fs/s3a/impl/HeaderProcessing.java | 13 -
.../hadoop/fs/s3a/impl/InternalConstants.java | 7 +
.../fs/s3a/impl/ListingOperationCallbacks.java | 11 +
.../hadoop/fs/s3a/impl/ProgressListenerEvent.java | 64 ++-
.../hadoop/fs/s3a/impl/RequestFactoryImpl.java | 42 ++
.../fs/s3a/impl/S3AFileSystemOperations.java | 102 ++++
.../hadoop/fs/s3a/impl/S3AMultipartUploader.java | 3 +-
.../apache/hadoop/fs/s3a/impl/S3AStoreBuilder.java | 16 +-
.../apache/hadoop/fs/s3a/impl/S3AStoreImpl.java | 384 +++++++++++++-
.../hadoop/fs/s3a/impl/UploadContentProviders.java | 569 +++++++++++++++++++++
.../fs/s3a/impl/logging/Log4JController.java | 52 ++
.../hadoop/fs/s3a/impl/logging/LogControl.java | 92 ++++
.../fs/s3a/impl/logging/LogControllerFactory.java | 98 ++++
.../package-info.java} | 15 +-
.../statistics/BlockOutputStreamStatistics.java | 3 +-
.../s3a/statistics/S3AInputStreamStatistics.java | 4 +
.../statistics/impl/StatisticsFromAwsSdkImpl.java | 43 ++
.../site/markdown/tools/hadoop-aws/connecting.md | 171 ++++++-
.../site/markdown/tools/hadoop-aws/encryption.md | 59 ++-
.../tools/hadoop-aws/third_party_stores.md | 19 +
.../tools/hadoop-aws/troubleshooting_s3a.md | 241 ++++++---
.../fs/contract/s3a/ITestS3AContractCreate.java | 19 +-
.../org/apache/hadoop/fs/s3a/CustomKeyring.java | 72 +++
.../hadoop/fs/s3a/ITestS3ABlockOutputArray.java | 7 +-
.../fs/s3a/ITestS3ABlockOutputByteBuffer.java | 2 +-
.../hadoop/fs/s3a/ITestS3ABlockOutputDisk.java | 2 +-
.../fs/s3a/ITestS3AClientSideEncryption.java | 167 +++++-
.../fs/s3a/ITestS3AClientSideEncryptionCustom.java | 89 ++++
.../hadoop/fs/s3a/ITestS3AConfiguration.java | 15 +-
.../hadoop/fs/s3a/ITestS3ACopyFromLocalFile.java | 12 +
.../ITestS3AEncryptionSSEKMSUserDefinedKey.java | 4 +-
.../hadoop/fs/s3a/ITestS3AEndpointRegion.java | 58 ++-
.../hadoop/fs/s3a/ITestS3AInputStreamLeakage.java | 161 ++++++
.../hadoop/fs/s3a/ITestS3AMiscOperations.java | 15 +-
.../apache/hadoop/fs/s3a/MockS3AFileSystem.java | 12 +-
.../apache/hadoop/fs/s3a/MultipartTestUtils.java | 2 +-
.../org/apache/hadoop/fs/s3a/S3ATestUtils.java | 30 ++
.../org/apache/hadoop/fs/s3a/TestDataBlocks.java | 233 +++++++--
.../java/org/apache/hadoop/fs/s3a/TestInvoker.java | 208 +++++++-
.../hadoop/fs/s3a/TestS3ABlockOutputStream.java | 19 +-
.../hadoop/fs/s3a/audit/AuditTestSupport.java | 12 +-
.../fs/s3a/audit/TestHttpReferrerAuditHeader.java | 33 +-
.../hadoop/fs/s3a/audit/TestLoggingAuditor.java | 41 ++
.../fs/s3a/audit/impl/ReferrerExtractor.java | 52 ++
.../hadoop/fs/s3a/commit/ITestUploadRecovery.java | 259 ++++++++++
.../commit/integration/ITestS3ACommitterMRJob.java | 4 +
.../s3a/commit/staging/TestStagingCommitter.java | 9 +
.../fileContext/ITestS3AFileContextStatistics.java | 30 +-
.../hadoop/fs/s3a/impl/ITestAwsSdkWorkarounds.java | 160 ++++++
.../fs/s3a/impl/ITestConnectionTimeouts.java | 132 ++++-
.../hadoop/fs/s3a/impl/TestClientManager.java | 1 +
.../hadoop/fs/s3a/impl/TestErrorTranslation.java | 33 +-
.../hadoop/fs/s3a/impl/TestRequestFactory.java | 78 ++-
.../s3a/impl/logging/TestLogControllerFactory.java | 214 ++++++++
.../fs/s3a/performance/AbstractS3ACostTest.java | 2 +-
.../fs/s3a/scale/AbstractSTestS3AHugeFiles.java | 85 +--
.../fs/s3a/scale/CountingProgressListener.java | 192 +++++++
.../ITestS3ABlockOutputStreamInterruption.java | 493 ++++++++++++++++++
.../hadoop/fs/s3a/scale/ITestS3AConcurrentOps.java | 94 ++--
.../fs/s3a/scale/ITestS3ADirectoryPerformance.java | 13 +-
.../fs/s3a/scale/ITestS3AHugeFilesNoMultipart.java | 10 +-
.../fs/s3a/statistics/TestErrorCodeMapping.java | 83 +++
.../s3a/test/MinimalListingOperationCallbacks.java | 8 +
.../test/MinimalWriteOperationHelperCallbacks.java | 45 +-
.../hadoop/fs/s3a/test/SdkFaultInjector.java | 262 ++++++++++
hadoop-tools/hadoop-azure/pom.xml | 29 +-
.../hadoop/fs/azurebfs/AbfsConfiguration.java | 10 +-
.../hadoop/fs/azurebfs/AzureBlobFileSystem.java | 4 +
.../fs/azurebfs/AzureBlobFileSystemStore.java | 2 +-
.../constants/FileSystemConfigurations.java | 2 +-
.../hadoop/fs/azurebfs/services/AbfsClient.java | 48 +-
.../services/AbfsManagedApacheHttpConnection.java | 16 +-
.../hadoop-azure/src/site/markdown/abfs.md | 4 +-
.../fs/azurebfs/ITestAbfsCustomEncryption.java | 4 +-
.../fs/azurebfs/ITestAbfsReadFooterMetrics.java | 19 +
.../azurebfs/ITestAbfsRestOperationException.java | 53 +-
.../ITestAzureBlobFileSystemAuthorization.java | 40 +-
.../ITestAzureBlobFileSystemChooseSAS.java | 11 +-
.../fs/azurebfs/ITestAzureBlobFileSystemE2E.java | 4 +
.../ITestAzureBlobFileSystemInitAndCreate.java | 3 +-
.../fs/azurebfs/ITestGetNameSpaceEnabled.java | 72 ++-
.../contract/ITestAbfsFileSystemContractSeek.java | 2 +
.../services/ITestApacheClientConnectionPool.java | 68 +++
.../services/ITestExponentialRetryPolicy.java | 14 +-
.../fs/azurebfs/services/TestAbfsClient.java | 141 +++++
.../azurebfs/services/TestAbfsRestOperation.java | 12 +-
.../hadoop/fs/azurebfs/utils/AbfsTestUtils.java | 17 +-
hadoop-tools/hadoop-compat-bench/pom.xml | 2 +-
hadoop-tools/hadoop-datajoin/pom.xml | 2 +-
hadoop-tools/hadoop-distcp/pom.xml | 4 +-
.../main/java/org/apache/hadoop/tools/DistCp.java | 28 +-
.../org/apache/hadoop/tools/mapred/CopyMapper.java | 9 +-
.../mapred/RetriableDirectoryCreateCommand.java | 33 +-
.../tools/mapred/RetriableFileCopyCommand.java | 43 +-
.../hadoop/tools/TestDistCpWithRawXAttrs.java | 130 +++++
.../org/apache/hadoop/tools/TestExternalCall.java | 5 +-
.../hadoop-dynamometer-blockgen/pom.xml | 2 +-
.../hadoop-dynamometer-infra/pom.xml | 2 +-
.../hadoop-dynamometer-workload/pom.xml | 2 +-
hadoop-tools/hadoop-extras/pom.xml | 2 +-
hadoop-tools/hadoop-federation-balance/pom.xml | 4 +-
hadoop-tools/hadoop-fs2img/pom.xml | 2 +-
hadoop-tools/hadoop-gridmix/pom.xml | 2 +-
.../hadoop/mapred/gridmix/TestGridMixClasses.java | 2 +-
hadoop-tools/hadoop-kafka/pom.xml | 2 +-
.../org/apache/hadoop/tools/rumen/JobBuilder.java | 16 +-
.../apache/hadoop/tools/rumen/JobHistoryUtils.java | 6 +-
.../org/apache/hadoop/tools/rumen/LoggedTask.java | 8 +-
.../hadoop/tools/rumen/LoggedTaskAttempt.java | 8 +-
hadoop-tools/hadoop-sls/pom.xml | 2 +-
.../java/org/apache/hadoop/yarn/sls/SLSRunner.java | 2 +-
hadoop-tools/hadoop-streaming/pom.xml | 2 +-
.../hadoop-yarn/dev-support/findbugs-exclude.xml | 8 -
.../hadoop-yarn/hadoop-yarn-api/pom.xml | 2 +-
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 18 +
.../pom.xml | 14 +-
.../application/TestAppCatalogSolrClient.java | 11 +-
.../pom.xml | 2 +-
.../yarn/applications/distributedshell/Client.java | 2 +-
.../unmanagedamlauncher/UnmanagedAMLauncher.java | 2 +-
.../dev-support/findbugs-exclude.xml | 5 +
.../hadoop-yarn-services-api/pom.xml | 2 +-
.../hadoop-yarn-services-core/pom.xml | 2 +-
.../containerlaunch/TestAbstractLauncher.java | 2 +-
.../hadoop-yarn/hadoop-yarn-client/pom.xml | 2 +-
.../apache/hadoop/yarn/client/cli/RouterCLI.java | 4 +-
.../hadoop-yarn/hadoop-yarn-common/pom.xml | 2 +-
.../src/main/resources/yarn-default.xml | 24 +
.../pom.xml | 6 +-
.../yarn/server/timeline/LeveldbTimelineStore.java | 2 +-
.../timeline/RollingLevelDBTimelineStore.java | 2 +-
.../timeline/security/TimelineACLsManager.java | 2 +-
.../hadoop-yarn-server-common/pom.xml | 2 +-
.../LocalityMulticastAMRMProxyPolicy.java | 2 +-
.../policies/dao/WeightedPolicyInfo.java | 2 +-
.../utils/FederationPolicyStoreInputValidator.java | 2 +-
.../federation/utils/FederationRegistryClient.java | 2 +-
.../utils/FederationStateStoreFacade.java | 4 +-
.../policygenerator/LoadBasedGlobalPolicy.java | 2 +-
.../hadoop-yarn-server-nodemanager/pom.xml | 2 +-
.../yarn/server/nodemanager/ContainerExecutor.java | 3 +-
.../server/nodemanager/LinuxContainerExecutor.java | 25 +-
.../resources/CGroupsCpuResourceHandlerImpl.java | 8 +-
.../localizer/ResourceLocalizationService.java | 14 +-
.../TestLinuxContainerExecutorWithMocks.java | 57 ++-
.../TestCGroupsCpuResourceHandlerImpl.java | 15 +-
.../linux/resources/TestCGroupsHandlerImpl.java | 2 +-
.../linux/resources/TestCGroupsV2HandlerImpl.java | 2 +-
.../resources/gpu/TestGpuResourceAllocator.java | 2 +-
.../linux/runtime/TestDockerContainerRuntime.java | 10 +-
.../resourceplugin/com/nec/TestNECVEPlugin.java | 2 +-
.../hadoop-yarn-server-resourcemanager/pom.xml | 7 +-
.../server/resourcemanager/NodesListManager.java | 126 ++++-
.../server/resourcemanager/ResourceManager.java | 14 +-
.../resourcemanager/ResourceTrackerService.java | 2 +-
.../ProportionalCapacityPreemptionPolicy.java | 2 +-
.../ReservedContainerCandidatesSelector.java | 3 +
.../server/resourcemanager/rmapp/RMAppImpl.java | 2 +-
.../server/resourcemanager/rmnode/RMNodeImpl.java | 15 +-
.../scheduler/AbstractYarnScheduler.java | 200 +++++++-
.../scheduler/SchedulerApplicationAttempt.java | 3 +-
.../resourcemanager/scheduler/YarnScheduler.java | 5 +-
.../scheduler/activities/ActivitiesManager.java | 2 +-
.../scheduler/activities/AppAllocation.java | 2 +-
.../scheduler/capacity/CapacityScheduler.java | 46 +-
.../CapacitySchedulerQueueCapacityHandler.java | 2 +-
.../allocator/RegularContainerAllocator.java | 5 +-
.../scheduler/common/fica/FiCaSchedulerApp.java | 8 +-
.../scheduler/fair/FairScheduler.java | 9 +-
.../scheduler/placement/AppPlacementAllocator.java | 2 +-
.../placement/MultiNodeSortingManager.java | 2 +-
.../TestResourceTrackerService.java | 63 +++
.../scheduler/TestAbstractYarnScheduler.java | 355 +++++++++++++
.../scheduler/capacity/TestCapacityScheduler.java | 38 +-
.../TestCapacitySchedulerAutoQueueCreation.java | 55 +-
.../scheduler/capacity/TestUtils.java | 14 +
.../TestFSConfigToCSConfigArgumentHandler.java | 2 +-
.../converter/TestFSConfigToCSConfigConverter.java | 2 +-
.../converter/TestQueuePlacementConverter.java | 2 +-
.../hadoop-yarn-server-router/pom.xml | 2 +-
.../yarn/server/router/RouterServerUtil.java | 2 +-
.../rmadmin/FederationRMAdminInterceptor.java | 4 +-
.../yarn/server/router/webapp/AppsBlock.java | 2 +-
.../router/webapp/FederationInterceptorREST.java | 2 +-
.../yarn/server/router/webapp/NodesBlock.java | 2 +-
.../yarn/server/router/webapp/RouterBlock.java | 2 +-
.../router/clientrm/TestRouterYarnClientUtils.java | 2 +-
.../clientrm/TestSequentialRouterPolicy.java | 2 +-
.../TestableFederationRMAdminInterceptor.java | 2 +-
.../server/router/secure/TestSecureLogins.java | 2 +-
.../subcluster/TestFederationSubCluster.java | 2 +-
.../hadoop-yarn-server-sharedcachemanager/pom.xml | 2 +-
.../hadoop-yarn-server-tests/pom.xml | 2 +-
.../pom.xml | 2 +-
.../pom.xml | 32 +-
.../TestDocumentStoreCollectionCreator.java | 11 +-
.../TestDocumentStoreTimelineReaderImpl.java | 19 +-
.../TestDocumentStoreTimelineWriterImpl.java | 22 +-
.../cosmosdb/TestCosmosDBDocumentStoreReader.java | 14 +-
.../cosmosdb/TestCosmosDBDocumentStoreWriter.java | 18 +-
.../pom.xml | 4 +-
.../pom.xml | 2 +-
.../hadoop-yarn-server-timelineservice/pom.xml | 2 +-
.../hadoop-yarn-server-web-proxy/pom.xml | 2 +-
.../hadoop-yarn/hadoop-yarn-ui/pom.xml | 4 +-
pom.xml | 16 +-
395 files changed, 12537 insertions(+), 2163 deletions(-)
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/WithErasureCoding.java
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/LeakReporter.java
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/store/ByteBufferInputStream.java
rename
{hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl
=>
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security}/CustomizedCallbackHandler.java
(50%)
delete mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslConstants.java
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslMechanismFactory.java
copy
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ProgressListenerEvent.java
=>
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/package-info.java
(77%)
copy
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/test/MinimalWriteOperationHelperCallbacks.java
=>
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/RunnableRaisingIOE.java
(57%)
create mode 100644
hadoop-common-project/hadoop-common/src/site/markdown/HadoopDocker.md
copy
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ABlockOutputByteBuffer.java
=>
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/constants/ConfigConstants.java
(64%)
copy
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ProgressListenerEvent.java
=>
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/constants/package-info.java
(76%)
create mode 100644
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/impl/TestLeakReporter.java
create mode 100644 hadoop-dist/src/main/compose/hadoop/.env
create mode 100644 hadoop-dist/src/main/compose/hadoop/config
create mode 100644 hadoop-dist/src/main/compose/hadoop/docker-compose.yaml
create mode 100644 hadoop-dist/src/main/docker/Dockerfile
copy
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ProgressListenerEvent.java
=>
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/package-info.java
(77%)
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapterMockitoUtil.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestRedundantEditLogInputStream.java
create mode 100644
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/.LineRecordReader.java.swp
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AwsSdkWorkarounds.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/BaseS3AFileSystemOperations.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CSEMaterials.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CSES3AFileSystemOperations.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CSEUtils.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/CSEV1CompatibleS3AFileSystemOperations.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/EncryptionS3ClientFactory.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/S3AFileSystemOperations.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/UploadContentProviders.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/logging/Log4JController.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/logging/LogControl.java
create mode 100644
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/logging/LogControllerFactory.java
copy
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/{ProgressListenerEvent.java
=> logging/package-info.java} (77%)
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/CustomKeyring.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AClientSideEncryptionCustom.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AInputStreamLeakage.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/audit/impl/ReferrerExtractor.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestUploadRecovery.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestAwsSdkWorkarounds.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/logging/TestLogControllerFactory.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/CountingProgressListener.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3ABlockOutputStreamInterruption.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/statistics/TestErrorCodeMapping.java
create mode 100644
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/test/SdkFaultInjector.java
create mode 100644
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsClient.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
