This is an automated email from the ASF dual-hosted git repository. stevel pushed a change to branch feature-HADOOP-18073-s3a-sdk-upgrade in repository https://gitbox.apache.org/repos/asf/hadoop.git
omit 75220b77387 fixes issues after rebase omit 1f137d3283c HADOOP-18565. Complete outstanding items for the AWS SDK V2 upgrade. (#5421) omit fb6d12a9a59 adds in unit check for bytes omit ce02671de4a update GET range referrer header logic to V2 omit 644a32f9d27 updates new providers in TestS3AAWSCredentialsProvider to V2 omit 0338fd89f6d Add unit tests for SelectEventStreamPublisher omit 38dff867b9d Add unit tests for BlockingEnumeration omit 0c2fd282526 Move TransferManager initialization to ClientFactory omit 1ab7e6827d0 Reinstate old constants omit aa1cf19ffaa Move MultiObjectDeleteException to impl omit 68510c8ab12 HADOOP-18073. Address review comments. (#31) omit f2bf08cae13 HADOOP-18073. Upgrade AWS SDK to v2 in S3A [work in progress] add 482897a0f65 Fix typos in HDFS documents. (#5665) add 8e173851410 HDFS-17009. RBF: state store putAll should also return failed records (#5664) add 5ddaf2e133d YARN-11493. [Federation] ConfiguredRMFailoverProxyProvider Supports Randomly Select an Router. (#5651) add 03163f9de26 HDFS-17011. Fix the metric of "HttpPort" at DataNodeInfo (#5657). Contributed by Zhaohui Wang. add 350dcaf6167 HDFS-17010. Add a subtree test to TestSnapshotDiffReport. (#5656) add 78cc528739c HDFS-17012.Remove unused DFSConfigKeys#DFS_DATANODE_PMEM_CACHE_DIRS_DEFAULT. (#5659). Contributed by JiangHua Zhu. add ff8eac517a3 YARN-11463. Node Labels root directory creation doesn't have a retry logic - 2nd addendum (#5670) add bba663038d9 YARN-8898. Fix FederationInterceptor#allocate to set application priority in allocateResponse. (#5645) add f6770dee476 HDFS-16979. RBF: Add proxyuser port in hdfsauditlog (#5552). Contributed by liuguanghua. add 4627242c449 HADOOP-18652. Path.suffix raises NullPointerException (#5653). Contributed by Patrick Grandjean. add 441fb232939 HDFS-17018. Improve dfsclient log format. (#5668). Contributed by Xianming Lei. add 339bc7b3a6e HDFS-16653. Improve error messages in ShortCircuitCache. (#5568). Contributed by ECFuzz. add 0914b3e7920 HDFS-16697. Add logs if resources are not available in NameNodeResourcePolicy. (#5569). Contributed by ECFuzz. add 5272ed86708 HADOOP-17518. Update the regex to A-Z (#5669). Contributed by Nishtha Shah. add 408dbf318e7 HDFS-17014. HttpFS Add Support getStatus API (#5660). Contributed by Hualong Zhang. add 9a524ede872 HDFS-17022. Fix the exception message to print the Identifier pattern (#5678). Contributed by Nishtha Shah. add 0110e24ed86 YARN-11496. Improve TimelineService log format. (#5677). Contributed by Xianming Lei. add af933f3a4f9 HDFS-17017. Fix the issue of arguments number limit in report command in DFSAdmin (#5667). Contributed by Haiyang Hu. add 5b22dc6aced HDFS-16909. Improve ReplicaMap#mergeAll method. (#5353). Contributed by ZhangHB. add afe850ca2c2 HADOOP-18746. Install Python 3 for Windows 10 docker image (#5679) add 3b65b5d68f8 HDFS-17020. RBF: mount table addAll should print failed records in std error (#5674) add a98d15804a9 MAPREDUCE-7419. Upgrade Junit 4 to 5 in hadoop-mapreduce-client-common (#5028). Contributed by Ashutosh Gupta. add aeb3f6f1a84 YARN-11490. Reverting YARN-11211 and eliminating the use of DefaultMetricsSystem during configuration validation (#5644) add e9740cb17ae HDFS-16908. Fix javadoc of field IncrementalBlockReportManager#readyToSend. (#5351). Contributed by farmmamba. add e6b54f7f683 Revert "HADOOP-18706. Improve S3ABlockOutputStream recovery (#5563)" add b977065cc4c YARN-11478. [Federation] SQLFederationStateStore Support Store ApplicationSubmitData. (#5663) add 97afb33c73b YARN-11276. Add LRU cache for RMWebServices.getApps. (#4793) add 86c250a54a5 YARN-7720. Race condition between second app attempt and UAM timeout when first attempt node is down. (#5672) add 124313d2159 HDFS-17026. RBF: NamenodeHeartbeatService should update JMX report with configurable frequency. (#5691). Contributed by hchaverri. add 9acf462d268 HDFS-17000. Fix faulty loop condition in TestDFSStripedOutputStreamUpdatePipeline (#5699). Contributed by Marcono1234. add d92a5815f45 HDFS-17027. RBF: Adds auto-msync support for clients connecting to routers. (#5693) add f8b7ddf69c5 HDFS-16996. Fix flaky testFsCloseAfterClusterShutdown in TestFileCreation (#5697). Contributed by Nishtha Shah. add 9f1e23cc675 HDFS-17031. Reduce some repeated codes in RouterRpcServer. (#5701). Contributed by Chengwei Wang. add 160b9fc3c92 HADOOP-18755. openFile builder new optLong() methods break hbase-filesystem (#5704) add 03a499821c9 HADOOP-18207. Introduce hadoop-logging module (#5503) add e0a339223a8 HADOOP-18709. Add curator based ZooKeeper communication support over SSL/TLS into the common library. Contributed by Ferenc Erdelyi add ee94f6cdcbc YARN-11277. Trigger log-dir deletion by size for NonAggregatingLogHandler. (#4797) add 1d0c9ab4330 Revert "HADOOP-18207. Introduce hadoop-logging module (#5503)" add 2243cfd225c HDFS-17028. RBF: Optimize debug logs of class ConnectionPool and other related class. (#5694). Contributed by farmmamba. add e69a077af84 YARN-11497 : Support removal of only selective node states in untracked removal flow (#5681) add 241398de3b2 YARN-11492. Improve createJerseyClient#setConnectTimeout Code. (#5636). Contributed by Shilun Fan. add 5d6ca13c5cf HDFS-16983. Fix concat operation doesn't honor dfs.permissions.enabled (#5561). Contributed by caozhiqiang. add d9980ab40ff HDFS-17029. Support getECPolices API in WebHDFS (#5698). Contributed by Hualong Zhang. add fd3c3ae0686 YARN-11500. Fix typos in hadoop-yarn-server-common#federation. (#5702) add e6937d70760 YARN-11425. [Hotfix] YARN-11425. Modify Expiration Time Unit error. (#5712) add 9de13f879af YARN-11502. Refactor AMRMProxy#FederationInterceptor#registerApplicationMaster. (#5705) add 1dbaba8e70b HADOOP-18740. S3A prefetch cache blocks should be accessed by RW locks (#5675) add 0c209961f82 HDFS-17019. Optimize the logic for reconfigure slow peer enable for Namenode" (#5671) add ddae78b0ec7 HDFS-17035. FsVolumeImpl#getActualNonDfsUsed may return negative value. (#5708). Contributed by farmmamba. add 0e6bd09ae3c HDFS-17003. Erasure Coding: invalidate wrong block after reporting bad blocks from datanode (#5643). Contributed by hfutatzhanghb. add 7bb09f10109 HADOOP-18752. Change fs.s3a.directory.marker.retention to "keep" (#5689) add 9c989515ba4 HDFS-17037. Consider nonDfsUsed when running balancer. (#5715). Contributed by Shuyan Zhang. add 7a45ef41647 MAPREDUCE-7435. Manifest Committer OOM on abfs (#5519) add 35158db7115 HDFS-17023. RBF: Record proxy time when call invokeConcurrent method. (#5683). Contributed by farmmamba. add fb16e00da0e HADOOP-18718. Fix several maven build warnings (#5592). Contributed by Dongjoon Hyun. add a409d52ef8c YARN-6648. BackPort [GPG] Add SubClusterCleaner in Global Policy Generator. (#5676) add 2794fe264b5 YARN-8898. [Addendum] Improve NodeManager#TestFederationInterceptor Setup Code (#5732) add a4c3d48c31b YARN-11504. [Federation] YARN Federation Supports Non-HA mode. (#5722) add e1bb4acd368 HADOOP-18761. Remove mysql-connector-java (#5731) add 3fbadc5d505 HDFS-16946. Fix getTopTokenRealOwners to return String (#5696). Contributed by Nishtha Shah. add f0c4286e3ef HDFS-17041. RBF: Fix putAll impl for mysql and file based state stores (#5723) add a75e3788685 HADOOP-18756. S3A prefetch - CachingBlockManager to use AtomicBoolean for closed flag (#5718) add f4b70b353df YARN-11429. Improve the updateTestDataAutomatically in TestRMWebServices. (#5736) add f4accec379a YARN-11504. [Addendum] [Federation] YARN Federation Supports Non-HA mode. (#5746) add ab47a658bd9 HDFS-17030. Limit wait time for getHAServiceState in ObserverReaderProxy (#5700) add 1a918a6b960 HDFS-17047. BlockManager#addStoredBlock should log storage id when AddBlockResult is REPLACED (#5742). Contributed by farmmamba. add 5dd37b28149 HDFS-17045. File renamed from a snapshottable dir to a non-snapshottable dir cannot be deleted. (#5738) add 2fe3b2a73f6 HADOOP-18763. Upgrade aws-java-sdk to 1.12.367 (#5741) add 02027c8dccc HDFS-17051. Fix wrong time unit in TestFileAppend4#recoverFile (#5749). Contributed by Zhaohui Wang. add 427366b73bf HDFS-17042 Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode (#5730) add 9a7d1b49e2f HDFS-17043. HttpFS implementation for getAllErasureCodingPolicies (#5734). Contributed by Hualong Zhang. add ba08f26a152 HADOOP-18774. Add .vscode to gitignore. (#5756). Contributed by Xiaoqiao He. add 7c54a783435 YARN-11506. The formatted yarn queue list is displayed on CLI (#5716). Contributed by Lu Yuan. add e14c52c25f2 HDFS-17053. Optimize method BlockInfoStriped#findSlot to reduce time complexity. (#5757). Contributed by farmmamba. add 35d77a6359c YARN-11505. [Federation] Add Steps To Set up a Test Cluster. (#5751) add 3d2cddd235c YARN-11516. Improve FederationStateStoreFacade#existsApplicationHomeSubCluster/existsReservationHomeSubCluster Log Level. (#5755) add 5c02f21f2e8 YARN-11510. [Federation] Fix NodeManager#TestFederationInterceptor Flaky Unit Test. (#5733) add 3f4de919bd8 YARN-11511. Improve TestRMWebServices test config and data. (#5745) add cebcb44d377 HADOOP-18713. Update solr from 8.8.2 to 8.11.2 (#5459). Contributed by Xuesen Liang. add c82ea52e4c5 YARN-11513: Applications submitted to ambiguous queue fail during recovery if "Specified" Placement Rule is used (#5748) add eb88b9ff21e YARN-11498. Exclude jettison from jersey-json artifact as on older version is being pulled (#5623) add 1c15987ee38 MAPREDUCE-7441. Fix race condition in closing FadvisedFileRegion. Contributed by Benjamin Teke add e239d40ab1b Post release update add 161f80810fe HDFS-17057. Add DataNode maintenance states to Federation UI (#5775) add a4cf4c3778f YARN-11517. Improve Federation#RouterCLI DeregisterSubCluster Code. (#5766) add a85272c33d6 HADOOP-18751. Fix incorrect output path in javadoc build phase (#5688) add 03902f5ef00 HDFS-17055 Export HAState as a metric from Namenode for monitoring (#5764) add 21d9c4727c1 HADOOP-18773. Upgrade maven-shade-plugin to 3.4.1 (#5750) add 56ef05a9ca5 HADOOP-18782. Upgrade to snappy-java 1.1.10.1 due to CVEs (#5773) add 0d057e27c3c MAPREDUCE-7432. Make manifest committer default on abfs and gcs stores (#5378) add 5db7107b773 HADOOP-18764. fs.azure.buffer.dir to be under Yarn container path on yarn applications (#5737) add 8a529901504 YARN-11519. [Federation] Add RouterAuditLog to log4j.properties. (#5785) add 4bd873b816d HDFS-17044. Set size of non-exist block to NO_ACK when process FBR or IBR to avoid useless report from DataNode. (#5735). Contributed by Haiyang Hu. add 8fb611c1f70 HDFS-17062. HA NameNode Web UI should show last HA Transition time (#5791). Contributed by Zhaohui Wang. add 59a7836d13b HADOOP-18789. Remove ozone from hadoop dev support. (#5800). Contributed by Xiaoqiao He. add 750c0fc6318 HDFS-17052. Improve BlockPlacementPolicyRackFaultTolerant to avoid choose nodes failed when no enough Rack. (#5759). Contributed by Hualong Zhang and Shuyan Zhang. add 5a35fb5a726 HADOOP-18783. Upgrade to netty 4.1.94 due to CVE (#5774). Contributed by PJ Fanning. add 9659d0720dc HDFS-17065. Fix typos in hadoop-hdfs-project (#5796) add e8590adb7b4 HADOOP-18779. Improve hadoop-function.sh#status script. (#5762) add 6042d599042 YARN-11090. [GPG] Support Secure Mode. (#5782) add 178d046ce61 YARN-11526. [Client] Add a unit test to FormattingCLIUtils (#5801) add a36d8adfd18 YARN-11000. Replace queue resource calculation logic in updateClusterResource (#5783) add e3683a954ff HADOOP-18793. S3A StagingCommitter does not clean up staging-uploads directory (#5818) add 3f983e49e05 HDFS-17073. Enhance the warning message output for BlockGroupNonStripedChecksumComputer#compute" (#5817). Contributed by Haiyang Hu. add b673ebfec25 HDFS-17070. Remove unused import in DataNodeMetricHelper.java. (#5812). Contributed by farmmamba. add a84284e9742 HDFS-17064. Document the usage of the new Balancer "sortTopNodes" and "hotBlockTimeInterval" parameter (#5794). Contributed by Haiyang Hu. add 37b2d36edd2 HDFS-17033. Update fsck to display stale state info of blocks accurately. (#5815). Contributed by Wang Yuanben. add a822a3c70ba YARN-11528. Lock triple-beam to the version compatible with node.js 12 to avoid compilation error. (#5827). Contributed by Masatake Iwasaki add 4e699f03835 HDFS-17076. Remove the unused method isSlownodeByNameserviceId in DataNode (#5824). Contributed by Haiyang Hu. add 6843f8e4e0f HADOOP-18794. ipc.server.handler.queue.size missing from core-default.xml (#5819). Contributed by WangYuanben. add fac7d26c5d7 HADOOP-18781. ABFS backReference passed down to streams to avoid GC closing the FS. (#5780) add c13d92996df YARN-11464: TestFSQueueConverter#testAutoCreateV2FlagsInWeightMode has a missing dot before auto-queue-creation-v2.enabled for method call assertNoValueForQueues (#5792) add 33b1677e9e4 YARN-11524. Improve the Policy Description in Federation.md. (#5797) add 8b88e9f8f4f YARN-11509. The FederationInterceptor#launchUAM Added retry logic. (#5727) add 680af873777 YARN-11515. [Federation] Improve DefaultRequestInterceptor#init Code. (#5752) add 325f7e5fdb4 YARN-11529 Add metrics for ContainerMonitorImpl. (#5828) add 4c8d048f256 HDFS-17082. Add documentation for provisionSnapshotTrash command to HDFSCommands.md and HdfsSnapshots.md" (#5834). Contributed by Haiyang Hu. add dfb351c3a8c HDFS-17083. Support getErasureCodeCodecs API in WebHDFS (#5836). Contributed by Hualong Zhang. add c1d19c39517 HDFS-17081. EC: Add logic for striped blocks in isSufficientlyReplicated (#5833). Contributed by Haiyang Hu. add e7d74f3d594 HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache (#5754) add c457c445b77 HDFS-17068. Datanode should record last directory scan time. (#5809). Contributed by farmmamba. add 8dd9c874e14 HDFS-17086. Fix the parameter settings in TestDiskspaceQuotaUpdate#updateCountForQuota (#5842). Contributed by Haiyang Hu. add b95595158fe HADOOP-18801. Delete path directly when it can not be parsed in trash. (#5744). Contributed by farmmamba. add c44823dadb7 HDFS-17075. Reconfig disk balancer parameters for datanode (#5823). Contributed by Haiyang Hu. add 38ac2f73499 HADOOP-18809. S3A prefetch read/write file operations should guard channel close (#5853) add b3130056f5a HADOOP-18808. LogExactlyOnce to add a debug() method (#5850) add 0fe1863094d HDFS-17091. Blocks on DECOMMISSIONING DNs should be sorted properly in LocatedBlocks. (#5849) add 8df1616f534 HDFS-17088. Improve the debug verifyEC and dfsrouteradmin commands in HDFSCommands.md (#5846) add b6b259066fd HADOOP-18757. S3A Committer only finalizes the commits in a single thread (#5706) add 84dd6247686 YARN-11531. [Federation] Code cleanup for NodeManager#amrmproxy. (#5841) add 23ecc32d3ad YARN-11525. [Federation] Router CLI Supports Save the SubClusterPolicyConfiguration Of Queues. (#5816) add 7ba2bd63058 HDFS-17094. EC: Fix bug in block recovery when there are stale datanodes. (#5854) add 193ff1c24e5 YARN-11533. CapacityScheduler CapacityConfigType changed in legacy queue allocation mode (#5852) add 82c8070e93a YARN-9877 - Intermittent TIME_OUT of LogAggregationReport (#5784) add 104a323f6f6 HADOOP-18795. S3A DelegationToken plugin to expand return type of binding (#5821) add c35f31640ec HADOOP-18807. Close child file systems in ViewFileSystem when cache is disabled. (#5847) add acb31b6757f HADOOP-18073. Upgrade AWS SDK to v2 in S3A [work in progress] This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (75220b77387) \ N -- N -- N refs/heads/feature-HADOOP-18073-s3a-sdk-upgrade (acb31b6757f) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .gitignore | 1 + LICENSE-binary | 66 +- dev-support/docker/Dockerfile_windows_10 | 6 + dev-support/git-jira-validation/README.md | 6 +- .../git_jira_fix_version_check.py | 2 +- .../hadoop-client-check-invariants/pom.xml | 1 - .../hadoop-client-check-test-invariants/pom.xml | 1 - .../jdiff/Apache_Hadoop_Common_3.3.6.xml | 40994 +++++++++++++++++++ hadoop-common-project/hadoop-common/pom.xml | 9 +- .../hadoop-common/src/main/bin/hadoop-functions.sh | 7 + .../hadoop-common/src/main/conf/log4j.properties | 15 + .../apache/hadoop/fs/CommonConfigurationKeys.java | 8 + .../hadoop/fs/CommonConfigurationKeysPublic.java | 9 + .../main/java/org/apache/hadoop/fs/FSBuilder.java | 56 +- .../src/main/java/org/apache/hadoop/fs/Path.java | 7 +- .../org/apache/hadoop/fs/TrashPolicyDefault.java | 18 +- .../hadoop/fs/impl/AbstractFSBuilderImpl.java | 16 +- .../org/apache/hadoop/fs/impl/BackReference.java | 41 +- .../fs/impl/prefetch/CachingBlockManager.java | 12 +- .../hadoop/fs/impl/prefetch/PrefetchConstants.java | 44 + .../fs/impl/prefetch/SingleFilePerBlockCache.java | 310 +- .../hadoop/fs/statistics/IOStatisticsSetters.java | 75 + .../hadoop/fs/statistics/IOStatisticsSnapshot.java | 30 +- .../impl/ForwardingIOStatisticsStore.java | 3 +- .../fs/statistics/impl/IOStatisticsStore.java | 48 +- .../statistics/impl/IOStatisticsStoreBuilder.java | 11 + .../impl/IOStatisticsStoreBuilderImpl.java | 12 + .../org/apache/hadoop/fs/store/LogExactlyOnce.java | 11 + .../org/apache/hadoop/fs/viewfs/InodeTree.java | 5 + .../apache/hadoop/fs/viewfs/ViewFileSystem.java | 34 + .../java/org/apache/hadoop/ipc/CallerContext.java | 2 +- .../org/apache/hadoop/ipc/ProcessingDetails.java | 15 +- .../main/java/org/apache/hadoop/ipc/Server.java | 17 +- .../hadoop/ipc/metrics/RpcDetailedMetrics.java | 25 +- .../org/apache/hadoop/ipc/metrics/RpcMetrics.java | 9 + .../metrics2/lib/MutableRatesWithAggregation.java | 4 +- .../hadoop/util/curator/ZKCuratorManager.java | 222 +- .../hadoop/util/functional/RemoteIterators.java | 124 +- .../src/main/resources/core-default.xml | 52 +- .../hadoop-common/src/site/markdown/Metrics.md | 7 +- .../site/markdown/release/3.3.6/CHANGELOG.3.3.6.md | 176 + .../markdown/release/3.3.6/RELEASENOTES.3.3.6.md | 31 + .../test/java/org/apache/hadoop/fs/TestPath.java | 7 + .../test/java/org/apache/hadoop/fs/TestTrash.java | 50 + .../hadoop/fs/impl/prefetch/TestBlockCache.java | 6 +- .../fs/statistics/TestIOStatisticsSetters.java | 189 + .../hadoop/fs/store/TestFSBuilderSupport.java | 42 +- .../test/java/org/apache/hadoop/ipc/TestRPC.java | 80 + .../hadoop/metrics2/lib/TestMutableMetrics.java | 15 +- .../org/apache/hadoop/test/MetricsAsserts.java | 13 + .../util/curator/TestSecureZKCuratorManager.java | 214 + .../util/curator/resources/data/ssl/keystore.jks | Bin 0 -> 4160 bytes .../util/curator/resources/data/ssl/truststore.jks | Bin 0 -> 994 bytes .../util/functional/TestRemoteIterators.java | 54 + hadoop-dist/pom.xml | 3 - .../java/org/apache/hadoop/hdfs/DFSClient.java | 37 +- .../apache/hadoop/hdfs/DFSStripedOutputStream.java | 8 +- .../java/org/apache/hadoop/hdfs/DFSUtilClient.java | 8 +- .../org/apache/hadoop/hdfs/client/HdfsUtils.java | 4 +- .../hdfs/client/impl/BlockReaderFactory.java | 20 +- .../hdfs/client/impl/BlockReaderLocalLegacy.java | 6 +- .../hadoop/hdfs/client/impl/DfsClientConf.java | 2 +- .../hadoop/hdfs/client/impl/LeaseRenewer.java | 28 +- .../namenode/ha/ObserverReadProxyProvider.java | 97 +- .../ha/RouterObserverReadProxyProvider.java | 223 + .../hadoop/hdfs/server/protocol/StorageReport.java | 2 +- .../hdfs/shortcircuit/ShortCircuitCache.java | 13 +- .../org/apache/hadoop/hdfs/web/JsonUtilClient.java | 42 + .../apache/hadoop/hdfs/web/WebHdfsFileSystem.java | 27 + .../hadoop/hdfs/web/resources/GetOpParam.java | 2 + .../hadoop/fs/http/client/HttpFSFileSystem.java | 30 + .../apache/hadoop/fs/http/server/FSOperations.java | 69 + .../fs/http/server/HttpFSParametersProvider.java | 2 + .../apache/hadoop/fs/http/server/HttpFSServer.java | 14 + .../http/server/metrics/HttpFSServerMetrics.java | 10 + .../java/org/apache/hadoop/lib/util/Check.java | 6 +- .../hadoop/fs/http/client/BaseTestHttpFSWith.java | 72 +- .../java/org/apache/hadoop/lib/util/TestCheck.java | 10 + hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml | 23 +- .../metrics/FederationRPCPerformanceMonitor.java | 8 +- .../hdfs/server/federation/metrics/RBFMetrics.java | 19 +- .../federation/router/ConnectionContext.java | 9 +- .../federation/router/ConnectionManager.java | 2 +- .../server/federation/router/ConnectionPool.java | 4 +- .../router/NamenodeHeartbeatService.java | 75 +- .../server/federation/router/RBFConfigKeys.java | 3 + .../server/federation/router/RouterRpcClient.java | 5 +- .../server/federation/router/RouterRpcServer.java | 40 +- .../security/token/SQLConnectionFactory.java | 1 - .../store/driver/StateStoreOperationResult.java | 79 + .../store/driver/StateStoreRecordOperations.java | 5 +- .../store/driver/impl/StateStoreBaseImpl.java | 2 +- .../store/driver/impl/StateStoreFileBaseImpl.java | 34 +- .../store/driver/impl/StateStoreMySQLImpl.java | 16 +- .../driver/impl/StateStoreSerializableImpl.java | 17 + .../store/driver/impl/StateStoreZooKeeperImpl.java | 10 +- .../federation/store/impl/MountTableStoreImpl.java | 6 +- .../protocol/AddMountTableEntriesResponse.java | 10 + .../pb/AddMountTableEntriesResponsePBImpl.java | 11 + .../hadoop/hdfs/tools/federation/RouterAdmin.java | 2 +- .../src/main/proto/FederationProtocol.proto | 1 + .../src/main/resources/hdfs-rbf-default.xml | 10 + .../src/main/webapps/router/federationhealth.html | 5 +- .../metrics/TestNameserviceRPCMetrics.java | 23 + .../federation/router/TestObserverWithRouter.java | 267 +- .../federation/router/TestRouterAdminCLI.java | 5 + .../router/TestRouterNamenodeMonitoring.java | 40 +- .../server/federation/router/TestRouterRpc.java | 7 + .../security/TestRouterSecurityManager.java | 53 +- .../store/FederationStateStoreTestUtils.java | 4 +- .../store/driver/TestStateStoreDriverBase.java | 26 +- .../store/driver/TestStateStoreMySQL.java | 2 +- .../store/records/MockStateStoreDriver.java | 11 +- .../dev-support/jdiff/Apache_Hadoop_HDFS_3.3.6.xml | 835 + hadoop-hdfs-project/hadoop-hdfs/pom.xml | 4 +- .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 1 - .../main/java/org/apache/hadoop/hdfs/DFSUtil.java | 13 +- .../hadoop/hdfs/server/balancer/Balancer.java | 4 +- .../hdfs/server/balancer/BalancingPolicy.java | 20 +- .../server/blockmanagement/BlockInfoStriped.java | 3 +- .../hdfs/server/blockmanagement/BlockManager.java | 29 +- .../BlockPlacementPolicyRackFaultTolerant.java | 17 +- .../BlockUnderConstructionFeature.java | 8 + .../server/blockmanagement/DatanodeManager.java | 48 +- .../hdfs/server/datanode/BlockChecksumHelper.java | 4 +- .../hdfs/server/datanode/BlockPoolManager.java | 7 - .../hadoop/hdfs/server/datanode/DataNode.java | 58 +- .../hadoop/hdfs/server/datanode/DataXceiver.java | 4 +- .../hdfs/server/datanode/DirectoryScanner.java | 1 + .../hadoop/hdfs/server/datanode/DiskBalancer.java | 56 +- .../datanode/IncrementalBlockReportManager.java | 4 +- .../server/datanode/fsdataset/FsDatasetSpi.java | 6 + .../datanode/fsdataset/impl/FsDatasetImpl.java | 13 +- .../datanode/fsdataset/impl/FsVolumeImpl.java | 2 +- .../server/datanode/fsdataset/impl/ReplicaMap.java | 10 +- .../datanode/metrics/DataNodeMetricHelper.java | 8 +- .../server/datanode/metrics/DataNodeMetrics.java | 4 + .../server/datanode/metrics/FSDatasetMBean.java | 5 + .../hadoop/hdfs/server/namenode/BackupNode.java | 2 + .../hadoop/hdfs/server/namenode/FSDirConcatOp.java | 2 +- .../hdfs/server/namenode/FSDirWriteFileOp.java | 2 +- .../hadoop/hdfs/server/namenode/FSNamesystem.java | 14 +- .../hdfs/server/namenode/INodeReference.java | 12 +- .../hadoop/hdfs/server/namenode/NameNode.java | 25 + .../server/namenode/NameNodeResourcePolicy.java | 15 +- .../hadoop/hdfs/server/namenode/NamenodeFsck.java | 3 + .../web/resources/NamenodeWebHdfsMethods.java | 12 + .../org/apache/hadoop/hdfs/tools/DFSAdmin.java | 21 +- .../java/org/apache/hadoop/hdfs/tools/DFSck.java | 2 + .../java/org/apache/hadoop/hdfs/web/JsonUtil.java | 23 + .../src/main/webapps/hdfs/dfshealth.html | 3 + .../src/site/markdown/ArchivalStorage.md | 2 +- .../hadoop-hdfs/src/site/markdown/Federation.md | 2 +- .../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 13 +- .../site/markdown/HDFSHighAvailabilityWithQJM.md | 2 +- .../src/site/markdown/HdfsQuotaAdminGuide.md | 2 +- .../hadoop-hdfs/src/site/markdown/HdfsSnapshots.md | 18 +- .../hadoop-hdfs/src/site/markdown/LibHdfs.md | 2 +- .../src/site/markdown/TransparentEncryption.md | 2 +- .../hadoop-hdfs/src/site/markdown/WebHDFS.md | 101 + .../hadoop/fs/viewfs/TestViewFileSystemClose.java | 61 + .../hdfs/ReadStripedFileWithDecodingHelper.java | 22 +- .../TestClientProtocolForPipelineRecovery.java | 2 +- .../org/apache/hadoop/hdfs/TestDFSFinalize.java | 2 +- .../org/apache/hadoop/hdfs/TestDFSRollback.java | 2 +- .../TestDFSStripedOutputStreamUpdatePipeline.java | 2 +- .../org/apache/hadoop/hdfs/TestDFSUpgrade.java | 4 +- .../org/apache/hadoop/hdfs/TestFileAppend4.java | 4 +- .../org/apache/hadoop/hdfs/TestFileCreation.java | 2 + .../hadoop/hdfs/TestLeaseRecoveryStriped.java | 60 + .../hdfs/TestReadStripedFileWithDecoding.java | 127 +- .../hadoop/hdfs/server/balancer/TestBalancer.java | 74 +- .../server/blockmanagement/TestBlockManager.java | 80 + .../blockmanagement/TestSortLocatedBlock.java | 155 +- .../hdfs/server/datanode/SimulatedFSDataset.java | 28 +- .../hdfs/server/datanode/TestDataNodeMXBean.java | 1 + .../datanode/TestDataNodeReconfiguration.java | 57 + .../hdfs/server/datanode/TestDirectoryScanner.java | 19 + .../datanode/extdataset/ExternalDatasetImpl.java | 6 + .../datanode/fsdataset/impl/TestFsDatasetImpl.java | 8 +- .../server/diskbalancer/TestDiskBalancerRPC.java | 2 +- .../server/namenode/TestDiskspaceQuotaUpdate.java | 2 +- .../hadoop/hdfs/server/namenode/TestFsck.java | 86 + .../hdfs/server/namenode/TestHDFSConcat.java | 95 + .../server/namenode/TestNameNodeMetricsLogger.java | 2 + .../server/namenode/TestNameNodeReconfigure.java | 2 + .../namenode/TestNameNodeResourcePolicy.java | 14 +- .../namenode/TestReconstructStripedBlocks.java | 61 + .../hdfs/server/namenode/ha/TestHAMetrics.java | 55 + .../namenode/ha/TestObserverReadProxyProvider.java | 217 +- .../namenode/snapshot/TestRenameWithSnapshots.java | 3 + .../snapshot/TestSnapRootDescendantDiff.java | 38 +- .../namenode/snapshot/TestSnapshotDiffReport.java | 88 +- .../hdfs/shortcircuit/TestShortCircuitCache.java | 19 +- .../org/apache/hadoop/hdfs/tools/TestDFSAdmin.java | 10 +- .../org/apache/hadoop/hdfs/web/TestWebHDFS.java | 51 + .../jdiff/Apache_Hadoop_MapReduce_Common_3.3.6.xml | 113 + .../jdiff/Apache_Hadoop_MapReduce_Core_3.3.6.xml | 28994 +++++++++++++ .../Apache_Hadoop_MapReduce_JobClient_3.3.6.xml | 16 + .../hadoop-mapreduce-client-common/pom.xml | 15 + .../org/apache/hadoop/mapred/TestJobClient.java | 42 +- .../apache/hadoop/mapred/TestJobClientGetJob.java | 13 +- .../mapred/TestLocalDistributedCacheManager.java | 32 +- .../hadoop/mapred/TestLocalModeWithNewApis.java | 17 +- .../hadoop/mapred/TestMRWithDistributedCache.java | 136 +- .../apache/hadoop/mapreduce/TestTypeConverter.java | 81 +- .../hadoop/mapreduce/v2/TestRPCFactories.java | 19 +- .../hadoop/mapreduce/v2/TestRecordFactory.java | 23 +- .../hadoop/mapreduce/v2/api/records/TestIds.java | 33 +- .../v2/jobhistory/TestFileNameIndexUtils.java | 162 +- .../v2/jobhistory/TestJobHistoryUtils.java | 100 +- .../hadoop/mapreduce/v2/util/TestMRApps.java | 284 +- .../hadoop-mapreduce-client-core/pom.xml | 4 +- .../committer/manifest/ManifestCommitter.java | 8 +- .../manifest/ManifestCommitterConfig.java | 21 +- .../manifest/ManifestCommitterConstants.java | 22 +- .../committer/manifest/files/DiagnosticKeys.java | 3 + .../output/committer/manifest/files/DirEntry.java | 26 +- .../output/committer/manifest/files/FileEntry.java | 36 +- .../committer/manifest/impl/EntryFileIO.java | 569 + .../manifest/impl/LoadedManifestData.java | 99 + .../manifest/impl/ManifestCommitterSupport.java | 24 +- .../manifest/stages/AbstractJobOrTaskStage.java | 26 +- .../committer/manifest/stages/CleanupJobStage.java | 2 +- .../committer/manifest/stages/CommitJobStage.java | 201 +- .../stages/CreateOutputDirectoriesStage.java | 88 +- .../manifest/stages/LoadManifestsStage.java | 305 +- .../manifest/stages/RenameFilesStage.java | 58 +- .../committer/manifest/stages/StageConfig.java | 72 + .../manifest/stages/ValidateRenamedFilesStage.java | 51 +- .../src/main/resources/mapred-default.xml | 12 +- .../src/site/markdown/manifest_committer.md | 58 +- .../lib/output/TestFileOutputCommitter.java | 110 +- .../manifest/AbstractManifestCommitterTest.java | 6 +- .../manifest/ManifestCommitterTestSupport.java | 53 +- .../manifest/TestCreateOutputDirectoriesStage.java | 39 +- .../manifest/TestJobThroughManifestCommitter.java | 98 +- .../committer/manifest/TestLoadManifestsStage.java | 97 +- .../committer/manifest/TestRenameStageFailure.java | 32 +- .../committer/manifest/impl/TestEntryFileIO.java | 382 + .../apache/hadoop/mapred/FadvisedFileRegion.java | 102 +- hadoop-project-dist/pom.xml | 35 +- hadoop-project/pom.xml | 16 +- .../java/org/apache/hadoop/fs/s3a/Constants.java | 14 +- .../apache/hadoop/fs/s3a/S3ABlockOutputStream.java | 13 +- .../org/apache/hadoop/fs/s3a/S3ADataBlocks.java | 31 +- .../apache/hadoop/fs/s3a/S3AInstrumentation.java | 2 +- .../apache/hadoop/fs/s3a/WriteOperationHelper.java | 1 - .../org/apache/hadoop/fs/s3a/WriteOperations.java | 7 - .../delegation/AbstractDelegationTokenBinding.java | 35 +- .../s3a/auth/delegation/DelegationBindingInfo.java | 40 +- .../s3a/auth/delegation/S3ADelegationTokens.java | 40 +- .../hadoop/fs/s3a/commit/impl/CommitContext.java | 2 +- .../apache/hadoop/fs/s3a/commit/staging/Paths.java | 13 + .../fs/s3a/commit/staging/StagingCommitter.java | 32 +- .../hadoop/fs/s3a/impl/DirectoryPolicyImpl.java | 4 +- .../fs/s3a/prefetch/S3ACachingBlockManager.java | 11 +- .../site/markdown/tools/hadoop-aws/committers.md | 2 +- .../markdown/tools/hadoop-aws/directory_markers.md | 234 +- .../src/site/markdown/tools/hadoop-aws/index.md | 9 +- .../src/site/markdown/tools/hadoop-aws/s3guard.md | 10 +- .../src/site/markdown/tools/hadoop-aws/testing.md | 38 +- .../hadoop/fs/s3a/ITestS3ABlockOutputArray.java | 45 +- .../fs/s3a/ITestS3APrefetchingLruEviction.java | 243 + .../org/apache/hadoop/fs/s3a/TestDataBlocks.java | 2 +- .../hadoop/fs/s3a/TestS3ABlockOutputStream.java | 11 - .../fs/s3a/commit/AbstractITCommitProtocol.java | 24 + .../integration/ITestStagingCommitProtocol.java | 70 + .../ITestStagingCommitProtocolFailure.java | 2 + .../hadoop/fs/s3a/prefetch/S3APrefetchFakes.java | 6 +- .../s3a/prefetch/TestS3ACachingBlockManager.java | 20 +- .../hadoop/fs/azurebfs/AzureBlobFileSystem.java | 9 +- .../fs/azurebfs/AzureBlobFileSystemStore.java | 14 + .../commit/AbfsManifestStoreOperations.java | 14 +- .../fs/azurebfs/services/AbfsInputStream.java | 10 + .../azurebfs/services/AbfsInputStreamContext.java | 15 + .../fs/azurebfs/services/AbfsOutputStream.java | 21 + .../azurebfs/services/AbfsOutputStreamContext.java | 14 + .../fs/azurebfs/commit/AbfsCommitTestHelper.java | 11 + .../commit/ITestAbfsLoadManifestsStage.java | 14 + .../fs/azurebfs/services/ITestAbfsInputStream.java | 21 + .../azurebfs/services/ITestAbfsOutputStream.java | 78 + hadoop-tools/hadoop-benchmark/pom.xml | 7 - hadoop-tools/hadoop-distcp/pom.xml | 24 +- hadoop-tools/hadoop-federation-balance/pom.xml | 24 +- .../MySQL/FederationStateStoreStoredProcs.sql | 19 +- .../MySQL/FederationStateStoreTables.sql | 3 +- .../SQLServer/FederationStateStoreStoredProcs.sql | 21 +- .../SQLServer/FederationStateStoreTables.sql | 2 +- hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh | 2 +- .../hadoop-yarn/dev-support/findbugs-exclude.xml | 5 + .../jdiff/Apache_Hadoop_YARN_API_3.3.6.xml | 26440 ++++++++++++ .../jdiff/Apache_Hadoop_YARN_Client_3.3.6.xml | 3067 ++ .../jdiff/Apache_Hadoop_YARN_Common_3.3.6.xml | 3987 ++ .../Apache_Hadoop_YARN_Server_Common_3.3.6.xml | 1456 + .../apache/hadoop/yarn/conf/YarnConfiguration.java | 105 +- .../api/ResourceManagerAdministrationProtocol.java | 16 + .../api/protocolrecords/FederationQueueWeight.java | 169 + .../SaveFederationQueuePolicyRequest.java | 70 + .../SaveFederationQueuePolicyResponse.java | 32 +- .../resourcemanager_administration_protocol.proto | 1 + ...arn_server_resourcemanager_service_protos.proto | 10 + .../src/main/proto/yarn_protos.proto | 6 + .../package.json | 3 + .../apache/hadoop/yarn/client/cli/QueueCLI.java | 35 +- .../apache/hadoop/yarn/client/cli/RouterCLI.java | 255 +- .../yarn/client/util/FormattingCLIUtils.java | 277 + .../yarn/client/TestRMFailoverProxyProvider.java | 75 +- .../hadoop/yarn/client/cli/TestRouterCLI.java | 87 +- .../apache/hadoop/yarn/client/cli/TestYarnCLI.java | 35 +- .../yarn/client/util/TestFormattingCLIUtils.java | 79 + .../src/test/resources/FormattingResult | 23 + .../hadoop-yarn/hadoop-yarn-common/pom.xml | 8 +- .../apache/hadoop/yarn/client/AMRMClientUtils.java | 8 +- .../apache/hadoop/yarn/client/ClientRMProxy.java | 24 + .../client/ConfiguredRMFailoverProxyProvider.java | 47 +- .../org/apache/hadoop/yarn/client/RMProxy.java | 58 + .../yarn/nodelabels/store/AbstractFSNodeStore.java | 14 +- ...eManagerAdministrationProtocolPBClientImpl.java | 19 + ...ManagerAdministrationProtocolPBServiceImpl.java | 20 + .../impl/pb/FederationQueueWeightPBImpl.java | 132 + .../pb/SaveFederationQueuePolicyRequestPBImpl.java | 162 + .../SaveFederationQueuePolicyResponsePBImpl.java | 98 + .../yarn/util/AbstractLivelinessMonitor.java | 2 +- .../org/apache/hadoop/yarn/util/AppsCacheKey.java | 141 + .../org/apache/hadoop/yarn/util/CacheNode.java} | 28 +- .../java/org/apache/hadoop/yarn/util/LRUCache.java | 64 + .../src/main/resources/yarn-default.xml | 180 +- .../hadoop/yarn/conf/TestYarnConfiguration.java | 19 + .../nodelabels/TestFileSystemNodeLabelsStore.java | 26 +- .../resourcetypes/ResourceTypesTestHelper.java | 2 +- .../org/apache/hadoop/yarn/util/TestLRUCache.java | 49 + .../hadoop-yarn-server-common/pom.xml | 8 +- .../policies/ConfigurableFederationPolicy.java | 2 +- .../federation/policies/FederationPolicyUtils.java | 2 +- .../federation/policies/RouterPolicyFacade.java | 2 +- .../policies/amrmproxy/RejectAMRMProxyPolicy.java | 2 +- .../policies/dao/WeightedPolicyInfo.java | 4 +- .../policies/manager/FederationPolicyManager.java | 8 +- .../policies/manager/RejectAllPolicyManager.java | 2 +- .../policies/router/AbstractRouterPolicy.java | 2 +- .../store/impl/MemoryFederationStateStore.java | 26 + .../store/impl/SQLFederationStateStore.java | 66 +- .../federation/store/records/SubClusterIdInfo.java | 5 + .../impl/pb/ApplicationHomeSubClusterPBImpl.java | 3 +- .../federation/utils/FederationMethodWrapper.java | 2 +- .../utils/FederationStateStoreFacade.java | 57 +- .../yarn/server/uam/UnmanagedAMPoolManager.java | 25 +- .../server/uam/UnmanagedApplicationManager.java | 23 +- .../yarn/server/MockResourceManagerFacade.java | 16 +- .../TestFederationRMFailoverProxyProvider.java | 48 + .../TestLocalityMulticastAMRMProxyPolicy.java | 2 +- .../router/TestWeightedRandomRouterPolicy.java | 2 +- .../store/impl/FederationStateStoreBaseTest.java | 26 + .../store/impl/HSQLDBFederationStateStore.java | 20 +- .../store/impl/TestMemoryFederationStateStore.java | 34 - .../impl/TestZookeeperFederationStateStore.java | 33 - .../uam/TestUnmanagedApplicationManager.java | 64 +- .../pom.xml | 13 + .../GlobalPolicyGenerator.java | 70 + .../subclustercleaner/SubClusterCleaner.java | 113 + .../subclustercleaner/package-info.java} | 20 +- .../secure/AbstractGlobalPolicyGeneratorTest.java | 177 + .../secure/TestGpgSecureLogins.java | 50 + .../subclustercleaner/TestSubClusterCleaner.java | 121 + .../amrmproxy/AMRMProxyApplicationContext.java | 2 +- .../amrmproxy/AMRMProxyApplicationContextImpl.java | 2 +- .../nodemanager/amrmproxy/AMRMProxyService.java | 2 +- .../amrmproxy/AMRMProxyTokenSecretManager.java | 26 +- .../amrmproxy/AbstractRequestInterceptor.java | 4 +- .../amrmproxy/DefaultRequestInterceptor.java | 34 +- .../amrmproxy/FederationInterceptor.java | 458 +- .../amrmproxy/TokenAndRegisterResponse.java | 43 +- .../loghandler/NonAggregatingLogHandler.java | 49 +- .../monitor/ContainersMonitorImpl.java | 6 + .../nodemanager/metrics/NodeManagerMetrics.java | 8 + .../scheduler/DistributedScheduler.java | 6 +- .../nodemanager/amrmproxy/BaseAMRMProxyTest.java | 265 +- .../amrmproxy/TestAMRMProxyMetrics.java | 2 +- .../amrmproxy/TestAMRMProxyService.java | 160 +- .../amrmproxy/TestFederationInterceptor.java | 735 +- .../amrmproxy/TestFederationInterceptorSecure.java | 220 +- .../amrmproxy/TestableFederationInterceptor.java | 45 +- .../loghandler/TestNonAggregatingLogHandler.java | 99 + .../metrics/TestNodeManagerMetrics.java | 3 + .../yarn/server/resourcemanager/AdminService.java | 9 + .../server/resourcemanager/NodesListManager.java | 14 + .../yarn/server/resourcemanager/RMAppManager.java | 2 +- .../server/resourcemanager/ResourceManager.java | 27 + .../server/resourcemanager/rmapp/RMAppImpl.java | 9 +- .../rmapp/RMAppRunningOnNodeEvent.java | 14 + .../rmapp/attempt/AMLivelinessMonitor.java | 14 +- .../rmcontainer/RMContainerImpl.java | 2 +- .../scheduler/PartitionQueueMetrics.java | 2 +- .../resourcemanager/scheduler/QueueMetrics.java | 44 +- .../AbsoluteResourceCapacityCalculator.java | 3 +- .../capacity/AbstractAutoCreatedLeafQueue.java | 8 + .../scheduler/capacity/AbstractCSQueue.java | 89 +- .../scheduler/capacity/AbstractLeafQueue.java | 72 +- .../scheduler/capacity/AbstractParentQueue.java | 33 + .../scheduler/capacity/CSQueueMetrics.java | 80 +- .../scheduler/capacity/CSQueueUtils.java | 36 +- .../capacity/CapacitySchedulerConfigValidator.java | 9 +- .../capacity/CapacitySchedulerConfiguration.java | 4 + .../CapacitySchedulerQueueCapacityHandler.java | 21 +- .../capacity/CapacitySchedulerQueueManager.java | 3 +- .../scheduler/capacity/ManagedParentQueue.java | 31 +- .../scheduler/capacity/ParentQueue.java | 2 +- .../scheduler/capacity/QueueCapacityVector.java | 3 + .../capacity/ResourceCalculationDriver.java | 5 + .../GuaranteedOrZeroCapacityOverTimePolicy.java | 3 +- .../security/AMRMTokenSecretManager.java | 19 +- .../resourcemanager/webapp/RMWebServices.java | 43 + .../dao/helper/CapacitySchedulerInfoHelper.java | 52 +- .../server/resourcemanager/TestAppManager.java | 2 +- .../TestResourceTrackerService.java | 81 + .../TestWorkPreservingRMRestart.java | 126 +- .../rmapp/TestRMAppTransitions.java | 12 + .../CapacitySchedulerQueueCalculationTestBase.java | 2 +- .../capacity/CapacitySchedulerQueueHelpers.java | 2 +- .../capacity/CapacitySchedulerTestUtilities.java | 8 + .../TestAMAllocatedToNonExclusivePartition.java | 6 +- .../TestAbsoluteResourceConfiguration.java | 20 +- .../TestAbsoluteResourceWithAutoQueue.java | 8 + .../scheduler/capacity/TestApplicationLimits.java | 52 +- .../TestAutoCreatedQueueDeletionPolicy.java | 101 +- .../capacity/TestCSAllocateCustomResource.java | 1 + .../scheduler/capacity/TestCapacityScheduler.java | 20 +- .../TestCapacitySchedulerAutoCreatedQueueBase.java | 18 +- .../TestCapacitySchedulerAutoQueueCreation.java | 18 +- .../TestCapacitySchedulerConfigValidator.java | 109 +- .../TestCapacitySchedulerDynamicBehavior.java | 15 +- .../TestCapacitySchedulerLazyPreemption.java | 1 + .../TestCapacitySchedulerNewQueueAutoCreation.java | 21 +- .../TestCapacitySchedulerNodeLabelUpdate.java | 2 + .../capacity/TestCapacitySchedulerNodes.java | 1 + .../capacity/TestCapacitySchedulerQueues.java | 199 +- ...stCapacitySchedulerSchedulingRequestUpdate.java | 2 + ...stCapacitySchedulerWorkflowPriorityMapping.java | 8 + .../capacity/TestContainerAllocation.java | 2 + .../scheduler/capacity/TestLeafQueue.java | 38 +- .../TestMixedQueueResourceCalculation.java | 2 +- .../capacity/TestNodeLabelContainerAllocation.java | 10 +- .../scheduler/capacity/TestParentQueue.java | 33 +- .../capacity/TestQueueCapacityVector.java | 15 + .../scheduler/capacity/TestQueueMappings.java | 8 + .../scheduler/capacity/TestQueueParsing.java | 945 +- .../scheduler/capacity/TestReservationQueue.java | 15 +- .../scheduler/capacity/TestReservations.java | 8 + .../TestSchedulingRequestContainerAllocation.java | 1 + .../fair/converter/TestFSQueueConverter.java | 12 +- .../resourcemanager/webapp/TestRMWebServices.java | 74 + .../webapp/TestRMWebServicesCapacitySched.java | 433 +- ...estRMWebServicesCapacitySchedDynamicConfig.java | 398 +- ...ebServicesCapacitySchedLegacyQueueCreation.java | 94 + ...estRMWebServicesCapacitySchedulerMixedMode.java | 217 + .../TestRMWebServicesForCSWithPartitions.java | 204 +- .../TestRMWebServicesSchedulerActivities.java | 86 +- .../resourcemanager/webapp/TestWebServiceUtil.java | 268 + .../webapp/dynamic-testAbsoluteMode-0.json | 2935 ++ .../webapp/dynamic-testAbsoluteMode-16.json | 2935 ++ .../webapp/dynamic-testAbsoluteMode-32.json | 2935 ++ .../webapp/dynamic-testAbsoluteMode-laqc.json | 3292 ++ .../webapp/dynamic-testPercentageMode-0.json | 2935 ++ .../webapp/dynamic-testPercentageMode-16.json | 2935 ++ .../webapp/dynamic-testPercentageMode-32.json | 2935 ++ .../webapp/dynamic-testPercentageMode-laqc.json | 3292 ++ .../resources/webapp/dynamic-testWeightMode-0.json | 2935 ++ .../webapp/dynamic-testWeightMode-16.json | 2935 ++ .../webapp/dynamic-testWeightMode-32.json | 2935 ++ .../webapp/dynamic-testWeightMode-after-aqc.json | 7710 ++++ .../webapp/dynamic-testWeightMode-before-aqc.json | 2790 ++ ...mixed-testSchedulerAbsoluteAndPercentage-0.json | 2935 ++ ...ixed-testSchedulerAbsoluteAndPercentage-16.json | 2935 ++ ...ixed-testSchedulerAbsoluteAndPercentage-32.json | 2935 ++ ...tSchedulerAbsoluteAndPercentageAndWeight-0.json | 2935 ++ ...SchedulerAbsoluteAndPercentageAndWeight-16.json | 2935 ++ ...SchedulerAbsoluteAndPercentageAndWeight-32.json | 2935 ++ ...dulerAbsoluteAndPercentageAndWeightMixed-0.json | 2935 ++ ...ulerAbsoluteAndPercentageAndWeightMixed-16.json | 2935 ++ ...ulerAbsoluteAndPercentageAndWeightMixed-32.json | 2935 ++ .../mixed-testSchedulerAbsoluteAndWeight-0.json | 2935 ++ .../mixed-testSchedulerAbsoluteAndWeight-16.json | 2935 ++ .../mixed-testSchedulerAbsoluteAndWeight-32.json | 2935 ++ .../mixed-testSchedulerPercentageAndWeight-0.json | 2935 ++ .../mixed-testSchedulerPercentageAndWeight-16.json | 2935 ++ .../mixed-testSchedulerPercentageAndWeight-32.json | 2935 ++ .../webapp/scheduler-response-AbsoluteMode.json | 1739 - ...er-response-AbsoluteModeLegacyAutoCreation.json | 3043 +- .../scheduler-response-NodeLabelDefaultAPI.xml | 4162 +- .../scheduler-response-PerUserResources.json | 7343 +--- .../webapp/scheduler-response-PerUserResources.xml | 4987 +-- .../webapp/scheduler-response-PercentageMode.json | 1739 - ...-response-PercentageModeLegacyAutoCreation.json | 2569 +- .../webapp/scheduler-response-WeightMode.json | 2224 - ...onse-WeightModeWithAutoCreatedQueues-After.json | 6006 --- ...nse-WeightModeWithAutoCreatedQueues-Before.json | 2092 - .../test/resources/webapp/scheduler-response.json | 6488 +-- .../test/resources/webapp/scheduler-response.xml | 4150 +- .../hadoop/yarn/server/router/RouterMetrics.java | 31 + .../server/router/cleaner/SubClusterCleaner.java | 2 +- .../clientrm/DefaultClientRequestInterceptor.java | 28 +- .../rmadmin/DefaultRMAdminRequestInterceptor.java | 45 +- .../rmadmin/FederationRMAdminInterceptor.java | 138 +- .../router/rmadmin/RouterRMAdminService.java | 9 + .../webapp/DefaultRequestInterceptorREST.java | 1 + .../server/router/webapp/RouterWebServiceUtil.java | 34 +- .../yarn/server/router/TestRouterMetrics.java | 33 + .../PassThroughRMAdminRequestInterceptor.java | 8 + .../rmadmin/TestFederationRMAdminInterceptor.java | 97 + .../webapp/MockDefaultRequestInterceptorREST.java | 20 +- .../router/webapp/TestRouterWebServiceUtil.java | 84 + .../collector/NodeTimelineCollectorManager.java | 32 +- .../PerNodeTimelineCollectorsAuxService.java | 4 +- .../collector/TimelineCollectorManager.java | 10 +- .../collector/TimelineCollectorWebService.java | 6 +- .../reader/TimelineReaderServer.java | 4 +- .../reader/TimelineReaderWebServices.java | 133 +- ...TimelineReaderWhitelistAuthorizationFilter.java | 6 +- .../storage/FileSystemTimelineWriterImpl.java | 6 +- .../src/site/markdown/Federation.md | 392 +- pom.xml | 4 + 522 files changed, 217976 insertions(+), 43825 deletions(-) create mode 100644 hadoop-common-project/hadoop-common/dev-support/jdiff/Apache_Hadoop_Common_3.3.6.xml copy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppRunningOnNodeEvent.java => hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/BackReference.java (53%) create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/PrefetchConstants.java create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/IOStatisticsSetters.java rename {hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a => hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs}/statistics/impl/ForwardingIOStatisticsStore.java (97%) create mode 100644 hadoop-common-project/hadoop-common/src/site/markdown/release/3.3.6/CHANGELOG.3.3.6.md create mode 100644 hadoop-common-project/hadoop-common/src/site/markdown/release/3.3.6/RELEASENOTES.3.3.6.md create mode 100644 hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/statistics/TestIOStatisticsSetters.java create mode 100644 hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestSecureZKCuratorManager.java create mode 100644 hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/resources/data/ssl/keystore.jks create mode 100644 hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/resources/data/ssl/truststore.jks create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RouterObserverReadProxyProvider.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/StateStoreOperationResult.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/dev-support/jdiff/Apache_Hadoop_HDFS_3.3.6.xml create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemClose.java create mode 100644 hadoop-mapreduce-project/dev-support/jdiff/Apache_Hadoop_MapReduce_Common_3.3.6.xml create mode 100644 hadoop-mapreduce-project/dev-support/jdiff/Apache_Hadoop_MapReduce_Core_3.3.6.xml create mode 100644 hadoop-mapreduce-project/dev-support/jdiff/Apache_Hadoop_MapReduce_JobClient_3.3.6.xml create mode 100644 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/impl/EntryFileIO.java create mode 100644 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/impl/LoadedManifestData.java create mode 100644 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/impl/TestEntryFileIO.java copy hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/files/DiagnosticKeys.java => hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/DelegationBindingInfo.java (51%) create mode 100644 hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingLruEviction.java create mode 100644 hadoop-yarn-project/hadoop-yarn/dev-support/jdiff/Apache_Hadoop_YARN_API_3.3.6.xml create mode 100644 hadoop-yarn-project/hadoop-yarn/dev-support/jdiff/Apache_Hadoop_YARN_Client_3.3.6.xml create mode 100644 hadoop-yarn-project/hadoop-yarn/dev-support/jdiff/Apache_Hadoop_YARN_Common_3.3.6.xml create mode 100644 hadoop-yarn-project/hadoop-yarn/dev-support/jdiff/Apache_Hadoop_YARN_Server_Common_3.3.6.xml create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/FederationQueueWeight.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/SaveFederationQueuePolicyRequest.java copy hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/AddMountTableEntriesResponse.java => hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/SaveFederationQueuePolicyResponse.java (58%) create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/util/FormattingCLIUtils.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/util/TestFormattingCLIUtils.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/resources/FormattingResult create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/FederationQueueWeightPBImpl.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/SaveFederationQueuePolicyRequestPBImpl.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/SaveFederationQueuePolicyResponsePBImpl.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AppsCacheKey.java copy hadoop-yarn-project/hadoop-yarn/{hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppRunningOnNodeEvent.java => hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/CacheNode.java} (64%) create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LRUCache.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLRUCache.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/failover/TestFederationRMFailoverProxyProvider.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/subclustercleaner/SubClusterCleaner.java copy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/{hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppRunningOnNodeEvent.java => hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/subclustercleaner/package-info.java} (58%) create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/test/java/org/apache/hadoop/yarn/server/globalpolicygenerator/secure/AbstractGlobalPolicyGeneratorTest.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/test/java/org/apache/hadoop/yarn/server/globalpolicygenerator/secure/TestGpgSecureLogins.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/test/java/org/apache/hadoop/yarn/server/globalpolicygenerator/subclustercleaner/TestSubClusterCleaner.java copy hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/store/LogExactlyOnce.java => hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TokenAndRegisterResponse.java (50%) create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySchedLegacyQueueCreation.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySchedulerMixedMode.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestWebServiceUtil.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testAbsoluteMode-0.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testAbsoluteMode-16.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testAbsoluteMode-32.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testAbsoluteMode-laqc.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testPercentageMode-0.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testPercentageMode-16.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testPercentageMode-32.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testPercentageMode-laqc.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testWeightMode-0.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testWeightMode-16.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testWeightMode-32.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testWeightMode-after-aqc.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/dynamic-testWeightMode-before-aqc.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerAbsoluteAndPercentage-0.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerAbsoluteAndPercentage-16.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerAbsoluteAndPercentage-32.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerAbsoluteAndPercentageAndWeight-0.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerAbsoluteAndPercentageAndWeight-16.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerAbsoluteAndPercentageAndWeight-32.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerAbsoluteAndPercentageAndWeightMixed-0.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerAbsoluteAndPercentageAndWeightMixed-16.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerAbsoluteAndPercentageAndWeightMixed-32.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerAbsoluteAndWeight-0.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerAbsoluteAndWeight-16.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerAbsoluteAndWeight-32.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerPercentageAndWeight-0.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerPercentageAndWeight-16.json create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/mixed-testSchedulerPercentageAndWeight-32.json delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/scheduler-response-AbsoluteMode.json delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/scheduler-response-PercentageMode.json delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/scheduler-response-WeightMode.json delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/scheduler-response-WeightModeWithAutoCreatedQueues-After.json delete mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/scheduler-response-WeightModeWithAutoCreatedQueues-Before.json --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org