This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a change to branch HDDS-1564
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git.
omit a9b88c3 HDDS-2089: Add createPipeline CLI. (#1418)
omit 06492fa HDDS-1571. Create an interface for pipeline placement policy
to support network topologies. (#1395)
omit ebbf681 HDDS-1577. Add default pipeline placement policy
implementation. (#1366)
add 085af1a HDDS-2161. Create RepeatedKeyInfo structure to be saved in
deletedTable
add 2078440 HDDS-2160. Add acceptance test for ozonesecure-mr compose.
Contributed by Xiaoyu Yao. (#1490)
add 98fc3f1 HDDS-2159. Fix Race condition in ProfileServlet#pid.
add f07b133 HDDS-2081. Fix
TestRatisPipelineProvider#testCreatePipelinesDnExclude. Contributed by
Aravindan Vijayan. (#1506)
add 1da6dd0 HDDS-2167. Hadoop31-mr acceptance test is failing due to the
shading
add bd1b9e5 HDDS-2170. Add Object IDs and Update ID to Volume Object
(#1510)
add 06661b8 HDDS-2172.Ozone shell should remove description about REST
protocol support. Contributed by Siddharth Wagle.
add 8bfbc71 HDDS-2168. TestOzoneManagerDoubleBufferWithOMResponse
sometimes fails with out of memory error (#1509)
add ea36385 HDDS-2171. Dangling links in test report due to incompatible
realpath
add 17cbae1 HDDS-1738. Add nullable annotation for OMResponse classes
add 191d4f2 HDDS-2165. Freon fails if bucket does not exists
add 84c03a7 HDDS-2067. Create generic service facade with
tracing/metrics/logging support
add adec66f HDDS-2182. Fix checkstyle violations introduced by HDDS-1738
add f7d9fbd HDDS-2180. Add Object ID and update ID on VolumeList Object.
(#1526)
add 861fae2 HDDS-2174. Delete GDPR Encryption Key from metadata when a
Key is deleted
add 4eafc11 HDD-2193. Adding container related metrics in SCM.
add 5f6db5d HDDS-2179. ConfigFileGenerator fails with Java 10 or newer
add abdd2bd HDDS-2149. Replace findbugs with spotbugs
add 1a93d9d HDDS-2185. createmrenv failure not reflected in acceptance
test result
add 90d7cc4 HDDS-1146. Adding container related metrics in SCM. (#1541)
add 57dfd7c HDDS-2183. Container and pipline subcommands of scmcli should
be grouped
add 1f7d8bf HDDS-2153. Add a config to tune max pending requests in Ratis
leader
add b889902 HDDS-2202. Remove unused import in OmUtils
add 737546b HDDS-2207. Update Ratis to latest snapshot. Contributed by
Shashikant Bannerjee. (#1550)
add ffd2f8b HDDS-2205. checkstyle.sh reports wrong failure count
add 3582431 HDDS-1615. ManagedChannel references are being leaked in
ReplicationSupervisor.java. Contributed by Mukul Kumar Singh. (#1547)
add 6062a1a HDDS-2166. Some RPC metrics are missing from SCM prometheus
endpoint
add 911f8b8 HDDS-2210. ContainerStateMachine should not be marked
unhealthy if applyTransaction fails with closed container exception(#1552).
add 56d2873 HDDS-2187. ozone-mr test fails with No FileSystem for scheme
"o3fs"
add bdb3081 HDDS-2201. Rename VolumeList to UserVolumeInfo. (#1566)
add 0334f45 HDDS-2068. Make StorageContainerDatanodeProtocolService
message based
add cf69043 HDDS-2073. Make SCMSecurityProtocol message based.
Contributed by Elek, Marton.
add 3d013c5 HDDS-2227. GDPR key generation could benefit from
secureRandom. (#1574)
add 3a7bbdd HDDS-2162. Make OM Generic related configuration support HA
style config. (#1511)
add 1754b32 HDDS-2224. Fix loadup cache for cache cleanup policy NEVER.
(#1567)
add bd304bc HDDS-2019. Handle Set DtService of token in S3Gateway for OM
HA. (#1489)
add 2be51cf HDDS-2072. Make StorageContainerLocationProtocolService
message based Contributed by Elek, Marton.
add 67100ff HDDS-2228. Fix NPE in
OzoneDelegationTokenManager#addPersistedDelegat… (#1571)
add c8d3ca1 HDDS-2226. S3 Secrets should use a strong RNG. (#1572)
add 8560edf HDDS-2211. Collect docker logs if env fails to start (#1553)
add 2aace9e HDDS-2234. rat.sh fails due to ozone-recon-web/build files
(#1580)
add 5249f99 HDDS-2231. test-single.sh cannot copy results (#1575)
add 751c8d0 HDDS-1720 : Add ability to configure RocksDB logs for Ozone
Manager.
add 86ce58d HDDS-2200 : Recon does not handle the NULL snapshot from OM
DB cleanly.
add 1135b7c HDDS-2198. SCM should not consider containers in CLOSING
state to come out of safemode. (#1540)
add 2cd1864 HDDS-2223. Support ReadWrite lock in LockManager. (#1564)
add 7fbee75 HDDS-2225. SCM fails to start in most unsecure environments
due to leftover secure config
add 80ac600 HDDS-2222 (#1578)
add 1534e35 Revert "HDDS-2222 (#1578)" (#1594)
add a2ebf5a HDDS-2230. Invalid entries in ozonesecure-mr config
add 55f9233 HDDS-2140. Add robot test for GDPR feature
add 49ed516 HDDS-2199. In SCMNodeManager dnsToUuidMap cannot track
multiple DNs on the same host
add 5119799 HDDS-2216. Rename HADOOP_RUNNER_VERSION to
OZONE_RUNNER_VERSION in compose .env files.
add a6b8b16 HDDS-2222. Add a method to update ByteBuffer in
PureJavaCrc32/PureJavaCrc32C. (#1595)
add 6934402 HDDS-2230. Invalid entries in ozonesecure-mr config.
(Addendum)
add b6461c3 HDDS-2237. KeyDeletingService throws NPE if it's started too
early (#1584)
add c7232bb HDDS-2164 : om.db.checkpoints is getting filling up fast.
(#1536)
add e87d6df HDDS-2158. Fixing Json Injection Issue in JsonUtils. (#1486)
add 5583014 HDDS-2250. Generated configs missing from
ozone-filesystem-lib jars
add 37bc8b0 HDDS-2257. Fix checkstyle issues in ChecksumByteBuffer (#1603)
add 4d285c6 HDDS-2251. Add an option to customize unit.sh and
integration.sh parameters
add e70ea7b HDDS-2169. Avoid buffer copies while submitting client
requests in Ratis. Contributed by Tsz-wo Sze(#1517).
add ed47ffe HDDS-2252. Enable gdpr robot test in daily build
add 83a5715 HDDS-2239. Fix TestOzoneFsHAUrls (#1600)
add 719240c HDDS-2238. Container Data Scrubber spams log in empty cluster
add dd2bafa HDDS-2264. Improve output of TestOzoneContainer
add 796c057 HDDS-2259. Container Data Scrubber computes wrong checksum
add a7fdbf2 HDDS-2262. SLEEP_SECONDS: command not found
add e1eb29d HDDS-2245. Use dynamic ports for SCM in
TestSecureOzoneCluster Contributed by kevin su.
add 6a580fd HDDS-2260. Avoid evaluation of LOG.trace and LOG.debug
statement in the read/write path (HDDS). (#1612)
add 70cf448 HDDS-2244. Use new ReadWrite lock in OzoneManager. (#1589)
add 8ec4ba2 HDDS-2233 - Remove ByteStringHelper and refactor the code to
the place where it used (#1596)
add a15782a HDDS-2217. Remove log4j and audit configuration from the
docker-config files
add 3d5e915 HDDS-2217. Remove log4j and audit configuration from the
docker-config files
add 71cc36f Squashed commit of the following:
add 28e6203 HDDS-2265. integration.sh may report false negative
add 253faed HDDS-2266. Avoid evaluation of LOG.trace and LOG.debug
statement in the read/write path. (#1633)
add 9ff77cd HDDS-2269. Provide config for fair/non-fair for OM RW Lock.
(#1623)
add f89eda1 HDDS-1984. Fix listBucket API. (#1555)
add 640255a HDDS-1986. Fix listkeys API. (#1588)
add 2862cdd HDDS-2282. scmcli pipeline list command throws
NullPointerException. Contributed by Xiaoyu Yao. (#1642)
add 21ef389 HDDS-2213.Reduce key provider loading log level in
OzoneFileSystem#getAdditionalTokenIssuers (#1556)
add c8f14a5 HDDS-2287. Import common utility scripts and txt files from
Hadoop without history.
add e722e4d HDDS-2290. Rename pom.ozone.xml to pom.xml
new f236926 HDDS-1577. Add default pipeline placement policy
implementation. (#1366)
new baa5626 HDDS-1571. Create an interface for pipeline placement policy
to support network topologies. (#1395)
new 764fac5 HDDS-2089: Add createPipeline CLI. (#1418)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a9b88c3)
\
N -- N -- N refs/heads/HDDS-1564 (764fac5)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
BUILDING.txt | 511 +++++++++++++++++++
.../main/license/src/LICENSE.txt => LICENSE.txt | 53 +-
.../src/main/license/src/NOTICE.txt => NOTICE.txt | 5 +-
README.txt | 7 +
.../hadoop-ozone.sh => dev-support/bin/qbt | 7 +-
.../bin/smart-apply-patch | 7 +-
.../hadoop-ozone.sh => dev-support/bin/test-patch | 7 +-
dev-support/bin/yetus-wrapper | 188 +++++++
.../framework => dev-support/byteman}/README.md | 17 +-
dev-support/byteman/hadooprpc.btm | 44 ++
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 17 +-
.../hadoop/hdds/scm/XceiverClientManager.java | 11 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 67 +--
.../hdds/scm/client/ContainerOperationClient.java | 10 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 6 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 43 +-
.../apache/hadoop/hdds/scm/storage/BufferPool.java | 15 +
.../hadoop/hdds/scm/storage/CommitWatcher.java | 8 +-
.../common/dev-support/findbugsExcludeFile.xml | 5 +
hadoop-hdds/common/pom.xml | 4 +-
.../java/org/apache/hadoop/hdds/HddsUtils.java | 6 +-
.../apache/hadoop/hdds/conf/HddsConfServlet.java | 4 +-
.../function/FunctionWithServiceException.java | 21 +-
.../hdds/{cli => function}/package-info.java | 4 +-
.../SCMSecurityProtocolClientSideTranslatorPB.java | 104 ++--
.../SCMSecurityProtocolServerSideTranslatorPB.java | 132 -----
.../hdds/ratis/ContainerCommandRequestMessage.java | 107 ++++
.../org/apache/hadoop/hdds/ratis/RatisHelper.java | 17 +-
.../hadoop/hdds/scm/ByteStringConversion.java | 62 +++
.../apache/hadoop/hdds/scm/ByteStringHelper.java | 69 ---
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 5 +
.../hadoop/hdds/scm/container/ContainerInfo.java | 2 +-
.../hadoop/hdds/scm/net/NetworkTopologyImpl.java | 21 +-
.../apache/hadoop/hdds/scm/pipeline/Pipeline.java | 15 +-
...inerLocationProtocolClientSideTranslatorPB.java | 411 ++++++++-------
.../hdds/security/token/BlockTokenVerifier.java | 4 +-
.../security/token/OzoneBlockTokenSelector.java | 8 +-
.../authority/PKIProfiles/DefaultProfile.java | 4 +-
.../hdds/security/x509/keys/HDDSKeyGenerator.java | 6 +-
.../apache/hadoop/hdds/tracing/StringCodec.java | 4 +-
.../hadoop/hdds/utils/BackgroundService.java | 9 +-
.../apache/hadoop/hdds/utils/HddsVersionInfo.java | 6 +-
.../org/apache/hadoop/hdds/utils/LevelDBStore.java | 20 +-
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 24 +-
.../hadoop/hdds/utils/db/RDBCheckpointManager.java | 2 +-
.../hadoop/hdds/utils/db/RocksDBCheckpoint.java | 3 +-
.../hdds/utils/db/RocksDBConfiguration.java} | 48 +-
.../apache/hadoop/hdds/utils/db/TypedTable.java | 2 +-
.../hadoop/hdds/utils/db/cache/CacheKey.java | 11 +-
.../hadoop/hdds/utils/db/cache/TableCache.java | 9 +
.../hadoop/hdds/utils/db/cache/TableCacheImpl.java | 19 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 8 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 2 +
.../hadoop/ozone/common/ChecksumByteBuffer.java | 122 +++++
.../ozone/common/PureJavaCrc32ByteBuffer.java | 556 ++++++++++++++++++++
.../ozone/common/PureJavaCrc32CByteBuffer.java | 559 +++++++++++++++++++++
.../helpers/ContainerCommandRequestPBHelper.java | 16 +-
.../hadoop/ozone/lease/LeaseCallbackExecutor.java | 2 +-
.../apache/hadoop/ozone/lease/LeaseManager.java | 6 +-
.../org/apache/hadoop/ozone/lock/ActiveLock.java | 72 ++-
.../org/apache/hadoop/ozone/lock/LockManager.java | 183 ++++++-
.../hadoop/ozone/lock/PooledLockFactory.java | 7 +-
...inerLocationProtocolServerSideTranslatorPB.java | 395 ---------------
.../apache/hadoop/ozone/web/utils/JsonUtils.java | 5 +-
.../src/main/proto/SCMSecurityProtocol.proto | 96 ++--
.../src/main/proto/ScmBlockLocationProtocol.proto | 2 +-
.../proto/StorageContainerLocationProtocol.proto | 185 ++++---
.../common/src/main/resources/ozone-default.xml | 19 +
.../ratis/TestContainerCommandRequestMessage.java | 152 ++++++
.../hadoop/hdds/utils/db/TestDBStoreBuilder.java | 16 +-
.../ozone/common/TestChecksumByteBuffer.java | 102 ++++
.../apache/hadoop/ozone/lock/TestLockManager.java | 145 +++++-
.../hadoop/hdds/conf/ConfigFileGenerator.java | 5 +-
hadoop-hdds/container-service/pom.xml | 9 +-
.../ozone/container/common/impl/ContainerSet.java | 64 ++-
.../container/common/impl/HddsDispatcher.java | 11 +-
.../RandomContainerDeletionChoosingPolicy.java | 11 +-
...TopNOrderedContainerDeletionChoosingPolicy.java | 13 +-
.../common/statemachine/EndpointStateMachine.java | 4 +-
.../CloseContainerCommandHandler.java | 6 +-
.../commandhandler/DeleteBlocksCommandHandler.java | 28 +-
.../server/ratis/ContainerStateMachine.java | 72 ++-
.../transport/server/ratis/XceiverServerRatis.java | 13 +-
.../container/common/volume/HddsVolumeChecker.java | 14 +-
.../common/volume/ThrottledAsyncChecker.java | 8 +-
.../container/keyvalue/KeyValueBlockIterator.java | 6 +-
.../container/keyvalue/KeyValueContainerCheck.java | 34 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 117 +++--
.../container/keyvalue/helpers/ChunkUtils.java | 42 +-
.../container/keyvalue/impl/BlockManagerImpl.java | 8 +-
.../keyvalue/impl/ChunkManagerDummyImpl.java | 6 +-
.../container/keyvalue/impl/ChunkManagerImpl.java | 69 ++-
.../keyvalue/interfaces/ChunkManager.java | 2 +-
.../background/BlockDeletingService.java | 6 +-
.../container/ozoneimpl/ContainerController.java | 6 +-
.../container/ozoneimpl/ContainerDataScanner.java | 50 +-
.../ozoneimpl/ContainerDataScrubberMetrics.java | 4 +-
.../ozoneimpl/ContainerMetadataScanner.java | 19 +-
.../ContainerMetadataScrubberMetrics.java | 5 +-
.../ozoneimpl/ContainerScrubberConfiguration.java | 17 +
.../ozone/container/ozoneimpl/OzoneContainer.java | 8 +-
.../replication/GrpcReplicationClient.java | 6 +
...inerDatanodeProtocolClientSideTranslatorPB.java | 60 ++-
...inerDatanodeProtocolServerSideTranslatorPB.java | 115 +++--
.../proto/StorageContainerDatanodeProtocol.proto | 58 ++-
.../ozone/container/common/SCMTestUtils.java | 4 +-
.../container/common/impl/TestContainerSet.java | 18 +-
.../container/keyvalue/TestChunkManagerImpl.java | 69 +--
.../keyvalue/TestKeyValueContainerCheck.java | 69 ++-
.../ozoneimpl/TestContainerScrubberMetrics.java | 25 +-
.../container/ozoneimpl/TestOzoneContainer.java | 23 +-
hadoop-hdds/docs/content/beyond/Containers.md | 2 +-
hadoop-hdds/docs/content/start/FromSource.md | 2 +-
.../server/OzoneProtocolMessageDispatcher.java | 88 ++++
.../apache/hadoop/hdds/server/ProfileServlet.java | 10 +-
.../hadoop/hdds/server/PrometheusMetricsSink.java | 16 +-
.../hadoop/hdds/server/events/EventQueue.java | 2 +-
.../hdds/server/TestPrometheusMetricsSink.java | 77 ++-
hadoop-hdds/pom.xml | 18 +-
hadoop-hdds/server-scm/pom.xml | 4 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 10 +-
.../hdds/scm/block/SCMBlockDeletingService.java | 12 +-
.../scm/command/CommandStatusReportHandler.java | 12 +-
.../container/AbstractContainerReportHandler.java | 14 +-
.../scm/container/ContainerActionsHandler.java | 6 +-
.../hdds/scm/container/ContainerManager.java | 8 +
.../hdds/scm/container/ContainerReportHandler.java | 2 +
.../hdds/scm/container/ContainerStateManager.java | 4 +-
.../IncrementalContainerReportHandler.java | 16 +-
.../hdds/scm/container/SCMContainerManager.java | 48 +-
.../metrics/SCMContainerManagerMetrics.java | 144 ++++++
.../scm/{ => container/metrics}/package-info.java | 4 +-
.../scm/container/states/ContainerAttribute.java | 22 +-
.../scm/container/states/ContainerStateMap.java | 6 +-
.../apache/hadoop/hdds/scm/node/NodeManager.java | 8 +-
.../hadoop/hdds/scm/node/SCMNodeManager.java | 61 ++-
.../hdds/scm/pipeline/PipelineReportHandler.java | 4 +-
.../hdds/scm/pipeline/RatisPipelineProvider.java | 4 +-
.../hdds/scm/pipeline/RatisPipelineUtils.java | 4 +-
.../SCMSecurityProtocolServerSideTranslatorPB.java | 186 +++++++
...lockLocationProtocolServerSideTranslatorPB.java | 56 +--
...inerLocationProtocolServerSideTranslatorPB.java | 393 +++++++++++++++
.../hadoop/hdds/scm/protocol}/package-info.java | 8 +-
.../hdds/scm/safemode/ContainerSafeModeRule.java | 26 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 9 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 15 +-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 102 ++--
.../hdds/scm/server/SCMSecurityProtocolServer.java | 27 +-
.../hdds/scm/server/StorageContainerManager.java | 18 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 36 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 67 ++-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 6 +-
.../scm/server/TestSCMBlockProtocolServer.java | 3 +-
.../testutils/ReplicationNodeManagerMock.java | 5 +-
.../org/apache/hadoop/hdds/scm/cli/SCMCLI.java | 45 +-
.../hdds/scm/cli/container/CloseSubcommand.java | 7 +-
.../ContainerCommands.java} | 21 +-
.../hdds/scm/cli/container/CreateSubcommand.java | 5 +-
.../hdds/scm/cli/container/DeleteSubcommand.java | 7 +-
.../hdds/scm/cli/container/InfoSubcommand.java | 5 +-
.../hdds/scm/cli/container/ListSubcommand.java | 9 +-
.../cli/pipeline/ActivatePipelineSubcommand.java | 11 +-
.../scm/cli/pipeline/ClosePipelineSubcommand.java | 11 +-
.../cli/pipeline/DeactivatePipelineSubcommand.java | 11 +-
.../scm/cli/pipeline/ListPipelinesSubcommand.java | 11 +-
.../PipelineCommands.java} | 21 +-
.../client/io/BlockOutputStreamEntryPool.java | 22 +-
.../hadoop/ozone/client/io/KeyInputStream.java | 6 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 18 +-
hadoop-ozone/common/pom.xml | 4 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 154 +++---
.../apache/hadoop/ozone/om/OMMetadataManager.java | 8 +-
.../hadoop/ozone/om/S3SecretManagerImpl.java | 4 +-
.../ozone/om/codec/RepeatedOmKeyInfoCodec.java | 52 ++
...lumeListCodec.java => UserVolumeInfoCodec.java} | 13 +-
.../ozone/om/ha/OMFailoverProxyProvider.java | 6 +-
.../hadoop/ozone/om/helpers/OMRatisHelper.java | 4 +-
.../hadoop/ozone/om/helpers/OmVolumeArgs.java | 100 +++-
.../hadoop/ozone/om/helpers/RepeatedOmKeyInfo.java | 91 ++++
.../hadoop/ozone/om/lock/OzoneManagerLock.java | 139 ++++-
.../hadoop/ozone/security/GDPRSymmetricKey.java | 8 +-
.../security/OzoneBlockTokenSecretManager.java | 2 +-
.../OzoneDelegationTokenSecretManager.java | 11 +-
.../security/OzoneDelegationTokenSelector.java | 8 +-
.../hadoop/ozone/security/OzoneSecretManager.java | 13 +-
.../src/main/proto/OzoneManagerProtocol.proto | 18 +-
.../java/org/apache/hadoop/ozone/TestOmUtils.java | 79 ++-
.../ozone/security/TestGDPRSymmetricKey.java | 4 +-
hadoop-ozone/csi/pom.xml | 4 +-
.../dev-support/checks/_mvn_unit_report.sh | 21 +-
hadoop-ozone/dev-support/checks/blockade.sh | 2 +-
hadoop-ozone/dev-support/checks/build.sh | 2 +-
hadoop-ozone/dev-support/checks/checkstyle.sh | 4 +-
hadoop-ozone/dev-support/checks/findbugs.sh | 12 +-
hadoop-ozone/dev-support/checks/integration.sh | 6 +-
hadoop-ozone/dev-support/checks/unit.sh | 2 +-
.../dist/src/main/assemblies/ozone-src.xml | 7 +-
hadoop-ozone/dist/src/main/compose/ozone-hdfs/.env | 2 +-
.../main/compose/ozone-hdfs/docker-compose.yaml | 6 +-
.../dist/src/main/compose/ozone-hdfs/docker-config | 46 --
.../dist/src/main/compose/ozone-mr/common-config | 9 -
.../dist/src/main/compose/ozone-mr/hadoop27/.env | 2 +-
.../compose/ozone-mr/hadoop27/docker-compose.yaml | 8 +-
.../dist/src/main/compose/ozone-mr/hadoop31/.env | 2 +-
.../compose/ozone-mr/hadoop31/docker-compose.yaml | 8 +-
.../dist/src/main/compose/ozone-mr/hadoop32/.env | 2 +-
.../compose/ozone-mr/hadoop32/docker-compose.yaml | 8 +-
.../dist/src/main/compose/ozone-om-ha/.env | 2 +-
.../main/compose/ozone-om-ha/docker-compose.yaml | 10 +-
.../src/main/compose/ozone-om-ha/docker-config | 45 --
.../dist/src/main/compose/ozone-recon/.env | 2 +-
.../main/compose/ozone-recon/docker-compose.yaml | 8 +-
.../src/main/compose/ozone-recon/docker-config | 47 +-
.../dist/src/main/compose/ozone-topology/.env | 2 +-
.../compose/ozone-topology/docker-compose.yaml | 12 +-
.../src/main/compose/ozone-topology/docker-config | 49 --
hadoop-ozone/dist/src/main/compose/ozone/.env | 2 +-
.../src/main/compose/ozone/docker-compose.yaml | 6 +-
.../dist/src/main/compose/ozone/docker-config | 45 --
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 2 +
.../dist/src/main/compose/ozoneblockade/.env | 2 +-
.../main/compose/ozoneblockade/docker-compose.yaml | 8 +-
.../src/main/compose/ozoneblockade/docker-config | 45 --
hadoop-ozone/dist/src/main/compose/ozoneperf/.env | 2 +-
.../src/main/compose/ozoneperf/docker-compose.yaml | 10 +-
.../dist/src/main/compose/ozoneperf/docker-config | 13 -
.../dist/src/main/compose/ozones3-haproxy/.env | 2 +-
.../compose/ozones3-haproxy/docker-compose.yaml | 12 +-
.../src/main/compose/ozones3-haproxy/docker-config | 48 --
hadoop-ozone/dist/src/main/compose/ozones3/.env | 2 +-
.../src/main/compose/ozones3/docker-compose.yaml | 8 +-
.../dist/src/main/compose/ozones3/docker-config | 48 --
.../dist/src/main/compose/ozonescripts/.env | 2 +-
.../src/main/compose/ozonescripts/docker-config | 7 +-
.../dist/src/main/compose/ozonesecure-mr/.env | 2 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 50 +-
.../src/main/compose/ozonesecure-mr/docker-config | 77 +--
.../{ozone-mr/hadoop32 => ozonesecure-mr}/test.sh | 12 +-
.../dist/src/main/compose/ozonesecure/.env | 2 +-
.../main/compose/ozonesecure/docker-compose.yaml | 10 +-
.../src/main/compose/ozonesecure/docker-config | 53 --
.../dist/src/main/compose/ozonesecure/test.sh | 2 +
hadoop-ozone/dist/src/main/compose/test-all.sh | 4 +-
hadoop-ozone/dist/src/main/compose/test-single.sh | 2 +
hadoop-ozone/dist/src/main/compose/testlib.sh | 27 +-
hadoop-ozone/dist/src/main/dockerbin/entrypoint.sh | 2 +-
.../dist/src/main/smoketest/gdpr/gdpr.robot | 89 ++++
.../smoketest/{kinit.robot => kinit-hadoop.robot} | 2 +-
hadoop-ozone/dist/src/main/smoketest/kinit.robot | 5 +-
.../dist/src/main/smoketest/mapreduce.robot | 2 +-
.../{s3/__init__.robot => scmcli/pipeline.robot} | 13 +-
.../src/test/blockade/ozone/cluster.py | 4 +-
hadoop-ozone/insight/pom.xml | 9 +-
.../ozone/insight/BaseInsightSubCommand.java | 7 +-
.../scm/ScmProtocolBlockLocationInsight.java | 6 +-
...va => ScmProtocolContainerLocationInsight.java} | 18 +-
...nsight.java => ScmProtocolDatanodeInsight.java} | 27 +-
...nsight.java => ScmProtocolSecurityInsight.java} | 18 +-
.../metrics/TestSCMContainerManagerMetrics.java | 167 ++++++
.../scm/pipeline/TestRatisPipelineProvider.java | 26 +-
.../hadoop/ozone/TestSecureOzoneCluster.java | 13 +
.../rpc/TestContainerStateMachineFailures.java | 65 +++
.../client/rpc/TestOzoneRpcClientAbstract.java | 76 ++-
.../ozone/container/ContainerTestHelper.java | 11 +-
.../container/common/TestBlockDeletingService.java | 24 +-
.../common/impl/TestContainerPersistence.java | 53 +-
.../hadoop/ozone/dn/scrubber/TestDataScrubber.java | 7 +-
.../hadoop/ozone/om/TestOMDbCheckpointServlet.java | 4 -
.../ozone/om/TestOzoneManagerConfiguration.java | 9 +-
.../ozone/om/TestOzoneManagerRocksDBLogging.java | 97 ++++
hadoop-ozone/ozone-manager/pom.xml | 5 +-
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 19 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 106 ++--
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 59 +--
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 10 -
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 238 ++++++---
.../hadoop/ozone/om/OpenKeyCleanupService.java | 4 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 242 ++-------
.../apache/hadoop/ozone/om/PrefixManagerImpl.java | 11 +-
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 63 ++-
.../apache/hadoop/ozone/om/ha/OMHANodeDetails.java | 306 +++++++++++
.../hadoop/ozone/om/{ => ha}/OMNodeDetails.java | 2 +-
.../apache/hadoop/ozone/om/ha/package-info.java | 4 +-
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 8 +-
.../ozone/om/ratis/OzoneManagerRatisClient.java | 53 +-
.../ozone/om/ratis/OzoneManagerRatisServer.java | 8 +-
.../om/request/bucket/OMBucketCreateRequest.java | 12 +-
.../om/request/bucket/OMBucketDeleteRequest.java | 16 +-
.../request/bucket/OMBucketSetPropertyRequest.java | 13 +-
.../om/request/bucket/acl/OMBucketAclRequest.java | 6 +-
.../request/bucket/acl/OMBucketSetAclRequest.java | 4 +-
.../om/request/file/OMDirectoryCreateRequest.java | 4 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 4 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 4 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 4 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 7 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 4 +-
.../ozone/om/request/key/acl/OMKeyAclRequest.java | 6 +-
.../request/key/acl/prefix/OMPrefixAclRequest.java | 4 +-
.../request/s3/bucket/S3BucketCreateRequest.java | 30 +-
.../request/s3/bucket/S3BucketDeleteRequest.java | 13 +-
.../S3InitiateMultipartUploadRequest.java | 4 +-
.../multipart/S3MultipartUploadAbortRequest.java | 9 +-
.../S3MultipartUploadCommitPartRequest.java | 8 +-
.../S3MultipartUploadCompleteRequest.java | 4 +-
.../om/request/s3/security/S3GetSecretRequest.java | 6 +-
.../om/request/volume/OMVolumeCreateRequest.java | 22 +-
.../om/request/volume/OMVolumeDeleteRequest.java | 15 +-
.../ozone/om/request/volume/OMVolumeRequest.java | 32 +-
.../om/request/volume/OMVolumeSetOwnerRequest.java | 16 +-
.../om/request/volume/OMVolumeSetQuotaRequest.java | 6 +-
.../om/request/volume/acl/OMVolumeAclRequest.java | 4 +-
.../request/volume/acl/OMVolumeSetAclRequest.java | 6 +-
.../om/response/bucket/OMBucketCreateResponse.java | 8 +-
.../om/response/bucket/OMBucketDeleteResponse.java | 4 +-
.../bucket/OMBucketSetPropertyResponse.java | 7 +-
.../response/file/OMDirectoryCreateResponse.java | 3 +-
.../om/response/file/OMFileCreateResponse.java | 3 +-
.../om/response/key/OMAllocateBlockResponse.java | 6 +-
.../ozone/om/response/key/OMKeyCommitResponse.java | 7 +-
.../ozone/om/response/key/OMKeyCreateResponse.java | 3 +-
.../ozone/om/response/key/OMKeyDeleteResponse.java | 32 +-
.../ozone/om/response/key/OMKeyPurgeResponse.java | 4 +-
.../ozone/om/response/key/OMKeyRenameResponse.java | 6 +-
.../multipart/S3MultipartUploadAbortResponse.java | 21 +-
.../S3MultipartUploadCommitPartResponse.java | 45 +-
.../S3MultipartUploadCompleteResponse.java | 3 +-
.../om/response/volume/OMVolumeAclOpResponse.java | 3 +-
.../om/response/volume/OMVolumeCreateResponse.java | 13 +-
.../om/response/volume/OMVolumeDeleteResponse.java | 10 +-
.../response/volume/OMVolumeSetOwnerResponse.java | 12 +-
.../response/volume/OMVolumeSetQuotaResponse.java | 4 +-
.../om/snapshot/OzoneManagerSnapshotProvider.java | 2 +-
.../OzoneManagerHARequestHandlerImpl.java | 4 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 45 +-
.../protocolPB/OzoneManagerRequestHandler.java | 4 +-
.../ozone/security/acl/OzoneNativeAuthorizer.java | 8 +-
.../hadoop/ozone/web/ozShell/ObjectPrinter.java | 3 +-
.../org/apache/hadoop/ozone/web/ozShell/Shell.java | 3 +-
.../web/ozShell/bucket/AddAclBucketHandler.java | 5 +-
.../web/ozShell/bucket/GetAclBucketHandler.java | 4 +-
.../web/ozShell/bucket/RemoveAclBucketHandler.java | 7 +-
.../web/ozShell/bucket/SetAclBucketHandler.java | 5 +-
.../ozone/web/ozShell/keys/AddAclKeyHandler.java | 5 +-
.../ozone/web/ozShell/keys/GetAclKeyHandler.java | 4 +-
.../web/ozShell/keys/RemoveAclKeyHandler.java | 7 +-
.../ozone/web/ozShell/keys/SetAclKeyHandler.java | 5 +-
.../ozone/web/ozShell/token/GetTokenHandler.java | 2 +-
.../ozone/web/ozShell/token/PrintTokenHandler.java | 2 +-
.../web/ozShell/volume/AddAclVolumeHandler.java | 5 +-
.../web/ozShell/volume/GetAclVolumeHandler.java | 4 +-
.../web/ozShell/volume/RemoveAclVolumeHandler.java | 7 +-
.../web/ozShell/volume/SetAclVolumeHandler.java | 5 +-
.../hadoop/ozone/om/TestKeyDeletingService.java | 3 +
.../hadoop/ozone/om/TestOmMetadataManager.java | 417 +++++++++++++++
...TestOzoneManagerDoubleBufferWithOMResponse.java | 21 +-
.../om/ratis/TestOzoneManagerRatisServer.java | 2 +-
.../ozone/om/request/TestOMRequestUtils.java | 85 +++-
.../hadoop/ozone/om/request}/package-info.java | 4 +-
.../request/volume/TestOMVolumeCreateRequest.java | 16 +-
.../volume/TestOMVolumeSetOwnerRequest.java | 4 +-
.../ozone/om/response/TestOMResponseUtils.java | 8 +-
.../om/response/key/TestOMKeyDeleteResponse.java | 20 +-
.../hadoop/ozone/om/response}/package-info.java | 4 +-
.../s3/multipart/TestS3MultipartResponse.java | 3 +-
.../TestS3MultipartUploadAbortResponse.java | 19 +-
.../volume/TestOMVolumeCreateResponse.java | 7 +-
.../volume/TestOMVolumeDeleteResponse.java | 11 +-
.../volume/TestOMVolumeSetOwnerResponse.java | 17 +-
.../ozone/om/response/volume}/package-info.java | 4 +-
.../TestOzoneDelegationTokenSecretManager.java | 29 +-
hadoop-ozone/ozonefs-lib-current/pom.xml | 7 +-
hadoop-ozone/ozonefs-lib-legacy/pom.xml | 4 +-
hadoop-ozone/ozonefs/pom.xml | 9 +-
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 59 +--
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 4 +-
.../apache/hadoop/fs/ozone/OzoneFileSystem.java | 2 +-
hadoop-ozone/pom.xml | 15 +-
hadoop-ozone/recon/pom.xml | 4 +-
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 96 ++++
.../recon/recovery/ReconOmMetadataManagerImpl.java | 21 +-
.../spi/impl/ContainerDBServiceProviderImpl.java | 28 +-
.../spi/impl/OzoneManagerServiceProviderImpl.java | 33 +-
.../recon/spi/impl/ReconContainerDBProvider.java | 32 +-
.../ozone/recon/AbstractOMMetadataManagerTest.java | 2 +-
.../apache/hadoop/ozone/recon/TestReconUtils.java | 75 ++-
.../recovery/TestReconOmMetadataManagerImpl.java | 133 +++--
.../impl/TestOzoneManagerServiceProviderImpl.java | 35 +-
.../spi/impl/TestReconContainerDBProvider.java | 13 -
hadoop-ozone/s3gateway/pom.xml | 5 +-
.../apache/hadoop/ozone/s3/AWSV4AuthParser.java | 10 +-
.../hadoop/ozone/s3/OzoneClientProducer.java | 16 +-
.../hadoop/ozone/s3/OzoneServiceProvider.java | 50 +-
.../ozone/s3/exception/OS3ExceptionMapper.java | 4 +-
.../apache/hadoop/ozone/s3/util/OzoneS3Util.java | 44 ++
.../hadoop/ozone/s3/util/TestOzoneS3Util.java | 130 +++++
hadoop-ozone/tools/pom.xml | 9 +-
.../hadoop/ozone/freon/BaseFreonGenerator.java | 3 +-
.../org/apache/hadoop/ozone/scm/cli/SQLCLI.java | 4 +-
.../services/org.apache.hadoop.fs.FileSystem | 0
hadoop-ozone/upgrade/pom.xml | 5 +-
pom.ozone.xml => pom.xml | 29 +-
402 files changed, 9350 insertions(+), 3963 deletions(-)
create mode 100644 BUILDING.txt
copy hadoop-ozone/dist/src/main/license/src/LICENSE.txt => LICENSE.txt (82%)
copy hadoop-ozone/dist/src/main/license/src/NOTICE.txt => NOTICE.txt (91%)
create mode 100644 README.txt
copy hadoop-ozone/common/src/main/shellprofile.d/hadoop-ozone.sh =>
dev-support/bin/qbt (83%)
mode change 100644 => 100755
copy hadoop-ozone/common/src/main/shellprofile.d/hadoop-ozone.sh =>
dev-support/bin/smart-apply-patch (84%)
mode change 100644 => 100755
copy hadoop-ozone/common/src/main/shellprofile.d/hadoop-ozone.sh =>
dev-support/bin/test-patch (83%)
mode change 100644 => 100755
create mode 100755 dev-support/bin/yetus-wrapper
copy {hadoop-hdds/framework => dev-support/byteman}/README.md (63%)
create mode 100644 dev-support/byteman/hadooprpc.btm
copy
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/BooleanBiFunction.java
=>
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/function/FunctionWithServiceException.java
(67%)
copy hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/{cli =>
function}/package-info.java (88%)
delete mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocolPB/SCMSecurityProtocolServerSideTranslatorPB.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/ContainerCommandRequestMessage.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ByteStringConversion.java
delete mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ByteStringHelper.java
copy
hadoop-hdds/{container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/RatisServerConfiguration.java
=>
common/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDBConfiguration.java}
(51%)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumByteBuffer.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/PureJavaCrc32ByteBuffer.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/PureJavaCrc32CByteBuffer.java
delete mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/StorageContainerLocationProtocolServerSideTranslatorPB.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/ratis/TestContainerCommandRequestMessage.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/common/TestChecksumByteBuffer.java
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/OzoneProtocolMessageDispatcher.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/metrics/SCMContainerManagerMetrics.java
copy hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/{ =>
container/metrics}/package-info.java (87%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/SCMSecurityProtocolServerSideTranslatorPB.java
rename hadoop-hdds/{common/src/main/java/org/apache/hadoop/ozone/protocolPB =>
server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol}/ScmBlockLocationProtocolServerSideTranslatorPB.java
(85%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocolServerSideTranslatorPB.java
copy hadoop-hdds/{tools/src/main/java/org/apache/hadoop/hdds/scm/cli =>
server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol}/package-info.java
(88%)
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/{ReplicationManagerCommands.java
=> container/ContainerCommands.java} (73%)
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/{ReplicationManagerCommands.java
=> pipeline/PipelineCommands.java} (71%)
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/RepeatedOmKeyInfoCodec.java
rename
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/codec/{VolumeListCodec.java
=> UserVolumeInfoCodec.java} (78%)
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/RepeatedOmKeyInfo.java
copy hadoop-ozone/dist/src/main/compose/{ozone-mr/hadoop32 =>
ozonesecure-mr}/test.sh (83%)
create mode 100644 hadoop-ozone/dist/src/main/smoketest/gdpr/gdpr.robot
copy hadoop-ozone/dist/src/main/smoketest/{kinit.robot => kinit-hadoop.robot}
(94%)
copy hadoop-ozone/dist/src/main/smoketest/{s3/__init__.robot =>
scmcli/pipeline.robot} (74%)
copy
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/scm/{ScmProtocolBlockLocationInsight.java
=> ScmProtocolContainerLocationInsight.java} (73%)
copy
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/scm/{ScmProtocolBlockLocationInsight.java
=> ScmProtocolDatanodeInsight.java} (67%)
copy
hadoop-ozone/insight/src/main/java/org/apache/hadoop/ozone/insight/scm/{ScmProtocolBlockLocationInsight.java
=> ScmProtocolSecurityInsight.java} (75%)
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/metrics/TestSCMContainerManagerMetrics.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerRocksDBLogging.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ha/OMHANodeDetails.java
rename hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/{
=> ha}/OMNodeDetails.java (99%)
copy hadoop-ozone/{common =>
ozone-manager}/src/main/java/org/apache/hadoop/ozone/om/ha/package-info.java
(93%)
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java
copy hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/om
=>
ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request}/package-info.java
(92%)
copy {hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node =>
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response}/package-info.java
(92%)
copy hadoop-ozone/{common/src/test/java/org/apache/hadoop/ozone/om/lock =>
ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/volume}/package-info.java
(91%)
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/util/TestOzoneS3Util.java
copy hadoop-ozone/{ozonefs/src/test =>
tools/src/main}/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
(100%)
rename pom.ozone.xml => pom.xml (99%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
