This is an automated email from the ASF dual-hosted git repository.
szetszwo pushed a change to branch HDDS-4454
in repository https://gitbox.apache.org/repos/asf/ozone.git
discard 74ce5363bd HDDS-5666. Add option to createKey via streaming api in
Freon (#2574)
discard 246b0a6f2d HDDS-6500. [Ozone-Streaming] Buffer the PutBlockRequest at
the end of the stream. (#3229)
discard b23fbcf9bd HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput.
(#3195)
discard eeeddff8b9 HDDS-5798. [Ozone-Streaming] Setup TlsConf parameters.
(#3207)
discard ae91a3f869 HDDS-6461. Update Ratis version to 2.3.0-da5d868-SNAPSHOT.
(#3205)
discard 0d9b3e0e27 HDDS-6388. [Ozone-Streaming] Streaming write support both
pipeline model and star model (#3145)
discard c5a9686ee6 HDDS-6355. [Ozone-Streaming] Fix CheckStyle problem (#3119)
discard b281043220 HDDS-6229. [Ozone-Streaming] Data Channel abstraction on
datanode (#3023)
discard c13128155b HDDS-6282. Fix BlockDataStreamOutput#doFlushIfNeeded NPE
(#3060)
discard 32613fba33 HDDS-5487. [Ozone-Streaming] BlockDataStreamOutput support
FlushDelay. (#3002)
discard 4fa81b239c HDDS-6298. Add XceiverServerRatis stream config (#3070)
discard 1bbc76f47a HDDS-6138.[Ozone-Streaming] Define a limit on the size of
the retry bufferList. (#2946)
discard a65a63ee7d HDDS-6281. Update ratis version to 2.3.0-94db58b-SNAPSHOT
version (#3059)
discard 59eb086e9a HDDS-6178. [Ozone-Streaming] Fix NPE in HDDS-6139. (#2984)
discard 6925126ddc HDDS-6139. [Ozone-Streaming] Fix incorrect computation of
totalAckDataLength. (#2978)
discard d9d81f5b97 HDDS-6130. [Ozone-Streaming] When releaseBuffers will get
“Couldn 't find the required future” (#2939)
discard ac9815d47c HDDS-6039. Define a minimum packet size during streaming
writes. (#2883)
discard 4771a50e6a HDDS-5851. [Ozone-Streaming] Define a PutBlock/maxBuffer
fixed boundary for streaming writes. (#2866)
discard e1d9db9899 HDDS-5743. [Ozone-Streaming] Add option to write files via
streaming api in ofs and o3fs. (#2770)
discard 22d0553467 HDDS-5879. [Ozone-Streaming] OzoneBucket add the
createMultipartStreamKey method (#2760)
discard bb41750fdd HDDS-5961. [Ozone-Streaming] update the usage space of
Containers in the stream write (#2833)
discard 2a629f82c9 HDDS-5987. [Ozone-Streaming] Add XceiverClientRatis stream
config (#2841)
discard 0f5a55a62d HDDS-5763. Provide an Executor for each LocalStream in
ContainerStateMachine (#2782)
discard 2c1365e1b4 HDDS-5895. [Ozone-Streaming] Make
raft.server.data-stream.client.pool.size configurable (#2766)
discard 5049243ac1 HDDS-5674.[Ozone-Streaming] Handle client retries on
exception (#2701)
discard 247c4f2324 HDDS-5849. [Ozone-Streaming]Write exceptions occur after
checksum is enabled (#2729)
discard 7663ee5766 HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
discard 7b713b2b5f HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
discard 3479f54b69 HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
discard 0790843af0 HDDS-5599. [Ozone-Streaming]drop BufferPool and
ChunkBuffer to avoid buffer copying (#2557)
discard 3641548d56 HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
discard 14856e8808 HDDS-5480. [Ozone-Streaming] Client and server should
support stream setup. (#2452)
discard d32ae35318 HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
discard 133557f3c0 HDDS-5452. Add link method to ContainerStateMachine for
Ratis streaming (#2422)
discard 553346e7d2 HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
add 9055a11216 HDDS-6556. Check quota while allocate and update usedBytes
when commit. (#3286)
add a9df285447 HDDS-6563. Revert "HDDS-5867. Update quota when deleting
open keys" (#3282)
add 3889373070 HDDS-6703. Disable failing testPrepareDownedOM until Ratis
upgrade (#3387)
add d9c56aec00 HDDS-6675. Avoid repeating some S3 tests for different
bucket types (#3370)
add 73e69035e0 HDDS-6659: EC: Add BlockGroupLen info as part of PutBlock
in EC Writes for helping in recovery. (#3380)
add e90a44de2f HDDS-6602. Incorrect logic in
PipelineProvider.pickNodesNotUsed (#3318)
add d2c5631d2f HDDS-6716. Adopt RATIS-1560 (#3394)
add 6e0c85eb43 HDDS-6715. Avoid using GrpcClientProtocolService.LOG (#3393)
add d4d8e3c4e9 HDDS-6666. Make Hugo markdown image syntax add responsive
image class (#3364)
add ff351025bc HDDS-6709. Fix bucket usedBytes while versioning is true.
(#3388)
add 1419799bad HDDS-6678. Use secure random in production code (#3374)
add bfc361e1b8 HDDS-6507. Spotbugs transitive dependencies may result in
NoClassDefFound error (#3236)
add a32549cbc1 HDDS-6708. [FSO] Make normalize keyPath on key rename
request (#3386)
add 88b5e854be HDDS-6400. EC: Refactor ECKeyOutputStream for better code
reuse (#3372)
add bccb8c2c83 HDDS-6679. Further splits integration tests (#3373)
add 5459b49538 HDDS-6720. Add a function to get bucket layout info in
Request Validators (#3395)
add 91757d5145 HDDS-6734. ozone admin pipeline list CLI is not backward
compatible (#3406)
add 9f0766ecf1 HDDS-6619. Add freon command to run r/w mix workload using
ObjectStore APIs (#3383)
add 6edcc13900 HDDS-6680. Pre-Finalize behaviour for Bucket Layout
Feature. (#3377)
add 86ffd94735 HDDS-6704. EC: ReplicationManager - create version of
ContainerReplicaCounts applicable to EC (#3405)
new 2f128b9ac4 HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
new 7921b79095 HDDS-5452. Add link method to ContainerStateMachine for
Ratis streaming (#2422)
new c6a242e815 HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
new 354cc3e3a1 HDDS-5480. [Ozone-Streaming] Client and server should
support stream setup. (#2452)
new 5f2d2e973c HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
new ebae76879e HDDS-5599. [Ozone-Streaming]drop BufferPool and
ChunkBuffer to avoid buffer copying (#2557)
new 6c1f954481 HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
new 170c2d929e HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
new c210ad42e4 HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
new 532d9a34a3 HDDS-5849. [Ozone-Streaming]Write exceptions occur after
checksum is enabled (#2729)
new 47bcdf7d3f HDDS-5674.[Ozone-Streaming] Handle client retries on
exception (#2701)
new d6b3b5c8f0 HDDS-5895. [Ozone-Streaming] Make
raft.server.data-stream.client.pool.size configurable (#2766)
new 888f948e8c HDDS-5763. Provide an Executor for each LocalStream in
ContainerStateMachine (#2782)
new 9c71a2522a HDDS-5987. [Ozone-Streaming] Add XceiverClientRatis stream
config (#2841)
new 628a7a4010 HDDS-5961. [Ozone-Streaming] update the usage space of
Containers in the stream write (#2833)
new be48e28991 HDDS-5879. [Ozone-Streaming] OzoneBucket add the
createMultipartStreamKey method (#2760)
new 6608f39c7a HDDS-5743. [Ozone-Streaming] Add option to write files via
streaming api in ofs and o3fs. (#2770)
new de79a783b5 HDDS-5851. [Ozone-Streaming] Define a PutBlock/maxBuffer
fixed boundary for streaming writes. (#2866)
new 40cf7e9fe7 HDDS-6039. Define a minimum packet size during streaming
writes. (#2883)
new 35ae290054 HDDS-6130. [Ozone-Streaming] When releaseBuffers will get
“Couldn 't find the required future” (#2939)
new 159985138a HDDS-6139. [Ozone-Streaming] Fix incorrect computation of
totalAckDataLength. (#2978)
new 239ed3eae3 HDDS-6178. [Ozone-Streaming] Fix NPE in HDDS-6139. (#2984)
new c7e4ad51a8 HDDS-6281. Update ratis version to 2.3.0-94db58b-SNAPSHOT
version (#3059)
new 60fedb00f0 HDDS-6138.[Ozone-Streaming] Define a limit on the size of
the retry bufferList. (#2946)
new ed6a76185c HDDS-6298. Add XceiverServerRatis stream config (#3070)
new 5932655dcc HDDS-5487. [Ozone-Streaming] BlockDataStreamOutput support
FlushDelay. (#3002)
new 07d88767bd HDDS-6282. Fix BlockDataStreamOutput#doFlushIfNeeded NPE
(#3060)
new 7a6ed81645 HDDS-6229. [Ozone-Streaming] Data Channel abstraction on
datanode (#3023)
new fac96466c8 HDDS-6355. [Ozone-Streaming] Fix CheckStyle problem (#3119)
new 4ada8072ff HDDS-6388. [Ozone-Streaming] Streaming write support both
pipeline model and star model (#3145)
new 1b559fe471 HDDS-6461. Update Ratis version to 2.3.0-da5d868-SNAPSHOT.
(#3205)
new a53f57199d HDDS-5798. [Ozone-Streaming] Setup TlsConf parameters.
(#3207)
new 8995a692c2 HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput.
(#3195)
new 1ee1a9e62f HDDS-6500. [Ozone-Streaming] Buffer the PutBlockRequest at
the end of the stream. (#3229)
new 165158cc61 HDDS-5666. Add option to createKey via streaming api in
Freon (#2574)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (74ce5363bd)
\
N -- N -- N refs/heads/HDDS-4454 (165158cc61)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 35 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/post-commit.yml | 5 +-
dev-support/annotations/pom.xml | 4 +-
.../hdds/scm/storage/ECBlockOutputStream.java | 24 ++
.../io/ECBlockReconstructedStripeInputStream.java | 1 +
.../hadoop/hdds/client/ReplicationFactor.java | 8 +-
.../hadoop/hdds/freon/FakeClusterTopology.java | 1 +
.../hadoop/hdds/scm/net/NetworkTopologyImpl.java | 1 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 3 +
hadoop-hdds/container-service/pom.xml | 2 +-
.../common/report/ContainerReportPublisher.java | 1 +
.../common/report/PipelineReportPublisher.java | 1 +
.../ozoneimpl/ContainerDataScrubberMetrics.java | 1 +
hadoop-hdds/docs/dev-support/bin/generate-site.sh | 12 +-
.../docs/dev-support/bin/make_images_responsive.py | 57 ---
.../_markup/render-image.html} | 3 +-
.../client/DefaultCertificateClient.java | 9 +-
hadoop-hdds/server-scm/pom.xml | 2 +-
.../hadoop/hdds/scm/SCMCommonPlacementPolicy.java | 5 +-
.../hdds/scm/container/ContainerManagerImpl.java | 2 +
.../scm/container/ECContainerReplicaCount.java | 313 +++++++++++++++
.../hadoop/hdds/scm/pipeline/PipelineProvider.java | 13 +-
.../algorithms/RandomPipelineChoosePolicy.java | 1 +
.../states/TestECContainerReplicaCount.java | 432 +++++++++++++++++++++
.../scm/pipeline/TestRatisPipelineProvider.java | 4 +-
.../org/apache/ozone/test/GenericTestUtils.java | 1 +
.../scm/cli/pipeline/ListPipelinesSubcommand.java | 73 +++-
.../cli/pipeline/TestListPipelinesSubCommand.java | 46 ++-
.../ozone/client/io/ECBlockOutputStreamEntry.java | 4 +-
.../hadoop/ozone/client/io/ECKeyOutputStream.java | 343 +++++++---------
.../hadoop/ozone/client/TestOzoneECClient.java | 47 +++
.../hadoop/ozone/om/helpers/BucketLayout.java | 4 +
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 5 +-
.../dist/src/main/compose/ozonesecure/test.sh | 5 +-
.../non-rolling-upgrade/1.2.1-1.3.0/callback.sh | 2 +-
.../dist/src/main/smoketest/s3/bucketcreate.robot | 1 +
.../dist/src/main/smoketest/s3/bucketdelete.robot | 1 +
.../dist/src/main/smoketest/s3/buckethead.robot | 5 +-
.../dist/src/main/smoketest/s3/webui.robot | 1 +
.../dist/src/main/smoketest/upgrade/finalize.robot | 2 +-
hadoop-ozone/insight/pom.xml | 2 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 73 +++-
.../ozone/freon/TestOmBucketReadWriteFileOps.java | 2 +-
...leOps.java => TestOmBucketReadWriteKeyOps.java} | 146 +++----
.../apache/hadoop/ozone/om/OMUpgradeTestUtils.java | 88 +++++
.../hadoop/ozone/om/TestOMBucketLayoutUpgrade.java | 272 +++++++++++++
.../hadoop/ozone/om/TestOMUpgradeFinalization.java | 52 +--
.../hadoop/ozone/om/TestOzoneManagerPrepare.java | 4 +
hadoop-ozone/ozone-manager/pom.xml | 2 +-
.../om/request/bucket/OMBucketCreateRequest.java | 29 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 4 -
.../request/file/OMFileCreateRequestWithFSO.java | 4 -
.../om/request/key/OMAllocateBlockRequest.java | 19 +-
.../request/key/OMAllocateBlockRequestWithFSO.java | 19 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 45 +--
.../om/request/key/OMKeyCommitRequestWithFSO.java | 23 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 4 -
.../om/request/key/OMKeyCreateRequestWithFSO.java | 4 -
.../ozone/om/request/key/OMKeyRenameRequest.java | 15 +-
.../om/request/key/OMOpenKeysDeleteRequest.java | 26 +-
.../S3MultipartUploadCommitPartRequest.java | 12 +-
.../S3MultipartUploadCompleteRequest.java | 4 +-
.../om/request/validation/ValidationContext.java | 24 +-
.../response/file/OMFileCreateResponseWithFSO.java | 5 -
.../om/response/key/OMAllocateBlockResponse.java | 14 +-
.../key/OMAllocateBlockResponseWithFSO.java | 10 +-
.../ozone/om/response/key/OMKeyCreateResponse.java | 5 -
.../om/response/key/OMOpenKeysDeleteResponse.java | 13 +-
.../hadoop/ozone/om/upgrade/OMLayoutFeature.java | 6 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 3 +-
.../ozone/om/request/OMRequestTestUtils.java | 16 -
.../request/key/TestOMKeyRenameRequestWithFSO.java | 99 +++++
.../request/key/TestOMOpenKeysDeleteRequest.java | 55 +--
.../request/validation/TestRequestValidations.java | 66 +++-
.../response/key/TestOMAllocateBlockResponse.java | 2 +-
.../key/TestOMAllocateBlockResponseWithFSO.java | 2 +-
.../response/key/TestOMOpenKeysDeleteResponse.java | 59 +--
hadoop-ozone/ozonefs-common/pom.xml | 2 +-
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 4 +-
hadoop-ozone/recon/pom.xml | 2 +-
hadoop-ozone/s3gateway/pom.xml | 2 +-
.../apache/hadoop/ozone/s3/RequestIdentifier.java | 8 +-
hadoop-ozone/tools/pom.xml | 2 +-
...eOps.java => AbstractOmBucketReadWriteOps.java} | 178 +++------
.../hadoop/ozone/freon/BaseFreonGenerator.java | 1 +
.../hadoop/ozone/freon/ChunkManagerDiskWrite.java | 1 +
.../hadoop/ozone/freon/ContentGenerator.java | 1 +
.../hadoop/ozone/freon/DatanodeBlockPutter.java | 1 +
.../hadoop/ozone/freon/DatanodeChunkGenerator.java | 1 +
.../freon/FollowerAppendLogEntryGenerator.java | 3 +-
.../java/org/apache/hadoop/ozone/freon/Freon.java | 3 +-
.../hadoop/ozone/freon/HadoopDirTreeGenerator.java | 1 +
.../ozone/freon/HadoopNestedDirGenerator.java | 1 +
.../ozone/freon/LeaderAppendLogEntryGenerator.java | 3 +-
.../ozone/freon/OmBucketReadWriteFileOps.java | 220 ++---------
.../ozone/freon/OmBucketReadWriteKeyOps.java | 145 +++++++
.../hadoop/ozone/freon/RandomKeyGenerator.java | 1 +
.../apache/hadoop/ozone/freon/S3KeyGenerator.java | 1 +
.../hadoop/ozone/freon/SCMThroughputBenchmark.java | 1 +
.../containergenerator/GeneratorDatanode.java | 1 +
pom.xml | 60 ++-
100 files changed, 2231 insertions(+), 1085 deletions(-)
delete mode 100644 hadoop-hdds/docs/dev-support/bin/make_images_responsive.py
rename hadoop-hdds/docs/themes/ozonedoc/layouts/{shortcodes/image.html =>
_default/_markup/render-image.html} (84%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ECContainerReplicaCount.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/states/TestECContainerReplicaCount.java
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/{TestOmBucketReadWriteFileOps.java
=> TestOmBucketReadWriteKeyOps.java} (66%)
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/OMUpgradeTestUtils.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMBucketLayoutUpgrade.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyRenameRequestWithFSO.java
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/{OmBucketReadWriteFileOps.java
=> AbstractOmBucketReadWriteOps.java} (53%)
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OmBucketReadWriteKeyOps.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
