ivandika3 commented on code in PR #11519: URL: https://github.com/apache/ozone/pull/11519#discussion_r4290477390
########## hadoop-hdds/docs/content/design/protobuf-oneof-optimization.md: ########## @@ -0,0 +1,244 @@ +--- +title: Ozone Protobuf Oneof Optimization +summary: Use oneof groups in Proto files to Optimize memory usage in Ozone. +date: 2026-10-07 +jira: HDDS-16984 +status: draft +author: Rishabh Patel +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +## Problem Statement + +Ozone uses [Protobuf](https://protobuf.dev/) as its serialization framework. As part of its API surface, Ozone supports several types of requests and responses for each of its components (OM, SCM, DN). These requests are serialized via Protobuf. The existing Protobuf definitions for the Request/Response wrappers are suboptimal. As per the wrappers, any number of requests and responses can be part of the same Proto object. In practice, each wrapper Proto object is usually used for only a single type of request or response at a time. This proposal aims to address this discrepancy and optimize the Protobuf object design for the wrappers. + +Consider the [OMRequest](https://github.com/apache/ozone/blob/0a9bd0d999aca77d3bea8246ceb6e1c04672cb81/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto#L179-L333) proto object. It is a wrapper object that encapsulates all requests to the OM. It allows over a hundred types of requests. At a time, the OM will allow only one type of request like `CreateBucketRequest` or the `CreateKeyRequest` but not both. + +```proto +message OMRequest { + required Type cmdType = 1; // Type of the command + + // A string that identifies this command, we generate Trace ID in Ozone + // frontend and this allows us to trace that command all over ozone. + optional string traceID = 2; + + required string clientId = 3; + + optional UserInfo userInfo = 4; + optional uint32 version = 5; + + optional LayoutVersion layoutVersion = 6; + + optional ReadConsistencyHint readConsistencyHint = 7; + + optional CreateVolumeRequest createVolumeRequest = 11; + optional SetVolumePropertyRequest setVolumePropertyRequest = 12; + optional CheckVolumeAccessRequest checkVolumeAccessRequest = 13; + optional InfoVolumeRequest infoVolumeRequest = 14; + optional DeleteVolumeRequest deleteVolumeRequest = 15; + optional ListVolumeRequest listVolumeRequest = 16; + .... // many more optional Request fields +} +``` + +The protoc compiler however will [generate](https://protobuf.dev/reference/java/java-generated/#fields) the Java classes with a fixed layout. For each proto field, the generated classes will contain one reference to store the tag number and one reference to the object. + +```java +// optional .hadoop.ozone.CreateVolumeRequest createVolumeRequest = 11; +public static final int CREATEVOLUMEREQUEST_FIELD_NUMBER = 11; +private org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.CreateVolumeRequest createVolumeRequest_; +``` + +This results in memory being used for all such presence fields and reference fields. With the current design, the space occupied by each wrapper class increases linearly with the number of fields in the proto message definitions. +The field exists even if it is not set. A parsed `OMRequest` carrying a `CreateBucketRequest` still has a slot for `createVolumeRequest_`, for `deleteVolumeRequest_`, and for every other request type, all holding null. + +The memory cost comes from the generated Java wrapper class, not from serialized protobuf bytes or from allocating every child payload. In the current form, each `optional FooRequest fooRequest \= N` becomes its own instance field in every wrapper object. Even when the field value is `null`, the parent object still contains a fixed reference slot for it, plus presence bookkeeping. With dozens of mutually exclusive payload fields, every parsed or built wrapper object pays for many unused reference slots. + +``` +Today: one reference slot per request type With oneof: one slot plus a tag + ++--------------------------------+ +--------------------------------+ +| OMRequest 504 B | | OMRequest 96 B | ++--------------------------------+ +--------------------------------+ +| cmdType = CreateBucket | | cmdType = CreateBucket | +| traceID, clientId, userInfo .. | | traceID, clientId, userInfo .. | +| bitField0_ .. bitField3_ | | bitField0_ | +| createVolumeRequest_ null | | bodyCase_ = 21 | +| setVolumePropertyRequest_ null | | body_ ---------------------------> CreateBucketRequest +| checkVolumeAccessRequest_ null | +--------------------------------+ +| ... | +| createBucketRequest_ ----------> CreateBucketRequest +| ... | +| ~100 more slots, all null | ++--------------------------------+ +``` + + +## Protobuf `oneof` keyword + +Protobuf provides the [`oneof`](https://protobuf.dev/programming-guides/proto2/#oneof) keyword to enforce the presence of at most a single field within a group of fields. +In other words, it makes the group mutually exclusive. Setting one field invalidates the previously set field within the group. Setting at least one field is not required though. + +`oneof` changes that object layout. Instead of generating one reference field per possible payload, protoc stores all mutually exclusive payloads in a single shared `Object` field, together with a small discriminator indicating which field is active. The selected payload is still stored normally, but the unselected alternatives no longer consume per-instance reference slots in the wrapper object. + +```proto +private int bodyCase_ = 0; +private java.lang.Object body_; +``` + +This is useful for Ozone because request and response wrappers are created frequently and can pass through RPC, Ratis, queues, caches and GC-heavy paths. The change preserves the same field numbers and wire compatibility while reducing the in-memory footprint of each wrapper. The on-wire size does not change as optional fields are by-default [not transferred](https://protobuf.dev/programming-guides/encoding/#optional) over the wire. + +## Scope + +Eight dispatcher wrapper messages across four `.proto` files: + +| Proto file | Wrapper messages | +|:-----------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------| +| `https://github.com/apache/ozone/blob/master/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto` | `OMRequest`, `OMResponse` | +| `https://github.com/apache/ozone/blob/master/hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto` | `ContainerCommandRequestProto`, `ContainerCommandResponseProto` | +| `https://github.com/apache/ozone/blob/master/hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto` | `ScmContainerLocationRequest`, `ScmContainerLocationResponse` | +| `https://github.com/apache/ozone/blob/master/hadoop-hdds/interface-server/src/main/proto/ScmServerDatanodeHeartbeatProtocol.proto` | `SCMCommandProto`, `SCMDatanodeRequest, SCMDatanodeResponse` | +| `https://github.com/apache/ozone/blob/master/hadoop-hdds/interface-server/src/main/proto/ScmServerProtocol.proto` | `SCMBlockLocationRequest, SCMBlockLocationResponse` | + +As part of this change we will rearrange all the mutually exclusive requests into a `oneof` body within the same proto message. Some requests are however not mutually exclusive or are not supported by `oneof` bodies. + +For example today a `CreateVolumeRequest` implicitly piggybacks a `CreateTenantRequest` as a volume is automatically created for each new tenant. As such, the two requests are no longer mutually exclusive and both cannot be part of the `oneof` block. + +Similarly, `GetS3SecretRequest` is paired with `UpdateGetS3SecretRequest` and `TenantAssignUserAccessIdRequest` is paired with `UpdateGetS3SecretRequest`. + +Repeated fields like `SetSnapshotPropertyRequests` are also [not allowed](https://protobuf.dev/programming-guides/proto2/#oneof) to be part of the `oneof` block. +Refactoring `SetSnapshotPropertyRequests` into its own message with a repeated list and then placing that new message into the oneof block does not yield much benefit when compared to the cost of deprecating the old field. + +The shape of the change, using `OMRequest`: + +```proto +message OMRequest { + required Type cmdType = 1; + optional string traceID = 2; + required string clientId = 3; + optional UserInfo userInfo = 4; + optional uint32 version = 5; + optional LayoutVersion layoutVersion = 6; + optional ReadConsistencyHint readConsistencyHint = 7; + + // Mutually exclusive payloads. Same field numbers as today. + oneof body { + SetVolumePropertyRequest setVolumePropertyRequest = 12; + CheckVolumeAccessRequest checkVolumeAccessRequest = 13; + // ... + CreateBucketRequest createBucketRequest = 21; + // ... + CreateKeyRequest createKeyRequest = 31; + // ... + } + + // Not mutually exclusive with the block above, so they stay flat. + optional S3Authentication s3Authentication = 95; + optional CreateVolumeRequest createVolumeRequest = 11; + optional UpdateGetS3SecretRequest updateGetS3SecretRequest = 82; + repeated SetSnapshotPropertyRequest SetSnapshotPropertyRequests = 143; +} +``` + +## Compatibility Review Comment: Can protolock check this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
