[
https://issues.apache.org/jira/browse/HDDS-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-14949:
----------------------------------
Issue Type: Improvement (was: Bug)
> GrpcOmTransport should use shaded netty version from Ratis
> ----------------------------------------------------------
>
> Key: HDDS-14949
> URL: https://issues.apache.org/jira/browse/HDDS-14949
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
> Labels: pull-request-available
>
> *Problem*
> The OM gRPC path (`GrpcOzoneManagerServer` on OM and `GrpcOmTransport` in S3
> Gateway) is the only main RPC path that still uses unshaded `io.grpc` /
> `io.netty`. Every other gRPC path (datanode `XceiverServerGrpc` /
> `XceiverClientGrpc`, container replication, inter-SCM, Ratis) uses the
> ratis-thirdparty shaded copy. As a result:
> - **No native epoll.** The dist ships the unshaded
> `netty-transport-native-epoll` jar without its `linux-*` classifier, so it
> has no `.so`. Unshaded `Epoll.isAvailable()` is therefore always `false`, and
> `GrpcOzoneManagerServer` hard-codes `NioEventLoopGroup` /
> `NioServerSocketChannel` anyway. The shaded copy (`ratis-thirdparty-misc`)
> already bundles the epoll natives for `x86_64` and `aarch_64`.
> - **Two Netty/gRPC stacks per JVM in OM and S3 Gateway:** two sets of pooled
> allocators, two sets of default event-loop threads, and two tcnative loads.
> - **Dist and client classpath weight.** The dist carries the unshaded
> `grpc-*` and `netty-*` jars plus five `netty-tcnative-boringssl-static`
> classifier jars, all with matching `jar-report.txt` / `LICENSE.txt` entries.
> Applications that put the unshaded Ozone client on their classpath also pull
> unshaded `io.grpc` / `io.netty` in transitively.
> *Why moving to shaded gRPC is not a simple package rename*
> `OMRequest` / `OMResponse` are generated with unshaded `com.google.protobuf`
> (`hadoop-ozone/interface-client`, protobuf `${protobuf.version}`) and are
> shared with the Hadoop RPC path and the whole OM request pipeline. The
> `protoc-gen-grpc-java` stub `OzoneManagerServiceGrpc` builds its method
> descriptor with
> `io.grpc.protobuf.ProtoUtils.marshaller(OMRequest.getDefaultInstance())`. The
> shaded `ProtoUtils` requires a *shaded* protobuf `Message`, so rewriting
> `io.grpc` to the shaded prefix in the generated code (as
> `hadoop-hdds/interface-client` does) does not compile. Moving the OM protos
> to shaded protobuf is out of scope.
> *Proposed change*
> Replace the generated gRPC stub with a small hand-written binding that uses
> shaded gRPC and keeps the unshaded `OMRequest` / `OMResponse` messages.
> 1. **Method descriptor and marshaller** (new class in `hadoop-ozone/common`,
> e.g. `OzoneManagerServiceGrpcBinding`):
> - A shaded `MethodDescriptor<OMRequest, OMResponse>` of type `UNARY` with
> full method name **`hadoop.ozone.OzoneManagerService/submitRequest`**. This
> must match the name the generated stub uses today, so the wire format is
> unchanged.
> - `Marshaller<T>` for both messages:
> - `stream(T)`: returns an `InputStream` that also implements shaded
> `io.grpc.KnownLength`, so gRPC can frame the message without buffering it
> first. Write the message with `writeTo` / `toByteArray`.
> - `parse(InputStream)`: `OMRequest.parser().parseFrom(...)` /
> `OMResponse.parser().parseFrom(...)` on the unshaded messages.
> - A `ServerServiceDefinition` builder for the server side, using
> `ServerCalls.asyncUnaryCall`, and a blocking call helper for the client side,
> using `ClientCalls.blockingUnaryCall`.
> 2. **OM server:**
> - `OzoneManagerServiceGrpc` (ozone-manager) implements the handler against
> the new binding instead of extending the generated
> `OzoneManagerServiceImplBase`.
> - `GrpcOzoneManagerServer` switches to shaded `NettyServerBuilder`, picks
> `EpollEventLoopGroup` / `EpollServerSocketChannel` when `Epoll.isAvailable()`
> is true (falling back to NIO, mirroring `XceiverServerGrpc`), and logs the
> selected channel type.
> - TLS uses the shaded `GrpcSslContexts` with
> `SecurityConfig.getGrpcSslProvider()`, as on the datanode.
> 3. **Client:** `GrpcOmTransport` and `GrpcOMFailoverProxyProvider` move to
> shaded `NettyChannelBuilder` / `ManagedChannel` / `Status`, call through the
> binding, and use shaded TLS classes.
> 4. **Interceptors:** move `ClientAddressServerInterceptor`,
> `ClientAddressClientInterceptor` and `GrpcClientConstants` (ozone-common),
> and `GrpcMetricsServer\{Request,Response}Interceptor` and
> `GrpcMetricsServerTransportFilter` (hdds-framework), to shaded `io.grpc`.
> 5. **Build:**
> - Drop the `grpc-java` plugin execution from `hadoop-ozone/interface-client`
> so the unshaded `OzoneManagerServiceGrpc` stops being generated. Protobuf
> message generation is unchanged.
> - Remove unshaded `io.grpc` / `io.netty` dependencies from `ozone-common`,
> `ozone-manager`, `hdds-framework` and `s3gateway` where they become unused.
> 6. **Dist (follow-up or same PR):** remove the unshaded `grpc-*` / `netty-*`
> / `netty-tcnative-*` jars and their `jar-report.txt` / `LICENSE.txt` entries
> only if `dependency:tree` on the dist shows nothing else needs them at
> runtime.
> *Compatibility*
> - **Wire:** unchanged. Shading only changes Java class names, and the method
> name and message bytes stay the same. A mixed-version S3 Gateway and OM
> interoperate in both directions, and rolling upgrade is unaffected.
> - **Proto / RPC:** no `.proto` changes. Only the generated gRPC stub class is
> removed. It is internal (used only by OM and `GrpcOmTransport`).
> - **Config:** no new keys. The existing `ozone.om.grpc.*` keys (boss/worker
> group size, port, max message size) keep their meaning.
> *Risks / trade-offs*
> - **Netty and gRPC versions follow ratis-thirdparty.** Ozone pins unshaded
> Netty `4.1.136.Final`, but ratis-thirdparty ships `4.1.130.Final`, so this
> path moves back 6 patch releases. Future Netty / gRPC security fixes for OM
> gRPC then need a ratis-thirdparty release first.
> - **Netty system properties may stop applying.** Shading probably also
> relocates `io.netty.*` system-property names, so any `-Dio.netty.*` JVM flags
> operators use for OM / S3 Gateway would stop affecting this path. Verify
> before documenting.
> - **Test utilities are missing from the shaded jar.** `ratis-thirdparty-misc`
> does not ship shaded `grpc-inprocess` or `grpc-testing`.
> `TestS3GrpcOmTransport` and `TestGrpcOmTransportConcurrentFailover` use
> `InProcess\{Server,Channel}Builder` and `GrpcCleanupRule`, and need to switch
> to a shaded `NettyServerBuilder` on an ephemeral localhost port with explicit
> shutdown in `@AfterEach`.
> - **Overlap:** HDDS-16740 / HDDS-16741 touch the same `GrpcMetrics*`
> interceptors. Land those first or rebase on top of them.
> *Testing*
> - Unit tests:
> - marshaller round-trip for `OMRequest` / `OMResponse`, including a payload
> near `ozone.om.grpc.maximum.response.length`
> - the `MethodDescriptor` full name equals the old generated
> `OzoneManagerServiceGrpc.getSubmitRequestMethod().getFullMethodName()`,
> checked once against the old stub before removing it
> - Ported `TestS3GrpcOmTransport` / `TestGrpcOmTransportConcurrentFailover`;
> existing S3 Gateway integration tests and the `TestS3SDK` suite.
> - Secure compose cluster (gRPC TLS on): S3 Gateway → OM traffic works;
> mixed-version check with an old S3 Gateway against a new OM and the reverse.
> - Linux: OM log shows `EpollServerSocketChannel`; jstack shows epoll
> event-loop threads in OM and S3 Gateway and no unshaded `io.netty` threads.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
