Haonan Hou created RATIS-2638:
---------------------------------
Summary: Add a TLS handshake failure listener for gRPC transports
Key: RATIS-2638
URL: https://issues.apache.org/jira/browse/RATIS-2638
Project: Ratis
Issue Type: Improvement
Components: gRPC
Reporter: Haonan Hou
h3. Motivation
Applications embedding Ratis currently cannot reliably observe a gRPC TLS
handshake failure.
The failure is usually exposed later through an RPC as a
StatusRuntimeException. This does not reliably provide:
* the original TLS handshake exception;
* the local socket address;
* the remote socket address;
* whether the failure occurred on an inbound or outbound connection.
This information is needed by applications that audit failures to establish
trusted communication channels.
CC:2022 Part 2, section 18.2.4 (FTP_ITC.1), identifies the following as
minimum-level auditable information when FAU_GEN is included:
* failure of trusted channel functions;
* identification of the initiator and target of failed trusted channel
functions.
Reference:
https://www.commoncriteriaportal.org/files/ccfiles/CC2022PART2R1.pdf
For an inbound connection, the remote endpoint is the initiator and the local
endpoint is the target. For an outbound connection, the local endpoint is the
initiator and the remote endpoint is the target.
Ratis itself does not need to implement application-specific audit logging.
It only needs to expose enough transport information for an embedding
application to generate the audit event.
h3. Current limitations
ServerInterceptor and ClientInterceptor cannot observe this failure because a
TLS handshake may fail before an RPC is created.
ServerTransportFilter also does not provide the original TLS handshake
failure in a form suitable for this use case.
Inferring TLS failure from StatusRuntimeException messages is unreliable and
does not provide the server-side remote socket address.
h3. Proposed API
Ratis could provide an optional listener for TLS handshake failures.
For example:
{code:java}
@FunctionalInterface
public interface TlsHandshakeFailureListener {
void onFailure(TlsHandshakeFailureEvent event);
}
{code}
{code:java}
public interface TlsHandshakeFailureEvent {
Throwable getCause();
SocketAddress getLocalAddress();
SocketAddress getRemoteAddress();
boolean isInbound();
}
{code}
The exact API shape is open for discussion.
Since this is specific to the gRPC transport, the listener and event could be
placed in the ratis-grpc module. The listener could be registered through
Parameters or GrpcConfigKeys instead of being stored with certificate and key
material in GrpcTlsConfig.
h3. Required semantics
The notification should have the following behavior:
* Report only an initial TLS handshake failure.
* Preserve the original Throwable and its cause chain.
* Provide addresses from the actual physical connection.
* Report at most once per physical connection.
* A new failed reconnect may generate another event.
* Do not report RPC failures, RPC timeouts, connection refusal, HTTP/2 errors,
normal channel close, or successful TLS handshakes.
* An exception thrown by the listener must not replace the TLS exception or
change the channel closing behavior.
* The listener execution context should be documented.
* If invoked on a Netty event-loop thread, the listener must perform only
non-blocking work, such as enqueueing the event for another thread.
* No private key, password, trust store content, or other credential material
should be exposed.
* Existing behavior should remain unchanged when no listener is configured.
h3. Connection paths
The notification should cover TLS handshakes for all Ratis gRPC connection
paths, including:
* SERVER, CLIENT and ADMIN server listeners, whether separate or sharing the
same port;
* RaftClient connections;
* GrpcServerProtocolClient peer connections;
* heartbeat connections;
* GrpcStubPool connections.
The event does not need to identify SERVER, CLIENT or ADMIN because multiple
services may share one physical listener or ManagedChannel, and the TLS
handshake may fail before a specific RPC service is selected.
The event also does not need to provide a RaftPeerId. On an inbound connection
that fails during TLS negotiation, the peer identity may not yet be known
reliably.
h3. Implementation considerations
The preferred failure signal is the Netty SslHandler handshake future:
{code:java}
sslHandler.handshakeFuture().addListener(future -> {
if (!future.isSuccess()) {
Throwable cause = future.cause();
SocketAddress localAddress = channel.localAddress();
SocketAddress remoteAddress = channel.remoteAddress();
// Notify the application listener.
}
});
{code}
The listener should be attached to the SslHandler created for each physical
connection. This ensures that a shared SslContext still produces
connection-specific local and remote addresses.
A handshake future may also fail because the channel was closed before the
handshake completed. To avoid false reports, the implementation should verify
that the cause chain contains an SSLException. This includes errors such as
SSLHandshakeException and NotSslRecordException. A ClosedChannelException by
itself should not be reported as a TLS handshake failure.
A Ratis-only implementation may be possible by decorating the shaded gRPC
protocol negotiator and observing the SslHandler added to each Netty pipeline.
This would avoid changing ratis-thirdparty, although it may depend on shaded
gRPC internal APIs.
If using those internal APIs is considered too fragile, a stable handshake
listener extension in ratis-thirdparty could be considered instead.
A Ratis-only implementation would be preferred if it can preserve the
existing TLS executor and protocol negotiation behavior.
h3. Compatibility
The feature should be opt-in.
When no listener is configured:
* existing GrpcTlsConfig constructors should remain compatible;
* existing server and channel creation should remain unchanged;
* no audit-specific dependencies should be introduced;
* there should be no additional application-visible behavior.
h3. Suggested tests
The tests should use real gRPC/Netty TLS handshakes and cover:
* successful TLS handshake does not invoke the listener;
* client does not trust the server certificate;
* mTLS server does not trust the client certificate;
* mTLS client does not provide a certificate;
* plaintext client connects to a TLS server port;
* listener throws an exception;
* local and remote addresses are correct;
* inbound and outbound direction is correct;
* server, client, admin, heartbeat and pooled connection paths;
* reconnect reports once for each new physical connection;
* one physical connection never reports the same handshake failure twice.
I can contribute an implementation and integration tests after confirming the
preferred API and implementation approach.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
