https://bz.apache.org/bugzilla/show_bug.cgi?id=70091
Bug ID: 70091
Summary: HTTP/2 `:scheme` validation in 10.1.55 rejects
RFC-compliant translated requests from TLS-terminating
reverse proxies; need an opt-out
Product: Tomcat 10
Version: 10.1.55
Hardware: Macintosh
OS: Mac OS X 10.1
Status: NEW
Severity: major
Priority: P2
Component: Connectors
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ------
DESCRIPTION
===========
10.1.55 added (changelog, Coyote section):
"Add validation that the HTTP/2 :scheme pseudo-header is consistent with
the use (or not) of TLS. (markt)"
Implementation in java/org/apache/coyote/http2/Stream.java (10.1.55, ~line
551):
if ("https".equals(value) !=
handler.getProtocol().getHttp11Protocol().isSSLEnabled()) {
headerException = new StreamException(
sm.getString("stream.header.inconsistentScheme", getConnectionId(),
getIdAsString(),
value,
Boolean.toString(handler.getProtocol().getHttp11Protocol().isSSLEnabled())),
Http2Error.PROTOCOL_ERROR, getIdAsInt());
}
The check raises PROTOCOL_ERROR and resets the stream when the inbound :scheme
does not match the connector's SSLEnabled flag.
This breaks a very common deployment topology: a TLS-terminating reverse proxy
that forwards HTTP/2 cleartext (h2c) to the backend while preserving the
original request's :scheme: https. Concretely, this is what Cloud Foundry's
GoRouter does when a route destination has protocol: http2 -- the public hop
is HTTPS, GoRouter terminates TLS, and forwards as h2c to the container with
:scheme: https. The same pattern applies to Envoy, nginx, HAProxy, and
anything else doing edge-TLS + h2c upstream.
In our environment (SAP Java Buildpack 2.61.0, which bundled Tomcat 10.1.55)
every HTTP/2 request via GoRouter started failing immediately on upgrade from
10.1.54. Bisected to tomcat-coyote.jar 10.1.54 -> 10.1.55. Direct in-container
reproducer below.
WHY THIS IS OVER-STRICT PER RFC 9113
====================================
RFC 9113 section 8.3.1 defines :scheme:
"The :scheme pseudo-header field includes the scheme portion of the request
target. The scheme is taken from the target URI (Section 3.1 of [RFC3986])
when generating a request directly, or from the scheme of a translated
request (for example, see Section 3.3 of [HTTP/1.1])."
":scheme is not restricted to 'http' and 'https' schemed URIs. A proxy or
gateway can translate requests for non-HTTP schemes, enabling the use of
HTTP to interact with non-HTTP services."
:scheme reflects the request's target URI, not the wire's TLS state. The spec
explicitly contemplates a proxy/gateway translating requests, including
translating across schemes. There is no normative requirement that
:scheme: https imply the immediate transport is TLS -- and the most common
production deployments (TLS-offloading edge proxies forwarding h2c upstream)
rely on this distinction.
REPRODUCER (MINIMAL)
====================
Plain Tomcat 10.1.55, default <Connector> with <UpgradeProtocol
Http2Protocol/>,
no TLS:
curl --http2-prior-knowledge -H ':scheme: https' http://127.0.0.1:8080/
# curl: (55) Failed sending HTTP request
Same against 10.1.54 -> 200 OK.
End-to-end on Cloud Foundry: deploy any Java app via SJB 2.61 (Tomcat 10.1.55),
set the route destination to protocol: http2 (cf curl
/v3/routes/<guid>/destinations
-X PATCH ...), then curl https://<route>/ from outside. Hangs /
endpoint_failure
(http2: client conn could not be established) until upstream timeout. Switching
the destination back to protocol: http1 makes everything work, with 10.1.55
still in place -- confirming the trigger is the :scheme: https over
plaintext-TCP
combination, not anything else.
WHY WORKAROUNDS INSIDE THE CONNECTOR DON'T HELP
===============================================
- scheme="https" on the <Connector> does NOT affect this check. The codec
compares against connector.isSSLEnabled(), not the connector's scheme
attribute. (Verified.)
- SSLEnabled="true" would force Tomcat to attempt a real TLS handshake on the
plaintext socket and fail at connect time. Not a workaround.
- RemoteIpValve with protocolHeader="X-Forwarded-Proto" runs after
stream-header
validation and so cannot prevent the PROTOCOL_ERROR.
There is no string constant in Stream.class that suggests a
System.getProperty(...)
opt-out, and no Connector / Http2Protocol attribute controls this code path.
Operators currently have only two options: pin Tomcat to 10.1.54, or
reconfigure
their proxy/route to use HTTP/1.1 to the backend (losing h2c upstream).
PROPOSED FIX
============
Add an opt-out attribute on Http2Protocol so operators behind TLS-terminating
proxies can keep forwarding h2c with the original scheme:
<Connector port="8080" ...>
<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"
relaxedSchemeValidation="true" />
</Connector>
When set to true, Stream would skip the :scheme/SSLEnabled consistency check
(still validating that :scheme is present and a valid token, per 8.3.1's MUST).
Default false preserves current strict behavior for direct-TLS deployments.
Open to a different attribute name. The point is just to give operators a knob
they can enable when they know they're behind a TLS-terminating proxy and the
strictness is incorrect for their topology.
SEVERITY
========
Marking as major: this is a silent regression in a very common deployment
pattern (reverse-proxy + h2c upstream). The user-visible symptom is "all HTTP/2
requests fail/hang" with no obvious indicator pointing to :scheme, since the
PROTOCOL_ERROR is sent on the H2 stream and the application logs are silent.
We bisected for several days before locating it.
ENVIRONMENT IN WHICH WE OBSERVED IT
===================================
- Tomcat 10.1.55 (via SAP Java Buildpack sap_java_buildpack_jakarta_2_61)
- JRE: SapMachine 21
- Reverse proxy: Cloud Foundry GoRouter (h2 frontend, h2c upstream when
destination protocol: http2)
- Last working version: Tomcat 10.1.54 (same JRE, same proxy, same app,
same connector config)
REFERENCE: WHERE THE VALIDATION LIVES
=====================================
- java/org/apache/coyote/http2/Stream.java, ~line 551 (10.1.55 tag)
- Error message id: stream.header.inconsistentScheme
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
