bneradt opened a new issue, #13784: URL: https://github.com/apache/trafficserver/issues/13784
When a TLS origin closes without `close_notify`, ATS built with OpenSSL 4 can leave an HTTP/1.1 client waiting on an unfinished chunked response instead of promptly terminating the incomplete response. This was discovered while investigating the Fedora 44 → Fedora 45 CI migration and the fixture changes in #13763. Those changes explicitly frame empty fixture responses; they do not fix this core error-handling defect. ## Expected and actual behavior For an origin response whose completion depends on TLS closure, an unexpected TLS EOF should promptly fail the incomplete response and close the downstream connection as appropriate. An origin chunked response missing its terminal chunk should likewise fail promptly. ATS should not mark these reads successful or return the downstream connection to keep-alive with an unfinished response. Observed with OpenSSL 4: ATS receives `VC_EVENT_ERROR`, finishes the tunnel, and returns the downstream connection to keep-alive without completing the response. With `curl --max-time 4`, the client waits approximately four seconds and exits 28. That deadline is client-side; it is not a prompt rejection by ATS. ## Reproducer Use an ATS build linked to OpenSSL 4. Configure an HTTP/1.1 client listener on port 8080, disable the HTTP cache for this test (`proxy.config.http.cache.http: 0`), and disable origin certificate verification for the self-signed local test server (`proxy.config.ssl.client.verify.server.policy: DISABLED`). Add this remap: ```text map http://eof.example/ https://127.0.0.1:8443/ ``` Create a temporary origin certificate: ```sh openssl req -x509 -newkey rsa:2048 -nodes -days 1 \ -subj /CN=localhost -keyout origin.key -out origin.pem ``` Save and run this as `origin.py`. The default case has no HTTP body length. Pass `chunked` to reproduce an origin response missing its final chunk, or `length` for the complete-response control. The server intentionally closes without calling `unwrap()`/sending `close_notify`. ```python import socket import ssl import sys import time case = sys.argv[1] if len(sys.argv) > 1 else 'unframed' ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) ctx.load_cert_chain('origin.pem', 'origin.key') ctx.minimum_version = ctx.maximum_version = ssl.TLSVersion.TLSv1_3 header = b'HTTP/1.1 200 OK\r\nConnection: close\r\n' if case == 'chunked': response = header + b'Transfer-Encoding: chunked\r\n\r\n5\r\nhello\r\n' elif case == 'length': response = header + b'Content-Length: 5\r\n\r\nhello' else: response = header + b'\r\nhello' with socket.socket() as listener: listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) listener.bind(('127.0.0.1', 8443)) listener.listen(5) print('origin ready', flush=True) while True: raw, _ = listener.accept() try: with ctx.wrap_socket(raw, server_side=True) as conn: request = b'' while b'\r\n\r\n' not in request: data = conn.recv(4096) if not data: break request += data if request: conn.sendall(response) time.sleep(0.15) except (ssl.SSLError, ConnectionError): raw.close() ``` ```sh python3 origin.py # In another terminal, with ATS running: curl --http1.1 --noproxy '*' --max-time 4 -v \ -H 'Host: eof.example' http://127.0.0.1:8080/ ``` The unframed and truncated-chunked cases time out with OpenSSL 4. Restart the origin with `python3 origin.py length`: that case completes successfully. TLS 1.2, immediate close, and empty unframed bodies also reproduce the defect in the controlled matrix below. ## Controlled comparison Built identical ATS source with identical GCC 16/CMake settings in the same native ARM Fedora 45 container, selecting OpenSSL 3.5.7 or OpenSSL 4.0.2 headers/libraries. The Python 3.15/OpenSSL 4 origin and curl 8.21 were held constant. The ATS source was based on `2fa4963926a7a7deb2987fcbfa3b5c1e2d400872` with the Fedora 45 compatibility patches, including #13763. Actual process library mappings confirmed each build's own ATS library and the selected OpenSSL version. Each build exercised 40 cases: TLS 1.2 and 1.3, immediate close or a 150 ms delay, and these framing/shutdown combinations: | Origin response / shutdown | Cases per build | OpenSSL 3.5.7 | OpenSSL 4.0.2 | | --- | ---: | --- | --- | | No length, empty/nonempty body, `close_notify` | 8 | Completes | Completes | | Complete Content-Length or complete chunked body, abrupt TLS close | 12 | Completes | Completes | | No length, empty/nonempty body, abrupt TLS close | 8 | Accepted as complete | Client timeout | | Truncated Content-Length, clean/abrupt TLS close | 8 | Fails promptly | Fails promptly | | Chunked body missing final chunk, abrupt TLS close | 4 | Fails promptly | Client timeout | Repeating both runs without the SSL tracing observer produced identical curl outcomes and response bodies: 160 ATS exchanges in total. All valid-completion controls passed. OpenSSL 3's acceptance of the unframed abrupt-close cases is not proof of protocol correctness. [RFC 9112 §9.8](https://www.rfc-editor.org/rfc/rfc9112.html#section-9.8) distinguishes complete framed responses from responses requiring a proper TLS close. ## Why OpenSSL 4 exposes this ATS uses [`BIO_s_fastopen()`](https://github.com/apache/trafficserver/blob/2fa4963926a7a7deb2987fcbfa3b5c1e2d400872/src/iocore/net/BIO_fastopen.cc) for outbound TLS even when Fast Open is disabled. Its read callback returns zero at socket EOF but does not set `BIO_FLAGS_IN_EOF`; its delegated socket control callback therefore does not report EOF. OpenSSL 3.5.7 reports `SSL_ERROR_SYSCALL` with an empty error queue and errno zero in the isolated ATS-style BIO probe. ATS maps this to end-of-stream. OpenSSL 4's [legacy read adapter](https://github.com/openssl/openssl/blob/openssl-4.0.2/crypto/bio/bio_meth.c) automatically records EOF when that callback returns zero. The TLS layer consequently reports unexpected EOF, and [`SSL_get_error()`](https://docs.openssl.org/4.0/man3/SSL_get_error/#history) retains its connection error classification after the error queue is cleared. ATS delivers `VC_EVENT_ERROR` instead of `VC_EVENT_EOS`. An additional 144 isolated cases, including standard and ATS-style BIOs, reproduced these differences. ## ATS path to investigate - [`HttpTunnel::producer_handler_dechunked()`](https://github.com/apache/trafficserver/blob/2fa4963926a7a7deb2987fcbfa3b5c1e2d400872/src/proxy/http/HttpTunnel.cc) handles EOS/completion when generating downstream chunks, but skips `VC_EVENT_ERROR`. - `producer_handler_chunked()` also skips `VC_EVENT_ERROR`, so a missing terminal origin chunk need not set its truncation flag. - [`HttpSM::tunnel_handler_server()`](https://github.com/apache/trafficserver/blob/2fa4963926a7a7deb2987fcbfa3b5c1e2d400872/src/proxy/http/HttpSM.cc) processes the error through the EOS/truncation path. For an unknown-length response, `is_http_server_eos_truncation()` returns false. The handler sets `p->read_success = true` and calls `tunnel.local_finish_all(p)`. Local traces show the error, “finishing HTTP tunnel”, transaction teardown, and the downstream connection being returned to keep-alive. The connection closes only when the waiting curl client reaches its deadline. A fix should preserve prompt failure for TLS read errors, propagate truncation for incomplete chunked origins, and close/release client connections correctly. Regression tests should assert prompt failure, not merely a nonzero exit after the client's timeout, while retaining the valid framed and clean-shutdown controls. Related: #9880 concerns sending `close_notify` from the `forward_route` plugin; #13777 concerns a dropped HTTP/2 END_STREAM completion while a producer is throttled. This reproducer uses HTTP/1.1 to the TLS origin and receives an error event, so its triggering path differs from both. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
