ant5 opened a new issue, #13353: URL: https://github.com/apache/trafficserver/issues/13353
This is a hard to identify bug. From the end-user view it appeared in browser console as ``net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK)``. In ATS access log it may (*) appear as ``ERR_READ_ERROR/200`` The problematic URL sporadically changes. Site can show 0-3 errors of some URLs per page loading. When go to another page of the same site the situation repeat itself. There is a one catched problematic transaction client <->ATS assembled from the wire: # sample 1 ``` GET /wp-content/themes/epoxyworks/dist/quote-block-overlay-daFJl7Sw.svg HTTP/1.1 Host: www.epoxyworks.com Pragma: no-cache sec-ch-ua-platform: "Windows" User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 sec-ch-ua: "Google Chrome";v="149", "Chromium";v="149", "Not)A;Brand";v="24" sec-ch-ua-mobile: ?0 Accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8 Sec-Fetch-Site: same-origin Sec-Fetch-Mode: no-cors Sec-Fetch-Dest: image Referer: https://www.epoxyworks.com/ Accept-Encoding: gzip, deflate, br, zstd Accept-Language: en-US,en;q=0.9,ru;q=0.8,pl;q=0.7 Cookie: cookieyes-consent=consentid:MmxnZDMwaG5tdWJ4SVpmQVRPZjhGRzhoMXZMMUhwRWM,consent:no,action:,necessary:yes,functional:no,analytics:no,performance:no,advertisement:no; _ga=GA1.1.2040193634.1778861584; _clck=v0s5jz%5E2%5Eg62%5E0%5E2326; _ga_8ZFKGQ7DBS=GS2.1.s1781793747$o3$g1$t1781793790$j17$l0$h0 Via: 1.1 z101.orgs.link (squid) Cache-Control: no-cache Connection: close X-Forwarded-Proto: https -- HTTP/1.1 200 OK Date: Fri, 26 Jun 2026 10:15:42 GMT Content-Type: image/svg+xml Content-Length: 32578 Last-Modified: Fri, 29 May 2026 20:24:08 GMT Accept-Ranges: bytes Cache-Control: public, must-revalidate, proxy-revalidate, immutable, max-age=31536000, stale-while-revalidate=86400, stale-if-error=604800 Expires: Sat, 24 Oct 2026 10:15:41 GMT Vary: Accept-Encoding,User-Agent Content-Encoding: gzip Server: ATS/10.2.0 nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} cf-cache-status: MISS speculation-rules: "/cdn-cgi/speculation" report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=OPavTOqmnCcehb%2BByumlVeIAypgiLckUg3aRv1qg%2Bwh5nd6SKj1zthsQ30Msq87bgdws2nLBfAvOzYne0o5KkOkmEaqJMws0h3IeETdvPLKwFSsY7Yb68xuWZXWGJdqACD7dd40%3D"}]} cf-ray: a11b73c6abbb8055-ARN alt-svc: h3=":443"; ma=86400 Age: 0 Connection: Keep-Alive Via: http/1.1 traffic_server (ApacheTrafficServer/10.2.0) <svg width="1287" height="380" viewBox="0 0 1287 380" fill="none" xmlns="http://www.w3.org/2000/svg";> ... ``` Do you see the problem? Response contain "Content-Encoding: gzip" while the body is a plain svg. I assume that "Content-Length: 32578" is wrong and dedicated to gzip'ed variant of a content. I have another case but it is not so funny and more frustrating: # sample 2 ``` GET /wp-content/uploads/2017/05/IIT-Architecture-768x512.jpg.webp HTTP/1.1 Host: www.epoxyworks.com Pragma: no-cache sec-ch-ua-platform: "Windows" User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 sec-ch-ua: "Google Chrome";v="149", "Chromium";v="149", "Not)A;Brand";v="24" sec-ch-ua-mobile: ?0 Accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8 Sec-Fetch-Site: same-origin Sec-Fetch-Mode: no-cors Sec-Fetch-Dest: image Referer: https://www.epoxyworks.com/category/home-garden/architecture/ Accept-Encoding: gzip, deflate, br, zstd Accept-Language: en-US,en;q=0.9,ru;q=0.8,pl;q=0.7 Cookie: cookieyes-consent=consentid:MmxnZDMwaG5tdWJ4SVpmQVRPZjhGRzhoMXZMMUhwRWM,consent:no,action:,necessary:yes,functional:no,analytics:no,performance:no,advertisement:no; _ga=GA1.1.2040193634.1778861584; _clck=v0s5jz%5E2%5Eg62%5E0%5E2326; _ga_8ZFKGQ7DBS=GS2.1.s1781793747$o3$g1$t1781793790$j17$l0$h0; swpext86386=5fa1b333ba5fb75515384e19da19d4fc; PHPSESSID=bf7a9f0cb68ab7e1ab1dfaaf7d5b09f5 Via: 1.1 z101.orgs.link (squid) Cache-Control: no-cache Connection: close X-Forwarded-Proto: https HTTP/1.1 200 OK Date: Wed, 01 Jul 2026 14:28:56 GMT Content-Type: image/webp Content-Length: 46806 Last-Modified: Fri, 06 Jun 2025 19:44:44 GMT Accept-Ranges: bytes Cache-Control: public, must-revalidate, proxy-revalidate, immutable, max-age=31536000, stale-while-revalidate=86400, stale-if-error=604800 Expires: Thu, 29 Oct 2026 14:28:56 GMT Vary: Accept-Encoding Server: ATS/10.2.0 nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800} cf-cache-status: MISS speculation-rules: "/cdn-cgi/speculation" report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=7K0aITjElycFvh2agtTOgfruUFfXGeP9L4Bm6KZQjbjV580%2BrLaMUTcLqo9GbDU8nZw3N6JxCdO3rAAZCQUaQlradCvLn75q0BM5QlTz1tiIXvaGf6aRvMfB%2BjRu%2FreMN3QnAZ0%3D"}]} cf-ray: a146199ac9588291-ARN alt-svc: h3=":443"; ma=86400 Age: 0 Connection: Keep-Alive Via: https/2 traffic_server (ApacheTrafficServer/10.2.0) RIFF....WEBPVP8 . ... ``` Wireshark say that all this data is about 34Kb while "Content-Length: 46806" say it must be larger. This case appear in access log as ``1782916131.021 180 fd05:562e:5a23::212:2e01 ERR_READ_ERROR/200 952 GET https://www.epoxyworks.com/wp-content/uploads/2017/05/IIT-Architecture-768x512.jpg.webp - DIRECT/www.epoxyworks.com image/webp`` The ATS<->origin interaction contain no RST or something abnormal. Morever there is no TCP "dialog" for less than 44KB. So I don't now where "952" from. # sample 3 The same as sample 2. Bytes on wire is less than content-length. ``` GET /gtm.js?id=GTM-KLQLNPW8 HTTP/1.1 Host: www.googletagmanager.com Pragma: no-cache sec-ch-ua-platform: "Windows" User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 sec-ch-ua: "Google Chrome";v="149", "Chromium";v="149", "Not)A;Brand";v="24" sec-ch-ua-mobile: ?0 Accept: */* Sec-Fetch-Site: cross-site Sec-Fetch-Mode: no-cors Sec-Fetch-Dest: script Sec-Fetch-Storage-Access: active Referer: https://www.epoxyworks.com/ Accept-Encoding: gzip, deflate, br, zstd Accept-Language: en-US,en;q=0.9,ru;q=0.8,pl;q=0.7 Via: 1.1 z101.orgs.link (squid) Cache-Control: no-cache Connection: close X-Forwarded-Proto: https --- HTTP/1.1 200 OK Content-Type: application/javascript; charset=UTF-8 access-control-allow-origin: * access-control-allow-credentials: true access-control-allow-headers: Cache-Control Content-Encoding: gzip Vary: Accept-Encoding Date: Wed, 01 Jul 2026 15:52:41 GMT Expires: Wed, 01 Jul 2026 15:52:41 GMT Cache-Control: private, max-age=900 Last-Modified: Wed, 01 Jul 2026 15:00:00 GMT Strict-Transport-Security: max-age=31536000; includeSubDomains cross-origin-resource-policy: cross-origin Server: ATS/10.2.0 Content-Length: 127909 x-xss-protection: 0 alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Age: 0 Connection: Keep-Alive Via: https/2 traffic_server (ApacheTrafficServer/10.2.0) ............k[...(.._....K.....F..**...qC...4! &....o?OUw'....s.}.~....$}........,..(;..%p.^........N.*-.*(.~_..P hH.Gj....|w.Y..{y.ua....M"...........Uk......4.].Sk.../( ``` ``ERR_READ_ERROR/200`` in access log. # conclusion May be the problem is in update object in cache logic. It seems that objects in cache replaced by the object from response from origin regardless of it's Content-Encoding header. When the fresh object has arrived it replace previous object in a cache without updating Content-Length and Content-Encoding header. Another thread is in rather complex Vary header processing. In case of Vary in response and when an object from this response is cached, further cache may hit only if apropriate headers (typical Accept-Encoding and User-agent) in request is strictly matching corresponding headers of object in a cache. There may be (?) various form of object in cache that match a criteria. There may be gzip'ed svg and plain svg for the same request. So the issue may be related to building reply which contain headers from one object and body from another. May be I'm totally wrong. P.S. ATS 10.2.X, FreeBSD 14-STABLE --- * - a few weeks ago when I began investigation I don't find errors in access log related to problematic URLs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
