[ https://issues.apache.org/jira/browse/TS-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nikolai Gorchilov updated TS-3995: ---------------------------------- Description: Caching proxy running ATS 5.3.x (5.3.0, 5.3.1, 5.3.2 all fail) with proxy.config.http.cache.when_to_revalidate = 4 breaks akamaihd.net live streaming. The actual problem is that ATS rewrites origin response from 200 to 304, due to If-Modified-Since conditional header in client's request. As per ATS logic object is unmodified, but in fact it is. Most probably it's related to the fact that player and server somehow play with if-modified-since/last-modified headers pair to communicate position in the live stream. As result, Akamai player keeps repeating the said request, expecting it's 200, but getting 304 thus live video freezes forever, just a few seconds after start. IMHO when proxy.config.http.cache.when_to_revalidate = 4, ATS shall not interfere with origin response in this manner. Here's a debug log of request and response headers at different states in a single transaction: {noformat} +++++++++ Incoming Request +++++++++ -- State Machine Id: 168 GET http:///z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44 HTTP/1.1 Host: sshds5-lh.akamaihd.net Connection: keep-alive User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 X-Requested-With: ShockwaveFlash/19.0.0.226 Accept: */* DNT: 1 Referer: http://www.hotstar.com/ Accept-Encoding: gzip Accept-Language: en-US,en;q=0.8,bg;q=0.6 Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8 If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT +++++++++ Proxy's Request +++++++++ -- State Machine Id: 168 GET /z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44 HTTP/1.1 Host: sshds5-lh.akamaihd.net User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 X-Requested-With: ShockwaveFlash/19.0.0.226 Accept: */* DNT: 1 Referer: http://www.hotstar.com/ Accept-Encoding: gzip Accept-Language: en-US,en;q=0.8,bg;q=0.6 Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8 If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT +++++++++ Proxy's Request after hooks +++++++++ -- State Machine Id: 168 GET /z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44 HTTP/1.1 Host: sshds5-lh.akamaihd.net User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 X-Requested-With: ShockwaveFlash/19.0.0.226 Accept: */* DNT: 1 Referer: http://www.hotstar.com/ Accept-Encoding: gzip Accept-Language: en-US,en;q=0.8,bg;q=0.6 Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8 If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT +++++++++ Incoming O.S. Response +++++++++ -- State Machine Id: 168 HTTP/1.1 200 OK Server: AkamaiGHost Mime-Version: 1.0 Content-Type: video/abst Content-Length: 122 Last-Modified: Thu, 05 Nov 2015 02:30:28 GMT Expires: Thu, 05 Nov 2015 11:38:01 GMT Cache-Control: max-age=0, no-cache Pragma: no-cache Date: Thu, 05 Nov 2015 11:38:01 GMT Connection: keep-alive Set-Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; path=/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/; domain=sshds5-lh.akamaihd.net +++++++++ Base Header for Building Response +++++++++ -- State Machine Id: 168 HTTP/1.1 200 OK Server: AkamaiGHost Mime-Version: 1.0 Content-Type: video/abst Content-Length: 122 Last-Modified: Thu, 05 Nov 2015 02:30:28 GMT Expires: Thu, 05 Nov 2015 11:38:01 GMT Cache-Control: max-age=0, no-cache Pragma: no-cache Date: Thu, 05 Nov 2015 11:38:01 GMT Connection: keep-alive Set-Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; path=/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/; domain=sshds5-lh.akamaihd.net +++++++++ Proxy's Response 2 +++++++++ -- State Machine Id: 168 HTTP/1.1 304 Not Modified Date: Thu, 05 Nov 2015 11:38:01 GMT Expires: Thu, 05 Nov 2015 11:38:01 GMT Cache-Control: max-age=0, no-cache Connection: keep-alive +++++++++ Proxy's Response (Client Conditionals) +++++++++ -- State Machine Id: 168 HTTP/1.1 304 Not Modified Date: Thu, 05 Nov 2015 11:38:01 GMT Expires: Thu, 05 Nov 2015 11:38:01 GMT Cache-Control: max-age=0, no-cache Connection: keep-alive {noformat} was: Caching proxy running ATS 5.3.x (5.3.0, 5.3.1, 5.3.2 all fail) with proxy.config.http.cache.when_to_revalidate = 4 breaks akamaihd.net live streaming by overwriting non-cacheable origin response from 200 to 304 received against conditional request originating from the client. Here's a log of all request and response headers for a single transaction, collected via debug: {noformat} +++++++++ Incoming Request +++++++++ -- State Machine Id: 168 GET http:///z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44 HTTP/1.1 Host: sshds5-lh.akamaihd.net Connection: keep-alive User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 X-Requested-With: ShockwaveFlash/19.0.0.226 Accept: */* DNT: 1 Referer: http://www.hotstar.com/ Accept-Encoding: gzip Accept-Language: en-US,en;q=0.8,bg;q=0.6 Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8 If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT +++++++++ Proxy's Request +++++++++ -- State Machine Id: 168 GET /z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44 HTTP/1.1 Host: sshds5-lh.akamaihd.net User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 X-Requested-With: ShockwaveFlash/19.0.0.226 Accept: */* DNT: 1 Referer: http://www.hotstar.com/ Accept-Encoding: gzip Accept-Language: en-US,en;q=0.8,bg;q=0.6 Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8 If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT +++++++++ Proxy's Request after hooks +++++++++ -- State Machine Id: 168 GET /z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44 HTTP/1.1 Host: sshds5-lh.akamaihd.net User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 X-Requested-With: ShockwaveFlash/19.0.0.226 Accept: */* DNT: 1 Referer: http://www.hotstar.com/ Accept-Encoding: gzip Accept-Language: en-US,en;q=0.8,bg;q=0.6 Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8 If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT +++++++++ Incoming O.S. Response +++++++++ -- State Machine Id: 168 HTTP/1.1 200 OK Server: AkamaiGHost Mime-Version: 1.0 Content-Type: video/abst Content-Length: 122 Last-Modified: Thu, 05 Nov 2015 02:30:28 GMT Expires: Thu, 05 Nov 2015 11:38:01 GMT Cache-Control: max-age=0, no-cache Pragma: no-cache Date: Thu, 05 Nov 2015 11:38:01 GMT Connection: keep-alive Set-Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; path=/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/; domain=sshds5-lh.akamaihd.net +++++++++ Base Header for Building Response +++++++++ -- State Machine Id: 168 HTTP/1.1 200 OK Server: AkamaiGHost Mime-Version: 1.0 Content-Type: video/abst Content-Length: 122 Last-Modified: Thu, 05 Nov 2015 02:30:28 GMT Expires: Thu, 05 Nov 2015 11:38:01 GMT Cache-Control: max-age=0, no-cache Pragma: no-cache Date: Thu, 05 Nov 2015 11:38:01 GMT Connection: keep-alive Set-Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; path=/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/; domain=sshds5-lh.akamaihd.net +++++++++ Proxy's Response 2 +++++++++ -- State Machine Id: 168 HTTP/1.1 304 Not Modified Date: Thu, 05 Nov 2015 11:38:01 GMT Expires: Thu, 05 Nov 2015 11:38:01 GMT Cache-Control: max-age=0, no-cache Connection: keep-alive +++++++++ Proxy's Response (Client Conditionals) +++++++++ -- State Machine Id: 168 HTTP/1.1 304 Not Modified Date: Thu, 05 Nov 2015 11:38:01 GMT Expires: Thu, 05 Nov 2015 11:38:01 GMT Cache-Control: max-age=0, no-cache Connection: keep-alive {noformat} > "[hcoofsr] conditional request, 200 response, send back 304 if possible > [crc=304]" breaks akamaihd.net live streaming > --------------------------------------------------------------------------------------------------------------------- > > Key: TS-3995 > URL: https://issues.apache.org/jira/browse/TS-3995 > Project: Traffic Server > Issue Type: Bug > Components: Cache, Core, HTTP > Affects Versions: 5.3.2 > Reporter: Nikolai Gorchilov > > Caching proxy running ATS 5.3.x (5.3.0, 5.3.1, 5.3.2 all fail) with > proxy.config.http.cache.when_to_revalidate = 4 breaks akamaihd.net live > streaming. > The actual problem is that ATS rewrites origin response from 200 to 304, due > to If-Modified-Since conditional header in client's request. As per ATS logic > object is unmodified, but in fact it is. Most probably it's related to the > fact that player and server somehow play with if-modified-since/last-modified > headers pair to communicate position in the live stream. > As result, Akamai player keeps repeating the said request, expecting it's > 200, but getting 304 thus live video freezes forever, just a few seconds > after start. > IMHO when proxy.config.http.cache.when_to_revalidate = 4, ATS shall not > interfere with origin response in this manner. > Here's a debug log of request and response headers at different states in a > single transaction: > {noformat} > +++++++++ Incoming Request +++++++++ > -- State Machine Id: 168 > GET > http:///z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44 > HTTP/1.1 > Host: sshds5-lh.akamaihd.net > Connection: keep-alive > User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 > X-Requested-With: ShockwaveFlash/19.0.0.226 > Accept: */* > DNT: 1 > Referer: http://www.hotstar.com/ > Accept-Encoding: gzip > Accept-Language: en-US,en;q=0.8,bg;q=0.6 > Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; > hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8 > If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT > +++++++++ Proxy's Request +++++++++ > -- State Machine Id: 168 > GET > /z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44 > HTTP/1.1 > Host: sshds5-lh.akamaihd.net > User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 > X-Requested-With: ShockwaveFlash/19.0.0.226 > Accept: */* > DNT: 1 > Referer: http://www.hotstar.com/ > Accept-Encoding: gzip > Accept-Language: en-US,en;q=0.8,bg;q=0.6 > Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; > hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8 > If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT > +++++++++ Proxy's Request after hooks +++++++++ > -- State Machine Id: 168 > GET > /z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/464_209823ecd2922291-p.bootstrap?g=YCEAMIWDQZQT&hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8&hdcore=3.7.0&plugin=aasp-3.7.0.39.44 > HTTP/1.1 > Host: sshds5-lh.akamaihd.net > User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36 > X-Requested-With: ShockwaveFlash/19.0.0.226 > Accept: */* > DNT: 1 > Referer: http://www.hotstar.com/ > Accept-Encoding: gzip > Accept-Language: en-US,en;q=0.8,bg;q=0.6 > Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; > hdntl=exp=1446809835~acl=%2f*~data=hdntl~hmac=ebaedb13781605ce7f9f26b84e1346a7d43ecf0dfcc99e6b53e32487565ba3f8 > If-Modified-Since: Thu, 05 Nov 2015 02:30:28 GMT > +++++++++ Incoming O.S. Response +++++++++ > -- State Machine Id: 168 > HTTP/1.1 200 OK > Server: AkamaiGHost > Mime-Version: 1.0 > Content-Type: video/abst > Content-Length: 122 > Last-Modified: Thu, 05 Nov 2015 02:30:28 GMT > Expires: Thu, 05 Nov 2015 11:38:01 GMT > Cache-Control: max-age=0, no-cache > Pragma: no-cache > Date: Thu, 05 Nov 2015 11:38:01 GMT > Connection: keep-alive > Set-Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; > path=/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/; > domain=sshds5-lh.akamaihd.net > +++++++++ Base Header for Building Response +++++++++ > -- State Machine Id: 168 > HTTP/1.1 200 OK > Server: AkamaiGHost > Mime-Version: 1.0 > Content-Type: video/abst > Content-Length: 122 > Last-Modified: Thu, 05 Nov 2015 02:30:28 GMT > Expires: Thu, 05 Nov 2015 11:38:01 GMT > Cache-Control: max-age=0, no-cache > Pragma: no-cache > Date: Thu, 05 Nov 2015 11:38:01 GMT > Connection: keep-alive > Set-Cookie: _alid_=PmjLqgcUUqw6TP5gtK/xbg==; > path=/z/delayed/indvsa2015_INDVSSATEST1DAY1_1@336263/; > domain=sshds5-lh.akamaihd.net > +++++++++ Proxy's Response 2 +++++++++ > -- State Machine Id: 168 > HTTP/1.1 304 Not Modified > Date: Thu, 05 Nov 2015 11:38:01 GMT > Expires: Thu, 05 Nov 2015 11:38:01 GMT > Cache-Control: max-age=0, no-cache > Connection: keep-alive > +++++++++ Proxy's Response (Client Conditionals) +++++++++ > -- State Machine Id: 168 > HTTP/1.1 304 Not Modified > Date: Thu, 05 Nov 2015 11:38:01 GMT > Expires: Thu, 05 Nov 2015 11:38:01 GMT > Cache-Control: max-age=0, no-cache > Connection: keep-alive > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)