[squid-users] Squid issue of caching the m3u8 file

[LIU Yaning]

Dear All,

I would like to cache the .m3u8 file to be able to provide offline caching
service by Squid. The played HLS video streaming is the link as below:
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8

However, the .m3u8 file is not be cached probably because it is mentioned
as a no-cache, no-store, max-age=0 in the "Cache-Control" in the HTTP
header.

HTTP/1.1 200 OK

Server: Apache

ETag: "1d7168b4f49e75f76f3182f24bf075f6:1299516751"

Last-Modified: Mon, 07 Mar 2011 16:52:31 GMT

Expires: Fri, 02 Jun 2017 14:26:52 GMT

Cache-Control: max-age=0, no-cache, no-store

Pragma: no-cache

Date: Fri, 02 Jun 2017 14:26:52 GMT

Content-Length: 16046

Set-Cookie: AKID=77F9F1316ECCE780566608C5E514DE0A;expires=Fri, 26 Aug 2016
00:01:00 GMT; path=/; domain=qthttp.apple.com.edgesuite.net

Content-Type: application/x-mpegURL

Access-Control-Allow-Origin: *

I added a new rule for .m3u8 file in squid.conf, however, it is still not
working.

refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire
override-lastmod ignore-no-cache ignore-no-store

Does anyone know how to allow Squid caching the .m3u8 file? Thanks a lot in
advance.

Best Regards,
--
Yaning.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid issue of caching the m3u8 file

[Amos Jeffries]

On 05/06/17 23:54, LIU Yaning wrote:

Dear All,

I would like to cache the .m3u8 file to be able to provide offline 
caching service by Squid. The played HLS video streaming is the link 
as below:
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 

However, the .m3u8 file is not be cached probably because it is 
mentioned as a no-cache, no-store, max-age=0 in the "Cache-Control" in 
the HTTP header.
Nope. Only the CC:no-store is preventing caching. The other headers 
simply put boundaries on what is to be done with the content in the 
cache. In particular the "no-cache", max-age=0 and Expires values mean 
it has to be revalidated (REFRESH in your access.log) before any future 
uses - probably because of that Set-Cookie needing to be changed for 
different end-users.


HTTP/1.1 200 OK

Server: Apache

ETag: "1d7168b4f49e75f76f3182f24bf075f6:1299516751"

Last-Modified: Mon, 07 Mar 2011 16:52:31 GMT

Expires: Fri, 02 Jun 2017 14:26:52 GMT

Cache-Control: max-age=0, no-cache, no-store

Pragma: no-cache

Date: Fri, 02 Jun 2017 14:26:52 GMT

Content-Length: 16046

Set-Cookie: AKID=77F9F1316ECCE780566608C5E514DE0A;expires=Fri, 26 Aug 
2016 00:01:00 GMT; path=/; domain=qthttp.apple.com.edgesuite.net 



Content-Type: application/x-mpegURL

Access-Control-Allow-Origin: *

I added a new rule for .m3u8 file in squid.conf, however, it is still 
not working.


refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire 
override-lastmod ignore-no-cache ignore-no-store


Does anyone know how to allow Squid caching the .m3u8 file? Thanks a 
lot in advance.
What makes you think it is not caching? The ignore-no-store alone should 
be sufficient to allow current Squid versions to cache that object. You 
could perhapse add "store-stale" option on that config line. Which 
should make Squid cache object containing an Expires header with current 
or past values. refresh_pattern settings do not affect that cacheable vs 
non-cacheable decision. The no-cache header tells Squid the object needs 
revalidating before every use. However, be aware the tool at redbot.org 
tells me that this URL is badly broken in how it is using the ETag and 
Vary headers - in a way which can break the revalidation when these 
things are cached. Some of your clients may see very broken behaviour 
accessing this object unless you follow the no-store requirement or the 
server stops its broken ETag behaviour. PS. if you are using a Squid 
version much older than 3.5.24 I recommend an upgrade. With an urgency 
increasing the older your Squid is. Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid issue of caching the m3u8 file

[LIU Yaning]

Dear Amos,

Thanks a lot for your explanation and suggestion. I added the "store-stale"
to the refresh_pattern rule as:
refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire override-lastmod
ignore-no-cache ignore-no-store store-stale

However, I have checked the access.log, I am still getting TCP_Miss.
1496754869.963 13 192.168.0.100 TCP_MISS/200 16636 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg>
-
HIER_DIRECT/95.101.182.201
<http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q>
 application/x-mpegURL
1496754870.605 4 192.168.0.100 TCP_MISS/200 16636 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg>
-
HIER_DIRECT/95.101.182.201
<http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q>
 application/x-mpegURL
1496754871.194 15 192.168.0.100 TCP_MISS/200 16636 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg>
-
HIER_DIRECT/95.101.182.201
<http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q>
 application/x-mpegURL
1496754871.715 4 192.168.0.100 TCP_MISS/200 16636 GET
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
<http://www.google.com/url?q=http%3A%2F%2Fqthttp.apple.com.edgesuite.net%2F1010qwoeiuryfg%2F0150_vod.m3u8&sa=D&sntz=1&usg=AFQjCNHpiHy55EMeBIaMGhgEKRHanTrXxg>
-
HIER_DIRECT/95.101.182.201
<http://www.google.com/url?q=http%3A%2F%2F95.101.182.201&sa=D&sntz=1&usg=AFQjCNE9ZWXH7sOJgbqIA--MJwxobSp76Q>
 application/x-mpegURL

If I understand well, TCP_MISS/200 shows the content is not cached by
Squid. Could you please help me to see if anything I did wrong to make
.m3u8 not cached by Squid?


Date: Tue, 6 Jun 2017 14:08:11 +1200
From: Amos Jeffries 
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Squid issue of caching the m3u8 file
Message-ID: <5af5e6e9-4880-f58e-106b-9f11dd0a4...@treenet.co.nz>
Content-Type: text/plain; charset=utf-8; format=flowed

On 05/06/17 23:54, LIU Yaning wrote:
> Dear All,
>
> I would like to cache the .m3u8 file to be able to provide offline
> caching service by Squid. The played HLS video streaming is the link
> as below:
> http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8
> <http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8>
> However, the .m3u8 file is not be cached probably because it is
> mentioned as a no-cache, no-store, max-age=0 in the "Cache-Control" in
> the HTTP header.
Nope. Only the CC:no-store is preventing caching. The other headers
simply put boundaries on what is to be done with the content in the
cache. In particular the "no-cache", max-age=0 and Expires values mean
it has to be revalidated (REFRESH in your access.log) before any future
uses - probably because of that Set-Cookie needing to be changed for
different end-users.
>
> HTTP/1.1 200 OK
>
> Server: Apache
>
> ETag: "1d7168b4f49e75f76f3182f24bf075f6:1299516751"
>
> Last-Modified: Mon, 07 Mar 2011 16:52:31 GMT
>
> Expires: Fri, 02 Jun 2017 14:26:52 GMT
>
> Cache-Control: max-age=0, no-cache, no-store
>
> Pragma: no-cache
>
> Date: Fri, 02 Jun 2017 14:26:52 GMT
>
> Content-Length: 16046
>
> Set-Cookie: AKID=77F9F1316ECCE780566608C5E514DE0A;expires=Fri, 26 Aug
> 2016 00:01:00 GMT; path=/; domain=qthttp.apple.com.edgesuite.net
> <http://qthttp.apple.com.edgesuite.net/>
>
> Content-Type: application/x-mpegURL
>
> Access-Control-Allow-Origin: *
>
> I added a new rule for .m3u8 file in squid.conf, however, it is still
> not working.
>
> refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire
> override-lastmod ignore-no-cache ignore-no-store
>
> Does anyone know how to allow Squid caching the .m3u8 file? Thanks a
> lot in advance.
What makes you think it is not caching? The ignore-no-store alone should
be sufficient to allow current Squid versions to cache that object. You
could perhapse add "store-stale" option on that config line. Which
should make Squid cache object containing an Expires header with current
or past values. refresh_pattern settings do not affect that cacheable vs
non-cacheable de

Re: [squid-users] Squid issue of caching the m3u8 file

[Amos Jeffries]

On 07/06/17 01:33, LIU Yaning wrote:

Dear Amos,

Thanks a lot for your explanation and suggestion. I added the 
"store-stale" to the refresh_pattern rule as:
refresh_pattern -i \.(ts|m3u8)$ 120 90% 1000 override-expire 
override-lastmod ignore-no-cache ignore-no-store store-stale


However, I have checked the access.log, I am still getting TCP_Miss.
1496754869.963 13 192.168.0.100 TCP_MISS/200 16636 GET 
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 
 - 
HIER_DIRECT/95.101.182.201 
 application/x-mpegURL
1496754870.605 4 192.168.0.100 TCP_MISS/200 16636 GET 
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 
 - 
HIER_DIRECT/95.101.182.201 
 application/x-mpegURL
1496754871.194 15 192.168.0.100 TCP_MISS/200 16636 GET 
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 
 - 
HIER_DIRECT/95.101.182.201 
 application/x-mpegURL
1496754871.715 4 192.168.0.100 TCP_MISS/200 16636 GET 
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 
 - 
HIER_DIRECT/95.101.182.201 
 application/x-mpegURL


If I understand well, TCP_MISS/200 shows the content is not cached by 
Squid. Could you please help me to see if anything I did wrong to make 
.m3u8 not cached by Squid?


I'm out of ideas sorry.


FWIW: this is what I get from my test setup using your refresh_pattern:

1496766607.433   1284 ::1 TCP_MISS/200 16628 GET 
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 - 
HIER_DIRECT/119.224.143.41 application/x-mpegURL
1496766610.254902 ::1 TCP_REFRESH_UNMODIFIED/200 16499 GET 
http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 - 
HIER_DIRECT/119.224.143.41 application/x-mpegURL


I am currently testing Squid-4.0.20, but it should behave the same in 
the recent few v3.5 releases.


Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid issue of caching the m3u8 file

[joseph]

is Set-Cookie:  saved in cached file as well ?? amos




--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-issue-of-caching-the-m3u8-file-tp4682674p4682714.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid issue of caching the m3u8 file

[Amos Jeffries]

On 07/06/17 21:42, joseph wrote:

is Set-Cookie:  saved in cached file as well ?? amos


Yes it is. The header on the cached object only gets removed on delivery 
to a client.

Squid does not comply with the Cookie specifications in this regard.

With the extra wrinkle that the mandatory revalidation on these objects 
may provide a new Set-Cookie header that gets added for delivery to the 
new client after the old/cached header was removed. So you may still see 
Set-Cookie on the revalidated HIT (aka REFRESH_UNMODIFIED). Adding a new 
Set-Cookie is usually the purpose of the revalidation being required on 
these sort of objects.


Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid issue of caching the m3u8 file

[joseph]

so if the  server send   same obj  with new Cookie it will be miss  since the
Cookie dose not match in cached obj

regarding the topping sorry if i ask it will clear  all those question  in
all the question has Ben asked before
example cached file header

link   = 
http://sa.bbc.co.uk/bbc/bbc/s?name=SET-COUNTER.page&ml_name=webmodule&ml_version=63
header

Server: nginx
Date: Thu, 04 May 2017 16:14:41 GMT
Content-Type: image/gif
Content-Length: 43
Connection: close
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Pragma: no-cache
Cache-Control: no-cache
P3P: policyref="http://www.nedstat.com/w3c/p3p.xml";, CP="NOI DSP COR NID PSA
ADM OUR IND NAV COM"
so all those  should match right  to have hit  not  just the link and  some
of the header  like Cache-Control or so

even cookie and   p3p   am i right here
or  its just specific header should match ???

like his situation  for miss not only  the no-cache prevent HIT  its  cookie
as well
cookie on the link he provide  change on every single  click to that link

another question  what about  deleting  cookie  before save the content 
1 since  its same object they deliver so it will be  hit  dose it violate ??
2 or  damage the clients  web view ?? or anything bad







--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-issue-of-caching-the-m3u8-file-tp4682674p4682722.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid issue of caching the m3u8 file

[Amos Jeffries]

On 08/06/17 01:00, joseph wrote:

so if the  server send   same obj  with new Cookie it will be miss  since the
Cookie dose not match in cached obj


No, the Vary header does not say Cookie is part of the variance AFAICT. 
Just URL plus Accept-Encoding.


The Cookie/Set-Cookie being accurate is just related to whether the 
client browsing session is kept continuous or breaks. Depending on 
whether this transaction is part of a purchase or something similar that 
may be significant, or not.


At a guess since this is a media related object from Apple I suspect it 
is linked to an iTunes account of some sort. So the Cookie might be 
needed by the client for something. In that case getting a reply without 
one (as a pure cache HIT would appear to the client) may have problems.


That is just speculation though to show that this traffic behaviour is 
not completely unreasonable. Only the site and client software authors 
actually know for sure what is intended to be going on and why.



So, to get back on topic. Yes Squid should be caching it. But don't 
expect to see the letters "HIT" in the log anymore for this particular URL.


Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid issue of caching the m3u8 file

[joseph]

right 

lets say i have obj  with this header
if server  send the same  obj  with different Set-Cookie value that will be
MISS i was refairing to this
not to the vary if it has cookie sorry if i did not explain it correctly

Server: nginx
Date: Mon, 22 May 2017 15:44:59 GMT
Content-Type: image/gif
Content-Length: 43
Expires: Fri, 20 Mar 2009 00:00:00 GMT
P3P: CP="CUR ADM OUR NOR STA NID"
Set-Cookie:
ljtrtb=eJyrrgUAAXUA%2BQ%3D%3D;Path=/;Domain=.lijit.com;Expires=Thu,
01-Jan-1970 00:00:00 GMT
Set-Cookie:
lijit_retarget=eJyrrgUAAXUA%2BQ%3D%3D;Path=/;Domain=.lijit.com;Expires=Thu,
01-Jan-1970 00:00:00 GMT
X-Sovrn-Pod: ap3iad3



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-issue-of-caching-the-m3u8-file-tp4682674p4682724.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid issue of caching the m3u8 file

[Amos Jeffries]

On 08/06/17 02:04, joseph wrote:

right

lets say i have obj  with this header
if server  send the same  obj  with different Set-Cookie value that will be
MISS i was refairing to this
not to the vary if it has cookie sorry if i did not explain it correctly


Cookie(s) have nothing to do with MISS unless it is listed in Vary.

The fact that the object is coming from a server without involving any 
object in the proxies own cache is what makes a transaction be 
classified as MISS.


By comparison; things that involve both an object in the proxies cache 
and a server are REFRESH, and things that _only_ involve the proxy cache 
are HIT.


Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users