Re: Varnish wouldn't cache HLS fragments
please keep the mailing-list CC'd
you backend is telling Varnish to not cache:
-- BerespHeader Cache-Control: no-cache
which is acted upon in the built-in.vcl:
https://github.com/varnishcache/varnish-cache/blob/6.0/bin/varnishd/builtin.vcl#L161
more info here;
https://varnish-cache.org/docs/trunk/users-guide/vcl-built-in-code.html#vcl-built-in-code
and maybe this can help too:
https://info.varnish-software.com/blog/finally-understanding-built-in-vcl
--
Guillaume Quintard
On Sun, Jun 13, 2021 at 11:16 PM Hamidreza Hosseini
wrote:
> This is one of hls fragments that I want to be cached:
>
> > wget http://stream.test.local/hls/mystream/1623650629260.ts
>
> ```
> * << Request >> 32770
> - Begin req 32769 rxreq
> - Timestamp Start: 1623650670.552461 0.00 0.00
> - Timestamp Req: 1623650670.552461 0.00 0.00
> - VCL_useboot
> - ReqStart 192.168.200.10 58016 a0
> - ReqMethod GET
> - ReqURL /hls/mystream/1623650629260.ts
> - ReqProtocolHTTP/1.1
> - ReqHeader User-Agent: Wget/1.20.3 (linux-gnu)
> - ReqHeader Accept: */*
> - ReqHeader Accept-Encoding: identity
> - ReqHeader Host: stream.test.local
> - ReqHeader Connection: Keep-Alive
> - ReqHeader X-Forwarded-For: 192.168.200.10
> - VCL_call RECV
> - VCL_return hash
> - ReqUnset Accept-Encoding: identity
> - VCL_call HASH
> - VCL_return lookup
> - VCL_call MISS
> - VCL_return fetch
> - Link bereq 32771 fetch
> - Timestamp Fetch: 1623650670.557642 0.005181 0.005181
> - RespProtocol HTTP/1.1
> - RespStatus 200
> - RespReason OK
> - RespHeader Server: nginx/1.20.1
> - RespHeader Date: Mon, 14 Jun 2021 06:04:30 GMT
> - RespHeader Content-Type: video/mp2t
> - RespHeader Content-Length: 161868
> - RespHeader Last-Modified: Mon, 14 Jun 2021 06:03:51 GMT
> - RespHeader ETag: "60c6f147-2784c"
> - RespHeader Cache-Control: no-cache
> - RespHeader Access-Control-Allow-Origin: *
> - RespHeader Access-Control-Expose-Headers: Content-Length
> - RespHeader Accept-Ranges: bytes
> - RespHeader X-Varnish: 32770
> - RespHeader Age: 0
> - RespHeader Via: 1.1 varnish (Varnish/6.2)
> - VCL_call DELIVER
> - VCL_return deliver
> - Timestamp Process: 1623650670.557660 0.005199 0.18
> - Filters
> - RespHeader Connection: keep-alive
> - Timestamp Resp: 1623650670.558417 0.005956 0.000757
> - ReqAcct179 0 179 406 161868 162274
> - End
> ** << BeReq>> 32771
> -- Begin bereq 32770 fetch
> -- VCL_useboot
> -- Timestamp Start: 1623650670.552655 0.00 0.00
> -- BereqMethodGET
> -- BereqURL /hls/mystream/1623650629260.ts
> -- BereqProtocol HTTP/1.1
> -- BereqHeaderUser-Agent: Wget/1.20.3 (linux-gnu)
> -- BereqHeaderAccept: */*
> -- BereqHeaderHost: stream.test.local
> -- BereqHeaderX-Forwarded-For: 192.168.200.10
> -- BereqHeaderAccept-Encoding: gzip
> -- BereqHeaderX-Varnish: 32771
> -- VCL_call BACKEND_FETCH
> -- VCL_return fetch
> -- BackendOpen25 b1 {Backend_ip} 80 {Varnish_ip} 49734
> -- BackendStart {Backend_ip} 80
> -- Timestamp Bereq: 1623650670.552739 0.84 0.84
> -- Timestamp Beresp: 1623650670.557325 0.004669 0.004586
> -- BerespProtocol HTTP/1.1
> -- BerespStatus 200
> -- BerespReason OK
> -- BerespHeader Server: nginx/1.20.1
> -- BerespHeader Date: Mon, 14 Jun 2021 06:04:30 GMT
> -- BerespHeader Content-Type: video/mp2t
> -- BerespHeader Content-Length: 161868
> -- BerespHeader Last-Modified: Mon, 14 Jun 2021 06:03:51 GMT
> -- BerespHeader Connection: keep-alive
> -- BerespHeader ETag: "60c6f147-2784c"
> -- BerespHeader Cache-Control: no-cache
> -- BerespHeader Access-Control-Allow-Origin: *
> -- BerespHeader Access-Control-Expose-Headers: Content-Length
> -- BerespHeader Accept-Ranges: bytes
> -- TTLRFC 120 10 0 1623650671 1623650671 1623650670 0 0
> cacheable
> -- VCL_call BACKEND_RESPONSE
> -- TTLVCL 300 10 0 1623650671 cacheable
> -- TTLVCL 30 10 0 1623650671 cacheable
> -- TTLVCL 120 10 0 1623650671 cacheable
> -- TTLVCL 120 10 0 1623650671 uncacheable
> -- VCL_return deliver
> -- Filters
> -- Storagemalloc Transient
> -- Fetch_Body 3 length stream
> -- BackendReuse 25 b1
> -- Timestamp BerespBody: 1623650670.558352 0.005697 0.001
Re: Varnish wouldn't cache HLS fragments
Hi,
Can you share the output of "varnishlog -g request" for one of those
requests that should be cached please?
Cheers,
--
Guillaume Quintard
On Sun, Jun 13, 2021, 00:17 Hamidreza Hosseini
wrote:
> Hi,
> I put varnish in front of my http servers to serve Hls streaming, I want
> varnish cache the fragments but not .m3u8 manifest file,
> I configure it but it cache nothing!
> My configuration file:
>
> ```
> vcl 4.1;
>
> import directors;
>
>
> backend b1 {
> .host = "playback-02";
> .probe = {
> .url = "/";
> .timeout = 150 ms;
> .interval = 10s;
> .window = 6;
> .threshold = 5;
> }
> }
>
>
>
> sub vcl_init {
> # we use round robin director for our backend swift proxies
>
> new hls_cluster = directors.round_robin();
> hls_cluster.add_backend(b1);
>
> }
>
> acl purge {
> "localhost";
> }
>
>
> sub vcl_recv {
>
> set req.backend_hint = hls_cluster.backend();
> if (req.method == "PURGE") {
> if (!client.ip ~ purge) {
> return(synth(405,"Not allowed."));
> }
> return (purge);
> }
>
> if (req.url ~ "\.m3u8$") {
> return (pass);
> }
> }
>
>
>
>
>
> sub vcl_backend_response {
> # cache for half of a day
> set beresp.ttl=5m;
> # Don't cache 404 responses
>
> if (bereq.url ~ "\.(aac|dash|m4s|mp4|ts)$") {
> set beresp.ttl = 30s;
> }
>
> if ( beresp.status == 404 ) {
> set beresp.ttl = 120s;
> set beresp.uncacheable = true;
> return (deliver);
> }
> if (beresp.status == 500 || beresp.status == 502 || beresp.status ==
> 503 || beresp.status == 504)
> {
> set beresp.uncacheable = true;
> }
> }
>
> ```
>
> Varnish version:
> varnishd (varnish-6.0.7 revision 525d371e3ea0e0c38edd7baf0f80dc226560f26e)
> Copyright (c) 2006 Verdens Gang AS
> Copyright (c) 2006-2020 Varnish Software AS
>
> Distribution: Ubuntu 20.04 LTS
>
> ___
> varnish-misc mailing list
> varnish-misc@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
___
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
Varnish wouldn't cache HLS fragments
Hi,
I put varnish in front of my http servers to serve Hls streaming, I want
varnish cache the fragments but not .m3u8 manifest file,
I configure it but it cache nothing!
My configuration file:
```
vcl 4.1;
import directors;
backend b1 {
.host = "playback-02";
.probe = {
.url = "/";
.timeout = 150 ms;
.interval = 10s;
.window = 6;
.threshold = 5;
}
}
sub vcl_init {
# we use round robin director for our backend swift proxies
new hls_cluster = directors.round_robin();
hls_cluster.add_backend(b1);
}
acl purge {
"localhost";
}
sub vcl_recv {
set req.backend_hint = hls_cluster.backend();
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405,"Not allowed."));
}
return (purge);
}
if (req.url ~ "\.m3u8$") {
return (pass);
}
}
sub vcl_backend_response {
# cache for half of a day
set beresp.ttl=5m;
# Don't cache 404 responses
if (bereq.url ~ "\.(aac|dash|m4s|mp4|ts)$") {
set beresp.ttl = 30s;
}
if ( beresp.status == 404 ) {
set beresp.ttl = 120s;
set beresp.uncacheable = true;
return (deliver);
}
if (beresp.status == 500 || beresp.status == 502 || beresp.status == 503 ||
beresp.status == 504)
{
set beresp.uncacheable = true;
}
}
```
Varnish version:
varnishd (varnish-6.0.7 revision 525d371e3ea0e0c38edd7baf0f80dc226560f26e)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2020 Varnish Software AS
Distribution: Ubuntu 20.04 LTS
___
varnish-misc mailing list
varnish-misc@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
