Hi All, I have two things to clarify: 1) In earlier email (snipped below), Amos told that is caching and scheduled to download - does it mean that we got the answer and do some override? -------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > This is what I see when the download is in progress: > > KEY 44000000000000000902000000000000 > STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE > RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED
So file stored in memory and scheduled for removal. -------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2) With more debug_options enabled, I see that it is not caching because the response is part of authenticated flow. Is there a way I can override this? - Initially when this file download starts, it gets authorized (I think that is why I see HEAD request is sent to target which has Cache-Control: max-age=0) and then the subsequent GET requests (dont have any cache-control header) download the chunk using adjusted range and offset. - But the subsequent GET reply seems set with auth flag, see the code below snipped from source ( https://github.com/squid-cache/squid/blob/4f1c93a7a0d14eec223e199275ce570d840f71bc/src/http.cc ). // RFC 2068, sec 14.9.4 - MUST NOT cache any response with Authentication UNLESS certain CC controls are present // allow HTTP violations to IGNORE those controls (ie re-block caching Auth) if (request && (request->flags.auth || request->flags.authSent)) { if (!rep->cache_control) return decision.make(ReuseDecision::reuseNot, "authenticated and server reply missing Cache-Control"); - I tried adding override-expire in the cgi-bin refresh pattern, but that will override only for max-age in Cache-control but not relevant for auth flag. Please find the logs below: Please let me know if I am missing something. Thanks, Hari Refresh pattern (URL: https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhM...): refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 Logs: 2018/09/10 09:57:24.864| 11,2| http.cc(723) processReplyHeader: HTTP Server RESPONSE: --------- HTTP/1.1 206 Partial Content Date: Mon, 10 Sep 2018 09:57:20 GMT^M Server: Apache/2.2^M Content-Disposition: attachment; filename=somefile.iso;^M Last-Modified: Thu, 22 Mar 2018 02:11:11 GMT^M ETag: "4ad8a61-1193b000-567f6d2466dc0"^M Accept-Ranges: bytes^M Content-Length: 294342471^M Content-Range: bytes 549049-294891519/294891520^M Keep-Alive: timeout=5, max=100^M Connection: Keep-Alive^M Content-Type: application/unknown^M ^M ---------- 2018/09/10 09:57:24.864| 11,5| Client.cc(134) setVirginReply: 0x5560bf163d78 setting virgin reply to 0x5560bf162c20 2018/09/10 09:57:24.864| ctx: exit level 0 2018/09/10 09:57:24.864| 83,3| AccessCheck.cc(42) Start: adaptation off, skipping 2018/09/10 09:57:24.864| 11,5| Client.cc(969) adaptOrFinalizeReply: adaptationAccessCheckPending=0 2018/09/10 09:57:24.864| 11,5| Client.cc(152) setFinalReply: 0x5560bf163d78 setting final reply to 0x5560bf162c20 2018/09/10 09:57:24.864| 20,3| store.cc(1807) replaceHttpReply: StoreEntry::replaceHttpReply: https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtc 2018/09/10 09:57:24.864| ctx: enter level 0: ' https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtcoacGcTu6dv-dkLcT4lqtgvM70n8-ucJsj09lRYt_a0t7_M5 2018/09/10 09:57:24.864| 11,3| http.cc(907) haveParsedReplyHeaders: HTTP CODE: 206 2018/09/10 09:57:24.864| 73,3| HttpRequest.cc(664) storeId: sent back effectiveRequestUrl: https://dl.cisco.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtcoac 2018/09/10 09:57:24.864| 20,3| Controller.cc(386) peek: 76E544615E001DBF49EF0F94EE0A8F9A 2018/09/10 09:57:24.865| 20,4| Controller.cc(420) peek: cannot locate 76E544615E001DBF49EF0F94EE0A8F9A 2018/09/10 09:57:24.865| 20,3| store.cc(450) releaseRequest: 0 e:=p2IV/0x5560bf16ac60*3 2018/09/10 09:57:24.865| 20,3| store.cc(580) setPrivateKey: 01 e:=p2IV/0x5560bf16ac60*3 2018/09/10 09:57:24.865| 11,3| http.cc(982) haveParsedReplyHeaders: decided: do not cache and do not share because authenticated and server reply missing Cache-Control; HTTP status 206 e:=p2XIV/0x 2018/09/10 09:57:24.865| ctx: exit level 0 On Sat, Sep 8, 2018 at 8:13 PM Hariharan Sethuraman <srnh...@gmail.com> wrote: > But the partial data is been continuously sending back to client. Squid > didn't wait for the complete file to download > > On Sat, 8 Sep 2018, 20:08 Amos Jeffries, <squ...@treenet.co.nz> wrote: > >> On 8/09/18 7:44 PM, Hariharan Sethuraman wrote: >> > And here is my HTTP request and reply: >> > 1. HEAD request to read image information >> > 2. GET request to download the image >> > >> >> Not quite. GET partial / Range request to fetch the content. >> >> Squid converted it into a full request for the backend server due to >> range_offset_limit -1. But that does mean Squid had to download ~240MB >> of data before anything starts being sent to the client. >> >> >> >> > (2) >> > 2018/09/08 07:28:39.155| 11,2| client_side.cc(1278) parseHttpRequest: >> > HTTP Client REQUEST: >> > --------- >> > GET /DcKbz9kqMQXK-zp95pv9LH11kjhTpxOJsJ-1FYEL4 >> > Host: example.com:3129 <http://example.com:3129>^M >> > Range: bytes=242819145-^M >> > User-Agent: curl/7.56.1^M >> > Accept: */*^M >> > >> >> >> > 2018/09/08 07:28:39.938| 11,2| http.cc(2261) sendRequest: HTTP Server >> > REQUEST: >> > --------- >> > GET /DcKbz9kqMQXK-zp95pv9LH11kjhTpxOJsJ-1FYEL4 >> > User-Agent: curl/7.56.1^M >> > Accept: */*^M >> > Host: exampletarget.com <http://exampletarget.com>^M >> > Via: 1.1 jb7mgd (squid/4.2)^M >> > Surrogate-Capability: jb7mgd="Surrogate/1.0"^M >> > X-Forwarded-For: **.**.**.**^M >> > Cache-Control: max-age=0^M >> > Connection: keep-alive^M >> > >> > 2018/09/08 07:28:44.359| 11,2| http.cc(723) processReplyHeader: HTTP >> > Server RESPONSE: >> > --------- >> > HTTP/1.1 200 OK >> > Date: Sat, 08 Sep 2018 07:28:40 GMT^M >> > Server: Apache/2.2^M >> > Content-Disposition: attachment; filename=somefile.iso;^M >> > Last-Modified: Fri, 20 Jul 2018 18:23:10 GMT^M >> > ETag: "4a54c59-11653800-571726350bf80"^M >> > Accept-Ranges: bytes^M >> > Content-Length: 291846144^M >> > Keep-Alive: timeout=5, max=100^M >> > Connection: Keep-Alive^M >> > Content-Type: application/unknown^M >> > >> > 2018/09/08 07:28:44.361| 11,2| Stream.cc(267) sendStartOfMessage: HTTP >> > Client REPLY: >> > --------- >> > HTTP/1.1 206 Partial Content^M >> > Date: Sat, 08 Sep 2018 07:28:40 GMT^M >> > Server: Apache/2.2^M >> > Content-Disposition: attachment; filename=somefile.iso;^M >> > Last-Modified: Fri, 20 Jul 2018 18:23:10 GMT^M >> > ETag: "4a54c59-11653800-571726350bf80"^M >> > Accept-Ranges: bytes^M >> > Content-Type: application/unknown^M >> > X-Cache: MISS from jb7mgd^M >> > X-Cache-Lookup: MISS from jb7mgd:3128^M >> > Via: 1.1 jb7mgd (squid/4.2)^M >> > Connection: keep-alive^M >> > Content-Range: bytes 242819145-291846143/291846144^M >> > Content-Length: 49026999^M >> > >> > Thanks, >> > Hari >> > >> > On Sat, Sep 8, 2018 at 9:48 AM Hariharan Sethuraman wrote: >> > >> > Hi Amos, >> > >> > This is what I see when the download is in progress: >> > >> > KEY 44000000000000000902000000000000 >> > STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE >> > RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED >> >> So file stored in memory and scheduled for removal. >> >> > LV:1536379799 LU:1536379801 LM:1532110990 EX:-1 >> > 4 locks, 1 clients, 1 refs >> > Swap Dir -1, File 0XFFFFFFFF >> > GET >> > >> https://example.com/DhAskLOUpvRG2oeR_f_FxYTyLVHIN5esRF-LXOUKwkwyT0TOf6xO-AUm3KaM >> > inmem_lo: 99225582 >> > inmem_hi: 99324372 >> > swapout: 0 bytes queued >> > >> >> Amos >> >>
_______________________________________________ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users