Hi Amos, All, I continue to struggle with only receiving TCP_MISS when using squid-cache. I have now implemented a server which is responding to HTTP GETs. I have checked the endpoint with redbot.org and it states that :
1/The response allows all caches to store it 2/The response is fresh until 7 days from now The reason may still be served by a cache once it becomes stale. Therefore I believe the server side of this is now valid as a test for squid cache. My client, is routing calls to squidcache, and when it arrives there, I see the following log entry: 1666355825.841 106 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json Infact every single client call I make results in this TCP_MISS. I therefore think this may be to do with my squid.conf. Please see my squid conf below – paying particular attention to the refresh patterns I added specifically for this call (because it wasn’t working without these so I thought I would add these to see if it would work). Squid.conf: http_port 3128 acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 1025-65535 # unregistered ports acl CONNECT method CONNECT http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager # # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS # http_access allow localnet http_access allow localhost http_access deny all coredump_dir /squid/var/cache/squid cache_dir ufs /var/spool/squid 1024 16 256 refresh_pattern -i http:\/\/192.168.20.1:3001\/.* 10080 100% 43200 override-lastmod refresh_pattern -i http:\/\/192.168.20.1:3004\/.* 10080 100% 43200 override-lastmod refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 The complete output from the logs is : admin@zcxsys01:~$ docker logs squid-container 2022/10/21 12:16:58| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback. 2022/10/21 12:16:58| Set Current Directory to /etc/squid 2022/10/21 12:16:58| Creating missing swap directories 2022/10/21 12:16:58| /var/spool/squid exists 2022/10/21 12:16:58| /var/spool/squid/00 exists 2022/10/21 12:16:58| Making directories in /var/spool/squid/00 2022/10/21 12:16:58| /var/spool/squid/01 exists 2022/10/21 12:16:58| Making directories in /var/spool/squid/01 2022/10/21 12:16:58| /var/spool/squid/02 exists 2022/10/21 12:16:58| Making directories in /var/spool/squid/02 2022/10/21 12:16:58| /var/spool/squid/03 exists 2022/10/21 12:16:58| Making directories in /var/spool/squid/03 2022/10/21 12:16:58| /var/spool/squid/04 exists 2022/10/21 12:16:58| Making directories in /var/spool/squid/04 2022/10/21 12:16:59| /var/spool/squid/05 exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/05 2022/10/21 12:16:59| /var/spool/squid/06 exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/06 2022/10/21 12:16:59| /var/spool/squid/07 exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/07 2022/10/21 12:16:59| /var/spool/squid/08 exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/08 2022/10/21 12:16:59| /var/spool/squid/09 exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/09 2022/10/21 12:16:59| /var/spool/squid/0A exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/0A 2022/10/21 12:16:59| /var/spool/squid/0B exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/0B 2022/10/21 12:16:59| /var/spool/squid/0C exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/0C 2022/10/21 12:16:59| /var/spool/squid/0D exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/0D 2022/10/21 12:16:59| /var/spool/squid/0E exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/0E 2022/10/21 12:16:59| /var/spool/squid/0F exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/0F 2022/10/21 12:16:59| Removing PID file (/run/squid.pid) 2022/10/21 12:16:59| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback. 2022/10/21 12:16:58| Set Current Directory to /etc/squid 2022/10/21 12:16:58| Creating missing swap directories 2022/10/21 12:16:58| /var/spool/squid exists 2022/10/21 12:16:58| /var/spool/squid/00 exists 2022/10/21 12:16:58| Making directories in /var/spool/squid/00 2022/10/21 12:16:58| /var/spool/squid/01 exists 2022/10/21 12:16:58| Making directories in /var/spool/squid/01 2022/10/21 12:16:58| /var/spool/squid/02 exists 2022/10/21 12:16:58| Making directories in /var/spool/squid/02 2022/10/21 12:16:58| /var/spool/squid/03 exists 2022/10/21 12:16:58| Making directories in /var/spool/squid/03 2022/10/21 12:16:58| /var/spool/squid/04 exists 2022/10/21 12:16:58| Making directories in /var/spool/squid/04 2022/10/21 12:16:59| /var/spool/squid/05 exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/05 2022/10/21 12:16:59| /var/spool/squid/06 exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/06 2022/10/21 12:16:59| /var/spool/squid/07 exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/07 2022/10/21 12:16:59| /var/spool/squid/08 exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/08 2022/10/21 12:16:59| /var/spool/squid/09 exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/09 2022/10/21 12:16:59| /var/spool/squid/0A exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/0A 2022/10/21 12:16:59| /var/spool/squid/0B exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/0B 2022/10/21 12:16:59| /var/spool/squid/0C exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/0C 2022/10/21 12:16:59| /var/spool/squid/0D exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/0D 2022/10/21 12:16:59| /var/spool/squid/0E exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/0E 2022/10/21 12:16:59| /var/spool/squid/0F exists 2022/10/21 12:16:59| Making directories in /var/spool/squid/0F 2022/10/21 12:16:59| Removing PID file (/run/squid.pid) 2022/10/21 12:16:59| Set Current Directory to /etc/squid 2022/10/21 12:16:59| Starting Squid Cache version 5.2 for s390x-ibm-linux-gnu... 2022/10/21 12:16:59| Service Name: squid 2022/10/21 12:16:59| Process ID 11 2022/10/21 12:16:59| Process Roles: master worker 2022/10/21 12:16:59| With 1048576 file descriptors available 2022/10/21 12:16:59| Initializing IP Cache... 2022/10/21 12:16:59| DNS Socket created at 0.0.0.0, FD 8 2022/10/21 12:16:59| Adding nameserver 10.11.5.1 from /etc/resolv.conf 2022/10/21 12:16:59| Adding nameserver 10.11.5.2 from /etc/resolv.conf 2022/10/21 12:16:59| Adding domain test.sys.one from /etc/resolv.conf 2022/10/21 12:16:59| Logfile: opening log daemon:/var/log/squid/access.log 2022/10/21 12:16:59| Logfile Daemon: opening log /var/log/squid/access.log 2022/10/21 12:17:00| Unlinkd pipe opened on FD 14 2022/10/21 12:17:00| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec 2022/10/21 12:17:00| Store logging disabled 2022/10/21 12:17:00| Swap maxSize 1048576 + 262144 KB, estimated 100824 objects 2022/10/21 12:17:00| Target number of buckets: 5041 2022/10/21 12:17:00| Using 8192 Store buckets 2022/10/21 12:17:00| Max Mem size: 262144 KB 2022/10/21 12:17:00| Max Swap size: 1048576 KB 2022/10/21 12:17:00| Rebuilding storage in /var/spool/squid (dirty log) 2022/10/21 12:17:00| Using Least Load store dir selection 2022/10/21 12:17:00| Set Current Directory to /etc/squid 2022/10/21 12:17:00| Finished loading MIME types and icons. 2022/10/21 12:17:00| HTCP Disabled. 2022/10/21 12:17:00| Pinger socket opened on FD 19 2022/10/21 12:17:00| Squid plugin modules loaded: 0 2022/10/21 12:17:00| Adaptation support is off. 2022/10/21 12:17:00| Accepting HTTP Socket connections at conn2 local=0.0.0.0:3128 remote=[::] FD 17 flags=9 2022/10/21 12:17:00| Done reading /var/spool/squid swaplog (0 entries) 2022/10/21 12:17:00| Store rebuilding is 0.00% complete 2022/10/21 12:17:00| Finished rebuilding storage from disk. 2022/10/21 12:17:00| 0 Entries scanned 2022/10/21 12:17:00| 0 Invalid entries. 2022/10/21 12:17:00| 0 With invalid flags. 2022/10/21 12:17:00| 0 Objects loaded. 2022/10/21 12:17:00| 0 Objects expired. 2022/10/21 12:17:00| 0 Objects cancelled. 2022/10/21 12:17:00| 0 Duplicate URLs purged. 2022/10/21 12:17:00| 0 Swapfile clashes avoided. 2022/10/21 12:17:00| Took 0.12 seconds ( 0.00 objects/sec). 2022/10/21 12:17:00| Beginning Validation Procedure 2022/10/21 12:17:00| Completed Validation Procedure 2022/10/21 12:17:00| Validated 0 Entries 2022/10/21 12:17:00| store_swap_size = 0.00 KB 2022/10/21 12:17:00| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback. 2022/10/21 12:17:00| pinger: Initialising ICMP pinger ... 2022/10/21 12:17:00| pinger: ICMP socket opened. 2022/10/21 12:17:00| pinger: ICMPv6 socket opened 2022/10/21 12:17:01| storeLateRelease: released 0 objects 1666355216.966 213 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355220.808 93 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355221.369 66 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355222.770 254 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355224.514 82 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355225.518 80 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355225.963 65 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355226.366 64 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355226.715 69 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355227.371 315 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355227.700 56 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355239.913 207 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355244.729 96 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355245.014 63 10.1.1.70 TCP_MISS/200 60615 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355408.390 208 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355426.040 197 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355426.650 63 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355427.196 65 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355427.565 60 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355427.915 64 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355824.295 209 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355825.153 120 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json 1666355825.841 106 10.1.1.70 TCP_MISS/200 60646 GET http://192.168.20.1:3004/largeapp/largeapp/007005/bongo? - HIER_DIRECT/192.168.20.1 application/json I suspect something is wrong with my config – perhaps the rules are not working correctly. Please can you offer some advice as to why I can’t get this response to cache in squid-cache. Kind regards, Andy Armstrong 安迪 阿姆斯特朗 Principal Specialist for Z Technologies EMEA Squad Leader for Hybrid Cloud Worldwide Community Leader for Hybrid Cloud Member of the CTO Office Server & Storage EMEA Distinguished Technical Specialist – The Open Group IBM Master Inventor Mobile: +447500103874 From: squid-users <squid-users-boun...@lists.squid-cache.org> on behalf of Andy Armstrong <andy.armstr...@uk.ibm.com> Date: Thursday, 29 September 2022 at 15:00 To: Amos Jeffries <squ...@treenet.co.nz>, squid-users@lists.squid-cache.org <squid-users@lists.squid-cache.org> Subject: [EXTERNAL] Re: [squid-users] TCP_MISS only Hi, Excellent I understand and agree with what you are saying. Is this behaviour documented within the Squid documentation anywhere, or is this more ‘how does the HTTP specification handle caching’? ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, Excellent I understand and agree with what you are saying. Is this behaviour documented within the Squid documentation anywhere, or is this more ‘how does the HTTP specification handle caching’? I am moving forward with a HTTP GET to see if that works per my use case. I assume therefore that any other verb is simply not going to work out the box? Kind regards, Andy Armstrong 安迪 阿姆斯特朗 Principal Specialist for Z Technologies EMEA Squad Leader for Hybrid Cloud Worldwide Community Leader for Hybrid Cloud Member of the CTO Office Server & Storage EMEA Distinguished Technical Specialist – The Open Group IBM Master Inventor Mobile: +447500103874 From: squid-users <squid-users-boun...@lists.squid-cache.org> on behalf of Amos Jeffries <squ...@treenet.co.nz> Date: Thursday, 29 September 2022 at 13:06 To: squid-users@lists.squid-cache.org <squid-users@lists.squid-cache.org> Subject: [EXTERNAL] Re: [squid-users] TCP_MISS only On 28/09/22 07:56, Andy Armstrong wrote: > Okay – but what happens if you are communicating with a non REST > endpoint. You are still communicating over HTTP. To interact with and benefit from HTTP agents like caches you need to comply to the HTTP semantics they use. IMO, REST is just a useful tool to define (in abstract) an API's operation when considering what/how it needs to be implemented. > Consider a Web services endpoint for example where a request > is only interacted with via POST but the operation for example may > frequently be a read based function akin to a HTTP GET? That is by definition a broken implementation of HTTP. The agent is using a *delivery* API (POST) for retrieval (GET). If you can separate the delivery and fetch operations HTTP becomes much easier to use. > Is Squid just > simply not going to help cache those requests? Not *by default*, no. POST implies changing some arbitrary resource *other* than the URL presented. Based on data and logic which may not be provided in the request message URL+headers. To use POST with caching both the client *and* the server have to explicitly tell the HTTP cache agent(s) what to do on every single HTTP message. - The client has to tell the cache whether a stored response is able to be produced as reply, what object-ID it is trying to retrieve, what object-ID's it already knows about (if any), and how old the stored object is allowed to be. - The server has to tell the cache whether the response can be stored, what to use for a unique-ID of the reply object, how old it already is, how long it can be stored for, how and when to update it when it becomes stale. The Squid refresh_pattern can provide defaults for the storage times when they are omitted. But all the ID related things and whether to use cache at all can only come from the client/server. As you can see by limiting yourself to POST-only you have imposed a huge amount of complexity. Using GET instead for fetches makes all the above *optional* where now it is mandatory. > It is only helpful for > more strict alignment to REST principles? > You lost me here. Squid implements HTTP. REST is a very abstract simplification of basic HTTP/1.0 semantics. So the closer ones code aligns to REST the *easier* it is to implement HTTP properly. But HTTP/1.1+ are vastly more than REST. HTH Amos _______________________________________________ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users<http://lists.squid-cache.org/listinfo/squid-users> Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
_______________________________________________ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users