Re: [squid-users] Squid Cache 6.9 on Ubuntu 22.04.3 LTS. Not caching large files to disk.
You need to install certificates as well as on your clients. I don’t normally catch ISO. I only catch updates regarding windows. I have issues where I have to reserve the updates rather than download them over and over again it saves on bandwidth and costs. I wish you the best of luck, however you do need certificates for the ability to catch HTTPS. without certificates, it will not function so you must own the devices as well for this function. Windows is a different story as the updates come over just HTTPtherefore they can be caught without intercepting. I hope that helps if you’re using a transparent proxy. Sent from my iPhone > On Apr 12, 2024, at 09:30, PinPin Poola wrote: > > > Hi Jonathan, > > No, I didn't have a refresh_pattern for .ISO/etc, so thank you. BTW, what are > the "43800 100% 129600" values? > > I realised that I had not actually configured "SSL Bump" in that last > /etc/squid/squid.conf file I posted, as the access.log showed my https > connections as being tunnelled. 🙁 > > I have tried to enable SSL Bump as best I understand how to and my squid.conf > now looks like: > > acl localnet src 0.0.0.1-0.255.255.255 # RFC 1122 "this" network (LAN) > acl localnet src 10.0.0.0/8 # RFC 1918 local private network (LAN) > acl localnet src 100.64.0.0/10 # RFC 6598 shared address space (CGN) > acl localnet src 169.254.0.0/16 # RFC 3927 link-local (directly > plugged) machines > acl localnet src 172.16.0.0/12 # RFC 1918 local private network (LAN) > acl localnet src 192.168.0.0/16 # RFC 1918 local private network (LAN) > acl SSL_ports port 443 > acl Safe_ports port 80 # http > acl Safe_ports port 21 # ftp > acl Safe_ports port 443 # https > acl Safe_ports port 70 # gopher > acl Safe_ports port 210 # wais > acl Safe_ports port 1025-65535 # unregistered ports > acl Safe_ports port 280 # http-mgmt > acl Safe_ports port 488 # gss-http > acl Safe_ports port 591 # filemaker > acl Safe_ports port 777 # multiling http > http_access deny !Safe_ports > http_access deny CONNECT !SSL_ports > http_access allow localhost manager > http_access deny manager > http_access allow localhost > http_access allow localnet > http_access deny to_localhost > http_access deny to_linklocal > include /etc/squid/conf.d/*.conf > http_access deny all > http_port 3128 ssl-bump generate-host-certificates=on > dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myCA.pem > sslcrtd_program /usr/lib/squid/security_file_certgen -s /var/lib/ssl_db -M 4MB > ssl_bump peek all > ssl_bump splice all > coredump_dir /var/spool/squid > refresh_pattern ^ftp: 144020% 10080 > refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 > refresh_pattern . 0 20% 4320 > refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso)(\?|$) 43800 100% 129600 > shutdown_lifetime 10 seconds > maximum_object_size 35 GB > cache_mem 256 MB > maximum_object_size_in_memory 512 KB > cache_replacement_policy heap LFUDA > range_offset_limit -1 > quick_abort_min -1 KB > cache_dir aufs /var/spool/squid 15 16 256 min-size=1048576 > > I read in one blog that the cache_dir had to be listed after > maximum_object_size so I moved it. > > I also reduced the cache_dir / min-size value from 1 GB to 1 MB for testing > and switched to a smaller .ISO file as I was getting bored wating for the big > one to download repeatedly. > > So now: > > 1) A https download works, but is still tunnelled as mentioned above: > > root@client1 [ /tmp ]# wget -e https_proxy=10.40.1.250:3128 --ca-certificate > ~/myCA.pem > https://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso > --2024-04-12 15:42:44-- > https://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso > Connecting to 10.40.1.250:3128... connected. > Proxy request sent, awaiting response... 200 OK > Length: 1016070144 (969M) [application/x-iso9660-image] > Saving to: ‘ubuntu-18.04.6-live-server-amd64.iso’ > > ubuntu-18.04.6-live-server-amd64.iso > 100%[===>] > 969.00M 20.0MB/sin 53s > > 2024-04-12 15:43:37 (18.4 MB/s) - ‘ubuntu-18.04.6-live-server-amd64.iso’ > saved [1016070144/1016070144] > > and the access.log entry looks like this: > > 1712936617.285 52629 10.40.1.2 TCP_TUNNEL/200 1017438604 CONNECT > releases.ubuntu.com:443 - HIER_DIRECT/185.125.190.40 - > > > 2) A new http download works and is cached to disk now: > > root@client1 [ /tmp ]# wget -e http_proxy=10.40.1.250:3128 > http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso > --2024-04-12 15:44:15-- > http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso > Connecting to 10.40.1.250:3128... connected. > Proxy request sent, awaiting response... 200 OK > Length: 1016070144 (969M) [application/x-
Re: [squid-users] Squid Cache 6.9 on Ubuntu 22.04.3 LTS. Not caching large files to disk.
Hi Jonathan, No, I didn't have a refresh_pattern for .ISO/etc, so thank you. BTW, what are the "43800 100% 129600" values? I realised that I had not actually configured "SSL Bump" in that last /etc/squid/squid.conf file I posted, as the access.log showed my https connections as being tunnelled. 🙁 I have tried to enable SSL Bump as best I understand how to and my squid.conf now looks like: acl localnet src 0.0.0.1-0.255.255.255 # RFC 1122 "this" network (LAN) acl localnet src 10.0.0.0/8 # RFC 1918 local private network (LAN) acl localnet src 100.64.0.0/10 # RFC 6598 shared address space (CGN) acl localnet src 169.254.0.0/16 # RFC 3927 link-local (directly plugged) machines acl localnet src 172.16.0.0/12 # RFC 1918 local private network (LAN) acl localnet src 192.168.0.0/16 # RFC 1918 local private network (LAN) acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager http_access allow localhost http_access allow localnet http_access deny to_localhost http_access deny to_linklocal include /etc/squid/conf.d/*.conf http_access deny all http_port 3128 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myCA.pem sslcrtd_program /usr/lib/squid/security_file_certgen -s /var/lib/ssl_db -M 4MB ssl_bump peek all ssl_bump splice all coredump_dir /var/spool/squid refresh_pattern ^ftp: 144020% 10080 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso)(\?|$) 43800 100% 129600 shutdown_lifetime 10 seconds maximum_object_size 35 GB cache_mem 256 MB maximum_object_size_in_memory 512 KB cache_replacement_policy heap LFUDA range_offset_limit -1 quick_abort_min -1 KB cache_dir aufs /var/spool/squid 15 16 256 min-size=1048576 I read in one blog that the cache_dir had to be listed after maximum_object_size so I moved it. I also reduced the cache_dir / min-size value from 1 GB to 1 MB for testing and switched to a smaller .ISO file as I was getting bored wating for the big one to download repeatedly. So now: 1) A https download works, but is still tunnelled as mentioned above: root@client1 [ /tmp ]# wget -e https_proxy=10.40.1.250:3128 --ca-certificate ~/myCA.pem https://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso --2024-04-12 15:42:44-- https://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso Connecting to 10.40.1.250:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 1016070144 (969M) [application/x-iso9660-image] Saving to: ‘ubuntu-18.04.6-live-server-amd64.iso’ ubuntu-18.04.6-live-server-amd64.iso 100%[===>] 969.00M 20.0MB/sin 53s 2024-04-12 15:43:37 (18.4 MB/s) - ‘ubuntu-18.04.6-live-server-amd64.iso’ saved [1016070144/1016070144] and the access.log entry looks like this: 1712936617.285 52629 10.40.1.2 TCP_TUNNEL/200 1017438604 CONNECT releases.ubuntu.com:443 - HIER_DIRECT/185.125.190.40 - 2) A new http download works and is cached to disk now: root@client1 [ /tmp ]# wget -e http_proxy=10.40.1.250:3128 http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso --2024-04-12 15:44:15-- http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso Connecting to 10.40.1.250:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 1016070144 (969M) [application/x-iso9660-image] Saving to: ‘ubuntu-18.04.6-live-server-amd64.iso.1’ ubuntu-18.04.6-live-server-amd64.iso.1 100%[===>] 969.00M 16.0MB/sin 52s 2024-04-12 15:45:07 (18.6 MB/s) - ‘ubuntu-18.04.6-live-server-amd64.iso.1’ saved [1016070144/1016070144] and the access.log entry looks like this: 1712936707.689 52198 10.40.1.2 TCP_MISS/200 1016070508 GET http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso - HIER_DIRECT/185.125.190.40 application/x-iso9660-image 3) A subsequent http download of the same file does pull it from cache: root@client1 [ /tmp ]# wget -e http_proxy=10.40.1.250:3128 http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso --2024-04-12 15:45:23-- http://releases.ubuntu.com/18.04.6/ubuntu-18.04.6-live-server-amd64.iso Conne
[squid-users] Squid Cache 6.9 on Ubuntu 22.04.3 LTS. Not caching large files to disk.
I have moved on a pace since my first message yesterday - thank you all who helped. I can now happily download files from clients on my isolated network, through my new proxy. #fanfare However, I would really like to cache any file over 1 GB in size to disk, as the same file could get downloaded 100's of time a day by many different clients. The cache can purge/age out after a week or so, or when getting close to the 150 GB limit. I have configured cache_dir as below, but when I download a large 2 GB ISO file, I do not see it being cached within the /var/spool/squid directory structure and a subsequent download of the same file is no faster; so it is coming from Internet source. My full /etc/squid/squid.conf file looks like this: acl localnet src 0.0.0.1-0.255.255.255 # RFC 1122 "this" network (LAN) acl localnet src 10.0.0.0/8 # RFC 1918 local private network (LAN) acl localnet src 100.64.0.0/10 # RFC 6598 shared address space (CGN) acl localnet src 169.254.0.0/16 # RFC 3927 link-local (directly plugged) machines acl localnet src 172.16.0.0/12 # RFC 1918 local private network (LAN) acl localnet src 192.168.0.0/16 # RFC 1918 local private network (LAN) acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager http_access allow localhost http_access allow localnet http_access deny to_localhost http_access deny to_linklocal include /etc/squid/conf.d/*.conf http_access deny all http_port 3128 coredump_dir /var/spool/squid refresh_pattern ^ftp: 144020% 10080 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 shutdown_lifetime 10 seconds maximum_object_size 35 GB cache_dir aufs /var/spool/squid 15 16 256 min-size=1073741824 cache_mem 256 MB maximum_object_size_in_memory 512 KB cache_replacement_policy heap LFUDA range_offset_limit -1 quick_abort_min -1 KB I have plenty of disk space on my root partition: Filesystem Size Used Avail Use% Mounted on tmpfs 2.4G 1.2M 2.4G 1% /run /dev/mapper/ubuntu--vg-ubuntu--lv 364G 8.3G 341G 3% / tmpfs 12G 12K 12G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sda2 974M 252M 656M 28% /boot tmpfs 2.4G 4.0K 2.4G 1% /run/user/1000 I would really appreciate any pointers on what I am doing wrong? This is a test setup for now; so if there are security/best practice concerns about my config, I would like to be aware; but I need to get it working for now. Many Thanks Pin ___ squid-users mailing list squid-users@lists.squid-cache.org https://lists.squid-cache.org/listinfo/squid-users
