Re: [squid-users] analyzing cache in and out files
Em 30/09/15 04:13, Matus UHLAR - fantomas escreveu: the problem was iirc in caching partial objects http://wiki.squid-cache.org/Features/PartialResponsesCaching that problem could be avoided with properly setting range_offset_limit http://www.squid-cache.org/Doc/config/range_offset_limit/ but that also means that whole files instead of just their parts are fetched. it's quite possible that microsoft changed the windows updates to be smaller files, but I don't know anything about this, so I wonder if you really do cache windows updates, and how does the caching work related to informations above... On 30.09.15 11:08, Leonardo Rodrigues wrote: yes, i'm definitely caching windows update files !! [root@firewall ~]# cd /var/squid/ [root@firewall squid]# for i in `find . -type f`; do strings $i | head -3 | grep "http://";; done | grep windowsupdate | wc -l 824 and yes, i had to configure range_offset_limit: range_offset_limit 500 MB updates minimum_object_size 500 KB maximum_object_size 500 MB quick_abort_min -1 (being 'updates' the ACL with the URLs to be cached, basically windowsupdate and avast definition updates - the second one required further tweaks with storeid_rewrite for the CDN URLs) of course... BTW at one of my customers I noticed downloading the same HUGE files multiple times a day from a few machines - comodo antivirus. Some of updates have wven worse design... from access.log, i see a lot of TCP_HIT/206 (and just a few TCP_HIT/200), so it seems squid is able to get the fully cached file and provide the smaller pieces requested: [root@firewall squid]# grep "TCP_HIT/" access.log | grep windowsupdate | wc -l 9860 [root@firewall squid]# bzcat access.log.20150927.bz2 | grep "TCP_HIT/" | grep windowsupdate | wc -l 38584 can you provide maximum size of those files? having squid to download the WHOLE file at the very first request (even a partial request) may be bad, but considering it will be used later to provide the data for other requests, even partial ones, make things a little better. (this windowsupdate caching is running just for a few weeks, i expect HITs to grow a little more) watching this and providing information would be nice from you... -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Micro$oft random number generator: 0, 0, 0, 4.33e+67, 0, 0, 0... ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] analyzing cache in and out files
Em 30/09/15 04:13, Matus UHLAR - fantomas escreveu: the problem was iirc in caching partial objects http://wiki.squid-cache.org/Features/PartialResponsesCaching that problem could be avoided with properly setting range_offset_limit http://www.squid-cache.org/Doc/config/range_offset_limit/ but that also means that whole files instead of just their parts are fetched. it's quite possible that microsoft changed the windows updates to be smaller files, but I don't know anything about this, so I wonder if you really do cache windows updates, and how does the caching work related to informations above... yes, i'm definitely caching windows update files !! [root@firewall ~]# cd /var/squid/ [root@firewall squid]# for i in `find . -type f`; do strings $i | head -3 | grep "http://";; done | grep windowsupdate | wc -l 824 and yes, i had to configure range_offset_limit: range_offset_limit 500 MB updates minimum_object_size 500 KB maximum_object_size 500 MB quick_abort_min -1 (being 'updates' the ACL with the URLs to be cached, basically windowsupdate and avast definition updates - the second one required further tweaks with storeid_rewrite for the CDN URLs) from access.log, i see a lot of TCP_HIT/206 (and just a few TCP_HIT/200), so it seems squid is able to get the fully cached file and provide the smaller pieces requested: [root@firewall squid]# grep "TCP_HIT/" access.log | grep windowsupdate | wc -l 9860 [root@firewall squid]# bzcat access.log.20150927.bz2 | grep "TCP_HIT/" | grep windowsupdate | wc -l 38584 having squid to download the WHOLE file at the very first request (even a partial request) may be bad, but considering it will be used later to provide the data for other requests, even partial ones, make things a little better. (this windowsupdate caching is running just for a few weeks, i expect HITs to grow a little more) -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email gertru...@solutti.com.br My SPAMTRAP, do not email it ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] analyzing cache in and out files
Em 29/09/15 10:46, Matus UHLAR - fantomas escreveu: hmm, when did this change? IIRC that was big problem since updates use huge files and fetch only parts of them, which squid wasn't able to cache. But i'm off for a few years, maybe M$ finally fixed that up... On 29.09.15 13:57, Leonardo Rodrigues wrote: i'm not a squid expert, but it seems that things became much easier when squid becames fully HTTP/1.1 compliant. that wasn;t the problem... Caching huge files do not changed, that's needed for caching Windows Update files. Storage space, however, is becaming cheaper every year. In my setup, for example, i'm caching files up to 500Mb, i have absolutely no intention of caching ALL Windows Update files. the problem was iirc in caching partial objects http://wiki.squid-cache.org/Features/PartialResponsesCaching that problem could be avoided with properly setting range_offset_limit http://www.squid-cache.org/Doc/config/range_offset_limit/ but that also means that whole files instead of just their parts are fetched. it's quite possible that microsoft changed the windows updates to be smaller files, but I don't know anything about this, so I wonder if you really do cache windows updates, and how does the caching work related to informations above... -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Microsoft dick is soft to do no harm ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] analyzing cache in and out files
On 29/09/2015 20:51, Leonardo Rodrigues wrote: That's i was afraid, there's no tools to analyze the data. Anyway, thanks for the answer. These can be written. First there is a need to actually write the goal of the tool. Then learn the structure of the log.. then write a small app. I can think of one simple tool that reads a store.log file and can tell what was cached and was not erased until now aka still in the cache. Eliezer ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] analyzing cache in and out files
Em 28/09/15 17:55, Amos Jeffries escreveu: The store.log is the one recording what gets added and removed from cache. It is just that there are no available tools to do the analysis you are asking for. Most admin (and thus tools aimed at them) are more concerned with whether cached files are re-used (HITs and near-HITs) or not. That is recorded in the access.log and almost all analysis tools use that log in one format or another. That's i was afraid, there's no tools to analyze the data. Anyway, thanks for the answer. -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email gertru...@solutti.com.br My SPAMTRAP, do not email it ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] analyzing cache in and out files
Em 29/09/15 10:46, Matus UHLAR - fantomas escreveu: hmm, when did this change? IIRC that was big problem since updates use huge files and fetch only parts of them, which squid wasn't able to cache. But i'm off for a few years, maybe M$ finally fixed that up... i'm not a squid expert, but it seems that things became much easier when squid becames fully HTTP/1.1 compliant. Caching huge files do not changed, that's needed for caching Windows Update files. Storage space, however, is becaming cheaper every year. In my setup, for example, i'm caching files up to 500Mb, i have absolutely no intention of caching ALL Windows Update files. -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email gertru...@solutti.com.br My SPAMTRAP, do not email it ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] analyzing cache in and out files
On 28.09.15 15:59, Leonardo Rodrigues wrote: I have a running squid that, until some weeks ago, was not doing any kind of cache, it was just used for access controle rules. Now i have enabled it for windows updateand some specificURLs caching and it's just working fine. Em 29/09/15 07:42, Matus UHLAR - fantomas escreveu: windows updates are so badly designed that the only sane way to get them cached it running windows update server (WSUS). On 29.09.15 09:50, Leonardo Rodrigues wrote: WSUS works for corporate environments, not for all the others. And caching Windows Update with squid is pretty trivial actually, it doesnt even need URL rewriting as other services, youtube for example, do. And it works just fine !! hmm, when did this change? IIRC that was big problem since updates use huge files and fetch only parts of them, which squid wasn't able to cache. But i'm off for a few years, maybe M$ finally fixed that up... -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Linux is like a teepee: no Windows, no Gates and an apache inside... ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] analyzing cache in and out files
Em 29/09/15 07:42, Matus UHLAR - fantomas escreveu: On 28.09.15 15:59, Leonardo Rodrigues wrote: I have a running squid that, until some weeks ago, was not doing any kind of cache, it was just used for access controle rules. Now i have enabled it for windows updateand some specificURLs caching and it's just working fine. windows updates are so badly designed that the only sane way to get them cached it running windows update server (WSUS). WSUS works for corporate environments, not for all the others. And caching Windows Update with squid is pretty trivial actually, it doesnt even need URL rewriting as other services, youtube for example, do. And it works just fine !! -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email gertru...@solutti.com.br My SPAMTRAP, do not email it ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] analyzing cache in and out files
On 28.09.15 15:59, Leonardo Rodrigues wrote: I have a running squid that, until some weeks ago, was not doing any kind of cache, it was just used for access controle rules. Now i have enabled it for windows updateand some specificURLs caching and it's just working fine. windows updates are so badly designed that the only sane way to get them cached it running windows update server (WSUS). -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Linux IS user friendly, it's just selective who its friends are... ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] analyzing cache in and out files
On 29/09/2015 7:59 a.m., Leonardo Rodrigues wrote: > > Hi, > > I have a running squid that, until some weeks ago, was not doing any > kind of cache, it was just used for access controle rules. Now i have > enabled it for windows updateand some specificURLs caching and it's just > working fine. > > I was looking, however, for a way of tracking files that are getting > into the cache and excluded fromit. At first, i tough store_log would be > the way, but the comment on cache_store_log default squid.conf file > dissapointed me:"There are not really utilities to analyze this data" > > Which log coud i enable, if there's any, to help me analyze files > (and its URLs) getting into and out of the cache dirs ?? I'm using squid > 3.5.8 btw. The store.log is the one recording what gets added and removed from cache. It is just that there are no available tools to do the analysis you are asking for. Most admin (and thus tools aimed at them) are more concerned with whether cached files are re-used (HITs and near-HITs) or not. That is recorded in the access.log and almost all analysis tools use that log in one format or another. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] analyzing cache in and out files
Hi, I have a running squid that, until some weeks ago, was not doing any kind of cache, it was just used for access controle rules. Now i have enabled it for windows updateand some specificURLs caching and it's just working fine. I was looking, however, for a way of tracking files that are getting into the cache and excluded fromit. At first, i tough store_log would be the way, but the comment on cache_store_log default squid.conf file dissapointed me:"There are not really utilities to analyze this data" Which log coud i enable, if there's any, to help me analyze files (and its URLs) getting into and out of the cache dirs ?? I'm using squid 3.5.8 btw. Thanks ! -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email gertru...@solutti.com.br My SPAMTRAP, do not email it ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
