Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Eliezer Croitoru wrote > You can try to use the atime and not the mtime. Each time the fetcher script runs , all of request files will access and then atime will refreshed . I think for "request" directory it should be "mtime" and for "body" directory it should be "atime" . Eliezer Croitoru wrote > It is possible that some fetchers will consume lots of memory and some of > the requests are indeed un-needed but... don’t delete them. > Try to archive them and only then remove from them some by their age or > something similar. > Once you have the request you have the option to fetch files and since > it's such a small thing(max 64k per request) it's better to save and > archive first and later wonder if some file request is missing. But currently there is more than 23 files in old request directory . Maybe the garbage collector of GoLang will not release the memory after processing each file . Eliezer Croitoru wrote > * if you want me to test or analyze your archived requests archive them > inside a xz and send them over to me. I have sent you the request directory in previous private email . Thanks -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4682360.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Technically it depends on the GoLang Garbage collection. I will try to upgrade the software to GoLang V 1.8.1 in the next days but no promises on a specific date. The next article might help you: https://www.howtogeek.com/howto/ubuntu/delete-files-older-than-x-days-on-linux/ You can try to use the atime and not the mtime. In any case if you are running out of memory please dump the top -n1 into a file and then use some paste or attach the file so I would be able to read it since the lines are in a way not in a form it would be easy for me to read. It is possible that some fetchers will consume lots of memory and some of the requests are indeed un-needed but... don’t delete them. Try to archive them and only then remove from them some by their age or something similar. Once you have the request you have the option to fetch files and since it's such a small thing(max 64k per request) it's better to save and archive first and later wonder if some file request is missing. All The Bests, Eliezer * if you want me to test or analyze your archived requests archive them inside a xz and send them over to me. Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Wednesday, May 10, 2017 3:18 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store. I have deleted and recreate the request directory and see huge decrease in memory usage of the fetcher process . Did i do the right thing ? Is there anything that should i do after a while ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4682352.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
I have deleted and recreate the request directory and see huge decrease in memory usage of the fetcher process . Did i do the right thing ? Is there anything that should i do after a while ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4682352.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hello, The old problem appears again because i am running 6 instances at same time. TCP: out of memory -- consider tuning tcp_mem Here is commands you mentioned . root@squidbox:~# free -m totalusedfree shared buff/cache available Mem: 16037 13503 147 1712386 1652 Swap: 8179 5767603 root@squidbox:~# cat /proc/sys/net/ipv4/tcp_mem 381630 508844 763260 root@squidbox:~# top -n1 -b top - 15:07:11 up 2 days, 10:04, 1 user, load average: 3.50, 3.24, 3.13 Tasks: 190 total, 2 running, 188 sleeping, 0 stopped, 0 zombie %Cpu(s): 19.1 us, 6.2 sy, 0.5 ni, 67.5 id, 1.9 wa, 0.0 hi, 4.8 si, 0.0 st KiB Mem : 16422248 total, 166088 free, 13829624 used, 2426536 buff/cache KiB Swap: 8376316 total, 7786056 free, 590260 used. 1690840 avail Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 1620 proxy 20 0 3977368 3.606g 4708 S 46.7 23.0 1272:29 squid 4451 root 20 0 632240 592500584 S 26.7 3.6 404:11.34 ms-updates-fetc 2573 root 20 0 636068 594116612 S 20.0 3.6 403:12.45 ms-updates-fetc 3661 root 20 0 646392 607532500 S 20.0 3.7 427:31.70 ms-updates-fetc 4171 root 20 0 656876 619500604 S 20.0 3.8 460:54.17 ms-updates-fetc 3934 root 20 0 642160 604896492 S 13.3 3.7 411:05.97 ms-updates-fetc 4555 root 20 0 656732 619700 1548 R 13.3 3.8 478:16.31 ms-updates-fetc 14746 root 20 0 40552 3464 2892 R 6.7 0.0 0:00.01 top 1 root 20 0 185784 4972 3112 S 0.0 0.0 0:12.01 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.69 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:16.70 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 7 root 20 0 0 0 0 S 0.0 0.0 7:11.28 rcu_sched 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 9 root rt 0 0 0 0 S 0.0 0.0 0:00.23 migration/0 10 root rt 0 0 0 0 S 0.0 0.0 0:00.54 watchdog/0 11 root rt 0 0 0 0 S 0.0 0.0 0:00.50 watchdog/1 12 root rt 0 0 0 0 S 0.0 0.0 0:00.22 migration/1 13 root 20 0 0 0 0 S 0.0 0.0 0:52.29 ksoftirqd/1 15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H 16 root rt 0 0 0 0 S 0.0 0.0 0:00.48 watchdog/2 17 root rt 0 0 0 0 S 0.0 0.0 0:00.40 migration/2 18 root 20 0 0 0 0 S 0.0 0.0 10:22.44 ksoftirqd/2 20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H 21 root rt 0 0 0 0 S 0.0 0.0 0:00.49 watchdog/3 22 root rt 0 0 0 0 S 0.0 0.0 0:00.24 migration/3 23 root 20 0 0 0 0 S 0.0 0.0 0:16.35 ksoftirqd/3 25 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H 26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs 27 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns 28 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 perf 29 root 20 0 0 0 0 S 0.0 0.0 0:00.13 khungtaskd 30 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback 31 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd 32 root 39 19 0 0 0 S 0.0 0.0 0:35.21 khugepaged 33 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto 34 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd 35 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 36 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd 37 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ata_sff 38 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md 39 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 devfreq_wq 43 root 20 0 0 0 0 S 0.0 0.0 81:22.18 kswapd0 44 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 vmstat 45 root 20 0 0 0 0 S 0.0 0.0 0:00.17 fsnotify_mark 46 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ecryptfs-kthrea 62 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld 63 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 acpi_thermal_pm 64 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 65 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 66 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 67 root 0 -20 0 0 0
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hello, Thanks -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4682189.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Did you got my answer? You should be able to dispatch more then one fetcher but you should somehow manage them and restrict their amount and dispatch rate. Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Saturday, April 15, 2017 2:52 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store. Hello, I have sent the files you mentioned to your email 2 days ago . A little more investigation shows that some big files (~ 2GB ) are downloading slowly ( ~ 100KBytes/s) while some others downloading very faster. The problem is related to networking (BGP and IXP ) stuff and the fetcher script can not solve that . But is there a way to run more than one fetcher script at the same time to parallel downloading and not one by one ? There is free bandwidth but fetcher script takes a long time for some downloads . Thanks again for you support -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4682113.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hello, I have sent the files you mentioned to your email 2 days ago . A little more investigation shows that some big files (~ 2GB ) are downloading slowly ( ~ 100KBytes/s) while some others downloading very faster. The problem is related to networking (BGP and IXP ) stuff and the fetcher script can not solve that . But is there a way to run more than one fetcher script at the same time to parallel downloading and not one by one ? There is free bandwidth but fetcher script takes a long time for some downloads . Thanks again for you support -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4682113.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Thanks for reply. Eliezer Croitoru wrote > Also what is busy for you? The fetcher script is always downloading . For example right now i can see that a fetcher script is running for more than 3 days and it is downloading files one by one . Eliezer Croitoru wrote > Also what is busy for you? > Are you using a lock file ?( it might be possible that your server is > downloading in some loop and this is what causing this load) Yes . Everything looks fine in that mean . Eliezer Croitoru wrote > Did you upgraded to the latest version? Yes I will send you the files . Thanks -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4682033.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
I am not using it daily but I know that MS updates have more then one language support and it's pretty simple to verify the subject. Just send me privately the tar of the requests and headers files and I will try to see if something got changed or not. I do not believe that MS will change their system to such extent but I have not issue looking at the subject and try to verify what is causing for the load on your side. Just so you know that it might take me time to verify the issue. Also what is busy for you? Are you using a lock file ?( it might be possible that your server is downloading in some loop and this is what causing this load) Did you upgraded to the latest version? Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Thursday, April 6, 2017 5:39 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store. Hey Eliezer, Recently i have found that the fetcher script is very busy and it is always downloading . It seems that microsoft changed something . I am not sure and it is just a guess . Whats up at your servers ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4682002.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hey Eliezer, Recently i have found that the fetcher script is very busy and it is always downloading . It seems that microsoft changed something . I am not sure and it is just a guess . Whats up at your servers ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4682002.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hey Omid, For now the software is restricted only to windows updates which is protected and secured enough to sustain caching. About Mozilla, I need to verify it before I am doing anything about it. From my point of view it is hosted on Akamai and HSTS is restricting couple things on their service. I will try to look at it later without any promises. Do you have any starting points else then the domain itself? Have you tried to analyze some logs? Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Tuesday, September 6, 2016 5:48 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store. Hey Eliezer, According to these threads http://squid-web-proxy-cache.1019090.n4.nabble.com/range-offset-limit-not-working-as-expected-td4679355.html http://squid-web-proxy-cache.1019090.n4.nabble.com/TProxy-and-client-dst-passthru-td4670189.html Is there any chance that you implement something that may be used for other (206 partial) popular sites like download.cdn.mozilla.net . I think it has also same problem as windows update and has lots of uncachable requests . Thanks in advance . -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4679373.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hey Eliezer, According to these threads http://squid-web-proxy-cache.1019090.n4.nabble.com/range-offset-limit-not-working-as-expected-td4679355.html http://squid-web-proxy-cache.1019090.n4.nabble.com/TProxy-and-client-dst-passthru-td4670189.html Is there any chance that you implement something that may be used for other (206 partial) popular sites like download.cdn.mozilla.net . I think it has also same problem as windows update and has lots of uncachable requests . Thanks in advance . -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4679373.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hey Omid, I will comment inline. And there are couple details which we need to understand couple issues. Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Monday, July 25, 2016 12:15 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store. Hi, Thanks for support . recently i have seen a problem with version beta 0.2 . when fetcher is working the kernel logs lots of following error TCP: out of memory -- consider tuning tcp_mem # To verify the actual status we need the output of: $ free -m $ cat /proc/sys/net/ipv4/tcp_mem $ top -n1 -b $ cat /proc/net/sockstat $ cat /proc/sys/net/ipv4/tcp_max_orphans I think the problem is about orphaned connections which i mentioned before . Managed to try new version to see what happens. # If you have an orphaned connections on the machine with or without the MS updates proxy, you should consider to analyze the machine structure and load in general. If indeed there are orphan connections we need to verify if it's from the squid or my service or the combination of them together. Also i have a feature request . Please provide a configuration file for example in /etc/foldername or even beside the binary files to have selective options for both fetcher and logger. # With what options for the logger and fetcher? I have seen following change log beta 0.3 - 19/07/2016 + Upgraded the fetcher to honour private and no-store cache-control headers when fetching objects. As my point of view the more hits is better and there is no problem to store private and no-store objects if it helps to achieve more hits and bandwidth saving . So it would be fine to have an option in mentioned config file to change it myself . # I understand your way of looking at things but this is a very wrong way to look at cache and store. The problem with storing private and no-store responses is very simple. These files are temporary and exists for one request only(in most cases). Specifically for MS it is true and they do not use private files more then once. I do not wish to offend you or anyone by not honoring such a request but since it's a public service this is the definition of it. If you want to see the options of the fetcher and the service just add the "-h" option to see the available options. I have considered to use some log file but yet to get to the point which I have a specific format that I want to work with. I will try to see what can be done with log files and also what should be done to handle log rotation. Thanks again ## Resources * http://blog.tsunanet.net/2011/03/out-of-socket-memory.html ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hi, Thanks for support . recently i have seen a problem with version beta 0.2 . when fetcher is working the kernel logs lots of following error TCP: out of memory -- consider tuning tcp_mem I think the problem is about orphaned connections which i mentioned before . Managed to try new version to see what happens. Also i have a feature request . Please provide a configuration file for example in /etc/foldername or even beside the binary files to have selective options for both fetcher and logger . I have seen following change log beta 0.3 - 19/07/2016 + Upgraded the fetcher to honour private and no-store cache-control headers when fetching objects. As my point of view the more hits is better and there is no problem to store private and no-store objects if it helps to achieve more hits and bandwidth saving . So it would be fine to have an option in mentioned config file to change it myself . Thanks again -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678669.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hey Omid, After inspection of more data I have seen that there are couple cases which will result in disks space consumption. Windows Updates supports a variety of languages. When you have more then one or two languages the amount of cache is rapidly changes. To give some numbers to the picture: - Each Windows version have multiple versions(starter, home, professional, enterprise..) - Each cpu arch requires it's own updates(x86, x64) - Each Windows version can have a big update for multiple languages, depends on the locality of the system - Each Windows product such as office has it's own language packs and updates(some updates are huge..) Since I am not one of Microsoft Engineers or product\updates managers I cannot guarantee that my understanding of the subject is solid like the ground. But in the other hand since I do have background with HTTP and it's structure I can guarantee some assurance that my research can be understood by most if not any HTTP expert. Squid by it's nature honors specific caching rules and these are very general. To my understanding Squid was not built to satisfy each use case but it helps many of them. Since you also noticed that windows updates can consume lots of disk space then what you mentioned about last accessed time seems pretty reasonable for a cache. You have the choice on how to manage your store\cache according to whatever is required\needed. For example the command: find /cache1/body/v1/ -atime +7 -type f|wc -l Should give you some details about the files which was not accessed in the last week. We can try to enhance the above command\idea to calculate statistics in a way that will help us to get an idea of what files or updates are downloaded periodically. Currently only with the existence of the request files we can understand what responses belongs to what request. Let me know if you want me to compose some script that will help you to decide what files to purge. (I will probably write it in ruby) There is an option to "blacklist" a response from being fetched by the fetcher or to be used by the web-service but you will need to update to the latest version of the fetcher and to use the right cli option(don't remember now) or to use the command under a "true" pipe such as "true | /location/fetcher ..." to avoid any "pause" which it will cause. Thanks, Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Tuesday, July 19, 2016 1:59 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store. Eliezer Croitoru-2 wrote > Hey Omid, > > Indeed my preference is that if you can ask ask and I will try to give you > couple more details on the service and the subject. Hey Eliezer, 4.Current storage capacity is 500G andmore than 50% of it becomes full and growing fast . Is there any mechanism for garbage collection in your code ? If not is it good idea to remove files based on last access time (ls -ltu /cache1/body/v1/) ? should i also delete old files from header and request folders ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678581.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hey Omid, I will try to answer about the subject in general and it should contain the answers to what you have asked. Windows Updates can somehow be cached when combining squid StoreID and a refresh_pattern. However the nature of squid is to be a "cache", and in many cases since we can predict that we will need specific content more often it would be preferred to store the objects. The "tradeoff" is using the wire and the clients to fetch over and over again the same exact content while assuring consistency and integrity of it. For example most of Windows updates can be publically cached for 48 hours which should be enough for a "torrent"(some would call it DOS) of updates. A refresh_pattern which have the next options: ignore-no-store ignore-reload ignore-private ignore-must-revalidate override-expire override-lastmod Will cause in a way in a reduction of some bandwidth consumption by the clients but it kind of "breaks" some other features of the cache. Since squid 3.X there was a software change inside squid to prefer Integrity over caching due to changes in the nature of the Internet. MS are cache friendly in general and you will probably won't need override-lastmod and couple other options in the refresh_pattern definition. A refresh_pattern location in the squid.conf should not cause any difference on caching but it is important to place them like many FW and ACL rules: first seen and match HIT. This is since squid.conf parser places the refresh_patterns to be validated one at a time and each at a time from top to bottom inside the squid.conf file. To prevent de-duplication of content as Amos advised you should use the "cache" config directive. Take a peek at the docs: http://www.squid-cache.org/Doc/config/cache/ And also the example at: http://wiki.squid-cache.org/SquidFaq/SquidAcl#how_do_I_configure_Squid_not_t o_cache_a_specific_server.3F And remember to add after a cache deny, "cache allow all". About the "msip" acl you have added: It's not really needed and can also cause strange things if some request for\to another domain would be sent to this cache_peer. This is due to this service nature to return a 500 error on requests for non windows update domains. If you notice weird behavior with this store service like space consumption there are couple steps you should take: - stop the crontab of the fetcher(to rebase the situation) - verify that currently there are no stored responses which was supposed to be "private" ie use this script: https://gist.github.com/elico/5ae8920a4fbc813b415f8304cf1786db - verify how many unique requests are stored in the ie "ls /cache1/request/v1/|wc -l" The next step would be to examine the requests dump ie "tar cvfJ requests.tar.xz /cache1/request/v1/" and send me these dumps for analysis. If you need to filter the requests before sending them to me you will need to verify if there are cookies in the requests files. I believe that when we will have the numbers of responses that has private or public cache-control headers it would be much simpler as a start point before the next step. Just to mention that a Garbage Collection operation should be done before the actual full fetch. In my experiments I couldn't find evidence of a situation like you have but I assumed that some networks will have issues in this level. I will enhance my fetcher to avoid private content fetching but to be sure on the right move I need both the statistics and the requests dump. My code is not a cache which manage some level of the expiration or validation of the content. It's a simple http web service which was embedded with a special File System structure and a forward proxy. I will be available tonight at my skype: elico2013 Also on the squid irc channel at irc.freenode.net with the nick: elico And of-course my email. Just contact me so we would be able to understand the situation better. Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Tuesday, July 19, 2016 1:59 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store. Eliezer Croitoru-2 wrote > Hey Omid, > > Indeed my preference is that if you can ask ask and I will try to give you > couple more details on the service and the subject. Hey Eliezer, 1.I have refresh patterns from days before your code . Currently i prefer not to store windows updates in squid internal storage because of de-duplication . Now what should i do ? delete this refresh pattern ? or even create a pattern not to cache windows updates ? refresh_pattern -I (microsoft|windowsupdate)\.com/.*?\.(cab|exe|dll|ms[iuf]|asf|wm[va]|dat|zip| iso|psf)$ 10080 100% 172800 i
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
On 19/07/2016 10:58 p.m., Omid Kosari wrote: > Eliezer Croitoru-2 wrote >> Hey Omid, >> >> Indeed my preference is that if you can ask ask and I will try to give you >> couple more details on the service and the subject. > > Hey Eliezer, > > 1.I have refresh patterns from days before your code . Currently i prefer > not to store windows updates in squid internal storage because of > deduplication . Now what should i do ? delete this refresh pattern ? or even > create a pattern not to cache windows updates ? > > refresh_pattern -i > (microsoft|windowsupdate)\.com/.*?\.(cab|exe|dll|ms[iuf]|asf|wm[va]|dat|zip|iso|psf)$ > 10080 100% 172800 ignore-no-store ignore-reload ignore-private > ignore-must-revalidate override-expire override-lastmod > Either; cache deny ... Or (if your Squid supports it) store_miss deny ... The cache ACLs are again request-only ones. So based on dstdomain of WU services. The store_miss ACLs can be based on request or reply. So nice things like reply Content-Type header etc. can be used. If your refresh_pattern causes something to be a HIT in cache, then the store_miss stuff will never happen of course. Likewise, if the store_miss prevents something being added to cache the refresh_pattern will not then be able to have any effect on its cache entry. > 2.Is the position of your squid config important to prevent logical > conflicts? for example should it be before above refresh patterns to prevent > deduplication ? > > acl wu dstdom_regex \.download\.windowsupdate\.com$ > acl wu-rejects dstdom_regex stats > acl GET method GET > cache_peer 127.0.0.1 parent 8080 0 proxy-only no-tproxy no-digest no-query > no-netdb-exchange name=ms1 > cache_peer_access ms1 allow GET wu !wu-rejects > cache_peer_access ms1 deny all > never_direct allow GET wu !wu-rejects > never_direct deny all For these directives ordering is relevant only with regards to other lines of the same directive name. The exception being cache_peer_access; where the peer name field defines which lines are a sequential group. And the cache_peer definition line must come first. > > 3.Is it good idea to change your squid config as bellow to have more hits? > Or maybe it is big mistake ! > > acl msip dst 13.107.4.50 > acl wu dstdom_regex \.download\.windowsupdate\.com$ > \.download\.microsoft\.com$ > acl wu-rejects dstdom_regex stats > acl GET method GET > cache_peer 127.0.0.1 parent 8080 0 proxy-only no-tproxy no-digest no-query > no-netdb-exchange name=ms1 > cache_peer_access ms1 allow GET wu !wu-rejects > cache_peer_access ms1 allow GET msip !wu-rejects > cache_peer_access ms1 deny all > never_direct allow GET wu !wu-rejects > never_direct allow GET msip !wu-rejects > never_direct deny all Your question here is not clear. None of this config is directly related to HITs. With Eliezers setup HITs are a intentional by-product of the manipulatinoon happening in the peer. So you either use the peer and get what HITs it causes, or you don't. > > 4.Current storage capacity is 500G andmore than 50% of it becomes full and > growing fast . Is there any mechanism for garbage collection in your code ? > If not is it good idea to remove files based on last access time (ls -ltu > /cache1/body/v1/) ? should i also delete old files from header and request > folders ? > I'll leave that to Eliezer to answer. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Eliezer Croitoru-2 wrote > Hey Omid, > > Indeed my preference is that if you can ask ask and I will try to give you > couple more details on the service and the subject. Hey Eliezer, 1.I have refresh patterns from days before your code . Currently i prefer not to store windows updates in squid internal storage because of deduplication . Now what should i do ? delete this refresh pattern ? or even create a pattern not to cache windows updates ? refresh_pattern -i (microsoft|windowsupdate)\.com/.*?\.(cab|exe|dll|ms[iuf]|asf|wm[va]|dat|zip|iso|psf)$ 10080 100% 172800 ignore-no-store ignore-reload ignore-private ignore-must-revalidate override-expire override-lastmod 2.Is the position of your squid config important to prevent logical conflicts? for example should it be before above refresh patterns to prevent deduplication ? acl wu dstdom_regex \.download\.windowsupdate\.com$ acl wu-rejects dstdom_regex stats acl GET method GET cache_peer 127.0.0.1 parent 8080 0 proxy-only no-tproxy no-digest no-query no-netdb-exchange name=ms1 cache_peer_access ms1 allow GET wu !wu-rejects cache_peer_access ms1 deny all never_direct allow GET wu !wu-rejects never_direct deny all 3.Is it good idea to change your squid config as bellow to have more hits? Or maybe it is big mistake ! acl msip dst 13.107.4.50 acl wu dstdom_regex \.download\.windowsupdate\.com$ \.download\.microsoft\.com$ acl wu-rejects dstdom_regex stats acl GET method GET cache_peer 127.0.0.1 parent 8080 0 proxy-only no-tproxy no-digest no-query no-netdb-exchange name=ms1 cache_peer_access ms1 allow GET wu !wu-rejects cache_peer_access ms1 allow GET msip !wu-rejects cache_peer_access ms1 deny all never_direct allow GET wu !wu-rejects never_direct allow GET msip !wu-rejects never_direct deny all 4.Current storage capacity is 500G andmore than 50% of it becomes full and growing fast . Is there any mechanism for garbage collection in your code ? If not is it good idea to remove files based on last access time (ls -ltu /cache1/body/v1/) ? should i also delete old files from header and request folders ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678581.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
On 19/07/2016 5:50 p.m., Omid Kosari wrote: > Also i have seen that another guy did successfully something like that (not > exactly ) in this thread > http://squid-web-proxy-cache.1019090.n4.nabble.com/cache-peer-hit-miss-and-reject-td4661928.html > No. Niki's ACLs had nothing to do with HTTP reply state. The external ACL helper stored and used *request URL* information. The first request for any URL was a fixed value, only the second or later values varied based on the externally stored knowledge. It never predicts anything variable in advance, and never used reply headers. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Also i have seen that another guy did successfully something like that (not exactly ) in this thread http://squid-web-proxy-cache.1019090.n4.nabble.com/cache-peer-hit-miss-and-reject-td4661928.html -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678574.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Alex Rousskov wrote > On 07/18/2016 05:39 AM, Omid Kosari wrote: > >> acl mshit rep_header X-SHMSCDN HIT >> clientside_tos 0x30 mshit > > You cannot use response-based ACLs like rep_header with clientside_tos. > That directive is currently evaluated only at request processing time, > before there is a response. > >> 2016/07/18 16:26:31.927 kid1| WARNING: mshit ACL is used in context >> without >> an HTTP response. Assuming mismatch. > > ... which is what Squid is trying to tell you. > > > HTH, > > Alex. > > ___ > squid-users mailing list > squid-users@.squid-cache > http://lists.squid-cache.org/listinfo/squid-users Apart from that , can you confirm that we may use cutom header in rep_header ? Also the problem is acl mshit does not count att all . -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678566.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
On 07/18/2016 05:39 AM, Omid Kosari wrote: > acl mshit rep_header X-SHMSCDN HIT > clientside_tos 0x30 mshit You cannot use response-based ACLs like rep_header with clientside_tos. That directive is currently evaluated only at request processing time, before there is a response. > 2016/07/18 16:26:31.927 kid1| WARNING: mshit ACL is used in context without > an HTTP response. Assuming mismatch. ... which is what Squid is trying to tell you. HTH, Alex. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
About the mismatch log output I cannot say a thing since I have not researched it. And about an option to add a HIT HEADER you can use the next script: https://gist.github.com/elico/ac58073812b8cad14ef154d8730e22cb Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Monday, July 18, 2016 2:39 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store. Dear Eliezer, Unfortunately no success . I will describe what i did maybe i missed something . run the command perl -pi -e '$/=""; s/\r\n\r\n/\r\nX-SHMSCDN: HIT\r\n\r\n/;' /cache1/header/v1/* and verified that the text injected correctly squid config acl mshit rep_header X-SHMSCDN HIT clientside_tos 0x30 mshit but got the following popular log 2016/07/18 16:26:31.927 kid1| WARNING: mshit ACL is used in context without an HTTP response. Assuming mismatch. 2016/07/18 16:26:31.927 kid1| 28,3| Acl.cc(158) matches: checked: mshit = 0 One more thing . as i am not so familiar with perl , may i ask you to please edit it to ignore the files which already have the text ? Thanks -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678557.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Dear Eliezer, Unfortunately no success . I will describe what i did maybe i missed something . run the command perl -pi -e '$/=""; s/\r\n\r\n/\r\nX-SHMSCDN: HIT\r\n\r\n/;' /cache1/header/v1/* and verified that the text injected correctly squid config acl mshit rep_header X-SHMSCDN HIT clientside_tos 0x30 mshit but got the following popular log 2016/07/18 16:26:31.927 kid1| WARNING: mshit ACL is used in context without an HTTP response. Assuming mismatch. 2016/07/18 16:26:31.927 kid1| 28,3| Acl.cc(158) matches: checked: mshit = 0 One more thing . as i am not so familiar with perl , may i ask you to please edit it to ignore the files which already have the text ? Thanks -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678557.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hey Omid, Indeed my preference is that if you can ask ask and I will try to give you couple more details on the service and the subject. Windows updates are considered very static since they have Last-Modified and Date headers. Else then that, they support to validate and invalidate requests based on these times. This what makes it so simple to store them. Now the main issue(which you probably already know) is that clients are requesting partial content from a full static object. The full file or object is a resource and when you have the full object most web services can serve the partial content. Technically if the client software uses static ranges when accessing the a resource it would be very simple to "map" a range request into a specific object "ID" but the issue is that there are scenarios which the client ask for multiple ranges in the same request and everything get a bit complicated. >From a cache(squid) point of view when a client runs a "fetch" operation he also populate the cache. This is the most "preferred" way of handling cache population since it relies on a need which is somehow considered as required. Now when you look at it, in many cases it's a bit silly and can be considered in a way simple minded when you are talking about GB's of static content. When I look at MS updates I see lots of "Cache-Control: public,max-age=172800" in responses and it might be based on the assumption that the object is predicted to be a part of an update "torrent" of about 48 hours. The real world is far from this header and caches needs to be smarter in order to avoid re-population and re-deletion of existing content. Now since MS updates would be probably used over and over again by real clients it's sometimes would be good to just store them for a period of time. For example there aren't many Windows XP out there under paid support but if clients are still updating then it's right to have them. Now what I did was simply wrote a simple web-service which is also a forward proxy that is based on another file system rather compared to the standard ones. You have a storedata directory which can be changed in the command line. You have a listening port, and you also have some level of debug info. The store data have three important sub directories: request/v1/ header/v1/ response/v1/ Since it's a simple web service that relies on a specific file system structure it doesn't have the TOS and QOS features that are in a much lower level services have. Since you have full control on the web-service and the response headers are reliable you can safely use some kind of Internal response headers and be sure that MS and their CDN network will not use these and will "harm" your statistics. You will just need to use the concept which was mentioned in the HIT-MISS thread from 2012: acl mshit rep_header X-Company-Ms-Cache HIT clientside_tos 0x30 mshit And you can get wild with the name to verify that it will be 100% unique and will not collide with the CDN headers. Also you can use another term then HIT for example "INTERNAL-CACHE-SERVER" would probably not be coming from the up-stream CDN. Or even add a unique ID(#uuidgen) for this service that should never be mimicked. Since it's a web-service with static header files you will just need to use the perl script which I sent you in the last email to inject these headers into the response files. If the store-service is only piping the connection to the up-stream services the response will not include your customized embedded headers. The response headers are at: /var/storedata/header/v1/* The bodies are at: /var/storedata/response/v1/* Just as a side note: this store service was designed for MS updates only and using it for other services is prohibited in the code level. In the mean while I will look at the TOS\QOS options if at all. Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Sunday, July 17, 2016 9:34 AM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store. Dear Eliezer, Thanks for reply . 1. I am trying to understand but with your description it would be simpler 2. I already use logical volumes . Silly question 3. I don't want just a hit in log . I try to describe my need . Currently we have exclude the cache hits (based on TOS value) from our customers reserved bandwidth . For example you have 150Mbps internet link from our company and we have limitation for you on our QOS routers . But we have excluded cache hits from your 150M and you may have more than that if you are downloading from our cache hits . qos_flows local-hit=0x90 qos_flows sibling-hit=0x90 q
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Apart from previous email , maybe this is a bug or not but the fetcher does not release open files/sockets . Its number of open files just grows . currently i have added 'ulimit 65535' at the line 4 of fetch-task.sh to see what happens . before it was killed. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678536.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
It looks like the guy there is having the same request as I have. http://squid-web-proxy-cache.1019090.n4.nabble.com/cache-peer-communication-about-HIT-MISS-between-squid-and-and-non-squid-peer-td4600931.html -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678532.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Dear Eliezer, Thanks for reply . 1. I am trying to understand but with your description it would be simpler 2. I already use logical volumes . Silly question 3. I don't want just a hit in log . I try to describe my need . Currently we have exclude the cache hits (based on TOS value) from our customers reserved bandwidth . For example you have 150Mbps internet link from our company and we have limitation for you on our QOS routers . But we have excluded cache hits from your 150M and you may have more than that if you are downloading from our cache hits . qos_flows local-hit=0x90 qos_flows sibling-hit=0x90 qos_flows parent-hit=0x90 But the hits from your code could not be counted . Even you may help me to do that with linux iptables+squid it would be fine . Thanks -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678530.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hey Omid, 1. You should understand what you are doing and not blindly fetch downloads. The estimation is that you will need maximum of 100GB of storage for the whole "store" for a period of time. This is also due to this that Microsoft Windows Update service will not download files without a need. The fetcher should help you to download periodical updates but I assume that the updates have a limit... You should consider asking MS on what is expected to be in the downloads or when do download happen. 2. If you need more then one location you should use some logical volume to do that instead of spreading manually over more then one disk. This is based on the basic understanding that the service is a "web-service" which is serving files and you should treat it the same way like any other. When I am running a web-service and I need more then one disk I do not run to "spread" it manually but use some OS level tools. I do trust the OS and the logical volume management tools to do their work properly. When I will loss my trust in them I will stop using this OS, this is as simple as that. 3. The HITS are counted but I need to dig into the code to verify how a HIT is logged and how it can be counted manually. QOS or TOS, by what? How? The service how one way out and one way in.. If the requested file is in store you will not see outgoing traffic for the file. The right way to show a HIT in this service is to change the response headers file to have another header. This could be done manually using a tiny script but not as a part of the store software. An example to such addition would be: # perl -pi -e '$/=""; s/\r\n\r\n/\r\nX-Store-Hit: HIT\r\n\r\n/; /var/storedata/header/v1/fff8db4723842074ab8d8cc4ad20a0f97d47f6d849149c81c4e52abc727d43b5 And it will change the response headers and these can be seen in a squid access.log using a log format. I can think of other ways to report this but a question: If it works as expected and expected to always work, why would you want to see the HIT in a QOS or TOS? QOS and TOS levels of socket manipulation will require me to find a way to hack the simple web service and I will probably won’t go this way. I do know that you will be able to manipulate QOS or TOS in squid if some header exist in the response. I will might be able to look at the subject if there is a real technical\functional need for that in a long term usage. Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Friday, July 15, 2016 8:48 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store. Hi, Questions 1-What happens if disk or partition becomes full ? 2-Is there a way to use more than one location for store ? 3-Currently hits from your code , could not be counted .How i can use qos flows/tos mark those hits ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678524.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hi, Questions 1-What happens if disk or partition becomes full ? 2-Is there a way to use more than one location for store ? 3-Currently hits from your code , could not be counted .How i can use qos flows/tos mark those hits ? -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678524.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hey Omid, The key concept is that it is possible but not always worth the effort.. I have tested it to work for Windows 10 and for couple other platforms but I didn't verified how it will react to every version of Windows 7. I have tested how things works with WSUSOFFLINE and you will need to change the regex dstdomain into: acl wu dstdom_regex download\.windowsupdate\.com$ download\.microsoft\.com$ Now you need to have my latest updated version in order to avoid caching of MS AV updates which are critical and should never be cached for more then 1 hour. You can try to "seed" the cache using a client which will run WSUSOFFLINE but to my understanding it's not required since you will store more then you actually need. If one user is downloading an ancient or special update you don't need it stored unless you can predict it will be used\downloaded a lot. Let me know if you need some help with it. Eliezer Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Omid Kosari Sent: Thursday, July 14, 2016 2:59 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store. Hi, Great idea . I was looking for something like this for years and i was too lazy to start it myself ;) I am going to test your code in a multi thousand client ISP . It would more great if use the experiences of http://www.wsusoffline.net/ specially for your fetcher . It is GPL Also the ip address 13.107.4.50 is mainly used by microsoft for its download services . With services like https://www.virustotal.com/en-gb/ip-address/13.107.4.50/information/ we have found that other domains also used for update/download services . Maybe not bad if create special things for this ip address . Thanks in advance -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678492.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.
Hi, Great idea . I was looking for something like this for years and i was too lazy to start it myself ;) I am going to test your code in a multi thousand client ISP . It would more great if use the experiences of http://www.wsusoffline.net/ specially for your fetcher . It is GPL Also the ip address 13.107.4.50 is mainly used by microsoft for its download services . With services like https://www.virustotal.com/en-gb/ip-address/13.107.4.50/information/ we have found that other domains also used for update/download services . Maybe not bad if create special things for this ip address . Thanks in advance -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678492.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
