Re: [squid-users] Squid3: 100 % CPU load during object caching
Thanks a lot for your comprehensive testing. Now we are sure that the issue is independent of the underlying OS. I can easily reduce the maximum_object_size_in_memory. I have some constraints from our IT at the moment. I am confronted with a very bad disk IO in our BladeCenter, so I am forced to put as many objects in memory as possible in order to get adequate performance values. We are also rare with disk space. RAM is not an issue right now. Maybe Amos finds a way to fix the issue. Fortunately, we have a find a workaround. Have a nice weekend! Regards, Jens Gesendet: Freitag, 24. Juli 2015 um 19:01 Uhr Von: Marcus Kool marcus.k...@urlfilterdb.com An: Jens Offenbach wolle5...@gmx.de, squid-users@lists.squid-cache.org Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching On 07/24/2015 01:01 PM, Jens Offenbach wrote: @Marcus: I am not sure what exactly causes the problems, but could you please make a test with these two different settings: cache_mem 4 GB maximum_object_size_in_memory 1 GB I think this setting for maximum_object_size_in_memory is too high, independent of how the performance is. The tests also show that large objects cached on disk have a good performance. The perfect place for a large ISO image is the disk cache. I did the test with squid 3.5.6 and got the same result as you have: the download starts fast but quickly drops. Squid uses 100% CPU. wget displays 14 MB/sec ... 10 MB/sec ... 8 7 6 5 4 3 2 MB/sec and stays there for a long time. At 50% downloaded the speed drops more to 1 MB/sec and at the end of the download I got 500 KB/sec *average*. The second cached download was sustained 190 MB/sec and 120% CPU. I did a second test with cache_mem 4 GB maximum_object_size_in_memory 200 MB The download speed varied a lot: started with 30 MB/sec and went down and up many times between 6 MB/sec and 35 MB/sec. The final average download speed was 31 MB/sec. 100% CPU. The second cached download was sustained 190 MB/sec and 120% CPU. Third test with cache_mem 4 GB maximum_object_size_in_memory 8 MB The download speed started with 70 MB/sec and increased to 87 MB/sec. 100% CPU The second cached download was sustained 190 MB/sec and 120% CPU. 4th test with cache_mem 4 GB maximum_object_size_in_memory 32 MB The download speed started with 40 MB/sec and increased to 75 MB/sec. 100% CPU. The second cached download was sustained 190 MB/sec and 120% CPU. So Squid appears to have an issue with higher values of maximum_object_size_in_memory, the higher they are, the worse the performance. For now, I would not go beyond 16 MB. The question is, what is a reasonable size that you would like to be able to use for maximum_object_size_in_memory. Do you have any particular requirement for a high maximum_object_size_in_memory ? Marcus I think you will observe the behavior, that I was confronted with. The bad download rates of 500 KB/sec are gone, when I used the following settings: cache_mem 256 MB maximum_object_size_in_memory 16 MB I think Amos has an idea what seems to be the source of the problem: http://lists.squid-cache.org/pipermail/squid-users/2015-July/004728.html Regards, Jens Gesendet: Freitag, 24. Juli 2015 um 14:33 Uhr Von: Marcus Kool marcus.k...@urlfilterdb.com An: Jens Offenbach wolle5...@gmx.de, squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching On 07/24/2015 03:25 AM, Jens Offenbach wrote: I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same problem: 100 % CPU usage, 500 KB/sec download rate. Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr Von: Jens Offenbach wolle5...@gmx.de An: Marcus Kool marcus.k...@urlfilterdb.com, Eliezer Croitoru elie...@ngtech.co.il, Amos Jeffries squ...@treenet.co.nz, squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging on a real physical infrastructure. I get the same results on the physical machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) where Squid is running: 100 % CPU usage, 500 KB/sec download rate. All machines are idle and we have 1 GBit ethernet. The strace log from the physical test scenario can be found here, but I think it does not differ from the virtual test scenario: http://wikisend.com/download/293856/squid.strace2[http://wikisend.com/download/293856/squid.strace2] @Marcus: Have you verified that the file does not fit into memory and gets cached on disk? On which OS is Squid running? What are your build options of Squid (squid -v)? Is it possible that the issue is not part of 3.4.12? Do we have a regression? I screwed up earlier since the maximum_object_size was too low for the test with a 1 GB file and did a new test. The system has 64 GB memory and for sure the entire file is in the file system
Re: [squid-users] Squid3: 100 % CPU load during object caching
On 07/25/2015 05:25 AM, Jens Offenbach wrote: Thanks a lot for your comprehensive testing. Now we are sure that the issue is independent of the underlying OS. I can easily reduce the maximum_object_size_in_memory. I have some constraints from our IT at the moment. I am confronted with a very bad disk IO in our BladeCenter, so I am forced to put as many objects in memory as possible in order to get adequate performance values. We are also rare with disk space. RAM is not an issue right now. Maybe Amos finds a way to fix the issue. Fortunately, we have a find a workaround. If memory is plentyful, just make sure that the OS has a large file system cache. So reduce mem_cahce of Squid a little and tune the OS with vm.swappiness=10 in /etc/sysctl.conf Best regards Marcus Have a nice weekend! Regards, Jens Gesendet: Freitag, 24. Juli 2015 um 19:01 Uhr Von: Marcus Kool marcus.k...@urlfilterdb.com An: Jens Offenbach wolle5...@gmx.de, squid-users@lists.squid-cache.org Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching On 07/24/2015 01:01 PM, Jens Offenbach wrote: @Marcus: I am not sure what exactly causes the problems, but could you please make a test with these two different settings: cache_mem 4 GB maximum_object_size_in_memory 1 GB I think this setting for maximum_object_size_in_memory is too high, independent of how the performance is. The tests also show that large objects cached on disk have a good performance. The perfect place for a large ISO image is the disk cache. I did the test with squid 3.5.6 and got the same result as you have: the download starts fast but quickly drops. Squid uses 100% CPU. wget displays 14 MB/sec ... 10 MB/sec ... 8 7 6 5 4 3 2 MB/sec and stays there for a long time. At 50% downloaded the speed drops more to 1 MB/sec and at the end of the download I got 500 KB/sec *average*. The second cached download was sustained 190 MB/sec and 120% CPU. I did a second test with cache_mem 4 GB maximum_object_size_in_memory 200 MB The download speed varied a lot: started with 30 MB/sec and went down and up many times between 6 MB/sec and 35 MB/sec. The final average download speed was 31 MB/sec. 100% CPU. The second cached download was sustained 190 MB/sec and 120% CPU. Third test with cache_mem 4 GB maximum_object_size_in_memory 8 MB The download speed started with 70 MB/sec and increased to 87 MB/sec. 100% CPU The second cached download was sustained 190 MB/sec and 120% CPU. 4th test with cache_mem 4 GB maximum_object_size_in_memory 32 MB The download speed started with 40 MB/sec and increased to 75 MB/sec. 100% CPU. The second cached download was sustained 190 MB/sec and 120% CPU. So Squid appears to have an issue with higher values of maximum_object_size_in_memory, the higher they are, the worse the performance. For now, I would not go beyond 16 MB. The question is, what is a reasonable size that you would like to be able to use for maximum_object_size_in_memory. Do you have any particular requirement for a high maximum_object_size_in_memory ? Marcus I think you will observe the behavior, that I was confronted with. The bad download rates of 500 KB/sec are gone, when I used the following settings: cache_mem 256 MB maximum_object_size_in_memory 16 MB I think Amos has an idea what seems to be the source of the problem: http://lists.squid-cache.org/pipermail/squid-users/2015-July/004728.html Regards, Jens Gesendet: Freitag, 24. Juli 2015 um 14:33 Uhr Von: Marcus Kool marcus.k...@urlfilterdb.com An: Jens Offenbach wolle5...@gmx.de, squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching On 07/24/2015 03:25 AM, Jens Offenbach wrote: I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same problem: 100 % CPU usage, 500 KB/sec download rate. Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr Von: Jens Offenbach wolle5...@gmx.de An: Marcus Kool marcus.k...@urlfilterdb.com, Eliezer Croitoru elie...@ngtech.co.il, Amos Jeffries squ...@treenet.co.nz, squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging on a real physical infrastructure. I get the same results on the physical machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) where Squid is running: 100 % CPU usage, 500 KB/sec download rate. All machines are idle and we have 1 GBit ethernet. The strace log from the physical test scenario can be found here, but I think it does not differ from the virtual test scenario: http://wikisend.com/download/293856/squid.strace2[http://wikisend.com/download/293856/squid.strace2] @Marcus: Have you verified that the file does not fit into memory and gets cached on disk? On which OS is Squid running? What are your build options of Squid (squid -v)? Is it possible that the issue is not part of 3.4.12? Do we have
Re: [squid-users] Squid3: 100 % CPU load during object caching
I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same problem: 100 % CPU usage, 500 KB/sec download rate. Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr Von: Jens Offenbach wolle5...@gmx.de An: Marcus Kool marcus.k...@urlfilterdb.com, Eliezer Croitoru elie...@ngtech.co.il, Amos Jeffries squ...@treenet.co.nz, squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging on a real physical infrastructure. I get the same results on the physical machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) where Squid is running: 100 % CPU usage, 500 KB/sec download rate. All machines are idle and we have 1 GBit ethernet. The strace log from the physical test scenario can be found here, but I think it does not differ from the virtual test scenario: http://wikisend.com/download/293856/squid.strace2 @Marcus: Have you verified that the file does not fit into memory and gets cached on disk? On which OS is Squid running? What are your build options of Squid (squid -v)? Is it possible that the issue is not part of 3.4.12? Do we have a regression? @Amos, Eliezer Is someone able to reproduce the disk caching effect? Regards, Jens Gesendet: Donnerstag, 23. Juli 2015 um 20:08 Uhr Von: Marcus Kool marcus.k...@urlfilterdb.com An: Jens Offenbach wolle5...@gmx.de, Amos Jeffries squ...@treenet.co.nz, Eliezer Croitoru elie...@ngtech.co.il, squid-users@lists.squid-cache.org Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching The strace output shows this loop: Squid reads 16K-1 bytes from FD 13 webserver Squid writes 4 times 4K to FD 17 /var/cache/squid3/00/00/ Squid writes 4 times 4K to FD 12 browser But this loop does not explain the 100% CPU usage... Does Squid do a buffer reshuffle when it reads 16K-1 and writes 16K ? I did the download test with Squid 3.4.12 AUFS on an idle system with a 500 mbit connection and 1 CPU with 4 cores @ 3.7 GHz. The first download used 35% of 1 CPU core with a steady download speed of 62 MB/sec. The second (cached) download used 50% of 1 CPU core with a steady download speed of 87 MB/sec. I never looked at Squid CPU usage and do not know what is reasonable but it feels high. With respect to the 100% CPU issue of Jens, one factor is that Squid runs in a virtual machine. Squid in a virtual machine cannot be compared with a wget test since Squid allocates a lot of memory that the host must manage. This is a possible explanation for the fact that you see the performance going down and up. Can you do the same test on the host (i.e. not inside a VM). Marcus On 07/23/2015 10:39 AM, Jens Offenbach wrote: I have attached strace to Squid and waited until the download rate has decreased to 500 KB/sec. I used cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240. Here is the download link: http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace[http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace] I hope it can help you. *Gesendet:* Donnerstag, 23. Juli 2015 um 13:29 Uhr *Von:* Marcus Kool marcus.k...@urlfilterdb.com *An:* Jens Offenbach wolle5...@gmx.de, Eliezer Croitoru elie...@ngtech.co.il, Amos Jeffries squ...@treenet.co.nz, squid-users@lists.squid-cache.org *Betreff:* Re: [squid-users] Squid3: 100 % CPU load during object caching I am not sure if it is relevant, maybe it is: I am developing an ICAP daemon and after the ICAP server sends a 100 continue Squid sends the object to the ICAP server in small chunks of varying sizes: 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc. Note that the interval of receiving the chunks is 1/1000th of a second. It seems that Squid forwards the object to the ICAP server every time it receives one or a few TCP packets. I have a suspicion that in the scenario of 100% CPU, large #write calls and low throughput a similar thing is happening: Squid physically stores a small part of the object many times, i.e. every time one or a few TCP packets arrive. Amos, is there a debug setting that can confirm/reject this suspicion? Marcus On 07/23/2015 04:25 AM, Jens Offenbach wrote: A test with ROCK cache_dir rock /var/cache/squid3 51200 gives very confusing results. I cleared the cache: rm -rf /var/cache/squid3/* squid -z squid http_proxy=http://139.2.57.120:3128/[http://139.2.57.120:3128/][http://139.2.57.120:3128/[http://139.2.57.120:3128/]] wget http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar The download starts with 10 MB/sec and stays constant for 1 minutes, then it drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the download rate returns back to 10 MB/sec very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the download rates
Re: [squid-users] Squid3: 100 % CPU load during object caching
I have found something out... Hopefully, it helps to reproduce and solve the issue. I got it working with a good download rate, but very high CPU usage on Squid 3.3.8 and Squid 3.5.6. There seems to be problem with large files that get cached on disk in combination with memory caching. When I use these settings, memory usage of Squid grows step-by-step with 100% CPU usage and 500 KB/sec download rate: # MEMORY CACHE OPTIONS # - maximum_object_size_in_memory 1 GB memory_replacement_policy heap LFUDA cache_mem 4 GB # DISK CACHE OPTIONS # - maximum_object_size 10 GB cache_replacement_policy heap GDSF cache_dir aufs /var/cache/squid3 25600 16 256 I decided to turn off memory caching completely and used the following settings: # MEMORY CACHE OPTIONS # - maximum_object_size_in_memory 0 GB memory_replacement_policy heap LFUDA cache_mem 0 GB # DISK CACHE OPTIONS # - maximum_object_size 10 GB cache_replacement_policy heap GDSF cache_dir aufs /var/cache/squid3 25600 16 256 Now, I get stable and high download rates even on a cache miss. @Marcus: Could you please post your squid.config Gesendet: Freitag, 24. Juli 2015 um 08:25 Uhr Von: Jens Offenbach wolle5...@gmx.de An: squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same problem: 100 % CPU usage, 500 KB/sec download rate. Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr Von: Jens Offenbach wolle5...@gmx.de An: Marcus Kool marcus.k...@urlfilterdb.com, Eliezer Croitoru elie...@ngtech.co.il, Amos Jeffries squ...@treenet.co.nz, squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging on a real physical infrastructure. I get the same results on the physical machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) where Squid is running: 100 % CPU usage, 500 KB/sec download rate. All machines are idle and we have 1 GBit ethernet. The strace log from the physical test scenario can be found here, but I think it does not differ from the virtual test scenario: http://wikisend.com/download/293856/squid.strace2 @Marcus: Have you verified that the file does not fit into memory and gets cached on disk? On which OS is Squid running? What are your build options of Squid (squid -v)? Is it possible that the issue is not part of 3.4.12? Do we have a regression? @Amos, Eliezer Is someone able to reproduce the disk caching effect? Regards, Jens Gesendet: Donnerstag, 23. Juli 2015 um 20:08 Uhr Von: Marcus Kool marcus.k...@urlfilterdb.com An: Jens Offenbach wolle5...@gmx.de, Amos Jeffries squ...@treenet.co.nz, Eliezer Croitoru elie...@ngtech.co.il, squid-users@lists.squid-cache.org Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching The strace output shows this loop: Squid reads 16K-1 bytes from FD 13 webserver Squid writes 4 times 4K to FD 17 /var/cache/squid3/00/00/ Squid writes 4 times 4K to FD 12 browser But this loop does not explain the 100% CPU usage... Does Squid do a buffer reshuffle when it reads 16K-1 and writes 16K ? I did the download test with Squid 3.4.12 AUFS on an idle system with a 500 mbit connection and 1 CPU with 4 cores @ 3.7 GHz. The first download used 35% of 1 CPU core with a steady download speed of 62 MB/sec. The second (cached) download used 50% of 1 CPU core with a steady download speed of 87 MB/sec. I never looked at Squid CPU usage and do not know what is reasonable but it feels high. With respect to the 100% CPU issue of Jens, one factor is that Squid runs in a virtual machine. Squid in a virtual machine cannot be compared with a wget test since Squid allocates a lot of memory that the host must manage. This is a possible explanation for the fact that you see the performance going down and up. Can you do the same test on the host (i.e. not inside a VM). Marcus On 07/23/2015 10:39 AM, Jens Offenbach wrote: I have attached strace to Squid and waited until the download rate has decreased to 500 KB/sec. I used cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240. Here is the download link: http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace[http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace][http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace[http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace]] I hope it can help you
Re: [squid-users] Squid3: 100 % CPU load during object caching
On 24/07/2015 6:08 a.m., Marcus Kool wrote: The strace output shows this loop: Squid reads 16K-1 bytes from FD 13webserver Squid writes 4 times 4K to FD 17 /var/cache/squid3/00/00/ Squid writes 4 times 4K to FD 12 browser But this loop does not explain the 100% CPU usage... Does Squid do a buffer reshuffle when it reads 16K-1 and writes 16K ? Yes, several (UFS / AUFS 3, or diskd 5). TCP buffer - FD 13 read buffer FD 13 read buffer - 4x mem_node (4KB each) ** walk the length of the in-memory part of the object to find where to attach the mem_node. (once per each node?) - this has been a big CPU hog in the past (Squid-2 did it twice per node insertion) 4x mem_node - SHM memory buffer - diskd only, AUFS uses mem_node directly SHM memory buffer - FD 17 disk write latency - happens with both diskd (single treaded) and AUFS (x64 threads) - wait latency until completion event is seen by Squid ... 4x mem_node write() copy to FD 12 TCP buffer (OS dependent) If you are doing any kind of ICAP processing you can add +3 copies per service processing the transaction body. I did the download test with Squid 3.4.12 AUFS on an idle system with a 500 mbit connection and 1 CPU with 4 cores @ 3.7 GHz. The first download used 35% of 1 CPU core with a steady download speed of 62 MB/sec. The second (cached) download used 50% of 1 CPU core with a steady download speed of 87 MB/sec. I never looked at Squid CPU usage and do not know what is reasonable but it feels high. With respect to the 100% CPU issue of Jens, one factor is that Squid runs in a virtual machine. Squid in a virtual machine cannot be compared with a wget test since Squid allocates a lot of memory that the host must manage. This is a possible explanation for the fact that you see the performance going down and up. Can you do the same test on the host (i.e. not inside a VM). Marcus On 07/23/2015 10:39 AM, Jens Offenbach wrote: I have attached strace to Squid and waited until the download rate has decreased to 500 KB/sec. I used cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240. Here is the download link: http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace I hope it can help you. *Gesendet:* Donnerstag, 23. Juli 2015 um 13:29 Uhr *Von:* Marcus Kool marcus.k...@urlfilterdb.com *An:* Jens Offenbach wolle5...@gmx.de, Eliezer Croitoru elie...@ngtech.co.il, Amos Jeffries squ...@treenet.co.nz, squid-users@lists.squid-cache.org *Betreff:* Re: [squid-users] Squid3: 100 % CPU load during object caching I am not sure if it is relevant, maybe it is: I am developing an ICAP daemon and after the ICAP server sends a 100 continue Squid sends the object to the ICAP server in small chunks of varying sizes: 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc. Note that the interval of receiving the chunks is 1/1000th of a second. It seems that Squid forwards the object to the ICAP server every time it receives one or a few TCP packets. I have a suspicion that in the scenario of 100% CPU, large #write calls and low throughput a similar thing is happening: Squid physically stores a small part of the object many times, i.e. every time one or a few TCP packets arrive. Amos, is there a debug setting that can confirm/reject this suspicion? After a bit more thought and Marcus feedback ; store.cc, mem_node operations, and fd.cc and comm.cc are probably all worth watching. debug_options ALL,9 will get you everything Squid has to offer of course. But be aware that the debugging itself adds a horribly large amount of overheads for each line logged. At the highest levels it may noticably impact the high-speed core routines you are trying to measure by skewing latency into those with more debugs() statements. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid3: 100 % CPU load during object caching
On 24/07/2015 7:49 p.m., Jens Offenbach wrote: I have found something out... Hopefully, it helps to reproduce and solve the issue. I got it working with a good download rate, but very high CPU usage on Squid 3.3.8 and Squid 3.5.6. There seems to be problem with large files that get cached on disk in combination with memory caching. When I use these settings, memory usage of Squid grows step-by-step with 100% CPU usage and 500 KB/sec download rate: # MEMORY CACHE OPTIONS # - maximum_object_size_in_memory 1 GB memory_replacement_policy heap LFUDA cache_mem 4 GB # DISK CACHE OPTIONS # - maximum_object_size 10 GB cache_replacement_policy heap GDSF cache_dir aufs /var/cache/squid3 25600 16 256 I decided to turn off memory caching completely and used the following settings: # MEMORY CACHE OPTIONS # - maximum_object_size_in_memory 0 GB memory_replacement_policy heap LFUDA cache_mem 0 GB # DISK CACHE OPTIONS # - maximum_object_size 10 GB cache_replacement_policy heap GDSF cache_dir aufs /var/cache/squid3 25600 16 256 Now, I get stable and high download rates even on a cache miss. Damn. That gives me ~90% confidence its the mem_node walking as new 4KB chunks of memory are appended to the memory copy of the object. I would expect to see a reduced effect in 3.5 that kicks in around maximum_object_size_in_memory. Since the memory copies are now split into cache_mem objects, vs transients (disk cache only, or totally non-cacheable). With the transients getting their unnecessary in-memory sections pruned away regularly. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid3: 100 % CPU load during object caching
@Marcus: I am not sure what exactly causes the problems, but could you please make a test with these two different settings: cache_mem 4 GB maximum_object_size_in_memory 1 GB I think you will observe the behavior, that I was confronted with. The bad download rates of 500 KB/sec are gone, when I used the following settings: cache_mem 256 MB maximum_object_size_in_memory 16 MB I think Amos has an idea what seems to be the source of the problem: http://lists.squid-cache.org/pipermail/squid-users/2015-July/004728.html Regards, Jens Gesendet: Freitag, 24. Juli 2015 um 14:33 Uhr Von: Marcus Kool marcus.k...@urlfilterdb.com An: Jens Offenbach wolle5...@gmx.de, squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching On 07/24/2015 03:25 AM, Jens Offenbach wrote: I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same problem: 100 % CPU usage, 500 KB/sec download rate. Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr Von: Jens Offenbach wolle5...@gmx.de An: Marcus Kool marcus.k...@urlfilterdb.com, Eliezer Croitoru elie...@ngtech.co.il, Amos Jeffries squ...@treenet.co.nz, squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging on a real physical infrastructure. I get the same results on the physical machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) where Squid is running: 100 % CPU usage, 500 KB/sec download rate. All machines are idle and we have 1 GBit ethernet. The strace log from the physical test scenario can be found here, but I think it does not differ from the virtual test scenario: http://wikisend.com/download/293856/squid.strace2 @Marcus: Have you verified that the file does not fit into memory and gets cached on disk? On which OS is Squid running? What are your build options of Squid (squid -v)? Is it possible that the issue is not part of 3.4.12? Do we have a regression? I screwed up earlier since the maximum_object_size was too low for the test with a 1 GB file and did a new test. The system has 64 GB memory and for sure the entire file is in the file system cache. The disk system is HW RAID-1 with 1 GB cache. The OS is Linux 3.10, CentOS 7 latest patches. New test: test system: 1 CPU with 4 cores/8 threads @ 3.7 GHz, 64 GB memory, AUFS, 1 Gbit pipe, 500 mbit guaranteed with Squid 3.4.12 : 1st download starts with 90 MB/sec and halfway drops to 30 MB/sec. My guess is that the file system cache got stressed and slowed things down. 2nd cached download with 190 MB/sec sustained and 120% CPU time. With Squid 3.5.6 : 1st download starts with 90 MB/sec sustained and 80% CPU time. 2nd cached download with 190 MB/sec sustained and 120% CPU time. As a comparison, I did dd if=test of=test2 bs=4k which uses 100% CPU time and has a throughput of 1200 MB/sec. With bs=16k the throughput is 1300 MB/sec and with bs=64k the throughput is 1400 MB/sec. relevant parameters : read_ahead_gap 64 KB cache_mem 256 MB maximum_object_size_in_memory 8 MB maximum_object_size 8000 MB cache_dir aufs /local/squid34/cache 1 32 256 cache_swap_low 92 cache_swap_high 93 # also ICAP daemon and URL rewriter configured debug_options ALL,1 93,3 61,9 configure options: '--prefix=/local/squid35' '--disable-ipv6' '--enable-fd-config' '--with-maxfd=3200' '--enable-async-io=64' '--enable-storeio=aufs' '--with-pthreads' '--enable-removal-policies=lru' '--disable-auto-locale' '--enable-default-err-language=English' '--enable-err-languages=Dutch English Portuguese' '--with-openssl' '--enable-ssl' '--enable-ssl-crtd' '--enable-cachemgr-hostname=localhost' '--enable-cache-digests' '--enable-follow-x-forwarded-for' '--enable-xmalloc-statistics' '--disable-hostname-checks' '--enable-epoll' '--enable-icap-client' '--enable-useragent-log' '--enable-referer-log' '--enable-stacktraces' '--enable-underscores' '--disable-icmp' '--mandir=/usr/local/share' 'CC=gcc' 'CFLAGS=-g -O2 -Wall -march=native' 'CXXFLAGS=-g -O2 -Wall -march=native' --enable-ltdl-convenience As you can see the cache_mem is small, If Amos finds it useful, I can do another test with a larger cache_mem. Jens, since all your tests have a drop to 500 KB/sec I think the cause is somewhere is the configuration (Squid and/or OS). Marcus @Amos, Eliezer Is someone able to reproduce the disk caching effect? Regards, Jens Gesendet: Donnerstag, 23. Juli 2015 um 20:08 Uhr Von: Marcus Kool marcus.k...@urlfilterdb.com An: Jens Offenbach wolle5...@gmx.de, Amos Jeffries squ...@treenet.co.nz, Eliezer Croitoru elie...@ngtech.co.il, squid-users@lists.squid-cache.org Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching The strace output shows this loop: Squid reads 16K-1 bytes from FD 13 webserver Squid writes 4 times 4K to FD 17 /var/cache/squid3/00/00/ Squid
Re: [squid-users] Squid3: 100 % CPU load during object caching
On 07/24/2015 01:01 PM, Jens Offenbach wrote: @Marcus: I am not sure what exactly causes the problems, but could you please make a test with these two different settings: cache_mem 4 GB maximum_object_size_in_memory 1 GB I think this setting for maximum_object_size_in_memory is too high, independent of how the performance is. The tests also show that large objects cached on disk have a good performance. The perfect place for a large ISO image is the disk cache. I did the test with squid 3.5.6 and got the same result as you have: the download starts fast but quickly drops. Squid uses 100% CPU. wget displays 14 MB/sec ... 10 MB/sec ... 8 7 6 5 4 3 2 MB/sec and stays there for a long time. At 50% downloaded the speed drops more to 1 MB/sec and at the end of the download I got 500 KB/sec *average*. The second cached download was sustained 190 MB/sec and 120% CPU. I did a second test with cache_mem 4 GB maximum_object_size_in_memory 200 MB The download speed varied a lot: started with 30 MB/sec and went down and up many times between 6 MB/sec and 35 MB/sec. The final average download speed was 31 MB/sec. 100% CPU. The second cached download was sustained 190 MB/sec and 120% CPU. Third test with cache_mem 4 GB maximum_object_size_in_memory 8 MB The download speed started with 70 MB/sec and increased to 87 MB/sec. 100% CPU The second cached download was sustained 190 MB/sec and 120% CPU. 4th test with cache_mem 4 GB maximum_object_size_in_memory 32 MB The download speed started with 40 MB/sec and increased to 75 MB/sec. 100% CPU. The second cached download was sustained 190 MB/sec and 120% CPU. So Squid appears to have an issue with higher values of maximum_object_size_in_memory, the higher they are, the worse the performance. For now, I would not go beyond 16 MB. The question is, what is a reasonable size that you would like to be able to use for maximum_object_size_in_memory. Do you have any particular requirement for a high maximum_object_size_in_memory ? Marcus I think you will observe the behavior, that I was confronted with. The bad download rates of 500 KB/sec are gone, when I used the following settings: cache_mem 256 MB maximum_object_size_in_memory 16 MB I think Amos has an idea what seems to be the source of the problem: http://lists.squid-cache.org/pipermail/squid-users/2015-July/004728.html Regards, Jens Gesendet: Freitag, 24. Juli 2015 um 14:33 Uhr Von: Marcus Kool marcus.k...@urlfilterdb.com An: Jens Offenbach wolle5...@gmx.de, squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching On 07/24/2015 03:25 AM, Jens Offenbach wrote: I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same problem: 100 % CPU usage, 500 KB/sec download rate. Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr Von: Jens Offenbach wolle5...@gmx.de An: Marcus Kool marcus.k...@urlfilterdb.com, Eliezer Croitoru elie...@ngtech.co.il, Amos Jeffries squ...@treenet.co.nz, squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging on a real physical infrastructure. I get the same results on the physical machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) where Squid is running: 100 % CPU usage, 500 KB/sec download rate. All machines are idle and we have 1 GBit ethernet. The strace log from the physical test scenario can be found here, but I think it does not differ from the virtual test scenario: http://wikisend.com/download/293856/squid.strace2 @Marcus: Have you verified that the file does not fit into memory and gets cached on disk? On which OS is Squid running? What are your build options of Squid (squid -v)? Is it possible that the issue is not part of 3.4.12? Do we have a regression? I screwed up earlier since the maximum_object_size was too low for the test with a 1 GB file and did a new test. The system has 64 GB memory and for sure the entire file is in the file system cache. The disk system is HW RAID-1 with 1 GB cache. The OS is Linux 3.10, CentOS 7 latest patches. New test: test system: 1 CPU with 4 cores/8 threads @ 3.7 GHz, 64 GB memory, AUFS, 1 Gbit pipe, 500 mbit guaranteed with Squid 3.4.12 : 1st download starts with 90 MB/sec and halfway drops to 30 MB/sec. My guess is that the file system cache got stressed and slowed things down. 2nd cached download with 190 MB/sec sustained and 120% CPU time. With Squid 3.5.6 : 1st download starts with 90 MB/sec sustained and 80% CPU time. 2nd cached download with 190 MB/sec sustained and 120% CPU time. As a comparison, I did dd if=test of=test2 bs=4k which uses 100% CPU time and has a throughput of 1200 MB/sec. With bs=16k the throughput is 1300 MB/sec and with bs=64k the throughput is 1400 MB/sec. relevant parameters : read_ahead_gap 64 KB cache_mem 256 MB maximum_object_size_in_memory 8 MB
Re: [squid-users] Squid3: 100 % CPU load during object caching
I am not sure if it is relevant, maybe it is: I am developing an ICAP daemon and after the ICAP server sends a 100 continue Squid sends the object to the ICAP server in small chunks of varying sizes: 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc. Note that the interval of receiving the chunks is 1/1000th of a second. It seems that Squid forwards the object to the ICAP server every time it receives one or a few TCP packets. I have a suspicion that in the scenario of 100% CPU, large #write calls and low throughput a similar thing is happening: Squid physically stores a small part of the object many times, i.e. every time one or a few TCP packets arrive. Amos, is there a debug setting that can confirm/reject this suspicion? Marcus On 07/23/2015 04:25 AM, Jens Offenbach wrote: A test with ROCK cache_dir rock /var/cache/squid3 51200 gives very confusing results. I cleared the cache: rm -rf /var/cache/squid3/* squid -z squid http_proxy=http://139.2.57.120:3128/ wget http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar The download starts with 10 MB/sec and stays constant for 1 minutes, then it drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the download rate returns back to 10 MB/sec very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec. During caching progress, we have 100 % CPU usage and a disk IO that is corresponds with the download rate. For further investigations I give you my build properties: squid -v Squid Cache: Version 3.5.6 Service Name: squid configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3' '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-e s i' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/cache/squid3' '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr Von: Eliezer Croitoru elie...@ngtech.co.il An: squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching On 22/07/2015 21:59, Eliezer Croitoru wrote: Hey Jens, I have tested the issue with LARGE ROCK and not AUFS or UFS. Using squid or not my connection to the server is about 2.5 MBps (20Mbps). Squid is sitting on an intel atom with SSD drive and on a HIT case the download speed is more then doubled to 4.5 MBps(36Mbps). I have not tried it with AUFS yet. And I must admit that AUFS beats rock cache with speed. I have tried rock with basic cache_dir rock /var/spool/squid 8000 vs cache_dir aufs /var/spool/squid 8000 16 256 and the aufs cache HIT results more then doubles 3 the speed rock gave with default settings. So about 15MBps which is 120Mbps. I do not seem to feel what Jens feels but the 100% CPU might be because of spinning disk hangs while reading the file from disk. Amos, I remember that there were some suggestions how to tune large rock. Any hints? I can test it and make it a suggestion for big files. Eliezer ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid3: 100 % CPU load during object caching
I have attached strace to Squid and waited until the download rate has decreased to 500 KB/sec. I used cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240. Here is the download link: http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace I hope it can help you. Gesendet:Donnerstag, 23. Juli 2015 um 13:29 Uhr Von:Marcus Kool marcus.k...@urlfilterdb.com An:Jens Offenbach wolle5...@gmx.de, Eliezer Croitoru elie...@ngtech.co.il, Amos Jeffries squ...@treenet.co.nz, squid-users@lists.squid-cache.org Betreff:Re: [squid-users] Squid3: 100 % CPU load during object caching I am not sure if it is relevant, maybe it is: I am developing an ICAP daemon and after the ICAP server sends a 100 continue Squid sends the object to the ICAP server in small chunks of varying sizes: 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc. Note that the interval of receiving the chunks is 1/1000th of a second. It seems that Squid forwards the object to the ICAP server every time it receives one or a few TCP packets. I have a suspicion that in the scenario of 100% CPU, large #write calls and low throughput a similar thing is happening: Squid physically stores a small part of the object many times, i.e. every time one or a few TCP packets arrive. Amos, is there a debug setting that can confirm/reject this suspicion? Marcus On 07/23/2015 04:25 AM, Jens Offenbach wrote: A test with ROCK cache_dir rock /var/cache/squid3 51200 gives very confusing results. I cleared the cache: rm -rf /var/cache/squid3/* squid -z squid http_proxy=http://139.2.57.120:3128/ wget http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar The download starts with 10 MB/sec and stays constant for 1 minutes, then it drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the download rate returns back to 10 MB/sec very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec. During caching progress, we have 100 % CPU usage and a disk IO that is corresponds with the download rate. For further investigations I give you my build properties: squid -v Squid Cache: Version 3.5.6 Service Name: squid configure options: --build=x86_64-linux-gnu --prefix=/usr --includedir=/include --mandir=/share/man --infodir=/share/info --sysconfdir=/etc --localstatedir=/var --libexecdir=/lib/squid3 --srcdir=. --disable-maintainer-mode --disable-dependency-tracking --disable-silent-rules --datadir=/usr/share/squid3 --sysconfdir=/etc/squid3 --mandir=/usr/share/man --enable-inline --enable-async-io=8 --enable-storeio=ufs,aufs,diskd,rock --enable-removal-policies=lru,heap --enable-delay-pools --enable-cache-digests --enable-underscores --enable-icap-client --enable-follow-x-forwarded-for --enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB --enable-auth-digest=file,LDAP --enable-auth-negotiate=kerberos,wrapper --enable-auth-ntlm=fake,smb_lm --enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group --enable-url-rewrite-helpers=fake --enable-eui --enable-e s i --enable-icmp --enable-zph-qos --enable-ecap --disable-translation --with-swapdir=/var/cache/squid3 --with-logdir=/var/log/squid3 --with-pidfile=/var/run/squid3.pid --with-filedescriptors=65536 --with-large-files --with-default-user=proxy --enable-linux-netfilter build_alias=x86_64-linux-gnu CFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now CPPFLAGS=-D_FORTIFY_SOURCE=2 CXXFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr Von: Eliezer Croitoru elie...@ngtech.co.il An: squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching On 22/07/2015 21:59, Eliezer Croitoru wrote: Hey Jens, I have tested the issue with LARGE ROCK and not AUFS or UFS. Using squid or not my connection to the server is about 2.5 MBps (20Mbps). Squid is sitting on an intel atom with SSD drive and on a HIT case the download speed is more then doubled to 4.5 MBps(36Mbps). I have not tried it with AUFS yet. And I must admit that AUFS beats rock cache with speed. I have tried rock with basic cache_dir rock /var/spool/squid 8000 vs cache_dir aufs /var/spool/squid 8000 16 256 and the aufs cache HIT results more then doubles 3 the speed rock gave with default settings. So about 15MBps which is 120Mbps. I do not seem to feel what Jens feels but the 100% CPU might be because of spinning disk hangs while reading the file from disk. Amos, I remember that there were some suggestions how to tune large rock. Any hints? I can test it and make it a suggestion for big files. Eliezer
Re: [squid-users] Squid3: 100 % CPU load during object caching
It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging on a real physical infrastructure. I get the same results on the physical machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) where Squid is running: 100 % CPU usage, 500 KB/sec download rate. All machines are idle and we have 1 GBit ethernet. The strace log from the physical test scenario can be found here, but I think it does not differ from the virtual test scenario: http://wikisend.com/download/293856/squid.strace2 @Marcus: Have you verified that the file does not fit into memory and gets cached on disk? On which OS is Squid running? What are your build options of Squid (squid -v)? Is it possible that the issue is not part of 3.4.12? Do we have a regression? @Amos, Eliezer Is someone able to reproduce the disk caching effect? Regards, Jens Gesendet: Donnerstag, 23. Juli 2015 um 20:08 Uhr Von: Marcus Kool marcus.k...@urlfilterdb.com An: Jens Offenbach wolle5...@gmx.de, Amos Jeffries squ...@treenet.co.nz, Eliezer Croitoru elie...@ngtech.co.il, squid-users@lists.squid-cache.org Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching The strace output shows this loop: Squid reads 16K-1 bytes from FD 13 webserver Squid writes 4 times 4K to FD 17 /var/cache/squid3/00/00/ Squid writes 4 times 4K to FD 12 browser But this loop does not explain the 100% CPU usage... Does Squid do a buffer reshuffle when it reads 16K-1 and writes 16K ? I did the download test with Squid 3.4.12 AUFS on an idle system with a 500 mbit connection and 1 CPU with 4 cores @ 3.7 GHz. The first download used 35% of 1 CPU core with a steady download speed of 62 MB/sec. The second (cached) download used 50% of 1 CPU core with a steady download speed of 87 MB/sec. I never looked at Squid CPU usage and do not know what is reasonable but it feels high. With respect to the 100% CPU issue of Jens, one factor is that Squid runs in a virtual machine. Squid in a virtual machine cannot be compared with a wget test since Squid allocates a lot of memory that the host must manage. This is a possible explanation for the fact that you see the performance going down and up. Can you do the same test on the host (i.e. not inside a VM). Marcus On 07/23/2015 10:39 AM, Jens Offenbach wrote: I have attached strace to Squid and waited until the download rate has decreased to 500 KB/sec. I used cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240. Here is the download link: http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace I hope it can help you. *Gesendet:* Donnerstag, 23. Juli 2015 um 13:29 Uhr *Von:* Marcus Kool marcus.k...@urlfilterdb.com *An:* Jens Offenbach wolle5...@gmx.de, Eliezer Croitoru elie...@ngtech.co.il, Amos Jeffries squ...@treenet.co.nz, squid-users@lists.squid-cache.org *Betreff:* Re: [squid-users] Squid3: 100 % CPU load during object caching I am not sure if it is relevant, maybe it is: I am developing an ICAP daemon and after the ICAP server sends a 100 continue Squid sends the object to the ICAP server in small chunks of varying sizes: 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc. Note that the interval of receiving the chunks is 1/1000th of a second. It seems that Squid forwards the object to the ICAP server every time it receives one or a few TCP packets. I have a suspicion that in the scenario of 100% CPU, large #write calls and low throughput a similar thing is happening: Squid physically stores a small part of the object many times, i.e. every time one or a few TCP packets arrive. Amos, is there a debug setting that can confirm/reject this suspicion? Marcus On 07/23/2015 04:25 AM, Jens Offenbach wrote: A test with ROCK cache_dir rock /var/cache/squid3 51200 gives very confusing results. I cleared the cache: rm -rf /var/cache/squid3/* squid -z squid http_proxy=http://139.2.57.120:3128/[http://139.2.57.120:3128/] wget http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar The download starts with 10 MB/sec and stays constant for 1 minutes, then it drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the download rate returns back to 10 MB/sec very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec. During caching progress, we have 100 % CPU usage and a disk IO that is corresponds with the download rate. For further investigations I give you my build properties: squid -v Squid Cache: Version 3.5.6 Service Name: squid configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking
Re: [squid-users] Squid3: 100 % CPU load during object caching
On 23/07/2015 11:29 p.m., Marcus Kool wrote: I am not sure if it is relevant, maybe it is: I am developing an ICAP daemon and after the ICAP server sends a 100 continue Squid sends the object to the ICAP server in small chunks of varying sizes: 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc. Note that the interval of receiving the chunks is 1/1000th of a second. It seems that Squid forwards the object to the ICAP server every time it receives one or a few TCP packets. I have a suspicion that in the scenario of 100% CPU, large #write calls and low throughput a similar thing is happening: Squid physically stores a small part of the object many times, i.e. every time one or a few TCP packets arrive. If testing with low traffic (one connection) that guess is correct. Squid is so fast that it cycles through the whole allocate-memory-read()-process-write()-to-icap cycle in under a millisecond. It takes a few dozen busy clients in parallel to slow Slow down enough to increase the chunk sizes. Amos, is there a debug setting that can confirm/reject this suspicion? Not that I recall at the moment. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid3: 100 % CPU load during object caching
I have not tested rock yet, but I will give it a try and will report the results as soon as possible. I have tested Squid 3.5.6 in three different virtualized environments (VMware vSphere, OpenStack Icehouse, OpenStack Kilo). For clarification: The 100 % CPU usage and the low download speed occurs only when there is NO cache hit and when the object is added to disk cache FOR THE FIRST TIME. When the object is in the cache, the download speed and the CPU usage of Squid (3.3.8 and 3.5.6) is optimal. I have tested it with an Ubuntu ISO image (= 1 GB). This is my current squid.config. # ACCESS CONTROLS # - acl localnetsrc 139.2.0.0/16 acl localnetsrc 193.96.112.0/21 acl localnetsrc 192.109.216.0/24 acl localnetsrc 100.1.4.0/22 acl localnetsrc 10.0.0.0/8 acl localnetsrc 172.16.0.0/12 acl localnetsrc 192.168.0.0/16 http_access allow manager localhost http_access deny manager http_access allow localnet http_access allow localhost http_access deny all # NETWORK OPTIONS # - http_port 0.0.0.0:3128 # MEMORY CACHE OPTIONS # - maximum_object_size_in_memory 128 MB memory_replacement_policy heap LFUDA cache_mem 4 GB # DISK CACHE OPTIONS # - maximum_object_size 10 GB cache_replacement_policy heap GDSF cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240 # LOGFILE OPTIONS # - access_log daemon:/var/log/squid3/access.log squid cache_store_log daemon:/var/log/squid3/store.log # OPTIONS FOR TROUBLESHOOTING # - cache_log /var/log/squid3/cache.log coredump_dir /var/log/squid3 # OPTIONS FOR TUNING THE CACHE # - cache allow all # OPTIONS INFLUENCING REQUEST FORWARDING # - always_direct allow all # MISCELLANEOUS # - memory_pools off Thank you very much for your help! It is good to know that you will take care of my problem. Regards, Jens Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr Von: Eliezer Croitoru elie...@ngtech.co.il An: squid-users@lists.squid-cache.org Cc: Amos Jeffries squ...@treenet.co.nz Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching On 22/07/2015 21:59, Eliezer Croitoru wrote: Hey Jens, I have tested the issue with LARGE ROCK and not AUFS or UFS. Using squid or not my connection to the server is about 2.5 MBps (20Mbps). Squid is sitting on an intel atom with SSD drive and on a HIT case the download speed is more then doubled to 4.5 MBps(36Mbps). I have not tried it with AUFS yet. And I must admit that AUFS beats rock cache with speed. I have tried rock with basic cache_dir rock /var/spool/squid 8000 vs cache_dir aufs /var/spool/squid 8000 16 256 and the aufs cache HIT results more then doubles 3 the speed rock gave with default settings. So about 15MBps which is 120Mbps. I do not seem to feel what Jens feels but the 100% CPU might be because of spinning disk hangs while reading the file from disk. Amos, I remember that there were some suggestions how to tune large rock. Any hints? I can test it and make it a suggestion for big files. Eliezer ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid3: 100 % CPU load during object caching
Ok. I am sorry, Please, try this one: http://wikisend.com/download/413650/squid.strace Regards, Jens Gesendet: Donnerstag, 23. Juli 2015 um 16:25 Uhr Von: Marcus Kool marcus.k...@urlfilterdb.com An: Jens Offenbach wolle5...@gmx.de Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching I get a 403 Access Denied error trying to download the trace file Marcus On 07/23/2015 10:39 AM, Jens Offenbach wrote: I have attached strace to Squid and waited until the download rate has decreased to 500 KB/sec. I used cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240. Here is the download link: http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace I hope it can help you. *Gesendet:* Donnerstag, 23. Juli 2015 um 13:29 Uhr *Von:* Marcus Kool marcus.k...@urlfilterdb.com *An:* Jens Offenbach wolle5...@gmx.de, Eliezer Croitoru elie...@ngtech.co.il, Amos Jeffries squ...@treenet.co.nz, squid-users@lists.squid-cache.org *Betreff:* Re: [squid-users] Squid3: 100 % CPU load during object caching I am not sure if it is relevant, maybe it is: I am developing an ICAP daemon and after the ICAP server sends a 100 continue Squid sends the object to the ICAP server in small chunks of varying sizes: 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc. Note that the interval of receiving the chunks is 1/1000th of a second. It seems that Squid forwards the object to the ICAP server every time it receives one or a few TCP packets. I have a suspicion that in the scenario of 100% CPU, large #write calls and low throughput a similar thing is happening: Squid physically stores a small part of the object many times, i.e. every time one or a few TCP packets arrive. Amos, is there a debug setting that can confirm/reject this suspicion? Marcus On 07/23/2015 04:25 AM, Jens Offenbach wrote: A test with ROCK cache_dir rock /var/cache/squid3 51200 gives very confusing results. I cleared the cache: rm -rf /var/cache/squid3/* squid -z squid http_proxy=http://139.2.57.120:3128/[http://139.2.57.120:3128/] wget http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar The download starts with 10 MB/sec and stays constant for 1 minutes, then it drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the download rate returns back to 10 MB/sec very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec. During caching progress, we have 100 % CPU usage and a disk IO that is corresponds with the download rate. For further investigations I give you my build properties: squid -v Squid Cache: Version 3.5.6 Service Name: squid configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3' '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-e s i' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/cache/squid3' '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr Von: Eliezer Croitoru elie...@ngtech.co.il An: squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching On 22/07/2015 21:59, Eliezer Croitoru wrote: Hey Jens, I have tested the issue with LARGE ROCK and not AUFS or UFS. Using squid or not my connection to the server is about 2.5 MBps (20Mbps). Squid is sitting on an intel atom with SSD drive and on a HIT case the download speed is more
Re: [squid-users] Squid3: 100 % CPU load during object caching
A test with ROCK cache_dir rock /var/cache/squid3 51200 gives very confusing results. I cleared the cache: rm -rf /var/cache/squid3/* squid -z squid http_proxy=http://139.2.57.120:3128/ wget http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar The download starts with 10 MB/sec and stays constant for 1 minutes, then it drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the download rate returns back to 10 MB/sec very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec. During caching progress, we have 100 % CPU usage and a disk IO that is corresponds with the download rate. For further investigations I give you my build properties: squid -v Squid Cache: Version 3.5.6 Service Name: squid configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3' '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/cache/squid3' '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr Von: Eliezer Croitoru elie...@ngtech.co.il An: squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching On 22/07/2015 21:59, Eliezer Croitoru wrote: Hey Jens, I have tested the issue with LARGE ROCK and not AUFS or UFS. Using squid or not my connection to the server is about 2.5 MBps (20Mbps). Squid is sitting on an intel atom with SSD drive and on a HIT case the download speed is more then doubled to 4.5 MBps(36Mbps). I have not tried it with AUFS yet. And I must admit that AUFS beats rock cache with speed. I have tried rock with basic cache_dir rock /var/spool/squid 8000 vs cache_dir aufs /var/spool/squid 8000 16 256 and the aufs cache HIT results more then doubles 3 the speed rock gave with default settings. So about 15MBps which is 120Mbps. I do not seem to feel what Jens feels but the 100% CPU might be because of spinning disk hangs while reading the file from disk. Amos, I remember that there were some suggestions how to tune large rock. Any hints? I can test it and make it a suggestion for big files. Eliezer ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid3: 100 % CPU load during object caching
Can you share the relevant squid.conf settings? Just to reproduce.. I have a dedicated testing server here which I can test the issue on. 8GB archive which might be an ISO and can be cached on AUFS\UFS and LARGE ROCK cache types. I am pretty sure that the maximum cache object size is one thing to change and waht more? From What I understand it should not be different for 2GB cached archive and to 8 GB cached archive. I have a local copy of centos 7 ISO which should be a test worthy object. Anything more you can add to the test subject? Eliezer On 22/07/2015 16:24, Jens Offenbach wrote: I checked the bug you have mentioned and I think I am confronted with the same issue. I was able to build and test Squid 3.5.6 on Ubuntu 14.04.2 x84_64. I observed the same behavior. I have tested an 8 GB archive file and I get 100 % CPU usage and a download rate of nearly 500 KB/sec when the object gets cached. I have attached strace to the running process, but I killed it after 30 minutes. The whole takes hours, although we have a 1-GBit ethernet: Process 4091 attached Process 4091 detached % time seconds usecs/call calls errors syscall -- --- --- - - 78.83 2.622879 1 1823951 write 12.29 0.408748 2 228029 2 read 6.18 0.205663 0 912431 1 epoll_wait 2.58 0.085921 0 456020 epoll_ctl 0.09 0.002919 0 6168 brk 0.02 0.000623 2 356 openat 0.01 0.000286 0 712 getdents 0.00 0.71 1 91 getrusage 0.00 0.38 0 362 close 0.00 0.03 2 2 sendto 0.00 0.01 0 3 1 recvfrom 0.00 0.00 0 2 open 0.00 0.00 0 3 stat 0.00 0.00 0 1 1 rt_sigreturn 0.00 0.00 0 1 kill 0.00 0.00 0 4 fcntl 0.00 0.00 0 2 2 unlink 0.00 0.00 0 1 getppid -- --- --- - - 100.00 3.327152 3428139 7 total Can I do anything that helps to get ride of this problem? Gesendet: Dienstag, 21. Juli 2015 um 17:37 Uhr Von: Amos Jeffries squ...@treenet.co.nz An: Jens Offenbach wolle5...@gmx.de, squid-users@lists.squid-cache.org squid-users@lists.squid-cache.org Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching On 22/07/2015 12:31 a.m., Jens Offenbach wrote: Thank you very much for your detailed explainations. We want to use Squid in order to accelerate our automated software setup processes via Puppet. Actually Squid will host only a very short amount of large objects (10-20). Its purpose is not to cache web traffic or little objects. Ah, Squid does not host, it caches. The difference may seem trivial at first glance but it is the critical factor between whether a proxy or a local web server is the best tool for the job. From my own experiences with Puppet, yes Squid is the right tool. But only because the Puppet server was using relatively slow python code to generate objects and not doing server-side caching on its own. If that situation has changed in recent years then Squids usefulness will also have changed. The hit-ratio for all the hosted objects will be very high, because most of our VMs require the same software stack. I will update mit config regarding to your comments! Thanks a lot! But actually I have still no idea, why the download rates are so unsatisfying. We are sill in the test phase. We have only one client that requests a large object from Squid and the transfer rates are lower than 1 MB/sec during cache build-up without any form of concurrency. Have vou got an idea what could be the source of the problem here? Why causes the Squid process 100 % CPU usage. I did not see any config causing the known 100% CPU bugs to be encountered in your case (eg. HTTPS going through delay pools guarantees 100% CPU). Which leads me to think its probably related to memory shuffling. (http://bugs.squid-cache.org/show_bug.cgi?id=3189 https://3c.gmx.net/mail/client/dereferrer?redirectUrl=http%3A%2F%2Fbugs.squid-cache.org%2Fshow_bug.cgi%3Fid%3D3189 appears to be the same and still unidentified) As for speed, if the CPU is maxed out by one particular action Squid wont have time for much other work. So things go slow. On the other hand Squid is also optimized for relatively high traffic usage. For very small client counts (such as under-10) it is effectively running in idle mode 99% of the time. The I/O event loop starts pausing for 10ms blocks waiting to see if some more useful amount of work can be done at the end of the wait. That can lead to apparent network slowdown as TCP gets up to 10ms delay per packet. But that should not be visible in CPU numbers. That said, 1 client can still max out Squid CPU and/or NIC throughput capacity on a single request if its pushing/pulling packets fast enough. If you can attach the strace tool to Squid when its consuming the CPU there might be some better hints about where to look. Cheers Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http
Re: [squid-users] Squid3: 100 % CPU load during object caching
I will send you my current settings tomorrow. I have used AUFS as caching format, but I have also tested UFS. The format seems to have no influence on the issue. I have tested the 1 GB Ubuntu 15.04 image (ubuntu-15.04-desktop-amd64.iso). This is the link http://releases.ubuntu.com/15.04/ubuntu-15.04-desktop-amd64.iso. If you want to stress caching more with large files. You can use one of those: https://surfer.nmr.mgh.harvard.edu/fswiki/Download But I think the Centos 7 ISO are large enough, In my test scenario, I have put all files on an internal web server with gives them in stable 120 MB/sec. So the problem does not come from a slow network connection. I have checked network connectivity with Iperf3 (= 900 MBit/sec) and made a direct wget without Squid. The file gets downloaded in high speed. Adding Squid in the communication flow which caches the file on the first request and the issue occurrs. After some minutes, the download rate drops to 500 KByte/sec and stays on this level together with 100 % CPU load. The download rate corresponds with the disk IO. The file gets written with 500 KByte/sec. Thank you very much! Gesendet: Mittwoch, 22. Juli 2015 um 18:28 Uhr Von: Eliezer Croitoru elie...@ngtech.co.il An: squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching Can you share the relevant squid.conf settings? Just to reproduce.. I have a dedicated testing server here which I can test the issue on. 8GB archive which might be an ISO and can be cached on AUFS\UFS and LARGE ROCK cache types. I am pretty sure that the maximum cache object size is one thing to change and waht more? From What I understand it should not be different for 2GB cached archive and to 8 GB cached archive. I have a local copy of centos 7 ISO which should be a test worthy object. Anything more you can add to the test subject? Eliezer On 22/07/2015 16:24, Jens Offenbach wrote: I checked the bug you have mentioned and I think I am confronted with the same issue. I was able to build and test Squid 3.5.6 on Ubuntu 14.04.2 x84_64. I observed the same behavior. I have tested an 8 GB archive file and I get 100 % CPU usage and a download rate of nearly 500 KB/sec when the object gets cached. I have attached strace to the running process, but I killed it after 30 minutes. The whole takes hours, although we have a 1-GBit ethernet: Process 4091 attached Process 4091 detached % time seconds usecs/call calls errors syscall -- --- --- - - 78.83 2.622879 1 1823951 write 12.29 0.408748 2 228029 2 read 6.18 0.205663 0 912431 1 epoll_wait 2.58 0.085921 0 456020 epoll_ctl 0.09 0.002919 0 6168 brk 0.02 0.000623 2 356 openat 0.01 0.000286 0 712 getdents 0.00 0.71 1 91 getrusage 0.00 0.38 0 362 close 0.00 0.03 2 2 sendto 0.00 0.01 0 3 1 recvfrom 0.00 0.00 0 2 open 0.00 0.00 0 3 stat 0.00 0.00 0 1 1 rt_sigreturn 0.00 0.00 0 1 kill 0.00 0.00 0 4 fcntl 0.00 0.00 0 2 2 unlink 0.00 0.00 0 1 getppid -- --- --- - - 100.00 3.327152 3428139 7 total Can I do anything that helps to get ride of this problem? Gesendet: Dienstag, 21. Juli 2015 um 17:37 Uhr Von: Amos Jeffries squ...@treenet.co.nz An: Jens Offenbach wolle5...@gmx.de, squid-users@lists.squid-cache.org squid-users@lists.squid-cache.org Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching On 22/07/2015 12:31 a.m., Jens Offenbach wrote: Thank you very much for your detailed explainations. We want to use Squid in order to accelerate our automated software setup processes via Puppet. Actually Squid will host only a very short amount of large objects (10-20). Its purpose is not to cache web traffic or little objects. Ah, Squid does not host, it caches. The difference may seem trivial at first glance but it is the critical factor between whether a proxy or a local web server is the best tool for the job. From my own experiences with Puppet, yes Squid is the right tool. But only because the Puppet server was using relatively slow python code to generate objects and not doing server-side caching on its own. If that situation has changed in recent years then Squids usefulness will also have changed. The hit-ratio for all the hosted objects will be very high, because most of our VMs require the same software stack. I will update mit config regarding to your comments! Thanks a lot! But actually I have still no idea, why the download rates are so unsatisfying. We are sill in the test phase. We have only one client that requests a large object from Squid and the transfer rates are lower than 1 MB/sec during cache build-up without any form of concurrency. Have vou got an idea what could be the source of the problem here? Why causes the Squid process 100 % CPU usage. I did
Re: [squid-users] Squid3: 100 % CPU load during object caching
Hey Jens, I have tested the issue with LARGE ROCK and not AUFS or UFS. Using squid or not my connection to the server is about 2.5 MBps (20Mbps). Squid is sitting on an intel atom with SSD drive and on a HIT case the download speed is more then doubled to 4.5 MBps(36Mbps). I have not tried it with AUFS yet. My testing machine is an ARCH linux with self compiled squid with the replacement of diskd to rock from the compilation options of arch linux. You can take a look at the HIT log at: http://paste.ngtech.co.il/pnhkglgsu Eliezer On 22/07/2015 21:07, Jens Offenbach wrote: I will send you my current settings tomorrow. I have used AUFS as caching format, but I have also tested UFS. The format seems to have no influence on the issue. I have tested the 1 GB Ubuntu 15.04 image (ubuntu-15.04-desktop-amd64.iso). This is the link http://releases.ubuntu.com/15.04/ubuntu-15.04-desktop-amd64.iso. If you want to stress caching more with large files. You can use one of those: https://surfer.nmr.mgh.harvard.edu/fswiki/Download But I think the Centos 7 ISO are large enough, In my test scenario, I have put all files on an internal web server with gives them in stable 120 MB/sec. So the problem does not come from a slow network connection. I have checked network connectivity with Iperf3 (= 900 MBit/sec) and made a direct wget without Squid. The file gets downloaded in high speed. Adding Squid in the communication flow which caches the file on the first request and the issue occurrs. After some minutes, the download rate drops to 500 KByte/sec and stays on this level together with 100 % CPU load. The download rate corresponds with the disk IO. The file gets written with 500 KByte/sec. Thank you very much! Gesendet: Mittwoch, 22. Juli 2015 um 18:28 Uhr Von: Eliezer Croitoru elie...@ngtech.co.il An: squid-users@lists.squid-cache.org Betreff: Re: [squid-users] Squid3: 100 % CPU load during object caching Can you share the relevant squid.conf settings? Just to reproduce.. I have a dedicated testing server here which I can test the issue on. 8GB archive which might be an ISO and can be cached on AUFS\UFS and LARGE ROCK cache types. I am pretty sure that the maximum cache object size is one thing to change and waht more? From What I understand it should not be different for 2GB cached archive and to 8 GB cached archive. I have a local copy of centos 7 ISO which should be a test worthy object. Anything more you can add to the test subject? Eliezer On 22/07/2015 16:24, Jens Offenbach wrote: I checked the bug you have mentioned and I think I am confronted with the same issue. I was able to build and test Squid 3.5.6 on Ubuntu 14.04.2 x84_64. I observed the same behavior. I have tested an 8 GB archive file and I get 100 % CPU usage and a download rate of nearly 500 KB/sec when the object gets cached. I have attached strace to the running process, but I killed it after 30 minutes. The whole takes hours, although we have a 1-GBit ethernet: Process 4091 attached Process 4091 detached % time seconds usecs/call calls errors syscall -- --- --- - - 78.83 2.622879 1 1823951 write 12.29 0.408748 2 228029 2 read 6.18 0.205663 0 912431 1 epoll_wait 2.58 0.085921 0 456020 epoll_ctl 0.09 0.002919 0 6168 brk 0.02 0.000623 2 356 openat 0.01 0.000286 0 712 getdents 0.00 0.71 1 91 getrusage 0.00 0.38 0 362 close 0.00 0.03 2 2 sendto 0.00 0.01 0 3 1 recvfrom 0.00 0.00 0 2 open 0.00 0.00 0 3 stat 0.00 0.00 0 1 1 rt_sigreturn 0.00 0.00 0 1 kill 0.00 0.00 0 4 fcntl 0.00 0.00 0 2 2 unlink 0.00 0.00 0 1 getppid -- --- --- - - 100.00 3.327152 3428139 7 total Can I do anything that helps to get ride of this problem? Gesendet: Dienstag, 21. Juli 2015 um 17:37 Uhr Von: Amos Jeffries squ...@treenet.co.nz An: Jens Offenbach wolle5...@gmx.de, squid-users@lists.squid-cache.org squid-users@lists.squid-cache.org Betreff: Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching On 22/07/2015 12:31 a.m., Jens Offenbach wrote: Thank you very much for your detailed explainations. We want to use Squid in order to accelerate our automated software setup processes via Puppet. Actually Squid will host only a very short amount of large objects (10-20). Its purpose is not to cache web traffic or little objects. Ah, Squid does not host, it caches. The difference may seem trivial at first glance but it is the critical factor between whether a proxy or a local web server is the best tool for the job. From my own experiences with Puppet, yes Squid is the right tool. But only because the Puppet server was using relatively slow python code to generate objects and not doing server-side caching on its own. If that situation has changed in recent years then Squids usefulness will also have changed. The hit-ratio for all the hosted objects will be very high, because most of our
Re: [squid-users] Squid3: 100 % CPU load during object caching
I checked the bug you have mentioned and I think I am confronted with the same issue. I was able to build and test Squid 3.5.6 on Ubuntu 14.04.2 x84_64. I observed the same behavior. I have tested an 8 GB archive file and I get 100 % CPU usage and a download rate of nearly 500 KB/sec when the object gets cached. I have attached strace to the running process, but I killed it after 30 minutes. The whole takes hours, although we have a 1-GBit ethernet: Process 4091 attached Process 4091 detached % time seconds usecs/call calls errors syscall -- --- --- - - 78.83 2.622879 1 1823951 write 12.29 0.408748 2 228029 2 read 6.18 0.205663 0 912431 1 epoll_wait 2.58 0.085921 0 456020 epoll_ctl 0.09 0.002919 0 6168 brk 0.02 0.000623 2 356 openat 0.01 0.000286 0 712 getdents 0.00 0.71 1 91 getrusage 0.00 0.38 0 362 close 0.00 0.03 2 2 sendto 0.00 0.01 0 3 1 recvfrom 0.00 0.00 0 2 open 0.00 0.00 0 3 stat 0.00 0.00 0 1 1 rt_sigreturn 0.00 0.00 0 1 kill 0.00 0.00 0 4 fcntl 0.00 0.00 0 2 2 unlink 0.00 0.00 0 1 getppid -- --- --- - - 100.00 3.327152 3428139 7 total Can I do anything that helps to get ride of this problem? Gesendet:Dienstag, 21. Juli 2015 um 17:37 Uhr Von:Amos Jeffries squ...@treenet.co.nz An:Jens Offenbach wolle5...@gmx.de, squid-users@lists.squid-cache.org squid-users@lists.squid-cache.org Betreff:Re: Aw: Re: [squid-users] Squid3: 100 % CPU load during object caching On 22/07/2015 12:31 a.m., Jens Offenbach wrote: Thank you very much for your detailed explainations. We want to use Squid in order to accelerate our automated software setup processes via Puppet. Actually Squid will host only a very short amount of large objects (10-20). Its purpose is not to cache web traffic or little objects. Ah, Squid does not host, it caches. The difference may seem trivial at first glance but it is the critical factor between whether a proxy or a local web server is the best tool for the job. >From my own experiences with Puppet, yes Squid is the right tool. But only because the Puppet server was using relatively slow python code to generate objects and not doing server-side caching on its own. If that situation has changed in recent years then Squids usefulness will also have changed. The hit-ratio for all the hosted objects will be very high, because most of our VMs require the same software stack. I will update mit config regarding to your comments! Thanks a lot! But actually I have still no idea, why the download rates are so unsatisfying. We are sill in the test phase. We have only one client that requests a large object from Squid and the transfer rates are lower than 1 MB/sec during cache build-up without any form of concurrency. Have vou got an idea what could be the source of the problem here? Why causes the Squid process 100 % CPU usage. I did not see any config causing the known 100% CPU bugs to be encountered in your case (eg. HTTPS going through delay pools guarantees 100% CPU). Which leads me to think its probably related to memory shuffling. (http://bugs.squid-cache.org/show_bug.cgi?id=3189 appears to be the same and still unidentified) As for speed, if the CPU is maxed out by one particular action Squid wont have time for much other work. So things go slow. On the other hand Squid is also optimized for relatively high traffic usage. For very small client counts (such as under-10) it is effectively running in idle mode 99% of the time. The I/O event loop starts pausing for 10ms blocks waiting to see if some more useful amount of work can be done at the end of the wait. That can lead to apparent network slowdown as TCP gets up to 10ms delay per packet. But that should not be visible in CPU numbers. That said, 1 client can still max out Squid CPU and/or NIC throughput capacity on a single request if its pushing/pulling packets fast enough. If you can attach the strace tool to Squid when its consuming the CPU there might be some better hints about where to look. Cheers Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid3: 100 % CPU load during object caching
On 22/07/2015 21:59, Eliezer Croitoru wrote: Hey Jens, I have tested the issue with LARGE ROCK and not AUFS or UFS. Using squid or not my connection to the server is about 2.5 MBps (20Mbps). Squid is sitting on an intel atom with SSD drive and on a HIT case the download speed is more then doubled to 4.5 MBps(36Mbps). I have not tried it with AUFS yet. And I must admit that AUFS beats rock cache with speed. I have tried rock with basic cache_dir rock /var/spool/squid 8000 vs cache_dir aufs /var/spool/squid 8000 16 256 and the aufs cache HIT results more then doubles 3 the speed rock gave with default settings. So about 15MBps which is 120Mbps. I do not seem to feel what Jens feels but the 100% CPU might be because of spinning disk hangs while reading the file from disk. Amos, I remember that there were some suggestions how to tune large rock. Any hints? I can test it and make it a suggestion for big files. Eliezer ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid3: 100 % CPU load during object caching
On 22/07/2015 12:31 a.m., Jens Offenbach wrote: Thank you very much for your detailed explainations. We want to use Squid in order to accelerate our automated software setup processes via Puppet. Actually Squid will host only a very short amount of large objects (10-20). Its purpose is not to cache web traffic or little objects. Ah, Squid does not host, it caches. The difference may seem trivial at first glance but it is the critical factor between whether a proxy or a local web server is the best tool for the job. From my own experiences with Puppet, yes Squid is the right tool. But only because the Puppet server was using relatively slow python code to generate objects and not doing server-side caching on its own. If that situation has changed in recent years then Squids usefulness will also have changed. The hit-ratio for all the hosted objects will be very high, because most of our VMs require the same software stack. I will update mit config regarding to your comments! Thanks a lot! But actually I have still no idea, why the download rates are so unsatisfying. We are sill in the test phase. We have only one client that requests a large object from Squid and the transfer rates are lower than 1 MB/sec during cache build-up without any form of concurrency. Have vou got an idea what could be the source of the problem here? Why causes the Squid process 100 % CPU usage. I did not see any config causing the known 100% CPU bugs to be encountered in your case (eg. HTTPS going through delay pools guarantees 100% CPU). Which leads me to think its probably related to memory shuffling. (http://bugs.squid-cache.org/show_bug.cgi?id=3189 appears to be the same and still unidentified) As for speed, if the CPU is maxed out by one particular action Squid wont have time for much other work. So things go slow. On the other hand Squid is also optimized for relatively high traffic usage. For very small client counts (such as under-10) it is effectively running in idle mode 99% of the time. The I/O event loop starts pausing for 10ms blocks waiting to see if some more useful amount of work can be done at the end of the wait. That can lead to apparent network slowdown as TCP gets up to 10ms delay per packet. But that should not be visible in CPU numbers. That said, 1 client can still max out Squid CPU and/or NIC throughput capacity on a single request if its pushing/pulling packets fast enough. If you can attach the strace tool to Squid when its consuming the CPU there might be some better hints about where to look. Cheers Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid3: 100 % CPU load during object caching
On 21/07/2015 7:59 p.m., Jens Offenbach wrote: I am running Squid3 3.3.8 on Ubuntu 14.04. Squid3 has been installed from the Ubuntu package repository. In my scenario, Squid has to cache big files = 1 GB. At the moment, I am getting very bad transfer rates lower that 1 MB/sec. I have checked the connectivity using iperf3. It gives my a bandwith of 853 Mbits/sec between the nodes. I have tried to investigate the problem and recognized that when there is no cache hit for a requested object, the Squid process reaches shortly after startup 100 % of one CPU core. The download rate drops down to 1 MB/sec. When I have a cache hit, I only get 30 MB/sec in my download. Is there someting wrong with my config? I have already used Squid 3.3.14. I get the same result. Unfortunately, I was not able to build Squid 3.5.5 and 3.5.6. Squid-3 is better able to cope with large objects than Squid-2 was. But there are still significant problems. Firstly, you only have space in memory for 4x 1GB objects. Total. if you are dealing with such large objects at any regular frequency, you need at least a much larger cache_mem setting. Secondly, consider that Squid-3.3 places *all* active transactions into cache_mem. 4GB of memory cache can store ~4 million x 1KB transactions, or only 4 x 1GB transactions. If you have a cache full of small objects happily sitting in memory. Then a requests for a 1GB object comes in, a hugh number of those small objects need to be pushed out of memory cache onto disk, the memory reallocated for use by the big one, and possibly 1GB object loaded from disk into memory cache. Then consider that GB sized object sitting in cache as it gets near to being the oldest in memory. The next request is probably a puny little 0-1KB object, Squid may have to repeat all the GB size shufflings to and from disk just to make memory space for that KB. As you can imagine any one part of that process takes a lot of work and time with a big object involved as compared to only small objects being involved. The whole set of actions can be excruciatingly painful if the proxy is busy. Thirdly, you also only have 88GB of disk cache total. Neither that nor the memory cache is sufficient to be trying to cache GB sized objects. The tradeoff is whether one GB size object is going to get enough HITs often enough to be worth not caching the million or so smaller objects that could be taking its place. For most uses the tradeoff only makes sense with high traffic on the large objects and/or TB of disk space. My rule-of-thumb advice for caching is to keep it so that you can store at least a few thousand maximum-sized objects at once in the allocated size. So 4GB memory cache reasonable for 1MB size objects, 80GB disk cache is reasonable for ~100MB sized objects. That keeps almost all web page traffic able to be in memory, bigger but popular media/video objects on disk. And the big things like Windows Service Packs or whole DVD downloads get slower network fetches as needed. If those latter are actually a problem for you get a bigger disk cache, you *will* need it. And a free audit for your config... Here is my squid.conf: # ACCESS CONTROLS # acl intranetsrc 139.2.0.0/16 acl intranetsrc 193.96.112.0/21 acl intranetsrc 192.109.216.0/24 acl intranetsrc 100.1.4.0/22 acl localnetsrc 10.0.0.0/8 acl localnetsrc 172.16.0.0/12 acl localnetsrc 192.168.0.0/16 acl localnetsrc fc00::/7 acl localnetsrc fe80::/10 acl to_intranet dst 139.2.0.0/16 acl to_intranet dst 193.96.112.0/21 acl to_intranet dst 192.109.216.0/24 acl to_intranet dst 100.1.4.0/22 acl to_localnet dst 10.0.0.0/8 acl to_localnet dst 172.16.0.0/12 acl to_localnet dst 192.168.0.0/16 acl to_localnet dst fc00::/7 acl to_localnet dst fe80::/10 The intended purpose behind the localnet and to_localnet ACLs is that they are matching your intranet / LAN / local network ranges. The ones we distribute are just common standard ranges. You can simplify your config by adding the intranet ranges to localnet and dropping all the 'intranet' ACLs. ... BUT ... http_access allow manager localhost http_access deny manager http_access allow localnet http_access allow localhost http_access deny all ... noting how the intranet ACLs are not used to permit access through the proxy. Maybe just dropping them entirely is better. If this is a working proxy they are not being used. # NETWORK OPTIONS # http_port 0.0.0.0:3128 # OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM # cache_peer proxy.mycompany.de parent 8080 0 no-query no-digest # MEMORY CACHE OPTIONS #
Re: [squid-users] Squid3: 100 % CPU load during object caching
On 21/07/2015 10:59, Jens Offenbach wrote: Is there someting wrong with my config? I have already used Squid 3.3.14. I get the same result. Unfortunately, I was not able to build Squid 3.5.5 and 3.5.6. What was the issue? I am using 3.5.6 on 14.04.2 64 bit. Eliezer ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
