Re: [squid-users] squid 3.x expected max throughput
On Apr 11, 2013, at 12:13 PM, Amos Jeffries squ...@treenet.co.nz wrote: On 11/04/2013 12:23 a.m., Youssef Ghorbal wrote: I was aware of that page. As you said, it's often RPS so it's not relevant for me. It is more relevant than you seem to think. Squid processing overheads are tied tightly to the request parsing and ACL testing processes. These are relatively fixed overheads for each request regardless of the request size. Thank you for the explanation. It confirms what I was expecting myself. What I meant, is that in my case I have ONE big object (the infamous 1G file) and 1 client and I want to know what to expect on the troughput point of view. As kinkie said, in my scenario, Squid is basically a network pipe. the overhead (ACL parsing etc is done one time) Maybe, the question here is how does Squid handle data buffers, or at which rate does it perform server reads on the one side and to client writes on the other. IIRC the 50Mbps traffic was on average HTTP request traffic objects in a medium sized ISP - we do not have any good detaisl on what that average was though. With very little extra overheads the same proxy on same hardware might also reach 100Mbps. All that is required is the avg object size to go from, for example, 16KB to 32KB which are both within the bounds of average HTTP traffic sizes (11-63KB). Yousef
Re: [squid-users] squid 3.x expected max throughput
On Apr 10, 2013, at 10:37 AM, Kinkie gkin...@gmail.com wrote: On Tue, Apr 9, 2013 at 11:35 PM, Youssef Ghorbal d...@pasteur.fr wrote: Hello, Is there any recent figures about max throughput to be expected from a squid 3.x install (on recent hardware) in the scenario of a single stream downloading a large file ( 1GB) (read not cacheable) I'm aware that's not a performance metric per se, but it's one of the scenarios we have to deal with. Few weeks ago, Amos talked about 50Mb/s (client + server) for a squid 3.1 Hi, 50mb/s seems a very conservative estimate to me; in that scenario Squid is essentially acting as a network pipe. Assuming this is a lab (and we can thus ignore bandwidth and latency on the internet link), the expectation is that this kind of scenario for squid will be CPU and network I/O bound, so in order to give any sensible answer we'd need to know what kind of network interface you would use (fast-ethernet? Giga-ethernet copper? Giga-ethernet fiber? Even faster?), what kind of CPU and what kind of system architecture (server-class? pc-class? virtual?) I'm agree, it's a LAB and we can ignore bandwidth and latency of Internet link. And it's exactly what you said it's a scenario where Squid is essentially acting as a network pipe. Here is a summary of the hardware : - 1Gb ethernet NIC (copper) : the same of client and server traffic. - 16GB RAM - 2 CPUs Quad Core (Xeon E5420 2.5Ghz per core) : I think that the many cores are not relevent since a single stream will eventually be handled by a single core. - FreeBSD 8.3 amd64 What I'm seeing right now is ~50Mb/s on 3.1 release (as Amos said earlier) which seems very conservative estimate to me too, and I was seeking infos on what can be expected in a perfect world. If it's the current best figures I can get, that's fine and I'll not be looking to optmise anything else. If it can do a lot better (let's say 10 times better) I'll try to investgate time to reach this (upgrade to the last release, tune the configuration, tune the system etc) Youssef
Re: [squid-users] squid 3.x expected max throughput
I was aware of that page. As you said, it's often RPS so it's not relevant for me. Youssef On Apr 10, 2013, at 2:02 PM, Kinkie gkin...@gmail.com wrote: You probably want to check http://wiki.squid-cache.org/KnowledgeBase/Benchmarks Unfortunately the benchmarks reported are often expressed as RPS and not bandwidth. On Wed, Apr 10, 2013 at 1:21 PM, Youssef Ghorbal d...@pasteur.fr wrote: On Apr 10, 2013, at 10:37 AM, Kinkie gkin...@gmail.com wrote: On Tue, Apr 9, 2013 at 11:35 PM, Youssef Ghorbal d...@pasteur.fr wrote: Hello, Is there any recent figures about max throughput to be expected from a squid 3.x install (on recent hardware) in the scenario of a single stream downloading a large file ( 1GB) (read not cacheable) I'm aware that's not a performance metric per se, but it's one of the scenarios we have to deal with. Few weeks ago, Amos talked about 50Mb/s (client + server) for a squid 3.1 Hi, 50mb/s seems a very conservative estimate to me; in that scenario Squid is essentially acting as a network pipe. Assuming this is a lab (and we can thus ignore bandwidth and latency on the internet link), the expectation is that this kind of scenario for squid will be CPU and network I/O bound, so in order to give any sensible answer we'd need to know what kind of network interface you would use (fast-ethernet? Giga-ethernet copper? Giga-ethernet fiber? Even faster?), what kind of CPU and what kind of system architecture (server-class? pc-class? virtual?) I'm agree, it's a LAB and we can ignore bandwidth and latency of Internet link. And it's exactly what you said it's a scenario where Squid is essentially acting as a network pipe. Here is a summary of the hardware : - 1Gb ethernet NIC (copper) : the same of client and server traffic. - 16GB RAM - 2 CPUs Quad Core (Xeon E5420 2.5Ghz per core) : I think that the many cores are not relevent since a single stream will eventually be handled by a single core. - FreeBSD 8.3 amd64 What I'm seeing right now is ~50Mb/s on 3.1 release (as Amos said earlier) which seems very conservative estimate to me too, and I was seeking infos on what can be expected in a perfect world. If it's the current best figures I can get, that's fine and I'll not be looking to optmise anything else. If it can do a lot better (let's say 10 times better) I'll try to investgate time to reach this (upgrade to the last release, tune the configuration, tune the system etc) Youssef
[squid-users] squid 3.x expected max throughput
Hello, Is there any recent figures about max throughput to be expected from a squid 3.x install (on recent hardware) in the scenario of a single stream downloading a large file ( 1GB) (read not cacheable) I'm aware that's not a performance metric per se, but it's one of the scenarios we have to deal with. Few weeks ago, Amos talked about 50Mb/s (client + server) for a squid 3.1 Thank you Youssef
Re: [squid-users] investigate squid eating 100% CPU
[…] client-squid : 31Mbps. […] Squid-server : 28Mbps Total: 59Mbps. Which is slightly higher than the known good performance limit for Squid-3.1. Which is up to ~50Mbps, tuning both in Squid and the system can reach around 100Mbps IIRC. But that sort of numbers you are looking at specific traffic types as well. Ah ! Thank you very much for this valuable figures. 50Mb/s is not what I was really hoping/expecting. I was naively thinking that Squid can handle multi 100Mb/s out of the box (at least with no ACLs, and other fancy stuff) We have 2x100Mb/s Internet connections in front of this proxy. select_loops = 2316.561716/sec snip syscalls.sock.reads = 1316.883520/sec syscalls.sock.writes = 1770.096831/sec It looks like sending and receiving data in ~2KB chunks, just over one packet per cycle. syscalls.sock.recvfroms = 58.203913/sec syscalls.sock.sendtos = 30.974986/sec cpu_time = 117.934176 seconds wall_time = 300.048557 seconds cpu_usage = 39.305030% CPU usage at 40%. Squid-3.1 does use around 30% CPU to push data at line rate, which is what this set of counters seems to be showing. We have done a bunch of work aimed at improving things all over Squid in the more recent releases, if you want to assist the developers tracking this down please upgrade to the latest release before delving too deeply into the analysis as all fixes need to go in through 3.HEAD anyway and you maybe repeating already done work (only might there is still CPU issues hiding in the latest development code). Point taken, I'll do that. What do things look like when its at or close to 100% CPU? Basically, some users complain about browsing slowing down. But the main complaint is about download speeds of large files (~50MB - GB) (but this can be explaind by the natural limits of Squid) Also any sign of the OS swapping virtual memory when the CPU climbs? that is a known performance killer for a number of reasons we can't fix easily. Already checked this area, no swapping involved. Amos Youssef
[squid-users] investigate squid eating 100% CPU
Hello, We have a Squid 3.1.23 running on a FreeBSD 8.3 (amd64) The proxy is used to handle web access for ~2500 workstations and in pure proxy/filter (squidGaurd) mode with no cache (all disk caching is disabled) It's not a tranparent/intercepting proxy, just a plain explicit proxy mode. What we see, is that the squid process is using 100% of CPU (userland CPU usage, not kernel) all the time. Even in late night when the whole traffic is very minimalistic. What I'm looking for is some advice on how to track down what is causing this CPU misbehaviour. Maybe it's some stupid config option not suitable for this kind of setup, maybe a bug etc. What would be the tools/methodology that I can use to profile the running process. Any help/suggestion would be really appreciated. Youssef Ghorbal squid -v Squid Cache: Version 3.1.23 configure options: '--with-default-user=squid' '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/var/squid' '--sysconfdir=/usr/local/etc/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid/squid.pid' '--enable-removal-policies=lru heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll' '--disable-translation' '--disable-ecap' '--disable-loadable-modules' '--enable-auth=basic digest negotiate ntlm' '--enable-basic-auth-helpers=DB NCSA PAM MSNT SMB squid_radius_auth LDAP YP' '--enable-digest-auth-helpers=password ldap' '--enable-external-acl-helpers=ip_user session unix_group wbinfo_group ldap_group' '--enable-ntlm-auth-helpers=smb_lm' '--enable-storeio=ufs diskd aufs' '--enable-disk-io=AIO Blocking DiskDaemon DiskThreads' '--enable-delay-pools' '--enable-icap-client' '--enable-kqueue' '--with-large-files' '--enable-stacktraces' '--disable-optimizations' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=amd64-portbld-freebsd8.3' 'build_alias=amd64-portbld-freebsd8.3' 'CC=cc' 'CFLAGS=-pipe -I/usr/local/include -g -g -DLDAP_DEPRECATED' 'LDFLAGS= -L/usr/local/lib' 'CPPFLAGS=' 'CXX=c++' 'CXXFLAGS=-pipe -I/usr/local/include -g -g -DLDAP_DEPRECATED' 'CPP=cpp' --with-squid=/wrkdirs/usr/ports/www/squid31/work/squid-3.1.23 --enable-ltdl-convenience
Re: [squid-users] investigate squid eating 100% CPU
On Mar 26, 2013, at 1:50 PM, FredB fredbm...@free.fr wrote: Are you using delay_pool ? Nope, we are not using delay_pools.
Re: [squid-users] investigate squid eating 100% CPU
The current FreeBSD ports available for squid are squid31 and squid32 I'll be able to upgrade to the latest 3.2 but not further. Youssef - On Mar 26, 2013, at 1:50 PM, Amos Jeffries squ...@treenet.co.nz wrote: The first step in debugging any problem like this is to upgrade to the latest version and see if it has been resolved. The current latest is Squid-3.3.3. Amos On 27/03/2013 1:33 a.m., Alexandre Chappaz wrote: Hi, you can activate the full debug launch squid -k debug with the service running, and check what comes in the cache.log. squid -k parse will audit your config file. Look for WARNING in the output of this command. the cachemanager can be usefull to see the actual activity of your squid : squidclient localhost mgr:5min gives you the last 5 min stats. (see if the n° of req/s is coherent with what you expect ) Bonne chance Alex 2013/3/26 Youssef Ghorbal d...@pasteur.fr: Hello, We have a Squid 3.1.23 running on a FreeBSD 8.3 (amd64) The proxy is used to handle web access for ~2500 workstations and in pure proxy/filter (squidGaurd) mode with no cache (all disk caching is disabled) It's not a tranparent/intercepting proxy, just a plain explicit proxy mode. What we see, is that the squid process is using 100% of CPU (userland CPU usage, not kernel) all the time. Even in late night when the whole traffic is very minimalistic. What I'm looking for is some advice on how to track down what is causing this CPU misbehaviour. Maybe it's some stupid config option not suitable for this kind of setup, maybe a bug etc. What would be the tools/methodology that I can use to profile the running process. Any help/suggestion would be really appreciated. Youssef Ghorbal squid -v Squid Cache: Version 3.1.23 configure options: '--with-default-user=squid' '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/var/squid' '--sysconfdir=/usr/local/etc/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid/squid.pid' '--enable-removal-policies=lru heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll' '--disable-translation' '--disable-ecap' '--disable-loadable-modules' '--enable-auth=basic digest negotiate ntlm' '--enable-basic-auth-helpers=DB NCSA PAM MSNT SMB squid_radius_auth LDAP YP' '--enable-digest-auth-helpers=password ldap' '--enable-external-acl-helpers=ip_user session unix_group wbinfo_group ldap_group' '--enable-ntlm-auth-helpers=smb_lm' '--enable-storeio=ufs diskd aufs' '--enable-disk-io=AIO Blocking DiskDaemon DiskThreads' '--enable-delay-pools' '--enable-icap-client' '--enable-kqueue' '--with-large-files' '--enable-stacktraces' '--disable-optimizations' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=amd64-portbld-freebsd8.3' 'build_alias=amd64-portbld-freebsd8.3' 'CC=cc' 'CFLAGS=-pipe -I/usr/local/include -g -g -DLDAP_DEPRECATED' 'LDFLAGS= -L/usr/local/lib' 'CPPFLAGS=' 'CXX=c++' 'CXXFLAGS=-pipe -I/usr/local/include -g -g -DLDAP_DEPRECATED' 'CPP=cpp' --with-squid=/wrkdirs/usr/ports/www/squid31/work/squid-3.1.23 --enable-ltdl-convenience
Re: [squid-users] investigate squid eating 100% CPU
On Mar 26, 2013, at 1:33 PM, Alexandre Chappaz alexandrechap...@gmail.com wrote: Hi, you can activate the full debug launch squid -k debug with the service running, and check what comes in the cache.log. I'll give it a try. How to stop debug by the way ? just squid -k debug again ? squid -k parse will audit your config file. Look for WARNING in the output of this command. This command does not retrun any warning. the cachemanager can be usefull to see the actual activity of your squid : squidclient localhost mgr:5min gives you the last 5 min stats. (see if the n° of req/s is coherent with what you expect ) Here after the output of the mgr:5min It show that we are around 168 req/s for a cpu usage around 99% I don't think that 160 req/s is such a big number that can explain full CPU time. sample_start_time = 1364316830.58546 (Tue, 26 Mar 2013 16:53:50 GMT) sample_end_time = 1364317130.157822 (Tue, 26 Mar 2013 16:58:50 GMT) client_http.requests = 168.400939/sec client_http.hits = 20.203314/sec client_http.errors = 0.013329/sec client_http.kbytes_in = 479.431347/sec client_http.kbytes_out = 4713.703475/sec client_http.all_median_svc_time = 0.150482 seconds client_http.miss_median_svc_time = 0.167753 seconds client_http.nm_median_svc_time = 0.030657 seconds client_http.nh_median_svc_time = 0.097357 seconds client_http.hit_median_svc_time = 0.074090 seconds server.all.requests = 148.814088/sec server.all.errors = 0.00/sec server.all.kbytes_in = 4349.643949/sec server.all.kbytes_out = 466.209055/sec server.http.requests = 132.799387/sec server.http.errors = 0.00/sec server.http.kbytes_in = 2908.297586/sec server.http.kbytes_out = 339.334374/sec server.ftp.requests = 0.013329/sec server.ftp.errors = 0.00/sec server.ftp.kbytes_in = 4.145295/sec server.ftp.kbytes_out = 0.003332/sec server.other.requests = 16.001371/sec server.other.errors = 0.00/sec server.other.kbytes_in = 1437.204400/sec server.other.kbytes_out = 126.868017/sec icp.pkts_sent = 0.00/sec icp.pkts_recv = 0.00/sec icp.queries_sent = 0.00/sec icp.replies_sent = 0.00/sec icp.queries_recv = 0.00/sec icp.replies_recv = 0.00/sec icp.replies_queued = 0.00/sec icp.query_timeouts = 0.00/sec icp.kbytes_sent = 0.00/sec icp.kbytes_recv = 0.00/sec icp.q_kbytes_sent = 0.00/sec icp.r_kbytes_sent = 0.00/sec icp.q_kbytes_recv = 0.00/sec icp.r_kbytes_recv = 0.00/sec icp.query_median_svc_time = 0.00 seconds icp.reply_median_svc_time = 0.00 seconds dns.median_svc_time = 0.058152 seconds unlink.requests = 0.00/sec page_faults = 0.00/sec select_loops = 85.811603/sec select_fds = 0.00/sec average_select_fd_period = 0.00/fd median_select_fds = 0.00 swap.outs = 0.00/sec swap.ins = 0.00/sec swap.files_cleaned = 0.00/sec aborted_requests = 2.335894/sec syscalls.disk.opens = 0.00/sec syscalls.disk.closes = 0.00/sec syscalls.disk.reads = 0.00/sec syscalls.disk.writes = 0.00/sec syscalls.disk.seeks = 0.00/sec syscalls.disk.unlinks = 0.00/sec syscalls.sock.accepts = 81.882903/sec syscalls.sock.sockets = 68.897201/sec syscalls.sock.connects = 65.285063/sec syscalls.sock.binds = 68.897201/sec syscalls.sock.closes = 131.846369/sec syscalls.sock.reads = 982.381577/sec syscalls.sock.writes = 1513.369196/sec syscalls.sock.recvfroms = 57.544291/sec syscalls.sock.sendtos = 34.428607/sec cpu_time = 299.780455 seconds wall_time = 300.099276 seconds cpu_usage = 99.893761%
Re: [squid-users] investigate squid eating 100% CPU
the cachemanager can be usefull to see the actual activity of your squid : squidclient localhost mgr:5min gives you the last 5 min stats. (see if the n° of req/s is coherent with what you expect ) Here after the output of the mgr:5min It show that we are around 168 req/s for a cpu usage around 99% I don't think that 160 req/s is such a big number that can explain full CPU time. Forgot to mention that it somtimes drops to ~30% CPU for some periods. For example right now we are : client_http.requests = 160.370710/sec cpu_usage = 39.305030% Which makes me think of some kind of requests getting it to go crazy. [complete output here] sample_start_time = 1364317190.222942 (Tue, 26 Mar 2013 16:59:50 GMT) sample_end_time = 1364317490.271499 (Tue, 26 Mar 2013 17:04:50 GMT) client_http.requests = 160.370710/sec client_http.hits = 17.510499/sec client_http.errors = 0.016664/sec client_http.kbytes_in = 278.614904/sec client_http.kbytes_out = 3589.275718/sec client_http.all_median_svc_time = 0.092188 seconds client_http.miss_median_svc_time = 0.097357 seconds client_http.nm_median_svc_time = 0.000911 seconds client_http.nh_median_svc_time = 0.016481 seconds client_http.hit_median_svc_time = 0.000911 seconds server.all.requests = 145.123178/sec server.all.errors = 0.00/sec server.all.kbytes_in = 3294.903365/sec server.all.kbytes_out = 269.543039/sec server.http.requests = 130.565534/sec server.http.errors = 0.00/sec server.http.kbytes_in = 2860.440352/sec server.http.kbytes_out = 179.021024/sec server.ftp.requests = 0.00/sec server.ftp.errors = 0.00/sec server.ftp.kbytes_in = 0.00/sec server.ftp.kbytes_out = 0.00/sec server.other.requests = 14.557644/sec server.other.errors = 0.00/sec server.other.kbytes_in = 434.463013/sec server.other.kbytes_out = 90.525348/sec icp.pkts_sent = 0.00/sec icp.pkts_recv = 0.00/sec icp.queries_sent = 0.00/sec icp.replies_sent = 0.00/sec icp.queries_recv = 0.00/sec icp.replies_recv = 0.00/sec icp.replies_queued = 0.00/sec icp.query_timeouts = 0.00/sec icp.kbytes_sent = 0.00/sec icp.kbytes_recv = 0.00/sec icp.q_kbytes_sent = 0.00/sec icp.r_kbytes_sent = 0.00/sec icp.q_kbytes_recv = 0.00/sec icp.r_kbytes_recv = 0.00/sec icp.query_median_svc_time = 0.00 seconds icp.reply_median_svc_time = 0.00 seconds dns.median_svc_time = 0.014637 seconds unlink.requests = 0.00/sec page_faults = 0.00/sec select_loops = 2316.561716/sec select_fds = 0.00/sec average_select_fd_period = 0.00/fd median_select_fds = 0.00 swap.outs = 0.00/sec swap.ins = 0.00/sec swap.files_cleaned = 0.00/sec aborted_requests = 2.159650/sec syscalls.disk.opens = 0.00/sec syscalls.disk.closes = 0.00/sec syscalls.disk.reads = 0.00/sec syscalls.disk.writes = 0.00/sec syscalls.disk.seeks = 0.00/sec syscalls.disk.unlinks = 0.00/sec syscalls.sock.accepts = 105.942852/sec syscalls.sock.sockets = 66.179288/sec syscalls.sock.connects = 62.956477/sec syscalls.sock.binds = 66.179288/sec syscalls.sock.closes = 124.793135/sec syscalls.sock.reads = 1316.883520/sec syscalls.sock.writes = 1770.096831/sec syscalls.sock.recvfroms = 58.203913/sec syscalls.sock.sendtos = 30.974986/sec cpu_time = 117.934176 seconds wall_time = 300.048557 seconds cpu_usage = 39.305030%
Re: [squid-users] investigate squid eating 100% CPU
On Mar 26, 2013, at 1:19 PM, Youssef Ghorbal d...@pasteur.fr wrote: Hello, We have a Squid 3.1.23 running on a FreeBSD 8.3 (amd64) The proxy is used to handle web access for ~2500 workstations and in pure proxy/filter (squidGaurd) mode with no cache (all disk caching is disabled) It's not a tranparent/intercepting proxy, just a plain explicit proxy mode. What we see, is that the squid process is using 100% of CPU (userland CPU usage, not kernel) all the time. Even in late night when the whole traffic is very minimalistic. What I'm looking for is some advice on how to track down what is causing this CPU misbehaviour. Maybe it's some stupid config option not suitable for this kind of setup, maybe a bug etc. What would be the tools/methodology that I can use to profile the running process. Any help/suggestion would be really appreciated. Forgot to ask, is there any well know squid CPU bound operations I can start to focus on in order to narrow down the problem ? ACL checks ? Peer selection ? Youssef
