Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
On 15/10/2015 7:34 p.m., Reet Vyas wrote: > HI > > I am getting this error in my squid, > > Squid Version : 3.5.3, > > Can anyone help me out with this cause browsing is slow when I get this in > cache.log file. > Two problems; #0 it is not an error. Errors get logged with the word "error" or "ERROR" or "FATAL". #1 your disk I/O queue was not big enough for the traffic being processed through it. Squid is increasing the queue size automatically, and warning you about the change. That is all. On older Squid you can expect a few (maybe a few dozen) on startup, and on the first time it goes through a traffic peak. queue growth is exponential. It is only a problem if it continues for a long time. #2 your Squid version is outdated. Current Squid version is 3.5.10 and includes a number of bug fixes relating to disk I/O issues. Any one of those could be contributing to the "slow". Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] squidaio_queue_request : warning - queue congestion
On 2/02/2012 7:11 p.m., Mustafa Raji wrote: --- On Wed, 2/1/12, Amos Jeffrieswrote: From: Amos Jeffries On 02.02.2012 02:17, Mustafa Raji wrote: hi squid user i use squid 3.1.11 with 3 cach_dir aufs 5000 MB and 4GB ram for computer and 512 MB for squid in cache_mem some times i gate this message in my cache.log please can you tell me what is the problem clientProcessRequest: Invalid Request 2012/02/01 08:18:01| clientProcessRequest: Invalid Request 2012/02/01 08:18:15| clientProcessRequest: Invalid Request 2012/02/01 08:29:32| clientProcessRequest: Invalid Request 2012/02/01 08:36:32| clientProcessRequest: Invalid Request 2012/02/01 08:40:34| clientProcessRequest: Invalid Request Somebody sending invalid HTTP requests to your proxy. 2012/02/01 08:57:38| squidaio_queue_request: WARNING - Queue congestion http://wiki.squid-cache.org/KnowledgeBase/QueueCongestion Amos Dear Amos please can you tell me what you mean in somebody sending invalid http requests, i read about this in internet and i figure this happen when there is a space in the http header so squid consider it as a invalide request is this what you mean or another thank That error message does not say what particularly was wrong. There are great many things which are invalid HTTP. If you care you will need to dig down deeper into the HTTP traffic syntax. Amos
Re: [squid-users] squidaio_queue_request : warning - queue congestion
On 02.02.2012 02:17, Mustafa Raji wrote: hi squid user i use squid 3.1.11 with 3 cach_dir aufs 5000 MB and 4GB ram for computer and 512 MB for squid in cache_mem some times i gate this message in my cache.log please can you tell me what is the problem clientProcessRequest: Invalid Request 2012/02/01 08:18:01| clientProcessRequest: Invalid Request 2012/02/01 08:18:15| clientProcessRequest: Invalid Request 2012/02/01 08:29:32| clientProcessRequest: Invalid Request 2012/02/01 08:36:32| clientProcessRequest: Invalid Request 2012/02/01 08:40:34| clientProcessRequest: Invalid Request Somebody sending invalid HTTP requests to your proxy. 2012/02/01 08:57:38| squidaio_queue_request: WARNING - Queue congestion http://wiki.squid-cache.org/KnowledgeBase/QueueCongestion Amos
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
Mike Diggins wrote: OS: RHEL 5.3 64 Bit Squid: squid-2.7.STABLE9 Decided to do a very light load test on a new Squid server (specs above). I'm running the AUFS file system cache_dir aufs /usr/local/squid/var/cache 2 16 256 The test ran for about an hour with about 10 clients loading random web pages. The CPU/ WAIT IO never got above 2%, so I was surprised to see a few of these warnings in cache.log: Squid spends a huge amount of its time processing the HTTP headers to figure out what needs doing. Disk IO is only a very small and often short portion of the time for most requests. squidaio_queue_request: WARNING - Queue congestion Saw about 5 or 6 of these over the hour but they seemed to slow down (or stop) as time went on. I did some reading and found this: Queue congestion starts appearing if there are 8 operations queued and not yet handled by the IO threads. The limit is then doubled (to 16) before the next warning appears. And so on... Does this mean the warning limits are doubled each time? If so, that would seem to just mask the problem, no? I'm not sure if I have a problem or not. Summary: 5-6 seems okay to me. Slowing and stopping is even better. Details: Yes it keeps doubling. The queue being mentioned is the buffer between NIC bandwidth and disk bandwidth. It is self-regulating. There is a client waiting for each operation to complete and if they are delayed in the queue the client sees longer loading times and is slower to fetch the next object. On top of that, the warning gets shown at the time when increasing RAM buffers to cope with any extra load going into the future. With slow disks or fast NIC plus some few MB of client traffic it can be expected to overflow and be expanded automatically a few times. Also seen shortly after startup if doing a DIRTY cache rebuild or at the buildup to peak network traffic while Squid learns what sort of queue buffer is needed to cope. It is only a problem if it continues and does not go away after these events when the disk(s) should start catching up again. That type of behaviour means the disk is too slow for the *average* client traffic load being pushed through it. Multiple disks or proxies can help if that is the case. I don't have it easily at hand, but there is a formula for disk bandwidth you can calculate from the RPMs. (4KB per RPM or something). That is a very rough estimate of how much traffic is flowing to/from the disk per second before this warning shows up the first time, even a small sustained increase above that causes the queue to grow a few times. Large amounts of extra AIO threads can appear to help. But do so by having more buffer space in RAM used by all the threads. In reality its better to have only fine tuning and multiple disks. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.7 Beta testers wanted for 3.2.0.1
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
ons 2010-02-17 klockan 21:40 -0800 skrev Tory M Blue: And sorry sleeping was just my way of citing the box shows no load, almost no IO 4-5 when I'm hitting it hard. I do not see this issue with lesser threads, it's only when I turn up the juice. But with turning up the connections per second I would expect to see some type of load and I see none. Anything in /var/log/messages? The above problem description is almost an exact match for Linux iptables connectiontracking table limit being hit. Regards Henrik
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
On Thu, Feb 18, 2010 at 12:27 AM, Henrik Nordstrom hen...@henriknordstrom.net wrote: ons 2010-02-17 klockan 21:40 -0800 skrev Tory M Blue: And sorry sleeping was just my way of citing the box shows no load, almost no IO 4-5 when I'm hitting it hard. I do not see this issue with lesser threads, it's only when I turn up the juice. But with turning up the connections per second I would expect to see some type of load and I see none. Anything in /var/log/messages? The above problem description is almost an exact match for Linux iptables connectiontracking table limit being hit. Regards Henrik Thanks Henrik, nothing in /var/logs/messages or even dmesg and iptables Not running. No rules in place, service shutdown. Thats not the culprit, with less than 12 children at the beginning of my run 2010/02/17 10:29:51| Completed Validation Procedure 2010/02/17 10:29:51| Validated 948594 Entries 2010/02/17 10:29:51| store_swap_size = 3794376k 2010/02/17 10:29:51| storeLateRelease: released 0 objects 2010/02/18 09:53:08| squidaio_queue_request: WARNING - Queue congestion 2010/02/18 09:53:12| squidaio_queue_request: WARNING - Queue congestion 2010/02/18 09:53:17| squidaio_queue_request: WARNING - Queue congestion Even dropped my thread count and as soon as my load test starts (get maybe 10 children launched), I get the error 2010/02/18 09:56:18| squidaio_queue_request: WARNING - Queue congestion 2010/02/18 09:56:28| squidaio_queue_request: WARNING - Queue congestion Okay I've found some issues that I had not seen before, Feb 18 18:37:06 kvm0 kernel: nf_conntrack: table full, dropping packet. I would like to kick the netfilter team and fedora team in the shins. The issue was my squid boxes are virtual and the errors were being logged on the domain box (not domain as in MS). So now I'm trying to go through the system and remove all this garbage. This server does not need to track the connections and or log them. There does not seem to be a simple way to disable, just a lot of sysctl options and I'm unclear if these will do it entirely. net.ipv4.conf.all.rp_filter=0 net.ipv4.conf.all.arp_filter=0 net.ipv4.conf.default.rp_filter=0 net.ipv4.conf.default.arp_filter=0 net.ipv4.conf.lo.rp_filter=0 net.ipv4.conf.lo.arp_filter=0 net.ipv4.conf.eth0.rp_filter=0 net.ipv4.conf.eth0.arp_filter=0 net.ipv4.conf.eth1.rp_filter=0 net.ipv4.conf.eth1.arp_filter=0 net.ipv4.conf.br0.rp_filter=0 net.ipv4.conf.br0.arp_filter=0 net.ipv4.conf.br1.rp_filter=0 net.ipv4.conf.br1.arp_filter=0 net.ipv4.conf.vnet0.rp_filter=0 net.ipv4.conf.vnet0.arp_filter=0 net.ipv4.conf.vnet1.rp_filter=0 net.ipv4.conf.vnet1.arp_filter=0 But I'll be quiet here for a few, until I need assistance from the squid community. I'm still seeing the queue congestion, but if it actually doubles the thresholds each time, I may get to a good place, or can be okay to ignore the messages. Obviously the queue congestion was not causing the 500's, the dropping of packets by netfilter was Thanks Tory
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
tor 2010-02-18 klockan 11:49 -0800 skrev Tory M Blue: Okay I've found some issues that I had not seen before, Feb 18 18:37:06 kvm0 kernel: nf_conntrack: table full, dropping packet. And this is exactly what I wanted you to look out for... I would like to kick the netfilter team and fedora team in the shins. The issue was my squid boxes are virtual and the errors were being logged on the domain box (not domain as in MS). So now I'm trying to go through the system and remove all this garbage. This server does not need to track the connections and or log them. There does not seem to be a simple way to disable, just a lot of sysctl options and I'm unclear if these will do it entirely. There is no sysctl to block conntrack. What you need is to either a) Make sure conntrack is not loaded in the kernel. b) If conntrack needs to be loaded then make sure to add suitable NOTRACK rules in iptables to avoid tracking any flows that do not need to be tracked.. Regards Henrik
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
On Tue, Feb 16, 2010 at 7:38 PM, Tory M Blue tmb...@gmail.com wrote: /usr/local/squid/etc/squid/squid.conf ?? So it's really odd. Not getting anything to stdin/stdout But don't want to get too into the config piece when the big deal seems to be the congestion. Why more congestion with faster disks and I'm just thinking if there is actually another config being loaded, any optimizations in the non-loaded one are useless. Amos Ahh Appears squid by design, loads a lot of default params, before it actually reads the config file. So the parse lines are repeated for various items, the default and than your configs from squid.conf. Just needed to look farther down the debug output. So that is a red herring. Need to figure out this queue congestion as SSD's and a sleeping box. I'm missing something Tory
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
On Wed, 17 Feb 2010 10:33:27 -0800, Tory M Blue tmb...@gmail.com wrote: On Tue, Feb 16, 2010 at 7:38 PM, Tory M Blue tmb...@gmail.com wrote: /usr/local/squid/etc/squid/squid.conf ?? So it's really odd. Not getting anything to stdin/stdout But don't want to get too into the config piece when the big deal seems to be the congestion. Why more congestion with faster disks and I'm just thinking if there is actually another config being loaded, any optimizations in the non-loaded one are useless. Amos Ahh Appears squid by design, loads a lot of default params, before it actually reads the config file. So the parse lines are repeated for various items, the default and than your configs from squid.conf. Just needed to look farther down the debug output. So that is a red herring. Phew. Need to figure out this queue congestion as SSD's and a sleeping box. I'm missing something Tory Sleeping? I hope that was just a word you added ... Suddenly pushing N directory and file reads at a disk which is in sleep mode would cause an immediately extreme queue buildup. We have no experience with handling squid loads in such a case and I doubt it would be fully clean. Anyway, the -X trace should have a lot of garbage and some deep details about what exactly is being queued to happen. Henrik seems to have re-appeared and he has more disk IO experience then me so may have an idea whet to look for ... ? Amos
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
tor 2010-02-18 klockan 14:51 +1300 skrev Amos Jeffries: Henrik seems to have re-appeared and he has more disk IO experience then me so may have an idea whet to look for ... ? My first reaction is to run a small benchmark in parallel to squid performing a reasonably slow sequence of random reads over the disks and measuring the response time.. SSD disks in particular may have a habit of randomly block all requests for a while during/after a period of writes.. and OS write buffering and batching may even add to this queue latency problem. But I have not read up on the whole problem description. Also keep in mind that as Amos mentioned earlier these Queue congestion warnings come more frequently after start as there is a filter preventing the logs from getting overwhelmed with this warning. Regards Henrik
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
2010/2/17 Henrik Nordström hen...@henriknordstrom.net: tor 2010-02-18 klockan 14:51 +1300 skrev Amos Jeffries: Henrik seems to have re-appeared and he has more disk IO experience then me so may have an idea whet to look for ... ? My first reaction is to run a small benchmark in parallel to squid performing a reasonably slow sequence of random reads over the disks and measuring the response time.. SSD disks in particular may have a habit of randomly block all requests for a while during/after a period of writes.. and OS write buffering and batching may even add to this queue latency problem. But I have not read up on the whole problem description. Also keep in mind that as Amos mentioned earlier these Queue congestion warnings come more frequently after start as there is a filter preventing the logs from getting overwhelmed with this warning. Thanks Henrik and Amos I'll do whatever testing is needed. As it's really odd.and I can't use these servers/disks until I can get this problem identified. And sorry sleeping was just my way of citing the box shows no load, almost no IO 4-5 when I'm hitting it hard. I do not see this issue with lesser threads, it's only when I turn up the juice. But with turning up the connections per second I would expect to see some type of load and I see none. and the -X -d I dont see anything but that error, is there another log file that needs to be enabled vs cache.log and squid.out? I'll try some various things to see what I can see. I know time dd tests and bonnie++ and some FIO seem to do just fine. It's only squid that seems to be having an issue with me or my setup :) Tory
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
On Tue, 16 Feb 2010 14:30:06 -0800, Tory M Blue tmb...@gmail.com wrote: I'm starting to lose my mind here. New hardware test bed including a striped set of SSD's Same hardware, controller etc as my other squid servers, just added SSD's for testing. I've used default threads and I've built with 24 threads. And what's blowing my mind is I get the error immediately upon startup of my cache server (what?) and when I start banging on it with over 75 connections p/sec.. The issue with the well if you only see a few ignore, is that I actually get 500 errors when this happens. So something is going on and I'm not sure what. No Load No I/O wait. Fedora 12 Squid2.7Stable7 Dual Core 6gigs of ram Striped SSD's And did I mention no wait and zero load when this happens? configure options: '--host=i686-pc-linux-gnu' '--build=i686-pc-linux-gnu' '--target=i386-redhat-linux' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--libexecdir=/usr/lib/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--disable-dependency-tracking' '--enable-arp-acl' '--enable-follow-x-forwarded-for' '--enable-auth=basic,digest,negotiate' '--enable-basic-auth-helpers=NCSA,PAM,getpwnam,SASL' '--enable-digest-auth-helpers=password' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-external-acl-helpers=ip_user,session,unix_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-ident-lookups' '--with-large-files' '--enable-linux-netfilter' '--enable-referer-log' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log' '--enable-wccpv2' '--with-aio' '--with-maxfd=16384' '--with-dl' '--with-openssl' '--with-pthreads' '--with-aufs-threads=24' 'build_alias=i686-pc-linux-gnu' 'host_alias=i686-pc-linux-gnu' 'target_alias=i386-redhat-linux' 'CFLAGS=-fPIE -Os -g -pipe -fsigned-char -O2 -g -march=i386 -mtune=i686' 'LDFLAGS=-pie' 2010/02/16 14:15:49| Starting Squid Cache version 2.7.STABLE7 for i686-pc-linux-gnu... 2010/02/16 14:15:49| Process ID 19222 2010/02/16 14:15:49| With 4096 file descriptors available 2010/02/16 14:15:49| Using epoll for the IO loop 2010/02/16 14:15:49| Performing DNS Tests... 2010/02/16 14:15:49| Successful DNS name lookup tests... 2010/02/16 14:15:49| DNS Socket created at 0.0.0.0, port 52964, FD 6 2010/02/16 14:15:49| User-Agent logging is disabled. 2010/02/16 14:15:49| Referer logging is disabled. 2010/02/16 14:15:49| Unlinkd pipe opened on FD 10 2010/02/16 14:15:49| Swap maxSize 32768000 + 102400 KB, estimated 2528492 objects 2010/02/16 14:15:49| Target number of buckets: 126424 2010/02/16 14:15:49| Using 131072 Store buckets 2010/02/16 14:15:49| Max Mem size: 102400 KB 2010/02/16 14:15:49| Max Swap size: 32768000 KB 2010/02/16 14:15:49| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec 2010/02/16 14:15:49| Store logging disabled 2010/02/16 14:15:49| Rebuilding storage in /cache (CLEAN) 2010/02/16 14:15:49| Using Least Load store dir selection 2010/02/16 14:15:49| Set Current Directory to /var/spool/squid 2010/02/16 14:15:49| Loaded Icons. 2010/02/16 14:15:50| Accepting accelerated HTTP connections at 0.0.0.0, port 80, FD 13. 2010/02/16 14:15:50| Accepting ICP messages at 0.0.0.0, port 3130, FD 14. 2010/02/16 14:15:50| Accepting SNMP messages on port 3401, FD 15. 2010/02/16 14:15:50| WCCP Disabled. 2010/02/16 14:15:50| Ready to serve requests. 2010/02/16 14:15:50| Configuring host,domain.com Parent host.domain.com/80/0 2010/02/16 14:15:50| Store rebuilding is 0.4% complete 2010/02/16 14:16:05| Store rebuilding is 66.1% complete 2010/02/16 14:16:12| Done reading /cache swaplog (948540 entries) 2010/02/16 14:16:12| Finished rebuilding storage from disk. 2010/02/16 14:16:12|948540 Entries scanned 2010/02/16 14:16:12| 0 Invalid entries. 2010/02/16 14:16:12| 0 With invalid flags. 2010/02/16 14:16:12|948540 Objects loaded. 2010/02/16 14:16:12| 0 Objects expired. 2010/02/16 14:16:12| 0 Objects cancelled. 2010/02/16 14:16:12| 0 Duplicate URLs purged. 2010/02/16 14:16:12| 0 Swapfile clashes avoided. 2010/02/16 14:16:12| Took 23.0 seconds (41316.8 objects/sec). Hmm, this may be part of a hint. the other clean loads I've seen posted recently, even on old hardware were in or very close to the millions of objects per second. All its really doing so far is loading a text file into memory from disk... 2010/02/16 14:16:12| Beginning Validation Procedure 2010/02/16 14:16:13|262144 Entries Validated so far.
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
2010/02/16 14:18:15| squidaio_queue_request: WARNING - Queue congestion 2010/02/16 14:18:26| squidaio_queue_request: WARNING - Queue congestion What can I look for, if I don't believe it's IO wait or load (the box is sleeping), what else can it be. I thought creating a new build with 24 threads would help but it has not (I can rebuild with 10 threads vs the default 18 (is that right?) I guess. Each of the warnings doubles the previous queue size, so I think its time we took this to the next level of debug. Please run a startup with the option -X and lets see what squid is really trying to do there. Amos Okay not seeing anything exciting here. Nothing new with -X and/or with both -X and -d 2010/02/16 16:17:51| squidaio_queue_request: WARNING - Queue congestion 2010/02/16 16:17:59| squidaio_queue_request: WARNING - Queue congestion No additional information was provided other than what appears to be something odd between my config and what squid is loading into it's config. for example; conf file :maximum_object_size 1024 KB What it says it's parsing: 2010/02/16 16:12:07| parse_line: maximum_object_size 4096 KB conf file: cache_mem 100 MB What it says it's parsing: 2010/02/16 16:12:07| parse_line: cache_mem 8 MB This may not be the answer, but it's odd for sure ( Nothing more on the queue congestion, no idea why this is happening. 2010/02/16 16:12:07| Memory pools are 'off'; limit: 0.00 MB 2010/02/16 16:12:07| cachemgrRegister: registered mem 2010/02/16 16:12:07| cbdataInit 2010/02/16 16:12:07| cachemgrRegister: registered cbdata 2010/02/16 16:12:07| cachemgrRegister: registered events 2010/02/16 16:12:07| cachemgrRegister: registered squidaio_counts 2010/02/16 16:12:07| cachemgrRegister: registered diskd 2010/02/16 16:12:07| diskd started 2010/02/16 16:12:07| authSchemeAdd: adding basic 2010/02/16 16:12:07| authSchemeAdd: adding digest 2010/02/16 16:12:07| authSchemeAdd: adding negotiate 2010/02/16 16:12:07| parse_line: authenticate_cache_garbage_interval 1 hour 2010/02/16 16:12:07| parse_line: authenticate_ttl 1 hour 2010/02/16 16:12:07| parse_line: authenticate_ip_ttl 0 seconds 2010/02/16 16:12:07| parse_line: authenticate_ip_shortcircuit_ttl 0 seconds 2010/02/16 16:12:07| parse_line: acl_uses_indirect_client on 2010/02/16 16:12:07| parse_line: delay_pool_uses_indirect_client on 2010/02/16 16:12:07| parse_line: log_uses_indirect_client on 2010/02/16 16:12:07| parse_line: ssl_unclean_shutdown off 2010/02/16 16:12:07| parse_line: sslproxy_version 1 2010/02/16 16:12:07| parse_line: zph_mode off 2010/02/16 16:12:07| parse_line: zph_local 0 2010/02/16 16:12:07| parse_line: zph_sibling 0 2010/02/16 16:12:07| parse_line: zph_parent 0 2010/02/16 16:12:07| parse_line: zph_option 136 2010/02/16 16:12:07| parse_line: dead_peer_timeout 10 seconds 2010/02/16 16:12:07| parse_line: cache_mem 8 MB 2010/02/16 16:12:07| parse_line: maximum_object_size_in_memory 8 KB 2010/02/16 16:12:07| parse_line: memory_replacement_policy lru 2010/02/16 16:12:07| parse_line: cache_replacement_policy lru 2010/02/16 16:12:07| parse_line: store_dir_select_algorithm least-load 2010/02/16 16:12:07| parse_line: max_open_disk_fds 0 2010/02/16 16:12:07| parse_line: minimum_object_size 0 KB 2010/02/16 16:12:07| parse_line: maximum_object_size 4096 KB 2010/02/16 16:12:07| parse_line: cache_swap_low 90 2010/02/16 16:12:07| parse_line: cache_swap_high 95 2010/02/16 16:12:07| parse_line: update_headers on 2010/02/16 16:12:07| parse_line: logfile_daemon /usr/lib/squid/logfile-daemon 2010/02/16 16:12:07| parse_line: cache_log /var/logs/cache.log 2010/02/16 16:12:07| parse_line: cache_store_log /var/logs/store.log 2010/02/16 16:12:07| parse_line: logfile_rotate 10 2010/02/16 16:12:07| parse_line: emulate_httpd_log off 2010/02/16 16:12:07| parse_line: log_ip_on_direct on 2010/02/16 16:12:07| parse_line: mime_table /etc/squid/mime.conf 2010/02/16 16:12:07| parse_line: log_mime_hdrs off 2010/02/16 16:12:07| parse_line: pid_filename /var/logs/squid.pid 2010/02/16 16:12:07| parse_line: debug_options ALL,1 2010/02/16 16:12:07| parse_line: log_fqdn off 2010/02/16 16:12:07| parse_line: client_netmask 255.255.255.255 2010/02/16 16:12:07| parse_line: strip_query_terms on 2010/02/16 16:12:07| parse_line: buffered_logs off 2010/02/16 16:12:07| parse_line: netdb_filename /var/logs/netdb.state 2010/02/16 16:12:07| parse_line: ftp_user Squid@ 2010/02/16 16:12:07| parse_line: ftp_list_width 32 2010/02/16 16:12:07| parse_line: ftp_passive on 2010/02/16 16:12:07| parse_line: ftp_sanitycheck on 2010/02/16 16:12:07| parse_line: ftp_telnet_protocol on 2010/02/16 16:12:07| parse_line: diskd_program /usr/lib/squid/diskd-daemon 2010/02/16 16:12:07| parse_line: unlinkd_program /usr/lib/squid/unlinkd 2010/02/16 16:12:07| parse_line: storeurl_rewrite_children 5 2010/02/16 16:12:07| parse_line: storeurl_rewrite_concurrency 0 2010/02/16 16:12:07| parse_line: url_rewrite_children 5 2010/02/16 16:12:07| parse_line: url_rewrite_concurrency 0 2010/02/16
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
On Tue, 16 Feb 2010 16:24:22 -0800, Tory M Blue tmb...@gmail.com wrote: 2010/02/16 14:18:15| squidaio_queue_request: WARNING - Queue congestion 2010/02/16 14:18:26| squidaio_queue_request: WARNING - Queue congestion What can I look for, if I don't believe it's IO wait or load (the box is sleeping), what else can it be. I thought creating a new build with 24 threads would help but it has not (I can rebuild with 10 threads vs the default 18 (is that right?) I guess. Each of the warnings doubles the previous queue size, so I think its time we took this to the next level of debug. Please run a startup with the option -X and lets see what squid is really trying to do there. Amos Okay not seeing anything exciting here. Nothing new with -X and/or with both -X and -d 2010/02/16 16:17:51| squidaio_queue_request: WARNING - Queue congestion 2010/02/16 16:17:59| squidaio_queue_request: WARNING - Queue congestion No additional information was provided other than what appears to be something odd between my config and what squid is loading into it's config. for example; conf file :maximum_object_size 1024 KB What it says it's parsing: 2010/02/16 16:12:07| parse_line: maximum_object_size 4096 KB conf file: cache_mem 100 MB What it says it's parsing: 2010/02/16 16:12:07| parse_line: cache_mem 8 MB This may not be the answer, but it's odd for sure ( Nothing more on the queue congestion, no idea why this is happening. To stdout/stderr or cache.log? I think if thats to stdout/stderr might be the defaults loading. There should be two in that case. The later one correct. Though it may be worth double checking for other locations of squid.conf. Amos
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
On Tue, Feb 16, 2010 at 4:45 PM, Amos Jeffries squ...@treenet.co.nz wrote: On Tue, 16 Feb 2010 16:24:22 -0800, Tory M Blue tmb...@gmail.com wrote: 2010/02/16 14:18:15| squidaio_queue_request: WARNING - Queue congestion 2010/02/16 14:18:26| squidaio_queue_request: WARNING - Queue congestion What can I look for, if I don't believe it's IO wait or load (the box is sleeping), what else can it be. I thought creating a new build with 24 threads would help but it has not (I can rebuild with 10 threads vs the default 18 (is that right?) I guess. Each of the warnings doubles the previous queue size, so I think its time we took this to the next level of debug. Please run a startup with the option -X and lets see what squid is really trying to do there. Amos Okay not seeing anything exciting here. Nothing new with -X and/or with both -X and -d 2010/02/16 16:17:51| squidaio_queue_request: WARNING - Queue congestion 2010/02/16 16:17:59| squidaio_queue_request: WARNING - Queue congestion No additional information was provided other than what appears to be something odd between my config and what squid is loading into it's config. for example; conf file :maximum_object_size 1024 KB What it says it's parsing: 2010/02/16 16:12:07| parse_line: maximum_object_size 4096 KB conf file: cache_mem 100 MB What it says it's parsing: 2010/02/16 16:12:07| parse_line: cache_mem 8 MB This may not be the answer, but it's odd for sure ( Nothing more on the queue congestion, no idea why this is happening. To stdout/stderr or cache.log? I think if thats to stdout/stderr might be the defaults loading. There should be two in that case. The later one correct. Though it may be worth double checking for other locations of squid.conf. Amos That's from cache.log and I only have one squid.conf in /etc/squid and the only other squid.conf is the http configuration for cachemgr in /etc/httpd/conf.d So it's really odd. Not getting anything to stdin/stdout But don't want to get too into the config piece when the big deal seems to be the congestion. Why more congestion with faster disks and almost no load. I'm willing to run tests, tweak, rebuild with various settings whatever, just would like to figure this out Tory
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
On Tue, 16 Feb 2010 17:00:33 -0800, Tory M Blue tmb...@gmail.com wrote: On Tue, Feb 16, 2010 at 4:45 PM, Amos Jeffries squ...@treenet.co.nz wrote: On Tue, 16 Feb 2010 16:24:22 -0800, Tory M Blue tmb...@gmail.com wrote: 2010/02/16 14:18:15| squidaio_queue_request: WARNING - Queue congestion 2010/02/16 14:18:26| squidaio_queue_request: WARNING - Queue congestion What can I look for, if I don't believe it's IO wait or load (the box is sleeping), what else can it be. I thought creating a new build with 24 threads would help but it has not (I can rebuild with 10 threads vs the default 18 (is that right?) I guess. Each of the warnings doubles the previous queue size, so I think its time we took this to the next level of debug. Please run a startup with the option -X and lets see what squid is really trying to do there. Amos Okay not seeing anything exciting here. Nothing new with -X and/or with both -X and -d 2010/02/16 16:17:51| squidaio_queue_request: WARNING - Queue congestion 2010/02/16 16:17:59| squidaio_queue_request: WARNING - Queue congestion No additional information was provided other than what appears to be something odd between my config and what squid is loading into it's config. for example; conf file :maximum_object_size 1024 KB What it says it's parsing: 2010/02/16 16:12:07| parse_line: maximum_object_size 4096 KB conf file: cache_mem 100 MB What it says it's parsing: 2010/02/16 16:12:07| parse_line: cache_mem 8 MB This may not be the answer, but it's odd for sure ( Nothing more on the queue congestion, no idea why this is happening. To stdout/stderr or cache.log? I think if thats to stdout/stderr might be the defaults loading. There should be two in that case. The later one correct. Though it may be worth double checking for other locations of squid.conf. Amos That's from cache.log and I only have one squid.conf in /etc/squid and the only other squid.conf is the http configuration for cachemgr in /etc/httpd/conf.d /usr/local/squid/etc/squid/squid.conf ?? So it's really odd. Not getting anything to stdin/stdout But don't want to get too into the config piece when the big deal seems to be the congestion. Why more congestion with faster disks and I'm just thinking if there is actually another config being loaded, any optimizations in the non-loaded one are useless. Amos
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
/usr/local/squid/etc/squid/squid.conf ?? So it's really odd. Not getting anything to stdin/stdout But don't want to get too into the config piece when the big deal seems to be the congestion. Why more congestion with faster disks and I'm just thinking if there is actually another config being loaded, any optimizations in the non-loaded one are useless. Amos Nope, only /etc/squid/squid.conf /etc/squid/squid.conf.default. I've done a find on my system and no others. I'm going to run debug on my SDA 2.7stable13 boxen to see if I see something similar. But I still don't think the config is going to cause the queue congestion on an idle box. Tory
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
Tory M Blue wrote: Squid 2.7Stable7 F12 AUFS on a ext3 FS 6gigs ram dual proc cache_dir aufs /cache 32000 16 256 FilesystemSize Used Avail Use% Mounted on /dev/vda2 49G 3.8G 42G 9% /cache configure options: '--host=i686-pc-linux-gnu' '--build=i686-pc-linux-gnu' '--target=i386-redhat-linux' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--libexecdir=/usr/lib/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid' '--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' '--enable-arp-acl' '--enable-follow-x-forwarded-for' '--enable-auth=basic,digest,ntlm,negotiate' '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,getpwnam,multi-domain-NTLM,SASL,squid_radius_auth' '--enable-ntlm-auth-helpers=no_check,fakeauth' '--enable-digest-auth-helpers=password,ldap' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--with-large-files' '--enable-linux-netfilter' '--enable-referer-log' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log' '--enable-wccpv2' '--enable-esi' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl' '--with-openssl' '--with-pthreads' 'build_alias=i686-pc-linux-gnu' 'host_alias=i686-pc-linux-gnu' 'target_alias=i386-redhat-linux' 'CFLAGS=-fPIE -Os -g -pipe -fsigned-char -O2 -g -march=i386 -mtune=i686' 'LDFLAGS=-pie' No load to speak of, very little iowait. Threads were configured as the default. This is running on a striped pair of SSD's and is only a test script that (ya it's hitting it a bit hard), but nothing that squid nor my hardware should have an issue with. What exactly is this test script doing then? How many requests, of what type, is it pumping into squid? over what time period? (going by the cachemgr dump below I see your Squid is processing an average 1190 requests per second. but how intensive the peak load is unknown.) If the disks can take it, you could bump the --with-aufs-threads=N up a bit and raise the ceiling. The default is 16 per cache_dir configured. Queue congestion starts appearing if there are 8 operations queued and not yet handled by the IO threads. The limit is then doubled (to 16) before the next warning appears. And so on... Congestion can be hit easily is a sudden peak in load until Squid adjusts to your regular traffic. If the queues fill too much you get a more serious Disk I/O overloading instead. I've searched and really there does not appear to be a solid answer, except running out of cpu or running out of iops, neither appears to be the case here. Figured if it was a thread issue, I would see a bottleneck on my server? (ya?). Also the if it only happens a couple Queue congestion is what I'd call a leaf thread/process bottleneck. of times ignore it. This is just some testing and I believe this congestion is possibly causing the 500 errors I'm seeing while running my script. Which are? Any pointers, where to look etc? (2.7stable6 on fc6/xen kernel) had no such issues (yes, the SSD's are a new variable (but otherwise identical hardware). Thanks Tory avg-cpu: %user %nice %system %iowait %steal %idle 13.130.00 40.664.800.00 41.41 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn vda1539.50 11516.00 6604.00 23032 13208 avg-cpu: %user %nice %system %iowait %steal %idle 13.600.00 38.29 11.080.00 37.03 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn vda1385.07 11080.60 0.00 22272 0 [...@cache01 ~]$ free total used free sharedbuffers cached Mem: 5039312 4217684617544 0 50372 183900 -/+ buffers/cache: 1874964851816 Swap: 7143416 07143416 Totals since cache startup: sample_time = 1266055655.813016 (Sat, 13 Feb 2010 10:07:35 GMT) client_http.requests = 810672 client_http.hits = 683067 client_http.errors = 0 client_http.kbytes_in = 171682 client_http.kbytes_out = 2145472 client_http.hit_kbytes_out = 1809060 server.all.requests = 127606 server.all.errors = 0 server.all.kbytes_in = 321960 server.all.kbytes_out = 38104 server.http.requests = 127606 server.http.errors = 0 server.http.kbytes_in = 321960 server.http.kbytes_out = 38104
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
On Wed, Mar 4, 2009 at 6:59 AM, Shekhar Gupta shekharsaha...@gmail.com wrote: it's showing client_http.requests = 102.190863/sec It's strange. The load doesn't seem to be that high. What's the CPU load for squid, and how is iowait on your system? (both can be found using the 'top' command) -- /kinkie
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
For Last 1 hr . cpu_time = 86.267886 seconds wall_time = 3600.031220 seconds cpu_usage = 2.396309% On Thu, Mar 5, 2009 at 2:07 PM, Kinkie gkin...@gmail.com wrote: On Wed, Mar 4, 2009 at 6:59 AM, Shekhar Gupta shekharsaha...@gmail.com wrote: it's showing client_http.requests = 102.190863/sec It's strange. The load doesn't seem to be that high. What's the CPU load for squid, and how is iowait on your system? (both can be found using the 'top' command) -- /kinkie
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
In cachemgr, you can see the queue in ASYNC IO Functions Counters. What number do you see during peak load, or when the error shows in Cache.Log? Are you using 280GB cache_dir on one HDD? Or multiple HDDs? What's your HDD interface type (IDE, SATA, SCSI, etc.)? Regards HASSAN - Original Message - From: Shekhar Gupta shekharsaha...@gmail.com To: Kinkie gkin...@gmail.com Cc: squid-users@squid-cache.org Sent: Thursday, March 05, 2009 19:18 Subject: Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion For Last 1 hr . cpu_time = 86.267886 seconds wall_time = 3600.031220 seconds cpu_usage = 2.396309% On Thu, Mar 5, 2009 at 2:07 PM, Kinkie gkin...@gmail.com wrote: On Wed, Mar 4, 2009 at 6:59 AM, Shekhar Gupta shekharsaha...@gmail.com wrote: it's showing client_http.requests = 102.190863/sec It's strange. The load doesn't seem to be that high. What's the CPU load for squid, and how is iowait on your system? (both can be found using the 'top' command) -- /kinkie
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
Any suggestion on this ... On Wed, Mar 4, 2009 at 11:29 AM, Shekhar Gupta shekharsaha...@gmail.com wrote: it's showing client_http.requests = 102.190863/sec On Wed, Mar 4, 2009 at 1:53 AM, Chris Robertson crobert...@gci.net wrote: Shekhar Gupta wrote: can you please tell me how to find hit/sec . squidclient mgr:utilization Chris
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
On Tue, Mar 3, 2009 at 11:03 AM, Shekhar Gupta shekharsaha...@gmail.com wrote: I am getting this log in cache.log file . I am running with 280GB cach_dir with aufs ... Can any one suggest what can be done to resolve this . Is your cache overloaded? 280 Gb is a LOT.. how many hits/second are you getting? -- /kinkie
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
can you please tell me how to find hit/sec . On Tue, Mar 3, 2009 at 7:00 PM, Kinkie gkin...@gmail.com wrote: On Tue, Mar 3, 2009 at 11:03 AM, Shekhar Gupta shekharsaha...@gmail.com wrote: I am getting this log in cache.log file . I am running with 280GB cach_dir with aufs ... Can any one suggest what can be done to resolve this . Is your cache overloaded? 280 Gb is a LOT.. how many hits/second are you getting? -- /kinkie
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
Shekhar Gupta wrote: can you please tell me how to find hit/sec . squidclient mgr:utilization Chris
Re: [squid-users] squidaio_queue_request: WARNING - Queue congestion
it's showing client_http.requests = 102.190863/sec On Wed, Mar 4, 2009 at 1:53 AM, Chris Robertson crobert...@gci.net wrote: Shekhar Gupta wrote: can you please tell me how to find hit/sec . squidclient mgr:utilization Chris
