Re: [squid-users] socket failure: (24) Too many open files
Thank you for quickturnover! I bumped FD from 8k to 16k. Thanks, Naresh -Original Message- From: Eliezer Croitoru [mailto:elie...@ngtech.co.il] Sent: Monday, October 8, 2018 3:18 PM To: Cherukuri, Naresh Cc: 'squid-users@lists.squid-cache.org' Subject: Re: [squid-users] socket failure: (24) Too many open files I recommend Squid upgrade if possible due to couple bugs as something.. Try to bump the server to 32k open file descriptors and see what happens. Depends on the load on the server it might need at peek times more then 8k. The cache manage info page should give you couple technical details on the status of the service. It can also give some statistics which might shed some light on the scenario. Others might be able to give you some more detail on the relevant cache manager pages. https://wiki.squid-cache.org/Features/CacheManager Eliezer On 2018-10-05 18:57, Cherukuri, Naresh wrote: > Hello Squid Group, > > I am using squid 3.5.20 as a proxy server. I Increased the memory > from 12 GB to 32 GB and Max file descriptors from "4096" to "8192" and > deployed this server into production on 09/26/2018". > > I don't have any problem from the past 10 days everything working as > expected till today. Now after 10 days for the first time, I got > following errors on cache log today. Can someone advise/suggest any > ideas here? > > Error(s) in /var/log/squid/cache.log: 2018/10/05 11:03:59 kid1| > comm_open: socket failure: (24) Too many open files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many > open files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many > open files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many > open files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many > open files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many > open files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many > open files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many > open files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many > open files > > [n*@squidprod ~]$ free -m > > total used free sharedbuffers > cached > > Mem: 32004 15907 16097138295 > 14132 > > -/+ buffers/cache: 1480 30524 > > Swap:24999 0 24999 > > Thanks, > > Naresh > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users -- Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] socket failure: (24) Too many open files
On 9/10/18 2:14 AM, Cherukuri, Naresh wrote: > Yes, I have also splunk running on this machine. > > >> What do "cat /proc/sys/fs/file-max" and "cat /proc/sys/fs/file-nr" tell you? > > [root@squidprod ~]# cat /proc/sys/fs/file-nr > 47360 3256314 > [root@squidprod ~]# cat /proc/sys/fs/file-max > 3256314 > >> How many users do you have, what sort of number of connections per > second/minute/hour (whatever is convenient for you to express) do you have > going through this machine? > > I am not sure, but I would say more than 3000 connections per minute. > The Squid "info" manager report (squidclient mgr:info) contains the req/min details. Since you are running out of FD the number Squid reports as being used will be a lower bound on how many it could be handling if there were enough FD. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] socket failure: (24) Too many open files
I recommend Squid upgrade if possible due to couple bugs as something.. Try to bump the server to 32k open file descriptors and see what happens. Depends on the load on the server it might need at peek times more then 8k. The cache manage info page should give you couple technical details on the status of the service. It can also give some statistics which might shed some light on the scenario. Others might be able to give you some more detail on the relevant cache manager pages. https://wiki.squid-cache.org/Features/CacheManager Eliezer On 2018-10-05 18:57, Cherukuri, Naresh wrote: Hello Squid Group, I am using squid 3.5.20 as a proxy server. I Increased the memory from 12 GB to 32 GB and Max file descriptors from "4096" to "8192" and deployed this server into production on 09/26/2018". I don't have any problem from the past 10 days everything working as expected till today. Now after 10 days for the first time, I got following errors on cache log today. Can someone advise/suggest any ideas here? Error(s) in /var/log/squid/cache.log: 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open files 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open files 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open files 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open files 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open files 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open files 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open files 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open files 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open files [n*@squidprod ~]$ free -m total used free sharedbuffers cached Mem: 32004 15907 16097138295 14132 -/+ buffers/cache: 1480 30524 Swap:24999 0 24999 Thanks, Naresh ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users -- Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: elie...@ngtech.co.il ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] socket failure: (24) Too many open files
Yes, I have also splunk running on this machine. >What do "cat /proc/sys/fs/file-max" and "cat /proc/sys/fs/file-nr" tell you? [root@squidprod ~]# cat /proc/sys/fs/file-nr 47360 3256314 [root@squidprod ~]# cat /proc/sys/fs/file-max 3256314 >How many users do you have, what sort of number of connections per second/minute/hour (whatever is convenient for you to express) do you have going through this machine? I am not sure, but I would say more than 3000 connections per minute. Thanks, Naresh -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Antony Stone Sent: Friday, October 5, 2018 5:06 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] socket failure: (24) Too many open files On Friday 05 October 2018 at 22:08:16, Cherukuri, Naresh wrote: > For just squid process open files count. > > [root@squidprod ~]# lsof -c squid | wc -l > 4385 Squid is not the only thing running on this machine... > [root@squidprod ~]# lsof | wc -l > 10875 But you seem to have sufficient file descriptors configured *in Squid* (but maybe not in the O/S): > [root@squidprod ~]# cat /etc/squid/squid.conf | grep "max_filedescriptors" > max_filedescriptors 8192 So, Squid can have 8192 FDs. > ulimit -a value > [root@squidprod ~]# ulimit -a > open files (-n) 8192 ...and the system will provide 8192 FDs for every process combined... > [root@squidprod ~]# lsof | wc -l > 10875 I reckon that may well be your problem - you have a system-wide limit of 8192 file descriptors, and yet you are trying to use 10875 open files (this will include local pipes, sockets, etc, so it's understandable that it's higher, but it indicates you're going over the limit). > Thanks, > Naresh And, as I asked previously: What do "cat /proc/sys/fs/file-max" and "cat /proc/sys/fs/file-nr" tell you? How many users do you have, what sort of number of connections per second/minute/hour (whatever is convenient for you to express) do you have going through this machine? Antony. -- A good conversation is like a miniskirt; short enought to retain interest, but long enough to cover the subject. - Celeste Headlee Please reply to the list; please *don't* CC me. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] socket failure: (24) Too many open files
On Friday 05 October 2018 at 22:08:16, Cherukuri, Naresh wrote: > For just squid process open files count. > > [root@squidprod ~]# lsof -c squid | wc -l > 4385 Squid is not the only thing running on this machine... > [root@squidprod ~]# lsof | wc -l > 10875 But you seem to have sufficient file descriptors configured *in Squid* (but maybe not in the O/S): > [root@squidprod ~]# cat /etc/squid/squid.conf | grep "max_filedescriptors" > max_filedescriptors 8192 So, Squid can have 8192 FDs. > ulimit -a value > [root@squidprod ~]# ulimit -a > open files (-n) 8192 ...and the system will provide 8192 FDs for every process combined... > [root@squidprod ~]# lsof | wc -l > 10875 I reckon that may well be your problem - you have a system-wide limit of 8192 file descriptors, and yet you are trying to use 10875 open files (this will include local pipes, sockets, etc, so it's understandable that it's higher, but it indicates you're going over the limit). > Thanks, > Naresh And, as I asked previously: What do "cat /proc/sys/fs/file-max" and "cat /proc/sys/fs/file-nr" tell you? How many users do you have, what sort of number of connections per second/minute/hour (whatever is convenient for you to express) do you have going through this machine? Antony. -- A good conversation is like a miniskirt; short enought to retain interest, but long enough to cover the subject. - Celeste Headlee Please reply to the list; please *don't* CC me. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] socket failure: (24) Too many open files
Antony, For just squid process open files count. [root@squidprod ~]# lsof -c squid | wc -l 4385 Thanks, Naresh -Original Message- From: Cherukuri, Naresh Sent: Friday, October 5, 2018 3:52 PM To: 'Antony Stone'; squid-users@lists.squid-cache.org Subject: RE: [squid-users] socket failure: (24) Too many open files Thanks for quick turnover! Please find following details that you requested. >On what Operating System? Operating system : Red Hat 7.0 > I Increased the memory from 12 GB to 32 GB > You mean you put more memory into the server, or you re-configured something > in software (if so, what)? We put more memory into server. Before the server has 12 GB and we increase to 32GB. > and Max file descriptors from "4096" to "8192" [root@squidprod ~]# cat /etc/squid/squid.conf | grep "max_filedescriptors" max_filedescriptors 8192 >ulimit -a value [root@squidprod ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 255941 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 8192 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 255941 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [root@squidprod ~]# lsof | wc -l 10875 Thanks, Naresh -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Antony Stone Sent: Friday, October 5, 2018 12:35 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] socket failure: (24) Too many open files On Friday 05 October 2018 at 17:57:39, Cherukuri, Naresh wrote: > Hello Squid Group, > > > I am using squid 3.5.20 as a proxy server. On what Operating System? > I Increased the memory from 12 GB to 32 GB You mean you put more memory into the server, or you re-configured something in software (if so, what)? > and Max file descriptors from "4096" to "8192" How did you do that? What does "ulimit -a" tell yoou? > and deployed this server into production on 09/26/2018". > > I don't have any problem from the past 10 days everything working as > expected till today. How many users do you have, what sort of number of connections per second/minute/hour (whatever is convenient for you to express) do you have going through this machine? > Now after 10 days for the first time, I got following errors on cache log > today. Can someone advise/suggest any ideas here? > > > Error(s) in /var/log/squid/cache.log: 2018/10/05 11:03:59 kid1| comm_open: > socket failure: (24) Too many open files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files What do "cat /proc/sys/fs/file-max" and "cat /proc/sys/fs/file-nr" tell you? > [n*@squidprod ~]$ free -m > total used free sharedbuffers cached > Mem: 32004 15907 16097138295 14132 > -/+ buffers/cache: 1480 30524 > Swap:24999 0 24999 However that has nothing to do with files or descriptors. What does domething ike "ulimit -a" or "lsof | wc" tell you? Antony. -- Angela Merkel arrives at Paris airport. "Nationality?" asks the immigration officer. "German," she replies. "Occupation?" "No, just here for a summit conference." Please reply to the list; please *don't* CC me. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] socket failure: (24) Too many open files
Thanks for quick turnover! Please find following details that you requested. >On what Operating System? Operating system : Red Hat 7.0 > I Increased the memory from 12 GB to 32 GB > You mean you put more memory into the server, or you re-configured something > in software (if so, what)? We put more memory into server. Before the server has 12 GB and we increase to 32GB. > and Max file descriptors from "4096" to "8192" [root@squidprod ~]# cat /etc/squid/squid.conf | grep "max_filedescriptors" max_filedescriptors 8192 >ulimit -a value [root@squidprod ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 255941 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 8192 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 255941 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [root@squidprod ~]# lsof | wc -l 10875 Thanks, Naresh -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Antony Stone Sent: Friday, October 5, 2018 12:35 PM To: squid-users@lists.squid-cache.org Subject: Re: [squid-users] socket failure: (24) Too many open files On Friday 05 October 2018 at 17:57:39, Cherukuri, Naresh wrote: > Hello Squid Group, > > > I am using squid 3.5.20 as a proxy server. On what Operating System? > I Increased the memory from 12 GB to 32 GB You mean you put more memory into the server, or you re-configured something in software (if so, what)? > and Max file descriptors from "4096" to "8192" How did you do that? What does "ulimit -a" tell yoou? > and deployed this server into production on 09/26/2018". > > I don't have any problem from the past 10 days everything working as > expected till today. How many users do you have, what sort of number of connections per second/minute/hour (whatever is convenient for you to express) do you have going through this machine? > Now after 10 days for the first time, I got following errors on cache log > today. Can someone advise/suggest any ideas here? > > > Error(s) in /var/log/squid/cache.log: 2018/10/05 11:03:59 kid1| comm_open: > socket failure: (24) Too many open files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files What do "cat /proc/sys/fs/file-max" and "cat /proc/sys/fs/file-nr" tell you? > [n*@squidprod ~]$ free -m > total used free sharedbuffers cached > Mem: 32004 15907 16097138295 14132 > -/+ buffers/cache: 1480 30524 > Swap:24999 0 24999 However that has nothing to do with files or descriptors. What does domething ike "ulimit -a" or "lsof | wc" tell you? Antony. -- Angela Merkel arrives at Paris airport. "Nationality?" asks the immigration officer. "German," she replies. "Occupation?" "No, just here for a summit conference." Please reply to the list; please *don't* CC me. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] socket failure: (24) Too many open files
On Friday 05 October 2018 at 17:57:39, Cherukuri, Naresh wrote: > Hello Squid Group, > > > I am using squid 3.5.20 as a proxy server. On what Operating System? > I Increased the memory from 12 GB to 32 GB You mean you put more memory into the server, or you re-configured something in software (if so, what)? > and Max file descriptors from "4096" to "8192" How did you do that? What does "ulimit -a" tell yoou? > and deployed this server into production on 09/26/2018". > > I don't have any problem from the past 10 days everything working as > expected till today. How many users do you have, what sort of number of connections per second/minute/hour (whatever is convenient for you to express) do you have going through this machine? > Now after 10 days for the first time, I got following errors on cache log > today. Can someone advise/suggest any ideas here? > > > Error(s) in /var/log/squid/cache.log: 2018/10/05 11:03:59 kid1| comm_open: > socket failure: (24) Too many open files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files > > 2018/10/05 11:03:59 kid1| comm_open: socket failure: (24) Too many open > files What do "cat /proc/sys/fs/file-max" and "cat /proc/sys/fs/file-nr" tell you? > [n*@squidprod ~]$ free -m > total used free sharedbuffers cached > Mem: 32004 15907 16097138295 14132 > -/+ buffers/cache: 1480 30524 > Swap:24999 0 24999 However that has nothing to do with files or descriptors. What does domething ike "ulimit -a" or "lsof | wc" tell you? Antony. -- Angela Merkel arrives at Paris airport. "Nationality?" asks the immigration officer. "German," she replies. "Occupation?" "No, just here for a summit conference." Please reply to the list; please *don't* CC me. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
