Re: [squid-users] What is a reasonable size for squid.conf?
On Sun, Jun 29, 2014 at 3:57 AM, Owen Crow owen.c...@gmail.com wrote: Consider this a reply to Kinkie and Eliezer. Yes, I expect my setup is unusual, but that's why I'm trying to get advice from others who might have a similar setup. I run the proxy as the main destination for a wildcard DNS. This is our many tenants use URLs in the wildcard domain (lets call it *.wild.com) and the proxy connects them to the various backend services based on the hostname such as: acme-www.wild.com connects to the WWW server for Acme customer beta-www.wild.com connects to a similar but different WWW server for Beta customer. For each customer there are 5-10 unique hostnames to keep the services separate. We do this as it is much simpler than URL-rewriting (or at least it seemed so to me at the beginning). In addition, our proxy listens on about 8 different ports (80/443/8080, etc) for different services. The different ports require 7 ACLs that excludes the other ports that are not for that one service/port combination. I can get more specific if anyone is interested. I use make+M4 macros to generate the squid.conf file from a source file and then separate all the customers into individual configuration files based on a conf.d directory. Hi, yes, it could be interesting. Not the full configuration, which is most likely confidential. But the template you are using for a single entry may be interesting and maybe give us enough information to understand if there are opportunities for optimization. Kinkie
Re: [squid-users] What is a reasonable size for squid.conf?
On 06/27/2014 10:51 PM, Owen Crow wrote: My squid.conf is currently clocking in 60k lines (not including comments or blank lines). Combined with the conf files in my conf.d directory, I have a total of 89k lines in configuration. Hey, Consider that I am Writing from my own mind, knowledge and experience. This is no way close to the original conf file of squid...(the last time I have seen it) You have a very special setup. There is no documentation about it and I assume that the admin assumes couple levels of sanity of the file structure VS complexity of the software. I have not seen Samba settings file that will consume 30k of settings line just because on many environments it's not really sane to run such a server. I cannot compare squid to other services but it's not the first time that this kind of question is being asked. I am not sure what you are expecting, are you asking for less time or just to know more about the subject? I am also not sure that squid.conf is the place to configure couple special setups. There are cases which squid is not the right tool for the the task. So consider: - You are expecting too much from squid(which is overwhelming, in the good way..) - You are using a special setup which is more complex then the basic. - You are maybe wrongly comparing grep or similar tools which are not parsing a complex conf file. In any of the above cases 20 seconds can be considered a reasonable time to load a complex proxy setup. There are software systems which I have seen that takes more then 20 seconds to just show a tiny sign of life due to their complexity. From my point of view when you go above the basic squid conf file you should ask your self if you expect these static rules to stay for a long time and if not then you should consider using another way to implement the idea. If you can share the reason which the conf file is so big some might give you an idea about it. Eliezer
Re: [squid-users] What is a reasonable size for squid.conf?
On Fri, Jun 27, 2014 at 9:51 PM, Owen Crow owen.c...@gmail.com wrote: I am running a non-caching reverse proxy using version 3.3.10. My squid.conf is currently clocking in 60k lines (not including comments or blank lines). Combined with the conf files in my conf.d directory, I have a total of 89k lines in configuration. Hi Owen, I suspect you have embedded in your squid.conf some very long ACL, haven't you? If so, what type is it, and how many lines? As a general advice, you may want to consider moving these ACLs to external files and reference them from the config-file. I have definitely noticed -k reconfigure calls taking on the order of 20 seconds to run when it used to be less than a couple seconds. (Same results with -k test). 20 seconds is quite a bit. What has changed in the configuration file since then? I've tried searching for anything related to max lines and similar, but it usually talks about squid.conf configuration options and not the file itself. If this is not documented per se, are there any anecdotal examples that have this many lines or more? I only see this growing over time. There is no hard limit to the configuration file that I know of. Are you experiencing any performance issues other than during reconfiguration? -- Kinkie
Re: [squid-users] What is a reasonable size for squid.conf?
Consider this a reply to Kinkie and Eliezer. Yes, I expect my setup is unusual, but that's why I'm trying to get advice from others who might have a similar setup. I run the proxy as the main destination for a wildcard DNS. This is our many tenants use URLs in the wildcard domain (lets call it *.wild.com) and the proxy connects them to the various backend services based on the hostname such as: acme-www.wild.com connects to the WWW server for Acme customer beta-www.wild.com connects to a similar but different WWW server for Beta customer. For each customer there are 5-10 unique hostnames to keep the services separate. We do this as it is much simpler than URL-rewriting (or at least it seemed so to me at the beginning). In addition, our proxy listens on about 8 different ports (80/443/8080, etc) for different services. The different ports require 7 ACLs that excludes the other ports that are not for that one service/port combination. I can get more specific if anyone is interested. I use make+M4 macros to generate the squid.conf file from a source file and then separate all the customers into individual configuration files based on a conf.d directory. Zero caching is happening, it is all just forwarded traffic. When I started with ~50 customers, squid cpu was 5% at all times. Now with closer to 200 customers it sticks around 20%, so I'm just thinking about the future. And if things keep going well, I can only expect the number to rise. Thanks for all the replies! And if anyone is wondering how I counted lines I used something like this to eliminate comments and blank lines: egrep -vc '(^#|^$)' squid.conf Owen On Sat, Jun 28, 2014 at 3:50 PM, Kinkie gkin...@gmail.com wrote: On Fri, Jun 27, 2014 at 9:51 PM, Owen Crow owen.c...@gmail.com wrote: I am running a non-caching reverse proxy using version 3.3.10. My squid.conf is currently clocking in 60k lines (not including comments or blank lines). Combined with the conf files in my conf.d directory, I have a total of 89k lines in configuration. Hi Owen, I suspect you have embedded in your squid.conf some very long ACL, haven't you? If so, what type is it, and how many lines? As a general advice, you may want to consider moving these ACLs to external files and reference them from the config-file. I have definitely noticed -k reconfigure calls taking on the order of 20 seconds to run when it used to be less than a couple seconds. (Same results with -k test). 20 seconds is quite a bit. What has changed in the configuration file since then? I've tried searching for anything related to max lines and similar, but it usually talks about squid.conf configuration options and not the file itself. If this is not documented per se, are there any anecdotal examples that have this many lines or more? I only see this growing over time. There is no hard limit to the configuration file that I know of. Are you experiencing any performance issues other than during reconfiguration? -- Kinkie
[squid-users] What is a reasonable size for squid.conf?
I am running a non-caching reverse proxy using version 3.3.10. My squid.conf is currently clocking in 60k lines (not including comments or blank lines). Combined with the conf files in my conf.d directory, I have a total of 89k lines in configuration. I have definitely noticed -k reconfigure calls taking on the order of 20 seconds to run when it used to be less than a couple seconds. (Same results with -k test). I've tried searching for anything related to max lines and similar, but it usually talks about squid.conf configuration options and not the file itself. If this is not documented per se, are there any anecdotal examples that have this many lines or more? I only see this growing over time. Thanks, Owen
Re: [squid-users] What is a reasonable size for squid.conf?
My squid.conf is 3380 bytes, and 99 total lines, with around 35 lines blank or commented out. If you had been upgrading from any 3.1 or older squid, they had a LOT of unnecessary lines in there for TAG related entries and excess documentation of every little line. Mike On 6/27/2014 2:51 PM, Owen Crow wrote: I am running a non-caching reverse proxy using version 3.3.10. My squid.conf is currently clocking in 60k lines (not including comments or blank lines). Combined with the conf files in my conf.d directory, I have a total of 89k lines in configuration. I have definitely noticed -k reconfigure calls taking on the order of 20 seconds to run when it used to be less than a couple seconds. (Same results with -k test). I've tried searching for anything related to max lines and similar, but it usually talks about squid.conf configuration options and not the file itself. If this is not documented per se, are there any anecdotal examples that have this many lines or more? I only see this growing over time. Thanks, Owen
