Re: [squid-users] Squid is very slow after moving to production environment

[Amos Jeffries]

On 13/04/18 05:55, Roberto Carna wrote:
> People, I can't test de new proxy in the production environment
> because I affect the users. I think is a good idea to add 10/15 users
> to my new proxy, and test it with users from my IT area. Maybe the
> problem is DansguardianI don't know.
> 
> I'm seeing pfSense use Squidguard in place of Dansguardianis this
> a better option to block sites and with better performance???
> 

SquidGuard is deprecated software and no longer maintained. Modern Squid
can do almost everything it provided, and the remaining cases can/should
use ufdbguard instead.

DansGuardian is also in a similar position. I'm not sure if it is being
maintained still or not, there is an e2Guardian fork project that has a
lot more recent updates though.


As for performance it depends on what you have them doing:

* URL-rewrite helpers (eg SquidGuard) work by having Squid generate a
second transaction from the "redirected" URL results. That slows down
and uses more memory than regular request processing in Squid.

* chained proxies (eg DansGuardian) require the traffic to be formatted
as HTTP traffic during each hop delivery and re-parsed re-processed by
every proxy along the way. Which naturally adds a bunch of delay overheads.

As a general rule; the less you have SquidGuard doing the more efficient
it is. The less you have DansGuardian doing the more those re-parse
overheads reduce any performance gains.


Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[MK2018]

Roberto Carna wrote
> Thanks to everybody...
> 
> I've reviewed what you tell me. I've executed "squid -k parse" and
> everything is ok, and I've restarted de Squid entire server.
> 
> When I use the server with IP#1, it works OK, is fastbut when I
> change its IP to IP#2 (the IP from the current Squid that I want to
> replace), the navigation is very very slow, just 20/30 concurrent
> users.
> 
> So I think the Squid configuration parameters are OK, because with
> IP#1 the proxy runs perfectly.
> 
> Why just an IP change affected the performance of web browsing 
> Maybe because of something relative to Dansguardian ???
> 
> Thanks and regards !!!

From your description, this looks like a loadbalancing issue, specifically
if you are using DNS round-robin to loadbalance the 2 servers. In most
cases, users will hit the second (or last IP), because DNS round-robin works
from the bottom up.

To get away from guess work, please examine all your log files (cache.log,
access.log,...etc) they will give you a clear picture of what really
happens.

Another quick guess: a "slow" squid is usually an indication of a
"repeatedly crashing" squid, due to overload or system configuration issues.

Logs are your friend.



--
Sent from: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[Amos Jeffries]

On 11/04/18 07:10, Roberto Carna wrote:
> Thanks to everybody...
> 
> I've reviewed what you tell me. I've executed "squid -k parse" and
> everything is ok, and I've restarted de Squid entire server.
> 
> When I use the server with IP#1, it works OK, is fastbut when I
> change its IP to IP#2 (the IP from the current Squid that I want to
> replace), the navigation is very very slow, just 20/30 concurrent
> users.
> 
> So I think the Squid configuration parameters are OK, because with
> IP#1 the proxy runs perfectly.

Then the issue is probably not with Squid. Something outside Squid is
causing the issue - either the VM itself, or the network setup.

> 
> Why just an IP change affected the performance of web browsing 

We do not know the answer to that. None of the info so far shows any
sign of such a problem. Something you have not thought to provide yet
contains the clues.

Perhapse taking a look through the available logs (both Squid and
others) might find better information and ideas.


> Maybe because of something relative to Dansguardian ???
> 

Maybe yes, maybe no. see above.

Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[Roberto Carna]

Thanks to everybody...

I've reviewed what you tell me. I've executed "squid -k parse" and
everything is ok, and I've restarted de Squid entire server.

When I use the server with IP#1, it works OK, is fastbut when I
change its IP to IP#2 (the IP from the current Squid that I want to
replace), the navigation is very very slow, just 20/30 concurrent
users.

So I think the Squid configuration parameters are OK, because with
IP#1 the proxy runs perfectly.

Why just an IP change affected the performance of web browsing 
Maybe because of something relative to Dansguardian ???

Thanks and regards !!!

2018-04-10 15:32 GMT-03:00 joseph :
> hi also lower maximum_object_size_in_memory 4096 KB  to
> maximum_object_size_in_memory 1 MB  higher not wise
>
>
>
> -
> **
> * Crash to the future  
> **
> --
> Sent from: 
> http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[joseph]

hi also lower maximum_object_size_in_memory 4096 KB  to 
maximum_object_size_in_memory 1 MB  higher not wise 



-
** 
* Crash to the future  
**
--
Sent from: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[Eliezer Croitoru]

Well about Cloned VM's acting slower than the original...
I clearly tested it more then once and it's not true and it's a myth.
The only issue I have seen with such cloned systems(I have a very large cluster 
of cloned squid instances) is when the admin over-commit the physical machine.
There is another thing in the hypervisor's world that some admins just do not 
take into account:
- Squid can heavily load a specific CPU.
- You cannot expect the virtualization platform to "create" cycles that do not 
exist.
- You cannot expect the virtualization platform to.. make the disks or the 
network perform more than they have avaliable.

I have a fleet of more than 10 hypervisors which run's more than 90 VM's and 
from them more then 20 percent have Squid-Cache and other services on them.
The only time I had issues was when one of the VM's that was running a java 
based service took a hit of more then 50k requests per second and took\claimed 
brutally more CPU and RAM to spare the other VM's and... all the other VM's 
just crashed with a kernel panic while this specific VM "controlled" or 
"dominated" the hypervisor resources.

All The Bests,
Eliezer


Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il



-Original Message-
From: squid-users <squid-users-boun...@lists.squid-cache.org> On Behalf Of Amos 
Jeffries
Sent: Tuesday, April 10, 2018 13:09
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Squid is very slow after moving to production 
environment

On 10/04/18 07:58, Roberto Carna wrote:
> Dear Antony, both proxies are virtual machines in the same DMZthey
> use the same DNS, the same firewall, the same Internet link, the same
> IP but different MAC Address.


FYI: there were issues some years back with VMs that were cloned
operating VERY much slower for no apparent reason than the original
image they were cloned from.

If you are making production as a clone of the testing you may want to
try a non-clone to see if the problem disappears.

Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[Matus UHLAR - fantomas]

On 09.04.18 16:53, Roberto Carna wrote:

Dear Periko, so here is what you ask to me:

CPU x 8
RAM x 12 GB
HD x 50 GB

And this is /etc/squid/squid.conf file:



cache_mem 4096 MB


what is squid's real memory usage?
It can be much much more than 4G, 4G is only cache, but squid also uses
buffers and indexes.


memory_replacement_policy lru


I would use heap gdsfhere for betterhit ratio, but this should not be a
problem


cache_dir aufs /var/spool/squid 25000 16 256


What's squid CPU usage?
here can be a problem. aufs cache_dir can be only used by one process.
Maybe you should try rock store for cache_Dir


fqdncache_size 4096


I don't see any reason to specify this. too low fqdn cache can result into
repeated DNS fetches.


acl manager proto cache_object


doesn't squid complain here? the "manager" acl is predefined since 3.4 iirc.
Are you sure squid uses this config file?


auth_param basic program /usr/lib/squid/squid_ldap_auth -b
"dc=company,dc=com,dc=ar" -f "uid=%s" -h ldap.company.com.ar -v 3
auth_param basic children 5


aren't there too few children? it can result into waiting for authentication
result before client is allowed.
what does squid log say?


acl QUERY urlpath_regex cgi-bin \? \.css \.asp \.aspx
cache deny QUERY


this is useless for a long time. urlpath_regex causes squid eat much of CPU.
disable this.


acl gedo dstdomain .gde.gob.ar
always_direct allow gedo


you have no cache peers defined. This is therefore useless.


I've just changed the new proxy to test environment and it works very
well againI get lost.


see the limits above. Some of them may be low for a production system.

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
WinError #9: Out of error messages.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[Matus UHLAR - fantomas]

On 10/04/18 07:58, Roberto Carna wrote:

Dear Antony, both proxies are virtual machines in the same DMZthey
use the same DNS, the same firewall, the same Internet link, the same
IP but different MAC Address.


On 10.04.18 22:09, Amos Jeffries wrote:

FYI: there were issues some years back with VMs that were cloned
operating VERY much slower for no apparent reason than the original
image they were cloned from.

If you are making production as a clone of the testing you may want to
try a non-clone to see if the problem disappears.


maybe using "linked clones" causes the problem.
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
(R)etry, (A)bort, (C)ancer
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[Amos Jeffries]

On 10/04/18 07:58, Roberto Carna wrote:
> Dear Antony, both proxies are virtual machines in the same DMZthey
> use the same DNS, the same firewall, the same Internet link, the same
> IP but different MAC Address.


FYI: there were issues some years back with VMs that were cloned
operating VERY much slower for no apparent reason than the original
image they were cloned from.

If you are making production as a clone of the testing you may want to
try a non-clone to see if the problem disappears.

Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[Rafael Akchurin]

Hello Roberto,

When Squid is "slow" like users complain first thing to check is always the DNS 
settings.
Also sometimes switching to "IPv4 DNS resolve first" helps.

Look for "squidclient mgr:idns" and "dns_v4_first on" on Squid wiki.

Hope others have better answers.

Best regards,
Rafael Akchurin
Diladele B.V.

https://www.diladele.com/


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Roberto Carna
Sent: Monday, April 9, 2018 9:59 PM
To: Antony Stone <antony.st...@squid.open.source.it>
Cc: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Squid is very slow after moving to production 
environment

Dear Antony, both proxies are virtual machines in the same DMZthey use the 
same DNS, the same firewall, the same Internet link, the same IP but different 
MAC Address.

Firewall rules are the same too.

The new proxy is slow because when users try to go to a web page, it is very 
slow in download the content page.about 1 minute to do it.

The Dansguardian configuration is te same too.

I've past my configuration in the previous mail.

Thanks a lot !!!



2018-04-09 16:36 GMT-03:00 Antony Stone <antony.st...@squid.open.source.it>:
> On Monday 09 April 2018 at 21:00:21, Roberto Carna wrote:
>
>> Dear, I have implemented a server with Dansguardian 10.2.1.1 and 
>> Squid 3.5.23-5.
>>
>> I've tested it with 5 users for along 2 months and always it worked OK.
>>
>> But today when a moved it to production environment, it worked but 
>> very very slow.
>
> 1. What is "very very slow"?  What difference are you noticing:
>
>  - limited bandwidth for downloads?
>
>  - high latency for reaching new URLs?
>
>  - reduced ability to handle new requests?
>
> Basically, how are you measuring the difference between test 
> performance and production performance?
>
> 2. Please explain your networking setups for the test and production
> environments:
>
>  - do they share the same Internet connection?
>
>  - do they both go through the same firewall?
>
>  - do they both use the same DNS server, or have their own DNS 
> servers, or what?
>
>  - are the same traffic rules implemented for each procy on the firewall/s?
>
>  - do you use any form of user authentication, and if so, please give 
> details
>
> 3. What volume of requests per hour / minute / day / whatever is 
> convenient did you have in the test environment, and what volume do 
> you have now in the production environment?
>
>> I've just changed hostname and IP, in order to match with the old 
>> proxy server and flush de ARP table of the firewall (because ths 
>> server has the same IP but different MAC Address)and no more. And 
>> let me say that in production environment, there are 30-40 users at 
>> all, it's not a big number of users at all.
>>
>> Where can I start to see in order to analyze the problem? Any idea to 
>> help me?
>
>
> Regards,
>
>
> Antony.
>
> --
> I thought I had type A blood, but it turned out to be a typo.
>
>Please reply to the list;
>  please *don't* CC me.
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[Periko Support]

Try to add this setting:

dns_v4_first on

Latter check settings and see if there is no issue with the setting.

squid -k parse.

Then reload:

squid -k reconfigure

Test.


On Mon, Apr 9, 2018 at 1:05 PM, Antony Stone
 wrote:
> On Monday 09 April 2018 at 21:58:52, Roberto Carna wrote:
>
>> Dear Antony, both proxies are virtual machines in the same DMZthey
>> use the same DNS, the same firewall, the same Internet link, the same
>> IP but different MAC Address.
>
> So, what is different between "test" and "production"?
>
>
> Antony.
>
> --
> "Remember: the S in IoT stands for Security."
>
>  - Jan-Piet Mens
>
>Please reply to the list;
>  please *don't* CC me.
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[Antony Stone]

On Monday 09 April 2018 at 21:53:26, Roberto Carna wrote:

> I've just changed the new proxy to test environment and it works very
> well againI get lost.

What does that change involve?  I'm trying to understand what is different 
between your "test" environment and your "production" environment, especially 
since you say they both have the same IP address.


Antony.

-- 
A user interface is like a joke.
If you have to explain it, it means it doesn't work.

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[Antony Stone]

On Monday 09 April 2018 at 21:58:52, Roberto Carna wrote:

> Dear Antony, both proxies are virtual machines in the same DMZthey
> use the same DNS, the same firewall, the same Internet link, the same
> IP but different MAC Address.

So, what is different between "test" and "production"?


Antony.

-- 
"Remember: the S in IoT stands for Security."

 - Jan-Piet Mens

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[Roberto Carna]

Dear Antony, both proxies are virtual machines in the same DMZthey
use the same DNS, the same firewall, the same Internet link, the same
IP but different MAC Address.

Firewall rules are the same too.

The new proxy is slow because when users try to go to a web page, it
is very slow in download the content page.about 1 minute to do it.

The Dansguardian configuration is te same too.

I've past my configuration in the previous mail.

Thanks a lot !!!



2018-04-09 16:36 GMT-03:00 Antony Stone :
> On Monday 09 April 2018 at 21:00:21, Roberto Carna wrote:
>
>> Dear, I have implemented a server with Dansguardian 10.2.1.1 and Squid
>> 3.5.23-5.
>>
>> I've tested it with 5 users for along 2 months and always it worked OK.
>>
>> But today when a moved it to production environment, it worked but
>> very very slow.
>
> 1. What is "very very slow"?  What difference are you noticing:
>
>  - limited bandwidth for downloads?
>
>  - high latency for reaching new URLs?
>
>  - reduced ability to handle new requests?
>
> Basically, how are you measuring the difference between test performance and
> production performance?
>
> 2. Please explain your networking setups for the test and production
> environments:
>
>  - do they share the same Internet connection?
>
>  - do they both go through the same firewall?
>
>  - do they both use the same DNS server, or have their own DNS servers, or
> what?
>
>  - are the same traffic rules implemented for each procy on the firewall/s?
>
>  - do you use any form of user authentication, and if so, please give details
>
> 3. What volume of requests per hour / minute / day / whatever is convenient
> did you have in the test environment, and what volume do you have now in the
> production environment?
>
>> I've just changed hostname and IP, in order to match with the old proxy
>> server and flush de ARP table of the firewall (because ths server has the 
>> same
>> IP but different MAC Address)and no more. And let me say that in
>> production environment, there are 30-40 users at all, it's not a big number
>> of users at all.
>>
>> Where can I start to see in order to analyze the problem? Any idea to help
>> me?
>
>
> Regards,
>
>
> Antony.
>
> --
> I thought I had type A blood, but it turned out to be a typo.
>
>Please reply to the list;
>  please *don't* CC me.
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid is very slow after moving to production environment

[Antony Stone]

On Monday 09 April 2018 at 21:00:21, Roberto Carna wrote:

> Dear, I have implemented a server with Dansguardian 10.2.1.1 and Squid
> 3.5.23-5.
> 
> I've tested it with 5 users for along 2 months and always it worked OK.
> 
> But today when a moved it to production environment, it worked but
> very very slow.

1. What is "very very slow"?  What difference are you noticing:

 - limited bandwidth for downloads?

 - high latency for reaching new URLs?

 - reduced ability to handle new requests?

Basically, how are you measuring the difference between test performance and 
production performance?

2. Please explain your networking setups for the test and production 
environments:

 - do they share the same Internet connection?

 - do they both go through the same firewall?

 - do they both use the same DNS server, or have their own DNS servers, or 
what?

 - are the same traffic rules implemented for each procy on the firewall/s?

 - do you use any form of user authentication, and if so, please give details

3. What volume of requests per hour / minute / day / whatever is convenient 
did you have in the test environment, and what volume do you have now in the 
production environment?

> I've just changed hostname and IP, in order to match with the old proxy
> server and flush de ARP table of the firewall (because ths server has the same
> IP but different MAC Address)and no more. And let me say that in
> production environment, there are 30-40 users at all, it's not a big number
> of users at all.
> 
> Where can I start to see in order to analyze the problem? Any idea to help
> me?


Regards,


Antony.

-- 
I thought I had type A blood, but it turned out to be a typo.

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Squid is very slow after moving to production environment

[Roberto Carna]

Dear, I have implemented a server with Dansguardian 10.2.1.1 and Squid 3.5.23-5.

I've tested it with 5 users for along 2 months and always it worked OK.

But today when a moved it to production environment, it worked but
very very slow. I've just changed hostname and IP, in order to match
with the old proxy server and flush de ARP table of the firewall
(because ths server has the same IP but different MAC Address)and
no more. And let me say that in production environment, there are
30-40 users at all, it's not a big number of users at all.

Where can I start to see in order to analyze the problem? Any idea to help me?

Thanking in advance, regards !!!

Robert
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users