Re: [squid-users] Squid as gateway

[Amos Jeffries]

On 08/07/17 04:14, erdosain9 wrote:

Hi.
It's possible to put the squid server as gateway??? and config to ear in
port 80 instead of 3128? This will work?



Yes. See .

Like that page name most of the documentation can be found by looking 
for the term "reverse proxy" rather than gateway.


Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Squid as gateway

[erdosain9]

Hi.
It's possible to put the squid server as gateway??? and config to ear in
port 80 instead of 3128? This will work?
Thanks to all.



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-as-gateway-tp4683022.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Huge amount of time_wait connections after upgrade from v2 to v3

[Ivan Larionov]

> On Jul 7, 2017, at 07:20, Eliezer Croitoru  wrote:
> 
> Hey Ivan,
> 
> How do you run these tests?
> With what application "ab" ?
> 

Apache Jmeter with test case written by our load test engineer. I'm not at work 
right now so can't say the exact scenario but afaik we were trying to reproduce 
our production load so it should be somehow close to the real life traffic.

> Thanks,
> Eliezer
> 
> 
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: elie...@ngtech.co.il
> 
> 
> 
> -Original Message-
> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On 
> Behalf Of Ivan Larionov
> Sent: Friday, July 7, 2017 17:07
> To: Amos Jeffries 
> Cc: squid-users@lists.squid-cache.org
> Subject: Re: [squid-users] Huge amount of time_wait connections after upgrade 
> from v2 to v3
> 
> Thank you for the fast reply.
> 
>>> On Jul 7, 2017, at 01:10, Amos Jeffries  wrote:
>>> 
>>> On 07/07/17 13:55, Ivan Larionov wrote:
>>> Hi. Sorry that I'm answering to the old thread. I was on vacation and 
>>> didn't have a chance to test the proposed solution.
>>> Dieter, yes, I'm on the old CentOS 6 based OS (Amazon Linux) but with a new 
>>> kernel 4.9.27.
>>> Amos, thank you for the suggestions about configure flags and squid config 
>>> options, I fixed all issues you pointed to.
>>> Unfortunately following workarounds didn't help:
>>> * client_idle_pconn_timeout 30 seconds
>>> * half_closed_clients on
>>> * client_persistent_connections off
>>> * server_persistent_connections off
>> 
>> TIME_WAIT is a sign that Squid is following the normal TCP process for 
>> closing connections, and doing so before the remote endpoint closes.
>> 
>> Disabling persistent connections increases the number of connections going 
>> through that process. So you definitely want those settings ON to reduce the 
>> WAIT states.
>> 
> 
> I understand that. I just wrote that I tried this options and they had no 
> effect. They didn't increase nor decrease number of TIME_WAIT connections. I 
> removed them when I started testing older versions.
> 
>> If the remote end is the one doing the closure, then you will see less 
>> TIME_WAIT, but CLOSE_WAIT will increase instead. The trick is in finding the 
>> right balance of timeouts on both client and server idle pconn to get the 
>> minimum of total WAIT states. That is network dependent.
>> 
>> Generally though forward/explicit and intercept proxies want 
>> client_idle_pconn_timeout to be shorter than server_idle_pconn_timeout. 
>> Reverse proxy want the opposite.
>> 
>> 
>> 
>>> However I assumed that this is a bug and that I can find older version 
>>> which worked fine. I started testing from 3.1.x all the way to 3.5.26 and 
>>> this is what I found:
>>> * All versions until 3.5.21 work fine. There no issues with huge amount of 
>>> TIME_WAIT connections under load.
>>> * 3.5.20 is the latest stable version.
>>> * 3.5.21 is the first broken version.
>>> * 3.5.23, 3.5.25, 3.5.26 are broken as well.
>>> This effectively means that bug is somewhere in between 3.5.20 and 3.5.21.
>>> I hope this helps and I hope you'll be able to find an issue. If you can 
>>> create a bug report based on this information and post it here it would be 
>>> awesome.
>> 
>> The changes in 3.5.21 were fixes to some common crashes and better caching 
>> behaviour. So I expect at least some of the change is due to higher traffic 
>> throughput on proxies previously restricted by those problems.
>> 
> 
> I can't imagine how throughput increase could result in 500 times more 
> TIME_WAIT connections count.
> 
> In our prod environment when we updated from 2.7.x to 3.5.25 we saw increase 
> from 100 to 1. This is 100x.
> 
> When I was load testing different versions yesterday I was always sending the 
> same amount of RPS to them. Update from 3.5.20 to 3.5.21 resulted in jump 
> from 20 to 1 TIME_WAIT count. This is 500x.
> 
> I know that time_wait is fine in general. Until you have too many of them.
> 
>> Amos
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
> 
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Dstdomain "there are more than 100 regular expressions"

[Eliezer Croitoru]

Hey,

The choice to work with regular expressions on domain names is kind of weird to 
me.
I would try to explain why I am thinking this way.
Let say I have a domain such as:
p.ngtech.co.il
and this domain has many sub domains and I want to catch each and every one of 
these domains how would I block them using a regex?
^s1\.p\.ngtech\.co\.il$
Or
^s[0-9]+\.p.ngtech\.co\.il$

What did I earned if the whole ngtech.co.il domain contains content that I 
don’t want?
There are very specific cases which regex are good for domains blacklisting but 
if you need them so much then just split them into two groups of 90 to overcome 
the squid warning.
If you are using a blacklist which offers you a blacklist with regex then I 
think you are using the wrong one or using it the wrong way.

If you want me to help you with this matter we can findout together if there is 
a much smarter and better way to block these sites then first making sure that 
the domain is not a match for more then 100 regex.

All The Bests,
Eliezer


Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il



-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of erdosain9
Sent: Thursday, July 6, 2017 16:23
To: squid-users@lists.squid-cache.org
Subject: [squid-users] Dstdomain "there are more than 100 regular expressions"

Hi. I have this in my cache.log

ad_block, is a list for block publicity. 

there is a best way to do that??

2017/07/06 10:35:49| /etc/squid/squid.conf line 55: acl ads dstdom_regex 
"/etc/squid/listas/ad_block.lst"
2017/07/06 10:35:49| WARNING: there are more than 100 regular expressions.
Consider using less REs or use rules without expressions like 'dstdomain'.


Thanks to all.



--
View this message in context: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Dstdomain-there-are-more-than-100-regular-expressions-tp4682999.html
Sent from the Squid - Users mailing list archive at Nabble.com.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Huge amount of time_wait connections after upgrade from v2 to v3

[Eliezer Croitoru]

Hey Ivan,

How do you run these tests?
With what application "ab" ?

Thanks,
Eliezer


Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il



-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Ivan Larionov
Sent: Friday, July 7, 2017 17:07
To: Amos Jeffries 
Cc: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] Huge amount of time_wait connections after upgrade 
from v2 to v3

Thank you for the fast reply.

> On Jul 7, 2017, at 01:10, Amos Jeffries  wrote:
> 
>> On 07/07/17 13:55, Ivan Larionov wrote:
>> Hi. Sorry that I'm answering to the old thread. I was on vacation and didn't 
>> have a chance to test the proposed solution.
>> Dieter, yes, I'm on the old CentOS 6 based OS (Amazon Linux) but with a new 
>> kernel 4.9.27.
>> Amos, thank you for the suggestions about configure flags and squid config 
>> options, I fixed all issues you pointed to.
>> Unfortunately following workarounds didn't help:
>> * client_idle_pconn_timeout 30 seconds
>> * half_closed_clients on
>> * client_persistent_connections off
>> * server_persistent_connections off
> 
> TIME_WAIT is a sign that Squid is following the normal TCP process for 
> closing connections, and doing so before the remote endpoint closes.
> 
> Disabling persistent connections increases the number of connections going 
> through that process. So you definitely want those settings ON to reduce the 
> WAIT states.
> 

I understand that. I just wrote that I tried this options and they had no 
effect. They didn't increase nor decrease number of TIME_WAIT connections. I 
removed them when I started testing older versions.

> If the remote end is the one doing the closure, then you will see less 
> TIME_WAIT, but CLOSE_WAIT will increase instead. The trick is in finding the 
> right balance of timeouts on both client and server idle pconn to get the 
> minimum of total WAIT states. That is network dependent.
> 
> Generally though forward/explicit and intercept proxies want 
> client_idle_pconn_timeout to be shorter than server_idle_pconn_timeout. 
> Reverse proxy want the opposite.
> 
> 
> 
>> However I assumed that this is a bug and that I can find older version which 
>> worked fine. I started testing from 3.1.x all the way to 3.5.26 and this is 
>> what I found:
>> * All versions until 3.5.21 work fine. There no issues with huge amount of 
>> TIME_WAIT connections under load.
>> * 3.5.20 is the latest stable version.
>> * 3.5.21 is the first broken version.
>> * 3.5.23, 3.5.25, 3.5.26 are broken as well.
>> This effectively means that bug is somewhere in between 3.5.20 and 3.5.21.
>> I hope this helps and I hope you'll be able to find an issue. If you can 
>> create a bug report based on this information and post it here it would be 
>> awesome.
> 
> The changes in 3.5.21 were fixes to some common crashes and better caching 
> behaviour. So I expect at least some of the change is due to higher traffic 
> throughput on proxies previously restricted by those problems.
> 

I can't imagine how throughput increase could result in 500 times more 
TIME_WAIT connections count.

In our prod environment when we updated from 2.7.x to 3.5.25 we saw increase 
from 100 to 1. This is 100x.

When I was load testing different versions yesterday I was always sending the 
same amount of RPS to them. Update from 3.5.20 to 3.5.21 resulted in jump from 
20 to 1 TIME_WAIT count. This is 500x.

I know that time_wait is fine in general. Until you have too many of them.

> Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] youtube videos and squid

[Eliezer Croitoru]

Hey Amos, Sergei,

I managed to write an ICAP service which works with a ruby\golang StorID helper 
and a redis DB that can help with the caching of YouTube videos.
However despite to the fact that I am able to tag urls with StoreID  squid is 
still not responding with a cache HIT but fetching from the original sources.

There are couple obstacles in this field and this specific solution I wrote is 
designed for PC and not IPAD or Andorid Based devices.
There is an issue with Android 6+(7..) and IOS 10 based devices which for some 
reason do not cope with SSL traffic interception.
If you do have a tiny server and you can run a tiny http service with 
nginx\apache I would recommend you to download the videos and serve them 
locally using chrome or another browser which works.
I found it more useful then intercepting and caching also, kids are usually 
enjoying from a more "stable" stash of movies\videos then others.
They can watch the same video over and over and it would be fun for them.
The teenagers are having issue's watching the same video over and over so..
I can recommend on a nice wordpress theme which can fit a "VOD" site and a 
video downloader which you can use to download specific videos or playlists.

This solution would be something like "netflix on a stick" and is very 
effective with a raspberry pi 3 with some external USB HDD. 

I would be happy to get any response to any of the ideas.

Thanks In Advance,
Eliezer

 * If you are willing to devote some time to debug the issue with the current 
helpers let me know and bump me if you think I missed an email and didn't 
responded(it happens when you have kids...)

Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Amos Jeffries
Sent: Friday, July 7, 2017 11:32
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] youtube videos and squid

On 07/07/17 15:40, Sergei G wrote:
> Hi guys,
> 
> I have a very specific scenario in mind to use squid for.
> 
> I have 2 kids (2.5 and 4 years old) that are watching iPads and really 
> using available Comcast bandwidth.  It does not help that they sometimes 
> just leave those iPads running.
> 
> They tend to re-watch youtube videos (click on the same icon that they 
> liked before).  And that makes me think that squid could help me with 
> caching off youtube content.  Am I correct?

Possibly. Google have actively been making it more difficult every year 
for quite a while.

These days it requires intercepting the YouTube HTTPS connections. That 
is only possible if the clients are not using Chrome or other Google 
apps to fetch the videos - otherwise you run up against the cert pinning 
wall.

After that you need some extra helper software to track the YT video 
fetching process and decipher what the actual video URL is from the mess 
of session traffic. That is being kept a bit of a secret these days, 
since every time G find out how it is being done they change the process 
to make it more obtuse and harder to do :-(

Eliezer has been trying to get a helper for that going most recently. 
There are also some other products I forget the name of right now 
(videobooster maybe), but should be easy to find that cache YouTube content.


> 
> If not then I have no reason to bother you anymore :)
> 
> If squid could help me, then could you point me to a an example 
> configuration that would work?
> 
> As far as hardware I have 2 options:
> 
> 1. I can install squid on a Raspberry PI 3, if package is readily 
> available.  that's my preferred solution.
> 2. I have an old server hardware with more power than RPI 3, but I don't 
> like to run it, because it is noisy.  It has FreeBSD 10 installed and I 
> can upgrade it to latest FreeBSD (11?) and isntall squid application 
> that way.
> 
> 
> Does squid run on RPI3?  FreeBSD?

Yes to both, and at the small scale you need the RPi3 should be able to 
cope with it.

Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Huge amount of time_wait connections after upgrade from v2 to v3

[Ivan Larionov]

Thank you for the fast reply.

> On Jul 7, 2017, at 01:10, Amos Jeffries  wrote:
> 
>> On 07/07/17 13:55, Ivan Larionov wrote:
>> Hi. Sorry that I'm answering to the old thread. I was on vacation and didn't 
>> have a chance to test the proposed solution.
>> Dieter, yes, I'm on the old CentOS 6 based OS (Amazon Linux) but with a new 
>> kernel 4.9.27.
>> Amos, thank you for the suggestions about configure flags and squid config 
>> options, I fixed all issues you pointed to.
>> Unfortunately following workarounds didn't help:
>> * client_idle_pconn_timeout 30 seconds
>> * half_closed_clients on
>> * client_persistent_connections off
>> * server_persistent_connections off
> 
> TIME_WAIT is a sign that Squid is following the normal TCP process for 
> closing connections, and doing so before the remote endpoint closes.
> 
> Disabling persistent connections increases the number of connections going 
> through that process. So you definitely want those settings ON to reduce the 
> WAIT states.
> 

I understand that. I just wrote that I tried this options and they had no 
effect. They didn't increase nor decrease number of TIME_WAIT connections. I 
removed them when I started testing older versions.

> If the remote end is the one doing the closure, then you will see less 
> TIME_WAIT, but CLOSE_WAIT will increase instead. The trick is in finding the 
> right balance of timeouts on both client and server idle pconn to get the 
> minimum of total WAIT states. That is network dependent.
> 
> Generally though forward/explicit and intercept proxies want 
> client_idle_pconn_timeout to be shorter than server_idle_pconn_timeout. 
> Reverse proxy want the opposite.
> 
> 
> 
>> However I assumed that this is a bug and that I can find older version which 
>> worked fine. I started testing from 3.1.x all the way to 3.5.26 and this is 
>> what I found:
>> * All versions until 3.5.21 work fine. There no issues with huge amount of 
>> TIME_WAIT connections under load.
>> * 3.5.20 is the latest stable version.
>> * 3.5.21 is the first broken version.
>> * 3.5.23, 3.5.25, 3.5.26 are broken as well.
>> This effectively means that bug is somewhere in between 3.5.20 and 3.5.21.
>> I hope this helps and I hope you'll be able to find an issue. If you can 
>> create a bug report based on this information and post it here it would be 
>> awesome.
> 
> The changes in 3.5.21 were fixes to some common crashes and better caching 
> behaviour. So I expect at least some of the change is due to higher traffic 
> throughput on proxies previously restricted by those problems.
> 

I can't imagine how throughput increase could result in 500 times more 
TIME_WAIT connections count.

In our prod environment when we updated from 2.7.x to 3.5.25 we saw increase 
from 100 to 1. This is 100x.

When I was load testing different versions yesterday I was always sending the 
same amount of RPS to them. Update from 3.5.20 to 3.5.21 resulted in jump from 
20 to 1 TIME_WAIT count. This is 500x.

I know that time_wait is fine in general. Until you have too many of them.

> Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] youtube videos and squid

[Amos Jeffries]

On 07/07/17 15:40, Sergei G wrote:

Hi guys,

I have a very specific scenario in mind to use squid for.

I have 2 kids (2.5 and 4 years old) that are watching iPads and really 
using available Comcast bandwidth.  It does not help that they sometimes 
just leave those iPads running.


They tend to re-watch youtube videos (click on the same icon that they 
liked before).  And that makes me think that squid could help me with 
caching off youtube content.  Am I correct?


Possibly. Google have actively been making it more difficult every year 
for quite a while.


These days it requires intercepting the YouTube HTTPS connections. That 
is only possible if the clients are not using Chrome or other Google 
apps to fetch the videos - otherwise you run up against the cert pinning 
wall.


After that you need some extra helper software to track the YT video 
fetching process and decipher what the actual video URL is from the mess 
of session traffic. That is being kept a bit of a secret these days, 
since every time G find out how it is being done they change the process 
to make it more obtuse and harder to do :-(


Eliezer has been trying to get a helper for that going most recently. 
There are also some other products I forget the name of right now 
(videobooster maybe), but should be easy to find that cache YouTube content.





If not then I have no reason to bother you anymore :)

If squid could help me, then could you point me to a an example 
configuration that would work?


As far as hardware I have 2 options:

1. I can install squid on a Raspberry PI 3, if package is readily 
available.  that's my preferred solution.
2. I have an old server hardware with more power than RPI 3, but I don't 
like to run it, because it is noisy.  It has FreeBSD 10 installed and I 
can upgrade it to latest FreeBSD (11?) and isntall squid application 
that way.



Does squid run on RPI3?  FreeBSD?


Yes to both, and at the small scale you need the RPi3 should be able to 
cope with it.


Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Huge amount of time_wait connections after upgrade from v2 to v3

[Amos Jeffries]

On 07/07/17 13:55, Ivan Larionov wrote:
Hi. Sorry that I'm answering to the old thread. I was on vacation and 
didn't have a chance to test the proposed solution.


Dieter, yes, I'm on the old CentOS 6 based OS (Amazon Linux) but with a 
new kernel 4.9.27.


Amos, thank you for the suggestions about configure flags and squid 
config options, I fixed all issues you pointed to.


Unfortunately following workarounds didn't help:

* client_idle_pconn_timeout 30 seconds
* half_closed_clients on
* client_persistent_connections off
* server_persistent_connections off



TIME_WAIT is a sign that Squid is following the normal TCP process for 
closing connections, and doing so before the remote endpoint closes.


Disabling persistent connections increases the number of connections 
going through that process. So you definitely want those settings ON to 
reduce the WAIT states.


If the remote end is the one doing the closure, then you will see less 
TIME_WAIT, but CLOSE_WAIT will increase instead. The trick is in finding 
the right balance of timeouts on both client and server idle pconn to 
get the minimum of total WAIT states. That is network dependent.


Generally though forward/explicit and intercept proxies want 
client_idle_pconn_timeout to be shorter than server_idle_pconn_timeout. 
Reverse proxy want the opposite.




However I assumed that this is a bug and that I can find older version 
which worked fine. I started testing from 3.1.x all the way to 3.5.26 
and this is what I found:


* All versions until 3.5.21 work fine. There no issues with huge amount 
of TIME_WAIT connections under load.

* 3.5.20 is the latest stable version.
* 3.5.21 is the first broken version.
* 3.5.23, 3.5.25, 3.5.26 are broken as well.

This effectively means that bug is somewhere in between 3.5.20 and 3.5.21.

I hope this helps and I hope you'll be able to find an issue. If you can 
create a bug report based on this information and post it here it would 
be awesome.


The changes in 3.5.21 were fixes to some common crashes and better 
caching behaviour. So I expect at least some of the change is due to 
higher traffic throughput on proxies previously restricted by those 
problems.


Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] /var/run/squid gone on every reboot?!

[Antony Stone]

On Friday 07 July 2017 at 08:22:29, Heiler Bemerguy wrote:

> 3.5. Everytime I must recreate this. What I'm missing?

Hm, dunno.

Give us a clue...

What operating system are you running under, and how did you install squid?

How does squid (try to) start at boot time?



Antony.

-- 
How I want a drink, alcoholic of course, after the heavy chapters involving 
quantum mechanics.

 - mnemonic for 3.14159265358979

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users