Re: [squid-users] Content Adaptation with HTTPs

[Amos Jeffries]

On 20/08/17 16:05, Christopher Ahrens wrote:


The current solution doesn't work for me since it only supports a very 
limited number of clients.  I am working with a charity that provides 
internet services to those with impaired vision, the intention of my 
project was to set up a semi-public proxy for recipient of the charity 
(EG, we would install DD-WRT like routers within their homes that would 
create a tunnel into our network so that they could browse the internet 
using off-the-shelf systems.  We recently received a large number of 
tablets form a corporate donor, the tablets themselves will work for our 
recipients, but unfortunately the internet at large does not.


FYI: If you can get the adaptation part to be small enough a non-caching 
Squid should be able to run on those WRT-like devices with under 32 MB 
of RAM needed. So the tunnel may not be necessary, just a way to update 
the software and its config.




We've looked into commercial systems in the past, but we cannot afford 
the cost of commercial systems, especially since we are unsure about the 
exact licensing that would be needed for our endeavor.  We have also 
been burnt in the past with commercial software where the project either 
goes dead, begins to require insanely expensive appliances, or the 
license price is sent sky-high.


Would it be possible to use a setup of Squid <-> Privoxy <-> Squid to 
execute this?  I figure we'd build an internal instance that will handle 
the client<->proxy part, Privoxy handles the content modification, then 
a second Squid instance to handle the web server<->proxy part.


Squid will only send SSL-Bump'ed HTTPS traffic over encrypted 
connections. So that is only possible if privoxy accepts TLS connections 
from Squid. In which case you probably do not need the second Squid, as 
privoxy would also be doing the HTTPS to-server part easily enough itself.





SO it looks like the solution would be to find a developer to write an 
ECAP to cycle through regexes to replace/remove HTML/CSS content.  So 
time to dig out my old C++ books and get to work...


If the existing ICAP/eCAP options are not suitable, then yes a custom 
one would be needed.


It is not as easy as a few regex replacements though. Adaptors are 
streamed the full on-wire HTTP message format with only minor 
sanitization by Squids parser. To alter the content you will have to 
deal with data encodings, object ranges, partially received objects. And 
it is best to assume everything is of infinite length unless explicitly 
told otherwise - so no buffer-then-adapt code.

 eCAP is simpler than ICAP, but still has to deal with these HTTP features.

Those are a big part of why available software is so sparse. The other 
part being that HTTP traffic payloads are copyright content, so there 
are legal issues with selling software for the purpose of altering 
copyright content sans authors permission.


Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] client-->iptables-->squid-proxy->another-proxy

[Amos Jeffries]

On 20/08/17 14:38, Diogenes S. Jesus wrote:>

* squid.conf:
---
acl localhost src 127.0.0.0/8 
acl localnet src 192.168.100.0/24  
192.168.101.0/24  172.16.0.0/12 


acl SSL_ports port 443
acl Safe_ports port 80# http
acl Safe_ports port 443# https
acl CONNECT method CONNECT

http_access allow  localhost localnet
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny all



Those http_access rules contain an impossible condition.

The src-IP cannot simultaneously be having a value in the 127/8 network 
range *and* in one of the RFC1918 ranges. So there is no way anything is 
ever allowed to use this proxy.


I suspect it was working due to a recently fixed bug where the CONNECT 
message was not consistently passed through http_access controls 
sometimes in the first SSL-Bump step. Do not expect that to work much 
longer.


Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] IPv6 and TPROXY

[Eliezer Croitoru]

I am still waiting for couple answers about the system and the setup.
Also to resolve the issue it will be required to know if the issue is on squid 
side or the kernel side(ipv6 related) or iptables rules.
All of the above will allow us to help Walter make this system work.

And Amos, about the part of avoiding using tproxy for the outgoing traffic and 
only use it to intercept the connections:
For a CentOS 6 system it's the only option to run an INTERCEPT proxy which 
hides the client IPv6 address so I think it's something that need to be 
documented somewhere in the wiki.
I would be happy to write the article if I would have known how to disable 
tproxy for the outgoing traffic.

Eliezer


Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il



-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Amos Jeffries
Sent: Sunday, August 20, 2017 03:45
To: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] IPv6 and TPROXY

On 20/08/17 12:08, Eliezer Croitoru wrote:
> You can use tproxy but you will need to somehow make it so squid will do 
> "NAT" instead of only tproxy or to findout what is causing the issue to 
> happen in the network layer of the connection.
> It can be a simple iptables rule which block traffic or another issue like 
> rp_filter.
> If you are up to it I will be willing to try and setup a more advanced ipv6 
> setup that might help to inspect the issue.
> 
> In the mean while I am missing one piece which maybe Amos can help with:
> Is it possible to use tproxy for interception but force a non tproxy 
> connection on the outgoing traffic?

I'm not sure what problem that would solve. If TPROXY is not working 
fully it wont magically start half-working.

AFAICS, Walters problem with TPROXY is that his firewall rules are setup 
for accepting only traffic with 2001::/16 IP addresses. With TPROXY the 
original 2a02::/16 IP remains present so the rules based on 2001::/16 
wont let the traffic into the proxy.

Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] IPv6 and TPROXY

[Amos Jeffries]

On 20/08/17 23:47, Eliezer Croitoru wrote:

I am still waiting for couple answers about the system and the setup.
Also to resolve the issue it will be required to know if the issue is on squid 
side or the kernel side(ipv6 related) or iptables rules.
All of the above will allow us to help Walter make this system work.

And Amos, about the part of avoiding using tproxy for the outgoing traffic and 
only use it to intercept the connections:
For a CentOS 6 system it's the only option to run an INTERCEPT proxy which 
hides the client IPv6 address so I think it's something that need to be 
documented somewhere in the wiki.


CentOS 6 still supplies kernel 2.6.32 apparently. Issues with those 
kernels are listed in the TPROXY wiki page:

"
TPROXYv4 support reached a usable form in 2.6.28. However several 
Kernels have various known bugs:


 * 2.6.28 to 2.6.32 have different rp_filter configuration. The 
rp_filter settings (0 or 1) for these kernels will silently block TPROXY 
if used on newer kernels.

 * 2.6.28 to 2.6.36 are known to have ICMP and TIME_WAIT issues.
 * 2.6.32 to 2.6.34 have bridging issues on some systems.
"




I would be happy to write the article if I would have known how to disable 
tproxy for the outgoing traffic.


There is nothing to document, it is not configurable.

When one is stuck with an ancient kernel the available modern features 
are naturally rather limited.


Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Content Adaptation with HTTPs

[Christopher Ahrens]

Amos Jeffries wrote:

On 20/08/17 16:05, Christopher Ahrens wrote:


The current solution doesn't work for me since it only supports a very
limited number of clients.  I am working with a charity that provides
internet services to those with impaired vision, the intention of my
project was to set up a semi-public proxy for recipient of the charity
(EG, we would install DD-WRT like routers within their homes that
would create a tunnel into our network so that they could browse the
internet using off-the-shelf systems.  We recently received a large
number of tablets form a corporate donor, the tablets themselves will
work for our recipients, but unfortunately the internet at large does
not.


FYI: If you can get the adaptation part to be small enough a non-caching
Squid should be able to run on those WRT-like devices with under 32 MB
of RAM needed. So the tunnel may not be necessary, just a way to update
the software and its config.


Part of it is to pre-shrink the size of the pages to prevent saturating 
the tunnel.  A lot of our recipients have low-cost internet connections 
(Usually between 1-5 Mbps).  From my personal experiences, the 
transformation are probably cutting about 75%-80% of excess garbage from 
website.


We're also looking at possibly building tiny x86 or ARM-based boxes that 
can be deployed to their homes to do caching to further reduce the load 
on their internet connections.  The biggest complaint we have is why it 
takes so long to load pictures and words especially since a lot of the 
pictures are the same page-to-page (I am having a very hard time arguing 
with them...)


We can get a lot of hardware from local companies, but not so much in 
the way of software or services






We've looked into commercial systems in the past, but we cannot afford
the cost of commercial systems, especially since we are unsure about
the exact licensing that would be needed for our endeavor.  We have
also been burnt in the past with commercial software where the project
either goes dead, begins to require insanely expensive appliances, or
the license price is sent sky-high.

Would it be possible to use a setup of Squid <-> Privoxy <-> Squid to
execute this?  I figure we'd build an internal instance that will
handle the client<->proxy part, Privoxy handles the content
modification, then a second Squid instance to handle the web
server<->proxy part.


Squid will only send SSL-Bump'ed HTTPS traffic over encrypted
connections. So that is only possible if privoxy accepts TLS connections
from Squid. In which case you probably do not need the second Squid, as
privoxy would also be doing the HTTPS to-server part easily enough itself.



Unfortunately Privoxy doesn't do HTTPs.  We looked into using it, but it 
can only do domain blocking for HTTPs, not content manipulation.







SO it looks like the solution would be to find a developer to write an
ECAP to cycle through regexes to replace/remove HTML/CSS content.  So
time to dig out my old C++ books and get to work...


If the existing ICAP/eCAP options are not suitable, then yes a custom
one would be needed.

It is not as easy as a few regex replacements though. Adaptors are
streamed the full on-wire HTTP message format with only minor
sanitization by Squids parser. To alter the content you will have to
deal with data encodings, object ranges, partially received objects. And
it is best to assume everything is of infinite length unless explicitly
told otherwise - so no buffer-then-adapt code.
 eCAP is simpler than ICAP, but still has to deal with these HTTP features.

Those are a big part of why available software is so sparse. The other
part being that HTTP traffic payloads are copyright content, so there
are legal issues with selling software for the purpose of altering
copyright content sans authors permission.



Yeah, I was a bit afraid that would be the case.  I was planning on 
seeing how GreaseMonkey and ABP handle data streams since they seem to 
be able to handle streaming media.  Or dig into Privoxy to see how 
things are done in there. Might find it to be easier to adapt it as an 
ICAP/ECAP by changing its input / output functions to be ICAP/ECAP 
interface rather than TCP.


For now, I'm thinking that I'll just let HTTPS pass through without 
modification and let Privoxy handle http.  Seems to be the easiest way 
to do things.



Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users


___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] IPv6 and TPROXY

[Eliezer Croitoru]

Hey Amos,

Leaving aside with very old kernels, I still don't know if this setup works in 
the routing level not to speak about tproxy interception.

The known issues are not relevant for the case if I will be able to test it and 
make sure the issue doesn’t apply to the latest CentOS 6 kernels.

Also even if CentOS have ancient kernel from the 2.X era it doesn't mean that 
more advanced OS versions are not affected by the same or similar issues.
CentOS 7 now uses 3.10 Linux kernel and it's not an ancient Kernel but also not 
the tip or mainline.

Also from what I have seen in the CentOS 7 and RHEL 7 and Netfilter man pages 
and other documentation it seems that a tproxy socket (IP_TRANSPARENT ie 19) is 
required for both trpoxy and REDIRECT ip6tables targets to work properly.

I have yet to test the REDIRECT with ipv6 on a CentOS 7 and I am not sure how 
it should\would work(even if it compiles..).
With ipv4 you would have used SO_ORIGINAL on the socket to know the original 
remote address but with tproxy and IP_TRANSPARENT based sockets from what I 
remember you had to use another option to know the original destination address.
It should be something like "get local address" of the socket(for tproxy) is 
the equivalent to get_sock_opt(..SO_ORIGINAL..).

Until I will try to test the ipv6 REDIRECT with squid intercept I will not know 
if it works the same as the ipv4 redirect and what the recommendation should be 
for general usage in the socket level and squid level.

And if there is no other option then using a transparent proxy socket for both 
tproxy and REDIRECT targets then the outgoing ip address for traffic usage 
should be configurable using some fast acls(leaving aside this specific thread 
use case).

Eliezer


Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il



-Original Message-
From: Amos Jeffries [mailto:squ...@treenet.co.nz] 
Sent: Sunday, August 20, 2017 18:32
To: Eliezer Croitoru ; squid-users@lists.squid-cache.org
Subject: Re: [squid-users] IPv6 and TPROXY

On 20/08/17 23:47, Eliezer Croitoru wrote:
> I am still waiting for couple answers about the system and the setup.
> Also to resolve the issue it will be required to know if the issue is on 
> squid side or the kernel side(ipv6 related) or iptables rules.
> All of the above will allow us to help Walter make this system work.
> 
> And Amos, about the part of avoiding using tproxy for the outgoing traffic 
> and only use it to intercept the connections:
> For a CentOS 6 system it's the only option to run an INTERCEPT proxy which 
> hides the client IPv6 address so I think it's something that need to be 
> documented somewhere in the wiki.

CentOS 6 still supplies kernel 2.6.32 apparently. Issues with those 
kernels are listed in the TPROXY wiki page:
"
TPROXYv4 support reached a usable form in 2.6.28. However several 
Kernels have various known bugs:

  * 2.6.28 to 2.6.32 have different rp_filter configuration. The 
rp_filter settings (0 or 1) for these kernels will silently block TPROXY 
if used on newer kernels.
  * 2.6.28 to 2.6.36 are known to have ICMP and TIME_WAIT issues.
  * 2.6.32 to 2.6.34 have bridging issues on some systems.
"



> I would be happy to write the article if I would have known how to disable 
> tproxy for the outgoing traffic.

There is nothing to document, it is not configurable.

When one is stuck with an ancient kernel the available modern features 
are naturally rather limited.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] wiki.squid-cache.org SSL configuration problem ...

[Kinkie]

Hi,
  it's been fixed last week. Thanks again for the heads-up!


On Tue, Aug 8, 2017 at 9:00 PM, Francesco Chemolli  wrote:
> On 8 Aug 2017, at 19:06, Walter H.  wrote:
>
> Hello,
>
> the intermediate certificate which is provided doen't go with the end
> entitiy certificate ...
>
> the intermediate that is provided:  Let's Encrypt Authority X1
> the intermediate that should be provided:  Let's Encrypt Authority X3
>
> for more see:
> https://www.ssllabs.com/ssltest/analyze.html?d=wiki.squid-cache.org&s=104.130.201.120
>
>
>
>
> Thanks for letting us know.
> We'll look into it ASAP.
>
> Francesco



-- 
Francesco
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] client-->iptables-->squid-proxy->another-proxy

[Diogenes S. Jesus]

Hi Amos. Thanks for pointing it out - but this has never been an
acl-related issue, more like a https_port / ssl-bump configuration question
when the upstream ssl request was not sending a "CONNECT www.example.org:443"
but a "GET htttps://www.example.org".

For the sake of testing one can simply get rid of the acls and set "allow
all", it wouldn't matter - this line "ssl_bump splice all" is the answer
most people were looking for I supposed.

Best regards.



On Sun, Aug 20, 2017 at 10:31 AM, Amos Jeffries 
wrote:

> On 20/08/17 14:38, Diogenes S. Jesus wrote:>
>
>> * squid.conf:
>> ---
>> acl localhost src 127.0.0.0/8 
>> acl localnet src 192.168.100.0/24 
>> 192.168.101.0/24  172.16.0.0/12 <
>> http://172.16.0.0/12>
>> acl SSL_ports port 443
>> acl Safe_ports port 80# http
>> acl Safe_ports port 443# https
>> acl CONNECT method CONNECT
>>
>> http_access allow  localhost localnet
>> http_access deny !Safe_ports
>> http_access deny CONNECT !SSL_ports
>> http_access deny all
>>
>>
> Those http_access rules contain an impossible condition.
>
> The src-IP cannot simultaneously be having a value in the 127/8 network
> range *and* in one of the RFC1918 ranges. So there is no way anything is
> ever allowed to use this proxy.
>
> I suspect it was working due to a recently fixed bug where the CONNECT
> message was not consistently passed through http_access controls sometimes
> in the first SSL-Bump step. Do not expect that to work much longer.
>
>
> Amos
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>



-- 



Diogenes S. de Jesus
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users