Re: [squid-users] server persistent connections and cache

[Amos Jeffries]

On 27/07/18 16:18, Alex Rousskov wrote:
> On 07/26/2018 09:15 PM, Amos Jeffries wrote:
>> On 27/07/18 13:31, Alex Rousskov wrote:
>>> On 07/26/2018 05:47 PM, Vishali Somaskanthan wrote:
>>>
 By re-use I meant to say that the server-connection S (TCP + SSL) is
 re-used across 2 client connections (C1 and C2), from the same client
 one after the other is torn down. I, presume that
 “/server_persistent_connection on/” allows for such a use-case.
>>>
>>> server_persistent_connection controls whether a single Squid-to-server
>>> connection can carry more than one request. It does not (or should not)
>>> control re-pinning.
> 
>> Also, pinned connections should never be added to the persistent pool
>> after pinning. Doing so is a bug IMO.
> 
> Agreed.
> 
> 
>> Sending the client handshake to a server makes the TLS end-to-end
>> stateful between the client<->server. So *that* is the first point at
>> which pinning is required by Squid.
> 
> At TLS level, yes. However, one could argue that Squid should honor a
> (higher level) client and server assumption that they are talking to
> each other (at HTTP+ level).

One could, but that is dependent on what the admin policy is.
Configuring persistence on/off is the way to control that, not pinning.

> We will probably break fewer transactions
> that way. With that idea in mind, the "first point" becomes establishing
> a TCP tunnel with the server, even if no client Hello pieces are forwarded.
> 

HTTP(S) being stateless, at that level any endpoint relying on implicit
state is non-compliant and buggy. HTTP(S) agents are explicitly required
to include any necessary state references in each message generated.

It is *nice* not to result in visible errors, but not a requirement we
should stick to at cost of proper behaviour. HTTP and extensions also
specify required error handling in most cases where it matters.


> 
>> Bumping at step1 does not involve any server information. So no pinning
>> required. The client is talking to *Squid* over TLS independent of how
>> the response is fetched.
> 
> True for TLS level. Not necessarily true for higher levels as discussed
> above.
> 

The client accepted proof that Squid *was* that origin (false or not)
order to reach said higher levels. Pinning at a later time due to higher
level stateful situation is not relevant to the bumping code actions and
not something we need to consider at the present C2 use of S after C1
should have pinned it.

The only cross-level requirement is to provide the surety that pinning
is a one-way action. Any level setting it makes the connections fate shared.


> 
 Case 1: We see that no pinning happens i.e. pinConnection() is not
 called at all. C1->S gets established, C1 is closed and then C2 re-uses S
> 
>>> The code just happens to work this way (evidently). It is not something
>>> I would rely on until the matter is discussed and settled.
> 
>> I think the described behaviour of C2 using S after C1 has pinned the
>> connection is a bug.
> 
> Your earlier "Sending the client handshake to a server" and "Bumping at
> step1" statements contradict this statement AFAICT because both C1 and
> C2 were bumped at step1 in "Case 1".
> 

 "after C1 has pinned" - if no pinning happens at all the whole
statement is irrelevant. Sorry for confusion.


> FWIW, I would default to honor the tunnel until somebody presents a
> convincing argument for making the behavior configurable. That is, like
> you said, treat C2 reusing S (or, more precisely, not closing S when C1
> is closed) as a Squid bug.
> 

What I'm most confused about here is why S is closed when C1 dies with
non-pinning.  _unless_ there is pinning between them they should not be
that closely fate sharing.

Is the HTTPS message on both C1 and S saying "Connection: close" perhapse?
 If so the closure relationship is an illusion.
 But then, why would C2 be using a connection currently in-use by C1 in
a way that starts *after* C1 closure? (so not collapsed forwarding AFAIK)

Something smells fishy.


> 
>> * S should not be pooled as persistent until the client which triggered
>> its TCP open has finished with it (eg after the TLS handshake completes).
> 
>> * Once the handshake begins sending client data, whichever client was
>> opening it should pin it.
> 
> I agree that pinned connections should not be pooled. Bugs
> notwithstanding, they are not pooled today IIRC.
> 
> I am not sure you meant that, but I doubt a hypothetical feature of
> pooling before-sending-handshake/before-pinning connections is a good
> idea, but it does not contradict "pinned connections are not pooled"
> principle. In practice, once the TCP connection is established, the
> requester pins it immediately (if needed) so there is no opportunity or
> need for pooling.
> 

Off-topic; TLS has connections that do not use handshakes at all, which
are becoming more common in TLS/1.3. So I believe such a feature may be
coming one day, but irrelevant right now.


> 
>> * 

Re: [squid-users] HSTS and HPKP

[Amos Jeffries]

On 27/07/18 16:10, Gordon Hsiao wrote:
> I'm running squid4.1 interception peek+splice mode.
> 
> Some sites with HSTS(max-age=0) will not work whenever squid is on, HSTS
> max-age=0 is supposed to turn off HSTS, but chrome/firefox will keep
> redirecting https<-->http until it failed(too many redirects). Once
> Squid is removed all is good.
> 
> I also searched various lists and squid's website, it's still unclear to
> me, for intercept proxy, can Squid deal with HSTS reliably these days?
> 

Handle yes. Reliably no.

Squid should be erasing the HSTS header completely whenever it can. The
problem is that HSTS can be delivered in several ways that Squid is not
in control of (spliced' traffic, non-HTTP protocols, and non-proxied
connections). You have to reliably seal off those other protocols and
connection types for the MITM proxy to have even a basic chance at success.

FWIW: any HSTS TTL value that gets through to the server breaks things.
Even though max-age=0 can be used to clear some of those other HSTS
avenues, it still breaks things just by turning on the HSTS handling at
the server.


> A similar questions is HPKP, or the pinning certificate, can Squid 4.1
> handle that?

No.

While HSTS was a train wreck from day-0, HPKP is technically closer to
how TLS was supposed to be used in the first place.

AFAIK, the only thing you can do in the presence of client application
using HPKP is splice. Server using it does not matter if the client is
not checking.

Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] server persistent connections and cache

[Alex Rousskov]

On 07/26/2018 09:15 PM, Amos Jeffries wrote:
> On 27/07/18 13:31, Alex Rousskov wrote:
>> On 07/26/2018 05:47 PM, Vishali Somaskanthan wrote:
>>
>>> By re-use I meant to say that the server-connection S (TCP + SSL) is
>>> re-used across 2 client connections (C1 and C2), from the same client
>>> one after the other is torn down. I, presume that
>>> “/server_persistent_connection on/” allows for such a use-case.
>>
>> server_persistent_connection controls whether a single Squid-to-server
>> connection can carry more than one request. It does not (or should not)
>> control re-pinning.

> Also, pinned connections should never be added to the persistent pool
> after pinning. Doing so is a bug IMO.

Agreed.


> Sending the client handshake to a server makes the TLS end-to-end
> stateful between the client<->server. So *that* is the first point at
> which pinning is required by Squid.

At TLS level, yes. However, one could argue that Squid should honor a
(higher level) client and server assumption that they are talking to
each other (at HTTP+ level). We will probably break fewer transactions
that way. With that idea in mind, the "first point" becomes establishing
a TCP tunnel with the server, even if no client Hello pieces are forwarded.


> Bumping at step1 does not involve any server information. So no pinning
> required. The client is talking to *Squid* over TLS independent of how
> the response is fetched.

True for TLS level. Not necessarily true for higher levels as discussed
above.


>>> Case 1: We see that no pinning happens i.e. pinConnection() is not
>>> called at all. C1->S gets established, C1 is closed and then C2 re-uses S

>> The code just happens to work this way (evidently). It is not something
>> I would rely on until the matter is discussed and settled.

> I think the described behaviour of C2 using S after C1 has pinned the
> connection is a bug.

Your earlier "Sending the client handshake to a server" and "Bumping at
step1" statements contradict this statement AFAICT because both C1 and
C2 were bumped at step1 in "Case 1".

FWIW, I would default to honor the tunnel until somebody presents a
convincing argument for making the behavior configurable. That is, like
you said, treat C2 reusing S (or, more precisely, not closing S when C1
is closed) as a Squid bug.


> * S should not be pooled as persistent until the client which triggered
> its TCP open has finished with it (eg after the TLS handshake completes).

> * Once the handshake begins sending client data, whichever client was
> opening it should pin it.

I agree that pinned connections should not be pooled. Bugs
notwithstanding, they are not pooled today IIRC.

I am not sure you meant that, but I doubt a hypothetical feature of
pooling before-sending-handshake/before-pinning connections is a good
idea, but it does not contradict "pinned connections are not pooled"
principle. In practice, once the TCP connection is established, the
requester pins it immediately (if needed) so there is no opportunity or
need for pooling.


> * The sequence of the above two should result in S being pinned before
> it is ever considered for the persistence pool. Which should cause it to
> be excluded from the servers pool.

Agreed.


> So IMO when these things are working properly, C2 server selection
> should never even see that S exists let alone be able to use it.

The open question is whether S should be pinned in the case where C1 is
bumped at step1 (i.e., "Case 1" in Vishali's email). If S should be
pinned, then I agree with the above statement. If S should not be
pinned, then S is just a regular Squid-to-server connection that can be
pooled.


Cheers,

Alex.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] HSTS and HPKP

[Gordon Hsiao]

I'm running squid4.1 interception peek+splice mode.

Some sites with HSTS(max-age=0) will not work whenever squid is on, HSTS
max-age=0 is supposed to turn off HSTS, but chrome/firefox will keep
redirecting https<-->http until it failed(too many redirects). Once Squid
is removed all is good.

I also searched various lists and squid's website, it's still unclear to
me, for intercept proxy, can Squid deal with HSTS reliably these days?

A similar questions is HPKP, or the pinning certificate, can Squid 4.1
handle that?

When no HSTS/HPKP is involved, it seems all sites work well.

Gordon
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] server persistent connections and cache

[Amos Jeffries]

On 27/07/18 13:31, Alex Rousskov wrote:
> On 07/26/2018 05:47 PM, Vishali Somaskanthan wrote:
> 
>> By re-use I meant to say that the server-connection S (TCP + SSL) is
>> re-used across 2 client connections (C1 and C2), from the same client
>> one after the other is torn down. I, presume that
>> “/server_persistent_connection on/” allows for such a use-case.
> 
> server_persistent_connection controls whether a single Squid-to-server
> connection can carry more than one request. It does not (or should not)
> control re-pinning.
> 

Also, pinned connections should never be added to the persistent pool
after pinning. Doing so is a bug IMO. See the sequence at the end of
this mail.

> 
>> Is my
>> understanding that Pinning means binding C1 to S and then if C1 is
>> closed, we unpin and then later if C2 is created, we can pin it again to S?
> 
> IIRC, a lot of code assumes that pinning ties C1 and S connection
> lifetimes together. I do not know whether all code assumes that. I do
> not know whether there is consensus that "same lifetimes" is the correct
> approach for all pinned connections.
> 

Correct (Alex). Pinning to us/Squid means the two connections have done
something stateful - which means shared fate is required from that point
onwards to avoid nasties outside Squid screwing with that statefulness.

Today I work to:
* If one dies they both do.
* A pinned server connection must not be usable by any other client
*after* being pinned.
* A connection not pinned may only be used by *one* client at a time.

There may be old code which does not check pinned state when it should
or makes wrong assumptions about a pinned connection usability. Please
fix as and when found.


> 
>> There is some confusion in my understanding this statement – “pinning
>> _all_ SslBump connections is easier than pinning some of them”, because
>> I see different behaviors when I bump at Step 1 (case 1) vs bump at Step
>> 2 (case 2).
> 
> Just because something is easier in retrospect, does not mean it was
> easy or even clear to the developers writing bits and pieces of that
> code. There are a lot of inconsistencies (and bugs) that we are slowly
> weeding out.
> 

Sending the client handshake to a server makes the TLS end-to-end
stateful between the client<->server. So *that* is the first point at
which pinning is required by Squid.

Peeking at step1 alone does not add statefulness to the client
connection being peeked.

Peeking at step2 requires client data sent to the server. So pinning is
required.

Bumping at step1 does not involve any server information. So no pinning
required. The client is talking to *Squid* over TLS independent of how
the response is fetched.

Bumping at step2 or step3 the client is talking to the specific server
over TLS.

Splice at any time requires tunneling data both ways. So pinning is
required in all cases.

Stare AFAIK does send *some* client state (filtered) to the server so
pinning is probably always required here - but may not be depending on
what the filtering was.


> 
>> Case 1: We see that no pinning happens i.e. pinConnection() is not
>> called at all. C1->S gets established, C1 is closed and then C2 re-uses S
>>
>> Case 2: We see that pinning happens i.e. httpsPeeked() calls
>> pinConnection(). Here, C1->S gets established. Closing C1 from the
>> client brings down S. Later, opening C2 opens a new server-connection S.
>>
>> Is this the expected behavior?
> 
> The code just happens to work this way (evidently). It is not something
> I would rely on until the matter is discussed and settled.
> 


I think the described behaviour of C2 using S after C1 has pinned the
connection is a bug.

* S should not be pooled as persistent until the client which triggered
its TCP open has finished with it (eg after the TLS handshake completes).

* Once the handshake begins sending client data, whichever client was
opening it should pin it.

* The sequence of the above two should result in S being pinned before
it is ever considered for the persistence pool. Which should cause it to
be excluded from the servers pool.

So IMO when these things are working properly, C2 server selection
should never even see that S exists let alone be able to use it.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Wpad problem (DNS)

[Amos Jeffries]

On 27/07/18 03:06, erdosain9 wrote:
> Hi to all.
> Im trying to put proxy trough DNS. Im working on a Windows Server 2012 r2.
> I follow a lot of tutorial... and cant do it.
> The best i have is this (and is strange).
> When the pc start i see in log of squid the ip of that pc.
> 
>  tail -f /var/log/squid/access.log | grep 192.168.6.22
> 1532616150.629 77 192.168.6.22 TCP_REFRESH_UNMODIFIED/200 316 GET
> http://www.msftncsi.com/ncsi.txt - HIER_DIRECT/200.81.17.41 text/plain
> 
> but, if i go throug a web browser, nothing appears in access.log... is like
> the things that the system search (is a windows 7) goes trough proxy, but
> not the thing that i search in the web browser (it's configured to "detect
> automatic").
> 
> I do this in windows server.
> Create a web with IIS, and put wpad.dat file. (create the mime)
> In the DNS, create a new zone wpad, and put a new record txt with this
> "service: wpad:!http://wpad..xxx:80/wpad.dat";
> and a CNAME in my domain with a A record name wpad, and fqdn: the hostname
> of the server.
> 

Okay, the TXT record is not one I've encountered before but the CNAME/A
records match what I know IE8 supported.

Does the wpad..xxx:80 server provide the wpad.dat file with the
correct mime type "application/x-ns-proxy-autoconfig" in HTTP response
message?

Does the wpad.dat file contain valid PAC syntax pointing receivers to
use the proxy?
  has more info on what the PAC file needs
to contain.

What OS is the client machine using?
 Presence of IE 8 hints that it is an older one (XP?) where there may be
ipconfig related tricks needed.


If you have not already done so that site also has troubleshooting tools
and how-to at .


Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] server persistent connections and cache

[Alex Rousskov]

On 07/26/2018 05:47 PM, Vishali Somaskanthan wrote:

> By re-use I meant to say that the server-connection S (TCP + SSL) is
> re-used across 2 client connections (C1 and C2), from the same client
> one after the other is torn down. I, presume that
> “/server_persistent_connection on/” allows for such a use-case.

server_persistent_connection controls whether a single Squid-to-server
connection can carry more than one request. It does not (or should not)
control re-pinning.


> Is my
> understanding that Pinning means binding C1 to S and then if C1 is
> closed, we unpin and then later if C2 is created, we can pin it again to S?

IIRC, a lot of code assumes that pinning ties C1 and S connection
lifetimes together. I do not know whether all code assumes that. I do
not know whether there is consensus that "same lifetimes" is the correct
approach for all pinned connections.


> There is some confusion in my understanding this statement – “pinning
> _all_ SslBump connections is easier than pinning some of them”, because
> I see different behaviors when I bump at Step 1 (case 1) vs bump at Step
> 2 (case 2).

Just because something is easier in retrospect, does not mean it was
easy or even clear to the developers writing bits and pieces of that
code. There are a lot of inconsistencies (and bugs) that we are slowly
weeding out.


> Case 1: We see that no pinning happens i.e. pinConnection() is not
> called at all. C1->S gets established, C1 is closed and then C2 re-uses S
> 
> Case 2: We see that pinning happens i.e. httpsPeeked() calls
> pinConnection(). Here, C1->S gets established. Closing C1 from the
> client brings down S. Later, opening C2 opens a new server-connection S.
> 
> Is this the expected behavior?

The code just happens to work this way (evidently). It is not something
I would rely on until the matter is discussed and settled.


HTH,

Alex.


> On Thu, Jul 26, 2018 at 2:13 PM, Alex Rousskov wrote:
> 
> On 07/26/2018 02:49 PM, Vishali Somaskanthan wrote:
> 
> > 1. Are there any security reasons behind /pinning the connection/
> when a
> > peek is done at Step1
> 
> I doubt there is some fundamental _security_ reason to pin if you bump
> without forwarding the TLS client information to the server. The reasons
> to pin in that case include:
> 
> * honoring client and server blind tunneling expectations
>   (which may result in better compatibility or even security);
> 
> * development convenience (pinning _all_ SslBump connections is easier
>   than pinning some of them).
> 
> 
> > Why is that done?
> 
> I speculate it was done for the reasons stated in the above bullets.
> 
> 
> > 2.a)  what is the scenario where a pinned connection can be reused? 
> 
> When a previously established Squid-to-server connection is used for the
> next request on the corresponding client-to-Squid connection.
> 
> 
> >    b) Which configure option is used to enable /#if
> STRICT_ORIGINAL_DST ?/
> 
> There is no user-friendly way to control that macro AFAICT. You may
> define it when building Squid from sources (e.g., by passing
> -DSTRICT_ORIGINAL_DST=1 to the compiler via CPPFLAGS). Please do not
> misinterpret my response as an indication that this is an officially
> supported feature.
> 
> 
> HTH,
> 
> Alex.
> 
> 
> 
> > On Wed, Jul 18, 2018 at 5:00 PM, Alex Rousskov wrote:
> >
> >     On 07/18/2018 03:03 PM, Vishali Somaskanthan wrote:
> >
> >     > I had a problem after sending too many requests to the same
> server where
> >     > my persistence stopped working suddenly.
> >
> >     Please note that there are many reasons why a proxy may close a
> >     connection. For pinned to-server connections (like those
> created by
> >     SslBump), it may not be possible to open a new to-server
> connection so
> >     Squid should close both from-client and to-server connections.
> >
> >     In general, a client should not rely on a connections staying
> persistent
> >     except in some very unusual/special circumstances.
> >
> >
> >     > 1. What is the relationship between the caching and the
> persistence
> >     > connection established?
> >
> >     Virtually none. Caching decisions are done primarily based on
> request
> >     and response headers.
> >
> >
> >     > 2. When will squid use cached results and when will it not
> if the cache
> >     > deny all directive weren't specified.
> >
> >     Squid will deliver a response from the cache when HTTP rules
> and Squid
> >     configuration allow a hit. The details are too complex to
> document here.
> >
> >
> >     > 3. However, this didn't work fully. Even with /cache deny all/
> >     > directive, persistence wasn't working when I peeked at the
> sslBump step
> >     > 1 and then Bumped. 
> >   

Re: [squid-users] server persistent connections and cache

[Vishali Somaskanthan]

Hi,

FYI, in all my examples below, one have the same client and same server



By re-use I meant to say that the server-connection S (TCP + SSL) is
re-used across 2 client connections (C1 and C2), from the same client one
after the other is torn down. I, presume that “*server_persistent_connection
on*” allows for such a use-case. Is my understanding that Pinning means
binding C1 to S and then if C1 is closed, we unpin and then later if C2 is
created, we can pin it again to S?



There is some confusion in my understanding this statement – “pinning _all_
SslBump connections is easier than pinning some of them”, because I see
different behaviors when I bump at Step 1 (case 1) vs bump at Step 2 (case
2).

Case 1: We see that no pinning happens i.e. pinConnection() is not called
at all. C1->S gets established, C1 is closed and then C2 re-uses S

Case 2: We see that pinning happens i.e. httpsPeeked() calls
pinConnection(). Here, C1->S gets established. Closing C1 from the client
brings down S. Later, opening C2 opens a new server-connection S.



Is this the expected behavior? Please explain.


Thank you.

On Thu, Jul 26, 2018 at 2:13 PM, Alex Rousskov <
rouss...@measurement-factory.com> wrote:

> On 07/26/2018 02:49 PM, Vishali Somaskanthan wrote:
>
> > 1. Are there any security reasons behind /pinning the connection/ when a
> > peek is done at Step1
>
> I doubt there is some fundamental _security_ reason to pin if you bump
> without forwarding the TLS client information to the server. The reasons
> to pin in that case include:
>
> * honoring client and server blind tunneling expectations
>   (which may result in better compatibility or even security);
>
> * development convenience (pinning _all_ SslBump connections is easier
>   than pinning some of them).
>
>
> > Why is that done?
>
> I speculate it was done for the reasons stated in the above bullets.
>
>
> > 2.a)  what is the scenario where a pinned connection can be reused?
>
> When a previously established Squid-to-server connection is used for the
> next request on the corresponding client-to-Squid connection.
>
>
> >b) Which configure option is used to enable /#if STRICT_ORIGINAL_DST
> ?/
>
> There is no user-friendly way to control that macro AFAICT. You may
> define it when building Squid from sources (e.g., by passing
> -DSTRICT_ORIGINAL_DST=1 to the compiler via CPPFLAGS). Please do not
> misinterpret my response as an indication that this is an officially
> supported feature.
>
>
> HTH,
>
> Alex.
>
>
>
> > On Wed, Jul 18, 2018 at 5:00 PM, Alex Rousskov wrote:
> >
> > On 07/18/2018 03:03 PM, Vishali Somaskanthan wrote:
> >
> > > I had a problem after sending too many requests to the same server
> where
> > > my persistence stopped working suddenly.
> >
> > Please note that there are many reasons why a proxy may close a
> > connection. For pinned to-server connections (like those created by
> > SslBump), it may not be possible to open a new to-server connection
> so
> > Squid should close both from-client and to-server connections.
> >
> > In general, a client should not rely on a connections staying
> persistent
> > except in some very unusual/special circumstances.
> >
> >
> > > 1. What is the relationship between the caching and the persistence
> > > connection established?
> >
> > Virtually none. Caching decisions are done primarily based on request
> > and response headers.
> >
> >
> > > 2. When will squid use cached results and when will it not if the
> cache
> > > deny all directive weren't specified.
> >
> > Squid will deliver a response from the cache when HTTP rules and
> Squid
> > configuration allow a hit. The details are too complex to document
> here.
> >
> >
> > > 3. However, this didn't work fully. Even with /cache deny all/
> > > directive, persistence wasn't working when I peeked at the sslBump
> step
> > > 1 and then Bumped.
> > > Persistence worked only when I directly did sslbump allow all
> (without
> > > peeking at first step).
> >
> > Bumping step should not affect connection persistence AFAICT. I do
> not
> > know why it does in your case. This could be a Squid bug or a
> > misunderstanding. If you can reproduce, Squid logs should contain
> > reasons for each connection closure (but it may be difficult to
> find).
> >
> >
> > HTH,
> >
> > Alex.
> >
> >
> >
> >
> > --
> > Regards,
> > Vishali Somaskanthan
> >
> >
> > ___
> > squid-users mailing list
> > squid-users@lists.squid-cache.org
> > http://lists.squid-cache.org/listinfo/squid-users
> >
>
>


-- 
Regards,
Vishali Somaskanthan
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] server persistent connections and cache

[Alex Rousskov]

On 07/26/2018 02:49 PM, Vishali Somaskanthan wrote:

> 1. Are there any security reasons behind /pinning the connection/ when a
> peek is done at Step1

I doubt there is some fundamental _security_ reason to pin if you bump
without forwarding the TLS client information to the server. The reasons
to pin in that case include:

* honoring client and server blind tunneling expectations
  (which may result in better compatibility or even security);

* development convenience (pinning _all_ SslBump connections is easier
  than pinning some of them).


> Why is that done?

I speculate it was done for the reasons stated in the above bullets.


> 2.a)  what is the scenario where a pinned connection can be reused? 

When a previously established Squid-to-server connection is used for the
next request on the corresponding client-to-Squid connection.


>    b) Which configure option is used to enable /#if STRICT_ORIGINAL_DST ?/

There is no user-friendly way to control that macro AFAICT. You may
define it when building Squid from sources (e.g., by passing
-DSTRICT_ORIGINAL_DST=1 to the compiler via CPPFLAGS). Please do not
misinterpret my response as an indication that this is an officially
supported feature.


HTH,

Alex.



> On Wed, Jul 18, 2018 at 5:00 PM, Alex Rousskov wrote:
> 
> On 07/18/2018 03:03 PM, Vishali Somaskanthan wrote:
> 
> > I had a problem after sending too many requests to the same server where
> > my persistence stopped working suddenly.
> 
> Please note that there are many reasons why a proxy may close a
> connection. For pinned to-server connections (like those created by
> SslBump), it may not be possible to open a new to-server connection so
> Squid should close both from-client and to-server connections.
> 
> In general, a client should not rely on a connections staying persistent
> except in some very unusual/special circumstances.
> 
> 
> > 1. What is the relationship between the caching and the persistence
> > connection established?
> 
> Virtually none. Caching decisions are done primarily based on request
> and response headers.
> 
> 
> > 2. When will squid use cached results and when will it not if the cache
> > deny all directive weren't specified. 
> 
> Squid will deliver a response from the cache when HTTP rules and Squid
> configuration allow a hit. The details are too complex to document here.
> 
> 
> > 3. However, this didn't work fully. Even with /cache deny all/
> > directive, persistence wasn't working when I peeked at the sslBump step
> > 1 and then Bumped. 
> > Persistence worked only when I directly did sslbump allow all (without
> > peeking at first step).
> 
> Bumping step should not affect connection persistence AFAICT. I do not
> know why it does in your case. This could be a Squid bug or a
> misunderstanding. If you can reproduce, Squid logs should contain
> reasons for each connection closure (but it may be difficult to find).
> 
> 
> HTH,
> 
> Alex.
> 
> 
> 
> 
> -- 
> Regards,
> Vishali Somaskanthan
> 
> 
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
> 

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] server persistent connections and cache

[Vishali Somaskanthan]

Hi,

Resuming the above conversation; When looking at the cache log and the
code, I find that when peek is done at step 1 and then bumped, the
connection gets pinned after *httpsPeeked() *is called.

Log:

*2018/07/23 11:40:29.572 kid1| 17,4| AsyncCallQueue.cc(55) fireNext:
entering ConnStateData::ConnStateData::httpsPeeked(local=4.16.205.42:33596
 remote=96.43.144.26:443
FD 15 flags=1, request=0x559f3b1a6ed0*5)*
* 40056 2018/07/23 11:40:29.572 kid1| 17,4| AsyncCall.cc(38) make: make
call ConnStateData::ConnStateData::httpsPeeked [call261]*
* 40057 2018/07/23 11:40:29.572 kid1| 45,9| cbdata.cc(419)
cbdataReferenceValid: 0x559f3b64b4a8*
* 40058 2018/07/23 11:40:29.572 kid1| 45,9| cbdata.cc(419)
cbdataReferenceValid: 0x559f3b64b4a8*
* 40059 2018/07/23 11:40:29.572 kid1| 45,9| cbdata.cc(419)
cbdataReferenceValid: 0x559f3b64b4a8*
* 40060 2018/07/23 11:40:29.572 kid1| 45,9| cbdata.cc(419)
cbdataReferenceValid: 0x559f3b64b4a8*
* 40061 2018/07/23 11:40:29.572 kid1| 17,4| AsyncJob.cc(123) callStart:
Http1::Server status in: [ job10]*
* 40062 2018/07/23 11:40:29.572 kid1| 45,9| cbdata.cc(419)
cbdataReferenceValid: 0x559f3b64b4a8*
* 40063 2018/07/23 11:40:29.572 kid1| 33,3| Pipeline.cc(35) front: Pipeline
0x559f3b64b4f0 front 0x559f3b1a9190*2*
* 40064 2018/07/23 11:40:29.572 kid1| 33,3| Pipeline.cc(35) front: Pipeline
0x559f3b64b4f0 front 0x559f3b1a9190*3*
* 40065 2018/07/23 11:40:29.572 kid1| 33,3| client_side.cc(4089)
unpinConnection:*
* 40066 2018/07/23 11:40:29.572 kid1| 33,3| client_side.cc(3927)
pinConnection: local=4.16.205.42:33596 
remote=96.43.144.26:443  FD 15 flags=1*

I assume that a pinned connection means a server connection which MUST have
a 1:1 relationship with some client connection. Accordingly, if the client
terminates, then the server connection should also be closing. From the
post:
http://lists.squid-cache.org/pipermail/squid-users/2015-June/004298.html

1. Are there any security reasons behind *pinning the connection* when a
peek is done at Step1 such that server connection get closed if
corresponding client closes. Why is that done?

2.a)  I see a stmt "reusing a pinned conn" in the function
selectPeerForIntercepted().
>From the logs, I dont find this function getting called. Under what
scenario will this get called? In other words, what is the scenario where a
pinned connection can be reused?

   b) Which configure option is used to enable *#if STRICT_ORIGINAL_DST ?*


*#if STRICT_ORIGINAL_DST*
*if (isIntercepted && useOriginalDst) {*
*selectPeerForIntercepted();*
*// 3.2 does not suppro re-wrapping inside CONNECT.*
*// our only alternative is to fake destination "found" and
continue with the forwarding.*
*startConnectionOrFail();*
*return;*
*}*
*#endif*


On Wed, Jul 18, 2018 at 5:00 PM, Alex Rousskov <
rouss...@measurement-factory.com> wrote:

> On 07/18/2018 03:03 PM, Vishali Somaskanthan wrote:
>
> > I had a problem after sending too many requests to the same server where
> > my persistence stopped working suddenly.
>
> Please note that there are many reasons why a proxy may close a
> connection. For pinned to-server connections (like those created by
> SslBump), it may not be possible to open a new to-server connection so
> Squid should close both from-client and to-server connections.
>
> In general, a client should not rely on a connections staying persistent
> except in some very unusual/special circumstances.
>
>
> > 1. What is the relationship between the caching and the persistence
> > connection established?
>
> Virtually none. Caching decisions are done primarily based on request
> and response headers.
>
>
> > 2. When will squid use cached results and when will it not if the cache
> > deny all directive weren't specified.
>
> Squid will deliver a response from the cache when HTTP rules and Squid
> configuration allow a hit. The details are too complex to document here.
>
>
> > 3. However, this didn't work fully. Even with /cache deny all/
> > directive, persistence wasn't working when I peeked at the sslBump step
> > 1 and then Bumped.
> > Persistence worked only when I directly did sslbump allow all (without
> > peeking at first step).
>
> Bumping step should not affect connection persistence AFAICT. I do not
> know why it does in your case. This could be a Squid bug or a
> misunderstanding. If you can reproduce, Squid logs should contain
> reasons for each connection closure (but it may be difficult to find).
>
>
> HTH,
>
> Alex.
>



-- 
Regards,
Vishali Somaskanthan
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Wpad problem (DNS)

[Walter H.]

On 26.07.2018 17:32, erdosain9 wrote:

Hi, thanks
I try Explorer 8.0 and Chrome 68.0...

this can be deactivated on browser side; then wpad is for the cats ...

Walter



smime.p7s
Description: S/MIME Cryptographic Signature
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Wpad problem (DNS)

[erdosain9]

Hi, thanks
I try Explorer 8.0 and Chrome 68.0...




--
Sent from: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Wpad problem (DNS)

[Antony Stone]

On Thursday 26 July 2018 at 17:06:20, erdosain9 wrote:

> Hi to all.
> Im trying to put proxy trough DNS. Im working on a Windows Server 2012 r2.
> I follow a lot of tutorial... and cant do it.
> The best i have is this (and is strange).
> When the pc start i see in log of squid the ip of that pc.
> 
>  tail -f /var/log/squid/access.log | grep 192.168.6.22
> 1532616150.629 77 192.168.6.22 TCP_REFRESH_UNMODIFIED/200 316 GET
> http://www.msftncsi.com/ncsi.txt - HIER_DIRECT/200.81.17.41 text/plain
> 
> but, if i go throug a web browser,

Which browser?

Have you tried others?

> nothing appears in access.log... is like the things that the system search
> (is a windows 7) goes trough proxy, but not the thing that i search in the
> web browser (it's configured to "detect automatic").
> 
> I do this in windows server.
> Create a web with IIS, and put wpad.dat file. (create the mime)
> In the DNS, create a new zone wpad, and put a new record txt with this
> "service: wpad:!http://wpad..xxx:80/wpad.dat";
> and a CNAME in my domain with a A record name wpad, and fqdn: the hostname
> of the server.
> 
> i unblock the wpad in the dns also.
> 
> And as i say, the system of the machine use the proxy, but not the web
> browser... so... some help???
> 
> Thanks to all!

Antony.

-- 
"I estimate there's a world market for about five computers."

 - Thomas J Watson, Chairman of IBM

   Please reply to the list;
 please *don't* CC me.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Wpad problem (DNS)

[erdosain9]

Hi to all.
Im trying to put proxy trough DNS. Im working on a Windows Server 2012 r2.
I follow a lot of tutorial... and cant do it.
The best i have is this (and is strange).
When the pc start i see in log of squid the ip of that pc.

 tail -f /var/log/squid/access.log | grep 192.168.6.22
1532616150.629 77 192.168.6.22 TCP_REFRESH_UNMODIFIED/200 316 GET
http://www.msftncsi.com/ncsi.txt - HIER_DIRECT/200.81.17.41 text/plain

but, if i go throug a web browser, nothing appears in access.log... is like
the things that the system search (is a windows 7) goes trough proxy, but
not the thing that i search in the web browser (it's configured to "detect
automatic").

I do this in windows server.
Create a web with IIS, and put wpad.dat file. (create the mime)
In the DNS, create a new zone wpad, and put a new record txt with this
"service: wpad:!http://wpad..xxx:80/wpad.dat";
and a CNAME in my domain with a A record name wpad, and fqdn: the hostname
of the server.

i unblock the wpad in the dns also.

And as i say, the system of the machine use the proxy, but not the web
browser... so... some help???

Thanks to all!



--
Sent from: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] A logging only ACL?

[Alex Rousskov]

On 07/26/2018 07:15 AM, Ralf Hildebrandt wrote:
> Before destroying our Squid proxy with an ACL, I'd like to LOG ACL hits
> instead of using "http_access deny" straight away:
> 
> --- nsip ---
> acl markURLhaus annotate_transaction accessRule=URLhaus
> acl URLhaus url_regex "/etc/squid5/generated-urlhaus.acl"
> http_access deny URLhaus markURLhaus
> --- nsip ---
> 
> How?

In two steps:

1. Make sure your being-tested access rule never denies. For example:

http_access deny URLhaus !markURLhaus

2. Use %note logformat code to log all annotations or, with an
accessRule parameter, just the accessRule annotation.

Alex.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] A logging only ACL?

[Matus UHLAR - fantomas]

On 26.07.18 15:15, Ralf Hildebrandt wrote:

Before destroying our Squid proxy with an ACL, I'd like to LOG ACL hits
instead of using "http_access deny" straight away:

--- nsip ---
acl markURLhaus annotate_transaction accessRule=URLhaus
acl URLhaus url_regex "/etc/squid5/generated-urlhaus.acl"
http_access deny URLhaus markURLhaus
--- nsip ---

How?


you can configure an ACL and only define it in access_log directive.
separate log file would be preferred for this.

there's also "note" directive that allows you log notes for requests
matching ACL. http://www.squid-cache.org/Doc/config/note/


Underlying problem: https://urlhaus.abuse.ch/ is offering
a plain-text URL list here https://urlhaus.abuse.ch/downloads/text/

But in squid I must used "url_regex" - meaning I'll have to escape the
likes of .^$*+?()[{\|


the main problem is that HTTPS URL is only known when you do SSL
deciphering, which may happen:
- in reverse proxy scenario (using squid for ssl offloading)
- when you use SSL bumping (which is quote complicated to implement).

also note that url_regex ACLs are quite CPU hungry.

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
  One OS to rule them all, One OS to find them, 
One OS to bring them all and into darkness bind them 
___

squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] A logging only ACL?

[Ralf Hildebrandt]

Before destroying our Squid proxy with an ACL, I'd like to LOG ACL hits
instead of using "http_access deny" straight away:

--- nsip ---
acl markURLhaus annotate_transaction accessRule=URLhaus
acl URLhaus url_regex "/etc/squid5/generated-urlhaus.acl"
http_access deny URLhaus markURLhaus
--- nsip ---

How?

Underlying problem: https://urlhaus.abuse.ch/ is offering 
a plain-text URL list here https://urlhaus.abuse.ch/downloads/text/

But in squid I must used "url_regex" - meaning I'll have to escape the
likes of .^$*+?()[{\|

-- 
Ralf Hildebrandt   Charite Universitätsmedizin Berlin
ralf.hildebra...@charite.deCampus Benjamin Franklin
https://www.charite.de Hindenburgdamm 30, 12203 Berlin
Geschäftsbereich IT, Abt. Netzwerk fon: +49-30-450.570.155
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Regression after upgrading 3.5.27 -> 4.1

[Andrea Venturoli]

On 7/25/18 7:07 PM, Andrea Venturoli wrote:

On 7/25/18 6:46 PM, Amos Jeffries wrote:


What is your "squid -v" output?

If --disable-http-violations is used then relaxed parser will not
include those "must never be transmitted in un-escaped form" (RFC 2396)
characters.


It's there!!!

Thanks for pointing me in the correct direction.
I'm off recompiling... will let you know if this solves.


I can confirm removing this flag solved my problem.

Thanks to all.

 bye
av.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users