[squid-users] generate-host-certificates=on fails to generate certificates for _some_ hosts

[Greg Hulands]

Hi,
I am setting up squid 5.0.3 and during testing I have found some websites fail 
to have their certificates generated correctly. I am able to go to sites like 
YouTube.com  and have the certificates for that be 
generated correctly, but when I try to go to some others, like arstechnica.com 
, they fail to generate and return the CA cert that 
squid is using to sign certificates with.

I turned the logging up on certificate stuff to 5 and have the cache log from 
trying to make a request here: 
https://gist.github.com/ghulands/f89b49bf180bfac86c98c46c4260f1eb 


My ssl-bump config is 

ssl_bump peek step1
ssl_bump bump all

Does anyone have any suggestions or insight on what might the problem be?

Thanks,
Greg


$ squid --version
Squid Cache: Version 5.0.3
Service Name: squid

This binary uses OpenSSL 1.1.1i  8 Dec 2020. For legal restrictions on 
distribution see https://www.openssl.org/source/license.html 


configure options:  '--with-default-user=squid' '--bindir=/usr/local/sbin' 
'--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' 
'--libexecdir=/usr/local/libexec/squid' '--localstatedir=/var' 
'--sysconfdir=/usr/local/etc/squid' '--with-logdir=/var/log/squid' 
'--with-pidfile=/var/run/squid/squid.pid' '--with-swapdir=/var/squid/cache' 
'--without-gnutls' '--with-included-ltdl' '--enable-auth' '--enable-zph-qos' 
'--enable-build-info' '--enable-loadable-modules' 
'--enable-removal-policies=lru heap' '--disable-epoll' 
'--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-translation' 
'--disable-arch-native' '--disable-strict-error-checking' '--enable-eui' 
'--enable-cache-digests' '--enable-delay-pools' '--disable-ecap' 
'--disable-esi' '--enable-follow-x-forwarded-for' '--with-mit-krb5=/usr/local' 
'CFLAGS=-I/usr/local/include -O2 -pipe  -fstack-protector-strong 
-fno-strict-aliasing ' 'LDFLAGS=-L/usr/local/lib  -pthread -L/usr/local/lib 
-lpcreposix -lpcre -Wl,-rpath,/usr/local/lib:/usr/lib -Wl,-rpath,/usr/local/lib 
-fstack-protector-strong ' 'LIBS=-lkrb5 -lgssapi_krb5 ' 
'KRB5CONFIG=/usr/local/bin/krb5-config' 
'krb5_config=/usr/local/bin/krb5-config' '--enable-htcp' '--enable-icap-client' 
'--enable-icmp' '--enable-ident-lookups' '--enable-ipv6' '--enable-kqueue' 
'--with-large-files' '--enable-http-violations' '--without-nettle' 
'--enable-snmp' '--enable-ssl' '--with-openssl=/usr/local' 
'--enable-security-cert-generators=file' 
'LIBOPENSSL_CFLAGS=-I/usr/local/include' 'LIBOPENSSL_LIBS=-lcrypto -lssl' 
'--enable-ssl-crtd' '--disable-stacktraces' '--disable-tdb' 
'--disable-ipf-transparent' '--enable-ipfw-transparent' 
'--disable-pf-transparent' '--without-nat-devpf' '--enable-forw-via-db' 
'--enable-wccp' '--enable-wccpv2' '--enable-auth-basic=DB SMB_LM NCSA PAM POP3 
RADIUS fake getpwnam NIS' '--enable-auth-digest=file' 
'--enable-external-acl-helpers=file_userip unix_group delayer' 
'--enable-auth-negotiate=kerberos wrapper' '--enable-auth-ntlm=fake SMB_LM' 
'--enable-storeio=aufs diskd rock ufs' '--enable-disk-io=DiskThreads DiskDaemon 
AIO Blocking IpcIo Mmapped' '--enable-log-daemon-helpers=file DB' 
'--enable-url-rewrite-helpers=fake LFS' '--enable-storeid-rewrite-helpers=file' 
'--enable-security-cert-validators=fake' '--prefix=/usr/local' 
'--mandir=/usr/local/man' '--disable-silent-rules' 
'--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd12.2' 
'build_alias=amd64-portbld-freebsd12.2' 'CC=cc' 'CPPFLAGS=-I/usr/local/include' 
'CXX=c++' 'CXXFLAGS=-O2 -pipe -fstack-protector-strong -fno-strict-aliasing  ' 
'CPP=cpp' --enable-ltdl-convenience___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] generate-host-certificates=on fails to generate certificates for _some_ hosts

[Alex Rousskov]

On 1/13/21 4:33 PM, Greg Hulands wrote:

> I am setting up squid 5.0.3 and during testing I have found some 
> websites fail to have their certificates generated correctly. I am
> able to go to sites like YouTube.com and have the certificates for
> that be generated correctly, but when I try to go to some others,
> like arstechnica.com, they fail to generate and return the CA cert
> that squid is using to sign certificates with.

Just to double check: Are you sure that the certificate the client gets
is the configured CA certificate? For example, do the two certificates
have the same fingerprint?


> I turned the logging up on certificate stuff to 5 and have the cache log
> from trying to make a request
> here: https://gist.github.com/ghulands/f89b49bf180bfac86c98c46c4260f1eb

The posted snippet shows successful TLS negotiation with the origin
server (FD 23) and a subsequently failed negotiation with the client (FD
21). The latter may have failed because the client did not like the
certificate generated by Squid, but I did not check the exact failure
reason carefully.

The snippet has no information about Squid sending the (generated)
certificates to the client, but Squid appears to receive some generated
certificate from the helper (crtGenRq3180846).

* If you are sure that the client gets a wrong certificate from Squid,
then I recommend posting an ALL,9 log of the problematic transaction.
With some luck, we may be able to see what went wrong with certificate
generation (or virgin certificate validation??).

* Otherwise, I recommend double checking what certificate the client
gets. If the client gets the correct generated certificate, then the
problem is not in certificate validation or generation.

Posting the certificate that the client actually gets may help a lot
with the triage as well.


HTH,

Alex.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] generate-host-certificates=on fails to generate certificates for _some_ hosts

[Alex Rousskov]

On 1/13/21 9:47 PM, Greg Hulands wrote:
> I have put the ALL,9 log
> here https://gist.github.com/ghulands/4a689db93fc87f9e7f69174f292f1914

> I can see it generates the certificate correctly,

Agreed. Squid receives (from the helper) a generated certificate with
the right wildcard CN, not a CA certificate.


> but couldn’t identify why it didn’t return the cert to the client.

Yeah... Squid is calling the code that should set the certificate for
the client connection. Unfortunately, I cannot easily tell whether that
code is using the right certificate -- the existing debugging may not
even reveal that detail.

If you see a different certificate received by the client -- something I
cannot verify from the logs -- then perhaps Squid incorrectly switched
the right certificate to a different one or Squid failed to set the
right certificate but forgot to report the problem (and the CA
certificate from the related context was used?). These are just wild
guesses.

If you do not get better suggestions for going forward, consider these
last-straw ideas:

* Testing with a client like openssl, try disabling TLS v1.3. It is
being used by the client in your logs. Perhaps there is something in TLS
v1.3 that requires special handing when talking to the client. I know
that Squid has problems with TLS v1.3 on the Squid-to-server
connections... (In your case, the Squid-to-server connection is TLS v1.2
AFAICT).

* Upgrade to the latest v5 or even v6. I see no relevant fixes in v5 but
I could miss them.

* If you are a developer, add more debugging or use gdb to find out what
happens with the Squid-to-client certificate. Otherwise, find a
developer who can do that for you.

Sorry I cannot think of any good options here.

Alex.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] generate-host-certificates=on fails to generate certificates for _some_ hosts

[Greg Hulands]

Hi Alex,
Thanks for the help. Comments inline.


> On Jan 13, 2021, at 2:23 PM, Alex Rousskov  
> wrote:
> 
> On 1/13/21 4:33 PM, Greg Hulands wrote:
> 
>> I am setting up squid 5.0.3 and during testing I have found some 
>> websites fail to have their certificates generated correctly. I am
>> able to go to sites like YouTube.com and have the certificates for
>> that be generated correctly, but when I try to go to some others,
>> like arstechnica.com, they fail to generate and return the CA cert
>> that squid is using to sign certificates with.
> 
> Just to double check: Are you sure that the certificate the client gets
> is the configured CA certificate? For example, do the two certificates
> have the same fingerprint?

Yes, I verified it’s the same certificate - fingerprints are a match.

> 
>> I turned the logging up on certificate stuff to 5 and have the cache log
>> from trying to make a request
>> here: https://gist.github.com/ghulands/f89b49bf180bfac86c98c46c4260f1eb
> 
> The posted snippet shows successful TLS negotiation with the origin
> server (FD 23) and a subsequently failed negotiation with the client (FD
> 21). The latter may have failed because the client did not like the
> certificate generated by Squid, but I did not check the exact failure
> reason carefully.
> 
> The snippet has no information about Squid sending the (generated)
> certificates to the client, but Squid appears to receive some generated
> certificate from the helper (crtGenRq3180846).
> 
> * If you are sure that the client gets a wrong certificate from Squid,
> then I recommend posting an ALL,9 log of the problematic transaction.
> With some luck, we may be able to see what went wrong with certificate
> generation (or virgin certificate validation??).

I have put the ALL,9 log here 
https://gist.github.com/ghulands/4a689db93fc87f9e7f69174f292f1914 


I can see it generates the certificate correctly, but couldn’t identify why it 
didn’t return the cert to the client.

> 
> * Otherwise, I recommend double checking what certificate the client
> gets. If the client gets the correct generated certificate, then the
> problem is not in certificate validation or generation.
> 
> Posting the certificate that the client actually gets may help a lot
> with the triage as well.

The certificate that gets returned is in the logs as it’s the CA cert.

Thanks,
Greg

> 
> 
> HTH,
> 
> Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] generate-host-certificates=on fails to generate certificates for _some_ hosts

[Greg Hulands]

Hey Alex,
Can you point me to the rough location in code where the certs are sent to the 
client.

I tried with TLS 1.2 with openssl s_client and it returned the certs the same.

Thanks,
Greg

> On Jan 13, 2021, at 8:44 PM, Alex Rousskov  
> wrote:
> 
> On 1/13/21 9:47 PM, Greg Hulands wrote:
>> I have put the ALL,9 log
>> here https://gist.github.com/ghulands/4a689db93fc87f9e7f69174f292f1914
> 
>> I can see it generates the certificate correctly,
> 
> Agreed. Squid receives (from the helper) a generated certificate with
> the right wildcard CN, not a CA certificate.
> 
> 
>> but couldn’t identify why it didn’t return the cert to the client.
> 
> Yeah... Squid is calling the code that should set the certificate for
> the client connection. Unfortunately, I cannot easily tell whether that
> code is using the right certificate -- the existing debugging may not
> even reveal that detail.
> 
> If you see a different certificate received by the client -- something I
> cannot verify from the logs -- then perhaps Squid incorrectly switched
> the right certificate to a different one or Squid failed to set the
> right certificate but forgot to report the problem (and the CA
> certificate from the related context was used?). These are just wild
> guesses.
> 
> If you do not get better suggestions for going forward, consider these
> last-straw ideas:
> 
> * Testing with a client like openssl, try disabling TLS v1.3. It is
> being used by the client in your logs. Perhaps there is something in TLS
> v1.3 that requires special handing when talking to the client. I know
> that Squid has problems with TLS v1.3 on the Squid-to-server
> connections... (In your case, the Squid-to-server connection is TLS v1.2
> AFAICT).
> 
> * Upgrade to the latest v5 or even v6. I see no relevant fixes in v5 but
> I could miss them.
> 
> * If you are a developer, add more debugging or use gdb to find out what
> happens with the Squid-to-client certificate. Otherwise, find a
> developer who can do that for you.
> 
> Sorry I cannot think of any good options here.
> 
> Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] generate-host-certificates=on fails to generate certificates for _some_ hosts

[Eliezer Croitoru]

Hey Greg,

I am trying to test it with 5.0.4 and it seems that this site works for me with 
SSL BUMP.
The CN and the SAN are the same so it makes sense that it should work the same 
on your proxy.
However I do see that this domain has 2 IP addresses which might affect what 
you see.
I am trying to verify this issue locally.

I wrote the next ruby script to help others with some insights.
https://github.com/elico/tls-check-script

Both ip addresses seem to give the same certificate.
I am using openssl to see the certificate:
openssl s_client -showcerts -servername arstechnica.com -connect 
arstechnica.com:443 /dev/null | openssl x509 -noout -text

Let me know if something specific is seen in your environment.

It shouldn't matter too much but, what OS are you running squid ontop and what 
is "squid -v" output?

Thanks,
Eliezer


Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: ngtech1...@gmail.com
Zoom: Coming soon


-Original Message-
From: squid-users  On Behalf Of Greg 
Hulands
Sent: Thursday, January 14, 2021 8:22 AM
To: Alex Rousskov 
Cc: squid-users@lists.squid-cache.org
Subject: Re: [squid-users] generate-host-certificates=on fails to generate 
certificates for _some_ hosts

Hey Alex,
Can you point me to the rough location in code where the certs are sent to the 
client.

I tried with TLS 1.2 with openssl s_client and it returned the certs the same.

Thanks,
Greg

> On Jan 13, 2021, at 8:44 PM, Alex Rousskov  
> wrote:
> 
> On 1/13/21 9:47 PM, Greg Hulands wrote:
>> I have put the ALL,9 log
>> here https://gist.github.com/ghulands/4a689db93fc87f9e7f69174f292f1914
> 
>> I can see it generates the certificate correctly,
> 
> Agreed. Squid receives (from the helper) a generated certificate with
> the right wildcard CN, not a CA certificate.
> 
> 
>> but couldn’t identify why it didn’t return the cert to the client.
> 
> Yeah... Squid is calling the code that should set the certificate for
> the client connection. Unfortunately, I cannot easily tell whether that
> code is using the right certificate -- the existing debugging may not
> even reveal that detail.
> 
> If you see a different certificate received by the client -- something I
> cannot verify from the logs -- then perhaps Squid incorrectly switched
> the right certificate to a different one or Squid failed to set the
> right certificate but forgot to report the problem (and the CA
> certificate from the related context was used?). These are just wild
> guesses.
> 
> If you do not get better suggestions for going forward, consider these
> last-straw ideas:
> 
> * Testing with a client like openssl, try disabling TLS v1.3. It is
> being used by the client in your logs. Perhaps there is something in TLS
> v1.3 that requires special handing when talking to the client. I know
> that Squid has problems with TLS v1.3 on the Squid-to-server
> connections... (In your case, the Squid-to-server connection is TLS v1.2
> AFAICT).
> 
> * Upgrade to the latest v5 or even v6. I see no relevant fixes in v5 but
> I could miss them.
> 
> * If you are a developer, add more debugging or use gdb to find out what
> happens with the Squid-to-client certificate. Otherwise, find a
> developer who can do that for you.
> 
> Sorry I cannot think of any good options here.
> 
> Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] generate-host-certificates=on fails to generate certificates for _some_ hosts

[Alex Rousskov]

On 1/14/21 1:22 AM, Greg Hulands wrote:

> Can you point me to the rough location in code where the certs are sent to 
> the client.

I would start with the following log line:

> 2021/01/13 18:09:11.655 kid1| 33,5| client_side.cc(2700) sslCrtdHandleReply: 
> Certificate for arstechnica.com was successfully recieved from ssl_crtd

You can see the exact code location of each low-level debug statement.
Unfortunately, Squid uses a somewhat unusual filename(linenumber)
notation for that, but the information is there -- client_side.cc line
2700. After the source code location, Squid prints the function name
where the message was generated -- sslCrtdHandleReply.

The above line points to a method that receives the (parsed) certificate
from the code that talks to the helper. IIRC, Squid then calls
Ssl::configureSSLUsingPkeyAndCertFromMemory() to apply the certificate
to the client-Squid TLS connection. It would be good to confirm that the
certificate Squid applies is the right one and that all the application
functions were successful (e.g., Squid may forget to check some OpenSSL
function result). This will require some development skills.


> I tried with TLS 1.2 with openssl s_client and it returned the certs the same.

Noted. You may prefer to test with TLS v1.2 (first) because it may
produce simpler debugging logs and create fewer uncertainties. The code
mentioned above is used for all TLS versions.


Good luck,

Alex.


>> On Jan 13, 2021, at 8:44 PM, Alex Rousskov 
>>  wrote:
>>
>> On 1/13/21 9:47 PM, Greg Hulands wrote:
>>> I have put the ALL,9 log
>>> here https://gist.github.com/ghulands/4a689db93fc87f9e7f69174f292f1914
>>
>>> I can see it generates the certificate correctly,
>>
>> Agreed. Squid receives (from the helper) a generated certificate with
>> the right wildcard CN, not a CA certificate.
>>
>>
>>> but couldn’t identify why it didn’t return the cert to the client.
>>
>> Yeah... Squid is calling the code that should set the certificate for
>> the client connection. Unfortunately, I cannot easily tell whether that
>> code is using the right certificate -- the existing debugging may not
>> even reveal that detail.
>>
>> If you see a different certificate received by the client -- something I
>> cannot verify from the logs -- then perhaps Squid incorrectly switched
>> the right certificate to a different one or Squid failed to set the
>> right certificate but forgot to report the problem (and the CA
>> certificate from the related context was used?). These are just wild
>> guesses.
>>
>> If you do not get better suggestions for going forward, consider these
>> last-straw ideas:
>>
>> * Testing with a client like openssl, try disabling TLS v1.3. It is
>> being used by the client in your logs. Perhaps there is something in TLS
>> v1.3 that requires special handing when talking to the client. I know
>> that Squid has problems with TLS v1.3 on the Squid-to-server
>> connections... (In your case, the Squid-to-server connection is TLS v1.2
>> AFAICT).
>>
>> * Upgrade to the latest v5 or even v6. I see no relevant fixes in v5 but
>> I could miss them.
>>
>> * If you are a developer, add more debugging or use gdb to find out what
>> happens with the Squid-to-client certificate. Otherwise, find a
>> developer who can do that for you.
>>
>> Sorry I cannot think of any good options here.
>>
>> Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users