Re: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[David Filip]

Well, it is not consistent ... sometimes when I stop and start it from the 
command line it works, and other times it doesn't, but every time it is getting 
the -Djavax.net .ssl.trustStore parameter ... which I check 
by doing a:

$ ps -aef | grep java | tr ' ' '\n'

which lists each parameter on a separate line.  So I wish it were that simple, 
but it does not appear to be so.  But specifically to answer your question, I 
use this script to toggle tomcat on and off:

# tomcat
#
# Start / Stop Tomcat Application Server
#
# - If tomcat is running, stop it
# - if tomcat is not running, start it
#
# 24-Apr-2010 - DEF, original coding
#

found=`ps -aef | grep /Library/Tomcat/bin/bootstrap.jar | grep -v grep | wc -l`

if [ $found -eq 0 ]
then
echo Starting Tomcat Application Server ...
sudo launchctl load /Library/LaunchDaemons/org.apache.tomcat.plist
else
echo Stopping Tomcat Application Server ...
sudo launchctl unload /Library/LaunchDaemons/org.apache.tomcat.plist
fi

And the org.apache.tomcat.plist contains:


http://www.apple.com/DTDs/PropertyList-1.0.dtd;>


Label
org.apache.tomcat
RunAtLoad

ProgramArguments

/Library/Tomcat/bin/catalina.sh
run

StandardOutPath
/tmp/Tomcat.log
UserName
tomcat



so it is using catalina.sh.  But right now, I just ran this command:

$ ps -aef | grep java | tr ' ' '\n' | grep -- '-D'
-Djava.util.logging.config.file=/Library/Tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Dbis.home=/home/infodesk
-Dbis.download=/tmp
-Dinfodoc.home=/home/infodesk
-Dinfodoc.download=/tmp
-Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts
-Djdk.tls.ephemeralDHKeySize=2048
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027
-Djava.endorsed.dirs=/Library/Tomcat/endorsed
-Dcatalina.base=/Library/Tomcat
-Dcatalina.home=/Library/Tomcat
-Djava.io.tmpdir=/Library/Tomcat/temp

and that file exists:

$ ls -l 
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts
-rw-rw-r--  1 root  wheel  115588 Dec  1  2019 
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts

but I am getting the error:

sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target

However, I may have previously misspoken, as I *thought* it was related to 
Apache 8.5.x, because I saw it on the new server with 8.5.57 but not an older 
server running 7.0.x, and when I upgraded the old 7.0.x server to 8.5.57 it 
immediately began to exhibit the problem.  So cause and effect, right?  But I 
just downgraded the old server back to 7.0.x, and I am still seeing the 
problem!  Ugh!

So I tried running the same app on an 8.5.37 server running on AWS Linux 2 
(similar to CentOS 7), and it works fine there, even after stopping and 
starting the Tomcat server 6 (!) times -- just because I don't trust anything 
right now.

My original thought -- which I did not previously share because I convinced 
myself it was crazy -- was that I first noticed the problem after I applied the 
latest macOS security patch.  However, once I saw that the Tomcat 7.0.x server 
with the same macOS security patch did not exhibit the problem, I ASSUMED it 
was related to the Tomcat version ... but as I said, now that I have downgraded 
that server back to 7.0.x, I am not still -- sometimes -- seeing the problem.  
Ugh.

So I am now taking the "new" server and restoring from a backup taken a week 
ago -- before I applied the macOS security patch -- to see if that makes a 
difference.

Given all of that, I can assure you that I am not drinking -- at least not 
while in front of a computer -- and I am not taking any drugs, and as far as I 
know, I am not clinically insane.  But I still can't explain all of the 
inconsistencies I am seeing, and the one thing that I always hate most when 
debugging a problem is lack of a consistent reproducibility.

> On Aug 8, 2020, at 5:17 PM, calder  wrote:
> 
> On Sat, Aug 8, 2020, 13:59 David Filip  > wrote:
> 
>> Hello Everyone!
>> 
>> I spent a large part of yesterday and this morning trying to debug an SSL
>> problem on Tomcat 8.5.57 to no avail.  I've seen some discussion on either
>> this problem or something related back in 2016, but wanted to confirm what
>> the "correct" solution might be, because I got lost in the threads.
>> 
>> I never had this problem with Tomcat 7.0.x, but it started once I upgraded
>> to 8.5.57 (same 

Re: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[calder]

On Sat, Aug 8, 2020, 13:59 David Filip  wrote:

> Hello Everyone!
>
> I spent a large part of yesterday and this morning trying to debug an SSL
> problem on Tomcat 8.5.57 to no avail.  I've seen some discussion on either
> this problem or something related back in 2016, but wanted to confirm what
> the "correct" solution might be, because I got lost in the threads.
>
> I never had this problem with Tomcat 7.0.x, but it started once I upgraded
> to 8.5.57 (same application code), and it is related to making outgoing SSL
> connections to web services.  And this is NOT related to a self-signed, but
> to a commercial (GoDaddy) SSL certificate, albeit on a server that I also
> run in the cloud.
>
> The exception is being thrown when trying to connect to an SSL protected
> web service is:
>
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
>
> although the exact same code worked (and still works on other servers)
> reliably under Tomcat 7.0.x for several years.
>
> Now, here is the weird part: after Google'ing around, I thought the
> problem might be that Tomcat 8.5.5 and later -- at least this is the gist
> that I got -- no longer finds the 'default' Java certificate store
> (cacerts), so I added the following to /bin/catalina.sh (running on a Mac
> 10.14 / Mojave):
>
> export
> -Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts
>
> The weird part is that this appeared to fix the problem, so I thought I
> was done.  Then, I rebooted, and the problem re-appeared!
>
> I stopped and started Tomcat, and the problem was resolved again.  I
> rebooted again, and the problem re-appeared.
>



When you "stopped and started Tomcat", how did restart it?  At the command
line using one of the Tomcat shell scripts?

My thought is, "whatever" fires up Tomcat after an iOS system reboot - that
startup process does not call catalina.sh.

But when you start Tomcat manually, using catalina.sh or startup.sh (which
calls catalina.sh), it works because the Java option is being set.



Previously, when it worked, I refreshed the page several times, and it kept
> working.  When it doesn't work, if I keep refreshing the page, it continues
> to throw the exception.
>
> Does this mean that some "worker threads" can find the certificate store,
> and others can't?  Or am I going down the wrong rabbit hole?
>
> So, any idea?  The intermittent nature is driving me crazy!
>
> And I have can reproduce the problem on two separate servers (both Mac
> 10.14 / Mojave, both Java 1.8.0), one (new server) running 8.5.57 and one
> (slightly older server) running 8.5.35.  But again, I have several 7.0.x
> instances where I've never seen this problem before.
>
> Also, the generic 'SSLPoke' always connects to the service, and it appears
> that if I run (mostly) the same code from the command line outside of
> Tomcat (javac / java) it always works.  And if I paste the web service URL
> into Safari or Chrome, it always works.  And if I use the web service URL
> with curl (just for good measure), it always works.  So it only seems to
> fall under Tomcat 8.5.x.
>
> Thanks in advance for any guidance, as I'm running out of things to Google
> and try.
>

Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[David Filip]

Hello Everyone!

I spent a large part of yesterday and this morning trying to debug an SSL 
problem on Tomcat 8.5.57 to no avail.  I've seen some discussion on either this 
problem or something related back in 2016, but wanted to confirm what the 
"correct" solution might be, because I got lost in the threads.

I never had this problem with Tomcat 7.0.x, but it started once I upgraded to 
8.5.57 (same application code), and it is related to making outgoing SSL 
connections to web services.  And this is NOT related to a self-signed, but to 
a commercial (GoDaddy) SSL certificate, albeit on a server that I also run in 
the cloud.

The exception is being thrown when trying to connect to an SSL protected web 
service is:

sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target

although the exact same code worked (and still works on other servers) reliably 
under Tomcat 7.0.x for several years.

Now, here is the weird part: after Google'ing around, I thought the problem 
might be that Tomcat 8.5.5 and later -- at least this is the gist that I got -- 
no longer finds the 'default' Java certificate store (cacerts), so I added the 
following to /bin/catalina.sh (running on a Mac 10.14 / Mojave):

export 
-Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/security/cacerts

The weird part is that this appeared to fix the problem, so I thought I was 
done.  Then, I rebooted, and the problem re-appeared!

I stopped and started Tomcat, and the problem was resolved again.  I rebooted 
again, and the problem re-appeared.

Previously, when it worked, I refreshed the page several times, and it kept 
working.  When it doesn't work, if I keep refreshing the page, it continues to 
throw the exception.

Does this mean that some "worker threads" can find the certificate store, and 
others can't?  Or am I going down the wrong rabbit hole?

So, any idea?  The intermittent nature is driving me crazy!

And I have can reproduce the problem on two separate servers (both Mac 10.14 / 
Mojave, both Java 1.8.0), one (new server) running 8.5.57 and one (slightly 
older server) running 8.5.35.  But again, I have several 7.0.x instances where 
I've never seen this problem before.

Also, the generic 'SSLPoke' always connects to the service, and it appears that 
if I run (mostly) the same code from the command line outside of Tomcat (javac 
/ java) it always works.  And if I paste the web service URL into Safari or 
Chrome, it always works.  And if I use the web service URL with curl (just for 
good measure), it always works.  So it only seems to fall under Tomcat 8.5.x.

Thanks in advance for any guidance, as I'm running out of things to Google and 
try.

Regards,

Dave.