----- Original Message ----- > From: "Christopher Schultz" <ch...@christopherschultz.net> > To: users@tomcat.apache.org > Sent: Friday, December 6, 2019 12:41:23 PM > Subject: Re: Tomcat - No Fork for debugging? > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Alex, > > On 12/6/19 10:57, Alex Scheel wrote: > > Apologies, this reply got away from me... :-) > > Not at all. This is very helpful. > > > Perhaps grab a fresh $BEVERAGE before reading. > > It's before noon, so I grabbed a fresh espresso. > > :) >
~snip older context~ > >> RHEL ships OpenJDK and upstream itself doesn't do the FIPS > >> certification; it is Red Hat that does the certification of NSS > >> on RHEL. JDK itself, under the FIPS configuration above, now does > >> no crypto. All the crypto is done via PKCS#11 to NSS. So there's > >> nothing (in OpenJDK) to certify any more. Unlike in the non-FIPS > >> mode above, there's lots of crypto that would need to be > >> certified. > > Aah, okay. This makes more sense to me, now. At this point, the JSSE > provider is just a pass-through to NSS, just like the JSSE wrapper for > OpenSSL. Yep! And that's essentially how JSS behaves as well. Different wrappers with different goals by different teams. Same underlying crypto. > > >> Oracle, in their JDK, might ship their own PKCS#11 provider > >> and/or certify NSS, I don't know. I know IBM ships their own FIPS > >> crypto provider, and probably the next most popular is > >> BouncyCastle, but I thought that might be losing popularity > >> because they might've stopped FIPS certifying it, who knows. > > IMHO FIPS is useless at this point unless you are required to use a > FIPS-certified module, which many people are unfortunately required to > use. The reason I think it's useless is because FIPS mandates the use > of certain algorithms and some of them shouldn't be used. While I personally agree, there's still a number of people who need it or claim they want it. But I will say it is getting better, and oh, idk, by 2025 or so we should have FIPS 140-5 (with X25519 allowed!) and a sane set of cipher suites and allowed primitives (between SP800-56B rev 2 and a few other publications). By then... who knows what we'll have outside of FIPS. :-) > >> But my key point is there's no "OpenJDK FIPS Certificate" on the > >> FIPS page: > > > >> https://csrc.nist.gov/projects/cryptographic-module-validation-progra > m/validated-modules/search > > > >> (Search Java) > > > >> But there is one for NSS ("NSS" - and then its "NSS Cryptographic > >> Module" on the results page). > > Yep. I saw that Oracle has FIPS-certified modules, but they aren't a > part of the standard Java distribution (and certainly not part of > OpenJDK). I guess you get what you pay for. > > >> So yes, the SunJSSE provider is still used but the key > >> differences are: > > > >> 1) The crypto is done by SunPKCS11-NSS-FIPS provider, 2) The NSS > >> that is used is FIPS certified, 3) It places additional > >> restrictions (more about that below). > > > >> If you don't really care 2 (the token "Certificate") and just > >> want to test stuff, 1 and 3 work on any distro (e.g., Ubuntu and > >> Fedora), and NSS everywhere understands FIPS mode. > > > >> Also the SunPKCS11 / SunPKCS11-NSS-FIPS providers are limited to > >> Linux, iirc. > > Well, who would bother with NSS on Windows? Doesn't Mozilla still ship Firefox with NSS as the provider on Windows? Don't get me wrong, OpenSSL has definitely won the server space especially on Linux, but NSS still has a non-zero presence of client software just by virtue of Firefox and CentOS / RHEL 7 boxes (things like curl use it there). Also I know of some upstream consumers of JSS who run it on Mac and Windows (and thus build NSS on both). It can be done. ~snip long backtrace~ > Aha. "What's wrong with wrapping the KeyManager?" > > Answer: it doesn't work: KeyManagementException > > I guess they don't want any funny business. > > I'll have to look at how the KeyManager is built/wrapped. I didn't > write that code so I'm not too familiar with it. I know we need to > wave our hands around certain things due to the flexibility Tomcat > allows for certificate-selection, etc. > > When you get that error, is the object of type > org/apache/tomcat/util/net/jsse/JSSEKeyManager? Yessir. jdb: stop at JSSEKeyManager:<init> and a couple of step ups / steps later and you can see it get passed to the init of SSLEngine that is using the FIPS SunJSSE and the exception thrown. (And you can validate the type of the passed KeyManager[]'s -- there's only one element in the array). > I wonder if we are jumping the gun and always using that KeyManager > even when it's not strictly necessary. Well. What your KeyManager does (and one of the things I've not liked about the actual KeyManager interface because it doesn't do) is that it gives control of the certificate to the user :-) The _user_ configures a certificate they want to use and they know they're getting that certificate. The default KeyManager interface (whether X509KeyManager or X509ExtendedKeyManager) leaves selection of the certificate to the SSLEngine / KeyManager itself, based on clues the SSLEngine gives it. That's giving control to the devloper of some library somewhere in your stack. (I understand why it exists, to facilitate SNI/eSNI, but...) So in a messy NSS DB, potentially with lots of certificates with lots of subject names, you're not exactly sure which certificate you'd get, under the strict KeyManager interface. IMO, Tomcat's JSSE KeyManager wrapper fixes that problem. And that's why wrapping KeyManagers is necessary while not limiting SNI potential. s/NSS DB/jks/ I guess in the above for a more... popular take. :-) > > >> My config roughly looks like: > > > >> <Connector name="Secure" port="8443" scheme="https" secure="true" > >> SSLEnabled="true" keyAlias="localhost" > >> keystorePass="keystorePass" keystoreType="PKCS11" > >> keystoreProvider="SunPKCS11-NSS-FIPS" > >> keyManagerAlgorithm="SunX509" truststoreAlgorithm="PKCS11" > >> truststoreProvider="SunJSSE" truststoreType="SunX509" > >> truststorePassword="truststorePassword" SSLProtocol="TLSv1.2" > >> maxThreads="1" /> > > > >> (Yeah, that's messy -- but it illustrates the point -- you can > >> probably remove a number of those elements -- *including* > >> keyAlias, and it'll still fail). > > > >> This is because, for all KeyManagers, Tomcat wraps it its own > >> KeyManager: > > > >> https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/u > til/net/SSLUtilBase.java#L378 > > > >> Probably the minimum untested change that I'd have proposed > >> would be to wrap it in an if block like you have ealier: > > > >> https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/u > til/net/SSLUtilBase.java#L195 > > Sounds like the same as my conclusion. I wouldn't want to make that > change myself without understanding the other implications. Right. But I also kinda want to leave it alone and use it to push back and say hey, this is how its used in the wild. > > >> Now the core of the _why_ is because that's how NSS behaves. In > >> FIPS mode, NSS doesn't let you import keys. You can either have > >> keys in a NSS DB, or you can wrap/unwrap them into one (or the > >> temporary, in-memory key store). But you can't just call > >> PK11_ImportSymKey (or the related function for private keys) -- > >> they'll fail! > > > >> That's because NSS was a FIPS level 2 certified provider, unlike > >> say, OpenSSL which only undergoes FIPS level 1 certification. > >> Some of those restrictions that level 2 places are around key > >> management and handling, including where they're sourced from and > >> how they're protected in transit. > > Gotcha. Yeah, any idiot can "openssl genkey" and do whatever they want > with the key including publishing it on their web site. I mean, its not that you _can't_ do that with NSS DBs, its just that its a lot harder to do, especially if that NSS DB is in FIPS mode. You'd have to be cognizant of what you're doing. You'd know you're extracting it (the only way is as a wrapped, password-protected .p12 file) and then using some other tool (potentially on a different system not in FIPS mode) to unwrap it and publish it. > > >> One of the RFEs I've been discussing with our JDK team have been > >> to loosen the restrictions of the SunJSSE provider in FIPS mode. > >> Restricting the origin of the KeyManager (while having good > >> intentions of preventing the above) means that some otherwise > >> valid use cases -- like selecting a single key alias from the > >> server.xml config -- won't work. Really they should only care > >> that the key used is from the SunPKCS11-NSS-FIPS provider, they > >> don't care how they got it and from which KeyManager it came > >> from. > > Well, if it didn't come from a SunJSSE KeyManager, then it can't prove > that it wasn't tampered-with -- or even completely replaced, which > would effectively be an "imported key". So it kind of does make sense. But you can. SunPKCS11-NSS-FIPS Keys (which a KeyManager just facilitates access to) are just pointers. You'd need to do a huge amount of unsafe and conscious work to tamper with that key -- usually using JNI code and accessing NSS primitives -- in a way that violates any safety concerns. And you could still do all of that work if you leave it in the NSS DB and leave their pointer alone (you'd just do it before or after loading the SunPKCS11-NSS-FIPS key from their KeyManager). My assertion is that the correct test isn't "is this the right KeyManager class" but "is this the right _key_ / _certificate_ class"? The same problem applies to TrustManagers, except in reverse. If you restrict what TrustManagers are used (IIRC, the SunJSSE + SunPKCS11-NSS-FIPS does), you can't easily add additional validation. One of our requirements is to do is full-chain OCSP checking (from CC certification). The default SunX509 TrustManager doesn't do that. Most people probably don't want that. Yeah, you can shoot yourself in the foot and write a silly TrustManager. Anyone can do that, but they'd know what they're doing. And NSS lets you export public keys, so additional trust managers are possible in theory. At any rate, these are OpenJDK/SunJSSE problems and not Tomcat problems. But it does tell you a little of what we have to work with. :-) ~snip older context~ > Can I just say that "KBKDF" is a hilarious concept to me? > > Now we just need a KBPDF and we can go full-circle. :D I don't disagree, but it does have some interesting use cases. Smart cards have already established trust via symmetric keys when you initially provisioned them. When they're out in the field, and you want to put new certificates on them, you can reuse the existing symmetric keys and create a "secure channel" (SCP 03 is the protocol name) using KBKDF. It lets you prove integrity of the channel and identity of the card assuming you trust the initial key and the hardware to not give up the long-term secret. It is also used in Kerberos for similar purposes in RFC 8009. > > >> I'd like to get to the point where either work but both need a > >> few bug fixes and/or RFEs first. :-) > > > > > > > >>>>> However, currently on RHEL the SunJSSE provider is a bit > >>>>> broken, so this FIPSImplementation isn't really useful. If > >>>>> you're interested, I'd like to get Tomcat to a place where > >>>>> it supports FIPS mode JDK without the need for a separate > >>>>> implementation, but first I need a working FIPS-mode JDK > >>>>> (currently it doesn't handshake)... ...and we're probably > >>>>> going the route of finishing JSS anyways. > > > > Hmm. Is your goal to get a working JSSE+FIPS implementation > > specifically with JSS or do you just want to get FIPS working with > > Tomcat? The OpenSSL provider can be put into FIPS mode, provided > > that it was build with FIPS support of course. > > > >> I think I covered OpenSSL above. We have considered it and it > >> would be possible. But we'd prefer a NSS backed provider. > > > >> So either JSSE+SunPKCS11-NSS-FIPS or JSSE+/-JSS. :-) > > Understood. > > Mark will probably read this novel and say "oh, that's easy" and just > implement it. Having not written the original code and not > understanding the various intricacies of why the replacement > KeyManager is being used, etc., I wouldn't want to break things for > large numbers of people. > > Have you tried hacking Tomcat to just never wrap the KeyManager and > use a stock JSSE one? Does it work in that case? No... But since you asked and its a small change... And its Friday... :-) I made this change since I'm lazy: diff --git a/java/org/apache/tomcat/util/net/SSLUtilBase.java b/java/org/apache/tomcat/util/net/SSLUtilBase.java index bcbe3903e8..d8635f757a 100644 --- a/java/org/apache/tomcat/util/net/SSLUtilBase.java +++ b/java/org/apache/tomcat/util/net/SSLUtilBase.java @@ -374,9 +374,6 @@ public abstract class SSLUtilBase implements SSLUtil { if ("JKS".equals(certificate.getCertificateKeystoreType())) { alias = alias.toLowerCase(Locale.ENGLISH); } - for(int i = 0; i < kms.length; i++) { - kms[i] = new JSSEKeyManager((X509KeyManager)kms[i], alias); - } } return kms; Obviously that won't merge upstream without more understanding. This starts without any exceptions (unlike before): Dec 06, 2019 3:28:37 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["https-jsse-nio-8443"] Dec 06, 2019 3:28:37 PM org.apache.catalina.startup.Catalina load INFO: Server initialization in [826] milliseconds Dec 06, 2019 3:28:37 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service [Catalina] Dec 06, 2019 3:28:37 PM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet engine: [Apache Tomcat/9.0.30-dev] Dec 06, 2019 3:28:37 PM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["https-jsse-nio-8443"] Dec 06, 2019 3:28:37 PM org.apache.catalina.startup.Catalina start INFO: Server startup in [40] milliseconds And we get to where I got with my TomcatJSSE FIPS mode implementation: [root@localhost tomcat]# wget https://localhost:8443 --2019-12-06 15:29:13-- https://localhost:8443/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8443... connected. GnuTLS: A TLS fatal alert has been received. GnuTLS: received alert [80]: Internal error Unable to establish SSL connection. And in the logs: Dec 06, 2019 3:29:13 PM org.apache.tomcat.util.net.NioEndpoint$SocketProcessor doRun SEVERE: Error running socket processor java.lang.RuntimeException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ATTRIBUTE_VALUE_INVALID at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1519) at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:499) at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:238) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1575) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: java.security.ProviderException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ATTRIBUTE_VALUE_INVALID at sun.security.pkcs11.P11KeyPairGenerator.generateKeyPair(P11KeyPairGenerator.java:424) at java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:697) at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:65) at sun.security.ssl.ServerHandshaker.setupEphemeralECDHKeys(ServerHandshaker.java:1516) at sun.security.ssl.ServerHandshaker.trySetCipherSuite(ServerHandshaker.java:1311) at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1108) at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:814) at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:221) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) at java.security.AccessController.doPrivileged(Native Method) at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) at org.apache.tomcat.util.net.SecureNioChannel.tasks(SecureNioChannel.java:443) at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:507) ... 7 more Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ATTRIBUTE_VALUE_INVALID at sun.security.pkcs11.wrapper.PKCS11.C_GenerateKeyPair(Native Method) at sun.security.pkcs11.P11KeyPairGenerator.generateKeyPair(P11KeyPairGenerator.java:416) ... 21 more So clearly the SunJSSE in FIPS mode needs a little bit more work. (https://bugzilla.redhat.com/show_bug.cgi?id=1780339) But I was correct about the KeyManager being the issue. \o/ So I think we should find Mark and ask what's the best path forward. I'd still like to see if we can get the JDK restrictions loosened just a little. I have a few downstream trackers I'm watching for that. But if those fail... I'm partial to detecting the class of the KeyManager (it'd be sun.security.ssl.SunX509KeyManagerImpl in the event we're using JSSE), but we'd need to couple that with another FIPS-mode check... probably it'd be sufficient to see if the SunPKCS11-NSS-FIPS provider is present. And check which provide we're using for the SSLEngine. However... non-PKCS#11 KeyStore's are not allowed in FIPS mode either (including both JKS and PKCS#12 files, go figure). For maximal usability, I'd suggest detecting bad configurations and helping users towards fixing their config. That'd be elsewhere and unrelated to this KeyManager change though. Really up to you folks how much you'd like to take upstream. I'm happy to help and contribute as desired, - Alex --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
