-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Alex,
On 12/6/19 10:57, Alex Scheel wrote: > Apologies, this reply got away from me... :-) Not at all. This is very helpful. > Perhaps grab a fresh $BEVERAGE before reading. It's before noon, so I grabbed a fresh espresso. :) > ----- Original Message ----- >> From: "Christopher Schultz" <ch...@christopherschultz.net> To: >> users@tomcat.apache.org Sent: Friday, December 6, 2019 9:51:52 >> AM Subject: Re: Tomcat - No Fork for debugging? >> > Alex, > > On 12/5/19 18:05, Alex Scheel wrote: >>>> ----- Original Message ----- >>>>> From: "Christopher Schultz" <ch...@christopherschultz.net> >>>>> To: users@tomcat.apache.org Sent: Thursday, December 5, >>>>> 2019 5:05:42 PM Subject: Re: Tomcat - No Fork for >>>>> debugging? >>>>> >>>> Alex, >>>> >>>> On 12/5/19 15:18, Alex Scheel wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> I've tried searching to no avail. >>>>>>> >>>>>>> I'm working on a(nother) SSL adapter. However, I've had >>>>>>> some issues with it. There's a native component and I'm >>>>>>> trying to tease apart its relationship with why the >>>>>>> client won't handshake. The stack traces aren't overly >>>>>>> helpful and I'd love to attach gdb to this and set a >>>>>>> few specific environment variables. >>>>>>> >>>>>>> Query: >>>>>>> >>>>>>> Does Tomcat have a mode where it won't fork to a >>>>>>> different user and will run with a limited number of >>>>>>> threads? That'd greatly improve my ability to debug. >>>>>>> Something similar to `radiusd -X` or `sshd -d`? >>>> >>>> You can accomplish this in two steps: >>>> >>>> 1. Change your <Connector> to have an <Executor> with only a >>>> single thread, or set maxThreads="1" on the <Connector> if >>>> you aren't separately configuring an <Executor> >>>> >>>> 2. From the command line, run: >>>> >>>> $ bin/catalina.sh run >>>> >>>>> I think that behaves better. Thanks! > > No problem. > >>>> This will run Tomcat directly in the terminal window (or, on >>>> Windows, open a second terminal window where it will run). >>>> >>>> You can see stdout in that terminal, and you can press CTRL-C >>>> to kill the process. >>>> >>>>>>> Rationale: >>>>>>> >>>>>>> NSS has support for logging calls to its PKCS#11 >>>>>>> interface to a file, based on the presence of >>>>>>> environment variables. When I set these environment >>>>>>> variables and directly call the JVM to start Tomcat: >>>>>>> >>>>>>> # java -classpath $CLASSPATH $FLAGS >>>>>>> org.apache.catalina.startup.Bootstrap start >>>>>>> >>>>>>> I see it logging calls when the JDK starts up, but when >>>>>>> I hit it with wget on the TLS port, the resulting >>>>>>> PKCS#11 calls aren't logged. When launching in gdb, I >>>>>>> get an error about /sbin/nologin doesn't understand the >>>>>>> -c option, which to me says that Bootstrap is trying to >>>>>>> fork and create a new shell (I'm running as root in a >>>>>>> VM and it wants to launch as the tomcat user), dropping >>>>>>> my environmental variables I want. >>>>>>> >>>>>>> Ideally (for debugging) I'd like to simplify this. Is >>>>>>> there a more direct entry route I can use perhaps? >>>> >>>> Is there a JSSE wrapper for NSS? You can just plug-in the >>>> crypto provider for the SSLContext instead of writing your >>>> own connector. Wait. You said "adapter". What kind of >>>> "adapter" are you writing? >>>> >>>>> Ah sorry, my terminology was loose there. I'm writing >>>>> another org.apache.tomcat.util.net.SSLImplementation >>>>> implementation. Heh. >>>> >>>>> I'm the maintainer of JSS, a NSS wrapper in Java. This is >>>>> mostly utilized by the Dogtag project. >>>> >>>>> I also co-maintain TomcatJSS, a JSS adapter for Tomcat < >>>>> 8.5. When used with Tomcat >= 8.5, it uses JSSE's SSLEngine >>>>> but initializes JSS based on configuration in server.xml >>>>> and use it for the keystore. Mostly this is because we've >>>>> not yet gotten around to adding a SSLEngine to JSS (but >>>>> will likely finish that work soon -- JSS was started at >>>>> Netscape back in the late 90s and predated the SSLEngine >>>>> being added to JVM interfaces in JDK version 5). RHEL 7 >>>>> still ships with Tomcat 7, but RHEL 8 only has Tomcat 9, so >>>>> the hard requirement for a SSLEngine is new. We're getting >>>>> there :) >>>> >>>>> With FIPS certification requirements though, we can't use >>>>> JSSE (since it isn't FIPS certified). > > I was pretty sure Java's crypto implementation was FIPS certified. > I haven't looked in a while, but a quick good search turned up: > > https://www2.cs.duke.edu/csed/java/jdk1.7/technotes/guides/security/js se > > /FIPS.html > > and > > https://docs.oracle.com/javase/6/docs/technotes/guides/security/enhanc em > > ents.html > > That last one suggests that Oracle Java already contains a JSSE > provider which uses NSS. DO I misunderstad? > >> Also relevant: > >> https://docs.oracle.com/javase/8/docs/technotes/guides/security/p11gu ide.html#NSS > >> > > >> Ugh... This is where I confess I'm not an expert on FIPS stuff. >> Here's my best understanding: > >> We're talking about the same provider (that's good!). The subtle >> point I think you're missing is that FIPS isn't the _default_ >> configuration and it imposes more restrictions when in FIPS mode. >> It also needs special configuration (which in RHEL looks like): > >> fips.provider.1=sun.security.pkcs11.SunPKCS11 >> ${java.home}/lib/security/nss.fips.cfg >> fips.provider.2=sun.security.provider.Sun >> fips.provider.3=sun.security.ec.SunEC >> fips.provider.4=com.sun.net.ssl.internal.ssl.Provider >> SunPKCS11-NSS-FIPS <--- "FIPS" > >> In comparison, the default (non-FIPS mode providers) are: > >> security.provider.1=sun.security.provider.Sun >> security.provider.2=sun.security.rsa.SunRsaSign >> security.provider.3=sun.security.ec.SunEC >> security.provider.4=com.sun.net.ssl.internal.ssl.Provider <--- >> "Non-FIPS" security.provider.5=com.sun.crypto.provider.SunJCE >> security.provider.6=sun.security.jgss.SunProvider >> security.provider.7=com.sun.security.sasl.Provider >> security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI >> security.provider.9=sun.security.smartcardio.SunPCSC >> security.provider.10=sun.security.pkcs11.SunPKCS11 >> ${java.home}/lib/security/nss.cfg > >> Now, where my understanding gets more fuzzy... > > >> RHEL ships OpenJDK and upstream itself doesn't do the FIPS >> certification; it is Red Hat that does the certification of NSS >> on RHEL. JDK itself, under the FIPS configuration above, now does >> no crypto. All the crypto is done via PKCS#11 to NSS. So there's >> nothing (in OpenJDK) to certify any more. Unlike in the non-FIPS >> mode above, there's lots of crypto that would need to be >> certified. Aah, okay. This makes more sense to me, now. At this point, the JSSE provider is just a pass-through to NSS, just like the JSSE wrapper for OpenSSL. >> Oracle, in their JDK, might ship their own PKCS#11 provider >> and/or certify NSS, I don't know. I know IBM ships their own FIPS >> crypto provider, and probably the next most popular is >> BouncyCastle, but I thought that might be losing popularity >> because they might've stopped FIPS certifying it, who knows. IMHO FIPS is useless at this point unless you are required to use a FIPS-certified module, which many people are unfortunately required to use. The reason I think it's useless is because FIPS mandates the use of certain algorithms and some of them shouldn't be used. >> But my key point is there's no "OpenJDK FIPS Certificate" on the >> FIPS page: > >> https://csrc.nist.gov/projects/cryptographic-module-validation-progra m/validated-modules/search > >> (Search Java) > >> But there is one for NSS ("NSS" - and then its "NSS Cryptographic >> Module" on the results page). Yep. I saw that Oracle has FIPS-certified modules, but they aren't a part of the standard Java distribution (and certainly not part of OpenJDK). I guess you get what you pay for. >> So yes, the SunJSSE provider is still used but the key >> differences are: > >> 1) The crypto is done by SunPKCS11-NSS-FIPS provider, 2) The NSS >> that is used is FIPS certified, 3) It places additional >> restrictions (more about that below). > >> If you don't really care 2 (the token "Certificate") and just >> want to test stuff, 1 and 3 work on any distro (e.g., Ubuntu and >> Fedora), and NSS everywhere understands FIPS mode. > >> Also the SunPKCS11 / SunPKCS11-NSS-FIPS providers are limited to >> Linux, iirc. Well, who would bother with NSS on Windows? >> That's how I understand the above and no, you don't seem to >> misunderstand. :) > >>>>> OpenJDK supports NSS via the Sun-PKCS11 provider and has a >>>>> SSLEngine implementation under the SunJSSE provider. It has >>>>> some restrictions on it that means it can't be used from >>>>> Tomcat's JSSE adapter just by swapping out the provider >>>>> name (for one, wrapping the KeyManager like Tomcat does >>>>> isn't allowed). > > Hmm. Maybe there's a better way to do it, then, and you don't have > to write so much code. What's the problem with wrapping a > KeyManager? > >> That's an RFE I've filed internally against the JDK team. To me, >> it makes no sense. It fails now with Tomcat's JSSEImplementation >> : > >> Dec 06, 2019 10:20:04 AM org.apache.catalina.util.LifecycleBase >> handleSubClassException SEVERE: Failed to initialize component >> [Connector[HTTP/1.1-8443]] >> org.apache.catalina.LifecycleException: Protocol handler >> initialization failed at >> org.apache.catalina.connector.Connector.initInternal(Connector.java:9 35) >> >> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) >> at >> org.apache.catalina.core.StandardService.initInternal(StandardService .java:530) >> >> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) >> at >> org.apache.catalina.core.StandardServer.initInternal(StandardServer.j ava:852) >> >> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) >> at org.apache.catalina.startup.Catalina.load(Catalina.java:633) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:62) >> >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) at >> org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:306) at >> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:491) >> Caused by: java.lang.IllegalArgumentException: FIPS mode: only >> SunJSSE KeyManagers may be used at >> org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(Abst ractJsseEndpoint.java:114) >> >> at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJs seEndpoint.java:85) >> at >> org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:216) >> >> at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1 043) >> at >> org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:540) >> >> at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Proto col.java:74) >> at >> org.apache.catalina.connector.Connector.initInternal(Connector.java:9 32) >> >> ... 12 more >> Caused by: java.security.KeyManagementException: FIPS mode: only >> SunJSSE KeyManagers may be used at >> sun.security.ssl.SSLContextImpl.chooseKeyManager(SSLContextImpl.java: 154) >> >> at sun.security.ssl.SSLContextImpl.engineInit(SSLContextImpl.java:71) >> at javax.net.ssl.SSLContext.init(SSLContext.java:282) at >> org.apache.tomcat.util.net.jsse.JSSESSLContext.init(JSSESSLContext.ja va:53) >> >> at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(Abstrac tJsseEndpoint.java:112) >> ... 18 more > Aha. "What's wrong with wrapping the KeyManager?" Answer: it doesn't work: KeyManagementException I guess they don't want any funny business. I'll have to look at how the KeyManager is built/wrapped. I didn't write that code so I'm not too familiar with it. I know we need to wave our hands around certain things due to the flexibility Tomcat allows for certificate-selection, etc. When you get that error, is the object of type org/apache/tomcat/util/net/jsse/JSSEKeyManager? I wonder if we are jumping the gun and always using that KeyManager even when it's not strictly necessary. >> My config roughly looks like: > >> <Connector name="Secure" port="8443" scheme="https" secure="true" >> SSLEnabled="true" keyAlias="localhost" >> keystorePass="keystorePass" keystoreType="PKCS11" >> keystoreProvider="SunPKCS11-NSS-FIPS" >> keyManagerAlgorithm="SunX509" truststoreAlgorithm="PKCS11" >> truststoreProvider="SunJSSE" truststoreType="SunX509" >> truststorePassword="truststorePassword" SSLProtocol="TLSv1.2" >> maxThreads="1" /> > >> (Yeah, that's messy -- but it illustrates the point -- you can >> probably remove a number of those elements -- *including* >> keyAlias, and it'll still fail). > >> This is because, for all KeyManagers, Tomcat wraps it its own >> KeyManager: > >> https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/u til/net/SSLUtilBase.java#L378 > >> Probably the minimum untested change that I'd have proposed >> would be to wrap it in an if block like you have ealier: > >> https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/u til/net/SSLUtilBase.java#L195 Sounds >> like the same as my conclusion. I wouldn't want to make that change myself without understanding the other implications. >> Now the core of the _why_ is because that's how NSS behaves. In >> FIPS mode, NSS doesn't let you import keys. You can either have >> keys in a NSS DB, or you can wrap/unwrap them into one (or the >> temporary, in-memory key store). But you can't just call >> PK11_ImportSymKey (or the related function for private keys) -- >> they'll fail! > >> That's because NSS was a FIPS level 2 certified provider, unlike >> say, OpenSSL which only undergoes FIPS level 1 certification. >> Some of those restrictions that level 2 places are around key >> management and handling, including where they're sourced from and >> how they're protected in transit. Gotcha. Yeah, any idiot can "openssl genkey" and do whatever they want with the key including publishing it on their web site. >> One of the RFEs I've been discussing with our JDK team have been >> to loosen the restrictions of the SunJSSE provider in FIPS mode. >> Restricting the origin of the KeyManager (while having good >> intentions of preventing the above) means that some otherwise >> valid use cases -- like selecting a single key alias from the >> server.xml config -- won't work. Really they should only care >> that the key used is from the SunPKCS11-NSS-FIPS provider, they >> don't care how they got it and from which KeyManager it came >> from. Well, if it didn't come from a SunJSSE KeyManager, then it can't prove that it wasn't tampered-with -- or even completely replaced, which would effectively be an "imported key". So it kind of does make sense. >> Those changes are still under discussion. > >> --- > >> There's also about ~4 or 5 other weird bugs in this SunJSSE FIPS >> interface that make it less than usable or cause it to fail to >> handshake in FIPS mode, but they don't really concern Tomcat. > >> Those are why I'm not proposing a PR yet. :-) > > >>>>> I was trying to see if the JDK's SSLEngine would work for >>>>> us. On RHEL, we FIPS certify our shipped NSS release so the >>>>> JDK's Sun-PKCS11-NSS-FIPS provider is FIPS certified >>>>> transitively (since it does no crypto... :). Since I'm not >>>>> a Tomcat developer (and am more familiar with TomcatJSS), I >>>>> stubbed out a new SSLImplementation that uses SunJSSE so I >>>>> could bypass the parts of the Tomcat JSSEImplementation >>>>> that don't work with the JDK SunJSSE provider. > > That sounds like it would work, but also sound like a huge amount > of plumbing. What can you tell me about the way Tomcat uses the > JSSE API that precludes FIPS support from the existing tools -- > including your own JSS? > >> Mostly that's discussed above w.r.t. SunJSSE + >> SunPKCS11-NSS-FIPS. > > >> Why JSS? > >> Dogtag is a self-hosted CA using NSS and came out of Netscape's >> certificate offerings before landing at Red Hat. We support RSA >> and ECDSA certificates and also have strong HSM support (thanks >> to NSS). In part, this means we can install a CA and have both >> the CA and the SSL server certificates for the CA interface never >> leave the HSM (or the NSS DB). "We are a CA" is basically all the explanation I needed :) >> Since (especially in FIPS mode) HSMs generally frown upon >> extracting keys, that makes using say, BouncyCastle or Tomcat's >> OpenSSL adapter harder, since we'd need to configure and support >> HSM access from both NSS and $PROVIDER simultaneously. Far easier >> to use a PKCS#11 provider with NSS (where we've already set up >> the HSM and NSS DB). That leaves well, JSS or >> SunJSSE+SunPKCS11-NSS. > >> For internal reasons, the FIPS mode SunJSSE provider wasn't >> well... supported until relatively recently. So for most of our >> time, we've been fine using JSS (via TomcatJSS's BIO adapter on >> Tomcat < 8.5). Its only recently that we've needed an NIO adapter >> (so JSS hasn't yet gotten a SSLEngine) -- and only recently that >> we've considered SunJSSE again. > > >> So: > >> 1) We can't use Tomcat's JSSE API because we don't have a >> SSLEngine in JSS yet, 2) We can't use SunJSSE+SunPKCS11-NSS-FIPS >> because it doesn't work. > >> Currently we ignore FIPS (on Tomcat >= 8.5) and use the regular, >> non-FIPS SunJSSE. The only thing we really use TomcatJSS for is >> to initialize JSS early enough in the Tomcat startup sequence >> (before the app starts up) -- so that we can use JSS as the >> keystore for Tomcat and SunJSSE (which the non-FIPS mode adapter >> is happy to do). > >> For the rest of the app, we do need JSS though. In part because >> its one of the few Java providers that provide CMAC and (soon) >> KBKDF, but we also use it for all our certificate issuance >> operations. Can I just say that "KBKDF" is a hilarious concept to me? Now we just need a KBPDF and we can go full-circle. >> I'd like to get to the point where either work but both need a >> few bug fixes and/or RFEs first. :-) > > > >>>>> However, currently on RHEL the SunJSSE provider is a bit >>>>> broken, so this FIPSImplementation isn't really useful. If >>>>> you're interested, I'd like to get Tomcat to a place where >>>>> it supports FIPS mode JDK without the need for a separate >>>>> implementation, but first I need a working FIPS-mode JDK >>>>> (currently it doesn't handshake)... ...and we're probably >>>>> going the route of finishing JSS anyways. > > Hmm. Is your goal to get a working JSSE+FIPS implementation > specifically with JSS or do you just want to get FIPS working with > Tomcat? The OpenSSL provider can be put into FIPS mode, provided > that it was build with FIPS support of course. > >> I think I covered OpenSSL above. We have considered it and it >> would be possible. But we'd prefer a NSS backed provider. > >> So either JSSE+SunPKCS11-NSS-FIPS or JSSE+/-JSS. :-) Understood. Mark will probably read this novel and say "oh, that's easy" and just implement it. Having not written the original code and not understanding the various intricacies of why the replacement KeyManager is being used, etc., I wouldn't want to break things for large numbers of people. Have you tried hacking Tomcat to just never wrap the KeyManager and use a stock JSSE one? Does it work in that case? - -chris -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3qksMACgkQHPApP6U8 pFj8bg/8C50dYqQIv7mpvJ3ntdChjfn2FQdcNrYqpbOSrREaXaeSHShGAoHyjDnN sP8UbVQ8jyie/RC1a8l2wKPozFMv21n6p8q6USU8N0g0PC2oYidJC+FqQx0MwSto oqrXYxAR+wcY/HZXmFH56xOG2Lbw9AcBO7NUgaRGFapoYvYXiAAh9x61rRrTXjOX 3enJx3L7r40F4b8hug8o42S87TMonWoURhBoIx1grAx29jq8eBymy/Hp+LuyD5GH PYZ3ICgCqg2vSqMeuYRbt+WswwPiWMfuFKcj5Kd7F63sQUValhEmLMPs7Rj/ZukI Ow5d7Han8nmSTtslzCIOCSOGj2eVTcqZOlGW9fs9tpSfKFVbdKFzFm7uGSQApv0j h8l08RK8kMBS567K8VGl5VHD9vGiTxfqME7e5SzRmrS/PfYnAAF40dmbulJbGHKV V7HupYASkIzRj2/+yht3h0auZUr/qe86LBGNUbwzSus9Mp++VfKNLmNxU26C6VlU G4SUQcPIIzgjmcSsW/asR3EvbjPdQoTAcWacFcXvWNqsiu9D3Aotb0VwQR8OnPSL dMQrXuHS+ynsc8Ywe40+x/g1i+HOSJPmfSYOI2QYCPdF4yMz/sNDdNQpMuvnlbot mvgPczhTRBy8pJiPwixwxuy93YmkblrGkJV9fUPfkLi913cPqIA= =HJ2y -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
