I guess I will have to investigate the RHEL 7.3 entropy issue separately (possibly as hobby project) and look for other options to make progress.
I still find it odd that something related to randomness (entropy generation) is so consistent (the slowness is equally slow or more on multiple RHEL 7.3 systems I have, maybe I need to look for machines from different data center or a physical 7.3 server). And yes, the 10 year certificate validity is just for testing purposes. 😊 Thank you for your inputs. Indeed helpful in evaluating our choices. Thanks, Amit -----Original Message----- From: Christopher Schultz <ch...@christopherschultz.net> Sent: Wednesday, November 25, 2020 9:42 PM To: users@tomcat.apache.org Subject: Re: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3 Amit, On 11/25/20 12:40, Amit Pande wrote: > Thank you Chris for the inputs. Admittedly, I didn’t know the internals of > Sun JCE/JSSE vs BC JCE. > > Pasting sample output and it indeed is taking minutes on RHEL 7.3. Not sure > if I am indeed missing some trick here. > > > RHEL 7.3 --------------- > > test-host-1:~ # date ; /usr/openv/java/jre/bin/keytool -providerpath > /root/Downloads/bc-fips-1.0.2.jar -providerclass > org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair > -keyalg RSA -keypass "Test123" -validity 3650 -dname > "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" > -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date > > Wed Nov 25 10:52:56 CST 2020 (START TIME) > > Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) > with a validity of 3,650 days > for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org [Storing > /tmp/test_bc.bcfks] > > Wed Nov 25 10:58:11 CST 2020 (END TIME) (Almost 6 minutes) > > test-host-1:~ # uname -a > Linux test-host-1 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT > 2016 x86_64 x86_64 x86_64 GNU/Linux test-host-1:~ # cat > /etc/redhat-release Red Hat Enterprise Linux Server release 7.3 > (Maipo) test-host-1:~ # test-host-1:~ # /usr/openv/java/jre/bin/java > -version java version "1.8.0_271" > Java(TM) SE Runtime Environment (build 1.8.0_271-b09) Java HotSpot(TM) > 64-Bit Server VM (build 25.271-b09, mixed mode) test-host-1:~ # > > > RHEL 7.2 ------------- > > [root@test-host-2 ~]# date ; /usr/openv/java/jre/bin/keytool > -providerpath /root/bc-fips-1.0.2.jar -providerclass > org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair > -keyalg RSA -keypass "Test123" -validity 3650 -dname > "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" > -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date > > Wed Nov 25 11:20:06 CST 2020 (START TIME) > > Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) > with a validity of 3,650 days > for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org [Storing > /tmp/test_bc.bcfks] > > Wed Nov 25 11:20:10 CST 2020 (END TIME) (Almost 4 seconds) > > [root@test-host-2 ~]# uname -a > Linux test-host-2 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 > EDT 2015 x86_64 x86_64 x86_64 GNU/Linux > [root@test-host-2 ~]# cat /etc/redhat-release Red Hat Enterprise Linux > Server release 7.2 (Maipo) > [root@test-host-2 ~]# > > Red Hat Enterprise Linux Server release 7.2 (Maipo) > [root@test-host-2 ~]# /usr/openv/java/jre/bin/java -version java > version "1.8.0_261" > Java(TM) SE Runtime Environment (build 1.8.0_261-b33) Java HotSpot(TM) > 64-Bit Server VM (build 25.261-b33, mixed mode) > [root@test-host-2 ~]# > > > Since the keytool is literally taking minutes (specifically on RHEL > 7.3 as you can see above), enabling FIPS OOTB has become a challenge > as it has resulted in some our test suites timing out. What I see is that on two servers were everything is the same (except for the RHEL version... weird; RHEL 7.2 and 7.3 have identical kernel versions?), two identical command-lines take different amounts of time. In both cases, you are using BC with FIPS enabled. The reason is that one of the servers is starved for entropy and the other one is not. Some operations require a LOT of entropy (like generating an RSA key) and others require less entropy, but each of those operations consumes some of the system's entropy as they occur. When you run out, processes block waiting for it, and it's only created slowly as random hardware events are sampled to generate that "high quality randomness". So it can take a while. Some servers have hardware that generates entropy faster, some servers have usage profiles that cause certain hardware events to occur more often and therefore keep the entropy pool full. Others are constantly starved for entropy. > Tomcat is very much a core of our product and configuration and > starting Tomcat in timely manner (FIPS or no FIPS) has been a critical > requirement. And now, with this issue, test suites timing out, hard to > convince to get the suite timeouts increased. 😊 Well, you don't have too many choices. You either fix your entropy problem (which isn't exactly straightforward), or you switch to a process which does not require so much entropy (which may mean disabling FIPS), or you allow for more time to complete those operations. Is the generation of RSA keys a part of your test suite? Maybe you can re-write your test suite to only generate the certificate if it's missing and/or is getting close to expiration. If you generate a new RSA key for each run of your test quite, you are certainly going to drain your entropy pool quickly and repeatedly. BTW, I'm assuming that you are using (or at least just /testing/) FIPS mode because you need a "high security" environment. Nothing about a 2048-bit RSA key that is valid for 10 years should be considered "high security". -chris > -----Original Message----- > From: Christopher Schultz <ch...@christopherschultz.net> > Sent: Wednesday, November 25, 2020 10:33 AM > To: users@tomcat.apache.org > Subject: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3 > > Amit, > > On 11/24/20 11:21, Amit Pande wrote: >> Probably not directly related to Tomcat but still sharing. Advanced >> apologies for that. >> >> I am using bouncy castle FIPS library and observed that specifically >> on RHEL 7.3, the library usage is causing tremendous slowness. > Note that BC is pure-Java, so you should pretty much expect "tremendous > slowness". The reason that the Sun JCE/JSSE provider is (or can be) very fast > is because it's using both native code (which is only marginally faster than > well-JIT'ed Java bytecode, if at all) and also hardware support when it's > available. There is no way that I know of for BC's pure-Java crypto > implementations to use hardware acceleration. AES on silicon is *much* faster > than AES in software. > >> e.g. below key tool command taking several minutes to finish. > > Uhh... that's not software versus hardware. You might expect an order of > magnitude difference between BC and hardware-accelerated SunJCE. But minutes > to generate a 2048-bit RSA key? Something is wrong. > >> keytool -providerpath /root/Downloads/bc-fips-1.0.2.jar >> -providerclass >> org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair >> -keyalg RSA -keypass "Test123" -validity 3650 -dname "CN=Test_HostName, >> ou=My Org Unit, o=My Org" -storepass "Test123" >> -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias test_entry >> >> However, when I add the JVM option >> -J-Djava.security.egd=file:/dev/./urandom instead of the default >> /dev/random, all problems go away. > Classic entropy problem. > >> It's rather strange that multiple 7.3 RHEL systems are showing this >> behavior very consistently. RHEL 7.2, 8.0 seem just fine in my >> testing. No issues when using /dev/random along with JKS type key >> stores (including RHEL 7.3). >> >> Even Tomcat is flagging the slowness to generate the secure random >> numbers. >> >> "17-Nov-2020 19:24:59.142 WARNING [Catalina-utility-2] >> org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation >> of SecureRandom instance for session ID generation using [SHA1PRNG] took >> [68,745] milliseconds." >> >> Apparently, according to FIPS experts we should not use /dev/./urandom. > > Correct. FIPS assumes that 100% of your cryptographic operations are > of the *utmost highly-classified state-secret-style > nuclear-launch-code > flashing-purple* security level. > > Sometimes you just don't need to have perfect entropy. > > For long-lived keys, really good entropy is a Good Thing. So to generate your > RSA keys, I'd say go ahead and use the "high quality entropy source". But for > randomness for session id generation, and for temporary, bulk encryption TLS > session keys, /dev/urandom is perfectly fine. > >> So, while I continue to explore further, sharing this in case any one >> has seen this behavior or be aware of such potential issue as it may >> tie in to their Tomcat (or web apps to be deployed in Tomcat). > You might be able to justify using /dev/urandom (which must be spelled > /dev/./urandom to force Java to actually use it) in your web application, but > I would go ahead and let keytool take as long as it needs. > > -chris > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
