RE: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Thank you Stefan, Chris for the inputs. As I understood from our security experts, there is no moving away from /dev/random (anything else isn't (strongly) FIPS compliant) for us. Thanks, Amit -Original Message- From: Christopher Schultz Sent: Wednesday, December 2, 2020 10:29 AM To: users@tomcat.apache.org Subject: Re: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3 Stefan, On 11/30/20 19:17, Stefan Mayr wrote: > Hi, > > Am 30.11.2020 um 17:09 schrieb Amit Pande: >> I guess I will have to investigate the RHEL 7.3 entropy issue separately >> (possibly as hobby project) and look for other options to make progress. >> >> I still find it odd that something related to randomness (entropy >> generation) is so consistent (the slowness is equally slow or more on >> multiple RHEL 7.3 systems I have, maybe I need to look for machines from >> different data center or a physical 7.3 server). >> >> And yes, the 10 year certificate validity is just for testing >> purposes. 😊 >> >> Thank you for your inputs. Indeed helpful in evaluating our choices. >> >> Thanks, >> Amit > > you might have a look at rng-tools (rngd) or haveged to boost your > entropy pool. > > We use haveged in a VMware virtualized environment and this reduces a > plain tomcat startup from multiple minutes to just a few secondes. > > I think Red Hat preferes rngd but there should be some articles on > access.redhat.com to help you depending on the used hypervisor. I would think long and hard about whether or not you want to use any of these tools. There are already ways to get "a lot of entropy really quickly" from the Linux kernel; specifically, /dev/urandom. The whole point of both /dev/random and /dev/urandom existing side by side is so that the application can pick whether it wants "high quality entropy" (by using /dev/random) or "good enough randomness" (by using /dev/urandom). Tools like haveged and rngd basically make /dev/random behave like /dev/urandom so the application can never have "high quality entropy" even when it asks for it. Have a look at this discussion on security.stackexchange to get you started down the path to paranoia: https://security.stackexchange.com/questions/34523 My question has always been "if these things are both safe and a good idea, why does the Linux kernel not implement them directly?" There must be a reason why the kernel devs have decided not to "speed up" /dev/random using the techniques used by both haveged and rngd. Maybe their argument is essentially "you can always just use haveged/rngd" but my guess is there is a more fundamental reason for not adopting these techniques directly in the kernel. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Stefan, On 11/30/20 19:17, Stefan Mayr wrote: Hi, Am 30.11.2020 um 17:09 schrieb Amit Pande: I guess I will have to investigate the RHEL 7.3 entropy issue separately (possibly as hobby project) and look for other options to make progress. I still find it odd that something related to randomness (entropy generation) is so consistent (the slowness is equally slow or more on multiple RHEL 7.3 systems I have, maybe I need to look for machines from different data center or a physical 7.3 server). And yes, the 10 year certificate validity is just for testing purposes. 😊 Thank you for your inputs. Indeed helpful in evaluating our choices. Thanks, Amit you might have a look at rng-tools (rngd) or haveged to boost your entropy pool. We use haveged in a VMware virtualized environment and this reduces a plain tomcat startup from multiple minutes to just a few secondes. I think Red Hat preferes rngd but there should be some articles on access.redhat.com to help you depending on the used hypervisor. I would think long and hard about whether or not you want to use any of these tools. There are already ways to get "a lot of entropy really quickly" from the Linux kernel; specifically, /dev/urandom. The whole point of both /dev/random and /dev/urandom existing side by side is so that the application can pick whether it wants "high quality entropy" (by using /dev/random) or "good enough randomness" (by using /dev/urandom). Tools like haveged and rngd basically make /dev/random behave like /dev/urandom so the application can never have "high quality entropy" even when it asks for it. Have a look at this discussion on security.stackexchange to get you started down the path to paranoia: https://security.stackexchange.com/questions/34523 My question has always been "if these things are both safe and a good idea, why does the Linux kernel not implement them directly?" There must be a reason why the kernel devs have decided not to "speed up" /dev/random using the techniques used by both haveged and rngd. Maybe their argument is essentially "you can always just use haveged/rngd" but my guess is there is a more fundamental reason for not adopting these techniques directly in the kernel. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Hi, Am 30.11.2020 um 17:09 schrieb Amit Pande: > I guess I will have to investigate the RHEL 7.3 entropy issue separately > (possibly as hobby project) and look for other options to make progress. > > I still find it odd that something related to randomness (entropy generation) > is so consistent (the slowness is equally slow or more on multiple RHEL 7.3 > systems I have, maybe I need to look for machines from different data center > or a physical 7.3 server). > > And yes, the 10 year certificate validity is just for testing purposes. 😊 > > Thank you for your inputs. Indeed helpful in evaluating our choices. > > Thanks, > Amit you might have a look at rng-tools (rngd) or haveged to boost your entropy pool. We use haveged in a VMware virtualized environment and this reduces a plain tomcat startup from multiple minutes to just a few secondes. I think Red Hat preferes rngd but there should be some articles on access.redhat.com to help you depending on the used hypervisor. Regards, Stefan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
I guess I will have to investigate the RHEL 7.3 entropy issue separately (possibly as hobby project) and look for other options to make progress. I still find it odd that something related to randomness (entropy generation) is so consistent (the slowness is equally slow or more on multiple RHEL 7.3 systems I have, maybe I need to look for machines from different data center or a physical 7.3 server). And yes, the 10 year certificate validity is just for testing purposes. 😊 Thank you for your inputs. Indeed helpful in evaluating our choices. Thanks, Amit -Original Message- From: Christopher Schultz Sent: Wednesday, November 25, 2020 9:42 PM To: users@tomcat.apache.org Subject: Re: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3 Amit, On 11/25/20 12:40, Amit Pande wrote: > Thank you Chris for the inputs. Admittedly, I didn’t know the internals of > Sun JCE/JSSE vs BC JCE. > > Pasting sample output and it indeed is taking minutes on RHEL 7.3. Not sure > if I am indeed missing some trick here. > > > RHEL 7.3 --- > > test-host-1:~ # date ; /usr/openv/java/jre/bin/keytool -providerpath > /root/Downloads/bc-fips-1.0.2.jar -providerclass > org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair > -keyalg RSA -keypass "Test123" -validity 3650 -dname > "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" > -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date > > Wed Nov 25 10:52:56 CST 2020 (START TIME) > > Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) > with a validity of 3,650 days > for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org [Storing > /tmp/test_bc.bcfks] > > Wed Nov 25 10:58:11 CST 2020 (END TIME) (Almost 6 minutes) > > test-host-1:~ # uname -a > Linux test-host-1 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT > 2016 x86_64 x86_64 x86_64 GNU/Linux test-host-1:~ # cat > /etc/redhat-release Red Hat Enterprise Linux Server release 7.3 > (Maipo) test-host-1:~ # test-host-1:~ # /usr/openv/java/jre/bin/java > -version java version "1.8.0_271" > Java(TM) SE Runtime Environment (build 1.8.0_271-b09) Java HotSpot(TM) > 64-Bit Server VM (build 25.271-b09, mixed mode) test-host-1:~ # > > > RHEL 7.2 - > > [root@test-host-2 ~]# date ; /usr/openv/java/jre/bin/keytool > -providerpath /root/bc-fips-1.0.2.jar -providerclass > org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair > -keyalg RSA -keypass "Test123" -validity 3650 -dname > "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" > -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date > > Wed Nov 25 11:20:06 CST 2020 (START TIME) > > Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) > with a validity of 3,650 days > for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org [Storing > /tmp/test_bc.bcfks] > > Wed Nov 25 11:20:10 CST 2020 (END TIME)(Almost 4 seconds) > > [root@test-host-2 ~]# uname -a > Linux test-host-2 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 > EDT 2015 x86_64 x86_64 x86_64 GNU/Linux > [root@test-host-2 ~]# cat /etc/redhat-release Red Hat Enterprise Linux > Server release 7.2 (Maipo) > [root@test-host-2 ~]# > > Red Hat Enterprise Linux Server release 7.2 (Maipo) > [root@test-host-2 ~]# /usr/openv/java/jre/bin/java -version java > version "1.8.0_261" > Java(TM) SE Runtime Environment (build 1.8.0_261-b33) Java HotSpot(TM) > 64-Bit Server VM (build 25.261-b33, mixed mode) > [root@test-host-2 ~]# > > > Since the keytool is literally taking minutes (specifically on RHEL > 7.3 as you can see above), enabling FIPS OOTB has become a challenge > as it has resulted in some our test suites timing out. What I see is that on two servers were everything is the same (except for the RHEL version... weird; RHEL 7.2 and 7.3 have identical kernel versions?), two identical command-lines take different amounts of time. In both cases, you are using BC with FIPS enabled. The reason is that one of the servers is starved for entropy and the other one is not. Some operations require a LOT of entropy (like generating an RSA key) and others require less entropy, but each of those operations consumes some of the system's entropy as they occur. When you run out, processes block waiting for it, and it's only created slowly as random hardware events are sampled to generate that "high quality randomness". So it can take a while. Some servers have hardware that generates entropy faster, some servers have usage profiles that cause certain hardware events to occur m
Re: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Amit, On 11/25/20 12:40, Amit Pande wrote: Thank you Chris for the inputs. Admittedly, I didn’t know the internals of Sun JCE/JSSE vs BC JCE. Pasting sample output and it indeed is taking minutes on RHEL 7.3. Not sure if I am indeed missing some trick here. RHEL 7.3 --- test-host-1:~ # date ; /usr/openv/java/jre/bin/keytool -providerpath /root/Downloads/bc-fips-1.0.2.jar -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair -keyalg RSA -keypass "Test123" -validity 3650 -dname "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date Wed Nov 25 10:52:56 CST 2020 (START TIME) Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 3,650 days for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org [Storing /tmp/test_bc.bcfks] Wed Nov 25 10:58:11 CST 2020 (END TIME) (Almost 6 minutes) test-host-1:~ # uname -a Linux test-host-1 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux test-host-1:~ # cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.3 (Maipo) test-host-1:~ # test-host-1:~ # /usr/openv/java/jre/bin/java -version java version "1.8.0_271" Java(TM) SE Runtime Environment (build 1.8.0_271-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode) test-host-1:~ # RHEL 7.2 - [root@test-host-2 ~]# date ; /usr/openv/java/jre/bin/keytool -providerpath /root/bc-fips-1.0.2.jar -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair -keyalg RSA -keypass "Test123" -validity 3650 -dname "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date Wed Nov 25 11:20:06 CST 2020 (START TIME) Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 3,650 days for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org [Storing /tmp/test_bc.bcfks] Wed Nov 25 11:20:10 CST 2020 (END TIME)(Almost 4 seconds) [root@test-host-2 ~]# uname -a Linux test-host-2 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux [root@test-host-2 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.2 (Maipo) [root@test-host-2 ~]# Red Hat Enterprise Linux Server release 7.2 (Maipo) [root@test-host-2 ~]# /usr/openv/java/jre/bin/java -version java version "1.8.0_261" Java(TM) SE Runtime Environment (build 1.8.0_261-b33) Java HotSpot(TM) 64-Bit Server VM (build 25.261-b33, mixed mode) [root@test-host-2 ~]# Since the keytool is literally taking minutes (specifically on RHEL 7.3 as you can see above), enabling FIPS OOTB has become a challenge as it has resulted in some our test suites timing out. What I see is that on two servers were everything is the same (except for the RHEL version... weird; RHEL 7.2 and 7.3 have identical kernel versions?), two identical command-lines take different amounts of time. In both cases, you are using BC with FIPS enabled. The reason is that one of the servers is starved for entropy and the other one is not. Some operations require a LOT of entropy (like generating an RSA key) and others require less entropy, but each of those operations consumes some of the system's entropy as they occur. When you run out, processes block waiting for it, and it's only created slowly as random hardware events are sampled to generate that "high quality randomness". So it can take a while. Some servers have hardware that generates entropy faster, some servers have usage profiles that cause certain hardware events to occur more often and therefore keep the entropy pool full. Others are constantly starved for entropy. Tomcat is very much a core of our product and configuration and starting Tomcat in timely manner (FIPS or no FIPS) has been a critical requirement. And now, with this issue, test suites timing out, hard to convince to get the suite timeouts increased. 😊 Well, you don't have too many choices. You either fix your entropy problem (which isn't exactly straightforward), or you switch to a process which does not require so much entropy (which may mean disabling FIPS), or you allow for more time to complete those operations. Is the generation of RSA keys a part of your test suite? Maybe you can re-write your test suite to only generate the certificate if it's missing and/or is getting close to expiration. If you generate a new RSA key for each run of your test quite, you are certainly going to drain your entropy pool quickly and repeatedly. BTW, I'm assuming that you are using (or at least just /testing/) FIPS mode because you need a "high security" environment. Nothing about a 2048-bit RSA key that is valid for 10 years should be considered "high security". -chris -Original Message- From: Christopher Schult
RE: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Thank you Chris for the inputs. Admittedly, I didn’t know the internals of Sun JCE/JSSE vs BC JCE. Pasting sample output and it indeed is taking minutes on RHEL 7.3. Not sure if I am indeed missing some trick here. RHEL 7.3 --- test-host-1:~ # date ; /usr/openv/java/jre/bin/keytool -providerpath /root/Downloads/bc-fips-1.0.2.jar -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair -keyalg RSA -keypass "Test123" -validity 3650 -dname "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date Wed Nov 25 10:52:56 CST 2020 (START TIME) Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 3,650 days for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org [Storing /tmp/test_bc.bcfks] Wed Nov 25 10:58:11 CST 2020 (END TIME) (Almost 6 minutes) test-host-1:~ # uname -a Linux test-host-1 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux test-host-1:~ # cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.3 (Maipo) test-host-1:~ # test-host-1:~ # /usr/openv/java/jre/bin/java -version java version "1.8.0_271" Java(TM) SE Runtime Environment (build 1.8.0_271-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode) test-host-1:~ # RHEL 7.2 - [root@test-host-2 ~]# date ; /usr/openv/java/jre/bin/keytool -providerpath /root/bc-fips-1.0.2.jar -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair -keyalg RSA -keypass "Test123" -validity 3650 -dname "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date Wed Nov 25 11:20:06 CST 2020 (START TIME) Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 3,650 days for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org [Storing /tmp/test_bc.bcfks] Wed Nov 25 11:20:10 CST 2020 (END TIME)(Almost 4 seconds) [root@test-host-2 ~]# uname -a Linux test-host-2 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux [root@test-host-2 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.2 (Maipo) [root@test-host-2 ~]# Red Hat Enterprise Linux Server release 7.2 (Maipo) [root@test-host-2 ~]# /usr/openv/java/jre/bin/java -version java version "1.8.0_261" Java(TM) SE Runtime Environment (build 1.8.0_261-b33) Java HotSpot(TM) 64-Bit Server VM (build 25.261-b33, mixed mode) [root@test-host-2 ~]# Since the keytool is literally taking minutes (specifically on RHEL 7.3 as you can see above), enabling FIPS OOTB has become a challenge as it has resulted in some our test suites timing out. Tomcat is very much a core of our product and configuration and starting Tomcat in timely manner (FIPS or no FIPS) has been a critical requirement. And now, with this issue, test suites timing out, hard to convince to get the suite timeouts increased. 😊 Thanks, Amit -Original Message- From: Christopher Schultz Sent: Wednesday, November 25, 2020 10:33 AM To: users@tomcat.apache.org Subject: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3 Amit, On 11/24/20 11:21, Amit Pande wrote: > Probably not directly related to Tomcat but still sharing. Advanced > apologies for that. > > I am using bouncy castle FIPS library and observed that specifically > on RHEL 7.3, the library usage is causing tremendous slowness. Note that BC is pure-Java, so you should pretty much expect "tremendous slowness". The reason that the Sun JCE/JSSE provider is (or can be) very fast is because it's using both native code (which is only marginally faster than well-JIT'ed Java bytecode, if at all) and also hardware support when it's available. There is no way that I know of for BC's pure-Java crypto implementations to use hardware acceleration. AES on silicon is *much* faster than AES in software. > e.g. below key tool command taking several minutes to finish. Uhh... that's not software versus hardware. You might expect an order of magnitude difference between BC and hardware-accelerated SunJCE. But minutes to generate a 2048-bit RSA key? Something is wrong. > keytool -providerpath /root/Downloads/bc-fips-1.0.2.jar -providerclass > org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair > -keyalg RSA -keypass "Test123" -validity 3650 -dname > "CN=Test_HostName, ou=My Org Unit, o=My Org" -storepass "Test123" > -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias test_entry > > However, when I add the JVM option > -J-Djava.security.egd=file:/dev/./urandom instead of the default > /dev/random, all problems go away. Classic entropy problem. > It's rather strange that multiple 7.3 RHEL systems are showing this > behavior very consistently. RHEL 7.2, 8.0 seem just fine in my > testing. No issues when u
