I posted this on Monday from my home account, and have some followups from my work account:
Recap: On a RHEL6 box with FIPS enabled at boot time Tomcat 6.0.37 and Tomcat Native Library 1.1.29 will not start if the APR listener is configured with 'FIPSMode="on". There appear to be two places that are causing an abort during initialization: 1) TCN ssl.c in fipsModeSet - the return from FIPS_mode_set() is 0, which triggers a exception 2) If I explicitly check for the current mode and skip the call to FIPS_mode_set() if already set to one then the code which pregenerates the temporary keys fails in 'initialize'. Specifically, the call to generate the RSA 512 bit key fails, which causes the routine to abort. A coworker here indicated that the 512 bit RSA key is invalid for FIPS mode. My initial fix to this was to have the JNI call in the AprLifecycleListener code try and see if FIPS was already enabled before calling fipsModeSet so it could log a suitable message. I don't know if there is a way for the TCN ssl.c code to return a non-error message back to the AprLifecycleListener startup or not. This solved issue #1 For issue #2 I just removed the line in the SSL_TMP_KEYS_INIT macro in TCN ssl.c generating the 512 bit RSA key. Might need to put some logic there so that in FIPS mode only FIPS legal key lengths are generated... -Rob
