Hi All, We are having a problem with a Tomcat client getting "bad_record_mac" exceptions when connecting to a server. Other applications are able to connect to that service so this seems specific to our client.
I have included a description of the problem, analysis steps taken/not yet taken, environment information, and client side logs below. Any insights or suggestions that you may have will be very welcome indeed. Thanks, Diarmuid Description of the problem ========================== On a Windows 2008 R2 64-bit Enterprise server running Java 7u67 and Tomcat 7.0.55, outbound SSL connections using TLSv1 and cipher suite TLS_RSA_WITH_AES_128_CBC_SHA are failing on the client side with "SEND TLSv1 ALERT: fatal, description = bad_record_mac" and javax.net.ssl.SSLException: Invalid Padding length errors. This uses one-way SSL and there is no Mutual Authentication involved. The SSL handshake appears to successfully establish a shared secret but fails when it tries to change the new established cipher suite. Analysis Steps ============== 1) On a non-production server with the same JRE/Tomcat/Windows setup connecting to a non-production version of the same service. This works ok. 2) On the same non-production server, remove the truststore/keystore settings from the Tomcat Java Options and connecting to a non-production version of the same service. Surprisingly, this works ok even though the endpoint uses a self-signed certificate. This connection also uses TLS_RSA_WITH_AES_128_CBC_SHA. 3) On the affected production server, run a standalone Java program that just opens an SSL input stream to the production service. This works ok. 3) Had a working session with the application team on the server side. What they see is: a) On our affected client, when I query their service from Internet Explorer I can successfully retrieve the WSDL and their web server records a GET request b) On our affected client, when I try to invoke a POST request from the Tomcat client application, they don't see any traffic on their web server and our client fails with the bad_record_mac/Invalid Padding Length error 4) Googled for "Invalid Padding length" and cipher TLS_RSA_WITH_AES_128_CBC_SHA. There are numerous references to this. The most solid is a known Java 7u51 issue recorded here https://bugs.openjdk.java.net/browse/JDK-8054316. However this and the majority of other TLS Padding links refer to javax.net.ssl.SSLHandshakeException or "javax.net.ssl.SSLException: Received fatal alert: handshake_failure", whereas our situation the handshake is completing and on the client we get a "javax.net.ssl.SSLException: Invalid Padding length". Aside from that, this article recommends using a different cipher suite, but our tests on a different server in step (2) above worked with TLS_RSA_WITH_AES_128_CBC_SHA. 5) Read Oracle documentation on debugging SSL/TLS connections here http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html. What I see from this is: a) the server is changing the cipher suite rather than the client b) in our client trace I don't see "Found trusted certificate", but based on removing the truststore on a different server in step (2) above my conclusion is that this shouldn't matter, or if it did matter then the connection would fail with an error like "No trusted cert found" 6) Queried "SEND TLSv1 ALERT: fatal, description = bad_record_mac" and "handling exception: javax.net.ssl.SSLException: Invalid Padding length" on Stackoverflow but didn't learn much. 7) Confirmed that other applications can connect to the service with no issues 8) Check for differences in Windows patches between the affected production server and working non-production server. Using WMIC to compare produced the list of patches below which are on the working server but not on the failing server. I am working on addressing those but analysis of them indicates that they relate to Microsoft Office, .NET, and GDI components that are not being used by Tomcat or our client application. http://support.microsoft.com/?kbid=3046017 http://support.microsoft.com/?kbid=3060716 http://support.microsoft.com/?kbid=3071756 http://support.microsoft.com/?kbid=3072305 http://support.microsoft.com/?kbid=3075220 http://support.microsoft.com/?kbid=3076895 http://support.microsoft.com/?kbid=3077657 http://support.microsoft.com/?kbid=3078071 http://support.microsoft.com/?kbid=3078601 http://support.microsoft.com/?kbid=3079757 http://support.microsoft.com/?kbid=3087985 9) Firewall analysis between the client and server shows a mix of tcp-fin and tcp-rst commands from the server. However, given that no traffic appears on the server-side web server logs it isn't clear if this is normal communication or an O/S level issue. 10) The truststore and keystore are not logged in the Tomcat logs. Our tomcat Java Options are below but even removing the double quotes makes no difference. I don't know if this is an issue given that the non-production test worked ok with no truststore in step (2) above -Dcatalina.home=d:\Tomcat_ENV4 -Dcatalina.base=d:\Tomcat_ENV4 -Djava.endorsed.dirs=d:\Tomcat_ENV4\endorsed -Djava.io.tmpdir=d:\Tomcat_ENV4\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=d:\Tomcat_ENV4\conf\logging.properties -XX:MaxPermSize=1024m -Dorg.apache.el.parser.COERCE_TO_ZERO=false -Dfile.encoding=UTF-8 -Djavax.net.ssl.trustStore="d:\Tomcat_ENV4\tomcat_truststore.jks" -Djavax.net.ssl.trustStorePassword="<snip>" -Djavax.net.ssl.keyStore="d:\Tomcat_ENV4\AWCHASSMESA002_keystore.jks -Djavax.net.ssl.keyStorePassword="<snip>" -Dhttps.protocols="TLSv1,TLSv1.1,TLSv1.2" -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9012 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -XX:+UseG1GC -XX:NewRatio=4 -XX:InitiatingHeapOccupancyPercent=75 -Djavax.net.debug=ssl:handshake 11) Googled the Tomcat Users forum with the phrases "tomcat_users SEND TLSv1 ALERT: fatal, description = bad_record_mac" and "tomcat_users SEND TLSv1 ALERT: fatal, description = bad_record_mac" This link http://security.stackexchange.com/questions/39844/getting-ssl-alert-write-fatal-bad-record-mac-during-openssl-handshake suggests that there might be: a) a mismatch between the server public and private keys - it suggests using wireshark to get the server public key for comparison with the server private key b) a buggy cryptographic algorithm on either the client or server - it suggests using openssl to analyse this Analysis not yet done ===================== 1) Use openssl on the affected server 2) Install Windows Network Monitor or Wireshark on the affected client server Client software versions ======================== 1) Java java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) JCE Unlimited Security: No 2) Tomcat Apache Tomcat/7.0.55 3) Operating System C:\>wmic os get name Name Microsoft Windows Server 2008 R2 Enterprise |C:\Windows|\Device\Harddisk0\Partition2 C:\>wmic os get buildnumber, buildtype, version BuildNumber BuildType Version 7601 Multiprocessor Free 6.1.7601 C:\>wmic os get servicepackmajorversion, servicepackminorversion ServicePackMajorVersion ServicePackMinorVersion 1 0 C:\>wmic os get operatingsystemsku OperatingSystemSKU 10 Server versions =============== Web server: IBM HTTP Server 6.1 Java: none OS: Solaris 10 DataPower: No Client Log File Extract using -Djavax.net.debug=ssl:handshake ============================================================= Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_128_CBC_SHA256 Allow unsafe renegotiation: false Allow legacy hello messages: true Is initial handshake: true Is secure renegotiation: false schedulerFactory_Worker-8, setSoTimeout(0) called %% No cached client session *** ClientHello, TLSv1.2 RandomCookie: GMT: 1423675093 bytes = { 59, 69, 104, 56, 82, 159, 163, 199, 164, 17, 202, 162, 164, 138, 227, 23, 72, 131, 9, 12, 170, 240, 164, 81, 78, 211, 18, 182 } Session ID: {} Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV] Compression Methods: { 0 } Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1} Extension ec_point_formats, formats: [uncompressed] Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA224withECDSA, SHA224withRSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA, MD5withRSA Extension server_name, server_name: [host_name:dbgsvc.com] *** schedulerFactory_Worker-8, WRITE: TLSv1.2 Handshake, length = 202 schedulerFactory_Worker-8, READ: TLSv1 Handshake, length = 874 *** ServerHello, TLSv1 RandomCookie: GMT: 85 bytes = { 248, 101, 199, 189, 146, 96, 91, 4, 35, 49, 184, 57, 70, 109, 49, 169, 202, 62, 196, 132, 153, 140, 11, 62, 238, 254, 62, 248 } Session ID: {0, 0, 75, 106, 51, 228, 118, 38, 226, 191, 94, 148, 108, 7, 1, 26, 35, 185, 139, 243, 88, 88, 88, 88, 85, 220, 143, 212, 0, 0, 1, 243} Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA Compression Method: 0 *** Warning: No renegotiation indication extension in ServerHello %% Initialized: [Session-1288, TLS_RSA_WITH_AES_128_CBC_SHA] ** TLS_RSA_WITH_AES_128_CBC_SHA *** Certificate chain chain [0] = [ [ Version: V3 Subject: CN=dbgsvc.com, OU=Middleware, O=dbg, C=US Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 Key: Sun RSA public key, 2048 bits modulus: 27416625621508503270554151522092741482020477489678745726307230823654660202238477697170912505823360308620180535745683658447432466260395370308135646165829908384391197464382820445267126581648037315502027378754032426878559408357098485112148908686750527323576567695641186144720588008421870121138420689230509512232184237525975953193108683130916369410204566978146692849095305070366472400972116710158959670624153993473014517157505719274868907893011902408775109335680879452493964731153184810064194656254612233563261344757525391145460858752257216209328834445515104802735995754986644156237721271016494185419847066126313124298401 public exponent: 65537 Validity: [From: Tue Jan 13 15:22:34 CAT 2015, To: Fri Jan 10 15:22:34 CAT 2025] Issuer: CN=dbgsvc.com, OU=Middleware, O=dbg, C=US SerialNumber: [ 54b51c1a] ] Algorithm: [SHA1withRSA] Signature: 0000: A3 75 13 D5 3B 9B B1 D5 44 39 2F 31 93 2E 9F 5F .u..;...D9/1..._ 0010: 86 61 4D 1F AA 0E 25 BA 70 26 13 93 F5 6E 73 7C .aM...%.p&...ns. 0020: 23 A5 E1 A0 96 7C F9 A0 A3 B4 09 B8 29 2C BC C0 #...........),.. 0030: 5F E4 9D 96 FE 18 B5 BF 05 B5 5D 16 87 E3 DB 77 _.........]....w 0040: 1C 92 5A C5 99 65 4B BE 68 76 AF C1 04 CA EF 6D ..Z..eK.hv.....m 0050: 2D 56 5C BB AE A4 87 4E EB 88 D7 60 0F 74 30 80 -V\....N...`.t0. 0060: 47 F8 E0 74 8B 8B 73 C2 84 87 9B 64 5E FC B7 D7 G..t..s....d^... 0070: 6E F8 F5 24 9B B2 10 2B 2B 94 33 26 06 BC 62 21 n..$...++.3&..b! 0080: 4D C1 B4 BB 98 35 5F BF 18 21 C6 C1 98 EA 20 D5 M....5_..!.... . 0090: 8E C3 4D C5 1A 9C B9 21 FA B1 1B 1D 74 05 31 1A ..M....!....t.1. 00A0: C0 19 CD E2 22 81 FC 36 57 D6 86 29 FD 76 FA 97 ...."..6W..).v.. 00B0: D8 F3 06 55 15 2D B8 87 FB 34 92 5E 7B 53 CA 17 ...U.-...4.^.S.. 00C0: 27 9F 6D 45 86 C3 FD 60 93 51 27 39 AA 95 9D 26 '.mE...`.Q'9...& 00D0: 8C DB CE FC 3F E8 FA 09 4B 49 AF E8 6A 91 C3 C6 ....?...KI..j... 00E0: 1D D6 48 D7 EF B5 55 B2 CC 1F 17 D0 37 DF CE 09 ..H...U.....7... 00F0: 6A 30 11 C0 28 05 E4 C6 88 F9 3E C6 26 36 58 D9 j0..(.....>.&6X. ] *** *** ServerHelloDone *** ClientKeyExchange, RSA PreMasterSecret, TLSv1 schedulerFactory_Worker-8, WRITE: TLSv1 Handshake, length = 262 SESSION KEYGEN: PreMaster Secret: 0000: 03 03 3C 7D 72 98 C9 9E 1C 14 6B CF CD EE 04 28 ..<.r.....k....( 0010: 4A 1A 25 5E 66 37 E4 56 98 66 83 60 51 E9 8A E0 J.%^f7.V.f.`Q... 0020: CA 4B 88 D1 9E 9F 34 FF B0 A3 02 E5 14 1D AC 44 .K....4........D CONNECTION KEYGEN: Client Nonce: 0000: 55 DC 8F D5 3B 45 68 38 52 9F A3 C7 A4 11 CA A2 U...;Eh8R....... 0010: A4 8A E3 17 48 83 09 0C AA F0 A4 51 4E D3 12 B6 ....H......QN... Server Nonce: 0000: 00 00 00 55 F8 65 C7 BD 92 60 5B 04 23 31 B8 39 ...U.e...`[.#1.9 0010: 46 6D 31 A9 CA 3E C4 84 99 8C 0B 3E EE FE 3E F8 Fm1..>.....>..>. Master Secret: 0000: 6D AC 3B 14 7B 97 FB 52 80 3D 82 10 AB E9 00 43 m.;....R.=.....C 0010: 06 01 F9 1C C2 62 8F C1 70 9B B6 E3 54 59 1B 7E .....b..p...TY.. 0020: 2B B9 F4 54 17 51 CA 99 D3 03 ED 17 E8 CE 93 30 +..T.Q.........0 Client MAC write Secret: 0000: B9 73 E9 31 9E 93 04 C7 A1 C1 2D 19 4F 32 BA D9 .s.1......-.O2.. 0010: 95 C8 8E FD .... Server MAC write Secret: 0000: 0C 47 2A 24 AA 73 67 A5 71 22 EF A8 AB 2D 3C 65 .G*$.sg.q"...-<e 0010: D5 A6 45 FE ..E. Client write key: 0000: 0D A4 BB 9F 81 2E EB C5 E7 57 56 03 CE BD A0 A9 .........WV..... Server write key: 0000: 6C 12 B0 7B 37 DB C8 0E AE 00 E1 40 5F A3 AE E3 l...7......@_... Client write IV: 0000: 71 F9 65 D0 9C 03 4F A2 F3 71 08 BD 12 EB F6 48 q.e...O..q.....H Server write IV: 0000: 8A F4 74 FC 17 5A 43 85 F2 2E CB E7 4B 55 23 04 ..t..ZC.....KU#. schedulerFactory_Worker-8, WRITE: TLSv1 Change Cipher Spec, length = 1 *** Finished verify_data: { 164, 98, 156, 19, 91, 248, 251, 65, 109, 13, 178, 66 } *** schedulerFactory_Worker-8, WRITE: TLSv1 Handshake, length = 48 schedulerFactory_Worker-8, READ: TLSv1 Change Cipher Spec, length = 1 schedulerFactory_Worker-8, READ: TLSv1 Alert, length = 32 %% Invalidated: [Session-1288, TLS_RSA_WITH_AES_128_CBC_SHA] schedulerFactory_Worker-8, SEND TLSv1 ALERT: fatal, description = bad_record_mac schedulerFactory_Worker-8, WRITE: TLSv1 Alert, length = 32 schedulerFactory_Worker-8, called closeSocket() schedulerFactory_Worker-8, handling exception: javax.net.ssl.SSLException: Invalid Padding length: 137 schedulerFactory_Worker-8, called close() schedulerFactory_Worker-8, called closeInternal(true)
