-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Diarmuid,

On 8/27/15 12:08 PM, dmccrthy wrote:
> We are having a problem with a Tomcat client getting
> "bad_record_mac" exceptions when connecting to a server.  Other
> applications are able to connect to that service so this seems
> specific to our client.
> 
> I have included a description of the problem, analysis steps
> taken/not yet taken, environment information, and client side logs
> below.
> 
> Any insights or suggestions that you may have will be very welcome
> indeed.
> 
> Thanks, Diarmuid
> 
> 
> 
> Description of the problem ==========================
> 
> On a Windows 2008 R2 64-bit Enterprise server running Java 7u67 and
> Tomcat 7.0.55, outbound SSL connections using TLSv1 and cipher
> suite TLS_RSA_WITH_AES_128_CBC_SHA are failing on the client side
> with "SEND TLSv1 ALERT:  fatal, description = bad_record_mac" and 
> javax.net.ssl.SSLException: Invalid Padding length errors.  This
> uses one-way SSL and there is no Mutual Authentication involved.
> The SSL handshake appears to successfully establish a shared secret
> but fails when it tries to change the new established cipher
> suite.

So, the server tries to make an outgoing HTTPS request using that
cipher suite? Can you connect to that (other) server directly without
any errors? With that cipher suite? With what tool?

Or, you can't reach Tomcat directly using some client. I think I'm
confused by the term "Tomcat client".

> Analysis Steps ==============
> 
> 1) On a non-production server with the same JRE/Tomcat/Windows
> setup connecting to a non-production version of the same service.
> This works ok.
> 
> 2) On the same non-production server, remove the
> truststore/keystore settings from the Tomcat Java Options and
> connecting to a non-production version of the same service.
> Surprisingly, this works ok even though the endpoint uses a
> self-signed certificate.  This connection also uses 
> TLS_RSA_WITH_AES_128_CBC_SHA.

Not surprising to me: if you make an outgoing HTTPS connection from
your application, Tomcat's truststore/keystore don't make any
difference. If you removed your "keystore" configuration and a client
could still connect to your /Tomcat/ server, then something is
definitely broken, because the connector should never have
successfully initialized at all.

> 3) On the affected production server, run a standalone Java program
> that just opens an SSL input stream to the production service.
> This works ok.
> 
> 3) Had a working session with the application team on the server
> side. What they see is:
> 
> a) On our affected client, when I query their service from
> Internet Explorer I can successfully retrieve the WSDL and their
> web server records a GET request
> 
> b) On our affected client, when I try to invoke a POST request from
> the Tomcat client application, they don't see any traffic on their
> web server and our client fails with the bad_record_mac/Invalid
> Padding Length error

The error happens before the TLS handshake completes, so you wouldn't
see anything in the access logs. You might want to check other logs,
because you'd want to know if you had clients failing TLS handshakes.

> 4) Googled for "Invalid Padding length" and cipher 
> TLS_RSA_WITH_AES_128_CBC_SHA.  There are numerous references to
> this.  The most solid is a known Java 7u51 issue recorded here 
> https://bugs.openjdk.java.net/browse/JDK-8054316.  However this and
> the majority of other TLS Padding links refer to 
> javax.net.ssl.SSLHandshakeException or
> "javax.net.ssl.SSLException: Received fatal alert:
> handshake_failure", whereas our situation the handshake is
> completing and on the client we get a "javax.net.ssl.SSLException:
> Invalid Padding length".

No, your handshake has not completed. Or, possibly, the handshake has
completed, but you aren't able to encrypt anything so it doesn't
matter that you completed the handshake.

> Aside from that, this article recommends using a different cipher
> suite, but our tests on a different server in step (2) above worked
> with TLS_RSA_WITH_AES_128_CBC_SHA.

Really you should be using ECDHE or EDHE ciphers these days. I'm not
sure what IIS will actually support (if you are using IIS). You should
allow the server to specify the cipher suite order and make clents use
the highest-grade cipher suite they can support.

> 5) Read Oracle documentation on debugging SSL/TLS connections here 
> http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/Re
adDebug.html.
>
> 
What I see from this is:
> 
> a) the server is changing the cipher suite rather than the client

It's a negotiation, not a grudge match. Neither side changes anything.
Did you mean the the server chooses? You're right: the server always
chooses, but sometimes it chooses the first cipher listed by the
client, or it can be told to do something else (use server preferred
order, for instance).

> b) in our client trace I don't see "Found trusted certificate",
> but based on removing the truststore on a different server in step
> (2) above my conclusion is that this shouldn't matter, or if it did
> matter then the connection would fail with an error like "No
> trusted cert found"
> 
> 6) Queried "SEND TLSv1 ALERT:  fatal, description = bad_record_mac"
> and "handling exception: javax.net.ssl.SSLException: Invalid
> Padding length" on Stackoverflow but didn't learn much.
> 
> 7) Confirmed that other applications can connect to the service
> with no issues

What is the "client"?

> 8) Check for differences in Windows patches between the affected
> production server and working non-production server.  Using WMIC to
> compare produced the list of patches below which are on the working
> server but not on the failing server.  I am working on addressing
> those but analysis of them indicates that they relate to Microsoft
> Office, .NET, and GDI components that are not being used by Tomcat
> or our client application.
> 
> http://support.microsoft.com/?kbid=3046017 
> http://support.microsoft.com/?kbid=3060716 
> http://support.microsoft.com/?kbid=3071756 
> http://support.microsoft.com/?kbid=3072305 
> http://support.microsoft.com/?kbid=3075220 
> http://support.microsoft.com/?kbid=3076895 
> http://support.microsoft.com/?kbid=3077657 
> http://support.microsoft.com/?kbid=3078071 
> http://support.microsoft.com/?kbid=3078601 
> http://support.microsoft.com/?kbid=3079757 
> http://support.microsoft.com/?kbid=3087985
> 
> 
> 9) Firewall analysis between the client and server shows a mix of
> tcp-fin and tcp-rst commands from the server.  However, given that
> no traffic appears on the server-side web server logs it isn't
> clear if this is normal communication or an O/S level issue.
> 
> 10) The truststore and keystore are not logged in the Tomcat logs.
> Our tomcat Java Options are below but even removing the double
> quotes makes no difference.  I don't know if this is an issue given
> that the non-production test worked ok with no truststore in step
> (2) above
> 
> -Dcatalina.home=d:\Tomcat_ENV4 -Dcatalina.base=d:\Tomcat_ENV4 
> -Djava.endorsed.dirs=d:\Tomcat_ENV4\endorsed 
> -Djava.io.tmpdir=d:\Tomcat_ENV4\temp 
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 
> -Djava.util.logging.config.file=d:\Tomcat_ENV4\conf\logging.properties
>
> 
- -XX:MaxPermSize=1024m
> -Dorg.apache.el.parser.COERCE_TO_ZERO=false -Dfile.encoding=UTF-8 
> -Djavax.net.ssl.trustStore="d:\Tomcat_ENV4\tomcat_truststore.jks" 
> -Djavax.net.ssl.trustStorePassword="<snip>" 
> -Djavax.net.ssl.keyStore="d:\Tomcat_ENV4\AWCHASSMESA002_keystore.jks
>
> 
- -Djavax.net.ssl.keyStorePassword="<snip>"
> -Dhttps.protocols="TLSv1,TLSv1.1,TLSv1.2" 
> -Dcom.sun.management.jmxremote 
> -Dcom.sun.management.jmxremote.port=9012 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Dcom.sun.management.jmxremote.authenticate=false -XX:+UseG1GC 
> -XX:NewRatio=4 -XX:InitiatingHeapOccupancyPercent=75 
> -Djavax.net.debug=ssl:handshake
> 
> 11) Googled the Tomcat Users forum with the phrases "tomcat_users
> SEND TLSv1 ALERT:  fatal, description = bad_record_mac" and
> "tomcat_users SEND TLSv1 ALERT:  fatal, description =
> bad_record_mac"
> 
> This link 
> http://security.stackexchange.com/questions/39844/getting-ssl-alert-wr
ite-fatal-bad-record-mac-during-openssl-handshake
>
> 
suggests
> that there might be:
> 
> a) a mismatch between the server public and private keys - it
> suggests using wireshark to get the server public key for
> comparison with the server private key

A mismatch between the server private and public keys is nearly
impossibly to accomplish.

> b) a buggy cryptographic algorithm on either the client or server -
> it suggests using openssl to analyse this

This is possible.

> Analysis not yet done =====================
> 
> 1) Use openssl on the affected server

Do this.

> 2) Install Windows Network Monitor or Wireshark on the affected
> client server

Don't waste your time with this.

> Client software versions ========================
> 
> 1) Java
> 
> java version "1.7.0_67" Java(TM) SE Runtime Environment (build
> 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04,
> mixed mode)
> 
> JCE Unlimited Security: No

Why not?

> 2) Tomcat
> 
> Apache Tomcat/7.0.55
> 
> 3) Operating System
> 
> C:\>wmic os get name
> 
> Name
> 
> Microsoft Windows Server 2008 R2 Enterprise 
> |C:\Windows|\Device\Harddisk0\Partition2
> 
> 
> C:\>wmic os get buildnumber, buildtype, version
> 
> BuildNumber  BuildType            Version
> 
> 7601         Multiprocessor Free  6.1.7601
> 
> 
> C:\>wmic os get servicepackmajorversion, servicepackminorversion
> 
> ServicePackMajorVersion  ServicePackMinorVersion
> 
> 1                        0
> 
> C:\>wmic os get operatingsystemsku
> 
> OperatingSystemSKU
> 
> 10
> 
> Server versions ===============
> 
> Web server: IBM HTTP Server 6.1 Java: none OS: Solaris 10 
> DataPower: No
> 
> 
> Client Log File Extract using -Djavax.net.debug=ssl:handshake
> 
> =============================================================
> 
> 
> Ignoring unavailable cipher suite:
> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA Ignoring unavailable cipher
> suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA Ignoring unavailable cipher
> suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA Ignoring unsupported
> cipher suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 Ignoring
> unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 
> Ignoring unsupported cipher suite:
> TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 Ignoring unsupported cipher
> suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 Ignoring unsupported
> cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 Ignoring
> unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 
> Ignoring unsupported cipher suite:
> TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 Ignoring unsupported cipher
> suite: TLS_RSA_WITH_AES_256_CBC_SHA256 Ignoring unavailable cipher
> suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA Ignoring unsupported
> cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 Ignoring
> unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 
> Ignoring unavailable cipher suite:
> TLS_DHE_DSS_WITH_AES_256_CBC_SHA Ignoring unsupported cipher suite:
> TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 Ignoring unsupported cipher
> suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 Ignoring unsupported
> cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 Ignoring
> unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA 
> Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA 
> Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_128_CBC_SHA256
> 
> Allow unsafe renegotiation: false Allow legacy hello messages:
> true Is initial handshake: true Is secure renegotiation: false 
> schedulerFactory_Worker-8, setSoTimeout(0) called
> 
> %% No cached client session
> 
> *** ClientHello, TLSv1.2
> 
> RandomCookie:  GMT: 1423675093 bytes = { 59, 69, 104, 56, 82, 159,
> 163, 199, 164, 17, 202, 162, 164, 138, 227, 23, 72, 131, 9, 12,
> 170, 240, 164, 81, 78, 211, 18, 182 }
> 
> Session ID:  {}
> 
> Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, 
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, 
> TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,
> TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, 
> TLS_DHE_RSA_WITH_AES_128_CBC_SHA,
> TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,
> TLS_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA,
> TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA,
> TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, 
> TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,
> SSL_RSA_WITH_3DES_EDE_CBC_SHA, 
> TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,
> TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, 
> SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,
> SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5,
> TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
> 
> Compression Methods:  { 0 }
> 
> Extension elliptic_curves, curve names: {secp256r1, sect163k1,
> sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1,
> sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1,
> sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1,
> sect193r1, sect193r2, secp224k1, sect239k1, secp256k1}
> 
> Extension ec_point_formats, formats: [uncompressed]
> 
> Extension signature_algorithms, signature_algorithms:
> SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA,
> SHA256withECDSA, SHA256withRSA, SHA224withECDSA, SHA224withRSA,
> SHA1withECDSA, SHA1withRSA, SHA1withDSA, MD5withRSA
> 
> Extension server_name, server_name: [host_name:dbgsvc.com]
> 
> ***
> 
> schedulerFactory_Worker-8, WRITE: TLSv1.2 Handshake, length = 202
> 
> schedulerFactory_Worker-8, READ: TLSv1 Handshake, length = 874
> 
> *** ServerHello, TLSv1
> 
> RandomCookie:  GMT: 85 bytes = { 248, 101, 199, 189, 146, 96, 91,
> 4, 35, 49, 184, 57, 70, 109, 49, 169, 202, 62, 196, 132, 153, 140,
> 11, 62, 238, 254, 62, 248 }
> 
> Session ID:  {0, 0, 75, 106, 51, 228, 118, 38, 226, 191, 94, 148,
> 108, 7, 1, 26, 35, 185, 139, 243, 88, 88, 88, 88, 85, 220, 143,
> 212, 0, 0, 1, 243}
> 
> Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA
> 
> Compression Method: 0
> 
> ***
> 
> Warning: No renegotiation indication extension in ServerHello
> 
> %% Initialized:  [Session-1288, TLS_RSA_WITH_AES_128_CBC_SHA]
> 
> ** TLS_RSA_WITH_AES_128_CBC_SHA
> 
> *** Certificate chain
> 
> chain [0] = [
> 
> [
> 
> Version: V3
> 
> Subject: CN=dbgsvc.com, OU=Middleware, O=dbg, C=US
> 
> Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5
> 
> 
> 
> Key:  Sun RSA public key, 2048 bits
> 
> modulus: 
> 2741662562150850327055415152209274148202047748967874572630723082365466
020223847769717091250582336030862018053574568365844743246626039537030813
564616582990838439119746438282044526712658164803731550202737875403242687
855940835709848511214890868675052732357656769564118614472058800842187012
113842068923050951223218423752597595319310868313091636941020456697814669
284909530507036647240097211671015895967062415399347301451715750571927486
890789301190240877510933568087945249396473115318481006419465625461223356
326134475752539114546085875225721620932883444551510480273599575498664415
6237721271016494185419847066126313124298401
>
>  public exponent: 65537
> 
> Validity: [From: Tue Jan 13 15:22:34 CAT 2015,
> 
> To: Fri Jan 10 15:22:34 CAT 2025]
> 
> Issuer: CN=dbgsvc.com, OU=Middleware, O=dbg, C=US
> 
> SerialNumber: [    54b51c1a]
> 
> 
> 
> ]
> 
> Algorithm: [SHA1withRSA]
> 
> Signature:
> 
> 0000: A3 75 13 D5 3B 9B B1 D5   44 39 2F 31 93 2E 9F 5F
> .u..;...D9/1..._
> 
> 0010: 86 61 4D 1F AA 0E 25 BA   70 26 13 93 F5 6E 73 7C
> .aM...%.p&...ns.
> 
> 0020: 23 A5 E1 A0 96 7C F9 A0   A3 B4 09 B8 29 2C BC C0
> #...........),..
> 
> 0030: 5F E4 9D 96 FE 18 B5 BF   05 B5 5D 16 87 E3 DB 77
> _.........]....w
> 
> 0040: 1C 92 5A C5 99 65 4B BE   68 76 AF C1 04 CA EF 6D
> ..Z..eK.hv.....m
> 
> 0050: 2D 56 5C BB AE A4 87 4E   EB 88 D7 60 0F 74 30 80
> -V\....N...`.t0.
> 
> 0060: 47 F8 E0 74 8B 8B 73 C2   84 87 9B 64 5E FC B7 D7
> G..t..s....d^...
> 
> 0070: 6E F8 F5 24 9B B2 10 2B   2B 94 33 26 06 BC 62 21
> n..$...++.3&..b!
> 
> 0080: 4D C1 B4 BB 98 35 5F BF   18 21 C6 C1 98 EA 20 D5
> M....5_..!.... .
> 
> 0090: 8E C3 4D C5 1A 9C B9 21   FA B1 1B 1D 74 05 31 1A
> ..M....!....t.1.
> 
> 00A0: C0 19 CD E2 22 81 FC 36   57 D6 86 29 FD 76 FA 97
> ...."..6W..).v..
> 
> 00B0: D8 F3 06 55 15 2D B8 87   FB 34 92 5E 7B 53 CA 17
> ...U.-...4.^.S..
> 
> 00C0: 27 9F 6D 45 86 C3 FD 60   93 51 27 39 AA 95 9D 26
> '.mE...`.Q'9...&
> 
> 00D0: 8C DB CE FC 3F E8 FA 09   4B 49 AF E8 6A 91 C3 C6
> ....?...KI..j...
> 
> 00E0: 1D D6 48 D7 EF B5 55 B2   CC 1F 17 D0 37 DF CE 09
> ..H...U.....7...
> 
> 00F0: 6A 30 11 C0 28 05 E4 C6   88 F9 3E C6 26 36 58 D9
> j0..(.....>.&6X.
> 
> 
> 
> ]
> 
> ***
> 
> *** ServerHelloDone
> 
> *** ClientKeyExchange, RSA PreMasterSecret, TLSv1
> 
> schedulerFactory_Worker-8, WRITE: TLSv1 Handshake, length = 262
> 
> SESSION KEYGEN:
> 
> PreMaster Secret:
> 
> 0000: 03 03 3C 7D 72 98 C9 9E   1C 14 6B CF CD EE 04 28
> ..<.r.....k....(
> 
> 0010: 4A 1A 25 5E 66 37 E4 56   98 66 83 60 51 E9 8A E0
> J.%^f7.V.f.`Q...
> 
> 0020: CA 4B 88 D1 9E 9F 34 FF   B0 A3 02 E5 14 1D AC 44
> .K....4........D
> 
> CONNECTION KEYGEN:
> 
> Client Nonce:
> 
> 0000: 55 DC 8F D5 3B 45 68 38   52 9F A3 C7 A4 11 CA A2
> U...;Eh8R.......
> 
> 0010: A4 8A E3 17 48 83 09 0C   AA F0 A4 51 4E D3 12 B6
> ....H......QN...
> 
> Server Nonce:
> 
> 0000: 00 00 00 55 F8 65 C7 BD   92 60 5B 04 23 31 B8 39
> ...U.e...`[.#1.9
> 
> 0010: 46 6D 31 A9 CA 3E C4 84   99 8C 0B 3E EE FE 3E F8
> Fm1..>.....>..>.
> 
> Master Secret:
> 
> 0000: 6D AC 3B 14 7B 97 FB 52   80 3D 82 10 AB E9 00 43
> m.;....R.=.....C
> 
> 0010: 06 01 F9 1C C2 62 8F C1   70 9B B6 E3 54 59 1B 7E
> .....b..p...TY..
> 
> 0020: 2B B9 F4 54 17 51 CA 99   D3 03 ED 17 E8 CE 93 30
> +..T.Q.........0
> 
> Client MAC write Secret:
> 
> 0000: B9 73 E9 31 9E 93 04 C7   A1 C1 2D 19 4F 32 BA D9
> .s.1......-.O2..
> 
> 0010: 95 C8 8E FD                                        ....
> 
> Server MAC write Secret:
> 
> 0000: 0C 47 2A 24 AA 73 67 A5   71 22 EF A8 AB 2D 3C 65
> .G*$.sg.q"...-<e
> 
> 0010: D5 A6 45 FE                                        ..E.
> 
> Client write key:
> 
> 0000: 0D A4 BB 9F 81 2E EB C5   E7 57 56 03 CE BD A0 A9
> .........WV.....
> 
> Server write key:
> 
> 0000: 6C 12 B0 7B 37 DB C8 0E   AE 00 E1 40 5F A3 AE E3
> l...7......@_...
> 
> Client write IV:
> 
> 0000: 71 F9 65 D0 9C 03 4F A2   F3 71 08 BD 12 EB F6 48
> q.e...O..q.....H
> 
> Server write IV:
> 
> 0000: 8A F4 74 FC 17 5A 43 85   F2 2E CB E7 4B 55 23 04
> ..t..ZC.....KU#.
> 
> schedulerFactory_Worker-8, WRITE: TLSv1 Change Cipher Spec, length
> = 1
> 
> *** Finished
> 
> verify_data:  { 164, 98, 156, 19, 91, 248, 251, 65, 109, 13, 178,
> 66 }
> 
> ***
> 
> schedulerFactory_Worker-8, WRITE: TLSv1 Handshake, length = 48
> 
> schedulerFactory_Worker-8, READ: TLSv1 Change Cipher Spec, length =
> 1
> 
> schedulerFactory_Worker-8, READ: TLSv1 Alert, length = 32
> 
> %% Invalidated:  [Session-1288, TLS_RSA_WITH_AES_128_CBC_SHA]
> 
> schedulerFactory_Worker-8, SEND TLSv1 ALERT:  fatal, description = 
> bad_record_mac
> 
> schedulerFactory_Worker-8, WRITE: TLSv1 Alert, length = 32
> 
> schedulerFactory_Worker-8, called closeSocket()
> 
> schedulerFactory_Worker-8, handling exception:
> javax.net.ssl.SSLException: Invalid Padding length: 137
> 
> schedulerFactory_Worker-8, called close()
> 
> schedulerFactory_Worker-8, called closeInternal(true)
> 

You might also want to try updating to 7.0.64 (latest as of this
writing). But at this point, you don't have a reproducible test case
outside of production, so it would be a wild guess and might not work.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJV32aXAAoJEBzwKT+lPKRYFVgP/3oQy1vw/6BKmbg6WLmXzfcx
zAuosap72NhcTNOVA8wHV8HWT7DqZNKcu/TRqagYBvBz6zTa0aCEQbXMMHpwHA1D
vuEil5+Xh/VN8Gp9n27UcfRgSIdg1ZoGtExYHJahX5/BKbQ6pHBWdx+qKPF7e5kW
94H04earvOlFmmCSfUf2a6YkATTBICWHPjcnfGrkQwADY4EbwzlEcFLhRvF9uVE0
hoEbmTnmS+ChfdyWJpXiDpM7MxSCVsiK67hVfohnl46PuvAsDfpLfmNetsxzMEv2
/0AbNkiHcX/txgJ5v1/bu2cYi9mYQ1/J0i5ueNJFuMqX7zoS40eLsI1MjVbeuT/m
AJPvjwLS8Tpkb37ByEWcJPyHSyz5vA0kPeVjG7fwdBYqSlyVFI8MFMnGCDuWmR1S
m8EYpaWmEcple7pGzvA6ii6cWR9yPygY465huoVeiTfOyXsMiyfx7rkqsRMuKwOv
N2pacMfFDmWsbYCldbTiB4wpKIo/a1mdjsJRNiEXlWPiobpUjwTbnqDXOg/IWVZF
hlvBaWD+71UaDWyBxsGuzywjFWiUHZUUoFyFIix+HrX0bi3CBfO8+e+X1idXac88
0ieFVSmhwt+LKvxTnZZDp/zhVjSjhyYYC7Sd2hEU3RcH17D2XUwwSdIYbLtW1I9x
+AlDGZosfVaiCZs1xZsH
=Xbln
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to