[
https://issues.apache.org/jira/browse/TS-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824216#comment-13824216
]
David Carlin edited comment on TS-2355 at 11/15/13 11:09 PM:
-------------------------------------------------------------
I don't see error 1408F10B before the crash like how is reported for squid.
Host crashed at 21:44 and the previous instance of 1408F10B is 2.5 hours
earlier. FYI - At this time (19:11) the host was running a build of ATS built
against OpenSSL 1.0.0. In between that time and the crash whose logs appear
below at 21:44, I had replaced ATS on the host with one built against OpenSSL
1.0.1e per suggestions on IRC.
{noformat}
[Nov 15 19:11:15.871] Server {0x2b5f1a931700} ERROR: SSL::25:error:1408F10B:SSL
routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:337:
{noformat}
SSL errors just before the crash:
{noformat}
[Nov 15 21:44:03.572] Server {0x2b524c807700} ERROR: SSL::27:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:03.955] Server {0x2b524da19700} ERROR: SSL::45:error:14094412:SSL
routines:SSL3_READ_BYTES:sslv3 alert bad certificate:s3_pkt.c:1256:SSL alert
number 42
[Nov 15 21:44:04.313] Server {0x2b524dc1b700} ERROR: SSL::47:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:08.201] Server {0x2b5244100700} NOTE: Traffic Server is skipping
the current log entry for squid.log because its size (87144) exceeds the
maximum payload space in a log buffer
[Nov 15 21:44:12.798] Server {0x2b5244e0d700} NOTE: Traffic Server is skipping
the current log entry for squid.log because its size (16528) exceeds the
maximum payload space in a log buffer
[Nov 15 21:44:14.101] Server {0x2b524cd0c700} ERROR: SSL::32:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:16.352] Server {0x2b524c504700} ERROR: SSL::24:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:16.600] Server {0x2b524c605700} ERROR: SSL::25:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:25.645] Server {0x2b524c807700} ERROR: SSL::27:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:27.051] Server {0x2b524cc0b700} ERROR: SSL::31:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:29.943] Server {0x2b524ce0d700} ERROR: SSL::33:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:31.863] Server {0x2b524cb0a700} ERROR: SSL::30:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:32.284] Server {0x2b524c605700} ERROR: SSL::25:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:33.168] Server {0x2b524c605700} ERROR: SSL::25:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:38.135] {0x2aaca1066640} STATUS: opened
/home/y/logs/trafficserver/diags.log
{noformat}
was (Author: dcarlin):
I don't see error 1408F10B before the crash like how is reported for squid.
Host crashed at 21:44 and the previous instance of 1408F10B is 2.5 hours
earlier. FYI - At this time (19:11) the host was running a build of ATS built
against OpenSSL 1.0.0. In between that time and the crash whose logs appear
below at 21:44, I had replaced ATS on the host with one built against OpenSSL
1.0.1e.
{noformat}
[Nov 15 19:11:15.871] Server {0x2b5f1a931700} ERROR: SSL::25:error:1408F10B:SSL
routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:337:
{noformat}
SSL errors just before the crash:
{noformat}
[Nov 15 21:44:03.572] Server {0x2b524c807700} ERROR: SSL::27:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:03.955] Server {0x2b524da19700} ERROR: SSL::45:error:14094412:SSL
routines:SSL3_READ_BYTES:sslv3 alert bad certificate:s3_pkt.c:1256:SSL alert
number 42
[Nov 15 21:44:04.313] Server {0x2b524dc1b700} ERROR: SSL::47:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:08.201] Server {0x2b5244100700} NOTE: Traffic Server is skipping
the current log entry for squid.log because its size (87144) exceeds the
maximum payload space in a log buffer
[Nov 15 21:44:12.798] Server {0x2b5244e0d700} NOTE: Traffic Server is skipping
the current log entry for squid.log because its size (16528) exceeds the
maximum payload space in a log buffer
[Nov 15 21:44:14.101] Server {0x2b524cd0c700} ERROR: SSL::32:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:16.352] Server {0x2b524c504700} ERROR: SSL::24:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:16.600] Server {0x2b524c605700} ERROR: SSL::25:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:25.645] Server {0x2b524c807700} ERROR: SSL::27:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:27.051] Server {0x2b524cc0b700} ERROR: SSL::31:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:29.943] Server {0x2b524ce0d700} ERROR: SSL::33:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:31.863] Server {0x2b524cb0a700} ERROR: SSL::30:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:32.284] Server {0x2b524c605700} ERROR: SSL::25:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:33.168] Server {0x2b524c605700} ERROR: SSL::25:error:140943E8:SSL
routines:SSL3_READ_BYTES:reason(1000):s3_pkt.c:1256:SSL alert number 0
[Nov 15 21:44:38.135] {0x2aaca1066640} STATUS: opened
/home/y/logs/trafficserver/diags.log
{noformat}
> ATS 4.0.x crashes when using OpenSSL 1.0.1e
> -------------------------------------------
>
> Key: TS-2355
> URL: https://issues.apache.org/jira/browse/TS-2355
> Project: Traffic Server
> Issue Type: Bug
> Components: SSL
> Reporter: David Carlin
>
> I upgraded some 4.0.1 and 4.0.2 hosts from OpenSSL 1.0.0 to 1.0.1e which is
> supposed to be ABI compatible. I see this crash about 10 times in a given 24
> hour period.
> I'm interested in OpenSSL 1.0.1e as there is a CPU usage improvement in my
> tests, and for TLS 1.2 support.
> I came across this squid bug with a very similar backtrace. The OpenSSL RT
> ticket says
> "I have discussed this situation with some Squid developers and we decided -
> after SSL error 1408F10B calling standard/raw read() instead of SSL_read()
> for empty socket buffer and this patch stopped crash Squid."
> http://rt.openssl.org/Ticket/Display.html?id=3128&user=guest&pass=guest
> {noformat}
> #0 0x0000003f842e7154 in EVP_DigestFinal_ex () from
> /usr/lib64/libcrypto.so.10
> #1 0x0000003f84636263 in tls1_final_finish_mac () from
> /usr/lib64/libssl.so.10
> #2 0x0000003f8462ad62 in ssl3_do_change_cipher_spec () from
> /usr/lib64/libssl.so.10
> #3 0x0000003f8462c7f7 in ssl3_read_bytes () from /usr/lib64/libssl.so.10
> #4 0x0000003f8462d5e2 in ssl3_get_message () from /usr/lib64/libssl.so.10
> #5 0x0000003f8461da1c in ssl3_get_cert_verify () from /usr/lib64/libssl.so.10
> #6 0x0000003f84621e78 in ssl3_accept () from /usr/lib64/libssl.so.10
> #7 0x00000000006711aa in SSLNetVConnection::sslServerHandShakeEvent
> (this=0x2aadd0024300,
> err=@0x2aacab940c5c) at SSLNetVConnection.cc:488
> #8 0x0000000000672b77 in SSLNetVConnection::sslStartHandShake
> (this=0x2aadd0024300,
> event=<value optimized out>, err=@0x2aacab940c5c) at
> SSLNetVConnection.cc:470
> #9 0x0000000000671dd2 in SSLNetVConnection::net_read_io
> (this=0x2aadd0024300, nh=
> 0x2aacaa02cbf0, lthread=0x2aacaa029010) at SSLNetVConnection.cc:217
> #10 0x000000000067b8c2 in NetHandler::mainNetEvent (this=0x2aacaa02cbf0,
> event=<value optimized out>, e=<value optimized out>) at UnixNet.cc:386
> #11 0x00000000006a335f in handleEvent (this=0x2aacaa029010, e=0x1230a30,
> calling_code=5)
> at I_Continuation.h:146
> #12 EThread::process_event (this=0x2aacaa029010, e=0x1230a30, calling_code=5)
> at UnixEThread.cc:141
> #13 0x00000000006a3d43 in EThread::execute (this=0x2aacaa029010) at
> UnixEThread.cc:265
> #14 0x00000000006a21fa in spawn_thread_internal (a=0x143ec30) at Thread.cc:88
> #15 0x00002aaca05b9851 in start_thread () from /lib64/libpthread.so.0
> #16 0x000000324f0e890d in clone () from /lib64/libc.so.6
> {noformat}
> {noformat}
> NOTE: Traffic Server received Sig 11: Segmentation fault
> /home/y/bin/traffic_server - STACK TRACE:
> /lib64/libpthread.so.0(+0x324f40f500)[0x2b523d64e500]
> /usr/lib64/libcrypto.so.10(EVP_DigestFinal_ex+0x24)[0x3f842e7154]
> /usr/lib64/libssl.so.10(tls1_final_finish_mac+0x233)[0x3f84636263]
> /usr/lib64/libssl.so.10(ssl3_do_change_cipher_spec+0x72)[0x3f8462ad62]
> /usr/lib64/libssl.so.10(ssl3_read_bytes+0xb57)[0x3f8462c7f7]
> /usr/lib64/libssl.so.10(ssl3_get_message+0x222)[0x3f8462d5e2]
> /usr/lib64/libssl.so.10(ssl3_get_cert_verify+0x6c)[0x3f8461da1c]
> /usr/lib64/libssl.so.10(ssl3_accept+0x788)[0x3f84621e78]
> /home/y/bin/traffic_server(_ZN17SSLNetVConnection23sslServerHandShakeEventERi+0x2a)[0x6711aa]
> /home/y/bin/traffic_server(_ZN17SSLNetVConnection17sslStartHandShakeEiRi+0x37)[0x672b77]
> /home/y/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x1f2)[0x671dd2]
> /home/y/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1f2)[0x67b8c2]
> /home/y/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a335f]
> /home/y/bin/traffic_server(_ZN7EThread7executeEv+0x4a3)[0x6a3d43]
> /home/y/bin/traffic_server[0x6a21fa]
> /lib64/libpthread.so.0(+0x324f407851)[0x2b523d646851]
> /lib64/libc.so.6(clone+0x6d)[0x324f0e890d]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
