[jira] [Commented] (TS-1598) Coring in SSL
[ https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504806#comment-13504806 ] James Peach commented on TS-1598: - Without more info, I don't think there's anything we can do here. Is this reproducible? Can you get a packet trace or verbose log? > Coring in SSL > - > > Key: TS-1598 > URL: https://issues.apache.org/jira/browse/TS-1598 > Project: Traffic Server > Issue Type: Bug > Components: SSL >Affects Versions: 3.2.0 > Environment: RHEL6.2 64bit >Reporter: Abhishek Nayani > Fix For: 3.2.0 > > > (gdb) bt > #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6 > #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10 > #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10 > #3 0x0066eaf7 in ssl_read_from_net (nh=, > vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at > SSLNetVConnection.cc:135 > #4 0x0066f3b0 in SSLNetVConnection::net_read_io > (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at > SSLNetVConnection.cc:288 > #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8, > event=, e=) at UnixNet.cc:381 > #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0, > calling_code=5) at I_Continuation.h:146 > #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5) > at UnixEThread.cc:142 > #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at > UnixEThread.cc:264 > #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88 > #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0 > #11 0x00390ace76dd in clone () from /lib64/libc.so.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1598) Coring in SSL
[ https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505195#comment-13505195 ] Abhishek Nayani commented on TS-1598: - This happens once or twice a day. I do not have particular request which can reproduce this. The traffic is predominantly WebDAV (CalDAV) over SSL. Let me know if you need any other info. I've a core but cannot attach it to this bug as it is > 3GB. > Coring in SSL > - > > Key: TS-1598 > URL: https://issues.apache.org/jira/browse/TS-1598 > Project: Traffic Server > Issue Type: Bug > Components: SSL >Affects Versions: 3.2.0 > Environment: RHEL6.2 64bit >Reporter: Abhishek Nayani > > (gdb) bt > #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6 > #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10 > #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10 > #3 0x0066eaf7 in ssl_read_from_net (nh=, > vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at > SSLNetVConnection.cc:135 > #4 0x0066f3b0 in SSLNetVConnection::net_read_io > (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at > SSLNetVConnection.cc:288 > #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8, > event=, e=) at UnixNet.cc:381 > #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0, > calling_code=5) at I_Continuation.h:146 > #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5) > at UnixEThread.cc:142 > #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at > UnixEThread.cc:264 > #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88 > #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0 > #11 0x00390ace76dd in clone () from /lib64/libc.so.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1598) Coring in SSL
[ https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505219#comment-13505219 ] James Peach commented on TS-1598: - ATS is calling: int rres = SSL_read(sslvc->ssl, b->end() + offset, (int)block_write_avail); And we end up here in OpenSSL: int ssl3_read_bytes(SSL *s, int type, unsigned char *buf, int len, int peek) ... if ((unsigned int)len > rr->length) n = rr->length; else n = (unsigned int)len; memcpy(buf,&(rr->data[rr->off]),n); So I'd guess that we screwed up IO buffer management somehow, or there's a OpenSSL bug that is screwing up the memcpy. Abhishek, what verion of OpenSSL are you using? Since you have a core, can you try to get the valued of "buf", "len" and "n" from ssl3_read_bytes? > Coring in SSL > - > > Key: TS-1598 > URL: https://issues.apache.org/jira/browse/TS-1598 > Project: Traffic Server > Issue Type: Bug > Components: SSL >Affects Versions: 3.2.0 > Environment: RHEL6.2 64bit >Reporter: Abhishek Nayani > > (gdb) bt > #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6 > #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10 > #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10 > #3 0x0066eaf7 in ssl_read_from_net (nh=, > vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at > SSLNetVConnection.cc:135 > #4 0x0066f3b0 in SSLNetVConnection::net_read_io > (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at > SSLNetVConnection.cc:288 > #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8, > event=, e=) at UnixNet.cc:381 > #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0, > calling_code=5) at I_Continuation.h:146 > #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5) > at UnixEThread.cc:142 > #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at > UnixEThread.cc:264 > #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88 > #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0 > #11 0x00390ace76dd in clone () from /lib64/libc.so.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1598) Coring in SSL
[ https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505353#comment-13505353 ] Abhishek Nayani commented on TS-1598: - OpenSSL: tps_openssl-0.9.8o_2 I am not able to get any data from that frame (ssl3_read_bytes). I've given Bryan Call access to these cores, so he might be able to get more info out of them. > Coring in SSL > - > > Key: TS-1598 > URL: https://issues.apache.org/jira/browse/TS-1598 > Project: Traffic Server > Issue Type: Bug > Components: SSL >Affects Versions: 3.2.0 > Environment: RHEL6.2 64bit >Reporter: Abhishek Nayani > > (gdb) bt > #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6 > #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10 > #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10 > #3 0x0066eaf7 in ssl_read_from_net (nh=, > vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at > SSLNetVConnection.cc:135 > #4 0x0066f3b0 in SSLNetVConnection::net_read_io > (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at > SSLNetVConnection.cc:288 > #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8, > event=, e=) at UnixNet.cc:381 > #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0, > calling_code=5) at I_Continuation.h:146 > #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5) > at UnixEThread.cc:142 > #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at > UnixEThread.cc:264 > #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88 > #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0 > #11 0x00390ace76dd in clone () from /lib64/libc.so.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1598) Coring in SSL
[
https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507515#comment-13507515
]
Bryan Call commented on TS-1598:
so the line that starts this mess is SSLNetVConnection.cc:135:
while (block_write_avail > 0) {
int rres = SSL_read(sslvc->ssl, b->end() + offset,
(int)block_write_avail); <--- this line
(gdb) info locals
rres =
offset =
s = 0x2b1f3c633350
sslvc = 0x2b1f3c633250
buf = @0x2b1f3c633378
bytes_read = 0
block_write_avail = 47430144907985 <--- if that is the number of blocks that
seem crazy high
__func__ = "ssl_read_from_net"
b = 0x2b1f6b16f9b0
event =
sslErr = 0
(gdb) p *b
$19 = { = { = {_vptr.ForceVFPTToTop = 0x6a41b0},
m_refcount = 1}, _start = 0x0, _end = 0x0,
_buf_end = 0x2b2330f05ad1 "\272j",
_location = 0x6bcaa8 "memory/IOBuffer/HttpClientSession.cc:235", data = {
m_ptr = 0x2b2088e31e00}, next = {m_ptr = 0x0}}
(gdb) p *sslvc->ssl
$20 = {version = 769, type = 8192, method = 0x3f96455840,
rbio = 0x2b1fbceed1c0, wbio = 0x2b1fbceed1c0, bbio = 0x0, rwstate = 1,
in_handshake = 0, handshake_func = 0x3f9621cd30 , server = 1,
new_session = 0, quiet_shutdown = 1, shutdown = 0, state = 3, rstate = 240,
init_buf = 0x0, init_msg = 0x2b235cf40a34, init_num = 0, init_off = 0,
packet = 0x2b1f78d17d23 "\027\003\001\b\t\n\n \n
\nhttp://calendarserver.org/ns/\"/>\n, session = 0x2b1fbd066e80,
generate_session_id = 0, verify_mode = 0, verify_callback = 0,
info_callback = 0, error = 0, error_code = 0, kssl_ctx = 0x2b1fbceed240,
psk_client_callback = 0, psk_server_callback = 0, ctx = 0x1916080,
debug = 0, verify_result = 0, ex_data = {sk = 0x2b1fbceed670, dummy = 0},
client_CA = 0x0, references = 1, options = 4194308, mode = 0,
max_cert_list = 102400, first_packet = 0, client_version = 769,
max_send_fragment = 16384, tlsext_debug_cb = 0, tlsext_debug_arg = 0x0,
tlsext_hostname = 0x0, servername_done = 0, tlsext_status_type = -1,
tlsext_status_expected = 0, tlsext_ocsp_ids = 0x0, tlsext_ocsp_exts = 0x0,
tlsext_ocsp_resp = 0x0, tlsext_ocsp_resplen = -1,
tlsext_ticket_expected = 0, tlsext_opaque_prf_input = 0x0,
tlsext_opaque_prf_input_len = 0, tlsext_session_ticket = 0x0,
tls_session_ticket_ext_cb = 0, tls_session_ticket_ext_cb_arg = 0x0,
tls_session_secret_cb = 0, tls_session_secret_cb_arg = 0x0,
initial_ctx = 0x1916080}
> Coring in SSL
> -
>
> Key: TS-1598
> URL: https://issues.apache.org/jira/browse/TS-1598
> Project: Traffic Server
> Issue Type: Bug
> Components: SSL
>Affects Versions: 3.2.0
> Environment: RHEL6.2 64bit
>Reporter: Abhishek Nayani
>
> (gdb) bt
> #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6
> #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10
> #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10
> #3 0x0066eaf7 in ssl_read_from_net (nh=,
> vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at
> SSLNetVConnection.cc:135
> #4 0x0066f3b0 in SSLNetVConnection::net_read_io
> (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at
> SSLNetVConnection.cc:288
> #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8,
> event=, e=) at UnixNet.cc:381
> #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0,
> calling_code=5) at I_Continuation.h:146
> #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5)
> at UnixEThread.cc:142
> #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at
> UnixEThread.cc:264
> #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88
> #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0
> #11 0x00390ace76dd in clone () from /lib64/libc.so.6
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1598) Coring in SSL
[
https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507563#comment-13507563
]
Bryan Call commented on TS-1598:
A few things to note:
1. The _start and _end are not set in the IOBufferBlock - default values of 0x0
2. When write_avail is called on the buffer block it takes the _buf_end pointer
and subtracts _end giving the crazy high number of bytes available to write
3. b->end() + offset in the code will be a very low memory address since _end
== 0x0
Looking at the MIOBuffer:
(gdb) p *buf.mbuf
$6 = {size_index = 47430144908000, water_mark = 0, _writer = {
m_ptr = 0x2b1f6b16f9b0}, readers = {{accessor = 0x0, mbuf = 0x0, block = {
m_ptr = 0x0}, start_offset = 0, size_limit = 9223372036854775807}, {
accessor = 0x0, mbuf = 0x0, block = {m_ptr = 0x0}, start_offset = 0,
size_limit = 9223372036854775807}, {accessor = 0x0, mbuf = 0x0, block = {
m_ptr = 0x0}, start_offset = 0, size_limit = 9223372036854775807}, {
accessor = 0x0, mbuf = 0x0, block = {m_ptr = 0x0}, start_offset = 0,
size_limit = 9223372036854775807}, {accessor = 0x0, mbuf = 0x0, block = {
m_ptr = 0x0}, start_offset = 0, size_limit = 9223372036854775807}},
_location = 0x6bcaa8 "memory/IOBuffer/HttpClientSession.cc:235"}
> Coring in SSL
> -
>
> Key: TS-1598
> URL: https://issues.apache.org/jira/browse/TS-1598
> Project: Traffic Server
> Issue Type: Bug
> Components: SSL
>Affects Versions: 3.2.0
> Environment: RHEL6.2 64bit
>Reporter: Abhishek Nayani
>
> (gdb) bt
> #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6
> #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10
> #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10
> #3 0x0066eaf7 in ssl_read_from_net (nh=,
> vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at
> SSLNetVConnection.cc:135
> #4 0x0066f3b0 in SSLNetVConnection::net_read_io
> (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at
> SSLNetVConnection.cc:288
> #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8,
> event=, e=) at UnixNet.cc:381
> #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0,
> calling_code=5) at I_Continuation.h:146
> #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5)
> at UnixEThread.cc:142
> #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at
> UnixEThread.cc:264
> #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88
> #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0
> #11 0x00390ace76dd in clone () from /lib64/libc.so.6
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1598) Coring in SSL
[ https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13606877#comment-13606877 ] Bryan Call commented on TS-1598: Yes, close the bug. I believe it was fixed or they aren't having this issue anymore... > Coring in SSL > - > > Key: TS-1598 > URL: https://issues.apache.org/jira/browse/TS-1598 > Project: Traffic Server > Issue Type: Bug > Components: SSL >Affects Versions: 3.2.0 > Environment: RHEL6.2 64bit >Reporter: Abhishek Nayani >Assignee: Bryan Call > Fix For: sometime > > > (gdb) bt > #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6 > #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10 > #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10 > #3 0x0066eaf7 in ssl_read_from_net (nh=, > vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at > SSLNetVConnection.cc:135 > #4 0x0066f3b0 in SSLNetVConnection::net_read_io > (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at > SSLNetVConnection.cc:288 > #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8, > event=, e=) at UnixNet.cc:381 > #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0, > calling_code=5) at I_Continuation.h:146 > #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5) > at UnixEThread.cc:142 > #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at > UnixEThread.cc:264 > #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88 > #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0 > #11 0x00390ace76dd in clone () from /lib64/libc.so.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
