[jira] [Commented] (TS-1598) Coring in SSL
[ https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504806#comment-13504806 ] James Peach commented on TS-1598: - Without more info, I don't think there's anything we can do here. Is this reproducible? Can you get a packet trace or verbose log? > Coring in SSL > - > > Key: TS-1598 > URL: https://issues.apache.org/jira/browse/TS-1598 > Project: Traffic Server > Issue Type: Bug > Components: SSL >Affects Versions: 3.2.0 > Environment: RHEL6.2 64bit >Reporter: Abhishek Nayani > Fix For: 3.2.0 > > > (gdb) bt > #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6 > #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10 > #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10 > #3 0x0066eaf7 in ssl_read_from_net (nh=, > vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at > SSLNetVConnection.cc:135 > #4 0x0066f3b0 in SSLNetVConnection::net_read_io > (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at > SSLNetVConnection.cc:288 > #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8, > event=, e=) at UnixNet.cc:381 > #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0, > calling_code=5) at I_Continuation.h:146 > #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5) > at UnixEThread.cc:142 > #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at > UnixEThread.cc:264 > #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88 > #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0 > #11 0x00390ace76dd in clone () from /lib64/libc.so.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1598) Coring in SSL
[ https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505195#comment-13505195 ] Abhishek Nayani commented on TS-1598: - This happens once or twice a day. I do not have particular request which can reproduce this. The traffic is predominantly WebDAV (CalDAV) over SSL. Let me know if you need any other info. I've a core but cannot attach it to this bug as it is > 3GB. > Coring in SSL > - > > Key: TS-1598 > URL: https://issues.apache.org/jira/browse/TS-1598 > Project: Traffic Server > Issue Type: Bug > Components: SSL >Affects Versions: 3.2.0 > Environment: RHEL6.2 64bit >Reporter: Abhishek Nayani > > (gdb) bt > #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6 > #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10 > #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10 > #3 0x0066eaf7 in ssl_read_from_net (nh=, > vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at > SSLNetVConnection.cc:135 > #4 0x0066f3b0 in SSLNetVConnection::net_read_io > (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at > SSLNetVConnection.cc:288 > #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8, > event=, e=) at UnixNet.cc:381 > #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0, > calling_code=5) at I_Continuation.h:146 > #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5) > at UnixEThread.cc:142 > #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at > UnixEThread.cc:264 > #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88 > #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0 > #11 0x00390ace76dd in clone () from /lib64/libc.so.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1598) Coring in SSL
[ https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505219#comment-13505219 ] James Peach commented on TS-1598: - ATS is calling: int rres = SSL_read(sslvc->ssl, b->end() + offset, (int)block_write_avail); And we end up here in OpenSSL: int ssl3_read_bytes(SSL *s, int type, unsigned char *buf, int len, int peek) ... if ((unsigned int)len > rr->length) n = rr->length; else n = (unsigned int)len; memcpy(buf,&(rr->data[rr->off]),n); So I'd guess that we screwed up IO buffer management somehow, or there's a OpenSSL bug that is screwing up the memcpy. Abhishek, what verion of OpenSSL are you using? Since you have a core, can you try to get the valued of "buf", "len" and "n" from ssl3_read_bytes? > Coring in SSL > - > > Key: TS-1598 > URL: https://issues.apache.org/jira/browse/TS-1598 > Project: Traffic Server > Issue Type: Bug > Components: SSL >Affects Versions: 3.2.0 > Environment: RHEL6.2 64bit >Reporter: Abhishek Nayani > > (gdb) bt > #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6 > #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10 > #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10 > #3 0x0066eaf7 in ssl_read_from_net (nh=, > vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at > SSLNetVConnection.cc:135 > #4 0x0066f3b0 in SSLNetVConnection::net_read_io > (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at > SSLNetVConnection.cc:288 > #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8, > event=, e=) at UnixNet.cc:381 > #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0, > calling_code=5) at I_Continuation.h:146 > #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5) > at UnixEThread.cc:142 > #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at > UnixEThread.cc:264 > #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88 > #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0 > #11 0x00390ace76dd in clone () from /lib64/libc.so.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1598) Coring in SSL
[ https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505353#comment-13505353 ] Abhishek Nayani commented on TS-1598: - OpenSSL: tps_openssl-0.9.8o_2 I am not able to get any data from that frame (ssl3_read_bytes). I've given Bryan Call access to these cores, so he might be able to get more info out of them. > Coring in SSL > - > > Key: TS-1598 > URL: https://issues.apache.org/jira/browse/TS-1598 > Project: Traffic Server > Issue Type: Bug > Components: SSL >Affects Versions: 3.2.0 > Environment: RHEL6.2 64bit >Reporter: Abhishek Nayani > > (gdb) bt > #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6 > #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10 > #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10 > #3 0x0066eaf7 in ssl_read_from_net (nh=, > vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at > SSLNetVConnection.cc:135 > #4 0x0066f3b0 in SSLNetVConnection::net_read_io > (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at > SSLNetVConnection.cc:288 > #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8, > event=, e=) at UnixNet.cc:381 > #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0, > calling_code=5) at I_Continuation.h:146 > #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5) > at UnixEThread.cc:142 > #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at > UnixEThread.cc:264 > #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88 > #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0 > #11 0x00390ace76dd in clone () from /lib64/libc.so.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1598) Coring in SSL
[ https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507515#comment-13507515 ] Bryan Call commented on TS-1598: so the line that starts this mess is SSLNetVConnection.cc:135: while (block_write_avail > 0) { int rres = SSL_read(sslvc->ssl, b->end() + offset, (int)block_write_avail); <--- this line (gdb) info locals rres = offset = s = 0x2b1f3c633350 sslvc = 0x2b1f3c633250 buf = @0x2b1f3c633378 bytes_read = 0 block_write_avail = 47430144907985 <--- if that is the number of blocks that seem crazy high __func__ = "ssl_read_from_net" b = 0x2b1f6b16f9b0 event = sslErr = 0 (gdb) p *b $19 = { = { = {_vptr.ForceVFPTToTop = 0x6a41b0}, m_refcount = 1}, _start = 0x0, _end = 0x0, _buf_end = 0x2b2330f05ad1 "\272j", _location = 0x6bcaa8 "memory/IOBuffer/HttpClientSession.cc:235", data = { m_ptr = 0x2b2088e31e00}, next = {m_ptr = 0x0}} (gdb) p *sslvc->ssl $20 = {version = 769, type = 8192, method = 0x3f96455840, rbio = 0x2b1fbceed1c0, wbio = 0x2b1fbceed1c0, bbio = 0x0, rwstate = 1, in_handshake = 0, handshake_func = 0x3f9621cd30 , server = 1, new_session = 0, quiet_shutdown = 1, shutdown = 0, state = 3, rstate = 240, init_buf = 0x0, init_msg = 0x2b235cf40a34, init_num = 0, init_off = 0, packet = 0x2b1f78d17d23 "\027\003\001\b\t\n\n \n \nhttp://calendarserver.org/ns/\"/>\n, session = 0x2b1fbd066e80, generate_session_id = 0, verify_mode = 0, verify_callback = 0, info_callback = 0, error = 0, error_code = 0, kssl_ctx = 0x2b1fbceed240, psk_client_callback = 0, psk_server_callback = 0, ctx = 0x1916080, debug = 0, verify_result = 0, ex_data = {sk = 0x2b1fbceed670, dummy = 0}, client_CA = 0x0, references = 1, options = 4194308, mode = 0, max_cert_list = 102400, first_packet = 0, client_version = 769, max_send_fragment = 16384, tlsext_debug_cb = 0, tlsext_debug_arg = 0x0, tlsext_hostname = 0x0, servername_done = 0, tlsext_status_type = -1, tlsext_status_expected = 0, tlsext_ocsp_ids = 0x0, tlsext_ocsp_exts = 0x0, tlsext_ocsp_resp = 0x0, tlsext_ocsp_resplen = -1, tlsext_ticket_expected = 0, tlsext_opaque_prf_input = 0x0, tlsext_opaque_prf_input_len = 0, tlsext_session_ticket = 0x0, tls_session_ticket_ext_cb = 0, tls_session_ticket_ext_cb_arg = 0x0, tls_session_secret_cb = 0, tls_session_secret_cb_arg = 0x0, initial_ctx = 0x1916080} > Coring in SSL > - > > Key: TS-1598 > URL: https://issues.apache.org/jira/browse/TS-1598 > Project: Traffic Server > Issue Type: Bug > Components: SSL >Affects Versions: 3.2.0 > Environment: RHEL6.2 64bit >Reporter: Abhishek Nayani > > (gdb) bt > #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6 > #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10 > #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10 > #3 0x0066eaf7 in ssl_read_from_net (nh=, > vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at > SSLNetVConnection.cc:135 > #4 0x0066f3b0 in SSLNetVConnection::net_read_io > (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at > SSLNetVConnection.cc:288 > #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8, > event=, e=) at UnixNet.cc:381 > #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0, > calling_code=5) at I_Continuation.h:146 > #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5) > at UnixEThread.cc:142 > #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at > UnixEThread.cc:264 > #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88 > #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0 > #11 0x00390ace76dd in clone () from /lib64/libc.so.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1598) Coring in SSL
[ https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507563#comment-13507563 ] Bryan Call commented on TS-1598: A few things to note: 1. The _start and _end are not set in the IOBufferBlock - default values of 0x0 2. When write_avail is called on the buffer block it takes the _buf_end pointer and subtracts _end giving the crazy high number of bytes available to write 3. b->end() + offset in the code will be a very low memory address since _end == 0x0 Looking at the MIOBuffer: (gdb) p *buf.mbuf $6 = {size_index = 47430144908000, water_mark = 0, _writer = { m_ptr = 0x2b1f6b16f9b0}, readers = {{accessor = 0x0, mbuf = 0x0, block = { m_ptr = 0x0}, start_offset = 0, size_limit = 9223372036854775807}, { accessor = 0x0, mbuf = 0x0, block = {m_ptr = 0x0}, start_offset = 0, size_limit = 9223372036854775807}, {accessor = 0x0, mbuf = 0x0, block = { m_ptr = 0x0}, start_offset = 0, size_limit = 9223372036854775807}, { accessor = 0x0, mbuf = 0x0, block = {m_ptr = 0x0}, start_offset = 0, size_limit = 9223372036854775807}, {accessor = 0x0, mbuf = 0x0, block = { m_ptr = 0x0}, start_offset = 0, size_limit = 9223372036854775807}}, _location = 0x6bcaa8 "memory/IOBuffer/HttpClientSession.cc:235"} > Coring in SSL > - > > Key: TS-1598 > URL: https://issues.apache.org/jira/browse/TS-1598 > Project: Traffic Server > Issue Type: Bug > Components: SSL >Affects Versions: 3.2.0 > Environment: RHEL6.2 64bit >Reporter: Abhishek Nayani > > (gdb) bt > #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6 > #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10 > #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10 > #3 0x0066eaf7 in ssl_read_from_net (nh=, > vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at > SSLNetVConnection.cc:135 > #4 0x0066f3b0 in SSLNetVConnection::net_read_io > (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at > SSLNetVConnection.cc:288 > #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8, > event=, e=) at UnixNet.cc:381 > #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0, > calling_code=5) at I_Continuation.h:146 > #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5) > at UnixEThread.cc:142 > #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at > UnixEThread.cc:264 > #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88 > #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0 > #11 0x00390ace76dd in clone () from /lib64/libc.so.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1598) Coring in SSL
[ https://issues.apache.org/jira/browse/TS-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13606877#comment-13606877 ] Bryan Call commented on TS-1598: Yes, close the bug. I believe it was fixed or they aren't having this issue anymore... > Coring in SSL > - > > Key: TS-1598 > URL: https://issues.apache.org/jira/browse/TS-1598 > Project: Traffic Server > Issue Type: Bug > Components: SSL >Affects Versions: 3.2.0 > Environment: RHEL6.2 64bit >Reporter: Abhishek Nayani >Assignee: Bryan Call > Fix For: sometime > > > (gdb) bt > #0 0x00390ac88c5b in memcpy () from /lib64/libc.so.6 > #1 0x003f962264ce in ssl3_read_bytes () from /usr/lib64/libssl.so.10 > #2 0x003f9670 in ?? () from /usr/lib64/libssl.so.10 > #3 0x0066eaf7 in ssl_read_from_net (nh=, > vc=0x2ada4437e0a0, lthread=0x2ada11ff2010, ret=@0x2ada174e5c10) at > SSLNetVConnection.cc:135 > #4 0x0066f3b0 in SSLNetVConnection::net_read_io > (this=0x2ada4437e0a0, nh=0x2ada11ff51e8, lthread=0x2ada11ff2010) at > SSLNetVConnection.cc:288 > #5 0x00676fb2 in NetHandler::mainNetEvent (this=0x2ada11ff51e8, > event=, e=) at UnixNet.cc:381 > #6 0x006a0ba4 in handleEvent (this=0x2ada11ff2010, e=0x24fdfc0, > calling_code=5) at I_Continuation.h:146 > #7 EThread::process_event (this=0x2ada11ff2010, e=0x24fdfc0, calling_code=5) > at UnixEThread.cc:142 > #8 0x006a16f3 in EThread::execute (this=0x2ada11ff2010) at > UnixEThread.cc:264 > #9 0x0069fae2 in spawn_thread_internal (a=0x268f1a0) at Thread.cc:88 > #10 0x00390b007851 in start_thread () from /lib64/libpthread.so.0 > #11 0x00390ace76dd in clone () from /lib64/libc.so.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira