[
https://issues.apache.org/jira/browse/TS-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328722#comment-15328722
]
James Peach commented on TS-4195:
---------------------------------
I see this maybe 25% of the time. Here's a suspicious backtrace in SSL code:
{code}
^C[TrafficManager] ==> Cleaning up and reissuing signal #2
traffic_server: Interrupt (Signal sent by the kernel 0 0)
traffic_server: Interrupt (Signal sent by kill() 12962 0)
*** Error in `traffic_server: Segmentation fault (Signal sent by the kernel
[(nil)])
traffic_server - STACK TRACE:
/opt/ats/bin/traffic_server':
/opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xc3)[0x50c999]
/lib64/libpthread.so.0(+0x10a00)[0x2ac70e453a00]
/lib64/libc.so.6(+0x3953d)[0x2ac70f13953d]
/lib64/libc.so.6(+0x39635)[0x2ac70f139635]
/opt/ats/bin/traffic_servertraffic_server: Aborted (Signal sent by tkill()
12968 99)
traffic_server - STACK TRACE:
/opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xc3)[0x50c999]
/lib64/libpthread.so.0(+0x10a00)[0x2ac70e453a00]
/lib64/libc.so.6(gsignal+0x38)[0x2ac70f134a28]
/lib64/libc.so.6(abort+0x16a)[0x2ac70f13662a]
/lib64/libc.so.6(+0x77d7a)[0x2ac70f177d7a]
/lib64/libc.so.6(+0x801ca)[0x2ac70f1801ca]
/lib64/libc.so.6(cfree+0x4c)[0x2ac70f18372c]
/opt/ats/lib/libtsutil.so.7(ats_free+0x28)[0x2ac70c856cea]
[0x53eaa8]
/opt/ats/bin/traffic_server(_Z8ssl_freePv/lib64/libpthread.so.0(+0x10a00)[0x2ac70e453a00]
+0x/lib64/libc.so.618()nanosleep[0x+0x7b92802d]
)[0x2ac70f1c84cd]
/lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x2ac70d5662ed]
/lib64/libcrypto.so.10(lh_free+0x57)[0x2ac70d61be97]
/lib64/libssl.so.10(SSL_CTX_free+0x7c)[0x2ac70d2c93ec]
/lib64/libc.so.6(sleep+0xd4)[0x2ac70f1c8364]
/opt/ats/bin/traffic_server(_Z14mgmt_sleep_seci+0x15)[0x710e29]
/opt/ats/bin/traffic_server(_Z17SSLReleaseContextP10ssl_ctx_st+0x18)[0x7be54b]
/opt/ats/bin/traffic_server(_ZN15SSLNetProcessor7cleanupEv+0x1c)[0x7af3b6]
/opt/ats/bin/traffic_server(_ZN15SSLNetProcessorD2Ev+0x24)[0x7af6f2]
/opt/ats/bin/traffic_server(_Z19startProcessManagerPv+0x137)[0x70b832]
/lib64/libpthread.so.0(+0x761a)[0x2ac70e44a61a]
/lib64/libc.so.6(clone+0x6d)[0x2ac70f20259d]
/lib64/libc.so.6(+0x395e8)[0x2ac70f1395e8]
/lib64/libc.so.6(+0x39635)[0x2ac70f139635]
[TrafficManager] ==> signal #2
{code}
> <ctrl-c>out of traffic_manager causes a double free in traffic_server
> ---------------------------------------------------------------------
>
> Key: TS-4195
> URL: https://issues.apache.org/jira/browse/TS-4195
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
> Reporter: Leif Hedstrom
> Assignee: Bryan Call
> Priority: Blocker
> Fix For: 7.0.0
>
>
> While testing stuff, I was running traffic_manager from command line, and
> then I get a crash from traffic_server:
> {code}
> root@loki 407/0 # ./bin/traffic_manager
> [E. Mgmt] log ==> [TrafficManager] using root directory '/opt/ats'
> traffic_server: using root directory '/opt/ats'
> ^C[TrafficManager] ==> Cleaning up and reissuing signal #2
> traffic_server: Interrupt (Signal sent by the kernel 0 0)
> 9083 sent by kill()*** Error in `/opt/ats/bin/traffic_server': corrupted
> double-linked list: 0x00000000028f8940 ***
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x77da5)[0x2ad58f3fcda5]
> /lib64/libc.so.6(+0x80c06)[0x2ad58f405c06]
> /lib64/libc.so.6(cfree+0x4c)[0x2ad58f408cac]
> /lib64/libc.so.6(+0x39685)[0x2ad58f3be685]
> /lib64/libc.so.6(+0x396a5)[0x2ad58f3be6a5]
> /opt/ats/bin/traffic_server[0x4e300atraffic_server: Segmentation fault
> (Address not mapped to object [0x55b02140])
> traffic_server - STACK TRACE:
> /lib64/libc.so.6(nanosleep+0x2d)[0x2ad58f44d7ad]
> /opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x8e)[0x4abece]
> /lib64/libpthread.so.0(+0x109f0)[0x2ad58e3709f0]
> /lib64/libc.so.6(sleep+0xd4)[0x2ad58f44d644]
> /opt/ats/bin/traffic_server(_Z19startProcessManagerPv+0xb1)[0x69b8a1]
> /lib64/libpthread.so.0(+0x760a)[0x2ad58e36760a]
> /lib64/libc.so.6(clone+0x6d)[0x2ad58f487a4d]
> ======= Memory map: ========
> /lib64/libc.so.6(+0x395ad)[0x2ad58f3be5ad]
> 00400000-008a6000 r-xp 00000000 00:24 1775473
> /opt/ats/bin/traffic_server
> 00aa6000-00ab3000 r--p 004a6000 00:24 1775473
> /opt/ats/bin/traffic_server
> 00ab3000-00ab9000 rw-p 004b3000 00:24 1775473
> /opt/ats/bin/traffic_server
> 00ab9000-01097000 rw-p 00000000 00:00 0
> 028dd000-02cb9000 rw-p 00000000 00:00 0
> [heap]
> 2ad58c52c000-2ad58c54d000 r-xp 00000000 00:24 1389899
> /usr/lib64/ld-2.22.so
> 2ad58c54d000-2ad58c550000 rw-p 00000000 00:00 0
> 2ad58c550000-2ad58c560000 rwxp 00000000 00:00 0
> 2ad58c56b000-2ad58c6ed000 rw-p 00000000 00:00 0
> 2ad58c6ed000-2ad58c6fd000 rwxp 00000000 00:00 0
> 2ad58c6fd000-2ad58c748000 rw-p 00000000 00:00 0
> 2ad58c74c000-2ad58c74d000 r--p 00020000 00:24 1389899
> /usr/lib64/ld-2.22.so
> 2ad58c74d000-2ad58c74e000 rw-p 00021000 00:24 1389899
> /usr/lib64/ld-2.22.so
> 2ad58c74e000-2ad58c74f000 rw-p 00000000 00:00 0
> 2ad58c74f000-2ad58c790000 r-xp 00000000 00:24 1775306
> /opt/ats/lib/libtsutil.so.6.2.0
> 2ad58c790000-2ad58c990000 ---p 00041000 00:24 1775306
> /opt/ats/lib/libtsutil.so.6.2.0
> 2ad58c990000-2ad58c991000 r--p 00041000 00:24 1775306
> /opt/ats/lib/libtsutil.so.6.2.0
> 2ad58c991000-2ad58c993000 rw-p 00042000 00:24 1775306
> /opt/ats/lib/libtsutil.so.6.2.0
> 2ad58c993000-2ad58c994000 rw-p 00000000 00:00 0
> 2ad58c994000-2ad58c9cb000 r-xp 00000000 00:24 1393339
> /usr/lib64/libhwloc.so.5.6.6
> 2ad58c9cb000-2ad58cbcb000 ---p 00037000 00:24 1393339
> /usr/lib64/libhwloc.so.5.6.6
> 2ad58cbcb000-2ad58cbcc000 r--p 00037000 00:24 1393339
> /usr/lib64/libhwloc.so.5.6.6
> 2ad58cbcc000-2ad58cbcd000 rw-p 00038000 00:24 1393339
> /usr/lib64/libhwloc.so.5.6.6
> 2ad58cbcd000-2ad58cd77000 r-xp 00000000 00:24 1441754
> /usr/lib64/libtcl8.6.so
> 2ad58cd77000-2ad58cf77000 ---p 001aa000 00:24 1441754
> /lib64/libc.so.6( /usr/lib64/libtcl8.6.so
> 2ad58cf77000-2ad58cf86000 r--p 001aa000 00:24 1441754
> /usr/lib64/libtcl8.6.so
> 2ad58cf86000-2ad58cf87000 rw-p 001b9000 00:24 1441754
> /usr/lib64/libtcl8.6.so
> 2ad58cf87000-2ad58cf88000 rw-p 00000000 00:00 0
> 2ad58cf88000-2ad58cf9f000 r-xp 00000000 00:24 1389936
> /usr/lib64/libresolv-2.22.so
> 2ad58cf9f000-2ad58d19f000 ---p 00017000 00:24 1389936
> /usr/lib64/libresolv-2.22.so
> 2ad58d19f000-2ad58d1a0000 r--p 00017000 00:24 1389936
> /usr/lib64/libresolv-2.22.so
> 2ad58d1a0000-2ad58d1a1000 rw-p 00018000 00:24 1389936
> /usr/lib64/libresolv-2.22.so
> 2ad58d1a1000-2ad58d1a3000 rw-p 00000000 00:00 0
> 2ad58d1a3000-2ad58d212000 r-xp 00000000 00:24 1635380
> /usr/lib64/libssl.so.1.0.2f
> 2ad58d212000-2ad58d411000 ---p 0006f000 00:24 1635380
> /usr/lib64/libssl.so.1.0.2f
> 2ad58d411000-2ad58d416000 r--p 0006e000 00:24 1635380
> /usr/lib64/libssl.so.1.0.2f
> 2ad58d416000-2ad58d41d000 rw-p 00073000 00:24 1635380
> /usr/lib64/libssl.so.1.0.2f
> 2ad58d41d000-2ad58d64d000 r-xp 00000000 00:24 1635378
> /usr/lib64/libcrypto.so.1.0.2f
> 2ad58d64d000-2ad58d84c000 ---p 00230000 00:24 1635378
> /usr/lib64/libcrypto.so.1.0.2f
> 2ad58d84c000-2ad58d868000 r--p 0022f000 00:24 1635378
> /usr/lib64/libcrypto.so.1.0.2f
> 2ad58d868000-2ad58d875000 rw-p 0024b000 00:24 1635378
> /usr/lib64/libcrypto.so.1.0.2f
> 2ad58d875000-2ad58d879000 rw-p 00000000 00:00 0
> 2ad58d879000-2ad58d87d000 r-xp 00000000 00:24 1390959
> /usr/lib64/libcap.so.2.24
> 2ad58d87d000-2ad58da7c000 ---p 00004000 00:24 1390959
> /usr/lib64/libcap.so.2.24
> 2ad58da7c000-2ad58da7d000 r--p 00003000 00:24 1390959
> /usr/lib64/libcap.so.2.24
> 2ad58da7d000-2ad58da7e000 rw-p 00004000 00:24 1390959
> /usr/lib64/libcap.so.2.24
> 2ad58da7e000-2ad58daed000 r-xp 00000000 00:24 1390389 +0x
> /usr/lib64/libpcre.so.1.2.6
> 2ad58daed000-2ad58dcec000 ---p 0006f000 00:24 1390389
> /usr/lib64/libpcre.so.1.2.6
> 2ad58dcec000-2ad58dced000 r--p 0006e000 00:24 1390389
> /usr/lib64/libpcre.so.1.2.6
> 2ad58dced000-2ad58dcee000 rw-p 0006f000 00:24 1390389
> /usr/lib64/libpcre.so.1.2.6
> 2ad58dcee000-2ad58dd13000 r-xp 00000000 00:24 1390456
> /usr/lib64/liblzma.so.5.2.1
> 2ad58dd13000-2ad58df12000 ---p 00025000 00:24 1390456
> /usr/lib64/liblzma.so.5.2.1
> 2ad58df12000-2ad58df13000 r--p 00024000 00:24 1390456
> /usr/lib64/liblzma.so.5.2.1
> 2ad58df13000-2ad58df14000 rw-p 00000000 00:00 0
> 2ad58df14000-2ad58df29000 r-xp 00000000 00:24 1390421
> /usr/lib64/libz.so.1.2.8
> 2ad58df29000-2ad58e128000 ---p 00015000 00:24 1390421
> /usr/lib64/libz.so.1.2.8
> 2ad58e128000-2ad58e129000 r--p 00014000 00:24 1390421
> /usr/lib64/libz.so.1.2.8
> 2ad58e129000-2ad58e12a000 rw-p 00015000 00:24 1390421
> /usr/lib64/libz.so.1.2.8
> 2ad58e12a000-2ad58e131000 r-xp 00000000 00:24 1389910
> /usr/lib64/libcrypt-2.22.so
> 2ad58e131000-2ad58e330000 ---p 00007000 00:24 1389910
> /usr/lib64/libcrypt-2.22.so
> 2ad58e330000-2ad58e331000 r--p 00006000 00:24 1389910
> /usr/lib64/libcrypt-2.22.so
> 2ad58e331000-2ad58e332000 rw-p 00007000 00:24 1389910
> /usr/lib64/libcrypt-2.22.so
> 2ad58e332000-2ad58e360000 rw-p 00000000 00:00 0
> 2ad58e360000-2ad58e378000 r-xp 00000000 00:24 1389934
> /usr/lib64/libpthread-2.22.so
> 2ad58e378000-2ad58e577000 ---p 00018000 00:24 1389934
> /usr/lib64/libpthread-2.22.so
> 2ad58e577000-2ad58e578000 r--p 00017000 00:24 1389934
> /usr/lib64/libpthread-2.22.so
> 2ad58e578000-2ad58e579000 rw-p 00018000 00:24 1389934
> /usr/lib64/libpthread-2.22.so
> 2ad58e579000-2ad58e57d000 rw-p 00000000 00:00 0
> 2ad58e57d000-2ad58e580000 r-xp 00000000 00:396a5)[0x2ad58f3be6a524 1389912
> /usr/lib64/libdl-2.22.so
> 2ad58e580000-2ad58e77f000 ---p 00003000 00:24 1389912
> /usr/lib64/libdl-2.22.so
> 2ad58e77f000-2ad58e780000 r--p 00002000 00:24 1389912
> /usr/lib64/libdl-2.22.so
> 2ad58e780000-2ad58e781000 rw-p 00003000 00:24 1389912
> /usr/lib64/libdl-2.22.so
> 2ad58e781000-2ad58e8df000 r-xp 00000000 00:24 1390463
> /usr/lib64/libxml2.so.2.9.3
> 2ad58e8df000-2ad58eadf000 ---p 0015e000 00:24 1390463
> /usr/lib64/libxml2.so.2.9.3
> 2ad58eadf000-2ad58eae7000 r--p 0015e000 00:24 1390463
> /usr/lib64/libxml2.so.2.9.3
> 2ad58eae7000-2ad58eae9000 rw-p 00166000 00:24 1390463
> /usr/lib64/libxml2.so.2.9.3
> 2ad58eae9000-2ad58eaea000 rw-p 00000000 00:00 0
> 2ad58eaea000-2ad58ec5c000 r-xp 00000000 00:24 1390225
> /usr/lib64/libstdc++.so.6.0.21
> 2ad58ec5c000-2ad58ee5c000 ---p 00172000 00:24 1390225
> /usr/lib64/libstdc++.so.6.0.21
> 2ad58ee5c000-2]
> ad58ee66000 r--p 00172000 00:24 1390225
> /usr/lib64/libstdc++.so.6.0.21
> 2ad58ee66000-2ad58ee68000 rw-p 0017c000 00:24 1390225
> /usr/lib64/libstdc++.so.6.0.21
> 2ad58ee68000-2ad58ee6c000 rw-p 00000000 00:00 0
> 2ad58ee6c000-2ad58ef6d000 r-xp 00000000 00:24 1389914
> /usr/lib64/libm-2.22.so
> 2ad58ef6d000-2ad58f16c000 ---p 00101000 00:24 1389914
> /usr/lib64/libm-2.22.so
> 2ad58f16c000-2ad58f16d000 r--p 00100000 00:24 1389914
> /usr/lib64/libm-2.22.so
> 2ad58f16d000-2ad58f16e000 rw-p 00101000 00:24 1389914
> /usr/lib64/libm-2.22.so
> 2ad58f16e000-2ad58f184000 r-xp 00000000 00:24 1381023
> /usr/lib64/libgcc_s-5.3.1-20151207.so.1
> 2ad58f184000-2ad58f383000 ---p 00016000 00:24 1381023
> /usr/lib64/libgcc_s-5.3.1-20151207.so.1
> 2ad58f383000-2ad58f384000 r--p 00015000 00:24 1381023
> /usr/lib64/libgcc_s-5.3.1-20151207.so.1
> 2ad58f384000-2ad58f385000 rw-p 00016000 00:24 1381023
> /usr/lib64/libgcc_s-5.3.1-20151207.so.1
> 2ad58f385000-2ad58f53c000 r-xp 00000000 00:24 1389906
> /usr/lib64/libc-2.22.so
> 2ad58f53c000-2ad58f73c000 ---p 001b7000 00:24 1389906
> /usr/lib64/libc-2.22.so
> 2ad58f73c000-2ad58f740000 r--p 001b7000 00:24 1389906
> /usr/lib64/libc-2.22.so
> 2ad58f740000-2ad58f742000 rw-p 001bb000 00:24 1389906
> /usr/lib64/libc-2.22.so
> 2ad58f742000-2ad58f746000 rw-p 00000000 00:00 0
> 2ad58f746000-2ad58f750000 r-xp 00000000 00:24 1393293
> /usr/lib64/libnuma.so.1.0.0
> 2ad58f750000-2ad58f94f000 ---p 0000a000 00:24 1393293
> /usr/lib64/libnuma.so.1.0.0
> 2ad58f94f000-2ad58f950000 r--p 00009000 00:24 1393293
> /usr/lib64/libnuma.so.1.0.0
> 2ad58f950000-2ad58f951000 rw-p 0000a000 00:24 1393293
> /usr/lib64/libnuma.so.1.0.0
> 2ad58f951000-2ad58f95a000 r-xp 00000000 00:24 1669444
> /usr/lib64/libltdl.so.7.3.1
> 2ad58f95a000-2ad58fb590/opt/ats/bin/traffic_server[0x4e300a]
> /lib64/libpthread.so.0
> 9091 sent by tkill()(+0x109f0)traffic_server[0x2ad58e3709f0]
> - STACK TRACE:
> /lib64/libc.so.6(epoll_wait+0x33)[0x2ad58f488043]
> /opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x8e)[0x4abece]
> /lib64/libpthread.so.0(+0x109f0)[0x2ad58e3709f0]
> /lib64/libc.so.6(gsignal+0x38)[0x2ad58f3b9a98]
> /lib64/libc.so.6(abort+0x16a)[0x2ad58f3bb69a]
> /opt/ats/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x6f)[0x77c53f]
> /opt/ats/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8a)[0x7c37ea]
> /opt/ats/bin/traffic_server(_ZN7EThread7executeEv+0x619)[0x7c4449]
> /opt/ats/bin/traffic_server(main+0x16fa)[0x4915ea]
> /lib64/libc.so.6(__libc_start_main+0xf0)[0x2ad58f3a5580]
> /lib64/libc.so.6(+0x77daa)[0x2ad58f3fcdaa]
> /opt/ats/bin/traffic_server(_start+0x29)[0x496b79]
> [TrafficManager] ==> signal #2
> {code}
> [~bcall] got one using ASAN, which seems to point towards a double free.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
