[ https://issues.apache.org/jira/browse/TS-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431872#comment-15431872 ]
Bryan Call commented on TS-4195: -------------------------------- I also see this one: {noformat} ==23255==ERROR: AddressSanitizer: attempting double-free on 0x619000000f80 in thread T0 ([ET_NET 0]): #0 0x2b24e55bbac0 in free (/lib64/libasan.so.3+0xc6ac0) #1 0x2b24e9155214 in __run_exit_handlers (/lib64/libc.so.6+0x39214) #2 0x2b24e9155234 in __GI_exit (/lib64/libc.so.6+0x39234) #3 0x587239 in proxy_signal_handler /home/bcall/dev/apache/trafficserver/proxy/Main.cc:409 #4 0x2b24e80fac2f (/lib64/libpthread.so.0+0x10c2f) #5 0x2b24e921f4b2 in __GI_epoll_wait (/lib64/libc.so.6+0x1034b2) #6 0xc31421 in NetHandler::mainNetEvent(int, Event*) /home/bcall/dev/apache/trafficserver/iocore/net/UnixNet.cc:423 #7 0xd13ce0 in Continuation::handleEvent(int, void*) /home/bcall/dev/apache/trafficserver/iocore/eventsystem/I_Continuation.h:153 #8 0xd13ce0 in EThread::process_event(Event*, int) /home/bcall/dev/apache/trafficserver/iocore/eventsystem/UnixEThread.cc:148 #9 0xd16bc6 in EThread::execute() /home/bcall/dev/apache/trafficserver/iocore/eventsystem/UnixEThread.cc:275 #10 0x49ac50 in main /home/bcall/dev/apache/trafficserver/proxy/Main.cc:1956 #11 0x2b24e913c730 in __libc_start_main (/lib64/libc.so.6+0x20730) #12 0x4aa598 in _start (/usr/local/bin/traffic_server+0x4aa598) 0x619000000f80 is located 0 bytes inside of 1040-byte region [0x619000000f80,0x619000001390) freed by thread T1 here: ================================================================= #0 0x2b24e55bbac0 in free (/lib64/libasan.so.3+0xc6ac0) #1 0x2b24e9155214 in __run_exit_handlers (/lib64/libc.so.6+0x39214) ==23255==ERROR: LeakSanitizer: detected memory leaks previously allocated by thread T0 ([ET_NET 0]) here: Direct leak of 257 byte(s) in 1 object(s) allocated from: #0 0x2b24e55bbfe0 in calloc (/lib64/libasan.so.3+0xc6fe0) #1 0x2b24e91553e8 in __new_exitfn (/lib64/libc.so.6+0x393e8) #0 0x2b24e55bbe20 in malloc (/lib64/libasan.so.3+0xc6e20) Thread T1 created by T0 ([ET_NET 0]) here: #1 0x2b24e64f30d5 in ats_malloc /home/bcall/dev/apache/trafficserver/lib/ts/ink_memory.cc:59 #2 0x4988d5 in ats_scoped_str::ats_scoped_str(unsigned long) ../lib/ts/ink_memory.h:442 #3 0x4988d5 in main /home/bcall/dev/apache/trafficserver/proxy/Main.cc:1608 #0 0x2b24e5526458 in pthread_create (/lib64/libasan.so.3+0x31458) #4 0x2b24e913c730 in __libc_start_main (/lib64/libc.so.6+0x20730) #1 0x49833c in ink_thread_create ../lib/ts/ink_thread.h:147 #2 0x49833c in ProcessManager::start() ../mgmt/ProcessManager.h:65 #3 0x49833c in initialize_process_manager /home/bcall/dev/apache/trafficserver/proxy/Main.cc:503 #4 0x49833c in main /home/bcall/dev/apache/trafficserver/proxy/Main.cc:1567 Direct leak of 40 byte(s) in 1 object(s) allocated from: #5 0x2b24e913c730 in __libc_start_main (/lib64/libc.so.6+0x20730) #0 0x2b24e55bbe20 in malloc (/lib64/libasan.so.3+0xc6e20) SUMMARY: AddressSanitizer: double-free (/lib64/libasan.so.3+0xc6ac0) in free #1 0x2b24e64f30d5 in ats_malloc /home/bcall/dev/apache/trafficserver/lib/ts/ink_memory.cc:59 ==23255==ABORTING #2 0xd13640 in Thread::start(char const*, unsigned long, void* (*)(void*), void*) /home/bcall/dev/apache/trafficserver/iocore/eventsystem/Thread.cc:92 [TrafficManager] ==> signal #2 {noformat} > <ctrl-c>out of traffic_manager causes a double free in traffic_server > --------------------------------------------------------------------- > > Key: TS-4195 > URL: https://issues.apache.org/jira/browse/TS-4195 > Project: Traffic Server > Issue Type: Bug > Components: Core > Reporter: Leif Hedstrom > Assignee: Bryan Call > Priority: Blocker > Fix For: 7.0.0 > > > While testing stuff, I was running traffic_manager from command line, and > then I get a crash from traffic_server: > {code} > root@loki 407/0 # ./bin/traffic_manager > [E. Mgmt] log ==> [TrafficManager] using root directory '/opt/ats' > traffic_server: using root directory '/opt/ats' > ^C[TrafficManager] ==> Cleaning up and reissuing signal #2 > traffic_server: Interrupt (Signal sent by the kernel 0 0) > 9083 sent by kill()*** Error in `/opt/ats/bin/traffic_server': corrupted > double-linked list: 0x00000000028f8940 *** > ======= Backtrace: ========= > /lib64/libc.so.6(+0x77da5)[0x2ad58f3fcda5] > /lib64/libc.so.6(+0x80c06)[0x2ad58f405c06] > /lib64/libc.so.6(cfree+0x4c)[0x2ad58f408cac] > /lib64/libc.so.6(+0x39685)[0x2ad58f3be685] > /lib64/libc.so.6(+0x396a5)[0x2ad58f3be6a5] > /opt/ats/bin/traffic_server[0x4e300atraffic_server: Segmentation fault > (Address not mapped to object [0x55b02140]) > traffic_server - STACK TRACE: > /lib64/libc.so.6(nanosleep+0x2d)[0x2ad58f44d7ad] > /opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x8e)[0x4abece] > /lib64/libpthread.so.0(+0x109f0)[0x2ad58e3709f0] > /lib64/libc.so.6(sleep+0xd4)[0x2ad58f44d644] > /opt/ats/bin/traffic_server(_Z19startProcessManagerPv+0xb1)[0x69b8a1] > /lib64/libpthread.so.0(+0x760a)[0x2ad58e36760a] > /lib64/libc.so.6(clone+0x6d)[0x2ad58f487a4d] > ======= Memory map: ======== > /lib64/libc.so.6(+0x395ad)[0x2ad58f3be5ad] > 00400000-008a6000 r-xp 00000000 00:24 1775473 > /opt/ats/bin/traffic_server > 00aa6000-00ab3000 r--p 004a6000 00:24 1775473 > /opt/ats/bin/traffic_server > 00ab3000-00ab9000 rw-p 004b3000 00:24 1775473 > /opt/ats/bin/traffic_server > 00ab9000-01097000 rw-p 00000000 00:00 0 > 028dd000-02cb9000 rw-p 00000000 00:00 0 > [heap] > 2ad58c52c000-2ad58c54d000 r-xp 00000000 00:24 1389899 > /usr/lib64/ld-2.22.so > 2ad58c54d000-2ad58c550000 rw-p 00000000 00:00 0 > 2ad58c550000-2ad58c560000 rwxp 00000000 00:00 0 > 2ad58c56b000-2ad58c6ed000 rw-p 00000000 00:00 0 > 2ad58c6ed000-2ad58c6fd000 rwxp 00000000 00:00 0 > 2ad58c6fd000-2ad58c748000 rw-p 00000000 00:00 0 > 2ad58c74c000-2ad58c74d000 r--p 00020000 00:24 1389899 > /usr/lib64/ld-2.22.so > 2ad58c74d000-2ad58c74e000 rw-p 00021000 00:24 1389899 > /usr/lib64/ld-2.22.so > 2ad58c74e000-2ad58c74f000 rw-p 00000000 00:00 0 > 2ad58c74f000-2ad58c790000 r-xp 00000000 00:24 1775306 > /opt/ats/lib/libtsutil.so.6.2.0 > 2ad58c790000-2ad58c990000 ---p 00041000 00:24 1775306 > /opt/ats/lib/libtsutil.so.6.2.0 > 2ad58c990000-2ad58c991000 r--p 00041000 00:24 1775306 > /opt/ats/lib/libtsutil.so.6.2.0 > 2ad58c991000-2ad58c993000 rw-p 00042000 00:24 1775306 > /opt/ats/lib/libtsutil.so.6.2.0 > 2ad58c993000-2ad58c994000 rw-p 00000000 00:00 0 > 2ad58c994000-2ad58c9cb000 r-xp 00000000 00:24 1393339 > /usr/lib64/libhwloc.so.5.6.6 > 2ad58c9cb000-2ad58cbcb000 ---p 00037000 00:24 1393339 > /usr/lib64/libhwloc.so.5.6.6 > 2ad58cbcb000-2ad58cbcc000 r--p 00037000 00:24 1393339 > /usr/lib64/libhwloc.so.5.6.6 > 2ad58cbcc000-2ad58cbcd000 rw-p 00038000 00:24 1393339 > /usr/lib64/libhwloc.so.5.6.6 > 2ad58cbcd000-2ad58cd77000 r-xp 00000000 00:24 1441754 > /usr/lib64/libtcl8.6.so > 2ad58cd77000-2ad58cf77000 ---p 001aa000 00:24 1441754 > /lib64/libc.so.6( /usr/lib64/libtcl8.6.so > 2ad58cf77000-2ad58cf86000 r--p 001aa000 00:24 1441754 > /usr/lib64/libtcl8.6.so > 2ad58cf86000-2ad58cf87000 rw-p 001b9000 00:24 1441754 > /usr/lib64/libtcl8.6.so > 2ad58cf87000-2ad58cf88000 rw-p 00000000 00:00 0 > 2ad58cf88000-2ad58cf9f000 r-xp 00000000 00:24 1389936 > /usr/lib64/libresolv-2.22.so > 2ad58cf9f000-2ad58d19f000 ---p 00017000 00:24 1389936 > /usr/lib64/libresolv-2.22.so > 2ad58d19f000-2ad58d1a0000 r--p 00017000 00:24 1389936 > /usr/lib64/libresolv-2.22.so > 2ad58d1a0000-2ad58d1a1000 rw-p 00018000 00:24 1389936 > /usr/lib64/libresolv-2.22.so > 2ad58d1a1000-2ad58d1a3000 rw-p 00000000 00:00 0 > 2ad58d1a3000-2ad58d212000 r-xp 00000000 00:24 1635380 > /usr/lib64/libssl.so.1.0.2f > 2ad58d212000-2ad58d411000 ---p 0006f000 00:24 1635380 > /usr/lib64/libssl.so.1.0.2f > 2ad58d411000-2ad58d416000 r--p 0006e000 00:24 1635380 > /usr/lib64/libssl.so.1.0.2f > 2ad58d416000-2ad58d41d000 rw-p 00073000 00:24 1635380 > /usr/lib64/libssl.so.1.0.2f > 2ad58d41d000-2ad58d64d000 r-xp 00000000 00:24 1635378 > /usr/lib64/libcrypto.so.1.0.2f > 2ad58d64d000-2ad58d84c000 ---p 00230000 00:24 1635378 > /usr/lib64/libcrypto.so.1.0.2f > 2ad58d84c000-2ad58d868000 r--p 0022f000 00:24 1635378 > /usr/lib64/libcrypto.so.1.0.2f > 2ad58d868000-2ad58d875000 rw-p 0024b000 00:24 1635378 > /usr/lib64/libcrypto.so.1.0.2f > 2ad58d875000-2ad58d879000 rw-p 00000000 00:00 0 > 2ad58d879000-2ad58d87d000 r-xp 00000000 00:24 1390959 > /usr/lib64/libcap.so.2.24 > 2ad58d87d000-2ad58da7c000 ---p 00004000 00:24 1390959 > /usr/lib64/libcap.so.2.24 > 2ad58da7c000-2ad58da7d000 r--p 00003000 00:24 1390959 > /usr/lib64/libcap.so.2.24 > 2ad58da7d000-2ad58da7e000 rw-p 00004000 00:24 1390959 > /usr/lib64/libcap.so.2.24 > 2ad58da7e000-2ad58daed000 r-xp 00000000 00:24 1390389 +0x > /usr/lib64/libpcre.so.1.2.6 > 2ad58daed000-2ad58dcec000 ---p 0006f000 00:24 1390389 > /usr/lib64/libpcre.so.1.2.6 > 2ad58dcec000-2ad58dced000 r--p 0006e000 00:24 1390389 > /usr/lib64/libpcre.so.1.2.6 > 2ad58dced000-2ad58dcee000 rw-p 0006f000 00:24 1390389 > /usr/lib64/libpcre.so.1.2.6 > 2ad58dcee000-2ad58dd13000 r-xp 00000000 00:24 1390456 > /usr/lib64/liblzma.so.5.2.1 > 2ad58dd13000-2ad58df12000 ---p 00025000 00:24 1390456 > /usr/lib64/liblzma.so.5.2.1 > 2ad58df12000-2ad58df13000 r--p 00024000 00:24 1390456 > /usr/lib64/liblzma.so.5.2.1 > 2ad58df13000-2ad58df14000 rw-p 00000000 00:00 0 > 2ad58df14000-2ad58df29000 r-xp 00000000 00:24 1390421 > /usr/lib64/libz.so.1.2.8 > 2ad58df29000-2ad58e128000 ---p 00015000 00:24 1390421 > /usr/lib64/libz.so.1.2.8 > 2ad58e128000-2ad58e129000 r--p 00014000 00:24 1390421 > /usr/lib64/libz.so.1.2.8 > 2ad58e129000-2ad58e12a000 rw-p 00015000 00:24 1390421 > /usr/lib64/libz.so.1.2.8 > 2ad58e12a000-2ad58e131000 r-xp 00000000 00:24 1389910 > /usr/lib64/libcrypt-2.22.so > 2ad58e131000-2ad58e330000 ---p 00007000 00:24 1389910 > /usr/lib64/libcrypt-2.22.so > 2ad58e330000-2ad58e331000 r--p 00006000 00:24 1389910 > /usr/lib64/libcrypt-2.22.so > 2ad58e331000-2ad58e332000 rw-p 00007000 00:24 1389910 > /usr/lib64/libcrypt-2.22.so > 2ad58e332000-2ad58e360000 rw-p 00000000 00:00 0 > 2ad58e360000-2ad58e378000 r-xp 00000000 00:24 1389934 > /usr/lib64/libpthread-2.22.so > 2ad58e378000-2ad58e577000 ---p 00018000 00:24 1389934 > /usr/lib64/libpthread-2.22.so > 2ad58e577000-2ad58e578000 r--p 00017000 00:24 1389934 > /usr/lib64/libpthread-2.22.so > 2ad58e578000-2ad58e579000 rw-p 00018000 00:24 1389934 > /usr/lib64/libpthread-2.22.so > 2ad58e579000-2ad58e57d000 rw-p 00000000 00:00 0 > 2ad58e57d000-2ad58e580000 r-xp 00000000 00:396a5)[0x2ad58f3be6a524 1389912 > /usr/lib64/libdl-2.22.so > 2ad58e580000-2ad58e77f000 ---p 00003000 00:24 1389912 > /usr/lib64/libdl-2.22.so > 2ad58e77f000-2ad58e780000 r--p 00002000 00:24 1389912 > /usr/lib64/libdl-2.22.so > 2ad58e780000-2ad58e781000 rw-p 00003000 00:24 1389912 > /usr/lib64/libdl-2.22.so > 2ad58e781000-2ad58e8df000 r-xp 00000000 00:24 1390463 > /usr/lib64/libxml2.so.2.9.3 > 2ad58e8df000-2ad58eadf000 ---p 0015e000 00:24 1390463 > /usr/lib64/libxml2.so.2.9.3 > 2ad58eadf000-2ad58eae7000 r--p 0015e000 00:24 1390463 > /usr/lib64/libxml2.so.2.9.3 > 2ad58eae7000-2ad58eae9000 rw-p 00166000 00:24 1390463 > /usr/lib64/libxml2.so.2.9.3 > 2ad58eae9000-2ad58eaea000 rw-p 00000000 00:00 0 > 2ad58eaea000-2ad58ec5c000 r-xp 00000000 00:24 1390225 > /usr/lib64/libstdc++.so.6.0.21 > 2ad58ec5c000-2ad58ee5c000 ---p 00172000 00:24 1390225 > /usr/lib64/libstdc++.so.6.0.21 > 2ad58ee5c000-2] > ad58ee66000 r--p 00172000 00:24 1390225 > /usr/lib64/libstdc++.so.6.0.21 > 2ad58ee66000-2ad58ee68000 rw-p 0017c000 00:24 1390225 > /usr/lib64/libstdc++.so.6.0.21 > 2ad58ee68000-2ad58ee6c000 rw-p 00000000 00:00 0 > 2ad58ee6c000-2ad58ef6d000 r-xp 00000000 00:24 1389914 > /usr/lib64/libm-2.22.so > 2ad58ef6d000-2ad58f16c000 ---p 00101000 00:24 1389914 > /usr/lib64/libm-2.22.so > 2ad58f16c000-2ad58f16d000 r--p 00100000 00:24 1389914 > /usr/lib64/libm-2.22.so > 2ad58f16d000-2ad58f16e000 rw-p 00101000 00:24 1389914 > /usr/lib64/libm-2.22.so > 2ad58f16e000-2ad58f184000 r-xp 00000000 00:24 1381023 > /usr/lib64/libgcc_s-5.3.1-20151207.so.1 > 2ad58f184000-2ad58f383000 ---p 00016000 00:24 1381023 > /usr/lib64/libgcc_s-5.3.1-20151207.so.1 > 2ad58f383000-2ad58f384000 r--p 00015000 00:24 1381023 > /usr/lib64/libgcc_s-5.3.1-20151207.so.1 > 2ad58f384000-2ad58f385000 rw-p 00016000 00:24 1381023 > /usr/lib64/libgcc_s-5.3.1-20151207.so.1 > 2ad58f385000-2ad58f53c000 r-xp 00000000 00:24 1389906 > /usr/lib64/libc-2.22.so > 2ad58f53c000-2ad58f73c000 ---p 001b7000 00:24 1389906 > /usr/lib64/libc-2.22.so > 2ad58f73c000-2ad58f740000 r--p 001b7000 00:24 1389906 > /usr/lib64/libc-2.22.so > 2ad58f740000-2ad58f742000 rw-p 001bb000 00:24 1389906 > /usr/lib64/libc-2.22.so > 2ad58f742000-2ad58f746000 rw-p 00000000 00:00 0 > 2ad58f746000-2ad58f750000 r-xp 00000000 00:24 1393293 > /usr/lib64/libnuma.so.1.0.0 > 2ad58f750000-2ad58f94f000 ---p 0000a000 00:24 1393293 > /usr/lib64/libnuma.so.1.0.0 > 2ad58f94f000-2ad58f950000 r--p 00009000 00:24 1393293 > /usr/lib64/libnuma.so.1.0.0 > 2ad58f950000-2ad58f951000 rw-p 0000a000 00:24 1393293 > /usr/lib64/libnuma.so.1.0.0 > 2ad58f951000-2ad58f95a000 r-xp 00000000 00:24 1669444 > /usr/lib64/libltdl.so.7.3.1 > 2ad58f95a000-2ad58fb590/opt/ats/bin/traffic_server[0x4e300a] > /lib64/libpthread.so.0 > 9091 sent by tkill()(+0x109f0)traffic_server[0x2ad58e3709f0] > - STACK TRACE: > /lib64/libc.so.6(epoll_wait+0x33)[0x2ad58f488043] > /opt/ats/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x8e)[0x4abece] > /lib64/libpthread.so.0(+0x109f0)[0x2ad58e3709f0] > /lib64/libc.so.6(gsignal+0x38)[0x2ad58f3b9a98] > /lib64/libc.so.6(abort+0x16a)[0x2ad58f3bb69a] > /opt/ats/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x6f)[0x77c53f] > /opt/ats/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8a)[0x7c37ea] > /opt/ats/bin/traffic_server(_ZN7EThread7executeEv+0x619)[0x7c4449] > /opt/ats/bin/traffic_server(main+0x16fa)[0x4915ea] > /lib64/libc.so.6(__libc_start_main+0xf0)[0x2ad58f3a5580] > /lib64/libc.so.6(+0x77daa)[0x2ad58f3fcdaa] > /opt/ats/bin/traffic_server(_start+0x29)[0x496b79] > [TrafficManager] ==> signal #2 > {code} > [~bcall] got one using ASAN, which seems to point towards a double free. -- This message was sent by Atlassian JIRA (v6.3.4#6332)