According to traffic_ctl (which, btw, I love being able to query the current configs, as well as metrics), it's already set to 'thread'. Is it worth switching to 'global' or is it only the "global"->"thread" direction that would help?
I've never used ASAN before. Do I just need to use the flags you mentioned (-M -f -F) or do I need to recompile ATS with --enable-asan? I'm looking at the PPT presentation about it, but otherwise details about ATS+ASAN are a bit thin in the search results. I was able to get ATS to compile with --enable-asan, but I had to remove the libjansson package from the box where I was building. With libjansson installed (via the default Bionic package), ATS tries to compile the url_signing module and blows up with "relocation R_X86_64_PC32 against symbol `stdin@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC" (it's complaining about libjansson needing to be recompiled with -fPIC, not ATS). Are there any ill effects from compiling ATS without libjansson (beyond not building that experimental module that I don't use anyway)? (Incidentally, it'd be great to have a --configure flag to selectively avoid arbitrary experimental modules. I use tslua, so I need to enable experimental modules, but I would've liked an easy way to tell the build process "don't build url_signing") I'll have to do some testing after the weekend to see if the overhead of ASAN is too much. I'm running ATS on very nice boxes, so I'm hoping it'll be fine. If things work well enough to run the ASAN build in production on a box or two, what info am I looking for? I'm assuming something in diags.log or manager.log. Thanks! On Thu, May 27, 2021 at 9:44 AM Bryan Call <[email protected]> wrote: > Are you using per thread or global session pools? If you are using > global, it might be worth a try to switch to per thread ( > https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/files/records.config.en.html?highlight=records.config#proxy-config-http-server-session-sharing-pool > ). > > You can try turning off keep alive as referenced in the issue, but I have > a feeling it won’t help. > > If you can run an ASAN in production with > the proxy.config.proxy_binary_opts set to "-M -f -F” that would really help > to tracking this issue down. There is some overhead with running ASAN and > I don’t know if your servers can handle it ( > https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/files/records.config.en.html?highlight=records.config#proxy-config-proxy-binary-opts > ). > > -Bryan > > > On May 24, 2021, at 4:12 PM, Mark Moseley <[email protected]> wrote: > > I've been chasing an issue for a while now. This is 8.1.1 running on > Ubuntu Focal. I'm seeing this same crash (Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE`) > across different nodes that are handling unrelated traffic, but also ones > that don't mention HTTP_SM_MAGIC_ALIVE. > > Background: > ATS is doing reverse proxy caching in front of web hosting backends, so a > completely random assortment of backend domains and a random assortment of > file types. Customer domains resolve to a group of 3-node clusters. I'm > seeing the same crash across different clusters, so it wouldn't be related > to the same backend origin domain(s). Backends are derived via remap+tslua, > doing DNS lookups. These boxes constantly run at 200-400 reqs/seq all day, > but I'm only seeing this crash anywhere from "once every couple days" to a > couple of times a day on a given node (so the other millions of requests > per day are processed just fine). ATS sits behind a front-end nginx > instance (so nginx->ats is all localhost; origins are remote but on the > same LAN). > > It sounds similar to https://github.com/apache/trafficserver/issues/4921 > but that ticket also says that it was fixed for the OP in 8.1, so that > could be a red herring. > > Any patches/settings I can use to mitigate this? Would turning off > keepalives help as a bandaid? (Outbound keepalives to the origins would > hurt a bit; inbound would probably be fine due to nginx->ats being on > localhost). > > I'm also fine upgrading to 9.x but I ran into the issue where the > connection count grows without bound, despite the connections not really > being open (https://github.com/apache/trafficserver/issues/7471) > > I don't have any core files handy and these boxes have anywhere from > 64gb-224gb ATS RAM cache, so I imagine a core would be horrifyingly > large. The error is always similar to this (most of these are from the same > box but that's just out of convenience; I'm seeing the same variety of > segfaults across all of my boxes): > > May 24 07:05:14 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 24 07:05:14 cache0302 traffic_server[30820]: Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 24 07:05:14 cache0302 traffic_manager[21310]: [May 24 07:05:14.660] > traffic_crashlo NOTE: crashlog started, target=30820, debug=false > syslog=true, uid=114 euid=0 > May 24 07:05:14 cache0302 traffic_crashlog[30822]: traffic_crashlo NOTE: > crashlog started, target=30820, debug=false syslog=true, uid=114 euid=0 > May 24 07:05:14 cache0302 traffic_crashlog[30822]: traffic_crashlo NOTE: > logging to 0x14e10a33ad20 > May 24 07:05:14 cache0302 traffic_manager[21310]: [May 24 07:05:14.660] > traffic_crashlo NOTE: logging to 0x14e10a33ad20 > May 24 07:05:17 cache0302 traffic_manager[21310]: [May 24 07:05:17.094] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-24-070514.log > May 24 07:05:17 cache0302 traffic_crashlog[30822]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-24-070514.log > May 24 07:05:17 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 24 07:05:17 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x15ff15ac4ef0] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x675351928980] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x675350a20fb7] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x675350a22921] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x6753535bdb08] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x6753535ba7d5] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x15ff15b3cd0b] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x15ff15d70d32] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x15ff15d72329] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x15ff15d6324c] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x15ff15d64d72] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x15ff15db0f92] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x15ff15db2233] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x15ff15db091a] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x67535191d6db] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x675350b0371f] > > Sometimes I get this one: > > May 22 21:30:01 cache0102 traffic_manager[6292]: Fatal: HttpSM.cc:5854 > <http://httpsm.cc:5854>: failed assertion `server_session == nullptr` > May 22 21:30:01 cache0102 traffic_server[16246]: Fatal: HttpSM.cc:5854 > <http://httpsm.cc:5854>: failed assertion `server_session == nullptr` > May 22 21:30:01 cache0102 traffic_manager[6292]: [May 22 21:30:01.658] > traffic_crashlo NOTE: crashlog started, target=16246, debug=false > syslog=true, uid=114 euid=0 > May 22 21:30:01 cache0102 traffic_crashlog[16260]: traffic_crashlo NOTE: > crashlog started, target=16246, debug=false syslog=true, uid=114 euid=0 > May 22 21:30:01 cache0102 traffic_crashlog[16260]: traffic_crashlo NOTE: > logging to 0x18473c4cb6e0 > May 22 21:30:01 cache0102 traffic_manager[6292]: [May 22 21:30:01.659] > traffic_crashlo NOTE: logging to 0x18473c4cb6e0 > May 22 21:30:04 cache0102 traffic_manager[6292]: [May 22 21:30:04.197] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-213001.log > May 22 21:30:04 cache0102 traffic_crashlog[16260]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-213001.log > May 22 21:30:04 cache0102 traffic_manager[6292]: traffic_server: received > signal 6 (Aborted) > May 22 21:30:04 cache0102 traffic_manager[6292]: traffic_server - STACK > TRACE: > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x10b76b432ef0] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x62a740d4b980] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x62a73fe43fb7] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x62a73fe45921] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x62a7429e0b08] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x62a7429dd7d5] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN6HttpSM17get_http_scheduleEiPv+0x0)[0x10b76b4a2250] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN6HttpSM22state_http_server_openEiPv+0x2e2)[0x10b76b4af562] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x110)[0x10b76b4aade0] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN12CheckConnect14handle_connectEiP5Event+0x649)[0x10b76b6dc8e9] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(+0x36cd32)[0x10b76b6ded32] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0xd30)[0x10b76b6e77c0] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x27f)[0x10b76b6cec0f] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x295)[0x10b76b6ceef5] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x729)[0x10b76b720409] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(+0x3ac91a)[0x10b76b71e91a] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x62a740d406db] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x62a73ff2671f] > > > I also see: > > May 24 02:05:18 cache0302 traffic_manager[21310]: [May 24 02:05:18.715] > traffic_crashlo NOTE: crashlog started, target=28956, debug=false > syslog=true, uid=114 euid=0 > May 24 02:05:18 cache0302 traffic_manager[21310]: [May 24 02:05:18.715] > traffic_crashlo NOTE: logging to 0x1a09c40167a0 > May 24 02:05:18 cache0302 traffic_crashlog[28962]: traffic_crashlo NOTE: > crashlog started, target=28956, debug=false syslog=true, uid=114 euid=0 > May 24 02:05:18 cache0302 traffic_crashlog[28962]: traffic_crashlo NOTE: > logging to 0x1a09c40167a0 > May 24 02:05:21 cache0302 traffic_manager[21310]: [May 24 02:05:21.582] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-24-020518.log > May 24 02:05:21 cache0302 traffic_crashlog[28962]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-24-020518.log > May 24 02:05:21 cache0302 traffic_manager[21310]: traffic_server: received > signal 11 (Segmentation fault) > May 24 02:05:21 cache0302 traffic_manager[21310]: > traffic_servertraffic_server - STACK TRACE: > May 24 02:05:21 cache0302 traffic_manager[21310]: - STACK TRACE: > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x14d500a4fef0] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x733afa434980] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM21setup_server_transferEv+0x620)[0x14d500abe110] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7OpenDir11close_writeEP7CacheVC+0x1b8)[0x14d500c20d18] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x225)[0x14d500ac4095] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7CacheVC17openWriteCloseDirEiP5Event+0xab)[0x14d500c4ae9b] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x604)[0x14d500ac4a24] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN11HttpCacheSM8end_bothEv+0xa2)[0x14d500ace812] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xe37)[0x14d500acbf47] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM9kill_thisEv+0x83)[0x14d500ac7463] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x4f)[0x14d500ab4bbf] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x17f)[0x14d500ac7e4f] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xba)[0x14d500ac3f2a] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x14d500cfbd32] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x604)[0x14d500ac4a24] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x14d500cfd329] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM33state_read_server_response_headerEiPv+0x46d)[0x14d500ac594d] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x14d500cee24c] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x110)[0x14d500ac7de0] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x14d500cefd72] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection19readSignalAndUpdateEi+0x52)[0x14d500d00242] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x14d500d3bf92] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x17b5)[0x14d500cc8ec5] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x14d500d3d233] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x264)[0x14d500cebbf4] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x14d500d3b91a] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x295)[0x14d500cebef5] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x733afa4296db] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x729)[0x14d500d3d409] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x733af960f71f] > > > Sometimes it takes a day or two to crash, but sometimes it's much more > rapid. Here's a bunch from the span of a couple of hours: > > May 22 22:34:33 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 22 22:34:33 cache0302 traffic_server[22524]: Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 22 22:34:33 cache0302 traffic_manager[21310]: [May 22 22:34:33.988] > traffic_crashlo NOTE: crashlog started, target=22524, debug=false > syslog=true, uid=114 euid=0 > May 22 22:34:33 cache0302 traffic_crashlog[22527]: traffic_crashlo NOTE: > crashlog started, target=22524, debug=false syslog=true, uid=114 euid=0 > May 22 22:34:33 cache0302 traffic_manager[21310]: [May 22 22:34:33.989] > traffic_crashlo NOTE: logging to 0x10ba7153b000 > May 22 22:34:33 cache0302 traffic_crashlog[22527]: traffic_crashlo NOTE: > logging to 0x10ba7153b000 > May 22 22:34:36 cache0302 traffic_manager[21310]: [May 22 22:34:36.755] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-223433.log > May 22 22:34:36 cache0302 traffic_crashlog[22527]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-223433.log > May 22 22:34:36 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 22 22:34:36 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1bedf9b81ef0] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x75b896f0f980] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x75b896007fb7] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x75b896009921] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x75b898ba4b08] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x75b898ba17d5] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1bedf9bf9d0b] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x1bedf9e2dd32] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1bedf9e2f329] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x418)[0x1bedf9e21f18] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1bedf9e6df92] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1bedf9e6f233] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x1bedf9e6d91a] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x75b896f046db] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x75b8960ea71f] > May 22 22:34:46 cache0302 traffic_server[8819]: NOTE: --- traffic_server > Starting --- > May 22 22:34:46 cache0302 traffic_server[8819]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 > on Apr 9 2021 at 15:49:55) > May 22 22:41:09 cache0302 traffic_manager[21310]: [May 22 22:41:09.044] > traffic_crashlo NOTE: crashlog started, target=8819, debug=false > syslog=true, uid=114 euid=0 > May 22 22:41:09 cache0302 traffic_manager[21310]: [May 22 22:41:09.044] > traffic_crashlo NOTE: logging to 0x13fb55f0e020 > May 22 22:41:09 cache0302 traffic_crashlog[8825]: traffic_crashlo NOTE: > crashlog started, target=8819, debug=false syslog=true, uid=114 euid=0 > May 22 22:41:09 cache0302 traffic_crashlog[8825]: traffic_crashlo NOTE: > logging to 0x13fb55f0e020 > May 22 22:41:10 cache0302 traffic_manager[21310]: [May 22 22:41:10.463] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-224109.log > May 22 22:41:10 cache0302 traffic_crashlog[8825]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-224109.log > May 22 22:41:10 cache0302 traffic_manager[21310]: traffic_server: received > signal 11 (Segmentation fault) > May 22 22:41:10 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x191e98ab9ef0] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x67dee0e2f980] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel19finish_all_internalEP18HttpTunnelProducerb+0x177)[0x191e98b88567] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM21tunnel_handler_serverEiP18HttpTunnelProducer+0x60c)[0x191e98b1888c] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel16producer_handlerEiP18HttpTunnelProducer+0x58d)[0x191e98b84ffd] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x14a)[0x191e98b85b5a] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36d222)[0x191e98d66222] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x56f)[0x191e98d672cf] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x492)[0x191e98d59f92] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x191e98da5f92] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x191e98da7233] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x191e98da591a] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x67dee0e246db] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x67dee000a71f] > May 22 22:41:14 cache0302 traffic_server[11594]: NOTE: --- traffic_server > Starting --- > May 22 22:41:14 cache0302 traffic_server[11594]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 > on Apr 9 2021 at 15:49:55) > May 22 23:04:37 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 22 23:04:37 cache0302 traffic_server[11594]: Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 22 23:04:37 cache0302 traffic_manager[21310]: [May 22 23:04:37.520] > traffic_crashlo NOTE: crashlog started, target=11594, debug=false > syslog=true, uid=114 euid=0 > May 22 23:04:37 cache0302 traffic_crashlog[11600]: traffic_crashlo NOTE: > crashlog started, target=11594, debug=false syslog=true, uid=114 euid=0 > May 22 23:04:37 cache0302 traffic_crashlog[11600]: traffic_crashlo NOTE: > logging to 0x1decbe248a90 > May 22 23:04:37 cache0302 traffic_manager[21310]: [May 22 23:04:37.520] > traffic_crashlo NOTE: logging to 0x1decbe248a90 > May 22 23:04:38 cache0302 traffic_manager[21310]: [May 22 23:04:38.880] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-230437.log > May 22 23:04:38 cache0302 traffic_crashlog[11600]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-230437.log > May 22 23:04:38 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 22 23:04:38 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1867ffb06ef0] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x747f2bd26980] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x747f2ae1efb7] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x747f2ae20921] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x747f2d9bbb08] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x747f2d9b87d5] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1867ffb7ed0b] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x1867ffdb2d32] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1867ffdb4329] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x1867ffda524c] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x1867ffda6d72] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1867ffdf2f92] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1867ffdf4233] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x1867ffdf291a] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x747f2bd1b6db] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x747f2af0171f] > May 22 23:04:43 cache0302 traffic_server[23705]: NOTE: --- traffic_server > Starting --- > May 22 23:04:43 cache0302 traffic_server[23705]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 > on Apr 9 2021 at 15:49:55) > May 22 23:29:41 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:5854 > <http://httpsm.cc:5854>: failed assertion `server_session == nullptr` > May 22 23:29:41 cache0302 traffic_server[23705]: Fatal: HttpSM.cc:5854 > <http://httpsm.cc:5854>: failed assertion `server_session == nullptr` > May 22 23:29:41 cache0302 traffic_manager[21310]: [May 22 23:29:41.142] > traffic_crashlo NOTE: crashlog started, target=23705, debug=false > syslog=true, uid=114 euid=0 > May 22 23:29:41 cache0302 traffic_crashlog[23707]: traffic_crashlo NOTE: > crashlog started, target=23705, debug=false syslog=true, uid=114 euid=0 > May 22 23:29:41 cache0302 traffic_crashlog[23707]: traffic_crashlo NOTE: > logging to 0x19db57d47390 > May 22 23:29:41 cache0302 traffic_manager[21310]: [May 22 23:29:41.142] > traffic_crashlo NOTE: logging to 0x19db57d47390 > May 22 23:29:42 cache0302 traffic_manager[21310]: [May 22 23:29:42.544] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-232941.log > May 22 23:29:42 cache0302 traffic_crashlog[23707]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-232941.log > May 22 23:29:42 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 22 23:29:42 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x147b5d156ef0] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x71f05ed35980] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x71f05de2dfb7] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x71f05de2f921] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x71f0609cab08] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x71f0609c77d5] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM17get_http_scheduleEiPv+0x0)[0x147b5d1c6250] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM22state_http_server_openEiPv+0x2e2)[0x147b5d1d3562] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x110)[0x147b5d1cede0] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN12CheckConnect14handle_connectEiP5Event+0x649)[0x147b5d4008e9] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x147b5d402d32] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0xd30)[0x147b5d40b7c0] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x27f)[0x147b5d3f2c0f] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x295)[0x147b5d3f2ef5] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x729)[0x147b5d444409] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x147b5d44291a] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x71f05ed2a6db] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x71f05df1071f] > May 22 23:29:46 cache0302 traffic_server[2493]: NOTE: --- traffic_server > Starting --- > May 22 23:29:46 cache0302 traffic_server[2493]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 > on Apr 9 2021 at 15:49:55) > May 22 23:43:15 cache0302 traffic_manager[21310]: [May 22 23:43:15.997] > traffic_crashlo NOTE: crashlog started, target=2493, debug=false > syslog=true, uid=114 euid=0 > May 22 23:43:15 cache0302 traffic_crashlog[2495]: traffic_crashlo NOTE: > crashlog started, target=2493, debug=false syslog=true, uid=114 euid=0 > May 22 23:43:15 cache0302 traffic_manager[21310]: [May 22 23:43:15.997] > traffic_crashlo NOTE: logging to 0x16dba7520110 > May 22 23:43:15 cache0302 traffic_crashlog[2495]: traffic_crashlo NOTE: > logging to 0x16dba7520110 > May 22 23:43:17 cache0302 traffic_manager[21310]: [May 22 23:43:17.331] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-234315.log > May 22 23:43:17 cache0302 traffic_crashlog[2495]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-234315.log > May 22 23:43:17 cache0302 traffic_manager[21310]: traffic_server: received > signal 11 (Segmentation fault) > May 22 23:43:17 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1e354ceb6ef0] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x6ac17bd5b980] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel19finish_all_internalEP18HttpTunnelProducerb+0x177)[0x1e354cf85567] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM21tunnel_handler_serverEiP18HttpTunnelProducer+0x60c)[0x1e354cf1588c] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel16producer_handlerEiP18HttpTunnelProducer+0x58d)[0x1e354cf81ffd] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x14a)[0x1e354cf82b5a] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36d222)[0x1e354d163222] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x56f)[0x1e354d1642cf] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x492)[0x1e354d156f92] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1e354d1a2f92] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1e354d1a4233] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x1e354d1a291a] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x6ac17bd506db] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x6ac17af3671f] > May 22 23:43:21 cache0302 traffic_server[9064]: NOTE: --- traffic_server > Starting --- > May 22 23:43:21 cache0302 traffic_server[9064]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 > on Apr 9 2021 at 15:49:55) > May 22 23:51:29 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 22 23:51:29 cache0302 traffic_server[9064]: Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 22 23:51:29 cache0302 traffic_manager[21310]: [May 22 23:51:29.199] > traffic_crashlo NOTE: crashlog started, target=9064, debug=false > syslog=true, uid=114 euid=0 > May 22 23:51:29 cache0302 traffic_crashlog[9066]: traffic_crashlo NOTE: > crashlog started, target=9064, debug=false syslog=true, uid=114 euid=0 > May 22 23:51:29 cache0302 traffic_crashlog[9066]: traffic_crashlo NOTE: > logging to 0x153045b30140 > May 22 23:51:29 cache0302 traffic_manager[21310]: [May 22 23:51:29.199] > traffic_crashlo NOTE: logging to 0x153045b30140 > May 22 23:51:30 cache0302 traffic_manager[21310]: [May 22 23:51:30.396] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-235129.log > May 22 23:51:30 cache0302 traffic_crashlog[9066]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-235129.log > May 22 23:51:30 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 22 23:51:30 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1cfb65150ef0] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x76d5135c1980] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x76d5126b9fb7] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x76d5126bb921] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x76d515256b08] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x76d5152537d5] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1cfb651c8d0b] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x1cfb653fcd32] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1cfb653fe329] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x1cfb653ef24c] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x1cfb653f0d72] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1cfb6543cf92] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1cfb6543e233] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x1cfb6543c91a] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x76d5135b66db] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x76d51279c71f] > May 22 23:51:33 cache0302 traffic_server[12951]: NOTE: --- traffic_server > Starting --- > May 22 23:51:33 cache0302 traffic_server[12951]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 > on Apr 9 2021 at 15:49:55) > May 23 00:08:16 cache0302 traffic_manager[21310]: [May 23 00:08:16.507] > traffic_crashlo NOTE: crashlog started, target=12951, debug=false > syslog=true, uid=114 euid=0 > May 23 00:08:16 cache0302 traffic_manager[21310]: [May 23 00:08:16.507] > traffic_crashlo NOTE: logging to 0x19c404cfdad0 > May 23 00:08:16 cache0302 traffic_crashlog[12953]: traffic_crashlo NOTE: > crashlog started, target=12951, debug=false syslog=true, uid=114 euid=0 > May 23 00:08:16 cache0302 traffic_crashlog[12953]: traffic_crashlo NOTE: > logging to 0x19c404cfdad0 > May 23 00:08:17 cache0302 traffic_manager[21310]: [May 23 00:08:17.776] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-23-000816.log > May 23 00:08:17 cache0302 traffic_crashlog[12953]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-23-000816.log > May 23 00:08:17 cache0302 traffic_manager[21310]: traffic_server: received > signal 11 (Segmentation fault) > May 23 00:08:17 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x19c0a1524ef0] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x6d52207fd980] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7OpenDir11close_writeEP7CacheVC+0x1b8)[0x19c0a16f5d18] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7CacheVC17openWriteCloseDirEiP5Event+0xab)[0x19c0a171fe9b] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN11HttpCacheSM8end_bothEv+0xa2)[0x19c0a15a3812] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM9kill_thisEv+0x83)[0x19c0a159c463] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x17f)[0x19c0a159ce4f] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x19c0a17d0d32] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x19c0a17d2329] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x19c0a17c324c] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x19c0a17c4d72] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x19c0a1810f92] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x19c0a1812233] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x19c0a181091a] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x6d52207f26db] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x6d521f9d871f] > May 23 00:08:21 cache0302 traffic_server[20789]: NOTE: --- traffic_server > Starting --- > May 23 00:08:21 cache0302 traffic_server[20789]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 > on Apr 9 2021 at 15:49:55) > May 23 00:14:46 cache0302 traffic_manager[21310]: [May 23 00:14:46.669] > traffic_crashlo NOTE: crashlog started, target=20789, debug=false > syslog=true, uid=114 euid=0 > May 23 00:14:46 cache0302 traffic_crashlog[20796]: traffic_crashlo NOTE: > crashlog started, target=20789, debug=false syslog=true, uid=114 euid=0 > May 23 00:14:46 cache0302 traffic_manager[21310]: [May 23 00:14:46.669] > traffic_crashlo NOTE: logging to 0x14f170d7b660 > May 23 00:14:46 cache0302 traffic_crashlog[20796]: traffic_crashlo NOTE: > logging to 0x14f170d7b660 > May 23 00:14:47 cache0302 traffic_manager[21310]: [May 23 00:14:47.825] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-23-001446.log > May 23 00:14:47 cache0302 traffic_crashlog[20796]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-23-001446.log > May 23 00:14:47 cache0302 traffic_manager[21310]: traffic_server: received > signal 11 (Segmentation fault) > May 23 00:14:47 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x18c9c6353ef0] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x69e99706b980] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel19finish_all_internalEP18HttpTunnelProducerb+0x177)[0x18c9c6422567] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM21tunnel_handler_serverEiP18HttpTunnelProducer+0x60c)[0x18c9c63b288c] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel16producer_handlerEiP18HttpTunnelProducer+0x58d)[0x18c9c641effd] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x14a)[0x18c9c641fb5a] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36d222)[0x18c9c6600222] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x56f)[0x18c9c66012cf] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x492)[0x18c9c65f3f92] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x18c9c663ff92] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x18c9c6641233] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x18c9c663f91a] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x69e9970606db] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x69e99624671f] > May 23 00:14:51 cache0302 traffic_server[23441]: NOTE: --- traffic_server > Starting --- > May 23 00:14:51 cache0302 traffic_server[23441]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 > on Apr 9 2021 at 15:49:55) > May 23 00:18:52 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 23 00:18:52 cache0302 traffic_server[23441]: Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 23 00:18:52 cache0302 traffic_manager[21310]: [May 23 00:18:52.120] > traffic_crashlo NOTE: crashlog started, target=23441, debug=false > syslog=true, uid=114 euid=0 > May 23 00:18:52 cache0302 traffic_manager[21310]: [May 23 00:18:52.120] > traffic_crashlo NOTE: logging to 0x12c5d40cced0 > May 23 00:18:52 cache0302 traffic_crashlog[23443]: traffic_crashlo NOTE: > crashlog started, target=23441, debug=false syslog=true, uid=114 euid=0 > May 23 00:18:52 cache0302 traffic_crashlog[23443]: traffic_crashlo NOTE: > logging to 0x12c5d40cced0 > May 23 00:18:53 cache0302 traffic_manager[21310]: [May 23 00:18:53.113] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-23-001852.log > May 23 00:18:53 cache0302 traffic_crashlog[23443]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-23-001852.log > May 23 00:18:53 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 23 00:18:53 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1b7f44513ef0] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7244e5c04980] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7244e4cfcfb7] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7244e4cfe921] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x7244e7899b08] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x7244e78967d5] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1b7f4458bd0b] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x1b7f447bfd32] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1b7f447c1329] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x418)[0x1b7f447b3f18] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1b7f447fff92] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1b7f44801233] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x1b7f447ff91a] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7244e5bf96db] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7244e4ddf71f] > May 23 00:18:56 cache0302 traffic_server[26362]: NOTE: --- traffic_server > Starting --- > May 23 00:18:56 cache0302 traffic_server[26362]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 > on Apr 9 2021 at 15:49:55) > May 23 00:24:24 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 23 00:24:24 cache0302 traffic_server[26362]: Fatal: HttpSM.cc:2533 > <http://httpsm.cc:2533>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 23 00:24:24 cache0302 traffic_manager[21310]: [May 23 00:24:24.239] > traffic_crashlo NOTE: crashlog started, target=26362, debug=false > syslog=true, uid=114 euid=0 > May 23 00:24:24 cache0302 traffic_manager[21310]: [May 23 00:24:24.240] > traffic_crashlo NOTE: logging to 0x190a947ba530 > May 23 00:24:24 cache0302 traffic_crashlog[26365]: traffic_crashlo NOTE: > crashlog started, target=26362, debug=false syslog=true, uid=114 euid=0 > May 23 00:24:24 cache0302 traffic_crashlog[26365]: traffic_crashlo NOTE: > logging to 0x190a947ba530 > May 23 00:24:25 cache0302 traffic_manager[21310]: [May 23 00:24:25.274] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-23-002424.log > May 23 00:24:25 cache0302 traffic_crashlog[26365]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-23-002424.log > May 23 00:24:25 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 23 00:24:25 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1cf361726ef0] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x6d01e9b4d980] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x6d01e8c45fb7] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x6d01e8c47921] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x6d01eb7e2b08] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x6d01eb7df7d5] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1cf36179ed0b] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x1cf3619d2d32] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1cf3619d4329] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x1cf3619c524c] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x1cf3619c6d72] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1cf361a12f92] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1cf361a14233] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x1cf361a1291a] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x6d01e9b426db] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x6d01e8d2871f] > May 23 00:24:28 cache0302 traffic_server[28956]: NOTE: --- traffic_server > Starting --- > May 23 00:24:28 cache0302 traffic_server[28956]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 > on Apr 9 2021 at 15:49:55) > > >
