This is due to the plugin reload feature added to ATS 9.0.0 (https://docs.trafficserver.apache.org/en/9.0.x/developer-guide/design-documents/reloading-plugins.en.html <https://docs.trafficserver.apache.org/en/9.0.x/developer-guide/design-documents/reloading-plugins.en.html>). You can disable the feature by setting proxy.config.plugin.dynamic_reload_mode to 0.
-Bryan > On May 11, 2021, at 5:08 PM, Mark Moseley <[email protected]> wrote: > > This is partially a FYI for Google-ing, but I'd love some guidance from the > ATS devs on how to fix this issue in a 'better' way, as well as if I should > reopen a 8.x github issue (see next paragraph). > > Background: > I run 8.1.1 in a fairly traffic-heavy environment with ATS (sitting behind > nginx) doing reverse-proxying to backend webservers. It's ubuntu bionic, > running a grsecurity-enabled kernel (I haven't been able to test the below > scenario on a non-grsec kernel). I've been seeing some crashes lately with > 8.1.1 that look like https://github.com/apache/trafficserver/issues/4921 > <https://github.com/apache/trafficserver/issues/4921>, i.e. "Fatal: > HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE`" (and yes, I > see the comment in the ticket that someone should reopen the ticket if they > see that on 8.1.x, but I haven't gotten around to it yet -- see the first > line of this email). > > Today: > So due to those 8.1.1 crashes, I was trying out 9.0.1 and ran into the same > issue that I initially ran into with 9.0.0: traffic_manager starts up and > then starts traffic_server, and traffic_server crashes with: > > [May 11 18:36:41.343] traffic_server ERROR: [ReverseProxy] failed to add > remap rule at /etc/trafficserver/remap.config line 198: > /run/trafficserver/37882f60-2d8b-45e0-b8d8-ed2208c4c221/usr/lib/trafficserver/modules/tslua.so: > failed to map segment from shared objectfailed to remove runtime copy: > Success > > and traffic_manager dutifully tries again and again forever. > > And we use Lua to do all of our remapping (basically, to set the correct DNS > name for ATS to look up), so it's a must-have for us. > > After some strace'ing, I figured out that Bionic by default has /run mounted > with 'noexec'. If I remount /run with 'exec', I don't get this error anymore > and everything loads up just fine. > > In the straces I was looking at, it looks like traffic_server is copying the > tslua.so module into that temp directory under /run/trafficserver and then > trying to mmap() the newly copied tslua.so file in /run and failing: > > openat(AT_FDCWD, > "/run/trafficserver/e667e0ea-fc3b-43fd-945d-63fc4d9dc6e4/usr/lib/trafficserver/modules/tslua.so", > O_RDONLY|O_CLOEXEC) = 38 > read(38, "\177ELF\2\1\1\0\0\0<deleted for brevity sake>0\0\22\0\0\0", 832) = > 832 > fstat(38, {st_mode=S_IFREG|0644, st_size=152304, ...}) = 0 > mmap(NULL, 2251784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 38, 0) = > -1 EPERM (Operation not permitted) > close(38) = 0 > > With /run mounted with 'exec', that mmap() succeeds and life is great. > > So for the ATS devs: > > * Is this just me (probably due to my somewhat obscure kernel)? Typically I'd > see system log entries and/or dmesg in the case of grsecurity blocking it, > but nothing here. > > * Did I miss something in the upgrade notes? I can't find anything about this > behavior (copying modules to a dir under /run/trafficserver, instead of > mmap'ing them directly from /usr/lib/). The configs I'm using were untouched > from 8.1.1 (haven't yet removed the deprecated stuff). Normally, I'd chalk > stuff up to operator error (and this still likely is), but this one seemed > kind of weird. > > * Is there something I can set to prevent that behavior? I'd rather not > remount boxes' /run (not to mention, it's done by systemd automatically, so > it's not clear where to modify mount flags for /run; everything has a *.mount > unit file except for /run itself). Ideally, traffic_server would just mmap > /usr/lib/trafficserver/modules/tslua.so directly. But if that's not possible, > is there a setting to override the location? I tried setting > proxy.config.local_state_dir to something outside of /run; lock files and > sockets get created but traffic_server immediately dies). I created > /tmp/trafficserver and used that for local_state_dir, but traffic_manager > complains that: > > Fatal: failed to connect management socket > '/run/trafficserver/processerver.sock': No such file or directory > > so it's still using /run/trafficserver for some things. I guess there's more > to it than just the proxy.config.local_state_dir setting. And "traffic_ctl > config match . | grep run" returns only the > "proxy.config.alarm.script_runtime" setting (If I override local_state_dir, > of course, from /run/trafficserver). > > * For the 8.1.1 "failed assertion `magic == HTTP_SM_MAGIC_ALIVE`" issue, > should I reopen that github issue or start a new one? > > Thanks!
