Awesome, that works perfectly. Thanks! I don't know what the mount option defaults are for /run on CentOS and other distros, but looking at some Ubuntu Bionic and Focal boxes, 'noexec' is the default for /run. It might be worth emitting an error about it if traffic_server crashes trying to load dynamic modules (and specifically when it fails that mmap() of the newly copied dynamic module) and proxy.config.local_state_dir is mounted on a 'noexec' volume. I.e. if traffic_server fails that mmap(), then it consults /proc/self/mounts to determine mount options for proxy.config.local_state_dir?
Though again, this could also just be specific to my environment that this is even an issue. On Wed, May 12, 2021 at 6:07 AM Bryan Call <[email protected]> wrote: > This is due to the plugin reload feature added to ATS 9.0.0 ( > https://docs.trafficserver.apache.org/en/9.0.x/developer-guide/design-documents/reloading-plugins.en.html). > You can disable the feature by > setting proxy.config.plugin.dynamic_reload_mode to 0. > > -Bryan > > > On May 11, 2021, at 5:08 PM, Mark Moseley <[email protected]> wrote: > > This is partially a FYI for Google-ing, but I'd love some guidance from > the ATS devs on how to fix this issue in a 'better' way, as well as if I > should reopen a 8.x github issue (see next paragraph). > > Background: > I run 8.1.1 in a fairly traffic-heavy environment with ATS (sitting behind > nginx) doing reverse-proxying to backend webservers. It's ubuntu bionic, > running a grsecurity-enabled kernel (I haven't been able to test the below > scenario on a non-grsec kernel). I've been seeing some crashes lately with > 8.1.1 that look like https://github.com/apache/trafficserver/issues/4921, > i.e. "Fatal: HttpSM.cc:2533 <http://httpsm.cc:2533>: failed assertion > `magic == HTTP_SM_MAGIC_ALIVE`" (and yes, I see the comment in the ticket > that someone should reopen the ticket if they see that on 8.1.x, but I > haven't gotten around to it yet -- see the first line of this email). > > Today: > So due to those 8.1.1 crashes, I was trying out 9.0.1 and ran into the > same issue that I initially ran into with 9.0.0: traffic_manager starts up > and then starts traffic_server, and traffic_server crashes with: > > [May 11 18:36:41.343] traffic_server ERROR: [ReverseProxy] failed to add > remap rule at /etc/trafficserver/remap.config line 198: > /run/trafficserver/37882f60-2d8b-45e0-b8d8-ed2208c4c221/usr/lib/trafficserver/modules/tslua.so: > failed to map segment from shared objectfailed to remove runtime copy: > Success > > and traffic_manager dutifully tries again and again forever. > > And we use Lua to do all of our remapping (basically, to set the correct > DNS name for ATS to look up), so it's a must-have for us. > > After some strace'ing, I figured out that Bionic by default has /run > mounted with 'noexec'. If I remount /run with 'exec', I don't get this > error anymore and everything loads up just fine. > > In the straces I was looking at, it looks like traffic_server is copying > the tslua.so module into that temp directory under /run/trafficserver and > then trying to mmap() the newly copied tslua.so file in /run and failing: > > openat(AT_FDCWD, > "/run/trafficserver/e667e0ea-fc3b-43fd-945d-63fc4d9dc6e4/usr/lib/trafficserver/modules/tslua.so", > O_RDONLY|O_CLOEXEC) = 38 > read(38, "\177ELF\2\1\1\0\0\0<deleted for brevity sake>0\0\22\0\0\0", 832) > = 832 > fstat(38, {st_mode=S_IFREG|0644, st_size=152304, ...}) = 0 > mmap(NULL, 2251784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 38, 0) > = -1 EPERM (Operation not permitted) > close(38) = 0 > > With /run mounted with 'exec', that mmap() succeeds and life is great. > > So for the ATS devs: > > * Is this just me (probably due to my somewhat obscure kernel)? Typically > I'd see system log entries and/or dmesg in the case of grsecurity blocking > it, but nothing here. > > * Did I miss something in the upgrade notes? I can't find anything about > this behavior (copying modules to a dir under /run/trafficserver, instead > of mmap'ing them directly from /usr/lib/). The configs I'm using were > untouched from 8.1.1 (haven't yet removed the deprecated stuff). Normally, > I'd chalk stuff up to operator error (and this still likely is), but this > one seemed kind of weird. > > * Is there something I can set to prevent that behavior? I'd rather not > remount boxes' /run (not to mention, it's done by systemd automatically, so > it's not clear where to modify mount flags for /run; everything has a > *.mount unit file except for /run itself). Ideally, traffic_server would > just mmap /usr/lib/trafficserver/modules/tslua.so directly. But if that's > not possible, is there a setting to override the location? I tried > setting proxy.config.local_state_dir to something outside of /run; lock > files and sockets get created but traffic_server immediately dies). I > created /tmp/trafficserver and used that for local_state_dir, but > traffic_manager complains that: > > Fatal: failed to connect management socket > '/run/trafficserver/processerver.sock': No such file or directory > > so it's still using /run/trafficserver for some things. I guess there's > more to it than just the proxy.config.local_state_dir setting. And > "traffic_ctl config match . | grep run" returns only the > "proxy.config.alarm.script_runtime" setting (If I override local_state_dir, > of course, from /run/trafficserver). > > * For the 8.1.1 "failed assertion `magic == HTTP_SM_MAGIC_ALIVE`" issue, > should I reopen that github issue or start a new one? > > Thanks! > > >
