Do you know if the /tmp/ directory is also noexec? I am wondering if there is a better place to copy the reloadable plugin on startup.
I created a new issue on this: https://github.com/apache/trafficserver/issues/7820 -Bryan > On May 12, 2021, at 7:26 AM, Mark Moseley <[email protected]> wrote: > > Awesome, that works perfectly. Thanks! > > I don't know what the mount option defaults are for /run on CentOS and other > distros, but looking at some Ubuntu Bionic and Focal boxes, 'noexec' is the > default for /run. It might be worth emitting an error about it if > traffic_server crashes trying to load dynamic modules (and specifically when > it fails that mmap() of the newly copied dynamic module) and > proxy.config.local_state_dir is mounted on a 'noexec' volume. I.e. if > traffic_server fails that mmap(), then it consults /proc/self/mounts to > determine mount options for proxy.config.local_state_dir? > > Though again, this could also just be specific to my environment that this is > even an issue. > > > On Wed, May 12, 2021 at 6:07 AM Bryan Call <[email protected] > <mailto:[email protected]>> wrote: > This is due to the plugin reload feature added to ATS 9.0.0 > (https://docs.trafficserver.apache.org/en/9.0.x/developer-guide/design-documents/reloading-plugins.en.html > > <https://docs.trafficserver.apache.org/en/9.0.x/developer-guide/design-documents/reloading-plugins.en.html>). > You can disable the feature by setting > proxy.config.plugin.dynamic_reload_mode to 0. > > -Bryan > > >> On May 11, 2021, at 5:08 PM, Mark Moseley <[email protected] >> <mailto:[email protected]>> wrote: >> >> This is partially a FYI for Google-ing, but I'd love some guidance from the >> ATS devs on how to fix this issue in a 'better' way, as well as if I should >> reopen a 8.x github issue (see next paragraph). >> >> Background: >> I run 8.1.1 in a fairly traffic-heavy environment with ATS (sitting behind >> nginx) doing reverse-proxying to backend webservers. It's ubuntu bionic, >> running a grsecurity-enabled kernel (I haven't been able to test the below >> scenario on a non-grsec kernel). I've been seeing some crashes lately with >> 8.1.1 that look like https://github.com/apache/trafficserver/issues/4921 >> <https://github.com/apache/trafficserver/issues/4921>, i.e. "Fatal: >> HttpSM.cc:2533 <http://httpsm.cc:2533/>: failed assertion `magic == >> HTTP_SM_MAGIC_ALIVE`" (and yes, I see the comment in the ticket that someone >> should reopen the ticket if they see that on 8.1.x, but I haven't gotten >> around to it yet -- see the first line of this email). >> >> Today: >> So due to those 8.1.1 crashes, I was trying out 9.0.1 and ran into the same >> issue that I initially ran into with 9.0.0: traffic_manager starts up and >> then starts traffic_server, and traffic_server crashes with: >> >> [May 11 18:36:41.343] traffic_server ERROR: [ReverseProxy] failed to add >> remap rule at /etc/trafficserver/remap.config line 198: >> /run/trafficserver/37882f60-2d8b-45e0-b8d8-ed2208c4c221/usr/lib/trafficserver/modules/tslua.so: >> failed to map segment from shared objectfailed to remove runtime copy: >> Success >> >> and traffic_manager dutifully tries again and again forever. >> >> And we use Lua to do all of our remapping (basically, to set the correct DNS >> name for ATS to look up), so it's a must-have for us. >> >> After some strace'ing, I figured out that Bionic by default has /run mounted >> with 'noexec'. If I remount /run with 'exec', I don't get this error anymore >> and everything loads up just fine. >> >> In the straces I was looking at, it looks like traffic_server is copying the >> tslua.so module into that temp directory under /run/trafficserver and then >> trying to mmap() the newly copied tslua.so file in /run and failing: >> >> openat(AT_FDCWD, >> "/run/trafficserver/e667e0ea-fc3b-43fd-945d-63fc4d9dc6e4/usr/lib/trafficserver/modules/tslua.so", >> O_RDONLY|O_CLOEXEC) = 38 >> read(38, "\177ELF\2\1\1\0\0\0<deleted for brevity sake>0\0\22\0\0\0", 832) = >> 832 >> fstat(38, {st_mode=S_IFREG|0644, st_size=152304, ...}) = 0 >> mmap(NULL, 2251784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 38, 0) = >> -1 EPERM (Operation not permitted) >> close(38) = 0 >> >> With /run mounted with 'exec', that mmap() succeeds and life is great. >> >> So for the ATS devs: >> >> * Is this just me (probably due to my somewhat obscure kernel)? Typically >> I'd see system log entries and/or dmesg in the case of grsecurity blocking >> it, but nothing here. >> >> * Did I miss something in the upgrade notes? I can't find anything about >> this behavior (copying modules to a dir under /run/trafficserver, instead of >> mmap'ing them directly from /usr/lib/). The configs I'm using were untouched >> from 8.1.1 (haven't yet removed the deprecated stuff). Normally, I'd chalk >> stuff up to operator error (and this still likely is), but this one seemed >> kind of weird. >> >> * Is there something I can set to prevent that behavior? I'd rather not >> remount boxes' /run (not to mention, it's done by systemd automatically, so >> it's not clear where to modify mount flags for /run; everything has a >> *.mount unit file except for /run itself). Ideally, traffic_server would >> just mmap /usr/lib/trafficserver/modules/tslua.so directly. But if that's >> not possible, is there a setting to override the location? I tried setting >> proxy.config.local_state_dir to something outside of /run; lock files and >> sockets get created but traffic_server immediately dies). I created >> /tmp/trafficserver and used that for local_state_dir, but traffic_manager >> complains that: >> >> Fatal: failed to connect management socket >> '/run/trafficserver/processerver.sock': No such file or directory >> >> so it's still using /run/trafficserver for some things. I guess there's more >> to it than just the proxy.config.local_state_dir setting. And "traffic_ctl >> config match . | grep run" returns only the >> "proxy.config.alarm.script_runtime" setting (If I override local_state_dir, >> of course, from /run/trafficserver). >> >> * For the 8.1.1 "failed assertion `magic == HTTP_SM_MAGIC_ALIVE`" issue, >> should I reopen that github issue or start a new one? >> >> Thanks! >
