On Wed, May 12, 2021 at 8:27 AM Bryan Call <[email protected]> wrote: > Do you know if the /tmp/ directory is also noexec? I am wondering if > there is a better place to copy the reloadable plugin on startup. > > I created a new issue on this: > https://github.com/apache/trafficserver/issues/7820 > > It looks like, barring any install-time customization, /tmp is just part of / in Focal/Bionic. But in my server farm, I've got plenty of types of boxes where we set up /tmp as its own mount. Though, for /tmp, if you do have a partition mounted on /tmp, you can at least set the mount options in fstab (versus systemd + /run).
Though, having server-dependent code in /tmp makes my sysadmin side feel a little ill :) I do like having the code be put onto a tmpfs spot though. Under /run would be ideal, if it weren't for the noexec > On May 12, 2021, at 7:26 AM, Mark Moseley <[email protected]> wrote: > > Awesome, that works perfectly. Thanks! > > I don't know what the mount option defaults are for /run on CentOS and > other distros, but looking at some Ubuntu Bionic and Focal boxes, 'noexec' > is the default for /run. It might be worth emitting an error about it if > traffic_server crashes trying to load dynamic modules (and specifically > when it fails that mmap() of the newly copied dynamic module) > and proxy.config.local_state_dir is mounted on a 'noexec' volume. I.e. if > traffic_server fails that mmap(), then it consults /proc/self/mounts to > determine mount options for proxy.config.local_state_dir? > > Though again, this could also just be specific to my environment that this > is even an issue. > > > On Wed, May 12, 2021 at 6:07 AM Bryan Call <[email protected]> wrote: > >> This is due to the plugin reload feature added to ATS 9.0.0 ( >> https://docs.trafficserver.apache.org/en/9.0.x/developer-guide/design-documents/reloading-plugins.en.html). >> You can disable the feature by >> setting proxy.config.plugin.dynamic_reload_mode to 0. >> >> -Bryan >> >> >> On May 11, 2021, at 5:08 PM, Mark Moseley <[email protected]> wrote: >> >> This is partially a FYI for Google-ing, but I'd love some guidance from >> the ATS devs on how to fix this issue in a 'better' way, as well as if I >> should reopen a 8.x github issue (see next paragraph). >> >> Background: >> I run 8.1.1 in a fairly traffic-heavy environment with ATS (sitting >> behind nginx) doing reverse-proxying to backend webservers. It's ubuntu >> bionic, running a grsecurity-enabled kernel (I haven't been able to test >> the below scenario on a non-grsec kernel). I've been seeing some crashes >> lately with 8.1.1 that look like >> https://github.com/apache/trafficserver/issues/4921, i.e. "Fatal: >> HttpSM.cc:2533 <http://httpsm.cc:2533/>: failed assertion `magic == >> HTTP_SM_MAGIC_ALIVE`" (and yes, I see the comment in the ticket that >> someone should reopen the ticket if they see that on 8.1.x, but I haven't >> gotten around to it yet -- see the first line of this email). >> >> Today: >> So due to those 8.1.1 crashes, I was trying out 9.0.1 and ran into the >> same issue that I initially ran into with 9.0.0: traffic_manager starts up >> and then starts traffic_server, and traffic_server crashes with: >> >> [May 11 18:36:41.343] traffic_server ERROR: [ReverseProxy] failed to add >> remap rule at /etc/trafficserver/remap.config line 198: >> /run/trafficserver/37882f60-2d8b-45e0-b8d8-ed2208c4c221/usr/lib/trafficserver/modules/tslua.so: >> failed to map segment from shared objectfailed to remove runtime copy: >> Success >> >> and traffic_manager dutifully tries again and again forever. >> >> And we use Lua to do all of our remapping (basically, to set the correct >> DNS name for ATS to look up), so it's a must-have for us. >> >> After some strace'ing, I figured out that Bionic by default has /run >> mounted with 'noexec'. If I remount /run with 'exec', I don't get this >> error anymore and everything loads up just fine. >> >> In the straces I was looking at, it looks like traffic_server is copying >> the tslua.so module into that temp directory under /run/trafficserver and >> then trying to mmap() the newly copied tslua.so file in /run and failing: >> >> openat(AT_FDCWD, >> "/run/trafficserver/e667e0ea-fc3b-43fd-945d-63fc4d9dc6e4/usr/lib/trafficserver/modules/tslua.so", >> O_RDONLY|O_CLOEXEC) = 38 >> read(38, "\177ELF\2\1\1\0\0\0<deleted for brevity sake>0\0\22\0\0\0", >> 832) = 832 >> fstat(38, {st_mode=S_IFREG|0644, st_size=152304, ...}) = 0 >> mmap(NULL, 2251784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 38, >> 0) = -1 EPERM (Operation not permitted) >> close(38) = 0 >> >> With /run mounted with 'exec', that mmap() succeeds and life is great. >> >> So for the ATS devs: >> >> * Is this just me (probably due to my somewhat obscure kernel)? Typically >> I'd see system log entries and/or dmesg in the case of grsecurity blocking >> it, but nothing here. >> >> * Did I miss something in the upgrade notes? I can't find anything about >> this behavior (copying modules to a dir under /run/trafficserver, instead >> of mmap'ing them directly from /usr/lib/). The configs I'm using were >> untouched from 8.1.1 (haven't yet removed the deprecated stuff). Normally, >> I'd chalk stuff up to operator error (and this still likely is), but this >> one seemed kind of weird. >> >> * Is there something I can set to prevent that behavior? I'd rather not >> remount boxes' /run (not to mention, it's done by systemd automatically, so >> it's not clear where to modify mount flags for /run; everything has a >> *.mount unit file except for /run itself). Ideally, traffic_server would >> just mmap /usr/lib/trafficserver/modules/tslua.so directly. But if that's >> not possible, is there a setting to override the location? I tried >> setting proxy.config.local_state_dir to something outside of /run; lock >> files and sockets get created but traffic_server immediately dies). I >> created /tmp/trafficserver and used that for local_state_dir, but >> traffic_manager complains that: >> >> Fatal: failed to connect management socket >> '/run/trafficserver/processerver.sock': No such file or directory >> >> so it's still using /run/trafficserver for some things. I guess there's >> more to it than just the proxy.config.local_state_dir setting. And >> "traffic_ctl config match . | grep run" returns only the >> "proxy.config.alarm.script_runtime" setting (If I override local_state_dir, >> of course, from /run/trafficserver). >> >> * For the 8.1.1 "failed assertion `magic == HTTP_SM_MAGIC_ALIVE`" issue, >> should I reopen that github issue or start a new one? >> >> Thanks! >> >> >> >
