On Mon, Oct 24, 2022 at 10:39:45AM +0200, Jean Baptiste Favre wrote: > Please provide some more details on you issue: > * which plugin do you use? Is there any specific configuration?
I am not YunQiang Su but I'm pretty sure we're talking about the same thing. Any remap plugin will fail to load; for example, add the following to /etc/trafficserver/remap.config (example from the manual): map http://a.com http://b.com @plugin=regex_remap.so @pparam=maps.reg With contents of /etc/trafficserver/maps.reg (again from the manual): ^/(ogre.*)/more http://www.ogre.com/$h/$0/$1 Global plugins are not affected by this issue. > * do you have some trafficserver's logs to provide? Here you are (/var/log/trafficserver/diags.log): [Dec 28 20:00:59.369] traffic_server ERROR: plugin 'regex_remap.so' failed to load: /run/trafficserver/adb67a84-e739-4e8f-bc9d-c4797cec16e4/usr/lib/trafficserver/modules/regex_remap.so: failed to map segment from shared object [Dec 28 20:00:59.369] traffic_server ERROR: [ReverseProxy] failed to add remap rule at /etc/trafficserver/remap.config line 199: /run/trafficserver/adb67a84-e739-4e8f-bc9d-c4797cec16e4/usr/lib/trafficserver/modules/regex_remap.so: failed to map segment from shared objectfailed to remove runtime copy: Success [Dec 28 20:00:59.369] traffic_server WARNING: something failed during BuildTable() -- check your remap plugins! [Dec 28 20:00:59.369] traffic_server FATAL: remap.config failed to load [Dec 28 19:46:23.246] traffic_server ERROR: plugin 'cachekey.so' failed to load: /run/trafficserver/c4edf320-f56b-4193-8554-7db2dccfa0a7/usr/lib/trafficserver/modules/cachekey.so: failed to map segment from shared object [Dec 28 19:46:23.246] traffic_server ERROR: [ReverseProxy] failed to add remap rule at /etc/trafficserver/remap.config line 197: /run/trafficserver/c4edf320-f56b-4193-8554-7db2dccfa0a7/usr/lib/trafficserver/modules/cachekey.so: failed to map segment from shared objectfailed to remove runtime copy: Success [Dec 28 19:46:23.246] traffic_server WARNING: something failed during BuildTable() -- check your remap plugins! [Dec 28 19:46:23.246] traffic_server FATAL: remap.config failed to load There are errors from dlopen. As the OP said, the culprit is that it copies the plugins from /usr/lib to /run and tries to dlopen them there. This is doomed to fail because, exactly as the OP said, /run is mounted with the "noexec" option. Upstream logic behind this is to dynamically reload plugins that are changed while the server is running. I guess it makes sense with custom plugins and some other scenarios but there's no way for this to work on Debian systems. The attached dirty hack removes this code but also leads to test failures of two test programs where this dynamic reloading functionality is tested. It is not a proper fix but at least makes remap plugins load again. Like the OP, I also believe this bug deserves RC severity. Additionally (and this is an entirely separate issue, completely unrelated with the problem described above), some global plugins fail to load as well, for various reasons: [Dec 28 19:34:21.066] traffic_server NOTE: loading plugin '/usr/lib/trafficserve r/modules/prefetch.so' [Dec 28 19:34:21.066] traffic_server ERROR: unable to find TSPluginInit function in '/usr/lib/trafficserver/modules/prefetch.so': /usr/lib/trafficserver/modules/prefetch.so: undefined symbol: TSPluginInit [Dec 28 19:34:21.066] traffic_server FATAL: unable to find TSPluginInit function in '/usr/lib/trafficserver/modules/prefetch.so': /usr/lib/trafficserver/modules/prefetch.so: undefined symbol: TSPluginInit There is indeed no such function, and according to the plugin API there should be. [Dec 29 14:31:12.743] traffic_server ERROR: [geoip_acl] No Geo library available! [Dec 29 14:31:12.743] traffic_server ERROR: [ReverseProxy] failed to add remap rule at /etc/trafficserver/remap.config line 206: failed to initialize plugin geoip_acl.so: Unknown plugin errorfailed to remove runtime copy: Success [Dec 29 14:31:12.744] traffic_server WARNING: something failed during BuildTable() -- check your remap plugins! [Dec 29 14:31:12.744] traffic_server FATAL: remap.config failed to load I have tested only those that are pcre-based.
--- trafficserver-9.2.3+ds.orig/proxy/http/remap/PluginDso.cc +++ trafficserver-9.2.3+ds/proxy/http/remap/PluginDso.cc @@ -72,7 +72,7 @@ result = false; } else { PluginDebug(_tag, "plugin '%s' effective path: %s", _configPath.c_str(), _effectivePath.c_str()); - +#if 0 /* Copy the installed plugin DSO to a runtime directory if dynamic reload enabled */ std::error_code ec; if (isDynamicReloadEnabled() && !copy(_effectivePath, _runtimePath, ec)) { @@ -88,9 +88,9 @@ fs::file_status fs = fs::status(_effectivePath, ec); _mtime = fs::modification_time(fs); PluginDebug(_tag, "plugin '%s' modification time %ld", _configPath.c_str(), _mtime); - +#endif /* Now attempt to load the plugin DSO */ - if ((_dlh = dlopen(_runtimePath.c_str(), RTLD_NOW | RTLD_LOCAL)) == nullptr) { + if ((_dlh = dlopen(_effectivePath.c_str(), RTLD_NOW | RTLD_LOCAL)) == nullptr) { #if defined(freebsd) || defined(openbsd) char *err = (char *)dlerror(); #else @@ -105,12 +105,13 @@ PluginError("plugin '%s' failed to load: %s", _configPath.c_str(), error.c_str()); } } - +#if 0 /* Remove the runtime DSO copy even if we succeed loading to avoid leftovers after crashes */ if (_preventiveCleaning) { clean(error); } - } +#endif + PluginDebug(_tag, "plugin '%s' finished loading DSO", _configPath.c_str()); return result; --- trafficserver-9.2.3+ds.orig/proxy/http/remap/Makefile.am +++ trafficserver-9.2.3+ds/proxy/http/remap/Makefile.am @@ -80,32 +80,7 @@ $(CXX_Clang_Tidy) TESTS = $(check_PROGRAMS) -check_PROGRAMS = test_PluginDso test_PluginFactory test_RemapPluginInfo test_NextHopStrategyFactory test_NextHopRoundRobin test_NextHopConsistentHash - -test_PluginDso_CPPFLAGS = $(AM_CPPFLAGS) -I$(abs_top_srcdir)/tests/include -DPLUGIN_DSO_TESTS -test_PluginDso_LIBTOOLFLAGS = --preserve-dup-deps -EXTRA_test_PluginDso_DEPENDENCIES = unit-tests/plugin_v1.la -test_PluginDso_LDADD = $(COMMON_PLUGINDSO_LDADDS) -test_PluginDso_LDFLAGS = $(AM_LDFLAGS) -test_PluginDso_SOURCES = \ - unit-tests/test_PluginDso.cc \ - unit-tests/plugin_testing_common.cc \ - PluginDso.cc - -test_PluginFactory_CPPFLAGS = $(AM_CPPFLAGS) -I$(abs_top_srcdir)/tests/include -DPLUGIN_DSO_TESTS -test_PluginFactory_LIBTOOLFLAGS = --preserve-dup-deps -EXTRA_test_PluginFactory_DEPENDENCIES = \ - unit-tests/plugin_v1.la \ - unit-tests/plugin_init_fail.la \ - unit-tests/plugin_instinit_fail.la -test_PluginFactory_LDADD = $(COMMON_PLUGINDSO_LDADDS) -test_PluginFactory_LDFLAGS = $(AM_LDFLAGS) -test_PluginFactory_SOURCES = \ - unit-tests/test_PluginFactory.cc \ - unit-tests/plugin_testing_common.cc \ - PluginFactory.cc \ - PluginDso.cc \ - RemapPluginInfo.cc +check_PROGRAMS = test_RemapPluginInfo test_NextHopStrategyFactory test_NextHopRoundRobin test_NextHopConsistentHash test_RemapPluginInfo_CPPFLAGS = $(AM_CPPFLAGS) -I$(abs_top_srcdir)/tests/include -DPLUGIN_DSO_TESTS test_RemapPluginInfo_LIBTOOLFLAGS = --preserve-dup-deps