[ https://issues.apache.org/jira/browse/TS-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leif Hedstrom updated TS-4266: ------------------------------ Summary: ATS memory statistics shows that memory utilization is doubled after “traffic_ctl config reload” (was: ATS memory statistics shows that memory utilization is doubled after “traffic_ctlconfig reload”. And it is failed as it cannot find enough memory.) > ATS memory statistics shows that memory utilization is doubled after > “traffic_ctl config reload” > ------------------------------------------------------------------------------------------------ > > Key: TS-4266 > URL: https://issues.apache.org/jira/browse/TS-4266 > Project: Traffic Server > Issue Type: Bug > Components: Lua > Reporter: Rajendra Kishore Bonumahanti > Assignee: Kit Chan > Fix For: 6.2.0 > > > ATS memory statistics shows memory utilization is doubled after “traffic_ctl > config reload”. We get “not enough memory” error in the subsequent attempt > and “config reload” fails. > ATS is configured with 100 map entries in remap.config, all share the same > lua script. > ATS is started: The memory information is.. > [root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total > total kB 1416092 670256 663736 > After 1st Config reload: > [root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total > total kB 1932660 1128084 1121544 > After 2nd config reload: It had failed with error “not enough memory” and > memory status as.. > [root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total > total kB 2170756 1167808 1160836 > Error displayed in diags.log: > ======================= > [Mar 8 23:27:27.580] Server {0x2af92498b700} WARNING: Failed to create new > instance for plugin /opt/trafficserver/libexec/trafficserver/tslua.so (not a > TS_SUCCESS return) > [Mar 8 23:27:27.580] Server {0x2af92498b700} WARNING: Could not add rule at > line #3; Aborting! > [Mar 8 23:27:27.580] Server {0x2af92498b700} WARNING: [ReverseProxy] Can't > create new remap instance for plugin > "/opt/trafficserver/libexec/trafficserver/tslua.so" - [ts_lua_add_module] > luaL_loadfile /opt/trafficserver/etc/trafficserver/lua/process_remap.lua > failed: not enough memory at line 3 > [Mar 8 23:27:27.580] Server {0x2af92498b700} WARNING: something failed > during BuildTable() -- check your remap plugins! > [Mar 8 23:27:27.595] Server {0x2af92498b700} WARNING: failed to reload > remap.config, not replacing! > Lua VM memory size at that time ,ts.debug(FUNCTION..'Lua VM memory: > '..collectgarbage("count")) > [Mar 8 23:27:27.579] Server {0x2af92498b700} DIAG: (ts_lua) __init__(): Lua > VM memory: 3629.7060546875 > This shows that Lua VMs are hitting the max capacity. > Solution: > ======= > I looked at the ts_lua code TSRemapDeleteInstance () [ts_lua.c ] and > ts_lua_del_module() [ts_lua_util.c] which does cleaning of the lua memory for > the instance. However the lua memory is not released and reused. > So, I have added code to start the garbage collector in ts_lua_del_module() . > int > ts_lua_del_module(ts_lua_instance_conf *conf, ts_lua_main_ctx *arr, int n) > { > …. > lua_newtable(L); > lua_replace(L, LUA_GLOBALSINDEX); /* L[GLOBAL] = EMPTY */ > lua_gc(L, LUA_GCCOLLECT, 0); > TSMutexUnlock(arr[i].mutexp); > } > return 0; > } > This has improved the situation. However, I also added garbage collection in > ts_lua_add_module() at the end. With these two additions, we have tested the > code, the memory utilization is stable and we could do config reload at lest > 100 times with the background load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)