Re: [PATCHS] Re: Possible Lua Memory Leak
On Wed, Apr 26, 2017 at 10:26:01AM -0400, Philip Seidel wrote: > Thanks for the help everyone. I have tested out the patches and I no > longer see any memory being leaked. How long does it take before an > official release will be cut? it depends a lot on the amount of important bugs that are fixed. We try to release every 1-2 months, but if we face a big bug we can release faster, and sometimes it's so calm that we forget for 3 months. Willy
Re: [PATCHS] Re: Possible Lua Memory Leak
Thanks for the help everyone. I have tested out the patches and I no longer see any memory being leaked. How long does it take before an official release will be cut? Thanks, Phil On Wed, Apr 26, 2017 at 9:13 AM, Willy Tarreau wrote: > On Wed, Apr 26, 2017 at 03:12:11PM +0200, thierry.fourn...@arpalert.org > wrote: > > oops, the backport to 1.6 was not clean. > > Ah OK thanks, our mails have crossed. I'm taking them all then. > > Willy >
Re: [PATCHS] Re: Possible Lua Memory Leak
On Wed, Apr 26, 2017 at 03:12:11PM +0200, thierry.fourn...@arpalert.org wrote: > oops, the backport to 1.6 was not clean. Ah OK thanks, our mails have crossed. I'm taking them all then. Willy
Re: [PATCHS] Re: Possible Lua Memory Leak
On Wed, Apr 26, 2017 at 02:41:18PM +0200, thierry.fourn...@arpalert.org wrote:
> I catch the bug. It affect all the versions of HAProxy embedding Lua.
> It is fixed.
>
> I join 2 main patches:
>
> - 0001*.patch: the bug fix, it should be backported
> - 0002*.patch: a nice cleanup which is useless to backport
Thanks Thierry, but I can't merge these ones, look, one is broken with an
unresolved conflict :
> diff --git a/src/hlua.c b/src/hlua.c
> index 4e35785..d276e3a 100644
> --- a/src/hlua.c
> +++ b/src/hlua.c
> @@ -3615,6 +3615,40 @@ static int hlua_applet_http_new(lua_State *L, struct
> appctx *ctx)
> return 1;
> }
>
> +<<< HEAD
> +===
> +__LJMP static int hlua_applet_http_set_priv(lua_State *L)
> +{
> + struct hlua_appctx *appctx = MAY_LJMP(hlua_checkapplet_http(L, 1));
> + struct stream *s = appctx->htxn.s;
> + struct hlua *hlua = &s->hlua;
> +
> + MAY_LJMP(check_args(L, 2, "set_priv"));
> +
> + /* Remove previous value. */
> + if (hlua->Mref != -1)
> + luaL_unref(L, LUA_REGISTRYINDEX, hlua->Mref);
> +
> + /* Get and store new value. */
> + lua_pushvalue(L, 2); /* Copy the element 2 at the top of the stack. */
> + hlua->Mref = luaL_ref(L, LUA_REGISTRYINDEX); /* pop the previously
> pushed value. */
> +
> + return 0;
> +}
> +
> +__LJMP static int hlua_applet_http_get_priv(lua_State *L)
> +{
> + struct hlua_appctx *appctx = MAY_LJMP(hlua_checkapplet_http(L, 1));
> + struct stream *s = appctx->htxn.s;
> + struct hlua *hlua = &s->hlua;
> +
> + /* Push configuration index in the stack. */
> + lua_rawgeti(L, LUA_REGISTRYINDEX, hlua->Mref);
> +
> + return 1;
> +}
> +
> +>>> c43224e... BUG/MEDIUM: lua: memory leak
Also I think that this function is not used anywhere else in the patch,
so please confirm.
Thanks,
Willy
Re: [PATCHS] Re: Possible Lua Memory Leak
oops, the backport to 1.6 was not clean. Thierry On Wed, 26 Apr 2017 14:41:18 +0200 thierry.fourn...@arpalert.org wrote: > I catch the bug. It affect all the versions of HAProxy embedding Lua. > It is fixed. > > I join 2 main patches: > > - 0001*.patch: the bug fix, it should be backported > - 0002*.patch: a nice cleanup which is useless to backport > > And I join the backport for the 1.6 version because it is not trivial. > > - 0001-*.1.6.patch > > Willy, can you merge these bugfix ? > > Thierry > > > On Wed, 26 Apr 2017 07:51:04 -0400 > Philip Seidel wrote: > > > Thierry -- > > > > Thanks for your help with this. I tried running with that line commented > > out and I still see the same behavior during my testing. Memory is still > > never reclaimed throughout the life of the haproxy process. > > > > Thanks, > > > > -Phil > > > > On Wed, Apr 26, 2017 at 7:09 AM, wrote: > > > > > It seems that the memleak doesn't appear when if remove the line which > > > sets the "priv" context do nil. > > > > > > So, when I comment this line: > > > > > >txn:set_priv(txnPrvState); > > > > > > The memleak disappear. Now if you want, you can apply this workaroud. > > > I try to understand the memleak. > > > > > > Thierry > > > > > > > > > On Wed, 26 Apr 2017 12:11:16 +0200 > > > Thierry Fournier wrote: > > > > > > > Hi, > > > > > > > > Thanks for the bug report. I can reproduce the bug. > > > > I will look for a fix. > > > > > > > > Thierry > > > > > > > > > > > > > On 26 Apr 2017, at 05:14, Philip Seidel > > > wrote: > > > > > > > > > > Possible Lua Memory Leak? > > > > > > > > > > We are running HAProxy version 1.7.2 (also tested 1.7.5) and are > > > loading a Lua file which makes use of 2 actions. The first action is > > > called on http-request and the second on http-response. In the simple > > > test > > > case we put together the request action stores some data in a table and > > > calls txn:set_priv to save the state for the transaction. The response > > > action calls txn:get_priv in order to access the state that was set in the > > > request action. All variables are set to nil once they are no longer > > > needed but it seems that no matter what kind of load HAProxy is receiving > > > we leak a little bit of memory with each request. Eventually, response > > > times begin to increase to the point where health checks to HAProxy begin > > > to fail and the server is unresponsive. We can take the instance out of > > > rotation and memory still doesn't get reclaimed despite all connections to > > > the frontend and backend being closed. We do see garbage collections > > > happening but it never cleans up enough to stabi > > > lize the instance. The leak is easily reproducible using a local test > > > instance and JMeter. The test uses 10 connections with keep-alive at > > > around 1500 r/s. It takes only a few minutes to consume over 1.5G of > > > memory. Any ideas as to what might be going on here? Is there something > > > wrong with how we are attempting to integrate these LUA actions? I am > > > happy to provide additional information if anyone is willing to assist > > > with > > > this. I have posted the configuration and other files on pastebin. > > > > > > > > > > Configuration - https://pastebin.com/SBtAEZ9R > > > > > Lua - https://pastebin.com/64Anbxm4 > > > > > > > > > > > > > > > > >From 7ccffd0a813e96dc5ef5baee9d42df422c29ee24 Mon Sep 17 00:00:00 2001 From: Thierry FOURNIER Date: Wed, 26 Apr 2017 14:25:58 +0200 Subject: [PATCH] BUG/MEDIUM: lua: memory leak The priv context is not cleaned when we set a new priv context. This is caused by a stupid swap between two parameter of the luaL_unref() function. workaround: use set_priv only once when we process a stream. This patch should be backported in version 1.7 and 1.6 --- src/hlua.c |2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/hlua.c b/src/hlua.c index 4e35785..a22af73 100644 --- a/src/hlua.c +++ b/src/hlua.c @@ -4646,7 +4646,7 @@ __LJMP static int hlua_set_priv(lua_State *L) /* Remove previous value. */ if (hlua->Mref != -1) - luaL_unref(L, hlua->Mref, LUA_REGISTRYINDEX); + luaL_unref(L, LUA_REGISTRYINDEX, hlua->Mref); /* Get and store new value. */ lua_pushvalue(L, 2); /* Copy the element 2 at the top of the stack. */ -- 1.7.10.4
[PATCHS] Re: Possible Lua Memory Leak
I catch the bug. It affect all the versions of HAProxy embedding Lua.
It is fixed.
I join 2 main patches:
- 0001*.patch: the bug fix, it should be backported
- 0002*.patch: a nice cleanup which is useless to backport
And I join the backport for the 1.6 version because it is not trivial.
- 0001-*.1.6.patch
Willy, can you merge these bugfix ?
Thierry
On Wed, 26 Apr 2017 07:51:04 -0400
Philip Seidel wrote:
> Thierry --
>
> Thanks for your help with this. I tried running with that line commented
> out and I still see the same behavior during my testing. Memory is still
> never reclaimed throughout the life of the haproxy process.
>
> Thanks,
>
> -Phil
>
> On Wed, Apr 26, 2017 at 7:09 AM, wrote:
>
> > It seems that the memleak doesn't appear when if remove the line which
> > sets the "priv" context do nil.
> >
> > So, when I comment this line:
> >
> >txn:set_priv(txnPrvState);
> >
> > The memleak disappear. Now if you want, you can apply this workaroud.
> > I try to understand the memleak.
> >
> > Thierry
> >
> >
> > On Wed, 26 Apr 2017 12:11:16 +0200
> > Thierry Fournier wrote:
> >
> > > Hi,
> > >
> > > Thanks for the bug report. I can reproduce the bug.
> > > I will look for a fix.
> > >
> > > Thierry
> > >
> > >
> > > > On 26 Apr 2017, at 05:14, Philip Seidel
> > wrote:
> > > >
> > > > Possible Lua Memory Leak?
> > > >
> > > > We are running HAProxy version 1.7.2 (also tested 1.7.5) and are
> > loading a Lua file which makes use of 2 actions. The first action is
> > called on http-request and the second on http-response. In the simple test
> > case we put together the request action stores some data in a table and
> > calls txn:set_priv to save the state for the transaction. The response
> > action calls txn:get_priv in order to access the state that was set in the
> > request action. All variables are set to nil once they are no longer
> > needed but it seems that no matter what kind of load HAProxy is receiving
> > we leak a little bit of memory with each request. Eventually, response
> > times begin to increase to the point where health checks to HAProxy begin
> > to fail and the server is unresponsive. We can take the instance out of
> > rotation and memory still doesn't get reclaimed despite all connections to
> > the frontend and backend being closed. We do see garbage collections
> > happening but it never cleans up enough to stabi
> > lize the instance. The leak is easily reproducible using a local test
> > instance and JMeter. The test uses 10 connections with keep-alive at
> > around 1500 r/s. It takes only a few minutes to consume over 1.5G of
> > memory. Any ideas as to what might be going on here? Is there something
> > wrong with how we are attempting to integrate these LUA actions? I am
> > happy to provide additional information if anyone is willing to assist with
> > this. I have posted the configuration and other files on pastebin.
> > > >
> > > > Configuration - https://pastebin.com/SBtAEZ9R
> > > > Lua - https://pastebin.com/64Anbxm4
> > > >
> > >
> > >
> >
>From ea1dc9b8dcb0bcf340132402d23648a2c397a4b3 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER
Date: Wed, 26 Apr 2017 14:25:58 +0200
Subject: [PATCH] BUG/MEDIUM: lua: memory leak
The priv context is not cleaned when we set a new priv context.
This is caused by a stupid swap between two parameter of the
luaL_unref() function.
workaround: use set_priv only once when we process a stream.
This patch should be backported in version 1.7 and 1.6
---
src/hlua.c | 36 +++-
1 file changed, 35 insertions(+), 1 deletion(-)
diff --git a/src/hlua.c b/src/hlua.c
index 4e35785..d276e3a 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -3615,6 +3615,40 @@ static int hlua_applet_http_new(lua_State *L, struct appctx *ctx)
return 1;
}
+<<<<<<< HEAD
+===
+__LJMP static int hlua_applet_http_set_priv(lua_State *L)
+{
+ struct hlua_appctx *appctx = MAY_LJMP(hlua_checkapplet_http(L, 1));
+ struct stream *s = appctx->htxn.s;
+ struct hlua *hlua = &s->hlua;
+
+ MAY_LJMP(check_args(L, 2, "set_priv"));
+
+ /* Remove previous value. */
+ if (hlua->Mref != -1)
+ luaL_unref(L, LUA_REGISTRYINDEX, hlua->Mref);
+
+ /* Get and store new value. */
+ lua_pushvalue(L, 2); /* Copy the element 2 at the top of the stack. */
+ hlua->Mref = luaL_ref(L, LUA_REGISTRYINDEX); /* pop the previously pushed value. */
+
+ return 0;
+}
+
+__LJMP static int hlua_applet_h
Re: Possible Lua Memory Leak
Thierry -- Thanks for your help with this. I tried running with that line commented out and I still see the same behavior during my testing. Memory is still never reclaimed throughout the life of the haproxy process. Thanks, -Phil On Wed, Apr 26, 2017 at 7:09 AM, wrote: > It seems that the memleak doesn't appear when if remove the line which > sets the "priv" context do nil. > > So, when I comment this line: > >txn:set_priv(txnPrvState); > > The memleak disappear. Now if you want, you can apply this workaroud. > I try to understand the memleak. > > Thierry > > > On Wed, 26 Apr 2017 12:11:16 +0200 > Thierry Fournier wrote: > > > Hi, > > > > Thanks for the bug report. I can reproduce the bug. > > I will look for a fix. > > > > Thierry > > > > > > > On 26 Apr 2017, at 05:14, Philip Seidel > wrote: > > > > > > Possible Lua Memory Leak? > > > > > > We are running HAProxy version 1.7.2 (also tested 1.7.5) and are > loading a Lua file which makes use of 2 actions. The first action is > called on http-request and the second on http-response. In the simple test > case we put together the request action stores some data in a table and > calls txn:set_priv to save the state for the transaction. The response > action calls txn:get_priv in order to access the state that was set in the > request action. All variables are set to nil once they are no longer > needed but it seems that no matter what kind of load HAProxy is receiving > we leak a little bit of memory with each request. Eventually, response > times begin to increase to the point where health checks to HAProxy begin > to fail and the server is unresponsive. We can take the instance out of > rotation and memory still doesn't get reclaimed despite all connections to > the frontend and backend being closed. We do see garbage collections > happening but it never cleans up enough to stabi > lize the instance. The leak is easily reproducible using a local test > instance and JMeter. The test uses 10 connections with keep-alive at > around 1500 r/s. It takes only a few minutes to consume over 1.5G of > memory. Any ideas as to what might be going on here? Is there something > wrong with how we are attempting to integrate these LUA actions? I am > happy to provide additional information if anyone is willing to assist with > this. I have posted the configuration and other files on pastebin. > > > > > > Configuration - https://pastebin.com/SBtAEZ9R > > > Lua - https://pastebin.com/64Anbxm4 > > > > > > > >
Re: Possible Lua Memory Leak
It seems that the memleak doesn't appear when if remove the line which sets the "priv" context do nil. So, when I comment this line: txn:set_priv(txnPrvState); The memleak disappear. Now if you want, you can apply this workaroud. I try to understand the memleak. Thierry On Wed, 26 Apr 2017 12:11:16 +0200 Thierry Fournier wrote: > Hi, > > Thanks for the bug report. I can reproduce the bug. > I will look for a fix. > > Thierry > > > > On 26 Apr 2017, at 05:14, Philip Seidel wrote: > > > > Possible Lua Memory Leak? > > > > We are running HAProxy version 1.7.2 (also tested 1.7.5) and are loading a > > Lua file which makes use of 2 actions. The first action is called on > > http-request and the second on http-response. In the simple test case we > > put together the request action stores some data in a table and calls > > txn:set_priv to save the state for the transaction. The response action > > calls txn:get_priv in order to access the state that was set in the request > > action. All variables are set to nil once they are no longer needed but it > > seems that no matter what kind of load HAProxy is receiving we leak a > > little bit of memory with each request. Eventually, response times begin > > to increase to the point where health checks to HAProxy begin to fail and > > the server is unresponsive. We can take the instance out of rotation and > > memory still doesn't get reclaimed despite all connections to the frontend > > and backend being closed. We do see garbage collections happening but it > > never cleans up enough to stabi lize the instance. The leak is easily reproducible using a local test instance and JMeter. The test uses 10 connections with keep-alive at around 1500 r/s. It takes only a few minutes to consume over 1.5G of memory. Any ideas as to what might be going on here? Is there something wrong with how we are attempting to integrate these LUA actions? I am happy to provide additional information if anyone is willing to assist with this. I have posted the configuration and other files on pastebin. > > > > Configuration - https://pastebin.com/SBtAEZ9R > > Lua - https://pastebin.com/64Anbxm4 > > > >
Re: Possible Lua Memory Leak
Hi, Thanks for the bug report. I can reproduce the bug. I will look for a fix. Thierry > On 26 Apr 2017, at 05:14, Philip Seidel wrote: > > Possible Lua Memory Leak? > > We are running HAProxy version 1.7.2 (also tested 1.7.5) and are loading a > Lua file which makes use of 2 actions. The first action is called on > http-request and the second on http-response. In the simple test case we put > together the request action stores some data in a table and calls > txn:set_priv to save the state for the transaction. The response action > calls txn:get_priv in order to access the state that was set in the request > action. All variables are set to nil once they are no longer needed but it > seems that no matter what kind of load HAProxy is receiving we leak a little > bit of memory with each request. Eventually, response times begin to > increase to the point where health checks to HAProxy begin to fail and the > server is unresponsive. We can take the instance out of rotation and memory > still doesn't get reclaimed despite all connections to the frontend and > backend being closed. We do see garbage collections happening but it never > cleans up enough to stabilize the instance. The leak is easily reproducible > using a local test instance and JMeter. The test uses 10 connections with > keep-alive at around 1500 r/s. It takes only a few minutes to consume over > 1.5G of memory. Any ideas as to what might be going on here? Is there > something wrong with how we are attempting to integrate these LUA actions? I > am happy to provide additional information if anyone is willing to assist > with this. I have posted the configuration and other files on pastebin. > > Configuration - https://pastebin.com/SBtAEZ9R > Lua - https://pastebin.com/64Anbxm4 >
Possible Lua Memory Leak
Possible Lua Memory Leak? We are running HAProxy version 1.7.2 (also tested 1.7.5) and are loading a Lua file which makes use of 2 actions. The first action is called on http-request and the second on http-response. In the simple test case we put together the request action stores some data in a table and calls txn:set_priv to save the state for the transaction. The response action calls txn:get_priv in order to access the state that was set in the request action. All variables are set to nil once they are no longer needed but it seems that no matter what kind of load HAProxy is receiving we leak a little bit of memory with each request. Eventually, response times begin to increase to the point where health checks to HAProxy begin to fail and the server is unresponsive. We can take the instance out of rotation and memory still doesn't get reclaimed despite all connections to the frontend and backend being closed. We do see garbage collections happening but it never cleans up enough to stabilize the instance. The leak is easily reproducible using a local test instance and JMeter. The test uses 10 connections with keep-alive at around 1500 r/s. It takes only a few minutes to consume over 1.5G of memory. Any ideas as to what might be going on here? Is there something wrong with how we are attempting to integrate these LUA actions? I am happy to provide additional information if anyone is willing to assist with this. I have posted the configuration and other files on pastebin. Configuration - https://pastebin.com/SBtAEZ9R Lua - https://pastebin.com/64Anbxm4
