Help with Probable Leaks
Hello I am experiencing a behavior when using DBI inside Dancer that seems to result on SCALAR leaks, but only when running inside Dancer. To help diagnosing I did a couple of tests that I try to explain below. First, I developed a sample small application in Dancer. It is available (temporarily) in http://paste.perldancer.org/2yIABT6Yreh5T. Basically, it has two routes. One that calls Devel::Gladiator and dumps the count for each type of variables currently available, and another route that returns OK. This route has a prepare statement (line 27) that is my problem. If I comment that line, and test calling the main route different times (calling the arena between calls) I get the result in http://paste.perldancer.org/sWhBrp2rh1lq. Basically, first 5 hits are discarded, just to make sure all variables and caches prepared by Dancer are filled in, and then I call the main route 100 times. You might notice the amount of scalars rarely changes, and at the end, the summary is negative (less two scalars than the amount we had when beginning the test). If I uncomment, then I get http://paste.perldancer.org/1mjhfOfwxEJxf, where you might notice almost each call increments a SCALAR into memory. At the end, the summary is 97 wasted scalars (almost the same number of hits). Although I imagine DBI does some kind of cache, it is weird it stores 97 copies for the same query. Nevertheless, I tried a similar approach without Dancer, and NO SCALAR gets lost at all. Also, tried to do, in the same route, two prepares, and the number of lost scalars doubles. If I do only one prepare but two execute/fetch, I get only one scalar lost. So, this is something about Dancer and DBI together. But I have no idea of what might be going on. If you need some more details, or that I run some test, please poke me at #dbi on irc.perl.org (ambs). Thanks, Alberto (aka ambs) -- Alberto Simões
Re: Help with Probable Leaks
I just forgot to explain how to run the Dancer app. Probably easier to use this app: http://paste.perldancer.org/1nX0n7dg6abTJ Also, the test with only DBI and no Dancer here: http://paste.perldancer.org/uzagWiCYXUsj that doesn't leak anything. mje_ on IRC did some experiments and found out that when using prepare_cached the scalar count doesn't change. But then, strange enough to have prepare leaking inside dancer, and not outside. :-| Cheers Alberto (aka ambs) On Tue, May 7, 2013 at 1:36 PM, Alberto Simões hashas...@gmail.com wrote: Hello I am experiencing a behavior when using DBI inside Dancer that seems to result on SCALAR leaks, but only when running inside Dancer. To help diagnosing I did a couple of tests that I try to explain below. First, I developed a sample small application in Dancer. It is available (temporarily) in http://paste.perldancer.org/2yIABT6Yreh5T. Basically, it has two routes. One that calls Devel::Gladiator and dumps the count for each type of variables currently available, and another route that returns OK. This route has a prepare statement (line 27) that is my problem. If I comment that line, and test calling the main route different times (calling the arena between calls) I get the result in http://paste.perldancer.org/sWhBrp2rh1lq. Basically, first 5 hits are discarded, just to make sure all variables and caches prepared by Dancer are filled in, and then I call the main route 100 times. You might notice the amount of scalars rarely changes, and at the end, the summary is negative (less two scalars than the amount we had when beginning the test). If I uncomment, then I get http://paste.perldancer.org/1mjhfOfwxEJxf, where you might notice almost each call increments a SCALAR into memory. At the end, the summary is 97 wasted scalars (almost the same number of hits). Although I imagine DBI does some kind of cache, it is weird it stores 97 copies for the same query. Nevertheless, I tried a similar approach without Dancer, and NO SCALAR gets lost at all. Also, tried to do, in the same route, two prepares, and the number of lost scalars doubles. If I do only one prepare but two execute/fetch, I get only one scalar lost. So, this is something about Dancer and DBI together. But I have no idea of what might be going on. If you need some more details, or that I run some test, please poke me at #dbi on irc.perl.org (ambs). Thanks, Alberto (aka ambs) -- Alberto Simões -- Alberto Simões
Re: Help with Probable Leaks
Hey again
Tim answered to this problem yesterday but I didn't understand his answer
till now. The database handler stores in ChildHandles a list of all created
handles. This list is only emptied when $dbh looses scope.
The count didn't appear in my standalone test because it was
ill-implemented. Fixed it and the leak is there as well.
Where, not a leak, but a problem I need to solve.
At the moment I might do:
$dbh-{ChildHandles} = [];
I am afraid of what might happen doing this. But as the docs say on older
Perl versions is attribute is undef, there should be no problems.
Ideas?
Alberto (aka ambs)
On Tue, May 7, 2013 at 4:11 PM, Alberto Simões hashas...@gmail.com wrote:
I just forgot to explain how to run the Dancer app. Probably easier to use
this app: http://paste.perldancer.org/1nX0n7dg6abTJ
Also, the test with only DBI and no Dancer here:
http://paste.perldancer.org/uzagWiCYXUsj
that doesn't leak anything.
mje_ on IRC did some experiments and found out that when using
prepare_cached the scalar count doesn't change.
But then, strange enough to have prepare leaking inside dancer, and not
outside. :-|
Cheers
Alberto (aka ambs)
On Tue, May 7, 2013 at 1:36 PM, Alberto Simões hashas...@gmail.comwrote:
Hello
I am experiencing a behavior when using DBI inside Dancer that seems to
result on SCALAR leaks, but only when running inside Dancer.
To help diagnosing I did a couple of tests that I try to explain below.
First, I developed a sample small application in Dancer. It is available
(temporarily) in http://paste.perldancer.org/2yIABT6Yreh5T. Basically,
it has two routes. One that calls Devel::Gladiator and dumps the count for
each type of variables currently available, and another route that returns
OK. This route has a prepare statement (line 27) that is my problem.
If I comment that line, and test calling the main route different times
(calling the arena between calls) I get the result in
http://paste.perldancer.org/sWhBrp2rh1lq. Basically, first 5 hits are
discarded, just to make sure all variables and caches prepared by Dancer
are filled in, and then I call the main route 100 times. You might notice
the amount of scalars rarely changes, and at the end, the summary is
negative (less two scalars than the amount we had when beginning the test).
If I uncomment, then I get http://paste.perldancer.org/1mjhfOfwxEJxf,
where you might notice almost each call increments a SCALAR into memory. At
the end, the summary is 97 wasted scalars (almost the same number of hits).
Although I imagine DBI does some kind of cache, it is weird it stores 97
copies for the same query. Nevertheless, I tried a similar approach without
Dancer, and NO SCALAR gets lost at all.
Also, tried to do, in the same route, two prepares, and the number of
lost scalars doubles. If I do only one prepare but two execute/fetch, I get
only one scalar lost.
So, this is something about Dancer and DBI together. But I have no idea
of what might be going on.
If you need some more details, or that I run some test, please poke me at
#dbi on irc.perl.org (ambs).
Thanks,
Alberto (aka ambs)
--
Alberto Simões
--
Alberto Simões
--
Alberto Simões
Relevance of ChildHandles
Hello
Any kind of handle on DBI has a $h-{ChildHandles} that store weakrefs to
all child handles created. For instance, on a $dbh, you will get a list of
weakrefs to $sth you create.
When one of those $sth gets out of scope, its count get to 0. What Perl
does to the weakref, is to change it to an undef value (Perl isn't able to
remove that element from the array).
Although a single undef takes less memory than a complete $sth object, it
takes memory space. And if we get a lot of these undef values, things get
worse.
I noticed in the documentation that older Perl versions that doesn't
support weaken does not implement ChildHandles. Probably that means this
list is not *that* relevant.
My question is: how relevant is this list? relevant enough to waste this
space?
If it is not relevant on all situations, couldn't it be possible to turn it
off? Or then, turn it on if you want to use it?
Just some thoughts.
Thanks,
Alberto
--
Alberto Simões
Re: Relevance of ChildHandles
On Tue, May 07, 2013 at 10:28:29PM +0100, Alberto Simões wrote:
Hello
Any kind of handle on DBI has a $h-{ChildHandles} that store weakrefs to
all child handles created. For
instance, on a $dbh, you will get a list of weakrefs to $sth you create.
When one of those $sth gets out of scope, its count get to 0. What Perl
does to the weakref, is to
change it to an undef value (Perl isn't able to remove that element from
the array).
Although a single undef takes less memory than a complete $sth object, it
takes memory space. And if we
get a lot of these undef values, things get worse.
The DBI purges undef entries from ChildHandles from time to time.
(Specifically whenever the number of entries is a multiple of 120.)
At this stage in the life of the DBI I think it's reasonable to assume
that there isn't a leak in the DBI itself. If there was then a lot of
people would be affected and complaining about it.
Tim.
