Hi,
Some new tests have recently been added to the low-memory test suite
that we have (you can run it with 'ant junit-lowmem'). I'm now seeing
two problems:
a) I get an OOME now and then
b) It takes a long time to complete
Obviously, this looks like a memory leak in Derby. That's also pretty
much correct, but it's what I'll call a "controlled memory leak". I
believe the test I added for DERBY-4137 [1] is causing both (a) and (b).
It worked fine before the latest test additions, but now the memory
pressure is too high. The memory occupied for the 120'000 transactions
amounted to around 4 MB.
In DERBY-4137, the use of a timer causes memory leaks. The leak will
"self-correct" when the specified timeout duration has passed. In the
test, this value has been set very high and there is no guarantee this
won't happen in real-world applications either. In real-world
applications the size of the memory leak is given by transaction rate
and timeout value (the higher the tx rate and the longer the timeout
value, the more memory is hogged).
The fix only reduced the amount of memory leaked. In Java 5.0 there's a
method that can be used to purge the queue, but using it seems to be a
little bit complicated (i.e. when to invoke it). In any case, it will be
easier to deal with this once we have sunset support for J2SE 1.4 [2].
I can think of the following actions to reduce the likelihood of this
test causing problems:
1) Run the test only as embedded (reduces memory usage to ~half)
2) Force execution order and run this test as the last test (this
fixes the problem today, but it may reoccur if/when another test eats up
more memory)
3) Reduce the timeout value such that the data can be garbage
collected shortly after the test has been run (timing/performance sensitive)
Further, the following actions remove the problem altogether:
i) Delete the test.
ii) Run test in a separate VM.
iii) Fix issue using Java 5.0 API.
(some details to consider here)
For starters I will implement the changes suggested in (1) and (2). I
will test with only (2) first to see if we are already at a point where
the tests require more memory than what is available. See DERBY-5394 [3].
--
Kristian
[1] https://issues.apache.org/jira/browse/DERBY-4137.
[2] If this code is to be run on small platforms, we're back to square
one. It's not a problem technically, the code will just be a lot more
verbose and complex due to the use of reflection.
[3] https://issues.apache.org/jira/browse/DERBY-5394.
