Daniel Tripp wrote:
> Hello all. I have run into some behaviour that looks like a memory
> leak. I don't know if it's a bug or if I'm just missing something, but
> it's causing my application to crash so I would greatly appreciate any
> ideas.
>
> Scenario: Using derby 10.1.1.0 in an embedded setup, I am inserting many
> tens of thousands of rows, and the memory used by my application keeps
> creeping up to the point of OutOfMemoryError. The following code
> reproduces this:
>
> ** InsertLots.java **
>
> import java.util.*;
> import java.sql.*;
> import org.apache.derby.jdbc.EmbeddedDriver;
>
> public class InsertLots {
> static public void main(String args_[]) throws Exception {
> new EmbeddedDriver();
> Connection conn =
> DriverManager.getConnection("jdbc:derby:testdb;create=true");
> try {
> conn.createStatement().execute(
> "CREATE TABLE leaktesttable (time
> VARCHAR(100) )");
> } catch(SQLException e) {} // in case table already
> exists
>
> PreparedStatement stmt = conn.prepareStatement(
> "INSERT into leaktesttable values (?)");
> for(int i=0; true; ++i) {
> stmt.setString(1, (new
> java.util.Date()).toString());
> stmt.execute();
> if(i % 5000 == 0) {
> System.gc();
> System.out.println(""+i+":
> "+Runtime.getRuntime().totalMemory());
> }
> }
> }
> }
>
>
> When run with -Xmx5M, I get this output:
>
> 0: 2568192
> 5000: 3510272
> 10000: 4206592
> 15000: 5357568
> 20000: 6225920
> 25000: 6225920
> 30000: 6225920
> 35000: 6225920
> 40000: 6225920
> 45000: 6225920
> java.lang.OutOfMemoryError
> Exception in thread "main"
>
> I used a profiler (Jprofiler) on it, and noticed a guilty-looking class:
> org.apache.derby.impl.store.raw.data.StoredRecordHeader. The number of
> instances of this class consistently increased by 1 for every insert.
>
> A few other things I noticed:
> - Shutting down derby does not free these instances. (In fact, it
> creates more! And I had high hopes for this as a work-around.)
> - A prepared statement behaves the same as a non-prepared statement in
> this regard.
> - Select statements don't cause instances of this class to increase.
> - Running the above program more than once (so that the database already
> has a lot of rows in it) sometimes causes the program to crash sooner.
>
> Any ideas?
Most likely this is just the page cache growing. The page cache defaults
to 1000 pages, which in this case will be 4k pages. You can adjust the
size using the property
derby.storage.pageCacheSize
http://db.apache.org/derby/docs/10.1/tuning/rtunproper81359.html
E.G.
java -Xmx5M -Dderby.storage.pageCacheSize=100 InsertLots
There is a jira issue for Derby to handle these type of situations
better, e.g. by automatically shrinking the cache. I have some changes
almost ready for connections, but other areas are open for other people
to work on.
http://issues.apache.org/jira/browse/DERBY-443
Hope this helps,
Dan.
