Thomas Vatter wrote:
The result of distributing application and database over two machines is:
Import of the critical data now works, no hanging at 1200 records
(I need to start derby with -Xmx1024m else an outofmemory exception
is thrown). My import algorithm has to be reviewed, it allocates
memory in a loop, 100mb when looping 1300 times. The original ordering
problem does not appear at this point.
Now the less good news. After work, I turned off the computer where
derby is installed. I know derby expects a shutdown, but I just
pressed CTL-C in its terminal window and made a shutdown for the os.
This morning I booted the computer and started derby. I was surprized
about the memory consumption in the task list. The derby process was
consuming the complete memory, 1024mb, resident part 30mb (os is
suse9.3). I tried an import of another spreadsheet with 1220 records
in this situation. The import worked. I displayed the newly imported
data in my application, the data are listed slowly - and the
original ordering problem appears. To check if the composite index on
the sorting columns was corrupted I dropped and recreated the index.
Same ordering problem. Perhaps the cancelling of the derby server in
the evening caused the problem, I will redo the whole thing without
stopping the server.
Just to experiment a little, I created a program that imported data into
Derby.
Only one table with three CHAR(100) columns. I inserted approximately 5
GB (of text) with the default Derby 10.1 and JVM settings. A count(*)
reported this:
ij> select count(*) from biginsert;
1
-----------
17476267
1 row selected
Memory usage was low at all times, and as you can see there are over 17
million rows in the database. So data volume per se should not be a
problem here, it must be something else causing your trouble (schema
layout? column types?).
For us to be able to help you, is there any possibility you can give us
more details about the database schema (tables, columns, types,
indexes/indices), the data itself and/or the import program?
The fact that you need to start the JVM with -Xmx1024m indicates
something is wrong. It can be Derby's fault, but it very hard to tell
without more information.
thanks,
--
Kristian
