Hi all!
On 30/05/14 16:36, Mark Feblowitz wrote:
After some amount of time I see a series of messages after update posts
WARN [xxxxxx] RC = 500 : Java heap space
And I’m seeing "java.lang.OutOfMemoryError: Java heap space” errors.
I also got this error yesterday on an important machine.
My setup is this: Fuseki 1.0.1 with a single TDB (no inference) that has
grown to 13GB and an additional jena-text index of 200MB. Fuseki is
given approx. 6GB of heap (-Xmx6000M) on a machine with 16GB RAM. The
machine is a virtual machine running 64bit CentOS 6, kernel
2.6.32-431.11.2.el6.x86_64, java -version gives this:
java version "1.7.0_51"
OpenJDK Runtime Environment (rhel-2.4.4.1.el6_5-x86_64 u51-b02)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)
At the time this happened there were no updates (we usually only update
during the night), just read-only SELECT and CONSTRUCT queries coming in
at around 8 queries per second on average.
Suddenly queries stop working and CPU usage rises to around 350% (the
machine has 4 cores). Errors like this appear in the Fuseki log:
2014-06-04 11:42:52,293 WARN Fuseki :: [14739636] RC = 500
: Java heap space
java.lang.OutOfMemoryError: Java heap space
2014-06-04 11:39:06,670 WARN Fuseki :: [14739587] RC = 500
: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-06-04 11:43:54,817 INFO Fuseki :: [14739587] 500 GC
overhead limit exceeded (1,032.111 s)
2014-06-04 12:24:56,868 WARN Fuseki :: [14739738] RC = 500
: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-06-04 12:24:04,660 WARN Fuseki :: [14739862] RC = 500
: Java heap space
java.lang.OutOfMemoryError: Java heap space
2014-06-04 12:43:44,167 INFO Fuseki :: [14739738] 500 GC
overhead limit exceeded (2,227.717 s)
2014-06-04 12:43:44,167 INFO Fuseki :: [14739862] 500 Java
heap space (2,227.719 s)
2014-06-04 14:33:30,906 WARN Fuseki :: [14740021] RC = 500
: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-06-04 14:34:21,722 INFO Fuseki :: [14740021] 500 GC
overhead limit exceeded (4,850.886 s)
The timestamps in the log entries are not always in order, as you can
see above. Sometimes it takes more than an hour for an individual query
to fail (see last entry).
Needless to say this is a bit nasty way of failing - the process is
running, consuming nearly all CPU, but responding to queries very slowly
or not at all. It would even be better if the process just died, so
something else could restart it. I am considering using a tool such as
Monit to watch the Fuseki process and restart it if it starts behaving
oddly.
Am I doing something obviously wrong here? Should I just give the JVM
even more memory, or adjust some of the other JVM options? Is there any
way to force the GC to fail faster, or otherwise avoid futile attempts
of freeing more memory? Even better, could Fuseki just stick to the
memory it is given?
Thanks,
Osma
--
Osma Suominen
D.Sc. (Tech), Information Systems Specialist
National Library of Finland
P.O. Box 26 (Teollisuuskatu 23)
00014 HELSINGIN YLIOPISTO
Tel. +358 50 3199529
osma.suomi...@helsinki.fi
http://www.nationallibrary.fi
