Depends on what "runs out of memory means".
If the container is being killed by the host, then it is likely some
process in the container is asking for memory (sbrk), and there is a
container limit (or ulimit?) being exceeded, so the container runtime or
the host kills the container.
One source of sbrk is a JVM growing the heap. It is not the only source
in the container OS.
There is also the direct memory space in a JVM - it's fixed size (it
sits below the heap).
Java avoids doing major GC when it can grow the heap. Java will use
heap until it approaches the heap limit rather than do a full GC so it
can use more memory that it needs. This is regardless of the memory
working set size. It sbrk's even when it does not need to - a
time/space tradeoff.
If it works at 4G Fuseki isn't leaking memory.
Is the container limited?
Is the container runtime limited?
Andy
On 30/09/2022 06:36, Lorenz Buehmann wrote:
From my understanding, a larger heap space for Fuseki should only be
necessary when doing reasoning or e.g. loading the geospatial index. A
TDB database on the other hand is backed by memory mapped files, i.e.
makes use of off-heap memory and let the OS do all the work.
Indeed, I cannot explain why assigning more heap makes the Fuseki thread
consume as much until OOM is reached.
We also have a Fuseki Docker deployment, and we assigned Fuseki way more
memory (64GB) because of generating a large scale spatial index needs
all geometry objects in memory once. But it didn't crash because of what
you describe with the query load (maybe we never had such a constant load).
Indeed, comparison is difficult, different machines, different Docker
container, different Fuseki version ...
I think Andy will have better explanations and maybe also others like
Rob or people already using Fuseki@Docker
On 29.09.22 16:45, Martynas Jusevičius wrote:
Still hasn't crashed, so less heap could be the solution in this case.
On Thu, Sep 29, 2022 at 3:12 PM Martynas Jusevičius
<[email protected]> wrote:
I've lowered the heap size to 4GB to leave more off-heap memory (6GB).
It's been an hour and OOMKilled hasn't happened yet unlike before.
MEM% in docker stats peaks around 70%.
On Thu, Sep 29, 2022 at 12:41 PM Martynas Jusevičius
<[email protected]> wrote:
OK the findings are weird so far...
Under constant query load on my local Docker, MEM% of the Fuseki
container reached 100% within 45 minutes and it got OOMKilled.
However, the Used heap "teeth" in VisualVM were below 3GB of the total
~8GB Heap size the whole time.
What does that tell us?
On Thu, Sep 29, 2022 at 11:58 AM Martynas Jusevičius
<[email protected]> wrote:
Hi Eugen,
I have the debugger working, I was trying to connect the profiler :)
Finally I managed to connect from VisualVM on Windows thanks to this
answer:
https://stackoverflow.com/questions/66222727/how-to-connect-to-jmx-server-running-inside-wsl2/71881475#71881475
I've launched an infinite curl loop to create some query load, but
what now? What should I be looking for in VisualVM?
On Thu, Sep 29, 2022 at 11:33 AM Eugen Stan
<[email protected]> wrote:
For debugging, you need to do the following:
* pass JVM options to enable debugging
* expose docker port for JVM debug you chose
https://stackoverflow.com/questions/138511/what-are-java-command-line-options-to-set-to-allow-jvm-to-be-remotely-debugged
You should be able to do all this without changing the image:
docker env
variables and docker port option.
Once container is started and port is listening, open (confirm with
docker ps) connect to it to debug.
Good luck,
On 29.09.2022 11:22, Martynas Jusevičius wrote:
On Thu, Sep 29, 2022 at 9:41 AM Lorenz Buehmann
<[email protected]> wrote:
You're working on an in-memory dataset?
No the datasets are TDB2-backed
Does it also happen with Jena 4.6.1?
Don't know :)
I wanted to run a profiler and tried connecting from VisualVM on
Windows to the Fuseki container but neither jstatd nor JMX
connections
worked...
Now I want to run VisualVM inside the container itself but this
requires changing the Docker image in a way that I haven't
figured out
yet.
On 28.09.22 20:23, Martynas Jusevičius wrote:
Hi,
We have a dockerized Fuseki 4.5.0 instance that is gradually
running
out of memory over the course of a few hours.
3 datasets, none larger than 100000 triples. The load is
negligible
(maybe a few bursts x 10 simple queries per minute), no updates.
Dockerfile:
https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile
Memory settings:
mem_limit: 10240m
JAVA_OPTIONS=-Xmx7700m -Xms7700m
Any advice?
Martynas
--
Eugen Stan
+40770 941 271 / https://www.netdava.com
