Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-08-17 Thread Dan Pritts
On Sat, Jul 31, 2021 at 5:41 AM Andy Seaborne wrote: > > > This isn’t what we have run into - the process’s RSS goes through the > roof, much bigger than the heap. Unfortunately I can’t reproduce it > easily, only in production. > > Quote possible - RSS is all process resident memory which inclu

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-31 Thread Andy Seaborne
On 30/07/2021 23:00, Dan Pritts wrote: On 30 Jul 2021, at 16:18, Andy Seaborne wrote: Hi Dan, Bit hazy about you deployment. Too many to track! I’m going to have to talk to your supervisor about this. :) What is this "supervisor" of which you speak? When there is a persistent databa

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-30 Thread Dan Pritts
On 30 Jul 2021, at 16:18, Andy Seaborne wrote: > Hi Dan, > > Bit hazy about you deployment. Too many to track! I’m going to have to talk to your supervisor about this. :) > When there is a persistent database, TDB uses memory mapped files. The space > shows as part of the JVM process (it's a

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-30 Thread Andy Seaborne
Hi Dan, Bit hazy about you deployment. Too many to track! When there is a persistent database, TDB uses memory mapped files. The space shows as part of the JVM process (it's a virtual memory mapping from the file system); it's not heap. It means a Java process can look huge - 100Gs is possib

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-30 Thread Dan Pritts
Andy Seaborne wrote: > Very roughly, Java takes about 0.5G over the heap size so 2.4GB is what to > expect before the runtime triggers a full GC and the full GC will reduce the > heap in uses. The process size will not shrink. We have experienced some sort of degenerate behavior with fuseki wher

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-29 Thread Andy Seaborne
On 29/07/2021 10:46, Marco Fiocco wrote: I'm building a docker image with openjdk:14-alpine and cannot enable ShenandoahGC, even with the experimental feature flag. It seems that OpenJDK must be compiled with some feature in order to support that. > However I've tuned the Java config with "-

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-29 Thread Martynas Jusevičius
The new container-related JDK features might be relevant here. https://developer.oracle.com/java/technologies/javase/8u191-relnotes.html We use this in docker-compose: -XX:+UseContainerSupport -XX:MaxRAMPercentage=75 On Thu, 29 Jul 2021 at 11.46, Marco Fiocco wrote: > I'm building a docker ima

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-29 Thread Marco Fiocco
I'm building a docker image with openjdk:14-alpine and cannot enable ShenandoahGC, even with the experimental feature flag. It seems that OpenJDK must be compiled with some feature in order to support that. However I've tuned the Java config with "-Xmx512m -Xms512m" and max reserved memory in my d

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-28 Thread Andy Seaborne
On 27/07/2021 22:17, Marco Fiocco wrote: Ok let me clarify the steps. I start Fuseki as a docker container with the config you saw earlier. Then I load the dataset with curl. After that I intend to use Fuseki as "read only" query only. That at moment, there is absolutely no query being done, b

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-28 Thread jerven Bolleman
Hi Marco, What you are seeing is normal behavior with the default GC options. There are other GC/JVM combinations that do not act that way. In any case if you want to know if it is a "MemoryLeak" or just normal GC behaviour I can recommend the java option: -XX:+HeapDumpOnOutOfMemoryError Whi

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-27 Thread Marco Fiocco
Ok let me clarify the steps. I start Fuseki as a docker container with the config you saw earlier. Then I load the dataset with curl. After that I intend to use Fuseki as "read only" query only. That at moment, there is absolutely no query being done, but still the memory allocated keeps growing.

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-27 Thread james anderson
> On 2021-07-27, at 19:08:25, Andy Seaborne wrote: > > > > On 27/07/2021 14:40, Laura Morales wrote: >> I can actually confirm that this was happening to me as well on a VM with >> 4GB RAM and TDB store (thus not in-memory). As more and more queries were >> made, RAM filled up, then the SWA

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-27 Thread Andy Seaborne
If the dataset is read-only, then it is always empty. The fuseki:serviceReadWriteGraphStore is the only way to get data into the database. The other two services are read-only. Are you sure it is the database growing and not Java loading classes? On a tight memory footprint, and because classe

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-27 Thread Marco Fiocco
On Tue, 27 Jul 2021 at 18:04, Andy Seaborne wrote: > > > On 27/07/2021 14:19, Marco Fiocco wrote: > > Hello, > > > > I'm running a in-memory Fuseki 3.16 server and I see that the allocated > memory keeps growing linearly indefinitely even if idle. > > That is strange because if there are no reque

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-27 Thread Andy Seaborne
://jena.apache.org/documentation/tdb/store-parameters.html How useful that is depends on the amount of data. Andy Sent: Tuesday, July 27, 2021 at 3:19 PM From: "Marco Fiocco" To: users@jena.apache.org Subject: In-memory Fuseki keeps growing memory indefinitely even if idle H

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-27 Thread Andy Seaborne
On 27/07/2021 14:19, Marco Fiocco wrote: Hello, I'm running a in-memory Fuseki 3.16 server and I see that the allocated memory keeps growing linearly indefinitely even if idle. That is strange because if there are no requests, it does no work. Initially I reserved 1GB of memory and I've

Re: In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-27 Thread Laura Morales
cronjob. > Sent: Tuesday, July 27, 2021 at 3:19 PM > From: "Marco Fiocco" > To: users@jena.apache.org > Subject: In-memory Fuseki keeps growing memory indefinitely even if idle > > Hello, > > I'm running a in-memory Fuseki 3.16 server and I see that the all

In-memory Fuseki keeps growing memory indefinitely even if idle

2021-07-27 Thread Marco Fiocco
Hello, I'm running a in-memory Fuseki 3.16 server and I see that the allocated memory keeps growing linearly indefinitely even if idle. Initially I reserved 1GB of memory and I've noticed that the process gets OOM killed every 2 hours. Now I've allocated 2GB because I've read somewhere that 2GB