Re: Limit memory usage of Fuseki server?

Tue, 10 Apr 2018 06:27:03 -0700 


We still have this issue:
Apr 10 15:54:29 -- kernel: Out of memory: Kill process 17136 (java) score 770 or sacrifice child Apr 10 15:54:29 -- kernel: Killed process 17136 (java) total-vm:9760620kB, anon-rss:3536204kB, file-rss:0kB Apr 10 15:54:29 -- systemd[1]: apache-jena-fuseki.service: Main process exited, code=killed, status=9/KILL Apr 10 15:54:29 -- systemd[1]: apache-jena-fuseki.service: Unit entered failed state. Apr 10 15:54:29 -- systemd[1]: apache-jena-fuseki.service: Failed with result 'signal'. 

Any idea how to solve please?

On 6.2.2018 19:34, Andy Seaborne wrote:

In this case, the details are on the command line:


ExecStart=/usr/lib/jvm/java-8-oracle/bin/java -jar
/opt/insight/jena/fuseki-server.jar --update --port 3030
--loc=/opt/insight/jena_data /ds


    Andy

On 06/02/18 16:11, Laura Morales wrote:
You should have another file for the dataset configuration too, inside "run/configuration/". 

Sent: Tuesday, February 06, 2018 at 4:13 PM
From: "Mikael Pesonen" <mikael.peso...@lingsoft.fi>
To: users@jena.apache.org
Subject: Re: Limit memory usage of Fuseki server?
# Licensed under the terms of http://www.apache.org/licenses/LICENSE-2.0

## Fuseki Server configuration file.

@prefix :        <#> .
@prefix fuseki: <http://jena.apache.org/fuseki#[http://jena.apache.org/fuseki#]> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#[http://www.w3.org/1999/02/22-rdf-syntax-ns#]> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#[http://www.w3.org/2000/01/rdf-schema#]> . @prefix ja: <http://jena.hpl.hp.com/2005/11/Assembler#[http://jena.hpl.hp.com/2005/11/Assembler#]> . 

[] rdf:type fuseki:Server ;
    # Example::
    # Server-wide query timeout.
    #
    # Timeout - server-wide default: milliseconds.
    # Format 1: "1000" -- 1 second timeout
    # Format 2: "10000,60000" -- 10s timeout to first result,
    #                            then 60s timeout for the rest of query.
    #
    # See javadoc for ARQ.queryTimeout for details.
    # This can also be set on a per dataset basis in the dataset assembler. 
    #
    # ja:context [ ja:cxtName "arq:queryTimeout" ;  ja:cxtValue "30000" ] ; 

    # Add any custom classes you want to load.
    # Must have a "public static void init()" method.
    # ja:loadClass "your.code.Class" ;

    # End triples.
    .



On 6.2.2018 16:58, Lorenz Buehmann wrote:

I guess Andy was hoping to see the Fuseki config file(s)


On 06.02.2018 15:56, Mikael Pesonen wrote:

Is it this one?

File: /lib/systemd/system/apache-jena-fuseki.service:

[Unit]

Description=Apache Jena Fuseki


[Service]

Type=simple

User=fuseki

Environment=JAVA_HOME=/usr/lib/jvm/java-8-oracle/

Environment=FUSEKI_HOME=/opt/insight/jena/

Environment=FUSEKI_BASE=/opt/insight/jena/run

ExecStart=/usr/lib/jvm/java-8-oracle/bin/java -jar
/opt/insight/jena/fuseki-server.jar --update --port 3030
--loc=/opt/insight/jena_data /ds


[Install]

WantedBy=multi-user.target


File: /lib/systemd/system/apache-jena-fuseki.path

[Path]

PathExistsGlob=/var/crash/*.crash

Unit=apache-jena-fuseki.service



On 6.2.2018 16:46, Andy Seaborne wrote:

What is the dataset/service setup?

On 06/02/18 14:33, Mikael Pesonen wrote:

I'm not good with Linux so don't know what setup means. We have
Ubuntu 14.04, Jena Fuseki 3.6.0 with default config.ttl and running
as a service.

Test server which is having both issues is running on Upcloud:

$ cat /proc/meminfo
MemTotal:        8692992 kB
MemFree:         6716504 kB
Buffers:           50052 kB
Cached:           508684 kB
SwapCached:            0 kB
Active:          1385768 kB
Inactive:         469640 kB
Active(anon):    1300740 kB
Inactive(anon):     9076 kB
Active(file):      85028 kB
Inactive(file):   460564 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                48 kB
Writeback:             0 kB
AnonPages:       1296688 kB
Mapped:           125308 kB
Shmem:             13128 kB
Slab:              30824 kB
SReclaimable:      16120 kB
SUnreclaim:        14704 kB
KernelStack:        1520 kB
PageTables:        12920 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4346496 kB
Committed_AS:    1669692 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       34708 kB
VmallocChunk:   34359694588 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1193984 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       46960 kB
DirectMap2M:     2574336 kB
DirectMap1G:     6291456 kB

Anything else?

The setup in which I have to concurrency error every time, is apache
web server running php, which is calling Fuseki GSP with curl. With
4 concurrent tests i can't reproduce the error with some trying,
with 5 concurrent tests error occurs every time.



On 6.2.2018 16:14, Andy Seaborne wrote:

I don't know what's going on.  I can't start to try to reproduce it
because I don't know what the trigger is.  It might be a previously
corrupted database or something messing with the DB behind the
server the effect is right but it might well be something else.

ajs6f asked for details of the setup - could you provide those
please? and a complete minimal example.  The fact you can't create
one points towards an environment/config/hardware issue, not a code
bug.

     Andy

On 06/02/18 13:14, Mikael Pesonen wrote:

Do you have now enough info for figuring out what's going on? It's
a bug that can be fixed?

I guess this is not related to concurrency issue I sent on another
thread? Both issues are happening on same setup.

Br

On 6.2.2018 14:18, Andy Seaborne wrote:

Expections in alloc-write seem to be a cause of you rother issues
so, maybe, this is the same rot cause with a different
manifestation.  It would have been useful to know this at the
start of the thread. It could be the broken alloc-write is simply
asking for a huge amount of RAM becaue it says:

report_vm_out_of_memory

so it is not the system killing Fuseki, its the JVM exiting.

With a single TDB, 12G is way too much heap (try 2G) but the
figure may include mapped files.

     Andy

On 06/02/18 09:56, Mikael Pesonen wrote:

Hi,

ok I'll try that. I had out of memory error on other server now.
Is there a place I could post log files, if they help?

[2018-02-06 11:48:51] BindingTDB ERROR get1(?o)
org.apache.jena.tdb.base.file.FileException: In the middle of an
alloc-write
      at
org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:311) 

...
[2018-02-06 11:48:52] BindingTDB ERROR get1(?o)
java.nio.BufferOverflowException
      at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:214)
      at sun.nio.ch.IOUtil.read(IOUtil.java:200)
      at
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:741)       at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727) 
      at
org.apache.jena.tdb.base.file.BufferChannelFile.read(BufferChannelFile.java:112) 




Stack: [0x00007f1956dda000,0x00007f1956edb000],
sp=0x00007f1956ed9720, free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
V  [libjvm.so+0xaba7ea] VMError::report_and_die()+0x2ba
V  [libjvm.so+0x4f9e3b] report_vm_out_of_memory(char const*,
int, unsigned long, VMErrorType, char const*)+0x8b
V  [libjvm.so+0x91b613] os::Linux::commit_memory_impl(char*,
unsigned long, bool)+0x103
V  [libjvm.so+0x91b6dc] os::pd_commit_memory(char*, unsigned
long, bool)+0xc
V  [libjvm.so+0x91518a] os::commit_memory(char*, unsigned long,
bool)+0x2a
V  [libjvm.so+0x9198af] os::pd_create_stack_guard_pages(char*,
unsigned long)+0x7f
V  [libjvm.so+0xa6049e] JavaThread::create_stack_guard_pages()+0x5e 
V  [libjvm.so+0xa69e14]  JavaThread::run()+0x34
V  [libjvm.so+0x91d9d8] java_start(Thread*)+0x108
C  [libpthread.so.0+0x8182]  start_thread+0xc2

On 6.2.2018 10:57, Laura Morales wrote:

kernel: Killed process 29037 (java) total-vm:12745796kB

12.7GB? Used by Fuseki? Or does this account other processes as
well? There is no way Fuseki is using 12GB of memory for a few
thousands of triples, unless maybe memory is leaked during
update operations. You should try try:

1. start Fuseki with your database
2. run a lot of live update queries (insert/delete/update data)
and monitor how your VM responds


Sent: Tuesday, February 06, 2018 at 9:22 AM
From: "Mikael Pesonen" <mikael.peso...@lingsoft.fi>
To: users@jena.apache.org
Subject: Re: Limit memory usage of Fuseki server?
Hi,

config.ttl is defaut, Fuseki is ran as service, here is the log
from
around the kill (notice fairly high VM)
   kernel: Out of memory: Kill process 29037 (java) score 807 or 
sacrifice child
   kernel: Killed process 29037 (java) total-vm:12745796kB,
anon-rss:3458140kB, file-rss:0kB

systemd[1]: apache-jena-fuseki.service: Main process exited,
code=killed, status=9/KILL
   systemd[1]: apache-jena-fuseki.service: Unit entered failed
state.
   systemd[1]: apache-jena-fuseki.service: Failed with result
'signal'.
Usage is not much, around few thousands of triplets, post, get and delete, using Fuseki's GSP over HTTP. Uptime was around one week. 

Sorry can't think of anything else to tell about the system.

Br


On 5.2.2018 18:08, ajs6f wrote:

Can you tell us more about your configuration, data, workload,
and the timing of the problem?

ajs6f


On Feb 5, 2018, at 7:08 AM, Laura Morales <laure...@mail.com>
wrote:

You can tell the Kernel to limit memory usage per process or
per user. If you set the limit too low your process will be
killed because it can't allocate more memory, but you can
tell the Kernel to only limit resident memory while using up
all the virtual memory it wants. You can do this with cgroups. 



Sent: Monday, February 05, 2018 at 12:56 PM
From: "Mikael Pesonen" <mikael.peso...@lingsoft.fi>
To: users@jena.apache.org
Subject: Limit memory usage of Fuseki server?
We have a cloud server with 4gb of memory and after a while
system is
killing Fuseki server java process for using up all memory.
Is it
possible to limit memory usage, and what would be the
performance
penalty for doing that?

Thanks.

--
Lingsoft - 30 years of Leading Language Management
www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]][http://www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]]] 


Speech Applications - Language Management - Translation -
Reader's and Writer's Tools - Text Tools - E-books and M-books 

Mikael Pesonen
System Engineer

e-mail: mikael.peso...@lingsoft.fi
Tel. +358 2 279 3300

Time zone: GMT+2

Helsinki Office
Eteläranta 10
FI-00130 Helsinki
FINLAND

Turku Office
Kauppiaskatu 5 A
FI-20100 Turku
FINLAND


--
Lingsoft - 30 years of Leading Language Management
www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]] 

Speech Applications - Language Management - Translation -
Reader's and Writer's Tools - Text Tools - E-books and M-books

Mikael Pesonen
System Engineer

e-mail: mikael.peso...@lingsoft.fi
Tel. +358 2 279 3300

Time zone: GMT+2

Helsinki Office
Eteläranta 10
FI-00130 Helsinki
FINLAND

Turku Office
Kauppiaskatu 5 A
FI-20100 Turku
FINLAND


--
Lingsoft - 30 years of Leading Language Management

www.lingsoft.fi[http://www.lingsoft.fi]
Speech Applications - Language Management - Translation - Reader's and Writer's Tools - Text Tools - E-books and M-books 

Mikael Pesonen
System Engineer

e-mail: mikael.peso...@lingsoft.fi
Tel. +358 2 279 3300

Time zone: GMT+2

Helsinki Office
Eteläranta 10
FI-00130 Helsinki
FINLAND

Turku Office
Kauppiaskatu 5 A
FI-20100 Turku
FINLAND



--
Lingsoft - 30 years of Leading Language Management

www.lingsoft.fi

Speech Applications - Language Management - Translation - Reader's and Writer's 
Tools - Text Tools - E-books and M-books

Mikael Pesonen
System Engineer

e-mail: mikael.peso...@lingsoft.fi
Tel. +358 2 279 3300

Time zone: GMT+2

Helsinki Office
Eteläranta 10
FI-00130 Helsinki
FINLAND

Turku Office
Kauppiaskatu 5 A
FI-20100 Turku
FINLAND

Reply via email to