I guess Andy was hoping to see the Fuseki config file(s)
On 06.02.2018 15:56, Mikael Pesonen wrote: > > Is it this one? > > File: /lib/systemd/system/apache-jena-fuseki.service: > > [Unit] > > Description=Apache Jena Fuseki > > > [Service] > > Type=simple > > User=fuseki > > Environment=JAVA_HOME=/usr/lib/jvm/java-8-oracle/ > > Environment=FUSEKI_HOME=/opt/insight/jena/ > > Environment=FUSEKI_BASE=/opt/insight/jena/run > > ExecStart=/usr/lib/jvm/java-8-oracle/bin/java -jar > /opt/insight/jena/fuseki-server.jar --update --port 3030 > --loc=/opt/insight/jena_data /ds > > > [Install] > > WantedBy=multi-user.target > > > File: /lib/systemd/system/apache-jena-fuseki.path > > [Path] > > PathExistsGlob=/var/crash/*.crash > > Unit=apache-jena-fuseki.service > > > > On 6.2.2018 16:46, Andy Seaborne wrote: >> What is the dataset/service setup? >> >> On 06/02/18 14:33, Mikael Pesonen wrote: >>> >>> I'm not good with Linux so don't know what setup means. We have >>> Ubuntu 14.04, Jena Fuseki 3.6.0 with default config.ttl and running >>> as a service. >>> >>> Test server which is having both issues is running on Upcloud: >>> >>> $ cat /proc/meminfo >>> MemTotal: 8692992 kB >>> MemFree: 6716504 kB >>> Buffers: 50052 kB >>> Cached: 508684 kB >>> SwapCached: 0 kB >>> Active: 1385768 kB >>> Inactive: 469640 kB >>> Active(anon): 1300740 kB >>> Inactive(anon): 9076 kB >>> Active(file): 85028 kB >>> Inactive(file): 460564 kB >>> Unevictable: 0 kB >>> Mlocked: 0 kB >>> SwapTotal: 0 kB >>> SwapFree: 0 kB >>> Dirty: 48 kB >>> Writeback: 0 kB >>> AnonPages: 1296688 kB >>> Mapped: 125308 kB >>> Shmem: 13128 kB >>> Slab: 30824 kB >>> SReclaimable: 16120 kB >>> SUnreclaim: 14704 kB >>> KernelStack: 1520 kB >>> PageTables: 12920 kB >>> NFS_Unstable: 0 kB >>> Bounce: 0 kB >>> WritebackTmp: 0 kB >>> CommitLimit: 4346496 kB >>> Committed_AS: 1669692 kB >>> VmallocTotal: 34359738367 kB >>> VmallocUsed: 34708 kB >>> VmallocChunk: 34359694588 kB >>> HardwareCorrupted: 0 kB >>> AnonHugePages: 1193984 kB >>> HugePages_Total: 0 >>> HugePages_Free: 0 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >>> DirectMap4k: 46960 kB >>> DirectMap2M: 2574336 kB >>> DirectMap1G: 6291456 kB >>> >>> Anything else? >>> >>> The setup in which I have to concurrency error every time, is apache >>> web server running php, which is calling Fuseki GSP with curl. With >>> 4 concurrent tests i can't reproduce the error with some trying, >>> with 5 concurrent tests error occurs every time. >>> >>> >>> >>> On 6.2.2018 16:14, Andy Seaborne wrote: >>>> I don't know what's going on. I can't start to try to reproduce it >>>> because I don't know what the trigger is. It might be a previously >>>> corrupted database or something messing with the DB behind the >>>> server the effect is right but it might well be something else. >>>> >>>> ajs6f asked for details of the setup - could you provide those >>>> please? and a complete minimal example. The fact you can't create >>>> one points towards an environment/config/hardware issue, not a code >>>> bug. >>>> >>>> Andy >>>> >>>> On 06/02/18 13:14, Mikael Pesonen wrote: >>>>> >>>>> Do you have now enough info for figuring out what's going on? It's >>>>> a bug that can be fixed? >>>>> >>>>> I guess this is not related to concurrency issue I sent on another >>>>> thread? Both issues are happening on same setup. >>>>> >>>>> Br >>>>> >>>>> On 6.2.2018 14:18, Andy Seaborne wrote: >>>>>> Expections in alloc-write seem to be a cause of you rother issues >>>>>> so, maybe, this is the same rot cause with a different >>>>>> manifestation. It would have been useful to know this at the >>>>>> start of the thread. It could be the broken alloc-write is simply >>>>>> asking for a huge amount of RAM becaue it says: >>>>>> >>>>>> report_vm_out_of_memory >>>>>> >>>>>> so it is not the system killing Fuseki, its the JVM exiting. >>>>>> >>>>>> With a single TDB, 12G is way too much heap (try 2G) but the >>>>>> figure may include mapped files. >>>>>> >>>>>> Andy >>>>>> >>>>>> On 06/02/18 09:56, Mikael Pesonen wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> ok I'll try that. I had out of memory error on other server now. >>>>>>> Is there a place I could post log files, if they help? >>>>>>> >>>>>>> [2018-02-06 11:48:51] BindingTDB ERROR get1(?o) >>>>>>> org.apache.jena.tdb.base.file.FileException: In the middle of an >>>>>>> alloc-write >>>>>>> at >>>>>>> org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:311) >>>>>>> >>>>>>> ... >>>>>>> [2018-02-06 11:48:52] BindingTDB ERROR get1(?o) >>>>>>> java.nio.BufferOverflowException >>>>>>> at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:214) >>>>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:200) >>>>>>> at >>>>>>> sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:741) >>>>>>> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727) >>>>>>> at >>>>>>> org.apache.jena.tdb.base.file.BufferChannelFile.read(BufferChannelFile.java:112) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Stack: [0x00007f1956dda000,0x00007f1956edb000], >>>>>>> sp=0x00007f1956ed9720, free space=1021k >>>>>>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, >>>>>>> C=native code) >>>>>>> V [libjvm.so+0xaba7ea] VMError::report_and_die()+0x2ba >>>>>>> V [libjvm.so+0x4f9e3b] report_vm_out_of_memory(char const*, >>>>>>> int, unsigned long, VMErrorType, char const*)+0x8b >>>>>>> V [libjvm.so+0x91b613] os::Linux::commit_memory_impl(char*, >>>>>>> unsigned long, bool)+0x103 >>>>>>> V [libjvm.so+0x91b6dc] os::pd_commit_memory(char*, unsigned >>>>>>> long, bool)+0xc >>>>>>> V [libjvm.so+0x91518a] os::commit_memory(char*, unsigned long, >>>>>>> bool)+0x2a >>>>>>> V [libjvm.so+0x9198af] os::pd_create_stack_guard_pages(char*, >>>>>>> unsigned long)+0x7f >>>>>>> V [libjvm.so+0xa6049e] JavaThread::create_stack_guard_pages()+0x5e >>>>>>> V [libjvm.so+0xa69e14] JavaThread::run()+0x34 >>>>>>> V [libjvm.so+0x91d9d8] java_start(Thread*)+0x108 >>>>>>> C [libpthread.so.0+0x8182] start_thread+0xc2 >>>>>>> >>>>>>> On 6.2.2018 10:57, Laura Morales wrote: >>>>>>>>> kernel: Killed process 29037 (java) total-vm:12745796kB >>>>>>>> 12.7GB? Used by Fuseki? Or does this account other processes as >>>>>>>> well? There is no way Fuseki is using 12GB of memory for a few >>>>>>>> thousands of triples, unless maybe memory is leaked during >>>>>>>> update operations. You should try try: >>>>>>>> >>>>>>>> 1. start Fuseki with your database >>>>>>>> 2. run a lot of live update queries (insert/delete/update data) >>>>>>>> and monitor how your VM responds >>>>>>>> >>>>>>>> >>>>>>>> Sent: Tuesday, February 06, 2018 at 9:22 AM >>>>>>>> From: "Mikael Pesonen" <mikael.peso...@lingsoft.fi> >>>>>>>> To: users@jena.apache.org >>>>>>>> Subject: Re: Limit memory usage of Fuseki server? >>>>>>>> Hi, >>>>>>>> >>>>>>>> config.ttl is defaut, Fuseki is ran as service, here is the log >>>>>>>> from >>>>>>>> around the kill (notice fairly high VM) >>>>>>>> >>>>>>>> kernel: Out of memory: Kill process 29037 (java) score 807 or >>>>>>>> sacrifice child >>>>>>>> kernel: Killed process 29037 (java) total-vm:12745796kB, >>>>>>>> anon-rss:3458140kB, file-rss:0kB >>>>>>>> >>>>>>>> systemd[1]: apache-jena-fuseki.service: Main process exited, >>>>>>>> code=killed, status=9/KILL >>>>>>>> systemd[1]: apache-jena-fuseki.service: Unit entered failed >>>>>>>> state. >>>>>>>> systemd[1]: apache-jena-fuseki.service: Failed with result >>>>>>>> 'signal'. >>>>>>>> >>>>>>>> Usage is not much, around few thousands of triplets, post, get and >>>>>>>> delete, using Fuseki's GSP over HTTP. Uptime was around one week. >>>>>>>> >>>>>>>> Sorry can't think of anything else to tell about the system. >>>>>>>> >>>>>>>> Br >>>>>>>> >>>>>>>> >>>>>>>> On 5.2.2018 18:08, ajs6f wrote: >>>>>>>>> Can you tell us more about your configuration, data, workload, >>>>>>>>> and the timing of the problem? >>>>>>>>> >>>>>>>>> ajs6f >>>>>>>>> >>>>>>>>>> On Feb 5, 2018, at 7:08 AM, Laura Morales <laure...@mail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> You can tell the Kernel to limit memory usage per process or >>>>>>>>>> per user. If you set the limit too low your process will be >>>>>>>>>> killed because it can't allocate more memory, but you can >>>>>>>>>> tell the Kernel to only limit resident memory while using up >>>>>>>>>> all the virtual memory it wants. You can do this with cgroups. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Sent: Monday, February 05, 2018 at 12:56 PM >>>>>>>>>> From: "Mikael Pesonen" <mikael.peso...@lingsoft.fi> >>>>>>>>>> To: users@jena.apache.org >>>>>>>>>> Subject: Limit memory usage of Fuseki server? >>>>>>>>>> We have a cloud server with 4gb of memory and after a while >>>>>>>>>> system is >>>>>>>>>> killing Fuseki server java process for using up all memory. >>>>>>>>>> Is it >>>>>>>>>> possible to limit memory usage, and what would be the >>>>>>>>>> performance >>>>>>>>>> penalty for doing that? >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Lingsoft - 30 years of Leading Language Management >>>>>>>>>> >>>>>>>>>> www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Speech Applications - Language Management - Translation - >>>>>>>>>> Reader's and Writer's Tools - Text Tools - E-books and M-books >>>>>>>>>> >>>>>>>>>> Mikael Pesonen >>>>>>>>>> System Engineer >>>>>>>>>> >>>>>>>>>> e-mail: mikael.peso...@lingsoft.fi >>>>>>>>>> Tel. +358 2 279 3300 >>>>>>>>>> >>>>>>>>>> Time zone: GMT+2 >>>>>>>>>> >>>>>>>>>> Helsinki Office >>>>>>>>>> Eteläranta 10 >>>>>>>>>> FI-00130 Helsinki >>>>>>>>>> FINLAND >>>>>>>>>> >>>>>>>>>> Turku Office >>>>>>>>>> Kauppiaskatu 5 A >>>>>>>>>> FI-20100 Turku >>>>>>>>>> FINLAND >>>>>>>>>> >>>>>>>> -- >>>>>>>> Lingsoft - 30 years of Leading Language Management >>>>>>>> >>>>>>>> www.lingsoft.fi[http://www.lingsoft.fi] >>>>>>>> >>>>>>>> Speech Applications - Language Management - Translation - >>>>>>>> Reader's and Writer's Tools - Text Tools - E-books and M-books >>>>>>>> >>>>>>>> Mikael Pesonen >>>>>>>> System Engineer >>>>>>>> >>>>>>>> e-mail: mikael.peso...@lingsoft.fi >>>>>>>> Tel. +358 2 279 3300 >>>>>>>> >>>>>>>> Time zone: GMT+2 >>>>>>>> >>>>>>>> Helsinki Office >>>>>>>> Eteläranta 10 >>>>>>>> FI-00130 Helsinki >>>>>>>> FINLAND >>>>>>>> >>>>>>>> Turku Office >>>>>>>> Kauppiaskatu 5 A >>>>>>>> FI-20100 Turku >>>>>>>> FINLAND >>>>>>> >>>>> >>> >
