I seem to have similar problems as M. Pesonen in the other chain.
Summary:
Fuseki server encountered a "disk full" situation (see log and error
report below) during update leading to crash. After restart, some parts
of the database are corrupted: dump and compact fail with
NodeTableTRDF/Read exceptions (see third error log below), as well as
some queries, but not all.
The corruption has taken place in another dataset than the one that was
being update when the disk full occurred.
Logs below,
best
Harri Kiiskinen
Fuseki log for "disk full" crash:
--------------------------------------------------------------------
fuseki-server[216149]: [2021-05-20 21:37:07] Fuseki INFO [182050]
Update
fuseki-server[216149]: #
fuseki-server[216149]: # A fatal error has been detected by the Java
Runtime Environment:
fuseki-server[216149]: #
fuseki-server[216149]: # SIGBUS (0x7) at pc=0x00007f2b608b7e15,
pid=216149, tid=768713
fuseki-server[216149]: #
fuseki-server[216149]: # JRE version: OpenJDK Runtime Environment
(11.0.11+9) (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
fuseki-server[216149]: # Java VM: OpenJDK 64-Bit Server VM
(11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing, tiered,
compressed oops, g1 gc, linux-amd64)
fuseki-server[216149]: # Problematic frame:
fuseki-server[216149]: # v ~StubRoutines::jint_disjoint_arraycopy
fuseki-server[216149]: #
fuseki-server[216149]: # Core dump will be written. Default location:
Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d
%P %E" (or dumping to //core.216149)
fuseki-server[216149]: #
fuseki-server[216149]: # An error report file with more information is
saved as:
fuseki-server[216149]: # /tmp/hs_err_pid216149.log
fuseki-server[216149]: Compiled method (c2) 1517007691 7583 4
org.apache.jena.dboe.base.page.PageBlockMgr::promoteDuplicate (67 bytes)
fuseki-server[216149]: total in heap
[0x00007f2b6868fc10,0x00007f2b68692840] = 11312
fuseki-server[216149]: relocation
[0x00007f2b6868fd88,0x00007f2b6868fea8] = 288
fuseki-server[216149]: main code
[0x00007f2b6868fec0,0x00007f2b68691580] = 5824
fuseki-server[216149]: stub code
[0x00007f2b68691580,0x00007f2b686915b8] = 56
fuseki-server[216149]: oops
[0x00007f2b686915b8,0x00007f2b686915d0] = 24
fuseki-server[216149]: metadata
[0x00007f2b686915d0,0x00007f2b686917d0] = 512
fuseki-server[216149]: scopes data
[0x00007f2b686917d0,0x00007f2b68692230] = 2656
fuseki-server[216149]: scopes pcs
[0x00007f2b68692230,0x00007f2b686926f0] = 1216
fuseki-server[216149]: dependencies
[0x00007f2b686926f0,0x00007f2b68692710] = 32
fuseki-server[216149]: handler table
[0x00007f2b68692710,0x00007f2b686927a0] = 144
fuseki-server[216149]: nul chk table
[0x00007f2b686927a0,0x00007f2b68692840] = 160
fuseki-server[216149]: Could not load hsdis-amd64.so; library not
loadable; PrintAssembly is disabled
fuseki-server[216149]: #
fuseki-server[216149]: # If you would like to submit a bug report,
please visit:
fuseki-server[216149]: #
https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
fuseki-server[216149]: #
--------------------------------------------------------------------
The error report /tmp/hs_err_pid216149.log:
----------------------------------------------------------------------
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x00007f2b608b7e15, pid=216149, tid=768713
#
# JRE version: OpenJDK Runtime Environment (11.0.11+9) (build
11.0.11+9-Ubuntu-0ubuntu2.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.11+9-Ubuntu-0ubuntu2.20.04,
mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# v ~StubRoutines::jint_disjoint_arraycopy
#
# Core dump will be written. Default location: Core dumps may be
processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping
to //core.216149)
#
# If you would like to submit a bug report, please visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
#
--------------- S U M M A R Y ------------
Command Line: -Xmx2G org.apache.jena.fuseki.cmd.FusekiCmd
--jetty-config=/etc/fuseki/fuseki-jetty-https.xml
Host: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz, 2 cores, 15G, Ubuntu
20.04.2 LTS
Time: Thu May 20 21:37:07 2021 EEST elapsed time: 1517007.667502 seconds
(17d 13h 23m 27s)
--------------- T H R E A D ---------------
Current thread (0x00007f2b38058000): JavaThread "qtp1764291958-26"
[_thread_in_Java, id=768713, stack(0x00007f2b4ca66000,0x00007f2b4cb67000)]
Stack: [0x00007f2b4ca66000,0x00007f2b4cb67000], sp=0x00007f2b4cb644d0,
free space=1017k
Native frames: (J=compiled Java code, A=aot compiled Java code,
j=interpreted, Vv=VM code, C=native code)
v ~StubRoutines::jint_disjoint_arraycopy
siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr:
0x00007f24a3e3a000
Register to memory mapping:
RAX=0x0000000093000007 points into unknown readable memory: 00
RBX=0x00000000f5bf4538 points into unknown readable memory:
0x0000000000000005 | 05 00 00 00 00 00 00 00
RCX=0x00000000000007eb is an unknown value
RDX=0xfffffffffffffc0f is an unknown value
RSP=0x00007f2b4cb644d0 is pointing into the stack for thread:
0x00007f2b38058000
RBP=0x00007f2b4cb644d0 is pointing into the stack for thread:
0x00007f2b38058000
RSI=0x00007f24a3e3bfa0 points into unknown readable memory:
0x0000000000000000 | 00 00 00 00 00 00 00 00
RDI=0x00007f24a3ddbfa0 points into unknown readable memory:
0xe9f2000029f10000 | 00 00 f1 29 00 00 f2 e9
R8 =0x0000000000000001 is an unknown value
R9 =0x00007f24a3dda000 points into unknown readable memory:
0x0000000093000007 | 07 00 00 93 00 00 00 00
R10=0x00007f2b608b8ba0 is at begin+0 in a stub
StubRoutines::unsafe_arraycopy [0x00007f2b608b8ba0, 0x00007f2b608b8bdb[
(59 bytes)
R11=0x0 is NULL
R12=0x0 is NULL
R13=0x000000000002a742 is an unknown value
R14=0x00000000bc102fa0 points into unknown readable memory:
0x000000000000000d | 0d 00 00 00 00 00 00 00
R15=0x00007f2b38058000 is a thread
Registers:
RAX=0x0000000093000007, RBX=0x00000000f5bf4538, RCX=0x00000000000007eb,
RDX=0xfffffffffffffc0f
RSP=0x00007f2b4cb644d0, RBP=0x00007f2b4cb644d0, RSI=0x00007f24a3e3bfa0,
RDI=0x00007f24a3ddbfa0
R8 =0x0000000000000001, R9 =0x00007f24a3dda000, R10=0x00007f2b608b8ba0,
R11=0x0000000000000000
R12=0x0000000000000000, R13=0x000000000002a742, R14=0x00000000bc102fa0,
R15=0x00007f2b38058000
RIP=0x00007f2b608b7e15, EFLAGS=0x0000000000010286,
CSGSFS=0x002b000000000033, ERR=0x0000000000000006
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007f2b4cb644d0)
0x00007f2b4cb644d0: 00007f24a3dda000 00007f2b686903b2
0x00007f2b4cb644e0: 00000000f5bf4578 00000000f5bf45b0
0x00007f2b4cb644f0: 00000000f5bf4940 00000000f5bf4900
0x00007f2b4cb64500: 0000000000000001 00001fac00002000
Instructions: (pc=0x00007f2b608b7e15)
0x00007f2b608b7d15: 83 4a ff ff ff 48 8b ca 48 c1 ea 02 f7 c1 01 00
0x00007f2b608b7d25: 00 00 74 0a 66 8b 44 4f fe 66 89 44 4e fe f7 c1
0x00007f2b608b7d35: 02 00 00 00 0f 84 57 00 00 00 8b 04 d7 89 04 d6
0x00007f2b608b7d45: e9 4c 00 00 00 48 8b 44 d7 f8 48 89 44 d6 f8 48
0x00007f2b608b7d55: ff ca 75 f1 48 33 c0 c5 f8 77 c9 c3 66 66 66 0f
0x00007f2b608b7d65: 1f 84 00 00 00 00 00 66 66 66 90 48 8b 44 d7 18
0x00007f2b608b7d75: 48 89 44 d6 18 48 8b 44 d7 10 48 89 44 d6 10 48
0x00007f2b608b7d85: 8b 44 d7 08 48 89 44 d6 08 48 8b 04 d7 48 89 04
0x00007f
---------------------------------------------------------------------
Error message when running tdb2.tdbcompact
-----------------------------------------------------------------
org.apache.jena.tdb2.TDBException: NodeTableTRDF/Read
at
org.apache.jena.tdb2.store.nodetable.NodeTableTRDF.readNodeFromTable(NodeTableTRDF.java:87)
at
org.apache.jena.tdb2.store.nodetable.NodeTableNative._retrieveNodeByNodeId(NodeTableNative.java:103)
at
org.apache.jena.tdb2.store.nodetable.NodeTableNative.getNodeForNodeId(NodeTableNative.java:52)
at
org.apache.jena.tdb2.store.nodetable.NodeTableCache._retrieveNodeByNodeId(NodeTableCache.java:206)
at
org.apache.jena.tdb2.store.nodetable.NodeTableCache.getNodeForNodeId(NodeTableCache.java:131)
at
org.apache.jena.tdb2.store.nodetable.NodeTableWrapper.getNodeForNodeId(NodeTableWrapper.java:52)
at
org.apache.jena.tdb2.store.nodetable.NodeTableInline.getNodeForNodeId(NodeTableInline.java:65)
at org.apache.jena.tdb2.lib.TupleLib.quad(TupleLib.java:112)
at org.apache.jena.tdb2.lib.TupleLib.quad(TupleLib.java:108)
at
org.apache.jena.tdb2.lib.TupleLib.lambda$convertToQuads$3(TupleLib.java:53)
at org.apache.jena.atlas.iterator.Iter$2.next(Iter.java:352)
at
org.apache.jena.atlas.iterator.IteratorWrapper.next(IteratorWrapper.java:36)
at
org.apache.jena.dboe.transaction.txn.IteratorTxnTracker.next(IteratorTxnTracker.java:39)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at org.apache.jena.tdb2.sys.CopyDSG.lambda$null$0(CopyDSG.java:38)
at org.apache.jena.system.Txn.exec(Txn.java:77)
at org.apache.jena.system.Txn.executeWrite(Txn.java:125)
at org.apache.jena.tdb2.sys.CopyDSG.lambda$copy$1(CopyDSG.java:36)
at org.apache.jena.system.Txn.exec(Txn.java:77)
at org.apache.jena.system.Txn.executeRead(Txn.java:115)
at org.apache.jena.tdb2.sys.CopyDSG.copy(CopyDSG.java:35)
at org.apache.jena.tdb2.sys.DatabaseOps.compact(DatabaseOps.java:236)
at org.apache.jena.tdb2.sys.DatabaseOps.compact(DatabaseOps.java:198)
at tdb2.tdbcompact.exec(tdbcompact.java:44)
at jena.cmd.CmdMain.mainMethod(CmdMain.java:92)
at jena.cmd.CmdMain.mainRun(CmdMain.java:58)
at jena.cmd.CmdMain.mainRun(CmdMain.java:45)
at tdb2.tdbcompact.main(tdbcompact.java:28)
Caused by: org.apache.thrift.protocol.TProtocolException: Unrecognized
type 0
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:144)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:60)
at
org.apache.jena.riot.thrift.wire.RDF_Term.standardSchemeReadValue(RDF_Term.java:433)
at org.apache.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:224)
at org.apache.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:213)
at org.apache.thrift.TUnion.read(TUnion.java:138)
at
org.apache.jena.tdb2.store.nodetable.NodeTableTRDF.readNodeFromTable(NodeTableTRDF.java:82)
... 27 more
-----------------------------------------------------------------------
--
Tutkijatohtori / post-doctoral researcher
Viral Culture in the Early Nineteenth-Century Europe (ViCE)
Movie Making Finland: Finnish fiction films as audiovisual big data,
1907–2017 (MoMaF)
Turun yliopisto / University of Turku
