[
https://issues.apache.org/jira/browse/JENA-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Obrist updated JENA-2354:
--------------------------------
Description:
I have observed that after upgrading from TDB 4.8.0 to 4.9.0, memory
consumption seems to have increased substantially.
Our application creates many TDB datasets, which are queried and updated
concurrently. To profile the memory increase, I have implemented a small
script, which creates up to 50 of our application-level objects, one after
another. Each of these objects is backed by a TDB dataset under the hood, which
gets created from scratch and filled with data (around 30k triples, 200MB
folder size).
The following shows the memory profiles when running the script against our
application for both TDB 4.8.0 / 4.9.0 and using 1G of heap space.
h3. TDB 4.8.0
Memory increases only slightly and the script completes the creation of all 50
objects / TDB datasets:
!tdb-4.8.0-run-2-memory.png|width=800,height=207!
h3. TDB 4.9.0
Memory increases fast, the application crashes with an OOM after having created
8 objects, during the creation of the 9th:
!tdb-4.9.0-run-6-memory.png|width=800,height=207!
Stack trace:
{code:java}
java.lang.OutOfMemoryError: Java heap space
at
com.github.benmanes.caffeine.cache.FrequencySketch.ensureCapacity(FrequencySketch.java:92)
at com.github.benmanes.caffeine.cache.SSMS.<init>(Unknown Source)
at
java.base/java.lang.invoke.LambdaForm$DMH/0x00000008028f7c40.newInvokeSpecial(LambdaForm$DMH)
at
java.base/java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(DelegatingMethodHandle$Holder)
at
java.base/java.lang.invoke.LambdaForm$MH/0x00000008028f7040.invoke_MT(LambdaForm$MH)
at
com.github.benmanes.caffeine.cache.LocalCacheFactory.loadFactory(LocalCacheFactory.java:86)
at
com.github.benmanes.caffeine.cache.LocalCacheFactory.newBoundedLocalCache(LocalCacheFactory.java:40)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3947)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3943)
at com.github.benmanes.caffeine.cache.Caffeine.build(Caffeine.java:1051)
at
org.apache.jena.atlas.lib.cache.CacheCaffeine.<init>(CacheCaffeine.java:62)
at org.apache.jena.atlas.lib.CacheFactory.createCache(CacheFactory.java:45)
at org.apache.jena.atlas.lib.CacheFactory.createCache(CacheFactory.java:33)
at
org.apache.jena.tdb2.store.nodetable.NodeTableCache.createCache(NodeTableCache.java:101)
at
org.apache.jena.tdb2.store.nodetable.NodeTableCache.<init>(NodeTableCache.java:95)
at
org.apache.jena.tdb2.store.nodetable.NodeTableCache.create(NodeTableCache.java:87)
at
org.apache.jena.tdb2.store.TDB2StorageBuilder.addNodeTableCache(TDB2StorageBuilder.java:302)
at
org.apache.jena.tdb2.store.TDB2StorageBuilder.buildNodeTable(TDB2StorageBuilder.java:287)
at
org.apache.jena.tdb2.store.TDB2StorageBuilder.buildStorage(TDB2StorageBuilder.java:189)
at
org.apache.jena.tdb2.store.TDB2StorageBuilder.build(TDB2StorageBuilder.java:97)
at org.apache.jena.tdb2.sys.StoreConnection.make(StoreConnection.java:95)
at
org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:63)
at
org.apache.jena.tdb2.sys.DatabaseOps.createSwitchable(DatabaseOps.java:139)
at org.apache.jena.tdb2.sys.DatabaseOps.create(DatabaseOps.java:86)
at
org.apache.jena.tdb2.sys.DatabaseConnection.build(DatabaseConnection.java:112)
at
org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:74)
at
org.apache.jena.tdb2.sys.DatabaseConnection$$Lambda$8502/0x00000008028e3c40.apply(Unknown
Source)
at
java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
at
org.apache.jena.tdb2.sys.DatabaseConnection.make(DatabaseConnection.java:74)
at
org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:63)
at
org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:54)
at org.apache.jena.tdb2.DatabaseMgr.DB_ConnectCreate(DatabaseMgr.java:41)
{code}
was:
I have observed that after upgrading from TDB 4.8.0 to 4.9.0, memory
consumption seems to have increased substantially.
Our application creates many TDB datasets, which are queried and updated
concurrently. To profile the memory increase, I have implemented a small
script, which creates up to 50 of our application-level objects, one after
another. Each of these objects is backed by a TDB dataset under the hood, which
gets created from scratch and filled with data (around 30k triples, 200MB
folder size).
The following shows the memory profiles when running the script against our
application for both TDB 4.8.0 / 4.9.0 and using 1G of heap space.
h3. TDB 4.8.0
Memory increases only slightly and the script completes the creation of all 50
objects / TDB datasets:
!tdb-4.8.0-run-2-memory.png!
h3. TDB 4.9.0
Memory increases fast, the application crashes with an OOM after having created
8 objects, during the creation of the 9th:
!tdb-4.9.0-run-6-memory.png!
Stack trace:
{code:java}
java.lang.OutOfMemoryError: Java heap space
at
com.github.benmanes.caffeine.cache.FrequencySketch.ensureCapacity(FrequencySketch.java:92)
at com.github.benmanes.caffeine.cache.SSMS.<init>(Unknown Source)
at
java.base/java.lang.invoke.LambdaForm$DMH/0x00000008028f7c40.newInvokeSpecial(LambdaForm$DMH)
at
java.base/java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(DelegatingMethodHandle$Holder)
at
java.base/java.lang.invoke.LambdaForm$MH/0x00000008028f7040.invoke_MT(LambdaForm$MH)
at
com.github.benmanes.caffeine.cache.LocalCacheFactory.loadFactory(LocalCacheFactory.java:86)
at
com.github.benmanes.caffeine.cache.LocalCacheFactory.newBoundedLocalCache(LocalCacheFactory.java:40)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3947)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3943)
at com.github.benmanes.caffeine.cache.Caffeine.build(Caffeine.java:1051)
at
org.apache.jena.atlas.lib.cache.CacheCaffeine.<init>(CacheCaffeine.java:62)
at org.apache.jena.atlas.lib.CacheFactory.createCache(CacheFactory.java:45)
at org.apache.jena.atlas.lib.CacheFactory.createCache(CacheFactory.java:33)
at
org.apache.jena.tdb2.store.nodetable.NodeTableCache.createCache(NodeTableCache.java:101)
at
org.apache.jena.tdb2.store.nodetable.NodeTableCache.<init>(NodeTableCache.java:95)
at
org.apache.jena.tdb2.store.nodetable.NodeTableCache.create(NodeTableCache.java:87)
at
org.apache.jena.tdb2.store.TDB2StorageBuilder.addNodeTableCache(TDB2StorageBuilder.java:302)
at
org.apache.jena.tdb2.store.TDB2StorageBuilder.buildNodeTable(TDB2StorageBuilder.java:287)
at
org.apache.jena.tdb2.store.TDB2StorageBuilder.buildStorage(TDB2StorageBuilder.java:189)
at
org.apache.jena.tdb2.store.TDB2StorageBuilder.build(TDB2StorageBuilder.java:97)
at org.apache.jena.tdb2.sys.StoreConnection.make(StoreConnection.java:95)
at
org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:63)
at
org.apache.jena.tdb2.sys.DatabaseOps.createSwitchable(DatabaseOps.java:139)
at org.apache.jena.tdb2.sys.DatabaseOps.create(DatabaseOps.java:86)
at
org.apache.jena.tdb2.sys.DatabaseConnection.build(DatabaseConnection.java:112)
at
org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:74)
at
org.apache.jena.tdb2.sys.DatabaseConnection$$Lambda$8502/0x00000008028e3c40.apply(Unknown
Source)
at
java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
at
org.apache.jena.tdb2.sys.DatabaseConnection.make(DatabaseConnection.java:74)
at
org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:63)
at
org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:54)
at org.apache.jena.tdb2.DatabaseMgr.DB_ConnectCreate(DatabaseMgr.java:41)
{code}
> OOM / increased memory consumption with TDB 4.9.0
> -------------------------------------------------
>
> Key: JENA-2354
> URL: https://issues.apache.org/jira/browse/JENA-2354
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB2
> Affects Versions: Jena 4.9.0
> Reporter: Damien Obrist
> Priority: Major
> Attachments: tdb-4.8.0-run-2-memory.png, tdb-4.9.0-run-6-memory.png
>
>
> I have observed that after upgrading from TDB 4.8.0 to 4.9.0, memory
> consumption seems to have increased substantially.
> Our application creates many TDB datasets, which are queried and updated
> concurrently. To profile the memory increase, I have implemented a small
> script, which creates up to 50 of our application-level objects, one after
> another. Each of these objects is backed by a TDB dataset under the hood,
> which gets created from scratch and filled with data (around 30k triples,
> 200MB folder size).
> The following shows the memory profiles when running the script against our
> application for both TDB 4.8.0 / 4.9.0 and using 1G of heap space.
> h3. TDB 4.8.0
> Memory increases only slightly and the script completes the creation of all
> 50 objects / TDB datasets:
> !tdb-4.8.0-run-2-memory.png|width=800,height=207!
> h3. TDB 4.9.0
> Memory increases fast, the application crashes with an OOM after having
> created 8 objects, during the creation of the 9th:
> !tdb-4.9.0-run-6-memory.png|width=800,height=207!
> Stack trace:
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> at
> com.github.benmanes.caffeine.cache.FrequencySketch.ensureCapacity(FrequencySketch.java:92)
> at com.github.benmanes.caffeine.cache.SSMS.<init>(Unknown Source)
> at
> java.base/java.lang.invoke.LambdaForm$DMH/0x00000008028f7c40.newInvokeSpecial(LambdaForm$DMH)
> at
> java.base/java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(DelegatingMethodHandle$Holder)
> at
> java.base/java.lang.invoke.LambdaForm$MH/0x00000008028f7040.invoke_MT(LambdaForm$MH)
> at
> com.github.benmanes.caffeine.cache.LocalCacheFactory.loadFactory(LocalCacheFactory.java:86)
> at
> com.github.benmanes.caffeine.cache.LocalCacheFactory.newBoundedLocalCache(LocalCacheFactory.java:40)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3947)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3943)
> at com.github.benmanes.caffeine.cache.Caffeine.build(Caffeine.java:1051)
> at
> org.apache.jena.atlas.lib.cache.CacheCaffeine.<init>(CacheCaffeine.java:62)
> at
> org.apache.jena.atlas.lib.CacheFactory.createCache(CacheFactory.java:45)
> at
> org.apache.jena.atlas.lib.CacheFactory.createCache(CacheFactory.java:33)
> at
> org.apache.jena.tdb2.store.nodetable.NodeTableCache.createCache(NodeTableCache.java:101)
> at
> org.apache.jena.tdb2.store.nodetable.NodeTableCache.<init>(NodeTableCache.java:95)
> at
> org.apache.jena.tdb2.store.nodetable.NodeTableCache.create(NodeTableCache.java:87)
> at
> org.apache.jena.tdb2.store.TDB2StorageBuilder.addNodeTableCache(TDB2StorageBuilder.java:302)
> at
> org.apache.jena.tdb2.store.TDB2StorageBuilder.buildNodeTable(TDB2StorageBuilder.java:287)
> at
> org.apache.jena.tdb2.store.TDB2StorageBuilder.buildStorage(TDB2StorageBuilder.java:189)
> at
> org.apache.jena.tdb2.store.TDB2StorageBuilder.build(TDB2StorageBuilder.java:97)
> at org.apache.jena.tdb2.sys.StoreConnection.make(StoreConnection.java:95)
> at
> org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:63)
> at
> org.apache.jena.tdb2.sys.DatabaseOps.createSwitchable(DatabaseOps.java:139)
> at org.apache.jena.tdb2.sys.DatabaseOps.create(DatabaseOps.java:86)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.build(DatabaseConnection.java:112)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:74)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection$$Lambda$8502/0x00000008028e3c40.apply(Unknown
> Source)
> at
> java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.make(DatabaseConnection.java:74)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:63)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:54)
> at org.apache.jena.tdb2.DatabaseMgr.DB_ConnectCreate(DatabaseMgr.java:41)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
