[
https://issues.apache.org/jira/browse/JENA-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17853071#comment-17853071
]
Damien Obrist commented on JENA-2354:
-------------------------------------
Hi [~andy], apologies for digging up this issue from last year.
{quote}What's wrong with [https://github.com/apache/jena/pull/1996?]
{quote}
We have now done some more analysis regarding an upgrade from TDB 4.8.0 to
4.9.0+ and its impact on memory usage. Your change from the PR already helps,
but it isn't enough for our use case. We load many TDB datasets of different
sizes (which we can't predict), and the large bulk of it have around 30k
triples. For such datasets, we have observed that the memory usage increased
significantly: the retained size in the profiler went from ~8MB to ~15MB. A lot
of our memory is wasted for the initial capacity.
As I had already mentioned, ideally, the initial cache capacity could be made
configurable via StoreParams. I went ahead and implemented the change in a PR:
https://github.com/apache/jena/pull/2524
What do you think about it? The PR incidentally also fixes some issues with the
StoreParams prefix node cache size properties (cf. second commit of the PR).
> OOM / increased memory consumption with TDB 4.9.0
> -------------------------------------------------
>
> Key: JENA-2354
> URL: https://issues.apache.org/jira/browse/JENA-2354
> Project: Apache Jena
> Issue Type: Bug
> Components: TDB2
> Affects Versions: Jena 4.9.0
> Reporter: Damien Obrist
> Assignee: Andy Seaborne
> Priority: Major
> Fix For: Jena 4.10.0
>
> Attachments: tdb-4.8.0-run-2-memory.png, tdb-4.9.0-run-6-memory.png,
> tdb-4.9.0-run-7-memory.png, tdb-490-memory-increase-4.8.0-memory.png,
> tdb-490-memory-increase-4.9.0-exception.png,
> tdb-490-memory-increase-4.9.0-memory.png, tdb-490-memory-increase.zip
>
>
> I have observed that after upgrading from TDB 4.8.0 to 4.9.0, memory
> consumption seems to have increased substantially.
> Our application creates many TDB datasets, which are queried and updated
> concurrently. To profile the memory increase, I have implemented a small
> script, which creates up to 50 of our application-level objects, one after
> another. Each of these objects is backed by a TDB dataset under the hood,
> which gets created from scratch and filled with data (around 30k triples,
> 200MB folder size).
> The following shows the memory profiles when running the script against our
> application for both TDB 4.8.0 / 4.9.0 and using 1G of heap space.
> h3. TDB 4.8.0
> Memory increases only slightly and the script completes the creation of all
> 50 objects / TDB datasets:
> !tdb-4.8.0-run-2-memory.png|width=800,height=207!
> h3. TDB 4.9.0
> Memory increases fast and the application crashes with an OOM after having
> created 8 objects / TDB datasets, during the creation of the 9th:
> !tdb-4.9.0-run-6-memory.png|width=800,height=207!
> Stack trace:
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> at
> com.github.benmanes.caffeine.cache.FrequencySketch.ensureCapacity(FrequencySketch.java:92)
> at com.github.benmanes.caffeine.cache.SSMS.<init>(Unknown Source)
> at
> java.base/java.lang.invoke.LambdaForm$DMH/0x00000008028f7c40.newInvokeSpecial(LambdaForm$DMH)
> at
> java.base/java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(DelegatingMethodHandle$Holder)
> at
> java.base/java.lang.invoke.LambdaForm$MH/0x00000008028f7040.invoke_MT(LambdaForm$MH)
> at
> com.github.benmanes.caffeine.cache.LocalCacheFactory.loadFactory(LocalCacheFactory.java:86)
> at
> com.github.benmanes.caffeine.cache.LocalCacheFactory.newBoundedLocalCache(LocalCacheFactory.java:40)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3947)
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3943)
> at com.github.benmanes.caffeine.cache.Caffeine.build(Caffeine.java:1051)
> at
> org.apache.jena.atlas.lib.cache.CacheCaffeine.<init>(CacheCaffeine.java:62)
> at
> org.apache.jena.atlas.lib.CacheFactory.createCache(CacheFactory.java:45)
> at
> org.apache.jena.atlas.lib.CacheFactory.createCache(CacheFactory.java:33)
> at
> org.apache.jena.tdb2.store.nodetable.NodeTableCache.createCache(NodeTableCache.java:101)
> at
> org.apache.jena.tdb2.store.nodetable.NodeTableCache.<init>(NodeTableCache.java:95)
> at
> org.apache.jena.tdb2.store.nodetable.NodeTableCache.create(NodeTableCache.java:87)
> at
> org.apache.jena.tdb2.store.TDB2StorageBuilder.addNodeTableCache(TDB2StorageBuilder.java:302)
> at
> org.apache.jena.tdb2.store.TDB2StorageBuilder.buildNodeTable(TDB2StorageBuilder.java:287)
> at
> org.apache.jena.tdb2.store.TDB2StorageBuilder.buildStorage(TDB2StorageBuilder.java:189)
> at
> org.apache.jena.tdb2.store.TDB2StorageBuilder.build(TDB2StorageBuilder.java:97)
> at org.apache.jena.tdb2.sys.StoreConnection.make(StoreConnection.java:95)
> at
> org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:63)
> at
> org.apache.jena.tdb2.sys.DatabaseOps.createSwitchable(DatabaseOps.java:139)
> at org.apache.jena.tdb2.sys.DatabaseOps.create(DatabaseOps.java:86)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.build(DatabaseConnection.java:112)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:74)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection$$Lambda$8502/0x00000008028e3c40.apply(Unknown
> Source)
> at
> java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.make(DatabaseConnection.java:74)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:63)
> at
> org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:54)
> at org.apache.jena.tdb2.DatabaseMgr.DB_ConnectCreate(DatabaseMgr.java:41)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
