[jira] [Commented] (OAK-3007) SegmentStore cache does not take "string" map into account
[ https://issues.apache.org/jira/browse/OAK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299717#comment-15299717 ] Alex Parvulescu commented on OAK-3007: -- Merged to 1.2 branch with r1745461. > SegmentStore cache does not take "string" map into account > -- > > Key: OAK-3007 > URL: https://issues.apache.org/jira/browse/OAK-3007 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segmentmk >Reporter: Thomas Mueller >Assignee: Michael Dürig > Labels: candidate_oak_1_0, doc-impacting, resilience, scalability > Fix For: 1.4, 1.3.3, 1.2.16 > > Attachments: OAK-3007-2.patch, OAK-3007-3.patch, OAK-3007.patch > > > The SegmentStore cache size calculation ignores the size of the field > Segment.string (a concurrent hash map). It looks like a regular segment in a > memory mapped file has the size 1024, no matter how many strings are loaded > in memory. This can lead to out of memory. There seems to be no way to limit > (configure) the amount of memory used by strings. In one example, 100'000 > segments are loaded in memory, and 5 GB are used for Strings in that map. > We need a way to configure the amount of memory used for that. This seems to > be basically a cache. OAK-2688 does this, but it would be better to have one > cache with a configurable size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3007) SegmentStore cache does not take "string" map into account
[ https://issues.apache.org/jira/browse/OAK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14620405#comment-14620405 ] Thomas Mueller commented on OAK-3007: - For future reference, a benchmark I ran was: {noformat} java -Dwarmup=5 -Druntime=2 -Dprofile=true -jar target/oak-run-1.4-SNAPSHOT.jar benchmark ReadPropertyTest Oak-Tar | grep "Segment.read" {noformat} This does not show readString, but many readTemplate calls. Possibly the following tests could be run as well: ReadDeepTreeTest ConcurrentReadDeepTreeTest ConcurrentReadRandomNodeAndItsPropertiesTest ConcurrentReadTest ConcurrentTraversalTest DescendantSearchTest GetDeepNodeTest GetNodeWithAdmin ReadPropertyTest SQL2DescendantSearchTest SQL2SearchTest SimpleSearchTest > SegmentStore cache does not take "string" map into account > -- > > Key: OAK-3007 > URL: https://issues.apache.org/jira/browse/OAK-3007 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segmentmk >Reporter: Thomas Mueller > Fix For: 1.3.3 > > Attachments: OAK-3007-2.patch, OAK-3007-3.patch, OAK-3007.patch > > > The SegmentStore cache size calculation ignores the size of the field > Segment.string (a concurrent hash map). It looks like a regular segment in a > memory mapped file has the size 1024, no matter how many strings are loaded > in memory. This can lead to out of memory. There seems to be no way to limit > (configure) the amount of memory used by strings. In one example, 100'000 > segments are loaded in memory, and 5 GB are used for Strings in that map. > We need a way to configure the amount of memory used for that. This seems to > be basically a cache. OAK-2688 does this, but it would be better to have one > cache with a configurable size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3007) SegmentStore cache does not take "string" map into account
[ https://issues.apache.org/jira/browse/OAK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14618694#comment-14618694 ] Thomas Mueller commented on OAK-3007: - While writing tests, I found and fixed some issues and changed the API a bit to simplify testing. Still working on this. > SegmentStore cache does not take "string" map into account > -- > > Key: OAK-3007 > URL: https://issues.apache.org/jira/browse/OAK-3007 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segmentmk >Reporter: Thomas Mueller > Fix For: 1.3.3 > > Attachments: OAK-3007-2.patch, OAK-3007.patch > > > The SegmentStore cache size calculation ignores the size of the field > Segment.string (a concurrent hash map). It looks like a regular segment in a > memory mapped file has the size 1024, no matter how many strings are loaded > in memory. This can lead to out of memory. There seems to be no way to limit > (configure) the amount of memory used by strings. In one example, 100'000 > segments are loaded in memory, and 5 GB are used for Strings in that map. > We need a way to configure the amount of memory used for that. This seems to > be basically a cache. OAK-2688 does this, but it would be better to have one > cache with a configurable size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3007) SegmentStore cache does not take "string" map into account
[ https://issues.apache.org/jira/browse/OAK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616318#comment-14616318 ] Michael Dürig commented on OAK-3007: In some ad-hoc testing this patch together with the one from OAK-3055 proved beneficial. Pending the failing ITs and your review of OAK-3055 I think we should apply the patches to trunk. As soon as we have enough longevity coverage we should also merge them to the branches. > SegmentStore cache does not take "string" map into account > -- > > Key: OAK-3007 > URL: https://issues.apache.org/jira/browse/OAK-3007 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segmentmk >Reporter: Thomas Mueller > Fix For: 1.3.3 > > Attachments: OAK-3007-2.patch, OAK-3007.patch > > > The SegmentStore cache size calculation ignores the size of the field > Segment.string (a concurrent hash map). It looks like a regular segment in a > memory mapped file has the size 1024, no matter how many strings are loaded > in memory. This can lead to out of memory. There seems to be no way to limit > (configure) the amount of memory used by strings. In one example, 100'000 > segments are loaded in memory, and 5 GB are used for Strings in that map. > We need a way to configure the amount of memory used for that. This seems to > be basically a cache. OAK-2688 does this, but it would be better to have one > cache with a configurable size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3007) SegmentStore cache does not take "string" map into account
[ https://issues.apache.org/jira/browse/OAK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616306#comment-14616306 ] Michael Dürig commented on OAK-3007: A problem I encountered with the patch is {{FileStore#FileStore(java.io.File, org.apache.jackrabbit.oak.spi.state.NodeState, int)}} throwing an {{ISAE}}. That constructor should create a {{FileStore}} without cache. So far it did so by specifying a cache size of 0 to the {{SegmentTracker}}. With the LIRS cache this stopped working. Just run {{CompactionAndCleanupTest#propertyRetention}} to see this. > SegmentStore cache does not take "string" map into account > -- > > Key: OAK-3007 > URL: https://issues.apache.org/jira/browse/OAK-3007 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segmentmk >Reporter: Thomas Mueller > Fix For: 1.3.3 > > Attachments: OAK-3007-2.patch, OAK-3007.patch > > > The SegmentStore cache size calculation ignores the size of the field > Segment.string (a concurrent hash map). It looks like a regular segment in a > memory mapped file has the size 1024, no matter how many strings are loaded > in memory. This can lead to out of memory. There seems to be no way to limit > (configure) the amount of memory used by strings. In one example, 100'000 > segments are loaded in memory, and 5 GB are used for Strings in that map. > We need a way to configure the amount of memory used for that. This seems to > be basically a cache. OAK-2688 does this, but it would be better to have one > cache with a configurable size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3007) SegmentStore cache does not take "string" map into account
[ https://issues.apache.org/jira/browse/OAK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616304#comment-14616304 ] Thomas Mueller commented on OAK-3007: - > Thomas Mueller, could you also add a unit test for StringCache? Sure. > I believe that the root cause for the compaction case filling up the cache is > is OAK-3075. but this patch should be applied nonetheless. As far as I understand, OAK-3075 is not the reason, but yes I understand that some other issue caused that this many strings are loaded. It would be nice to find the root cause. In order to find the root cause, the patch here (OAK-3007) could be applied and the compaction test could be re-run. That should no longer result in out-of-memory in SegmentStore cache, but in slow compaction, which in turn can be analyzed using a profiler. Or in yet another out-of-memory (but in a different place), which in turn can be analyzed... So the current patch of OAK-3007 should help to understand the _real_ problem. > but this patch should be applied nonetheless. Yes, I think applying this patch eventually to trunk (after writing more unit tests) would be good. Backporting it might not be all that urgent I guess. > SegmentStore cache does not take "string" map into account > -- > > Key: OAK-3007 > URL: https://issues.apache.org/jira/browse/OAK-3007 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segmentmk >Reporter: Thomas Mueller > Fix For: 1.3.3 > > Attachments: OAK-3007-2.patch, OAK-3007.patch > > > The SegmentStore cache size calculation ignores the size of the field > Segment.string (a concurrent hash map). It looks like a regular segment in a > memory mapped file has the size 1024, no matter how many strings are loaded > in memory. This can lead to out of memory. There seems to be no way to limit > (configure) the amount of memory used by strings. In one example, 100'000 > segments are loaded in memory, and 5 GB are used for Strings in that map. > We need a way to configure the amount of memory used for that. This seems to > be basically a cache. OAK-2688 does this, but it would be better to have one > cache with a configurable size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3007) SegmentStore cache does not take "string" map into account
[ https://issues.apache.org/jira/browse/OAK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614846#comment-14614846 ] Alex Parvulescu commented on OAK-3007: -- I believe that the root cause for the compaction case filling up the cache is is OAK-3075. but this patch should be applied nonetheless. > SegmentStore cache does not take "string" map into account > -- > > Key: OAK-3007 > URL: https://issues.apache.org/jira/browse/OAK-3007 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segmentmk >Reporter: Thomas Mueller > Fix For: 1.3.3 > > Attachments: OAK-3007-2.patch, OAK-3007.patch > > > The SegmentStore cache size calculation ignores the size of the field > Segment.string (a concurrent hash map). It looks like a regular segment in a > memory mapped file has the size 1024, no matter how many strings are loaded > in memory. This can lead to out of memory. There seems to be no way to limit > (configure) the amount of memory used by strings. In one example, 100'000 > segments are loaded in memory, and 5 GB are used for Strings in that map. > We need a way to configure the amount of memory used for that. This seems to > be basically a cache. OAK-2688 does this, but it would be better to have one > cache with a configurable size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3007) SegmentStore cache does not take "string" map into account
[ https://issues.apache.org/jira/browse/OAK-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14609783#comment-14609783 ] Michael Dürig commented on OAK-3007: I think this approach makes a lot of sense and we should give it a spin. [~tmueller], could you also add a unit test for {{StringCache}}? > SegmentStore cache does not take "string" map into account > -- > > Key: OAK-3007 > URL: https://issues.apache.org/jira/browse/OAK-3007 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segmentmk >Reporter: Thomas Mueller > Fix For: 1.3.2 > > Attachments: OAK-3007-2.patch, OAK-3007.patch > > > The SegmentStore cache size calculation ignores the size of the field > Segment.string (a concurrent hash map). It looks like a regular segment in a > memory mapped file has the size 1024, no matter how many strings are loaded > in memory. This can lead to out of memory. There seems to be no way to limit > (configure) the amount of memory used by strings. In one example, 100'000 > segments are loaded in memory, and 5 GB are used for Strings in that map. > We need a way to configure the amount of memory used for that. This seems to > be basically a cache. OAK-2688 does this, but it would be better to have one > cache with a configurable size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)