[jira] [Commented] (OAK-2723) FileStore does not scale because of precomputed graph on TarReader
[ https://issues.apache.org/jira/browse/OAK-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495795#comment-14495795 ] Michael Dürig commented on OAK-2723: Fixed in trunk: http://svn.apache.org/r1673664 FileStore does not scale because of precomputed graph on TarReader -- Key: OAK-2723 URL: https://issues.apache.org/jira/browse/OAK-2723 Project: Jackrabbit Oak Issue Type: Bug Components: oak-core Affects Versions: 1.1.8 Reporter: Andrei Dulvac Assignee: Michael Dürig Priority: Critical Fix For: 1.3.0, 1.2.1 Attachments: 0001-TarReader-fix-for-precomputed-graph.patch The {{FileStore}} keeps a reference to all {{TarReader}} object, one per each file. In my test, for an ~350 Gb repository, that was ~1100 tar files, with a {{TarReader}} for each. The problem is {{TarReader}} keeps a reference to a precomputed _graph_ {{ByteBuffer}}, which is not really used that much. That means that through the {{readers}} field, there's a reference to these _graphs_, which means they can't be GC'ed. The construction of {{FileStore}} is from oak-run: bq. FileStore store = new FileStore(directory, 256, TAR_STORAGE_MEMORY_MAPPED); The effect is you need more that 6GB of Ram just to instantiate the {{FileStore}} object. The attached patch fixes this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2723) FileStore does not scale because of precomputed graph on TarReader
[ https://issues.apache.org/jira/browse/OAK-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495852#comment-14495852 ] Michael Dürig commented on OAK-2723: Fixed in 1.2 at http://svn.apache.org/r1673673 FileStore does not scale because of precomputed graph on TarReader -- Key: OAK-2723 URL: https://issues.apache.org/jira/browse/OAK-2723 Project: Jackrabbit Oak Issue Type: Bug Components: oak-core Affects Versions: 1.1.8 Reporter: Andrei Dulvac Assignee: Michael Dürig Priority: Critical Fix For: 1.3.0, 1.2.1 Attachments: 0001-TarReader-fix-for-precomputed-graph.patch The {{FileStore}} keeps a reference to all {{TarReader}} object, one per each file. In my test, for an ~350 Gb repository, that was ~1100 tar files, with a {{TarReader}} for each. The problem is {{TarReader}} keeps a reference to a precomputed _graph_ {{ByteBuffer}}, which is not really used that much. That means that through the {{readers}} field, there's a reference to these _graphs_, which means they can't be GC'ed. The construction of {{FileStore}} is from oak-run: bq. FileStore store = new FileStore(directory, 256, TAR_STORAGE_MEMORY_MAPPED); The effect is you need more that 6GB of Ram just to instantiate the {{FileStore}} object. The attached patch fixes this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2723) FileStore does not scale because of precomputed graph on TarReader
[ https://issues.apache.org/jira/browse/OAK-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485503#comment-14485503 ] Michael Dürig commented on OAK-2723: Patch looks good from a first look. However it has some performance impact on online compaction. Most likely neglectable though. I will include the patch in my testing for OAK-2713 and give it a spin. FileStore does not scale because of precomputed graph on TarReader -- Key: OAK-2723 URL: https://issues.apache.org/jira/browse/OAK-2723 Project: Jackrabbit Oak Issue Type: Bug Components: oak-core Affects Versions: 1.1.8 Reporter: Andrei Dulvac Assignee: Michael Dürig Priority: Critical Fix For: 1.2.1 Attachments: 0001-TarReader-fix-for-precomputed-graph.patch The {{FileStore}} keeps a reference to all {{TarReader}} object, one per each file. In my test, for an ~350 Gb repository, that was ~1100 tar files, with a {{TarReader}} for each. The problem is {{TarReader}} keeps a reference to a precomputed _graph_ {{ByteBuffer}}, which is not really used that much. That means that through the {{readers}} field, there's a reference to these _graphs_, which means they can't be GC'ed. The construction of {{FileStore}} is from oak-run: bq. FileStore store = new FileStore(directory, 256, TAR_STORAGE_MEMORY_MAPPED); The effect is you need more that 6GB of Ram just to instantiate the {{FileStore}} object. The attached patch fixes this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2723) FileStore does not scale because of precomputed graph on TarReader
[ https://issues.apache.org/jira/browse/OAK-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483073#comment-14483073 ] Andrei Dulvac commented on OAK-2723: If you need a memory dump, I can provide that offline, it's not possible (or desirable) to attach it in jira. FileStore does not scale because of precomputed graph on TarReader -- Key: OAK-2723 URL: https://issues.apache.org/jira/browse/OAK-2723 Project: Jackrabbit Oak Issue Type: Bug Components: oak-core Affects Versions: 1.1.8 Reporter: Andrei Dulvac Attachments: 0001-TarReader-fix-for-precomputed-graph.patch The {{FileStore}} keeps a reference to all {{TarReader}} object, one per each file. In my test, for an ~350 Gb repository, that was ~1100 tar files, with a {{TarReader}} for each. The problem is {{TarReader}} keeps a reference to a precomputed _graph_ {{ByteBuffer}}, which is not really used that much. The effect is you need more that 6GB of Ram just to instantiate the {{FileStore}} object. The attached patch fixes this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)