[jira] [Commented] (OAK-7285) Reindexing using --doc-traversal-mode can OOM while aggregation in some cases
[
https://issues.apache.org/jira/browse/OAK-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381522#comment-16381522
]
Vikas Saurabh commented on OAK-7285:
Backported to 1.8 in [r1825623|https://svn.apache.org/r1825623].
> Reindexing using --doc-traversal-mode can OOM while aggregation in some cases
> -
>
> Key: OAK-7285
> URL: https://issues.apache.org/jira/browse/OAK-7285
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene, mongomk
>Affects Versions: 1.8.0
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Major
> Fix For: 1.9.0, 1.10, 1.8.3
>
>
> {{--doc-traversal-mode}} works on the notion of {{preferred}} children which
> is computed using path fragments that form aggregate rules.
> The idea is reading through aggregated paths should avoid keeping non useful
> nodes (for path being currently indexed) in memory.
> But, currently, in case, say when there multiple preferred children -
> {{jcr:content}}, {{metadata}}, then an index defn indexing parent of a very
> deep tree root would try to read in the whole tree before concluding that it
> doesn't have preferred children
> e.g. with preferred list - {{jcr:content}} and {{metadata}} and index looking
> for {{jcr:content}} indexing following structure
> {noformat}
> + /path/being/indexed
>+ very
> + very
> + very
>+ deep
>+ tree
> + /some-sibling
> {noformat}
> Currently, while looking for {{jcr:content}}, the code concludes that it
> doesn't exist only after reaching {{/some-sibling}} (or if number of children
> read of {{/path/being/indexed}} is >= num_preferred_children).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7285) Reindexing using --doc-traversal-mode can OOM while aggregation in some cases
[
https://issues.apache.org/jira/browse/OAK-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379015#comment-16379015
]
Vikas Saurabh commented on OAK-7285:
Thanks for the review [~chetanm]. Hopefully, changes done in
[r1825475|https://svn.apache.org/r1825475] would fit the bill.
> Reindexing using --doc-traversal-mode can OOM while aggregation in some cases
> -
>
> Key: OAK-7285
> URL: https://issues.apache.org/jira/browse/OAK-7285
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene, mongomk
>Affects Versions: 1.8.0
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Major
> Labels: candidate_oak_1_8
> Fix For: 1.9.0, 1.10
>
>
> {{--doc-traversal-mode}} works on the notion of {{preferred}} children which
> is computed using path fragments that form aggregate rules.
> The idea is reading through aggregated paths should avoid keeping non useful
> nodes (for path being currently indexed) in memory.
> But, currently, in case, say when there multiple preferred children -
> {{jcr:content}}, {{metadata}}, then an index defn indexing parent of a very
> deep tree root would try to read in the whole tree before concluding that it
> doesn't have preferred children
> e.g. with preferred list - {{jcr:content}} and {{metadata}} and index looking
> for {{jcr:content}} indexing following structure
> {noformat}
> + /path/being/indexed
>+ very
> + very
> + very
>+ deep
>+ tree
> + /some-sibling
> {noformat}
> Currently, while looking for {{jcr:content}}, the code concludes that it
> doesn't exist only after reaching {{/some-sibling}} (or if number of children
> read of {{/path/being/indexed}} is >= num_preferred_children).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7285) Reindexing using --doc-traversal-mode can OOM while aggregation in some cases
[
https://issues.apache.org/jira/browse/OAK-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378929#comment-16378929
]
Chetan Mehrotra commented on OAK-7285:
--
First thing we can do is make "preferredPathElements" a Set instead of iterable
which would simply the code later and make it more clear
{code}
if (pitr.hasNext() && isAncestor(path, pitr.peek().getPath())) {
NodeStateEntry nextEntry = pitr.next();
String nextEntryName =
PathUtils.getName(nextEntry.getPath());
if (preferred && !Iterables.contains(preferredPathElements,
nextEntryName)) {
return endOfData();
}
return nextEntry;
}
{code}
Here {{nextEntryName}} would child node name of immediate or descendent child.
It may break flow if we have 2 preferred path elements and first has a child
Say preferred path is "a", "b"
{noformat}
a
a/x
a/x/y
b
{noformat}
Here iterator would break early and would not return 'b'. So we should only
check for preferred condition only if the path is immediate child path.
> Reindexing using --doc-traversal-mode can OOM while aggregation in some cases
> -
>
> Key: OAK-7285
> URL: https://issues.apache.org/jira/browse/OAK-7285
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene, mongomk
>Affects Versions: 1.8.0
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Major
> Labels: candidate_oak_1_8
> Fix For: 1.9.0, 1.10
>
>
> {{--doc-traversal-mode}} works on the notion of {{preferred}} children which
> is computed using path fragments that form aggregate rules.
> The idea is reading through aggregated paths should avoid keeping non useful
> nodes (for path being currently indexed) in memory.
> But, currently, in case, say when there multiple preferred children -
> {{jcr:content}}, {{metadata}}, then an index defn indexing parent of a very
> deep tree root would try to read in the whole tree before concluding that it
> doesn't have preferred children
> e.g. with preferred list - {{jcr:content}} and {{metadata}} and index looking
> for {{jcr:content}} indexing following structure
> {noformat}
> + /path/being/indexed
>+ very
> + very
> + very
>+ deep
>+ tree
> + /some-sibling
> {noformat}
> Currently, while looking for {{jcr:content}}, the code concludes that it
> doesn't exist only after reaching {{/some-sibling}} (or if number of children
> read of {{/path/being/indexed}} is >= num_preferred_children).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OAK-7285) Reindexing using --doc-traversal-mode can OOM while aggregation in some cases
[
https://issues.apache.org/jira/browse/OAK-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377067#comment-16377067
]
Vikas Saurabh commented on OAK-7285:
Following test in {{FaltFileStoreIteratorTest}} shows the behavior (not
committing ignored test, as we'd probably need to restructure
{{FlatFileStoreIteartor}} ctor to take in preferred list instead of just size:
{noformat}
@Test
public void getChildNodeForNonPreferred() {
// have more than 1 preferred names
Set preferred = ImmutableSet.of("j:c", "md");
CountingIterable citr = createList(preferred,
asList("/a", "/a/b", "/a/c"));
FlatFileStoreIterator fitr = new FlatFileStoreIterator(citr.iterator(),
preferred.size());
NodeStateEntry a = fitr.next();
assertEquals("/a", a.getPath());
NodeState aNS = a.getNodeState();
aNS.getChildNode("j:c");
// Don't read whole tree to conclude that "j:c" doesn't exist (reading
/a/b should imply that it doesn't exist)
assertEquals(1, fitr.getBufferSize());
}
{noformat}
> Reindexing using --doc-traversal-mode can OOM while aggregation in some cases
> -
>
> Key: OAK-7285
> URL: https://issues.apache.org/jira/browse/OAK-7285
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene, mongomk
>Affects Versions: 1.8.0
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Major
> Labels: candidate_oak_1_8
> Fix For: 1.10
>
>
> {{--doc-traversal-mode}} works on the notion of {{preferred}} children which
> is computed using path fragments that form aggregate rules.
> The idea is reading through aggregated paths should avoid keeping non useful
> nodes (for path being currently indexed) in memory.
> But, currently, in case, say when there multiple preferred children -
> {{jcr:content}}, {{metadata}}, then an index defn indexing parent of a very
> deep tree root would try to read in the whole tree before concluding that it
> doesn't have preferred children
> e.g. with preferred list - {{j:c}} and {{md} and index looking for {{j:c}}
> indexing following structure
> {noformat}
> + /path/being/indexed
>+ very
> + very
> + very
>+ deep
>+ tree
> + /some-sibling
> {noformat}
> Currently, while looking for {{j:c}}, the code concludes that it doesn't
> exist only after reaching {{/some-sibling}} (or if number of children read of
> {{/path/being/indexed}} is >= num_preferred_children).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
