[
https://issues.apache.org/jira/browse/OAK-7947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16714647#comment-16714647
]
Vikas Saurabh commented on OAK-7947:
------------------------------------
[~tmueller], the patch you had attached seems quite risky to me (as it touches
quite a lot of places) and would solve "avoid opening index as long as possible
wrt index definitions". If index definition can potentially answer the query
and we want to open index to say get num docs or num docs per field then we
would still copy in all index files. I've a at least one comment on the patch
which I'd note at the end.
Maybe we could try a different approach - let index open happen as it happens
today but copy required files right away (synchronously) and schedule rest of
the files for later. Here's a snip of size sorted list of files from a 11G
{{damAssetLucene}} index that [~chibulcu] had provided me from an AEM isntance:
{noformat}
$ ls -lsSh
total 11G
4.5G -rw-r--r-- 1 vsaurabh vsaurabh 4.5G Nov 23 12:20 _101z.fdt
4.5G -rw-r--r-- 1 vsaurabh vsaurabh 4.5G Nov 23 13:43 _1zt4.fdt
580M -rw-r--r-- 1 vsaurabh vsaurabh 580M Nov 23 12:20 _101z.pos
579M -rw-r--r-- 1 vsaurabh vsaurabh 579M Nov 23 13:43 _1zt4.pos
177M -rw-r--r-- 1 vsaurabh vsaurabh 177M Nov 23 13:44 _20z0.cfs
106M -rw-r--r-- 1 vsaurabh vsaurabh 106M Nov 23 12:20 _1x4o.cfs
65M -rw-r--r-- 1 vsaurabh vsaurabh 65M Nov 23 13:44 _20bb.cfs
29M -rw-r--r-- 1 vsaurabh vsaurabh 29M Nov 23 13:44 _217z.cfs
16M -rw-r--r-- 1 vsaurabh vsaurabh 16M Nov 23 12:10 _101z.doc
16M -rw-r--r-- 1 vsaurabh vsaurabh 16M Nov 23 12:20 _1zt4.doc
6.7M -rw-r--r-- 1 vsaurabh vsaurabh 6.7M Nov 23 13:44 _21ef.cfs
6.5M -rw-r--r-- 1 vsaurabh vsaurabh 6.5M Nov 23 13:44 _216f.cfs
6.3M -rw-r--r-- 1 vsaurabh vsaurabh 6.3M Nov 23 12:20 _101z.tim
5.9M -rw-r--r-- 1 vsaurabh vsaurabh 5.9M Nov 23 13:43 _1zt4.tim
5.9M -rw-r--r-- 1 vsaurabh vsaurabh 5.9M Nov 23 13:44 _21cy.cfs
4.4M -rw-r--r-- 1 vsaurabh vsaurabh 4.4M Nov 23 13:44 _21ab.cfs
3.8M -rw-r--r-- 1 vsaurabh vsaurabh 3.8M Nov 23 13:44 _21e4.cfs
3.7M -rw-r--r-- 1 vsaurabh vsaurabh 3.7M Nov 23 13:44 _21du.cfs
3.0M -rw-r--r-- 1 vsaurabh vsaurabh 3.0M Nov 23 13:44 _21dk.cfs
2.6M -rw-r--r-- 1 vsaurabh vsaurabh 2.6M Nov 23 13:44 _21f1.cfs
648K -rw-r--r-- 1 vsaurabh vsaurabh 647K Nov 23 12:10 _101z.dvd
424K -rw-r--r-- 1 vsaurabh vsaurabh 421K Nov 23 12:20 _1zt4.dvd
380K -rw-r--r-- 1 vsaurabh vsaurabh 378K Nov 23 12:20 _101z.fdx
372K -rw-r--r-- 1 vsaurabh vsaurabh 369K Nov 23 13:43 _1zt4.fdx
120K -rw-r--r-- 1 vsaurabh vsaurabh 120K Nov 23 13:44 _21f7.cfs
120K -rw-r--r-- 1 vsaurabh vsaurabh 120K Nov 23 13:44 _21f4.cfs
....
....
{noformat}
Looking at
https://lucene.apache.org/core/4_7_1/core/org/apache/lucene/codecs/lucene46/package-summary.html,
{{fdt}} files are stored field data and {{pos}} is positional data for indexed
terms. Both these shouldn't get loaded only for cost evaluation afaict (we
should probably try to confirm this btw). These 2 form the biggest chunk of the
files - so, maybe only avoiding these to get copied over just to open an index
would save us a lot of time for first time index open. Additionally, I think
this approach is much less risky imo.
_patch review_
The changes in
{noformat}
public LuceneIndexDefinition getIndexDefinition(String indexPath){
{noformat}
when index isn't in index map is providing a definition which is visible in
tree and not from stored index definition that gets stored. This would change
the behavior of planner to start to use un-indexed index definition as well.
Afaics, the other changes are essentially doing lazy init and won't affect
behavior afaics - but it does make it a little brittle to control to avoid
index open (an unrelated part of code might start call some part that would in
turn happily open the index).
> Lazy loading of Lucene index files startup
> ------------------------------------------
>
> Key: OAK-7947
> URL: https://issues.apache.org/jira/browse/OAK-7947
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene, query
> Reporter: Thomas Mueller
> Assignee: Thomas Mueller
> Priority: Major
> Attachments: OAK-7947.patch
>
>
> Right now, all Lucene index binaries are loaded on startup (I think when the
> first query is run, to do cost calculation). This is a performance problem if
> the index files are large, and need to be downloaded from the data store.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
