[ https://issues.apache.org/jira/browse/LUCENE-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankit Jain updated LUCENE-8783: ------------------------------- Review Patch?: (was: Yes) > Support FST lazy loading for non-default Codecs > ----------------------------------------------- > > Key: LUCENE-8783 > URL: https://issues.apache.org/jira/browse/LUCENE-8783 > Project: Lucene - Core > Issue Type: New Feature > Components: core/FSTs > Reporter: Ankit Jain > Priority: Major > Fix For: 8.0, 8.x, master (9.0) > > > Currently, FST loads all the terms into heap memory during index open. This > causes frequent JVM OOM issues if the term size gets big. A better way of > doing this will be to lazily load FST using mmap. That ensures only the > required terms get loaded into memory. > > Lucene can expose API for providing list of fields to load terms offheap. I'm > planning to take following approach for this: > # Add a boolean property fstOffHeap in FieldInfo > # Pass list of offheap fields to lucene during index open (ALL can be > special keyword for loading ALL fields offheap) > # Initialize the fstOffHeap property during lucene index open > # FieldReader invokes default FST constructor or OffHeap constructor based > on fstOffHeap field > > I created a patch (that loads all fields offheap), did some benchmarks using > es_rally and results look good. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org