[
https://issues.apache.org/jira/browse/SOLR-2888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Weiss resolved SOLR-2888.
-------------------------------
Resolution: Fixed
In trunk.
> FSTSuggester refactoring: utf8 storage, external sorts (OOM prevention), code
> cleanups
> --------------------------------------------------------------------------------------
>
> Key: SOLR-2888
> URL: https://issues.apache.org/jira/browse/SOLR-2888
> Project: Solr
> Issue Type: Improvement
> Components: spellchecker
> Reporter: Dawid Weiss
> Assignee: Dawid Weiss
> Fix For: 4.0
>
> Attachments: SOLR-2888.patch, SOLR-2888.patch, SOLR-2888.patch
>
>
> This issue incorporates several problems:
> - utf16 was used previously to store and lookup terms, now it is utf8
> - the construction would OOM with large number of terms because of the need
> to sort entries. Sorter APIs have been added and an implementation of
> external (on-disk) sorting is also added (Robert Muir).
> - the FSTLookup class has been split and refactored into FSTCompletion and
> FSTCompletionBuilder, FSTCompletionLookup remains a facade connecting all the
> pieces and implements Lookup interface. For large inputs use
> FSTCompletionBuilder directly (and pre-bucket your input weights).
> - Automatic bucketing in FSTCompletionLookup has been changed from linear
> min/max discretization into dividing into ranges after all values have been
> sorted. This empirically handles all potential distributions quite well. If
> somebody needs something very specific, use FSTCompletionBuilder directly
> (providing buckets), construct the automaton and then load it with
> FSTCompletionLookup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
