[
https://issues.apache.org/jira/browse/LUCENE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrien Grand resolved LUCENE-3953.
----------------------------------
Resolution: Fixed
Fix Version/s: master (7.0)
This looks mostly due to FieldCache storing data in a dense way. Now that
FieldCache has been replaced with doc values, you would need many more fields
in order to trigger OOMEs, but disk usage and indexing speed would still suffer
from these sparse fields. I think we can consider it as fixed in master (7.0)
thanks to the new sparse APIs for doc values and the new doc values format.
That said, it is still advised to have dense fields whenever possible. We made
sparse fields more cost-efficient, but there is still a penalty compared to
dense fields.
> Using many fields in filters, sorts causes OOM
> ----------------------------------------------
>
> Key: LUCENE-3953
> URL: https://issues.apache.org/jira/browse/LUCENE-3953
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
> Affects Versions: 4.0-ALPHA
> Reporter: Des Lownds
> Fix For: master (7.0)
>
> Attachments: FieldCacheLimitPatch.patch
>
>
> I have an index with ~1M documents, and many uniquely named fields. Each
> document only has a few fields populated, but there is a large set of dynamic
> fields across all docs. In other words, theres many sparse fields.
> Eventually the jvm will fall over due to OOM errors, because lucene's
> fieldCache grows without bounds. Adding more heap does not help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
