[
https://issues.apache.org/jira/browse/LUCENE-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Willnauer updated LUCENE-3507:
------------------------------------
Attachment: LUCENE-3507.patch
here is a patch that merges sorted docvalues without loading into main memory.
This patch remaps the ordinals, prunes "unreferenced" ords during merge and
streams the new sorted datastructure into the new segment without loading into
memory. Yet, there is still memory used to remap ords etc. but far less than
what it used to require.
We also don't need to hash and later sort the merged values again during merge
since they are already sorted.
> Improve Memory Consumption for merging DocValues SortedBytes variants
> ---------------------------------------------------------------------
>
> Key: LUCENE-3507
> URL: https://issues.apache.org/jira/browse/LUCENE-3507
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0
>
> Attachments: LUCENE-3507.patch
>
>
> Currently SortedBytes are loaded into memory during merge which could be a
> potential trap. Instead of loading them into Heap memory we can merge those
> sorted values with much smaller memory and without loading all values into
> ram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
