[
https://issues.apache.org/jira/browse/LUCENE-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16456844#comment-16456844
]
Adrien Grand commented on LUCENE-4892:
--------------------------------------
I have a proof of concept (see patch) that passes the tests and uses a hash set
of integers in the sparse case (until around 3% of deletions) and a bitset
afterward. The hash set tries to leverage the fact that it stores dense
integers by using the index in the hash table as a source of information to
reconstruct the original value, so it can store data in a short[] rather than
an int[]. It allows the hash set to use around 6 bytes per value in spite of
enforcing a load factor of at most 50%.
I still need to do more testing but wanted to upload a preview of this patch in
case someone has feedback on the approach.
> Create a compressed LiveDocsFormat
> ----------------------------------
>
> Key: LUCENE-4892
> URL: https://issues.apache.org/jira/browse/LUCENE-4892
> Project: Lucene - Core
> Issue Type: New Feature
> Reporter: Adrien Grand
> Priority: Trivial
> Labels: gsoc2014
> Attachments: LUCENE-4892.patch
>
>
> There are lots of use-cases where the number of deleted documents is low.
> This makes live docs very dense and I think it would be interesting to study
> the impact of some bitmap compression techniques on performance (intuitively
> I imagine it will be slower, but since it would make data smaller maybe CPU
> caches could help us so I'd be curious to see how it would behave). This
> format would make a good addition to our CheapBastardCodec.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
