[
https://issues.apache.org/jira/browse/LUCENE-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837231#action_12837231
]
Renaud Delbru commented on LUCENE-1410:
---------------------------------------
I am reporting here some experiments I performed over the past weeks while I
was trying to improve the FOR implementation.
I re-implement the FOR algorithms by getting rid of the IntBuffer and working
directly with the byte array. I have implemented multiple variants, such as one
that directly performs transformation over bytes to create the uncompressed
data (what I call byte-level in the next), and one that first convert bytes
into integers, and then performs transformation on integers to create the
uncompress data (what I call integer-level in the next). The last one is very
similar to your original FOR implementation, but without the IntBuffer.
I think these results can be of interest for you, especially to optimise
certain cases (byte level manipulation for certain cases such as bit frame of
2, 4 or 8 seems more suitable). I have attached a file containing a summary of
the results for space consideration. I can provide you the raw results, and the
code if you would like to test it on your side.
However, I get very different results if I perform the benchmarks on a 64 bits
OS or 32 Bits OS (on a same computer, IBM T61, the only difference is that on
one computer Ubuntu 9.10 32 bits is installed, on the other one it is Ubuntu
9.10 64 bits).
I am a little bit puzzled by these results. I thought that removing the
IntBuffer and working directly with the byte array will be faster (as I noticed
in other compression algorithms, such as GroupVarInt). The IntBuffer you are
currently using is a view on a byte buffer. It therefore does the conversion
between byte to int, plus it does several checks (if the index is in the range
of the buffer) and function calls.
But it seems that with FOR this does not make too much a difference on large
integers (> 8 bits). Moreover, I observe a decrease of performance on 64 bits
OS.
Maybe, you have an idea about the difference in behavior.
> PFOR implementation
> -------------------
>
> Key: LUCENE-1410
> URL: https://issues.apache.org/jira/browse/LUCENE-1410
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Other
> Reporter: Paul Elschot
> Priority: Minor
> Attachments: autogen.tgz, LUCENE-1410-codecs.tar.bz2,
> LUCENE-1410b.patch, LUCENE-1410c.patch, LUCENE-1410d.patch,
> LUCENE-1410e.patch, TermQueryTests.tgz, TestPFor2.java, TestPFor2.java,
> TestPFor2.java
>
> Original Estimate: 21840h
> Remaining Estimate: 21840h
>
> Implementation of Patched Frame of Reference.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
