[
https://issues.apache.org/jira/browse/LUCENE-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992687#comment-12992687
]
hao yan commented on LUCENE-2903:
---------------------------------
Hi, Robert and Michael
In order to test if ByteBuffer/IntBuffer works better than int[]<->byte[]
conversion, I now separate them into 3 different codecs. All of them use the
same PForDelta implementation except that they use different
indexinput/indexoutput as follows.
1. PatchedFrameOfRef3 -> use in.readBytes(), it will convert int[] <-> byte[]
manually. Its corresponding java code is: PForDeltaFixedIntBlockCodec.java
2. PatchedFrameOfRef4 -> use in.readBytes(), it will convert int[] <-> byte[]
by ByteBuffer/IntBuffer. Its corresponding java code is:
PForDeltaFixedIntBlockWithByteBufferCodec.java
3. PatchedFrameOfRef5 -> use in.readInt() with a loop, it does not need
conversion. Its corresponding java code is:
PForDeltaFixedIntBlockWithReadIntCodec.java
I tested them against BulkVInt on MacOS. The detailed results are attached.
Here is the conclusion:
1) Yes, Michael and Robert, you guys are right! ByteBuffer/IntBuffer are faster
then my manual conversion btw byte[]/int[]. I guess the reason I thought they
were worse is that i did not separate codecs before, such that the test results
is not stable due to JVM/JIT.
2) Now, PatchedFrameOfRef4 is still worse than BulkVInt in many kinds of
queries. However, it seems that it can do better for fuzzy queries and
wildcardquery.
3) Of course, these PatchedFrameOfRef3,4,5 are all better than
PatchedFrameOfRef and FrameOfRef for almost all queries.
4) The new patched is just uploaded, please check them out.
The following is the experimental results for 0.1M data.
(1) bulkVInt VS patchedFrameOfRef4 (withByteBuffer, in.readBytes(..) )
QueryQPS bulkVIntQPS pathcedFrameofref4-withByteBuffer Pct diff
"united states" 389.26 361.79 -7.1%
"united states"~3 234.52 228.99 -2.4%
+nebraska +states 1138.95 992.06 -12.9%
+united +states 670.69 603.86 -10.0%
doctimesecnum:[10000 TO 60000] 415.28 447.83 7.8%
doctitle:.*[Uu]nited.* 496.03 522.47 5.3%
spanFirst(unit, 5) 1176.47 1086.96 -7.6%
spanNear([unit, state], 10, true) 502.26 423.73 -15.6%
states 1612.90 1453.49 -9.9%
u*d 167.95 171.17 1.9%
un*d 260.69 275.33 5.6%
uni* 602.41 577.37 -4.2%
unit* 1016.26 1041.67 2.5%
united states 617.28 549.45 -11.0%
united~0.6 12.22 12.93 5.9%
united~0.75 53.88 56.78 5.4%
unit~0.5 12.58 13.19 4.9%
unit~0.7 52.41 54.93 4.8%
(2) bulkVInt VS patchedFrameOfRef3 (with my own int[] <-> byte[] conversion,
still in.readBytes(..))
QueryQPS bulkVIntQPS pathcedFrameofref3 Pct diff
"united states" 388.50 363.24 -6.5%
"united states"~3 234.80 223.56 -4.8%
+nebraska +states 1138.95 1016.26 -10.8%
+united +states 671.14 607.90 -9.4%
doctimesecnum:[10000 TO 60000] 418.24 441.89 5.7%
doctitle:.*[Uu]nited.* 489.00 522.74 6.9%
spanFirst(unit, 5) 1246.88 1127.40 -9.6%
spanNear([unit, state], 10, true) 514.14 473.71 -7.9%
states 1612.90 1488.10 -7.7%
u*d 170.77 167.31 -2.0%
un*d 261.37 264.48 1.2%
uni* 609.38 602.41 -1.1%
unit* 1028.81 1052.63 2.3%
united states 614.25 564.33 -8.1%
united~0.6 12.05 12.11 0.5%
united~0.75 53.16 54.97 3.4%
unit~0.5 12.43 12.50 0.6%
unit~0.7 52.81 53.23 0.8%
(3) bulkVInt VS patchedFrameOfRef5 (with my own int[] <-> byte[] conversion,
still in.readBytes(..))
QueryQPS bulkVIntQPS pathcedFrameofref5-withReadInt Pct diff
"united states" 391.24 366.70 -6.3%
"united states"~3 235.40 235.07 -0.1%
+nebraska +states 1137.66 1072.96 -5.7%
+united +states 673.40 642.26 -4.6%
doctimesecnum:[10000 TO 60000] 414.25 407.66 -1.6%
doctitle:.*[Uu]nited.* 492.61 538.21 9.3%
spanFirst(unit, 5) 1253.13 1175.09 -6.2%
spanNear([unit, state], 10, true) 511.25 483.56 -5.4%
states 1642.04 1490.31 -9.2%
u*d 166.78 160.28 -3.9%
un*d 261.64 255.36 -2.4%
uni* 609.38 593.47 -2.6%
unit* 1026.69 1042.75 1.6%
united states 606.43 580.72 -4.2%
united~0.6 12.07 11.18 -7.4%
united~0.75 53.57 53.91 0.6%
unit~0.5 12.49 11.54 -7.6%
unit~0.7 52.75 51.76 -1.9%
> Improvement of PForDelta Codec
> ------------------------------
>
> Key: LUCENE-2903
> URL: https://issues.apache.org/jira/browse/LUCENE-2903
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: hao yan
> Attachments: LUCENE_2903.patch, LUCENE_2903.patch
>
>
> There are 3 versions of PForDelta implementations in the Bulk Branch:
> FrameOfRef, PatchedFrameOfRef, and PatchedFrameOfRef2.
> The FrameOfRef is a very basic one which is essentially a binary encoding
> (may result in huge index size).
> The PatchedFrameOfRef is the implmentation based on the original version of
> PForDelta in the literatures.
> The PatchedFrameOfRef2 is my previous implementation which are improved this
> time. (The Codec name is changed to NewPForDelta.).
> In particular, the changes are:
> 1. I fixed the bug of my previous version (in Lucene-1410.patch), where the
> old PForDelta does not support very large exceptions (since
> the Simple16 does not support very large numbers). Now this has been fixed in
> the new LCPForDelta.
> 2. I changed the PForDeltaFixedIntBlockCodec. Now it is faster than the other
> two PForDelta implementation in the bulk branch (FrameOfRef and
> PatchedFrameOfRef). The codec's name is "NewPForDelta", as you can see in the
> CodecProvider and PForDeltaFixedIntBlockCodec.
> 3. The performance test results are:
> 1) My "NewPForDelta" codec is faster then FrameOfRef and PatchedFrameOfRef
> for almost all kinds of queries, slightly worse then BulkVInt.
> 2) My "NewPForDelta" codec can result in the smallest index size among all 4
> methods, including FrameOfRef, PatchedFrameOfRef, and BulkVInt, and itself)
> 3) All performance test results are achieved by running with "-server"
> instead of "-client"
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
