[ https://issues.apache.org/jira/browse/LUCENE-4702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17018228#comment-17018228 ]
Adrien Grand commented on LUCENE-4702: -------------------------------------- I did some research on what's taking space in the terms dictionary, and while suffixes take a fair amount of space for text fields, it tends to rather be stats for ID fields, so I did a couple changes to also use LZ4 to do some run-length encoding for doc freqs (frequent runs of 1s for ids, and interestingly there are many runs of 1s for the body field of wikibigall too), suffix lengths, which are also frequently the same especially for ID fields (always the same for UUID or flake IDs and very little variance for auto-increment IDs). Finally we were wasting some space with the pulsing optimization too since we kept writing the delta of file pointers in spite of the fact that these deltas are almost always zeros for ID fields since we don't write postings in the doc file but in the terms dictionary. The compression is significantly better now as the size of the tim file goes down by 18% from 937MB to 767MB. Here are the stats for the body and id fields if you are curious: {code} "id" field index FST: 72 bytes terms: 6647577 terms 39885462 bytes (6.0 bytes/term) blocks: 189932 blocks 184655 terms-only blocks 5277 sub-block-only blocks 0 mixed blocks 0 floor blocks 189932 non-floor blocks 0 floor sub-blocks 14059850 term suffix bytes before compression (52.8 suffix-bytes/block) 10023973 compressed term suffix bytes (0.71 compression ratio - compression count by algorithm: NO_COMPRESSION: 189932) 6647577 term stats bytes before compression (11.7 stats-bytes/block) 2226414 compressed term stats bytes (0.33 compression ratio) 26962631 other bytes (142.0 other-bytes/block) {code} {code} "body" field index FST: 72 bytes terms: 46916528 terms 595069147 bytes (12.7 bytes/term) blocks: 1507239 blocks 1158537 terms-only blocks 471 sub-block-only blocks 348231 mixed blocks 318391 floor blocks 491775 non-floor blocks 1015464 floor sub-blocks 359880365 term suffix bytes before compression (196.3 suffix-bytes/block) 295898442 compressed term suffix bytes (0.82 compression ratio - compression count by algorithm: NO_COMPRESSION: 252273, LOWERCASE_ASCII: 1190011, LZ4: 64955) 94426201 term stats bytes before compression (45.1 stats-bytes/block) 68022105 compressed term stats bytes (0.72 compression ratio) 213996755 other bytes (142.0 other-bytes/block) {code} I see a 10% slowdown on PKLookup that I'll look into. > Terms dictionary compression > ---------------------------- > > Key: LUCENE-4702 > URL: https://issues.apache.org/jira/browse/LUCENE-4702 > Project: Lucene - Core > Issue Type: Wish > Reporter: Adrien Grand > Assignee: Adrien Grand > Priority: Trivial > Attachments: LUCENE-4702.patch, LUCENE-4702.patch > > Time Spent: 3h 20m > Remaining Estimate: 0h > > I've done a quick test with the block tree terms dictionary by replacing a > call to IndexOutput.writeBytes to write suffix bytes with a call to > LZ4.compressHC to test the peformance hit. Interestingly, search performance > was very good (see comparison table below) and the tim files were 14% smaller > (from 150432 bytes overall to 129516). > {noformat} > TaskQPS baseline StdDevQPS compressed StdDev > Pct diff > Fuzzy1 111.50 (2.0%) 78.78 (1.5%) > -29.4% ( -32% - -26%) > Fuzzy2 36.99 (2.7%) 28.59 (1.5%) > -22.7% ( -26% - -18%) > Respell 122.86 (2.1%) 103.89 (1.7%) > -15.4% ( -18% - -11%) > Wildcard 100.58 (4.3%) 94.42 (3.2%) > -6.1% ( -13% - 1%) > Prefix3 124.90 (5.7%) 122.67 (4.7%) > -1.8% ( -11% - 9%) > OrHighLow 169.87 (6.8%) 167.77 (8.0%) > -1.2% ( -15% - 14%) > LowTerm 1949.85 (4.5%) 1929.02 (3.4%) > -1.1% ( -8% - 7%) > AndHighLow 2011.95 (3.5%) 1991.85 (3.3%) > -1.0% ( -7% - 5%) > OrHighHigh 155.63 (6.7%) 154.12 (7.9%) > -1.0% ( -14% - 14%) > AndHighHigh 341.82 (1.2%) 339.49 (1.7%) > -0.7% ( -3% - 2%) > OrHighMed 217.55 (6.3%) 216.16 (7.1%) > -0.6% ( -13% - 13%) > IntNRQ 53.10 (10.9%) 52.90 (8.6%) > -0.4% ( -17% - 21%) > MedTerm 998.11 (3.8%) 994.82 (5.6%) > -0.3% ( -9% - 9%) > MedSpanNear 60.50 (3.7%) 60.36 (4.8%) > -0.2% ( -8% - 8%) > HighSpanNear 19.74 (4.5%) 19.72 (5.1%) > -0.1% ( -9% - 9%) > LowSpanNear 101.93 (3.2%) 101.82 (4.4%) > -0.1% ( -7% - 7%) > AndHighMed 366.18 (1.7%) 366.93 (1.7%) > 0.2% ( -3% - 3%) > PKLookup 237.28 (4.0%) 237.96 (4.2%) > 0.3% ( -7% - 8%) > MedPhrase 173.17 (4.7%) 174.69 (4.7%) > 0.9% ( -8% - 10%) > LowSloppyPhrase 180.91 (2.6%) 182.79 (2.7%) > 1.0% ( -4% - 6%) > LowPhrase 374.64 (5.5%) 379.11 (5.8%) > 1.2% ( -9% - 13%) > HighTerm 253.14 (7.9%) 256.97 (11.4%) > 1.5% ( -16% - 22%) > HighPhrase 19.52 (10.6%) 19.83 (11.0%) > 1.6% ( -18% - 25%) > MedSloppyPhrase 141.90 (2.6%) 144.11 (2.5%) > 1.6% ( -3% - 6%) > HighSloppyPhrase 25.26 (4.8%) 25.97 (5.0%) > 2.8% ( -6% - 13%) > {noformat} > Only queries which are very terms-dictionary-intensive got a performance hit > (Fuzzy, Fuzzy2, Respell, Wildcard), other queries including Prefix3 behaved > (surprisingly) well. > Do you think of it as something worth exploring? -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org