[ https://issues.apache.org/jira/browse/LUCENE-5914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14117986#comment-14117986 ]
Eks Dev commented on LUCENE-5914: --------------------------------- bq. Do you have pointers to emails/irc logs describing such issues? I do not know what the gold standard lucene usage is, but at least one use case I can describe, maybe it helps. I am not proposing anything here, just sharing experience. Think about the (typical lucene?) usage with structured data (e.g. indexing relational db, like product catalog or such) with many smallish fields and then retrieving 2k such documents to post-process them, classify, cluster them or whatnot (e.g. mahout and co.) - Default compression with CHUNK_SIZE makes it decompress 2k * CHUNK_SIZE/2 bytes on average in order to retrieve 2k Documents - Reducing chunk_size helps a lot, but there is a sweet-spot, and if you reduce it too much, you will not see enough compression and then your index is not fitting into cache , so you get hurt on IO. Ideally we should enable to use biggish chunk_size during compression to improve compression and decompress only single document (not depending on chunk_size), just like you proposed here (if I figured it out correctly?) Usually, such data is highly compressible (imagine all these low cardinality fields like color of something...) and even some basic compression does the magic. What we did? - Reduced chunk_size - As a bonus to improve compression, added plain static dictionary compression for a few fields in update chain (we store analysed fields) - When applicable, we pre-sort collection periodically before indexing (on low cardinality fields first).... this old db-admin secret weapon helps a lot Conclusion: compression is great, and anything that helps tweak this balance (CPU effort / IO effort) in different phases indexing/retrieving smoothly makes lucene use case coverage broader. (e.g. "I want to afford more CPU during indexing, and less CPU during retrieval", static coder being extreme case for this...) I am not sure I figured out exactly if and how this patch is going to help in a such cases (how to achieve reasonable compression if we do per document compression for small documents? Reusing dictionaries from previous chunks? static dictionaries... ). In any case, thanks for doing the heavy lifting here! I think you already did really great job with compression in lucene. PS: Ages ago, before lucene, when memory was really expensive, we had our own serialization (not in lucene) that simply had one static Huffman coder per field (with byte or word symbols), with code-table populated offline, that was great, simple option as it enabled reasonable compression for "slow changing collections" and really fast random access. > More options for stored fields compression > ------------------------------------------ > > Key: LUCENE-5914 > URL: https://issues.apache.org/jira/browse/LUCENE-5914 > Project: Lucene - Core > Issue Type: Improvement > Reporter: Adrien Grand > Assignee: Adrien Grand > Fix For: 4.11 > > Attachments: LUCENE-5914.patch > > > Since we added codec-level compression in Lucene 4.1 I think I got about the > same amount of users complaining that compression was too aggressive and that > compression was too light. > I think it is due to the fact that we have users that are doing very > different things with Lucene. For example if you have a small index that fits > in the filesystem cache (or is close to), then you might never pay for actual > disk seeks and in such a case the fact that the current stored fields format > needs to over-decompress data can sensibly slow search down on cheap queries. > On the other hand, it is more and more common to use Lucene for things like > log analytics, and in that case you have huge amounts of data for which you > don't care much about stored fields performance. However it is very > frustrating to notice that the data that you store takes several times less > space when you gzip it compared to your index although Lucene claims to > compress stored fields. > For that reason, I think it would be nice to have some kind of options that > would allow to trade speed for compression in the default codec. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org