jpountz commented on issue #12317: URL: https://github.com/apache/lucene/issues/12317#issuecomment-1578953436
Sorry for the lag! I've been out for some time but I am back now. In general, we don't like adding options to file formats and prefer to have full control to keep file formats easy to reason about and to test. The object that you are referring to (`LZ4CompressionHashTable`) only gets allocated once per field per segment, which is not much. So I wouldn't generally expect it to be a big contributor to a heap profile unless there are many small segments getting written, which could happen if you do frequent refreshes, have many fields, or many indices. In that case it's possible that LZ4 compression never gets used on some fields/segments because of the checks on prefix length and average suffix length, so your idea to lazily allocate this compression hash table might help? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
