serhiy-bzhezytskyy commented on issue #16479: URL: https://github.com/apache/lucene/issues/16479#issuecomment-5159476763
#16480 implements this, in three commits: stored fields, term vectors, and one unrelated-looking fix that came out of the same measurement. Two things worth adding to the description above. **The third commit was found by measuring, not by reading.** `.tim` was checked for comparison, since blocktree also runs LZ4 over bytes read from disk. It turns out not to need a chunk checksum — 0.6% of sampled single-byte corruptions returned a wrong result silently, against 48.5% for `.fdt`, because `.tim` is mostly metadata and a full `TermsEnum` scan exposes discrepancies. But two of the 320 samples landed in `LowercaseAsciiCompression#decompress`, where the exception offsets are accumulated from the data and used to index the output with no bound check — the same shape as the LZ4 match offset in #16478. That is now a checked `IOException` with a test. **Back-compat was verified in both directions rather than assumed**, including a mixed index holding segments of both format versions, which reads correctly with each segment verified per its own version. A segment acquires checksums when it is next merged, under the default merge policy; a segment that is never merged keeps the old layout, and `UpgradeIndexMergePolicy` compares `Version.LATEST` rather than a format version, so within one major version it will not force the rewrite. Both are noted in the commit message rather than left implicit. Also relevant to #10396: the measurement there was a byte-flipping test that had to be disabled because not all corruptions were detected. The test in this PR asserts the property instead — no corruption may return a wrong document — and it fails on `main` with `expected:<0> but was:<16>`, which is what that test was trying to express. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
