Hi everyone, We recently discovered a severe bug in the 9.4 release in the kNN vectors format: https://github.com/apache/lucene/issues/11858. Explaining the problem: when ingesting a lot of data, or when performing a force merge, segments can grow large. The format validation code accidentally uses an int instead of a long to compute the data size, so it can fail on these large segments. When format validation fails, the segment is essentially lost and unusable. For some client systems like Elasticsearch, it can send the whole index into a "failed" state, blocking further writes or searches.
I think this bug is sufficiently bad that we should perform a 9.4.1 release as soon as possible. The fix is just an update to the read-side validation code, there won't be any effect on the data format. This means it is safe to merge the fix into the existing 9.4 vectors format. The bug was introduced during the work to add quantization ( https://github.com/apache/lucene/pull/1054) and does not affect versions before 9.4. Let me know what you think! I could serve as release manager. (We should also follow up with a plan to prevent this from happening in the future -- maybe we need to regularly run larger-scale benchmarks?) Julie
