benwtrent commented on issue #13478: URL: https://github.com/apache/lucene/issues/13478#issuecomment-2160815181
I think I know the issue with the parallel merging. This only happens when we use a SortingCodecReader. The key issue is here: https://github.com/apache/lucene/commit/17c285d61743da0c06735e06235b20bd5aac4e14 This adjusted the caching as to where: ``` private <T> T getOrCreateNorms(String field, IOSupplier<T> supplier) throws IOException { return getOrCreate(field, true, supplier); } private synchronized <T> T getOrCreate(String field, boolean norms, IOSupplier<T> supplier) throws IOException { if ((field.equals(cachedField) && cacheIsNorms == norms) == false) { assert assertCreatedOnlyOnce(field, norms); cachedObject = supplier.get(); cachedField = field; cacheIsNorms = norms; } assert cachedObject != null; return (T) cachedObject; } private <T> T getOrCreateDV(String field, IOSupplier<T> supplier) throws IOException { return getOrCreate(field, false, supplier); } ``` This will cause a weird race condition as when merging norms will call `getOrCreateNorms` and in parallel, we could be calling `getOrCreateDV`, either of which will overwrite the other, and then potentially double cache. Parallel merging breaks these assumptions and could cause issues. @iverase @jpountz I propose we remove intra-merging parallelism from norms, terms, and doc values and do a 9.11.1 release. We can incrementally add those back in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
