xiangfu0 commented on code in PR #19900:
URL: https://github.com/apache/pinot/pull/19900#discussion_r4290272474
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/VectorIndexHandler.java:
##########
@@ -637,7 +734,7 @@ && hasCombinedFile(indexDir, columnName, backendType)) {
}
// Delete the marker file.
- FileUtils.deleteQuietly(inProgress);
+ FileUtils.forceDelete(inProgress);
Review Comment:
Good — this now matches `TextIndexHandler`'s end-of-build `forceDelete`, so
a failed marker cleanup surfaces instead of silently looping recovery. The
enclosing method already throws, and the marker is always present here, so no
false FileNotFound.
##########
README.md:
##########
@@ -175,6 +175,21 @@ To verify which profile is active, run:
If you encounter issues, check that the `protoc-gen-grpc-java` binary is
present in one of the expected locations and is executable.
+### Lucene index recovery during loading
+
+Normal segment preprocessing automatically rebuilds text and HNSW indexes
whose Lucene format cannot be read by
+the running version. Single-column text, multi-column text, and
directory/packed HNSW indexes are rebuilt using
+the segment's original forward-index values and existing table configuration;
no recovery flag is needed.
+Compatible indexes are reused, and subsequent loads do not rebuild the
recovered indexes again. Interrupted builds
+restart from the retained source values before the segment is loaded.
+
+Install this recovery release on the intended rollback target before upgrading
Lucene. After a downgrade, loading
+segments containing newer Lucene indexes rebuilds those indexes with the older
writer. Retain the original forward
+indexes: dictionary/inverted-index reconstruction can lose multi-value
ordering and duplicate values. When original
+values are unavailable, recovery fails before replacing the incompatible
index; restore a compatible segment or
+refresh it from retained source data. Rebuilding adds CPU/I/O and
segment-loading time. Existing preprocessing
+opt-outs remain in effect. Corruption and ordinary I/O errors fail loading
instead of triggering format recovery.
Review Comment:
Wording is accurate now: corruption/IO errors rethrow as
IllegalStateException (via `isIncompatible`) and fail the load rather than
triggering recovery.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
