xiangfu0 commented on code in PR #19900:
URL: https://github.com/apache/pinot/pull/19900#discussion_r4289611099
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/VectorIndexHandler.java:
##########
@@ -591,6 +685,9 @@ private void
createVectorIndexForColumn(SegmentDirectory.Writer segmentWriter, C
String columnName = columnMetadata.getColumnName();
VectorIndexConfig config =
_fieldIndexConfigs.get(columnName).getConfig(StandardIndexes.vector());
VectorBackendType backendType = config.resolveBackendType();
+ if (backendType == VectorBackendType.HNSW &&
hasHnswRecoveryMarker(columnName)) {
Review Comment:
Addressed in fbe82e7e25: successful vector rebuilds now use forceDelete for
the current marker, so cleanup failures propagate instead of silently leaving
recovery pending. All 31 focused vector/HNSW tests passed.
##########
README.md:
##########
@@ -175,6 +175,21 @@ To verify which profile is active, run:
If you encounter issues, check that the `protoc-gen-grpc-java` binary is
present in one of the expected locations and is executable.
+### Lucene index recovery during loading
+
+Normal segment preprocessing automatically rebuilds text and HNSW indexes
whose Lucene format cannot be read by
+the running version. Single-column text, multi-column text, and
directory/packed HNSW indexes are rebuilt using
+the segment's original forward-index values and existing table configuration;
no recovery flag is needed.
+Compatible indexes are reused, and subsequent loads do not rebuild the
recovered indexes again. Interrupted builds
+restart from the retained source values before the segment is loaded.
+
+Install this recovery release on the intended rollback target before upgrading
Lucene. After a downgrade, loading
+segments containing newer Lucene indexes rebuilds those indexes with the older
writer. Retain the original forward
+indexes: dictionary/inverted-index reconstruction can lose multi-value
ordering and duplicate values. When original
+values are unavailable, recovery fails before replacing the incompatible
index; restore a compatible segment or
+refresh it from retained source data. Rebuilding adds CPU/I/O and
segment-loading time. Existing preprocessing
+opt-outs remain in effect, and corruption or ordinary I/O errors are not
treated as format incompatibility.
Review Comment:
Clarified in fbe82e7e25: corruption and ordinary I/O errors fail loading
instead of triggering format recovery.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
