yupeng9 commented on a change in pull request #7860:
URL: https://github.com/apache/pinot/pull/7860#discussion_r762089400
##########
File path:
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java
##########
@@ -472,12 +472,14 @@ public boolean index(GenericRow row, @Nullable
RowMetadata rowMetadata)
throws IOException {
boolean canTakeMore;
if (isUpsertEnabled()) {
- row = handleUpsert(row, _numDocsIndexed);
-
- updateDictionary(row);
- addNewRow(row);
- // Update number of documents indexed at last to make the latest row
queryable
+ PartitionUpsertMetadataManager.RecordInfo recordInfo =
getRecordInfo(row, _numDocsIndexed);
+ GenericRow updatedRow =
_partitionUpsertMetadataManager.updateRecord(row, recordInfo);
+ updateDictionary(updatedRow);
+ addNewRow(updatedRow);
+ // Update number of documents indexed before handling the upsert
metadata so that the record becomes queryable
+ // once validated
canTakeMore = _numDocsIndexed++ < _capacity;
+ _partitionUpsertMetadataManager.addRecord(this, recordInfo);
Review comment:
I think the consistency issue is always there, regardless of the order.
Either we see duplicates, or we see missing records.
If we want to truly solve this problem, then we have to introduce a lock
mechanism on the segment update/read per PK, though it would lead to
performance implications.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
