tibrewalpratik17 commented on code in PR #13256:
URL: https://github.com/apache/pinot/pull/13256#discussion_r1623985038
##########
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/UpsertConfig.java:
##########
@@ -94,6 +94,9 @@ public enum ConsistencyMode {
@JsonPropertyDescription("Whether to drop out-of-order record")
private boolean _dropOutOfOrderRecord;
+ @JsonPropertyDescription("Whether to pause partial upsert table's partition
ingestion during commit")
+ private boolean _pausePartialUpsertPartitionIngestionDuringCommit;
+
Review Comment:
@rohityadav1993 can you also add it as a cluster level config where this
table-level config will override the cluster-level value? In Uber, we would
like to roll it out for almost all tables as data-correctness >
data-ingestion-SLAs. For tables where ingestion lag is in order of 10s of
minutes there we can make a call to use this table level config.
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -1316,7 +1322,13 @@ public void goOnlineFromConsuming(SegmentZKMetadata
segmentZKMetadata)
protected void downloadSegmentAndReplace(SegmentZKMetadata segmentZKMetadata)
throws Exception {
- closeStreamConsumers();
+ // for partial upsert tables, do not release
_partitionGroupConsumerSemaphore proactively and rely on onDestroy()
+ // to release the semaphore. This ensures new consuming segment is not
consuming until the segment replacement is
+ // complete.
+ if (!_realtimeTableDataManager.isPartialUpsertEnabled() ||
!_tableConfig.getUpsertConfig()
+ .isPausePartialUpsertPartitionIngestionDuringCommit()) {
Review Comment:
we should check if `tableConfig.getUpsertConfig` is not null as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
