tibrewalpratik17 opened a new pull request, #13257: URL: https://github.com/apache/pinot/pull/13257
label: `upsert` `bugfix` Saw this in our Uber cluster today where ingestion got stopped due to some exception in snapshot flow. The following was the error we encountered (pinot-version-build: 1.0.0): ``` java.lang.IllegalArgumentException: Path: /data/upinot/stream-pinot-server/dataDir/<tableName>/<tableName>__3__31__20240427T0642Z is not a directory at com.google.common.base.Preconditions.checkArgument(Preconditions.java:210) at org.apache.pinot.segment.spi.store.SegmentDirectoryPaths.findSegmentDirectory(SegmentDirectoryPaths.java:50) at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentImpl.getValidDocIdsSnapshotFile(ImmutableSegmentImpl.java:172) at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentImpl.hasValidDocIdsSnapshotFile(ImmutableSegmentImpl.java:152) at org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager.doTakeSnapshot(BasePartitionUpsertMetadataManager.java:667) at org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager.takeSnapshot(BasePartitionUpsertMetadataManager.java:637) at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:679) at java.base/java.lang.Thread.run(Thread.java:829) ``` Ideally the segment was actually deleted but somehow was still a part of `trackedSegments`. This patch prevents the consumption thread from erroring out if the snapshot flow has any encountered issue. cc @Jackie-Jiang @klsince -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
