xiangfu0 commented on code in PR #20214:
URL: https://github.com/apache/pinot/pull/20214#discussion_r4530013342
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java:
##########
@@ -642,18 +643,28 @@ public long getLatestPartitionOffset(int partitionId) {
return ((LongMsgOffset) latestOffset).getOffset();
}
- /// We use this method to clean up when a table is being removed. No updates
are expected at this time as all
- /// RealtimeSegmentManagers should be down now.
+ /// We use this method to clean up when a table is being removed. It does
not throw, so the caller can go on to stop
+ /// the table's consumers.
public void shutdown() {
// Now that segments can't report metric, destroy metric for this table
_metricsRemovalScheduler.shutdown(); // ScheduledExecutor is installed in
constructor so must always be cancelled
if (_ingestionDelayTrackingScheduler != null) {
- _ingestionDelayTrackingScheduler.shutdown();
+ // Stop the tracking task before closing the stream metadata providers
it fetches through: a stream consumer
+ // such as Kafka's is not safe for concurrent use, so closing it under
an in-flight fetch throws.
+ _ingestionDelayTrackingScheduler.shutdownNow();
Review Comment:
MINOR: the interrupt surfaces in the `catch` in `updateLatestStreamOffset`
as an ERROR with a stack trace, and it increments
`INGESTION_DELAY_LATEST_OFFSET_FETCH_ERRORS`. That happens on routine table
deletes (seen in the local test log). Consider skipping the log and meter when
`_realTimeTableDataManager.isShutDown()` or the thread is interrupted, so
alerts on that meter don't fire.
##########
pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTrackerTest.java:
##########
@@ -157,6 +165,54 @@ private IngestionDelayTracker createTracker() {
return new MockIngestionDelayTracker(_serverMetrics, REALTIME_TABLE_NAME,
REALTIME_TABLE_DATA_MANAGER);
}
+ /// Shutdown must stop an in-flight upstream offset fetch before closing the
provider it runs on: a stream consumer
+ /// such as Kafka's throws ConcurrentModificationException when closed under
a concurrent fetch.
+ @Test
+ public void testShutdownClosesProviderOnlyAfterInFlightFetchStops()
+ throws Exception {
+ IngestionDelayTracker tracker = new IngestionDelayTracker(_serverMetrics,
REALTIME_TABLE_NAME,
+ REALTIME_TABLE_DATA_MANAGER, 60_000, 5, () -> true);
+ AtomicBoolean inFetch = new AtomicBoolean();
+ AtomicBoolean closedDuringFetch = new AtomicBoolean();
+ CountDownLatch fetchStarted = new CountDownLatch(1);
+ CountDownLatch neverReleased = new CountDownLatch(1);
+ StreamMetadataProvider provider = mock(StreamMetadataProvider.class);
+ when(provider.supportsOffsetLag()).thenReturn(true);
+ when(provider.fetchLatestStreamOffset(anySet(),
anyLong())).thenAnswer(invocation -> {
+ inFetch.set(true);
+ fetchStarted.countDown();
+ try {
+ // Blocks until the tracking task is interrupted, like a consumer
waiting on the broker
+ neverReleased.await();
+ } finally {
+ inFetch.set(false);
+ }
+ return Map.of();
+ });
+ doAnswer(invocation -> {
+ closedDuringFetch.set(inFetch.get());
+ return null;
+ }).when(provider).close();
+ tracker._streamConfigIndexToStreamMetadataProvider.put(0, provider);
+ tracker._partitionsHostedByThisServer.put(0, true);
+
+ assertTrue(fetchStarted.await(10, TimeUnit.SECONDS), "The tracking task
should start a fetch");
+ tracker.shutdown();
+ assertFalse(closedDuringFetch.get(), "The provider was closed while a
fetch was still running on it");
Review Comment:
MINOR: this also passes if `close()` is never called. Adding
`verify(provider).close();` after the assert would pin down the other half of
the contract: the provider is closed, and only after the fetch stops.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]