xiangfu0 commented on code in PR #20214:
URL: https://github.com/apache/pinot/pull/20214#discussion_r4530013342


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTracker.java:
##########
@@ -642,18 +643,28 @@ public long getLatestPartitionOffset(int partitionId) {
     return ((LongMsgOffset) latestOffset).getOffset();
   }
 
-  /// We use this method to clean up when a table is being removed. No updates 
are expected at this time as all
-  /// RealtimeSegmentManagers should be down now.
+  /// We use this method to clean up when a table is being removed. It does 
not throw, so the caller can go on to stop
+  /// the table's consumers.
   public void shutdown() {
     // Now that segments can't report metric, destroy metric for this table
     _metricsRemovalScheduler.shutdown(); // ScheduledExecutor is installed in 
constructor so must always be cancelled
     if (_ingestionDelayTrackingScheduler != null) {
-      _ingestionDelayTrackingScheduler.shutdown();
+      // Stop the tracking task before closing the stream metadata providers 
it fetches through: a stream consumer
+      // such as Kafka's is not safe for concurrent use, so closing it under 
an in-flight fetch throws.
+      _ingestionDelayTrackingScheduler.shutdownNow();

Review Comment:
   MINOR: the interrupt surfaces in the `catch` in `updateLatestStreamOffset` 
as an ERROR with a stack trace, and it increments 
`INGESTION_DELAY_LATEST_OFFSET_FETCH_ERRORS`. That happens on routine table 
deletes (seen in the local test log). Consider skipping the log and meter when 
`_realTimeTableDataManager.isShutDown()` or the thread is interrupted, so 
alerts on that meter don't fire.



##########
pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/IngestionDelayTrackerTest.java:
##########
@@ -157,6 +165,54 @@ private IngestionDelayTracker createTracker() {
     return new MockIngestionDelayTracker(_serverMetrics, REALTIME_TABLE_NAME, 
REALTIME_TABLE_DATA_MANAGER);
   }
 
+  /// Shutdown must stop an in-flight upstream offset fetch before closing the 
provider it runs on: a stream consumer
+  /// such as Kafka's throws ConcurrentModificationException when closed under 
a concurrent fetch.
+  @Test
+  public void testShutdownClosesProviderOnlyAfterInFlightFetchStops()
+      throws Exception {
+    IngestionDelayTracker tracker = new IngestionDelayTracker(_serverMetrics, 
REALTIME_TABLE_NAME,
+        REALTIME_TABLE_DATA_MANAGER, 60_000, 5, () -> true);
+    AtomicBoolean inFetch = new AtomicBoolean();
+    AtomicBoolean closedDuringFetch = new AtomicBoolean();
+    CountDownLatch fetchStarted = new CountDownLatch(1);
+    CountDownLatch neverReleased = new CountDownLatch(1);
+    StreamMetadataProvider provider = mock(StreamMetadataProvider.class);
+    when(provider.supportsOffsetLag()).thenReturn(true);
+    when(provider.fetchLatestStreamOffset(anySet(), 
anyLong())).thenAnswer(invocation -> {
+      inFetch.set(true);
+      fetchStarted.countDown();
+      try {
+        // Blocks until the tracking task is interrupted, like a consumer 
waiting on the broker
+        neverReleased.await();
+      } finally {
+        inFetch.set(false);
+      }
+      return Map.of();
+    });
+    doAnswer(invocation -> {
+      closedDuringFetch.set(inFetch.get());
+      return null;
+    }).when(provider).close();
+    tracker._streamConfigIndexToStreamMetadataProvider.put(0, provider);
+    tracker._partitionsHostedByThisServer.put(0, true);
+
+    assertTrue(fetchStarted.await(10, TimeUnit.SECONDS), "The tracking task 
should start a fetch");
+    tracker.shutdown();
+    assertFalse(closedDuringFetch.get(), "The provider was closed while a 
fetch was still running on it");

Review Comment:
   MINOR: this also passes if `close()` is never called. Adding 
`verify(provider).close();` after the assert would pin down the other half of 
the contract: the provider is closed, and only after the fetch stops.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to