zhang-arvin opened a new pull request, #20072:
URL: https://github.com/apache/druid/pull/20072
## Purpose
Fixes #18779: KafkaIndexSupervisor enters UNHEALTHY state on recoverable
errors, blocking partition consumption.
## Problem
When a Kafka ingestion task's duration exceeds the configured
`taskDuration`, or when task-level operations (like `checkTaskDuration()`,
`updateTaskStatus()`, `checkCurrentTaskState()`,
`checkPendingCompletionTasks()`) fail with `ExecutionException` or
`InterruptedException`, the supervisor's `runInternal()` method catches all
exceptions and records them as throwable events via
`stateManager.recordThrowableEvent(e)`. After `unhealthinessThreshold`
consecutive failed runs, the supervisor transitions to `UNHEALTHY_SUPERVISOR`
state, which blocks all partition consumption.
These task-level exceptions are recoverable — they arise from individual
task communication issues (e.g., task timeouts, task unresponsiveness) and do
not indicate a supervisor-level failure. The supervisor should retry on the
next iteration rather than entering UNHEALTHY state.
## Fix
Modified `SeekableStreamSupervisor.runInternal()` to distinguish between
recoverable task-level errors and non-recoverable supervisor-level errors:
- **Recoverable**: `ExecutionException` and `InterruptedException` — logged
as warnings without calling `recordThrowableEvent()`, so the run is not marked
as failed.
- **Non-recoverable**: `StreamException` and other exceptions — still
recorded as throwable events, potentially triggering UNHEALTHY state.
This ensures that transient task communication failures don't cause the
supervisor to stop consuming partitions.
## Changes
-
`indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java`:
- Added import for `java.util.concurrent.ExecutionException`
- Modified `runInternal()` catch block to handle
`ExecutionException`/`InterruptedException` as recoverable errors
This PR has:
- [x] been self-reviewed.
- [x] added documentation for new or modified features or behaviors.
- [x] added Javadocs for most classes and all non-trivial methods.
- [x] added comments explaining the "why" and the intent of the code
wherever it is not obvious from reading the code.
- [x] added unit tests or modified existing tests to cover new code paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
