[
https://issues.apache.org/jira/browse/SPARK-54039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishanth updated SPARK-54039:
-----------------------------
Description:
Currently, log messages in the {{release()}} method of {{KafkaDataConsumer}} do
not include Spark task context information (e.g., TaskID).
This change adds task context details to the {{release()}} logs to improve
traceability and debugging of Kafka consumer lifecycle events.
We previously added similar context in
{{{}KafkaBatchReaderFactory.createReader{}}}, which has proven useful for
debugging. Including this information in {{release()}} helps correlate consumer
release metrics with the specific Spark tasks that borrowed the consumer.
was:
Currently, the log messages in {{KafkaDataConsumer}} do not include Spark task
context information (e.g., {{{}TaskID{}}}) during the {{release()}} phase.
This makes it harder to correlate consumer release metrics and timing with the
specific Spark tasks that used the consumer, especially when multiple tasks are
borrowing consumers concurrently.
This improvement adds task context details (such as TaskID) to the log
statements in the {{release()}} method to improve traceability and debugging of
Kafka consumer lifecycle events.
> SS | `Add TaskContext information to KafkaDataConsumer release() logs for
> better debugging`
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-54039
> URL: https://issues.apache.org/jira/browse/SPARK-54039
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.4.1, 3.5.0, 3.5.2, 4.0.0
> Reporter: Nishanth
> Priority: Major
> Labels: pull-request-available
>
> Currently, log messages in the {{release()}} method of {{KafkaDataConsumer}}
> do not include Spark task context information (e.g., TaskID).
> This change adds task context details to the {{release()}} logs to improve
> traceability and debugging of Kafka consumer lifecycle events.
> We previously added similar context in
> {{{}KafkaBatchReaderFactory.createReader{}}}, which has proven useful for
> debugging. Including this information in {{release()}} helps correlate
> consumer release metrics with the specific Spark tasks that borrowed the
> consumer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
