[
https://issues.apache.org/jira/browse/SPARK-54039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishanth updated SPARK-54039:
-----------------------------
Description:
Currently, the log messages in {{KafkaDataConsumer}} do not include Spark task
context information (e.g., {{{}TaskID{}}}) during the {{release()}} phase.
This makes it harder to correlate consumer release metrics and timing with the
specific Spark tasks that used the consumer, especially when multiple tasks are
borrowing consumers concurrently.
This improvement adds task context details (such as TaskID) to the log
statements in the {{release()}} method to improve traceability and debugging of
Kafka consumer lifecycle events.
was:
Kafka consumer logs generated by {{KafkaDataConsumer}} do not include the Spark
{{TaskID}} or task context information.
This makes it difficult to correlate specific Kafka fetch or poll operations
with the corresponding Spark tasks during debugging and performance
investigations—especially when multiple tasks consume from the same Kafka topic
partitions concurrently on different executors.
h3. *Current Behavior*
The current Kafka source consumer logging framework does not include {{TaskID}}
information in the {{KafkaDataConsumer}} class.
*Example log output:*
{code:java}
14:30:16 INFO Executor: Running task 83.0 in stage 32.0 (TID 4188)
14:35:19 INFO KafkaDataConsumer: From Kafka topicPartition=*******
groupId=******* read 12922 records through 80 polls (polled out 12979 records),
taking 299911.706198 ms, during time span of 303617.473973 ms <-- No Task
Context Here
14:35:19 INFO Executor: Finished task 83.0 in stage 32.0 (TID 4188){code}
Although the Executor logs include the task context (TID), the
KafkaDataConsumer logs do not, making it difficult to correlate Kafka read
durations or latency with the specific Spark tasks that executed them. This
limits visibility into which part of a job spends the most time consuming from
Kafka.
> SS | `Add TaskID to KafkaDataConsumer logs for better correlation between
> Spark tasks and Kafka operations`
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-54039
> URL: https://issues.apache.org/jira/browse/SPARK-54039
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.4.1, 3.5.0, 3.5.2, 4.0.0
> Reporter: Nishanth
> Priority: Major
> Labels: pull-request-available
>
> Currently, the log messages in {{KafkaDataConsumer}} do not include Spark
> task context information (e.g., {{{}TaskID{}}}) during the {{release()}}
> phase.
> This makes it harder to correlate consumer release metrics and timing with
> the specific Spark tasks that used the consumer, especially when multiple
> tasks are borrowing consumers concurrently.
> This improvement adds task context details (such as TaskID) to the log
> statements in the {{release()}} method to improve traceability and debugging
> of Kafka consumer lifecycle events.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
