[
https://issues.apache.org/jira/browse/KAFKA-19403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-19403:
------------------------------------
Component/s: (was: connect)
> We're facing issue in Kafka while reading data from Azure event hubs through
> Azure Databricks
> ---------------------------------------------------------------------------------------------
>
> Key: KAFKA-19403
> URL: https://issues.apache.org/jira/browse/KAFKA-19403
> Project: Kafka
> Issue Type: Bug
> Components: consumer, network
> Affects Versions: 3.3.1
> Environment: Production
> Reporter: karthickthavasiraj09
> Priority: Major
>
> * We are experiencing significant slowness while reading data from Azure
> Event Hubs using Azure Databricks. After conducting initial analysis with the
> Microsoft support team, they confirmed that the root cause appears to be
> related to Kafka. We are reaching out for your assistance in investigating
> and resolving this issue.
> Below are the key findings and debug logs provided by the Microsoft team:
> ** The data read operation took *49 minutes* in total.
> ** Out of this, only one task(spark task) {*}Task 143 alone took 46
> minutes{*}, indicating a bottleneck in this specific task.
> ** The job duration was {*}49 minutes and 30 seconds{*}.
> Relevant Log Snippets:
> 25/04/15 14:21:44 INFO KafkaBatchReaderFactoryWithRowBytesAccumulator:
> Creating Kafka reader topicPartition=<topic-name>-0 fromOffset=16511904
> untilOffset=16658164,
> for queryId=dd660d4d-05cc-4a8e-8f93-d202ec78fec3
> runId=af7eb711-7310-4788-85b7-0977fc0756b7 batchId=73 taskId=143 partitionId=0
> 25/04/15 15:07:21 INFO KafkaDataConsumer:
> From Kafka topicPartition=<topic-name>-0
> groupId=spark-kafka-source-da79e0fc-8ee5-40f5-a127-7b31766b3022--1737876659-executor
> read 146260 records through 4314 polls (polled out 146265 records), taking
> 2526471821132 ns,
> over a timespan of 2736294068630 ns.
>
> Additionally, the thread stack trace indicates that the task was mostly
> waiting on Kafka to respond. See the following thread details captured during
> the slowness:
> Executor task launch worker for task 0.0 in stage 147.0 (TID 143)
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> ...
> kafkashaded.org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1759)
> ...
> org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer.fetchRecord(KafkaDataConsumer.scala:517)
> *We kindly request the Kafka team to look into this issue. The task appears
> to be blocked for an extended period during Kafka polling. Any insights into
> why Kafka is not responding promptly or recommendations for configuration
> changes or optimizations would be greatly appreciated.*
> *Please let us know if any additional information or diagnostics are required
> from our end.*
> *Thank you for your support.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
