[
https://issues.apache.org/jira/browse/SPARK-49442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877397#comment-17877397
]
Jungtaek Lim commented on SPARK-49442:
--------------------------------------
OK, that's unrelated. We haven't got any report for this kind of complaint. I
recommend you to provide a minimal reproducer e.g. Apache Spark cluster &
Apache Kafka cluster (no vendor version and no cloud service version), topic
partition to 3-5 and increase topic-partition and prove that the metadata
requests increase linearly, with the detailed explanation about how you capture
the requests. If you are relying on any vendor rather than building the cluster
on your own, it'd be ideal to contact with the support.
> Complete Metadata requests on each micro batch causing Kafka issues
> -------------------------------------------------------------------
>
> Key: SPARK-49442
> URL: https://issues.apache.org/jira/browse/SPARK-49442
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.3.2
> Reporter: vipin Kumar
> Priority: Major
> Labels: Kafka, spark-streaming-kafka
>
> We have noticed that spark does complete metadata requests on each micro
> batch and this is causing high metadata requests on small micro batch
> intervals .
>
> For example Kafka with 1900 partitions and 10 sec micro batch we are seeing
> order of
> ~{*}360K{*} metadata requests / sec
> Same with job with 60 sec micro batch we are observing *~60K* meta data
> requests.
>
> Metadata requests are controlled by *metadata.max.age.ms* but these config
> have no effect on spark consumers by default its 5 mins still we are seeing
> these huge requests.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
