[
https://issues.apache.org/jira/browse/SPARK-25466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969262#comment-16969262
]
Gabor Somogyi commented on SPARK-25466:
---------------------------------------
Well, if you think there is a bug in Spark please collect the driver + executor
logs, open a new bug jira and attach it.
Having partial code on Stack Overflow which is half way DStreams, half way
Structured Streaming and it was originally throwing exception by default
doesn't help.
> Documentation does not specify how to set Kafka consumer cache capacity for SS
> ------------------------------------------------------------------------------
>
> Key: SPARK-25466
> URL: https://issues.apache.org/jira/browse/SPARK-25466
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, Structured Streaming
> Affects Versions: 2.3.0
> Reporter: Patrick McGloin
> Priority: Minor
>
> When hitting this warning with SS:
> 19-09-2018 12:05:27 WARN CachedKafkaConsumer:66 - KafkaConsumer cache
> hitting max capacity of 64, removing consumer for
> CacheKey(spark-kafka-source-e06c9676-32c6-49c4-80a9-2d0ac4590609--694285871-executor,MyKafkaTopic-30)
> If you Google you get to this page:
> https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> Which is for Spark Streaming and says to use this config item to adjust the
> capacity: "spark.streaming.kafka.consumer.cache.maxCapacity".
> This is a bit confusing as SS uses a different config item:
> "spark.sql.kafkaConsumerCache.capacity"
> Perhaps the SS Kafka documentation should talk about the consumer cache
> capacity? Perhaps here?
> https://spark.apache.org/docs/2.2.0/structured-streaming-kafka-integration.html
> Or perhaps the warning message should reference the config item. E.g
> 19-09-2018 12:05:27 WARN CachedKafkaConsumer:66 - KafkaConsumer cache
> hitting max capacity of 64, removing consumer for
> CacheKey(spark-kafka-source-e06c9676-32c6-49c4-80a9-2d0ac4590609--694285871-executor,MyKafkaTopic-30).
> *The cache size can be adjusted with the setting
> "spark.sql.kafkaConsumerCache.capacity".*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
