[
https://issues.apache.org/jira/browse/SPARK-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905991#comment-15905991
]
LvDongrong commented on SPARK-19863:
------------------------------------
I see your comment on that issue(SPARK-19185), and I am agree with you. Our
problem is different, our kafka Cluster cannot support so many connections,
which is established by cached consumers to the kafka ,because the num of our
topic and partition is large. So I think it is necessary not to use cached
consumer in some cases.
> Whether or not use CachedKafkaConsumer need to be configured, when you use
> DirectKafkaInputDStream to connect the kafka in a Spark Streaming application
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-19863
> URL: https://issues.apache.org/jira/browse/SPARK-19863
> Project: Spark
> Issue Type: Bug
> Components: DStreams, Input/Output
> Affects Versions: 2.1.0
> Reporter: LvDongrong
>
> Whether or not use CachedKafkaConsumer need to be configured, when you use
> DirectKafkaInputDStream to connect the kafka in a Spark Streaming
> application. In Spark 2.x, the kafka consumer was replaced by
> CachedKafkaConsumer (some KafkaConsumer will keep establishing the kafka
> cluster), and cannot change the way. In fact ,The KafkaRDD(used by
> DirectKafkaInputDStream to connect kafka) provide the parameter
> useConsumerCache to choose Whether to use the CachedKafkaConsumer, but the
> DirectKafkaInputDStream set the parameter true.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
