[
https://issues.apache.org/jira/browse/SPARK-20287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15971893#comment-15971893
]
Stephane Maarek commented on SPARK-20287:
-----------------------------------------
[~c...@koeninger.org] It makes sense. I didn't realized in the direct streams,
that the driver was in charge of assigning metadata to the executors to pull
data. Therefore yes you're right, it's "incompatible" with the Kafka way of
being "master-free", where each consumer really doesn't know and shouldn't care
about how many other consumers there are. I think this ticket can now be closed
(just re-open it if you don't believe so). Maybe it'll be worth opening a KIP
on Kafka to have some APIs to allow Spark to be a bit more "optimized", but it
all seems okay for now. Cheers!
> Kafka Consumer should be able to subscribe to more than one topic partition
> ---------------------------------------------------------------------------
>
> Key: SPARK-20287
> URL: https://issues.apache.org/jira/browse/SPARK-20287
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.1.0
> Reporter: Stephane Maarek
>
> As I understand and as it stands, one Kafka Consumer is created for each
> topic partition in the source Kafka topics, and they're cached.
> cf
> https://github.com/apache/spark/blob/master/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/CachedKafkaConsumer.scala#L48
> In my opinion, that makes the design an anti pattern for Kafka and highly
> unefficient:
> - Each Kafka consumer creates a connection to Kafka
> - Spark doesn't leverage the power of the Kafka consumers, which is that it
> automatically assigns and balances partitions amongst all the consumers that
> share the same group.id
> - You can still cache your Kafka consumer even if it has multiple partitions.
> I'm not sure about how that translates to the spark underlying RDD
> architecture, but from a Kafka standpoint, I believe creating one consumer
> per partition is a big overhead, and a risk as the user may have to increase
> the spark.streaming.kafka.consumer.cache.maxCapacity parameter.
> Happy to discuss to understand the rationale
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
