[
https://issues.apache.org/jira/browse/SPARK-26121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cody Koeninger resolved SPARK-26121.
------------------------------------
Resolution: Fixed
Assignee: Anastasios Zouzias
Fix Version/s: 3.0.0
resolved by pr https://github.com/apache/spark/pull/23103
> [Structured Streaming] Allow users to define prefix of Kafka's consumer group
> (group.id)
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-26121
> URL: https://issues.apache.org/jira/browse/SPARK-26121
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.4.0
> Reporter: Anastasios Zouzias
> Assignee: Anastasios Zouzias
> Priority: Minor
> Fix For: 3.0.0
>
>
> I run in the following situation with Spark Structure Streaming (SS) using
> Kafka.
>
> In a project that I work on, there is already a secured Kafka setup where ops
> can issue an SSL certificate per "[group.id|http://group.id/]";, which should
> be predefined (or its prefix to be predefined).
>
> On the other hand, Spark SS fixes the [group.id|http://group.id/] to
>
> val uniqueGroupId =
> s"spark-kafka-source-${UUID.randomUUID}-${metadataPath.hashCode}"
>
> see, i.e.,
>
> [https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L124]
>
> https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L81
>
> I guess Spark developers had a good reason to fix it, but is it possible to
> make configurable the prefix of the above uniqueGroupId
> ("spark-kafka-source")?
>
> The rational is that spark users are not forced to use the same certificate
> on group-ids of the form (spark-kafka-source-*).
>
> DoD:
> * Allow spark SS users to define the group.id prefix as input parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
