[
https://issues.apache.org/jira/browse/SPARK-19853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu resolved SPARK-19853.
----------------------------------
Resolution: Fixed
Assignee: Genmao Yu
Fix Version/s: 2.2.0
2.1.1
> Uppercase Kafka topics fail when startingOffsets are SpecificOffsets
> --------------------------------------------------------------------
>
> Key: SPARK-19853
> URL: https://issues.apache.org/jira/browse/SPARK-19853
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.1.0
> Reporter: Chris Bowden
> Assignee: Genmao Yu
> Priority: Trivial
> Fix For: 2.1.1, 2.2.0
>
>
> When using the KafkaSource with Structured Streaming, consumer assignments
> are not what the user expects if startingOffsets is set to an explicit set of
> topics/partitions in JSON where the topic(s) happen to have uppercase
> characters. When StartingOffsets is constructed, the original string value
> from options is transformed toLowerCase to make matching on "earliest" and
> "latest" case insensitive. However, the toLowerCase json is passed to
> SpecificOffsets for the terminal condition, so topic names may not be what
> the user intended by the time assignments are made with the underlying
> KafkaConsumer.
> From KafkaSourceProvider:
> {code}
> val startingOffsets =
> caseInsensitiveParams.get(STARTING_OFFSETS_OPTION_KEY).map(_.trim.toLowerCase)
> match {
> case Some("latest") => LatestOffsets
> case Some("earliest") => EarliestOffsets
> case Some(json) => SpecificOffsets(JsonUtils.partitionOffsets(json))
> case None => LatestOffsets
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
