[
https://issues.apache.org/jira/browse/KAFKA-7653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7653:
-----------------------------------
Description:
Implicit resolution/conversion of Serdes/Consumed etc is a big improvement for
the Scala Streams API. However in cases where a user needs to differentiate
between Key and Value serializer functionality (i.e. using the Schema
Registry), implicit resolution doesn't help and could cause issues.
e.g.
{code:java}
case class MouseClickEvent(pageId: Long, userId: String)
builder
// Long serde taken from implicit scope configured with
// `isKey` = true
.stream[Long, MouseClickEvent]("mouse-clicks")
.selectKey((_,v) => v.userId)
.groupByKey
.aggregate(() => 0L, (_: String, mce: MouseClickEvent, count: Long) => count
+ 1)
.toStream
// Same Long serde taken from implicit scope configured with
// `isKey` = true, even thought the `Long` value in this case
// will be the Value
.to("mouse-clicks-by-user")
{code}
It would be ideal if Key and Value Serde/SerdeWrapper types/type classes could
be introduced to overcome this limitation.
KIP-513:
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-513%3A+Distinguish+between+Key+and+Value+serdes+in+scala+wrapper+library+for+kafka+streams]
was:
Implicit resolution/conversion of Serdes/Consumed etc is a big improvement for
the Scala Streams API. However in cases where a user needs to differentiate
between Key and Value serializer functionality (i.e. using the Schema
Registry), implicit resolution doesn't help and could cause issues.
e.g.
{code:java}
case class MouseClickEvent(pageId: Long, userId: String)
builder
// Long serde taken from implicit scope configured with
// `isKey` = true
.stream[Long, MouseClickEvent]("mouse-clicks")
.selectKey((_,v) => v.userId)
.groupByKey
.aggregate(() => 0L, (_: String, mce: MouseClickEvent, count: Long) => count
+ 1)
.toStream
// Same Long serde taken from implicit scope configured with
// `isKey` = true, even thought the `Long` value in this case
// will be the Value
.to("mouse-clicks-by-user")
{code}
It would be ideal if Key and Value Serde/SerdeWrapper types/type classes could
be introduced to overcome this limitation.
> Streams-Scala: Add type level differentiation for Key and Value serdes.
> -----------------------------------------------------------------------
>
> Key: KAFKA-7653
> URL: https://issues.apache.org/jira/browse/KAFKA-7653
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Mark Tranter
> Assignee: Mark Tranter
> Priority: Minor
> Labels: kip, scala
>
> Implicit resolution/conversion of Serdes/Consumed etc is a big improvement
> for the Scala Streams API. However in cases where a user needs to
> differentiate between Key and Value serializer functionality (i.e. using the
> Schema Registry), implicit resolution doesn't help and could cause issues.
> e.g.
> {code:java}
> case class MouseClickEvent(pageId: Long, userId: String)
> builder
> // Long serde taken from implicit scope configured with
> // `isKey` = true
> .stream[Long, MouseClickEvent]("mouse-clicks")
> .selectKey((_,v) => v.userId)
> .groupByKey
> .aggregate(() => 0L, (_: String, mce: MouseClickEvent, count: Long) =>
> count + 1)
> .toStream
> // Same Long serde taken from implicit scope configured with
> // `isKey` = true, even thought the `Long` value in this case
> // will be the Value
> .to("mouse-clicks-by-user")
> {code}
> It would be ideal if Key and Value Serde/SerdeWrapper types/type classes
> could be introduced to overcome this limitation.
> KIP-513:
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-513%3A+Distinguish+between+Key+and+Value+serdes+in+scala+wrapper+library+for+kafka+streams]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
