[
https://issues.apache.org/jira/browse/KAFKA-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173506#comment-16173506
]
Ewen Cheslack-Postava commented on KAFKA-4714:
----------------------------------------------
[~dhananjaydp] You cannot do that with SMTs, you would need to define two
connectors as you describe. The KIP probably could have included some more
information about why. Your use case is really a special case of the more
general problem of a single record generating multiple outputs, with some of
those outputs possibly transformed in some way. Supporting this would
significantly complicate the system, in ways that it would be difficult to
provide the guarantees we provide today (e.g. at-least once delivery). It
messes with some core concepts in Connect, such as having unique (partition,
offset) values for each message. Complex processing topologies like this are
better suited to Kafka Streams, which was built just for that purpose.
Note that you *can* do this without SMTs if you write your own connector that
can safely handle each message resulting in multiple writes to some output
system. You could still use the transformations internally in that connector if
you wanted to reuse the existing transformation code, they just wouldn't be
configured via the standard mechanism in Connect.
As for the specific use case of the original message, if someone made a strong
case for it and a KIP describing how it would be implemented, sure. We're open
to plenty of ideas about how to improve the framework.
> Implement remaining KIP-66 SMTs
> -------------------------------
>
> Key: KAFKA-4714
> URL: https://issues.apache.org/jira/browse/KAFKA-4714
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Reporter: Ewen Cheslack-Postava
> Assignee: Ewen Cheslack-Postava
> Fix For: 0.11.0.0
>
>
> Three didn't make it for the 0.10.2.0 release: Flatten, Cast, and
> TimestampConverter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
