[jira] [Commented] (KAFKA-4714) Implement remaining KIP-66 SMTs
[ https://issues.apache.org/jira/browse/KAFKA-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173506#comment-16173506 ] Ewen Cheslack-Postava commented on KAFKA-4714: -- [~dhananjaydp] You cannot do that with SMTs, you would need to define two connectors as you describe. The KIP probably could have included some more information about why. Your use case is really a special case of the more general problem of a single record generating multiple outputs, with some of those outputs possibly transformed in some way. Supporting this would significantly complicate the system, in ways that it would be difficult to provide the guarantees we provide today (e.g. at-least once delivery). It messes with some core concepts in Connect, such as having unique (partition, offset) values for each message. Complex processing topologies like this are better suited to Kafka Streams, which was built just for that purpose. Note that you *can* do this without SMTs if you write your own connector that can safely handle each message resulting in multiple writes to some output system. You could still use the transformations internally in that connector if you wanted to reuse the existing transformation code, they just wouldn't be configured via the standard mechanism in Connect. As for the specific use case of the original message, if someone made a strong case for it and a KIP describing how it would be implemented, sure. We're open to plenty of ideas about how to improve the framework. > Implement remaining KIP-66 SMTs > --- > > Key: KAFKA-4714 > URL: https://issues.apache.org/jira/browse/KAFKA-4714 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.11.0.0 > > > Three didn't make it for the 0.10.2.0 release: Flatten, Cast, and > TimestampConverter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-4714) Implement remaining KIP-66 SMTs
[ https://issues.apache.org/jira/browse/KAFKA-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16171189#comment-16171189 ] Dhananjay Patkar commented on KAFKA-4714: - Thanks [~ewencp] for quick reply. I have a use case, wherein I need to map incoming raw message to standard format and persist, as well as persist raw message as is. Converting from raw message to standard format message is not always guaranteed, but I still need to persist raw message. I understand I can write multiple consumer to same topic, 1 consumer will persist raw messages as is and other will transform raw message into standard format and persist. Just wanted to know, is there way I can do it as a single consumer through SMT. ??The record is maintained internally, but it is not exposed to the connector?? Is there a possibility to expose raw message based on configuration during SMT? > Implement remaining KIP-66 SMTs > --- > > Key: KAFKA-4714 > URL: https://issues.apache.org/jira/browse/KAFKA-4714 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.11.0.0 > > > Three didn't make it for the 0.10.2.0 release: Flatten, Cast, and > TimestampConverter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-4714) Implement remaining KIP-66 SMTs
[ https://issues.apache.org/jira/browse/KAFKA-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16170244#comment-16170244 ] Ewen Cheslack-Postava commented on KAFKA-4714: -- [~dhananjaydp] The record is maintained internally, but it is not exposed to the connector. It's somewhat counter to the point of SMTs since in some cases you may be trying to do things like remove PII, in which case you definitely don't want to give the connector task a chance to see that data. Regarding support for a transformation after SinkTask.put(), that doesn't really make sense since the whole point of put() is that it passes the data to the task to be written to the external system -- not only is the message in a different system, it's likely been converted to some native format for that system and is no longer in Connect's data API format. Is there a reason you want the original value in the task? Can you describe your use case? > Implement remaining KIP-66 SMTs > --- > > Key: KAFKA-4714 > URL: https://issues.apache.org/jira/browse/KAFKA-4714 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.11.0.0 > > > Three didn't make it for the 0.10.2.0 release: Flatten, Cast, and > TimestampConverter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-4714) Implement remaining KIP-66 SMTs
[ https://issues.apache.org/jira/browse/KAFKA-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169907#comment-16169907 ] Dhananjay Patkar commented on KAFKA-4714: - Currently on the sink side, transformations are applied on the consumed record. ??For sink connectors, transformations are applied on the collection of SinkRecord before being provided to SinkTask.put().?? Is it possible to get original ConnectRecord object in consumer task? If this is not available then ,can we preserve original message or is it possible to add support for post "SinkTask.put()" transformation? > Implement remaining KIP-66 SMTs > --- > > Key: KAFKA-4714 > URL: https://issues.apache.org/jira/browse/KAFKA-4714 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.11.0.0 > > > Three didn't make it for the 0.10.2.0 release: Flatten, Cast, and > TimestampConverter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
