I think we need to restrict the functionality to one-message-at-a-time. Basically, connect gives very little guarantees about the size of the set of the composition (you may get same messages over and over, mix of old and new, etc)
In order to do useful things over a collection, you need better defined semantics of what's included. Kafka Streams is putting tons of effort into having good windowing semantics, and I think apps that require modification of collections are a better fit there. I'm willing to change my mind though (have been known to happen) - what are the comments about usage that point toward the collections approach? Gwen On Mon, Jul 11, 2016 at 3:32 PM, Nisarg Shah <snis...@gmail.com> wrote: > Thanks Jay, added that to the KIP. > > Besides reviewing the KIP as a whole, I wanted to know about what everyone > thinks about what data should be dealt at the Transformer level. Transform > the whole Collection of Records (giving the flexibility of modifying > messages across the set) OR > Transform messages one at a time, iteratively. This will restrict > modifications across messages. > > I’ll get a working sample ready soon, to have a look. There were some > comments about Transformer usage that pointed to the first approach, which > I prefer too given the flexibility. > > > On Jul 11, 2016, at 2:49 PM, Jay Kreps <j...@confluent.io> wrote: > > > > One minor thing, the Transformer interface probably needs a close() > method > > (i.e. the opposite of initialize). This would be used for any transformer > > that uses a resource like a file/socket/db connection/etc that needs to > be > > closed. You usually don't need this but when you do need it you really > need > > it. > > > > -Jay > > > > On Mon, Jul 11, 2016 at 1:47 PM, Nisarg Shah <snis...@gmail.com> wrote: > > > >> Hello, > >> > >> This KIP < > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-66:+Add+Kafka+Connect+Transformers+to+allow+transformations+to+messages > > > >> is for KAFKA-3209 <https://issues.apache.org/jira/browse/KAFKA-3209>. > >> It’s about capabilities to transform messages in Kafka Connect. > >> > >> Some design decisions need to be taken, so please advise me on the same. > >> Feel free to express any thoughts or concerns as well. > >> > >> Many many thanks to Ewen Cheslack-Postava. > >> > >> -Nisarg > >
