Need permissions to edit the wiki. Username is ‘snisarg’. Thanks, Nisarg.
> On Jun 28, 2016, at 09:08, Nisarg Shah <snis...@gmail.com> wrote: > > Hello, > > I need to create a page so that I can write a Kafka Improvement Proposal for > the below. My username is ‘snisarg’. > > Thanks, > Nisarg > >> On Jun 19, 2016, at 10:43 PM, Nisarg Shah <snis...@gmail.com >> <mailto:snis...@gmail.com>> wrote: >> >> Hello, >> >> I am looking to do https://issues.apache.org/jira/browse/KAFKA-3209 >> <https://issues.apache.org/jira/browse/KAFKA-3209>. I wanted feedback from >> the devs for the design that I’m proposing to put in place. Thanks a lot for >> all the discussions Ewen Cheslack-Postava. >> >> A gist of how I plan to do it is by using ‘Transformers’ that can be >> configurationally chained together and data will pass through them between a >> source and destination for Kafka Connect. >> >> To set up transformers, we propose using the properties to define >> Transformer classes one after the other. >> transformer=abc.Transformer1,xyz.Transformer2 >> >> Each transformer can get specific properties passed on from the same >> properties file, as it is with the Connectors. >> >> About the actual signature for the transformation function that does all the >> work, how’s this interface? >> public abstract class Transformer<T1, T2> { >> public abstract T2 transform(T1 t1); >> >> public void initialize(Map<String, String> props) {} >> } >> >> Approach 1: >> Functionally, the complete data can be passed. >> Just as the *Tasks get a complete List<*Record>, the transformer can get the >> same. The whole list passing makes rearranging or merging data possible. >> This can be helpful if transformations require looking up or down the >> messages. Allowing custom datatypes between transformers will allow custom >> objects to be passed around intermediate. Casting could be an issue. >> >> Approach 2: >> Taking a simplistic approach and doing a message by message transformation. >> The transformer could store data from the previous message, but not go down >> the list of messages. From the comments by Michael Graff, both approaches >> would work, but if down looking is required, we would have to go with >> Approach 1. >> >> I will also have a working change ready for Approach 1 very soon but till >> then, please give me your suggestions. >> >> Thanks, >> Nisarg. >> >> >> >> >
