[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743641#comment-15743641
 ] 

Nisarg Shah commented on KAFKA-3209:


Yeah sorry about that, I got caught up with school work. You would have a 
better depth of understanding of this problem as I haven't used this part in a 
professional environment. I would gladly collaborate with you. :) 

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>Assignee: Shikhar Bhushan
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743278#comment-15743278
 ] 

Nisarg Shah commented on KAFKA-3209:


[~shikhar] the proposal looks details and flexible in terms of what it can do. 
I see you've assigned this to yourself, would you also like to update KIP-66?

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>Assignee: Shikhar Bhushan
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674120#comment-15674120
 ] 

Nisarg Shah commented on KAFKA-3209:


Hey Juan. I have started working on this, have only been busy lately and will 
start again within a few weeks. You could help though. Have a look at the KIP 
(link above) and let's discuss suggestions and implementations. 

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>  Labels: needs-kip
> Fix For: 0.10.2.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

[Nisarg Shah]

Gwen, 

Yup, that sounds great! Instead of keeping it up to the transformers to handle 
null, we can instead have the topic as null. Sounds good. To get rid of a 
message, set the topic to a special one (could be as simple as null). 

Like I said before, the more interesting part would be ‘adding’ a new message 
to the existing list, based on say the current message in the transformer. Does 
that feature warrant to be included? 

> On Jul 14, 2016, at 22:25, Gwen Shapira  wrote:
> 
> I used to work on Apache Flume, where we used to allow users to filter
> messages completely in the transformation and then we got rid of it,
> because we spent too much time trying to help users who had "message
> loss", where the loss was actually a bug in the filter...
> 
> What we couldn't do in Flume, but perhaps can do in the simple
> transform for Connect is the ability to route messages to different
> topics, with "null" as one of the possible targets. This will allow
> you to implement a dead-letter-queue functionality and redirect
> messages that don't pass filter to an "errors" topic without getting
> rid of them completely, while also allowing braver users to get rid of
> messages by directing them to "null".
> 
> Does that make sense?
> 
> Gwen
> 
> On Thu, Jul 14, 2016 at 8:33 PM, Nisarg Shah  wrote:
>> Thank you for your inputs Gwen and Michael.
>> 
>> The original reason why I suggested a set based processing is because of the 
>> flexibility is provides. The JIRA had a comment by a user requesting a 
>> feature that could be achieved with this.
>> 
>> After reading Gwen and Michael's points, (I went through the documentation 
>> and the code in detail) and agree with what you have to say. Also, fewer 
>> guarantees make what I had in mind less certain and thus simplifying it to a 
>> single message based transformation would ensure that users who do require 
>> more flexibility with the transformations will automatically “turn to" Kafka 
>> Streams. The transformation logic on a message by message basis makes more 
>> sense.
>> 
>> One usecase that Kafka Connect could consider is adding or removing a 
>> message completely. (This was trivially possible with the collection 
>> passing). Should users be pointed towards Kafka Streams even for this use 
>> case? I think this is a very useful feature for Connect too, and I’ll try to 
>> rethink on the API too.
>> 
>> Removing a message is as easy as returning a null and having the next 
>> transformer skip it, but adding messages would involve say a queue between 
>> transformers and a method which says “pass message” to the next, which can 
>> be called multiple times from one “transform” function; a variation on the 
>> chain of responsibility design pattern.
>> 
>>> On Jul 12, 2016, at 12:54 AM, Michael Noll  wrote:
>>> 
>>> As Gwen said, my initial thought is that message transformations that are
>>> "more than trivial" should rather be done by Kafka Streams, rather than by
>>> Kafka Connect (for the reasons that Gwen mentioned).
>>> 
>>> Transforming one message at a time would be a good fit for Kafka Connect.
>>> An important use case is to remove sensitive data (such as PII) from an
>>> incoming data stream before it hits Kafka's persistent storage -- this use
>>> case can't be implemented well with Kafka Streams because, by design, Kafka
>>> Streams is meant to read its input data from Kafka (i.e. at the point when
>>> Kafka Streams could be used to removed sensitive data fields the data is
>>> already stored persistently in Kafka, and this might be a no-go depending
>>> on the use case).
>>> 
>>> I'm of course interested to hear what other people think.
>>> 
>>> 
>>> On Tue, Jul 12, 2016 at 6:06 AM, Gwen Shapira  wrote:
>>> 
>>>> I think we need to restrict the functionality to one-message-at-a-time.
>>>> 
>>>> Basically, connect gives very little guarantees about the size of the set
>>>> of the composition (you may get same messages over and over, mix of old and
>>>> new, etc)
>>>> 
>>>> In order to do useful things over a collection, you need better defined
>>>> semantics of what's included. Kafka Streams is putting tons of effort into
>>>> having good windowing semantics, and I think apps that require modification
>>>> of collections are a better fit there.
>>>> 
>>>> I'm willing to change my mind though (have been known to happe

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

[Nisarg Shah]

Thank you for your inputs Gwen and Michael.

The original reason why I suggested a set based processing is because of the 
flexibility is provides. The JIRA had a comment by a user requesting a feature 
that could be achieved with this.

After reading Gwen and Michael's points, (I went through the documentation and 
the code in detail) and agree with what you have to say. Also, fewer guarantees 
make what I had in mind less certain and thus simplifying it to a single 
message based transformation would ensure that users who do require more 
flexibility with the transformations will automatically “turn to" Kafka 
Streams. The transformation logic on a message by message basis makes more 
sense.

One usecase that Kafka Connect could consider is adding or removing a message 
completely. (This was trivially possible with the collection passing). Should 
users be pointed towards Kafka Streams even for this use case? I think this is 
a very useful feature for Connect too, and I’ll try to rethink on the API too.

Removing a message is as easy as returning a null and having the next 
transformer skip it, but adding messages would involve say a queue between 
transformers and a method which says “pass message” to the next, which can be 
called multiple times from one “transform” function; a variation on the chain 
of responsibility design pattern.

> On Jul 12, 2016, at 12:54 AM, Michael Noll  wrote:
> 
> As Gwen said, my initial thought is that message transformations that are
> "more than trivial" should rather be done by Kafka Streams, rather than by
> Kafka Connect (for the reasons that Gwen mentioned).
> 
> Transforming one message at a time would be a good fit for Kafka Connect.
> An important use case is to remove sensitive data (such as PII) from an
> incoming data stream before it hits Kafka's persistent storage -- this use
> case can't be implemented well with Kafka Streams because, by design, Kafka
> Streams is meant to read its input data from Kafka (i.e. at the point when
> Kafka Streams could be used to removed sensitive data fields the data is
> already stored persistently in Kafka, and this might be a no-go depending
> on the use case).
> 
> I'm of course interested to hear what other people think.
> 
> 
> On Tue, Jul 12, 2016 at 6:06 AM, Gwen Shapira  wrote:
> 
>> I think we need to restrict the functionality to one-message-at-a-time.
>> 
>> Basically, connect gives very little guarantees about the size of the set
>> of the composition (you may get same messages over and over, mix of old and
>> new, etc)
>> 
>> In order to do useful things over a collection, you need better defined
>> semantics of what's included. Kafka Streams is putting tons of effort into
>> having good windowing semantics, and I think apps that require modification
>> of collections are a better fit there.
>> 
>> I'm willing to change my mind though (have been known to happen) - what are
>> the comments about usage that point toward the collections approach?
>> 
>> Gwen
>> 
>> On Mon, Jul 11, 2016 at 3:32 PM, Nisarg Shah  wrote:
>> 
>>> Thanks Jay, added that to the KIP.
>>> 
>>> Besides reviewing the KIP as a whole, I wanted to know about what
>> everyone
>>> thinks about what data should be dealt at the Transformer level.
>> Transform
>>> the whole Collection of Records (giving the flexibility of modifying
>>> messages across the set) OR
>>> Transform messages one at a time, iteratively. This will restrict
>>> modifications across messages.
>>> 
>>> I’ll get a working sample ready soon, to have a look. There were some
>>> comments about Transformer usage that pointed to the first approach,
>> which
>>> I prefer too given the flexibility.
>>> 
>>>> On Jul 11, 2016, at 2:49 PM, Jay Kreps  wrote:
>>>> 
>>>> One minor thing, the Transformer interface probably needs a close()
>>> method
>>>> (i.e. the opposite of initialize). This would be used for any
>> transformer
>>>> that uses a resource like a file/socket/db connection/etc that needs to
>>> be
>>>> closed. You usually don't need this but when you do need it you really
>>> need
>>>> it.
>>>> 
>>>> -Jay
>>>> 
>>>> On Mon, Jul 11, 2016 at 1:47 PM, Nisarg Shah 
>> wrote:
>>>> 
>>>>> Hello,
>>>>> 
>>>>> This KIP <
>>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-66:+Add+Kafka+Connect+Transformers+to+allow+transformations+to+messages
>>>> 
>>>>> is for KAFKA-3209 <https://issues.apache.org/jira/browse/KAFKA-3209>.
>>>>> It’s about capabilities to transform messages in Kafka Connect.
>>>>> 
>>>>> Some design decisions need to be taken, so please advise me on the
>> same.
>>>>> Feel free to express any thoughts or concerns as well.
>>>>> 
>>>>> Many many thanks to Ewen Cheslack-Postava.
>>>>> 
>>>>> -Nisarg
>>> 
>>> 
>> 
> 
> 
> 
> -- 
> Best regards,
> Michael Noll
> 
> 
> 
> *Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
> Apache Kafka and Confluent Platform: www.confluent.io/download
> <http://www.confluent.io/download>*

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

[Nisarg Shah]

Thanks Jay, added that to the KIP. 

Besides reviewing the KIP as a whole, I wanted to know about what everyone 
thinks about what data should be dealt at the Transformer level. Transform the 
whole Collection of Records (giving the flexibility of modifying messages 
across the set) OR
Transform messages one at a time, iteratively. This will restrict modifications 
across messages.

I’ll get a working sample ready soon, to have a look. There were some comments 
about Transformer usage that pointed to the first approach, which I prefer too 
given the flexibility. 

> On Jul 11, 2016, at 2:49 PM, Jay Kreps  wrote:
> 
> One minor thing, the Transformer interface probably needs a close() method
> (i.e. the opposite of initialize). This would be used for any transformer
> that uses a resource like a file/socket/db connection/etc that needs to be
> closed. You usually don't need this but when you do need it you really need
> it.
> 
> -Jay
> 
> On Mon, Jul 11, 2016 at 1:47 PM, Nisarg Shah  wrote:
> 
>> Hello,
>> 
>> This KIP <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-66:+Add+Kafka+Connect+Transformers+to+allow+transformations+to+messages>
>> is for KAFKA-3209 <https://issues.apache.org/jira/browse/KAFKA-3209>.
>> It’s about capabilities to transform messages in Kafka Connect.
>> 
>> Some design decisions need to be taken, so please advise me on the same.
>> Feel free to express any thoughts or concerns as well.
>> 
>> Many many thanks to Ewen Cheslack-Postava.
>> 
>> -Nisarg

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371610#comment-15371610
 ] 

Nisarg Shah commented on KAFKA-3209:


Please give your comments and suggestions here
https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Add+Kafka+Connect+Transformers+to+allow+transformations+to+messages


> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>Priority: Critical
>  Labels: needs-kip
> Fix For: 0.10.1.0
>
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSS] KIP-66 Kafka Connect Transformers for messages

[Nisarg Shah]

Hello,

This KIP 

 is for KAFKA-3209 . It’s 
about capabilities to transform messages in Kafka Connect.

Some design decisions need to be taken, so please advise me on the same. Feel 
free to express any thoughts or concerns as well.

Many many thanks to Ewen Cheslack-Postava.

-Nisarg

Re: Kafka Connect Transformers

[Nisarg Shah]

Need to submit a KIP for https://issues.apache.org/jira/browse/KAFKA-3209. 
Please provide wiki write access to ‘snisarg’. 

Thanks,
Nisarg Shah.

> On Jun 28, 2016, at 6:27 PM, Nisarg Shah  wrote:
> 
> Need permissions to edit the wiki. Username is ‘snisarg’. 
> 
> Thanks,
> Nisarg.
> 
>> On Jun 28, 2016, at 09:08, Nisarg Shah > <mailto:snis...@gmail.com>> wrote:
>> 
>> Hello,
>> 
>> I need to create a page so that I can write a Kafka Improvement Proposal for 
>> the below. My username is ‘snisarg’. 
>> 
>> Thanks,
>> Nisarg
>> 
>>> On Jun 19, 2016, at 10:43 PM, Nisarg Shah >> <mailto:snis...@gmail.com>> wrote:
>>> 
>>> Hello,
>>> 
>>> I am looking to do https://issues.apache.org/jira/browse/KAFKA-3209 
>>> <https://issues.apache.org/jira/browse/KAFKA-3209>. I wanted feedback from 
>>> the devs for the design that I’m proposing to put in place. Thanks a lot 
>>> for all the discussions Ewen Cheslack-Postava.
>>> 
>>> A gist of how I plan to do it is by using ‘Transformers’ that can be 
>>> configurationally chained together and data will pass through them between 
>>> a source and destination for Kafka Connect.
>>> 
>>> To set up transformers, we propose using the properties to define 
>>> Transformer classes one after the other. 
>>> transformer=abc.Transformer1,xyz.Transformer2
>>> 
>>> Each transformer can get specific properties passed on from the same 
>>> properties file, as it is with the Connectors.
>>> 
>>> About the actual signature for the transformation function that does all 
>>> the work, how’s this interface? 
>>> public abstract class Transformer {
>>> public abstract T2 transform(T1 t1);
>>> 
>>> public void initialize(Map props) {}
>>> }
>>> 
>>> Approach 1:
>>> Functionally, the complete data can be passed. 
>>> Just as the *Tasks get a complete List<*Record>, the transformer can get 
>>> the same. The whole list passing makes rearranging or merging data 
>>> possible. This can be helpful if transformations require looking up or down 
>>> the messages. Allowing custom datatypes between transformers will allow 
>>> custom objects to be passed around intermediate. Casting could be an issue.
>>> 
>>> Approach 2: 
>>> Taking a simplistic approach and doing a message by message transformation. 
>>> The transformer could store data from the previous message, but not go down 
>>> the list of messages. From the comments by Michael Graff, both approaches 
>>> would work, but if down looking is required, we would have to go with 
>>> Approach 1. 
>>> 
>>> I will also have a working change ready for Approach 1 very soon but till 
>>> then, please give me your suggestions. 
>>> 
>>> Thanks,
>>> Nisarg.
>>> 
>>> 
>>> 
>>> 
>> 
> 

Re: Kafka Connect Transformers

[Nisarg Shah]

Need permissions to edit the wiki. Username is ‘snisarg’. 

Thanks,
Nisarg.

> On Jun 28, 2016, at 09:08, Nisarg Shah  wrote:
> 
> Hello,
> 
> I need to create a page so that I can write a Kafka Improvement Proposal for 
> the below. My username is ‘snisarg’. 
> 
> Thanks,
> Nisarg
> 
>> On Jun 19, 2016, at 10:43 PM, Nisarg Shah > <mailto:snis...@gmail.com>> wrote:
>> 
>> Hello,
>> 
>> I am looking to do https://issues.apache.org/jira/browse/KAFKA-3209 
>> <https://issues.apache.org/jira/browse/KAFKA-3209>. I wanted feedback from 
>> the devs for the design that I’m proposing to put in place. Thanks a lot for 
>> all the discussions Ewen Cheslack-Postava.
>> 
>> A gist of how I plan to do it is by using ‘Transformers’ that can be 
>> configurationally chained together and data will pass through them between a 
>> source and destination for Kafka Connect.
>> 
>> To set up transformers, we propose using the properties to define 
>> Transformer classes one after the other. 
>> transformer=abc.Transformer1,xyz.Transformer2
>> 
>> Each transformer can get specific properties passed on from the same 
>> properties file, as it is with the Connectors.
>> 
>> About the actual signature for the transformation function that does all the 
>> work, how’s this interface? 
>> public abstract class Transformer {
>> public abstract T2 transform(T1 t1);
>> 
>> public void initialize(Map props) {}
>> }
>> 
>> Approach 1:
>> Functionally, the complete data can be passed. 
>> Just as the *Tasks get a complete List<*Record>, the transformer can get the 
>> same. The whole list passing makes rearranging or merging data possible. 
>> This can be helpful if transformations require looking up or down the 
>> messages. Allowing custom datatypes between transformers will allow custom 
>> objects to be passed around intermediate. Casting could be an issue.
>> 
>> Approach 2: 
>> Taking a simplistic approach and doing a message by message transformation. 
>> The transformer could store data from the previous message, but not go down 
>> the list of messages. From the comments by Michael Graff, both approaches 
>> would work, but if down looking is required, we would have to go with 
>> Approach 1. 
>> 
>> I will also have a working change ready for Approach 1 very soon but till 
>> then, please give me your suggestions. 
>> 
>> Thanks,
>> Nisarg.
>> 
>> 
>> 
>> 
> 

Re: Kafka Connect Transformers

[Nisarg Shah]

Hello,

I need to create a page so that I can write a Kafka Improvement Proposal for 
the below. My username is ‘snisarg’. 

Thanks,
Nisarg

> On Jun 19, 2016, at 10:43 PM, Nisarg Shah  wrote:
> 
> Hello,
> 
> I am looking to do https://issues.apache.org/jira/browse/KAFKA-3209 
> <https://issues.apache.org/jira/browse/KAFKA-3209>. I wanted feedback from 
> the devs for the design that I’m proposing to put in place. Thanks a lot for 
> all the discussions Ewen Cheslack-Postava.
> 
> A gist of how I plan to do it is by using ‘Transformers’ that can be 
> configurationally chained together and data will pass through them between a 
> source and destination for Kafka Connect.
> 
> To set up transformers, we propose using the properties to define Transformer 
> classes one after the other. 
> transformer=abc.Transformer1,xyz.Transformer2
> 
> Each transformer can get specific properties passed on from the same 
> properties file, as it is with the Connectors.
> 
> About the actual signature for the transformation function that does all the 
> work, how’s this interface? 
> public abstract class Transformer {
> public abstract T2 transform(T1 t1);
> 
> public void initialize(Map props) {}
> }
> 
> Approach 1:
> Functionally, the complete data can be passed. 
> Just as the *Tasks get a complete List<*Record>, the transformer can get the 
> same. The whole list passing makes rearranging or merging data possible. This 
> can be helpful if transformations require looking up or down the messages. 
> Allowing custom datatypes between transformers will allow custom objects to 
> be passed around intermediate. Casting could be an issue.
> 
> Approach 2: 
> Taking a simplistic approach and doing a message by message transformation. 
> The transformer could store data from the previous message, but not go down 
> the list of messages. From the comments by Michael Graff, both approaches 
> would work, but if down looking is required, we would have to go with 
> Approach 1. 
> 
> I will also have a working change ready for Approach 1 very soon but till 
> then, please give me your suggestions. 
> 
> Thanks,
> Nisarg.
> 
> 
> 
> 

Kafka Connect Transformers

[Nisarg Shah]

Hello,

I am looking to do https://issues.apache.org/jira/browse/KAFKA-3209. I wanted 
feedback from the devs for the design that I’m proposing to put in place. 
Thanks a lot for all the discussions Ewen Cheslack-Postava.

A gist of how I plan to do it is by using ‘Transformers’ that can be 
configurationally chained together and data will pass through them between a 
source and destination for Kafka Connect.

To set up transformers, we propose using the properties to define Transformer 
classes one after the other. 
transformer=abc.Transformer1,xyz.Transformer2

Each transformer can get specific properties passed on from the same properties 
file, as it is with the Connectors.

About the actual signature for the transformation function that does all the 
work, how’s this interface? 
public abstract class Transformer {
public abstract T2 transform(T1 t1);

public void initialize(Map props) {}
}

Approach 1:
Functionally, the complete data can be passed. 
Just as the *Tasks get a complete List<*Record>, the transformer can get the 
same. The whole list passing makes rearranging or merging data possible. This 
can be helpful if transformations require looking up or down the messages. 
Allowing custom datatypes between transformers will allow custom objects to be 
passed around intermediate. Casting could be an issue.

Approach 2: 
Taking a simplistic approach and doing a message by message transformation. The 
transformer could store data from the previous message, but not go down the 
list of messages. From the comments by Michael Graff, both approaches would 
work, but if down looking is required, we would have to go with Approach 1. 

I will also have a working change ready for Approach 1 very soon but till then, 
please give me your suggestions. 

Thanks,
Nisarg.

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329669#comment-15329669
 ] 

Nisarg Shah commented on KAFKA-3209:


I'm still working on it. Got a little busy. I thought I'll get a working 
prototype ready so that it can be demonstrated and it's easier to convey how 
it'll work. 
I have it almost ready, working out small kinks of sending properties down the 
transformers behind the scenes. I will be sending out mails to the dev channel 
soon. 

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264644#comment-15264644
 ] 

Nisarg Shah commented on KAFKA-3209:


That does sound good. I realise it is not insanely complicated, but since I 
haven't done much in terms of open source, I thought I'll start with something 
like this. I also think this is useful nonetheless. 

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

[Nisarg Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264548#comment-15264548
 ] 

Nisarg Shah commented on KAFKA-3209:


Hey, I'm looking to contribute to open source and want to take this up. :) 

> Support single message transforms in Kafka Connect
> --
>
> Key: KAFKA-3209
> URL: https://issues.apache.org/jira/browse/KAFKA-3209
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Neha Narkhede
>
> Users should be able to perform light transformations on messages between a 
> connector and Kafka. This is needed because some transformations must be 
> performed before the data hits Kafka (e.g. filtering certain types of events 
> or PII filtering). It's also useful for very light, single-message 
> modifications that are easier to perform inline with the data import/export.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)