It should be possible using a custom Single Message Transform ( https://docs.confluent.io/current/connect/javadocs/org/apache/kafka/connect/transforms/Transformation.html). There is a good talk here about when SMTs are appropriate (and when not): https://kafka-summit.org/sessions/single-message-transformations-not-transformations-youre-looking/
You might also want to look at Kafka Streams for applying logic to a stream before you ingest it with Kafka Connect. -- Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff On Fri, 25 Jan 2019 at 13:29, Sönke Liebau <soenke.lie...@opencore.com.invalid> wrote: > Hi Martin, > > I don't think that Connect offers this functionality as is. However > you can implement custom SMTs, which should allow you to do what need. > Alternatively you could write a ConsumerInterceptor that filters out > messages based on your criteria - but: this would affect all running > Connectors as this setting can only be set for the entire Connect > worker. > Depending on your exact needs one of the two approaches might be > preferable. If this filter has to be applied to everything that > Connect reads than the Interceptor might be preferrable, as you'll > only have to define it once and then not specify it again for every > connector. > > Hope that helps. > > Best regards, > Sönke > > On Fri, Jan 25, 2019 at 1:58 PM mbschorer <bscho...@tyntec.com> wrote: > > > > Hi! > > > > I am evaluating the suitability of Kafka Connect to leverage our > internal event sourcing system. > > Since custom deserializers are supported, it looks quite promising so > far. > > > > One open question is, whether Kafka Connect is capable of filtering out, > i.e. skipping certain messages based on conditions on the value. > > Or does it pursue a take all or nothing approach? > > > > All I could find so far is single message transform, which didn't reveal > this feature. > > Does anybody in this mailing list have more knowledge regarding this? > > > > Thanks in advance! > > Martin > > > > > -- > Sönke Liebau > Partner > Tel. +49 179 7940878 > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany >
