I agree, if the desire is to keep the internal SMTs collection small then providing an ease of discovery like Gunnar suggestions would be extremely helpful.
Brandon Brown > On Nov 19, 2021, at 6:13 PM, Gunnar Morling > <gunnar.morl...@googlemail.com.invalid> wrote: > > Hi all, > > Just came across this thread, I hope the late reply is ok. > > FWIW, we're in a similar situation in Debezium, where users often request > new (Debezium-specific) SMTs, and we generally tend to recommend them to be > maintained by users themselves, unless they are truly generic. This > excludes a share of users though who aren't Java developers. > > What might help is having means of simple discoverability of externally > hosted SMTs, e.g. via some kind of catalog hosted on kafka.apache.org. That > way, people would have it easier to find and obtain SMTs from other places, > reducing the pressure to get them added to Apache Kafka proper. > > Best, > > --Gunnar > > > > >> Am So., 7. Nov. 2021 um 21:49 Uhr schrieb Brandon Brown < >> bran...@bbrownsound.com>: >> >> I like the idea of a select number of SMTs being offered and supported out >> of the box. The addition of SMTs via this process is nice because it allows >> for a rich set to be supported out of the box and without the need for >> extra work to deploy. >> >> Perhaps this is a spot where the community could express the interest of >> additional SMTs which maybe are available via an open source library and if >> enough usage occurs there could be a path to fold into the Kafka project at >> large? >> >> Brandon Brown >> >> >>>> On Nov 7, 2021, at 1:19 PM, Randall Hauch <rha...@gmail.com> wrote: >>> >>> We have had several requests to add more Connect Single Message >>> Transforms (SMTs) to the project. When SMTs were first introduced with >>> KIP-66 (ref 1) in Jun 2017, the KIP mentioned the following: >>> >>>> Criteria: SMTs that are shipped with Kafka Connect should be general >> enough to apply to many data sources & serialization formats. They should >> also be simple enough to not cause any additional library dependency to be >> introduced. >>>> Beyond those being initially included with this KIP, transformations >> can be adopted for inclusion in future with JIRA/ML discussion to weigh the >> tradeoffs. >>> >>> In the 4+ years that we've had SMTs in the project, we've only >>> enhanced the framework with KIP-585 (ref 2), and fixed the initial >>> SMTs (including KIP-437, ref 3). We recently have had quite a few >>> requests to add new SMTs; a few samples of these include: >>> * https://issues.apache.org/jira/browse/KAFKA-10299 >>> * https://issues.apache.org/jira/browse/KAFKA-9436 >>> * https://issues.apache.org/jira/browse/KAFKA-9318 >>> * https://issues.apache.org/jira/browse/KAFKA-12443 >>> >>> Adding new or changing existing SMTs to the Apache Kafka project come >>> with requirements. First, AK releases are infrequent and necessarily >>> involve the entire project. Second, adding an SMT is an API change and >>> therefore requires a KIP. Third, all changes in behavior to SMTs >>> included in an prior AK release must be backward compatible, and >>> adding or changing an SMT's configuration requires a KIP. This last >>> one is also challenging if we're limiting ourselves to truly general >>> SMTs, since these are notoriously difficult to get right the first >>> time. All of these aspects mean that it's difficult to add, maintain, >>> and evolve/improve SMTs in AK. And unless a bug fix is critical, we're >>> likely not to create a patch release for AK just to fix a bug in an >>> SMT, simply because of the effort involved. >>> >>> On the other hand, anyone can easily implement their own SMT and >>> deploy them as a Connect plugin, whether that's part of a connector >>> plugin or a separate plugin dedicated for one or SMTs. Interestingly, >>> it's far simpler to implement and maintain custom SMTs outside of AK, >>> especially since those plugins can be released and deployed in any >>> Connect runtime version since at least 0.11.0. And if custom SMTs are >>> maintained in a relatively small project, they can be released often. >>> >>> Finally, KIP-26 (ref 4) specifically rejected maintaining connector >>> implementations in the AK project. So we have precedence for choosing >>> not to accept implementations. >>> >>> Given the above, I wonder if the time has come for us to prefer only >>> maintaining the SMT framework and existing SMTs, and to decline adding >>> new SMTs. >>> >>> Thoughts? >>> >>> Best regards, >>> >>> Randall Hauch >>> >>> (1) >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect >>> (2) >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Filter+and+Conditional+SMTs >>> (3) >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-437%3A+Custom+replacement+for+MaskField+SMT >>> (4) >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767 >>
