Hi all, Just came across this thread, I hope the late reply is ok.
FWIW, we're in a similar situation in Debezium, where users often request new (Debezium-specific) SMTs, and we generally tend to recommend them to be maintained by users themselves, unless they are truly generic. This excludes a share of users though who aren't Java developers. What might help is having means of simple discoverability of externally hosted SMTs, e.g. via some kind of catalog hosted on kafka.apache.org. That way, people would have it easier to find and obtain SMTs from other places, reducing the pressure to get them added to Apache Kafka proper. Best, --Gunnar Am So., 7. Nov. 2021 um 21:49 Uhr schrieb Brandon Brown < bran...@bbrownsound.com>: > I like the idea of a select number of SMTs being offered and supported out > of the box. The addition of SMTs via this process is nice because it allows > for a rich set to be supported out of the box and without the need for > extra work to deploy. > > Perhaps this is a spot where the community could express the interest of > additional SMTs which maybe are available via an open source library and if > enough usage occurs there could be a path to fold into the Kafka project at > large? > > Brandon Brown > > > > On Nov 7, 2021, at 1:19 PM, Randall Hauch <rha...@gmail.com> wrote: > > > > We have had several requests to add more Connect Single Message > > Transforms (SMTs) to the project. When SMTs were first introduced with > > KIP-66 (ref 1) in Jun 2017, the KIP mentioned the following: > > > >> Criteria: SMTs that are shipped with Kafka Connect should be general > enough to apply to many data sources & serialization formats. They should > also be simple enough to not cause any additional library dependency to be > introduced. > >> Beyond those being initially included with this KIP, transformations > can be adopted for inclusion in future with JIRA/ML discussion to weigh the > tradeoffs. > > > > In the 4+ years that we've had SMTs in the project, we've only > > enhanced the framework with KIP-585 (ref 2), and fixed the initial > > SMTs (including KIP-437, ref 3). We recently have had quite a few > > requests to add new SMTs; a few samples of these include: > > * https://issues.apache.org/jira/browse/KAFKA-10299 > > * https://issues.apache.org/jira/browse/KAFKA-9436 > > * https://issues.apache.org/jira/browse/KAFKA-9318 > > * https://issues.apache.org/jira/browse/KAFKA-12443 > > > > Adding new or changing existing SMTs to the Apache Kafka project come > > with requirements. First, AK releases are infrequent and necessarily > > involve the entire project. Second, adding an SMT is an API change and > > therefore requires a KIP. Third, all changes in behavior to SMTs > > included in an prior AK release must be backward compatible, and > > adding or changing an SMT's configuration requires a KIP. This last > > one is also challenging if we're limiting ourselves to truly general > > SMTs, since these are notoriously difficult to get right the first > > time. All of these aspects mean that it's difficult to add, maintain, > > and evolve/improve SMTs in AK. And unless a bug fix is critical, we're > > likely not to create a patch release for AK just to fix a bug in an > > SMT, simply because of the effort involved. > > > > On the other hand, anyone can easily implement their own SMT and > > deploy them as a Connect plugin, whether that's part of a connector > > plugin or a separate plugin dedicated for one or SMTs. Interestingly, > > it's far simpler to implement and maintain custom SMTs outside of AK, > > especially since those plugins can be released and deployed in any > > Connect runtime version since at least 0.11.0. And if custom SMTs are > > maintained in a relatively small project, they can be released often. > > > > Finally, KIP-26 (ref 4) specifically rejected maintaining connector > > implementations in the AK project. So we have precedence for choosing > > not to accept implementations. > > > > Given the above, I wonder if the time has come for us to prefer only > > maintaining the SMT framework and existing SMTs, and to decline adding > > new SMTs. > > > > Thoughts? > > > > Best regards, > > > > Randall Hauch > > > > (1) > https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect > > (2) > https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Filter+and+Conditional+SMTs > > (3) > https://cwiki.apache.org/confluence/display/KAFKA/KIP-437%3A+Custom+replacement+for+MaskField+SMT > > (4) > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767 >
