Hi Greg, I really like this idea. It has the potential to both soften the learning curve for users coming up to speed with Kafka Connect, and address a pain point for both new and seasoned users experimenting with connector configs. Given that standalone mode is useful for docs and quickstarts in addition to the use cases you describe, I feel these improvements are warranted for inclusion in out-of-the-box Kafka Connect.
I would welcome one or several KIPS for these improvements, depending on the scope and independence of the changes. With regards to the changes proposed, we might also consider allowing standalone workers to accept JSON files (so that users who are rapidly iterating with connector configs that are ultimately destined for deployment onto a distributed cluster won't have to translate between formats), and adding some kind of mechanism for users to await the successful startup of a connector and/or the first offset commit(s) by its task(s) (so that it's clear whether a config "works" or not). Thanks for raising this. Cheers, Chris On Mon, Oct 24, 2022 at 3:09 PM Greg Harris <[email protected]> wrote: > Hey All, > > I'd like to share with you a potential improvement to the workflow for > Connect users that may make connector configurations easier to develop and > debug. The focus for this is on reducing the number of manual steps > involved in seeing the effects of a config change, so that each iteration > is faster and a user can work through errors/misconfigurations faster. > > The details should be found here: > > https://gitlab.gregswebserver.com/gharris1727/kafka-notes/-/wikis/Wishlist/ConfigCycleTime > > While exploring this idea, I implemented a change to the Connect Standalone > runner which watches the provided configuration files for changes, and > applies them to the worker as soon as they are discovered. The prototype > branch can be found here: > > https://github.com/gharris1727/kafka/tree/prototype-standalone-config-watcher > > I wanted to put this out in order to ask a question of the maintainers: > Does this sort of usability improvement belong in apache/kafka, or some > external project? Would a KIP (or several) to implement this functionality > upstream be warranted for further discussion? > > As-is, I believe that it is possible to implement all of this functionality > in an external utility and/or a RestExtension without any (significant) > upstream changes. This proposal would not constitute any strictly new > use-cases, but rather make the Standalone Mode more viable for the stated > use-case, which is for "ad hoc, small, or experimental jobs." > > Thanks in advance for your comments, > Greg Harris >
