Hi Zhiqiang, Thanks for your proposal. The idea is very interesting! Since almost all connectors are or will be externalized, the pluggable design you suggested could help reduce the development effort.
As you mentioned, the attached doc contains only your preliminary idea. I would suggest that you could turn it into a FLIP with more details and do the followings: 1. Describe a real conflict case that will benefit from the pluggable design 2. Describe where you want to modify the code, or provide a POC branch 3. Guideline to tell users how to use it, i.e. where (the plugins dir) should the connector jar be located, how does it look like with an example, etc. WDYT? Best regards, Jing On Fri, Jul 14, 2023 at 3:00 PM zhiqiang li <lizhiqiang....@gmail.com> wrote: > Hi devs, > > I have observed that in [1], connectors and formats are pluggable, > allowing user code to be easily integrated. The advantages of having > pluggable connectors are evident, as it helps avoid conflicts between > different versions of jar packages. If classloader isolation is not used, > shading becomes necessary to resolve conflicts, resulting in a significant > waste of development time. I understand that implementing this change may > require numerous API modifications, so I would like to discuss in this > email. > > > Plugins cannot access classes from other plugins or from Flink that have > not been specifically whitelisted. > > This strict isolation allows plugins to contain conflicting versions of > the same library without the need to relocate classes or to converge to > common versions. > > Currently, file systems and metric reporters are pluggable but, in the > future, connectors, formats, and even user code should also be pluggable. > > [2] It is my preliminary idea. > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/ > [2] > https://docs.google.com/document/d/1XP2fBpcntK0YIdQ_Ax7JV2MhNdebvkFxSiNJRp6WQ24/edit?usp=sharing > > > Best, > Zhiqiang > >
