Hi, zhiqiang Thanks for driving this proposal, it looks very interesting.
As we all know, Flink has a very powerful connector ecosystem, and as Jing said, all the connectors are currently being externalized, and it would be very useful to design a pluggable mechanism for connectors, which could solve the potential problem of multiple versions of a connector conflicting without the user having to relocate. In our company, we also have a rich set of connectors internally, and we need to handle them carefully to avoid class loading conflicts. I also agree with Jing saying that we need to turn it into a FLIP. In that case, the whole design will be very complete. Best, Ron Thor God <ptsx...@gmail.com> 于2023年8月10日周四 14:47写道： > I'm interested in this, we often have linker dependency conflicts and it > takes a lot of work to resolve dependency conflicts. > > Benchao Li <libenc...@apache.org> 于2023年7月25日周二 20:51写道： > > > I agree with Jing that the current doc is quite preliminary, and needs to > > be turned into a FLIP. > > > > I'll be very interested in this FLIP, and looking forward to it. We've > > suffered from reshading/relocating heavy connector dependencies for a > long > > time. > > > > Besides, we've implemented a plugin mechanism to load hive udf in our > batch > > SQL scenario internally, it saves us a lot of effort to handle the > > dependency conflicts. The most challenging part would be the > > (de)serialization of those classes loaded through plugins according to > our > > experience. (I noticed you have already considered this) > > > > Jing Ge <j...@ververica.com.invalid> 于2023年7月21日周五 06:44写道： > > > > > Hi Zhiqiang, > > > > > > Thanks for your proposal. The idea is very interesting! Since almost > all > > > connectors are or will be externalized, the pluggable design you > > suggested > > > could help reduce the development effort. > > > > > > As you mentioned, the attached doc contains only your preliminary > idea. I > > > would suggest that you could turn it into a FLIP with more details and > do > > > the followings: > > > > > > 1. Describe a real conflict case that will benefit from the pluggable > > > design > > > 2. Describe where you want to modify the code, or provide a POC branch > > > 3. Guideline to tell users how to use it, i.e. where (the plugins dir) > > > should the connector jar be located, how does it look like with an > > example, > > > etc. > > > > > > WDYT? > > > > > > Best regards, > > > Jing > > > > > > > > > On Fri, Jul 14, 2023 at 3:00 PM zhiqiang li <lizhiqiang....@gmail.com> > > > wrote: > > > > > > > Hi devs, > > > > > > > > I have observed that in [1], connectors and formats are pluggable, > > > > allowing user code to be easily integrated. The advantages of having > > > > pluggable connectors are evident, as it helps avoid conflicts between > > > > different versions of jar packages. If classloader isolation is not > > used, > > > > shading becomes necessary to resolve conflicts, resulting in a > > > significant > > > > waste of development time. I understand that implementing this change > > may > > > > require numerous API modifications, so I would like to discuss in > this > > > > email. > > > > > > > > > Plugins cannot access classes from other plugins or from Flink that > > > have > > > > not been specifically whitelisted. > > > > > This strict isolation allows plugins to contain conflicting > versions > > of > > > > the same library without the need to relocate classes or to converge > to > > > > common versions. > > > > > Currently, file systems and metric reporters are pluggable but, in > > the > > > > future, connectors, formats, and even user code should also be > > pluggable. > > > > > > > > [2] It is my preliminary idea. > > > > > > > > [1] > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/ > > > > [2] > > > > > > > > > > https://docs.google.com/document/d/1XP2fBpcntK0YIdQ_Ax7JV2MhNdebvkFxSiNJRp6WQ24/edit?usp=sharing > > > > > > > > > > > > Best, > > > > Zhiqiang > > > > > > > > > > > > > > > > > -- > > > > Best, > > Benchao Li > > >
