I'm interested in this, we often have linker dependency conflicts and it takes a lot of work to resolve dependency conflicts.
Benchao Li <libenc...@apache.org> 于2023年7月25日周二 20:51写道： > I agree with Jing that the current doc is quite preliminary, and needs to > be turned into a FLIP. > > I'll be very interested in this FLIP, and looking forward to it. We've > suffered from reshading/relocating heavy connector dependencies for a long > time. > > Besides, we've implemented a plugin mechanism to load hive udf in our batch > SQL scenario internally, it saves us a lot of effort to handle the > dependency conflicts. The most challenging part would be the > (de)serialization of those classes loaded through plugins according to our > experience. (I noticed you have already considered this) > > Jing Ge <j...@ververica.com.invalid> 于2023年7月21日周五 06:44写道： > > > Hi Zhiqiang, > > > > Thanks for your proposal. The idea is very interesting! Since almost all > > connectors are or will be externalized, the pluggable design you > suggested > > could help reduce the development effort. > > > > As you mentioned, the attached doc contains only your preliminary idea. I > > would suggest that you could turn it into a FLIP with more details and do > > the followings: > > > > 1. Describe a real conflict case that will benefit from the pluggable > > design > > 2. Describe where you want to modify the code, or provide a POC branch > > 3. Guideline to tell users how to use it, i.e. where (the plugins dir) > > should the connector jar be located, how does it look like with an > example, > > etc. > > > > WDYT? > > > > Best regards, > > Jing > > > > > > On Fri, Jul 14, 2023 at 3:00 PM zhiqiang li <lizhiqiang....@gmail.com> > > wrote: > > > > > Hi devs, > > > > > > I have observed that in [1], connectors and formats are pluggable, > > > allowing user code to be easily integrated. The advantages of having > > > pluggable connectors are evident, as it helps avoid conflicts between > > > different versions of jar packages. If classloader isolation is not > used, > > > shading becomes necessary to resolve conflicts, resulting in a > > significant > > > waste of development time. I understand that implementing this change > may > > > require numerous API modifications, so I would like to discuss in this > > > email. > > > > > > > Plugins cannot access classes from other plugins or from Flink that > > have > > > not been specifically whitelisted. > > > > This strict isolation allows plugins to contain conflicting versions > of > > > the same library without the need to relocate classes or to converge to > > > common versions. > > > > Currently, file systems and metric reporters are pluggable but, in > the > > > future, connectors, formats, and even user code should also be > pluggable. > > > > > > [2] It is my preliminary idea. > > > > > > [1] > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/ > > > [2] > > > > > > https://docs.google.com/document/d/1XP2fBpcntK0YIdQ_Ax7JV2MhNdebvkFxSiNJRp6WQ24/edit?usp=sharing > > > > > > > > > Best, > > > Zhiqiang > > > > > > > > > > > -- > > Best, > Benchao Li >
