Flink CDC Maintainer xbjt...@gmail.com Factory Interface Timo Walther <twal...@apache.org>
Plugin Interface Aleksey Pak <alek...@ververica.com> Connector Base Stephan Ewen <se...@apache.org> Flink File System Stephan Ewen <se...@apache.org> I have observed that in [1], connectors and formats are pluggable, allowing user code to be easily integrated. The advantages of having pluggable connectors are evident, as it helps avoid conflicts between different versions of jar packages. If classloader isolation is not used, shading becomes necessary to resolve conflicts, resulting in a significant waste of development time. I understand that implementing this change may require numerous API modifications, so I would like to discuss in this email whether it is feasible to make these changes in Flink 2.0. > Plugins cannot access classes from other plugins or from Flink that have not > been specifically whitelisted. > This strict isolation allows plugins to contain conflicting versions of the > same library without the need to relocate classes or to converge to common > versions. > Currently, file systems and metric reporters are pluggable but, in the > future, connectors, formats, and even user code should also be pluggable. I hope to be able to integrate into a set of APIs, which would require significant changes to the integration API. Is it possible to address this in Flink 2.0? However, due to [2] FLIP-321, Flink 1.18 should deprecate some APIs, and the community is uncertain about the existence of Flink 1.20. This may make it impossible to integrate a set of APIs. @Xintong Song <tonysong...@gmail.com> suggests continuing to use two APIs. I am aware of some API modifications, but considering the extensive changes involved, I am concerned that my understanding may not be comprehensive. I am willing to do the work, but I have not yet found a committer. [1] https://nightlies.apache.org/Flink/Flink-docs-master/docs/deployment/filesystems/overview/<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/> [2] FLIP-321 https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process Best, Zhiqiang
