Hi Emre, Thanks for clarifying it. Afaiac, it is a quite interesting proposal, especially for Flink job developers who are heavily using the Datastream API. Publishing the plugin in your Github would be a feasible way for the first move. As I mentioned previously, in order to help the community understand the plugin, you might want to describe all those attractive features you mentioned in e.g. the readme.md with more technical details. I personally was wondering how those connector compatibility rules will be defined and maintained, given that almost all connectors have been externalized.
Best regards, Jing On Mon, May 22, 2023 at 11:24 AM Kartoglu, Emre <kar...@amazon.co.uk.invalid> wrote: > Hi Jing, > > The proposed plugin would be used by Flink application developers, when > they are writing their Flink job. It would trigger during > compilation/packaging and would look for known incompatibilities, bad > practices, or bugs. > For instance one cause of frustration for our customers is connector > incompatibilities (specifically Kafka and Kinesis) with certain Flink > versions. This plugin would be a quick way to update a list of known > incompatibilities, bugs, bad practices, so customers get errors during > compilation/packaging and not after they've deployed their Flink job. > > From what you're saying, the FLIP route might not be the best way to go. > We might publish this plugin in our own GitHub namespace/group first, and > then get community acknowledgement/support for it. I believe working with > the Flink community on this is key as we'd need their support/opinion to do > this the right way and reach more Flink users. > > Thanks > Emre > > On 21/05/2023, 16:48, "Jing Ge" <j...@ververica.com.inva <mailto: > j...@ververica.com.inva>LID> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > Hi Emre, > > > Thanks for your proposal. It looks very interesting! Please pay attention > that most connectors have been externalized. Will your proposed plug be > used for building Flink Connectors or Flink itself? Furthermore, it would > be great if you could elaborate features wrt best practices so that we > could understand how the plugin will help us. > > > Afaik, FLIP is recommended for improvement ideas that will change public > APIs. I am not sure if a new maven plugin belongs to it. > > > Best regards, > Jing > > > On Tue, May 16, 2023 at 11:29 AM Kartoglu, Emre <kar...@amazon.co.uk.inva > <mailto:kar...@amazon.co.uk.inva>lid> > wrote: > > > > Hello all, > > > > Myself and 2 colleagues developed a Maven plugin (no support for Gradle > or > > other build tools yet) that we use internally to detect potential issues > in > > Flink apps at compilation/packaging stage: > > > > > > * Known connector version incompatibilities – so far covering Kafka > > and Kinesis > > * Best practices e.g. setting operator IDs > > > > We’d like to make this open-source. Ideally with the Flink community’s > > support/mention of it on the Flink website, so more people use it. > > > > Going forward, I believe we have at least the following options: > > > > * Get community support: Create a FLIP to discuss where the plugin > > should live, what kind of problems it should detect etc. > > * We still open-source it but without the community support (if the > > community has objections to officially supporting it for instance). > > > > Just wanted to gauge the feeling/thoughts towards this tool from the > > community before going ahead. > > > > Thanks, > > Emre > > > > > > > >
