Hi Israel, thanks a lot for reaching out! I'm very excited about your efforts to bring additional Azure support into Flink. There are ~50 threads on the user@ mailing list mentioning Azure -- that's good evidence that our users use Flink in Azure.
Regarding your questions: Do I need to create a FLIP [2] in order to make these changes to bring the > new capabilities or the individual JIRA issues are sufficient? For the two DataLake FS tickets, I don't see the need for a FLIP. I am thinking about targeting Flink versions 1.10 through 1.12 > For new connectors like this, how many versions can/should this be > integrated into? We generally don't add new features to old releases (unless there's a very good reason to backport the feature). Therefore, the new integrations will all go into the next major Flink release (in this case probably Flink 1.12 for the first tickets). Are there any upcoming changes to supported Java and Scala versions that I > need to be aware of? No, I'm not aware of any upcoming changes. Java 8 and Java 11 are the two versions we test against. My goal is to add the first two features [FLINK-18562] and [FLINK-18568] to > the existing file system capabilities [1] and then have the other > connectors FLINK-1856[3-7] exist as standalone plugins. I like the order in which you approach the tickets. I assigned you to the first ticket and commented on the second one. I'm also willing to help review your pull requests. What do you mean by "standalone plugins" in the context of connectors? Would you like to contribute these connectors to the main Flink codebase, or maintain them outside Flink but list them in flink-packages.org? Best, Robert On Wed, Jul 22, 2020 at 10:43 AM Israel Ekpo <israele...@gmail.com> wrote: > I have opened the following issues to track new efforts to bring additional > Azure Support in the following areas to the connectors ecosystem. > > My goal is to add the first two features [FLINK-18562] and [FLINK-18568] to > the existing file system capabilities [1] and then have the other > connectors FLINK-1856[3-7] exist as standalone plugins. > > As more users adopt the additional connectors, we could incrementally bring > them into the core code base if necessary with sufficient support. > > I am new to the process so that I have a few questions: > > Do I need to create a FLIP [2] in order to make these changes to bring the > new capabilities or the individual JIRA issues are sufficient? > > I am thinking about targeting Flink versions 1.10 through 1.12 > For new connectors like this, how many versions can/should this be > integrated into? > > Are there any upcoming changes to supported Java and Scala versions that I > need to be aware of? > > Any ideas or suggestions you have would be great. > > Below is a summary of the JIRA issues that were created to track the effort > > Add Support for Azure Data Lake Store Gen 2 in Flink File System > https://issues.apache.org/jira/browse/FLINK-18562 > > Add Support for Azure Data Lake Store Gen 2 in Streaming File Sink > https://issues.apache.org/jira/browse/FLINK-18568 > > Add Support for Azure Cosmos DB DataStream Connector > https://issues.apache.org/jira/browse/FLINK-18563 > > Add Support for Azure Event Hub DataStream Connector > https://issues.apache.org/jira/browse/FLINK-18564 > > Add Support for Azure Event Grid DataStream Connector > https://issues.apache.org/jira/browse/FLINK-18565 > > Add Support for Azure Cognitive Search DataStream Connector > https://issues.apache.org/jira/browse/FLINK-18566 > > Add Support for Azure Cognitive Search Table & SQL Connector > https://issues.apache.org/jira/browse/FLINK-18567 > > > [1] https://github.com/apache/flink/tree/master/flink-filesystems > [2] > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals >
