I would not go with 2); I think it'd just be messy . Here's another option:
Create another repository (aws-connector-base) (following the externalization model), add it as a sub-module to the downstream repositories, and make it part of the release process of said connector.
I.e., we never create a release for aws-connector-bose, but release it as part of the connector. This main benefit here is that we'd always be able to make changes to the aws-base code without delaying connector releases. I would assume that any added overhead due to _technically_ releasing the aws code multiple times to be negligible.
On 20/10/2022 22:38, Danny Cranmer wrote:
Hello all, Currently we have 2 AWS Flink connectors in the main Flink codebase (Kinesis Data Streams and Kinesis Data Firehose) and one new externalized connector in progress (DynamoDB). Currently all three of these use common AWS utilities from the flink-connector-aws-base module. Common code includes client builders, property keys, validation, utils etc. Once we externalize the connectors, leaving flink-connector-aws-base in the main Flink repository will restrict our ability to evolve the connectors quickly. For example, as part of the DynamoDB connector build we are considering adding a general retry strategy config that can be leveraged by all connectors. We would need to block on Flink 1.17 for this. In the past we have tried to keep the AWS SDK version consistent across connectors, with the externalization this is more likely to diverge. Option 1: I propose we create a new repository, flink-connector-aws, which we can move the flink-connector-aws-base module to and create a new flink-connector-aws-parent to manage SDK versions. Each of the externalized AWS connectors will depend on this new module and parent. Downside is an additional module to release per Flink version, however I will volunteer to manage this. Option 2: We can move the flink-connector-aws-base module and create flink-connector-parent within the flink-connector-shared-utils repo [2] Option 3: We do nothing. For option 1+2 we will follow the general externalized connector versioning strategy and rules. I am inclined towards option 1, and appreciate feedback from the community. [1] https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base [2] https://github.com/apache/flink-connector-shared-utils Thanks, Danny
