I agree with Jark. It would be easier for the further development and maintenance, if all aws related connectors and the base module are in the same repo. It might make sense to upgrade the flink-connector-dynamodb to flink-connector-aws and move the other modules including the flink-connector-aws-base into it. The aws sdk could be managed in flink-connector-aws-base. Any future common connector features could also be developed in the base module.
Best regards, Jing On Fri, Oct 21, 2022 at 1:26 PM Jark Wu <imj...@gmail.com> wrote: > How about creating a new repository flink-connector-aws and merging > dynamodb, kinesis firehouse into it? > This can reduce the maintenance for complex dependencies and make the > release easy. > I think the maintainers of aws-releated connectors are the same people. > > Best, > Jark > > > 2022年10月21日 17:41，Chesnay Schepler <ches...@apache.org> 写道： > > > > I would not go with 2); I think it'd just be messy . > > > > Here's another option: > > > > Create another repository (aws-connector-base) (following the > externalization model), add it as a sub-module to the downstream > repositories, and make it part of the release process of said connector. > > > > I.e., we never create a release for aws-connector-bose, but release it > as part of the connector. > > This main benefit here is that we'd always be able to make changes to > the aws-base code without delaying connector releases. > > I would assume that any added overhead due to _technically_ releasing > the aws code multiple times to be negligible. > > > > > > On 20/10/2022 22:38, Danny Cranmer wrote: > >> Hello all, > >> > >> Currently we have 2 AWS Flink connectors in the main Flink codebase > >> (Kinesis Data Streams and Kinesis Data Firehose) and one new > externalized > >> connector in progress (DynamoDB). Currently all three of these use > common > >> AWS utilities from the flink-connector-aws-base module. Common code > >> includes client builders, property keys, validation, utils etc. > >> > >> Once we externalize the connectors, leaving flink-connector-aws-base in > the > >> main Flink repository will restrict our ability to evolve the connectors > >> quickly. For example, as part of the DynamoDB connector build we are > >> considering adding a general retry strategy config that can be > leveraged by > >> all connectors. We would need to block on Flink 1.17 for this. > >> > >> In the past we have tried to keep the AWS SDK version consistent across > >> connectors, with the externalization this is more likely to diverge. > >> > >> Option 1: I propose we create a new repository, flink-connector-aws, > which > >> we can move the flink-connector-aws-base module to and create a new > >> flink-connector-aws-parent to manage SDK versions. Each of the > externalized > >> AWS connectors will depend on this new module and parent. Downside is an > >> additional module to release per Flink version, however I will > volunteer to > >> manage this. > >> > >> Option 2: We can move the flink-connector-aws-base module and create > >> flink-connector-parent within the flink-connector-shared-utils repo [2] > >> > >> Option 3: We do nothing. > >> > >> For option 1+2 we will follow the general externalized connector > versioning > >> strategy and rules. > >> > >> I am inclined towards option 1, and appreciate feedback from the > community. > >> > >> [1] > >> > https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base > >> [2] https://github.com/apache/flink-connector-shared-utils > >> > >> Thanks, > >> Danny > >> > > > >
