Thanks Chesnay for the suggestion, I will investigate this option. Related to the single repo idea, I have considered it in the past. Are you proposing we also use a single version between all connectors? If we have a single version then it makes sense to combine them in a single repo, if they are separate versions, then splitting them makes sense. This was discussed last year more generally [1] and the consensus was "we ultimately propose to have a single repository per connector".
Combining all AWS connectors into a single repo with a single version is inline with how the AWS SDK works, therefore AWS users are familiar with this approach. However it is frustrating that we would have to release all connectors to fix a bug or add a feature in one of them. Example: a user is using Kinesis Data Streams only (the most popular and mature connector), and we evolve the version from 1.x to 2.y (or 1.x to 1.y) for a DynamoDB change. I am torn and will think some more, but it would be great to hear other people's opinions. [1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm Thanks, Danny On Fri, Oct 21, 2022 at 3:11 PM Jing Ge <j...@ververica.com> wrote: > I agree with Jark. It would be easier for the further development and > maintenance, if all aws related connectors and the base module are in the > same repo. It might make sense to upgrade the flink-connector-dynamodb to > flink-connector-aws and move the other modules including the > flink-connector-aws-base into it. The aws sdk could be managed in > flink-connector-aws-base. Any future common connector features could also > be developed in the base module. > > Best regards, > Jing > > On Fri, Oct 21, 2022 at 1:26 PM Jark Wu <imj...@gmail.com> wrote: > >> How about creating a new repository flink-connector-aws and merging >> dynamodb, kinesis firehouse into it? >> This can reduce the maintenance for complex dependencies and make the >> release easy. >> I think the maintainers of aws-releated connectors are the same people. >> >> Best, >> Jark >> >> > 2022年10月21日 17:41，Chesnay Schepler <ches...@apache.org> 写道： >> > >> > I would not go with 2); I think it'd just be messy . >> > >> > Here's another option: >> > >> > Create another repository (aws-connector-base) (following the >> externalization model), add it as a sub-module to the downstream >> repositories, and make it part of the release process of said connector. >> > >> > I.e., we never create a release for aws-connector-bose, but release it >> as part of the connector. >> > This main benefit here is that we'd always be able to make changes to >> the aws-base code without delaying connector releases. >> > I would assume that any added overhead due to _technically_ releasing >> the aws code multiple times to be negligible. >> > >> > >> > On 20/10/2022 22:38, Danny Cranmer wrote: >> >> Hello all, >> >> >> >> Currently we have 2 AWS Flink connectors in the main Flink codebase >> >> (Kinesis Data Streams and Kinesis Data Firehose) and one new >> externalized >> >> connector in progress (DynamoDB). Currently all three of these use >> common >> >> AWS utilities from the flink-connector-aws-base module. Common code >> >> includes client builders, property keys, validation, utils etc. >> >> >> >> Once we externalize the connectors, leaving flink-connector-aws-base >> in the >> >> main Flink repository will restrict our ability to evolve the >> connectors >> >> quickly. For example, as part of the DynamoDB connector build we are >> >> considering adding a general retry strategy config that can be >> leveraged by >> >> all connectors. We would need to block on Flink 1.17 for this. >> >> >> >> In the past we have tried to keep the AWS SDK version consistent across >> >> connectors, with the externalization this is more likely to diverge. >> >> >> >> Option 1: I propose we create a new repository, flink-connector-aws, >> which >> >> we can move the flink-connector-aws-base module to and create a new >> >> flink-connector-aws-parent to manage SDK versions. Each of the >> externalized >> >> AWS connectors will depend on this new module and parent. Downside is >> an >> >> additional module to release per Flink version, however I will >> volunteer to >> >> manage this. >> >> >> >> Option 2: We can move the flink-connector-aws-base module and create >> >> flink-connector-parent within the flink-connector-shared-utils repo [2] >> >> >> >> Option 3: We do nothing. >> >> >> >> For option 1+2 we will follow the general externalized connector >> versioning >> >> strategy and rules. >> >> >> >> I am inclined towards option 1, and appreciate feedback from the >> community. >> >> >> >> [1] >> >> >> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base >> >> [2] https://github.com/apache/flink-connector-shared-utils >> >> >> >> Thanks, >> >> Danny >> >> >> > >> >>
