Hi Alexey – As relatively new users of Beam, we recently selected v1 over v2 because we didn’t think v2 currently (as of 2.24.0-snapshot) had feature parity especially the lack of a v2 based S3FileSystem and KinesisIO.Write.
Ideally we would have selected v2. On a related note, and not really Beam’s problem, but if I remember rightly, we also had a bit of trouble creating some LocalStack testcontainers based integration tests with v2… Cheers, David David Hollands Audience Platform – Audience Data Engineering david.holla...@bbc.co.uk<mailto:david.holla...@bbc.co.uk> BC5 C5, BBC Broadcast Centre, London, W12 7TQ From: Alexey Romanenko <aromanenko....@gmail.com> Reply to: "user@beam.apache.org" <user@beam.apache.org> Date: Tuesday, 15 September 2020 at 17:06 To: "user@beam.apache.org" <user@beam.apache.org> Subject: Re: [DISCUSS] Deprecation of AWS SDK v2 IO connectors I just want to cross-post it on users@ to find out which version of AWS SDK connectors is mostly used in user applications and if there are any strong objections to switch mostly to AWS SDK v2? Thank you for any feedback in advance. On 11 Sep 2020, at 19:13, Alexey Romanenko <aromanenko....@gmail.com<mailto:aromanenko....@gmail.com>> wrote: Hello, In Beam, there are two versions of AWS IO connectors for Java SDK - based on AWS SDK v1 [1] and v2 [2]. For now, they are pretty equal in terms of functionality, but since AWS SDK v2 is more modern (it's a major rewrite of the version 1.x code base, it’s built on top of Java 8+ and adds more features [3]), then it would be more logical to use only V2. Also, it’s not reasonable to support two versions of similar connectors, since it’s a big pain for us, and it will be more clear for users which package of AWS connectors to use . According to this, I’d propose to deprecate all Java AWS IO connectors V1 (+ KinesisIO which is in a different package for now) starting from Beam 2.25 and then add new features only to V2 connectors. Bug fixes should be applied to V2 connectors in the first order, and to V1 connectors if it’s only necessary. What are the community thoughts on this? Any pros and cons that I'm missing? [1] https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services [2] https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services2 [3] https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/welcome.html
