Localstack looks really cool. We have to consider resource consumption and latency for where it fits in the testing plan. It definitely seems useful for having ITs easily. For individual IOs, an in-process implementation can be much more performant without losing much accuracy (since it is also just another implementation that is not the real one) so if that is available I would also use one of those (any time you would mock, use an in-process fake instead). Like direct runner vs ULR. You don't need the ULR to test your transform's correctness. Localstack's README discusses the pro/con of each pretty well. Fault injection is pretty compelling and process isolation might reduce dependency troubles.
Kenn On Fri, Jul 10, 2020 at 4:45 AM Alexey Romanenko <aromanenko....@gmail.com> wrote: > I think that we should get back to this question since that time we have > more and more AWS-related IO connectors (mostly in Java SDK afaik). > > It would be great to have a dedicated Beam’s credentials to run all our > AWS-releasted ITs against real AWS instance, but till then, I’m +1 to run > such tests against 3rd party implementations, for example, “localstack”, as > most comprehensive one. Of course we can observe some potential discrepancy > in behaviour between real AWS and other implementations, but if it’s not > principal things then it should not be a stopper. I believe that regular > running ITs, especially for IO connectors, is a very important. > > On 9 Jul 2020, at 22:18, Luke Cwik <lc...@google.com> wrote: > > It has come up a few times[1, 2, 3, 4] and there have also been a few > comments over time about whether someone could donate AWS resources to the > project. > > 1: https://issues.apache.org/jira/browse/BEAM-601 > 2: https://issues.apache.org/jira/browse/BEAM-3373 > 3: https://issues.apache.org/jira/browse/BEAM-3550 > 4: https://issues.apache.org/jira/browse/BEAM-3032 > > On Thu, Jul 9, 2020 at 1:02 PM Mani Kolbe <maniko...@gmail.com> wrote: > >> Have you guys considered using localstack to run AWS service based >> integration tests? >> >> https://github.com/localstack/localstack >> >> On Thu, 9 Jul, 2020, 5:25 PM Piotr Szuberski, < >> piotr.szuber...@polidea.com> wrote: >> >>> Yeah, I meant KinesisIOIT tests. I'll do the same with the >>> cross-language it tests then. Thanks for your reply :) >>> >>> On 2020/07/08 17:13:11, Alexey Romanenko <aromanenko....@gmail.com> >>> wrote: >>> > If you mean Java KinesisIO tests, then unit tests are running on >>> Jenkins [1] and ITs are not running since it requires AWS credentials that >>> we don’t have dedicated to Beam for the moment. >>> > >>> > In the same time, you can run KinesisIOIT with your own credentials, >>> like we do in Talend (a company that I work for). >>> > >>> > [1] >>> https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/ >>> < >>> https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/ >>> > >>> > >>> > > On 8 Jul 2020, at 13:11, Piotr Szuberski < >>> piotr.szuber...@polidea.com> wrote: >>> > > >>> > > I'm writing KinesisIO external transform with python wrapper and I >>> found that the tests aren't executed anywhere in Jenkins. Am I wrong or >>> there is a reason for that? >>> > >>> > >>> >> >
