Hi Evan, Great to hear that you are going to contribute to Beam. Welcome!
For integration tests we mostly use k8s. Did you take a look on current implementation of ITs for ElasticsearchIO (e.g. [1]) and how it runs on Jenkins [2]? Also, perhaps worth to mention our very good guide about writing ITs for Beam [3] [1] https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch-tests/elasticsearch-tests-7/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java [2] https://github.com/apache/beam/tree/master/.test-infra/kubernetes/elasticsearch [3] https://beam.apache.org/documentation/io/testing/#i-o-transform-integration-tests --- Alexey > On 22 Apr 2021, at 15:54, Evan Galpin <evan.gal...@gmail.com> wrote: > > Hi folks! > > I'm Evan, and I'm fairly new to developing the Beam SDK. I've been a user for > a number of years and have done some private SDK customizations along the way > for my day job, but have recently been given the green light to contribute > back to the OSS repo 🙌 In particular, I've worked with ElasticsearchIO the > most. > > I'm looking ahead at how to potentially revamp the ElasticsearchIO test > suite. I wanted to double-check if there are any WIP efforts underway so as > to not duplicate effort or step on toes. > > Barring that, I'm thinking that using docker containers to support > integration testing could be really beneficial. Do we have examples of doing > just that in the Java SDK at the moment? Do the Jenkins VMs support docker? > Looking around briefly outside of Beam it appears there are a few options for > running Docker containers from gradle [1][2]. Or perhaps there's another > alternative to Docker+gradle entirely. > > Thanks in advance for your advice, > Evan > > [1] https://github.com/bmuschko/gradle-docker-plugin > <https://github.com/bmuschko/gradle-docker-plugin> > [2] https://github.com/palantir/gradle-docker > <https://github.com/palantir/gradle-docker>
