Hi all! As I've discussed previously on this list[1], ensuring that we have high quality IO Transforms is important to beam. We want to do this without adding too much burden on developers wanting to contribute. Below I have a concrete proposal for what an IO integration test would look like and an example integration test[4] that meets those requirements.
Proposal: we should require that an IO transform includes a passing integration test showing the IO can connect to real instance of the data store. We still want/expect comprehensive unit tests on an IO transform, but we would allow check ins with just some unit tests in the presence of an IT. To support that, we'll require the following pieces associated with an IT: 1. Dockerfile that can be used to create a running instance of the data store. We've previously discussed on this list that we would use docker images running inside kubernetes or mesos[2], and I'd prefer having a kubernetes/mesos script to start a given data store, but for a single instance data store, we can take a dockerfile and use it to create a simple kubernetes/mesos app. If you have questions about how maintaining the containers long term would work, check [2] as I discussed a detailed plan there. 2. Code to load test data on the data store created by #1. Needs to be self contained. For now, the easiest way to do this would be to have code inside of the IT. 3. The IT. I propose keeping this inside of the same module as the IO transform itself since having all the IO transform ITs in one module would mean there may be conflicts between different data store's dependencies. Integration tests will need connection information pointing to the data store it is testing. As discussed previously on this list[3], it should receive that connection information via TestPipelineOptions. I'd like to get something up and running soon so people checking in new IO transforms can start taking advantage of an IT framework. Thus, there are a couple simplifying assumptions in this plan. Pieces of the plan that I anticipate will evolve: 1. The test data load script - we would like to write these in a uniform way and especially ensure that the test data is cleaned up after the tests run. 2. Spinning up/down instances - for now, we'd likely need to do this manually. It'd be good to get an automated process for this. That's especially critical for performance tests with multiple nodes - there's no need to keep instances running for that. Integrating closer with PKB would be a good way to do both of these things, but first let's focus on getting some basic ITs running. As a concrete example of this proposal, I've written JDBC IO IT [4]. JdbcIOTest already did a lot of test setup, so I heavily re-used it. The key pieces: * The integration test is in JdbcIOIT. * JdbcIOIT reads the TestPipelineOptions defined in PostgresTestOptions. We may move the TestOptions files into a common place so they can be shared between tests. * Test data is created/cleaned up inside of the IT. * kubernetes/mesos scripts - I have provided examples of both under the "jdbc/src/test/resources" directory, but I'd like us to decide as a project which container orchestration service we want to use - I'll send mail about that shortly. thanks! Stephen [1] Integration Testing Sources https://lists.apache.org/thread.html/518d78478ae9b6a56d6a690033071aa6e3b817546499c4f0f18d247d@%3Cdev.beam.apache.org%3E [2] Container Orchestration software for hosting data stores https://lists.apache.org/thread.html/5825b35b895839d0b33b6c726c1de0e76bdb9653d1e913b1207c6c4d@%3Cdev.beam.apache.org%3E [3] Some Thoughts on IO Integration Tests https://lists.apache.org/thread.html/637803ccae9c9efc0f4ed01499f1a0658fa73e761ab6ff4e8fa7b469@%3Cdev.beam.apache.org%3E [4] JDBC IO IT using postgres https://github.com/ssisk/beam/tree/io-testing/sdks/java/io/jdbc - have not been reviewed yet, so may contain code errors, but it does run & pass :)
