[Discussion] Sharing test containers/infrastructure among projects
Hello everyone! Luca B, Andrea T. and I had a brief discussion / brainstorm today about adjusting part of our testing infrastructure code so that it is easier for all the projects to levage testing bits used throughout the project. The issue that led to this idea can be found here [1]. As it is common these days, a lot of projects use TestContainers [2] to set up the test infrastructure required for the tests to run. This is how the integration tests for the Camel Kafka Connector work [3]. In the Camel Kafka Connector, the test infrastructure contains "AbstractServices". They can be resolved either to containers managed by TestContainers or remote instances of the services. These services are injected into the test code using JUnit 5 extensions. It uses a generic approach and it is not tightly coupled to Kafka Connect or the Camel Kafka Connector project. In more concrete terms, the idea is to make this test infrastructure more easily accessible for other projects. This would involve moving this code to another project (Camel? Where?), adjusting the few parts that may be coupled (if any at all) and moving the part of the testing documentation [4] (check the "Simulating the Test Infrastructure" part). A few potential benefits of this idea are reducing our effort in handling the test infrastructure throughout the code base and simplifying how we assert the behavior of each project for each infrastructure (ie.: we may want to make sure that components behave the same way across all projects). What do you think? 1. https://github.com/apache/camel-quarkus/issues/1783 2. https://www.testcontainers.org/ 3. https://github.com/apache/camel-kafka-connector/tree/master/tests 4. https://camel.apache.org/camel-kafka-connector/latest/testing.html Kind regards -- Otavio R. Piske http://orpiske.net
Re: [Discussion] Sharing test containers/infrastructure among projects
Hi Otavio, as you know, big +1 on that idea. As a strategy I'm thinking about: 1. try to contribute to TestContainers to make it easy to configure some common tasks the implementations they ship and eventually provide some new ones. 2. create some new modules, don't ask me for a name, with the core infrastructure code and some specific impl. 3. see if we can easily hook such service in the camel-test* and eventually deprecate camel-testcontainers* --- Luca Burgazzoli On Fri, Sep 11, 2020 at 3:49 PM Otavio Rodolfo Piske wrote: > Hello everyone! > > Luca B, Andrea T. and I had a brief discussion / brainstorm today about > adjusting part of our testing infrastructure code so that it is easier for > all the projects to levage testing bits used throughout the project. The > issue that led to this idea can be found here [1]. > > As it is common these days, a lot of projects use TestContainers [2] to set > up the test infrastructure required for the tests to run. This is how the > integration tests for the Camel Kafka Connector work [3]. > > In the Camel Kafka Connector, the test infrastructure contains > "AbstractServices". They can be resolved either to containers managed by > TestContainers or remote instances of the services. These services are > injected into the test code using JUnit 5 extensions. It uses a generic > approach and it is not tightly coupled to Kafka Connect or the Camel Kafka > Connector project. > > In more concrete terms, the idea is to make this test infrastructure more > easily accessible for other projects. This would involve moving this code > to another project (Camel? Where?), adjusting the few parts that may be > coupled (if any at all) and moving the part of the testing documentation > [4] (check the "Simulating the Test Infrastructure" part). > > A few potential benefits of this idea are reducing our effort in handling > the test infrastructure throughout the code base and simplifying how we > assert the behavior of each project for each infrastructure (ie.: we may > want to make sure that components behave the same way across all projects). > > What do you think? > > 1. https://github.com/apache/camel-quarkus/issues/1783 > 2. https://www.testcontainers.org/ > 3. https://github.com/apache/camel-kafka-connector/tree/master/tests > 4. https://camel.apache.org/camel-kafka-connector/latest/testing.html > > Kind regards > -- > Otavio R. Piske > http://orpiske.net >
Re: [Discussion] Sharing test containers/infrastructure among projects
Thanks Luca, I think we could try getting started with this by doing #2 or #3 and then, investigate what/if any stuff, we can contribute to Test Containers. As far as I know, they have some restrictions to what they accept. An important one is that they only accept TestContainers with official images (which rule out a few of ours) and this rules out a few of those that we have (ie.: Artemis, Qpid Interconnect, Apache Hadoop, etc). We would need Apache to start to provide official images for those otherwise they wouldn't accept it. So, I think that's the biggest barrier for #1. If that's OK, I will send a PR with a proposal for moving the current ones in Camel Kafka Connector to Camel. Something along the lines of strategy #2 and #3. Kind regards On Wed, Sep 16, 2020 at 9:29 AM Luca Burgazzoli wrote: > Hi Otavio, > > as you know, big +1 on that idea. > > As a strategy I'm thinking about: > > 1. try to contribute to TestContainers to make it easy to configure some > common tasks the implementations they ship and eventually provide some new > ones. > 2. create some new modules, don't ask me for a name, with the core > infrastructure code and some specific impl. > 3. see if we can easily hook such service in the camel-test* and eventually > deprecate camel-testcontainers* > > --- > Luca Burgazzoli > > > On Fri, Sep 11, 2020 at 3:49 PM Otavio Rodolfo Piske > > wrote: > > > Hello everyone! > > > > Luca B, Andrea T. and I had a brief discussion / brainstorm today about > > adjusting part of our testing infrastructure code so that it is easier > for > > all the projects to levage testing bits used throughout the project. The > > issue that led to this idea can be found here [1]. > > > > As it is common these days, a lot of projects use TestContainers [2] to > set > > up the test infrastructure required for the tests to run. This is how the > > integration tests for the Camel Kafka Connector work [3]. > > > > In the Camel Kafka Connector, the test infrastructure contains > > "AbstractServices". They can be resolved either to containers managed by > > TestContainers or remote instances of the services. These services are > > injected into the test code using JUnit 5 extensions. It uses a generic > > approach and it is not tightly coupled to Kafka Connect or the Camel > Kafka > > Connector project. > > > > In more concrete terms, the idea is to make this test infrastructure more > > easily accessible for other projects. This would involve moving this code > > to another project (Camel? Where?), adjusting the few parts that may be > > coupled (if any at all) and moving the part of the testing documentation > > [4] (check the "Simulating the Test Infrastructure" part). > > > > A few potential benefits of this idea are reducing our effort in handling > > the test infrastructure throughout the code base and simplifying how we > > assert the behavior of each project for each infrastructure (ie.: we may > > want to make sure that components behave the same way across all > projects). > > > > What do you think? > > > > 1. https://github.com/apache/camel-quarkus/issues/1783 > > 2. https://www.testcontainers.org/ > > 3. https://github.com/apache/camel-kafka-connector/tree/master/tests > > 4. https://camel.apache.org/camel-kafka-connector/latest/testing.html > > > > Kind regards > > -- > > Otavio R. Piske > > http://orpiske.net > > > -- Otavio R. Piske http://orpiske.net
