Justin, what are your thoughts on leveraging this approach along with
long-lived Docker containers? I think the lifecycle would look like:
1. I need components A, B, C
2. If not started, start A, B, C
3. If started, clean/reset it
4. Setup pre-test state
5. Run test(s)
6. Clean/reset component state after test method and/or test class
completion (we may consider nuking this step - I assisted in designing an
acceptance testing framework in the past where we handled cleanup on the
front end. This meant that remediation for failed tests became simpler
because I still had the state from the failed test run)
Stopping/removing the containers could be a manual process or just a simple
script in the project root. We could also add a special module that runs
last that handles shutting down containers, if desired.
I know this has been a perennial thread for us. I can dig up the original
discuss threads on this as well if folks think they're still pertinent, but
I think we're pretty far removed now from that original point in time and
should just move forward with fresh perspectives.
On Wed, May 1, 2019 at 9:21 AM Justin Leet <justinjl...@gmail.com> wrote:
> Re: the integration testing point above, a strategy I used recently to
> alleviate a similar problem was to exploit JUnit 5's extensions. I haven't
> played with this at all in Metron's code, so 1) Take this with a several
> grains of salt and 2) Feel encouraged to point out anything
> wrong/problematic/ludicrous. My use case was pretty tame and easy to
> implement.
>
> Essentially, you can set up an extension backed by a singleton cluster (in
> the case I was doing, I had two extensions, one for a Kafka cluster and one
> for MiniDfs). The backing clusters expose some methods (i.e. create topic,
> get brokers, get file system, etc.), which lets the tests classes setup
> whatever they need. Classes that need them just use an annotation and can
> be setup under the hood to init only when tests in the current run suite
> actually use them and spin down after all are done. This saved ~1 min on a
> 4 minute build. It ends up being a pretty clean way to use them, although
> we might need to be a bit more sophisticated, since my case was less
> complicated. The main messiness is that this necessarily invites tests to
> interfere with each other (i.e. if multiple tests use the kafka topic
> "test", problems will be involved). We might be able to improve on this.
>
> We could likely do something similar with our InMemoryComponents. Right now
> these are spun up on a per-test basis, rather than the overall suite. This
> involves some setup in any class that wants them, just to be able to get a
> cluster. There's also some definition of spin up order and so on, which is
> already taken care of with the extensions (declaration order). The other
> catch in our case is that we have way more code covered by JUnit 4, which
> doesn't have the same capabilities. That migration doesn't necessarily
> have to be done all at once, but it is a potential substantial pain point.
>
> Basically, I'd hope to push management of our abstraction further back into
> actual JUnit so they're get more reuse across runs and it's easily
> understood what a test needs and uses right up front. In our case, the
> InMemoryComponents likely map to extensions.
>
> If we did something like this, it potentially solves a few problems
> * Makes it easy to build an integration test that says "Give me these
> components".
> * Keeps it alive across any test that needs them
> * Only spins it up if there are tests running that do need them. Only spins
> up the necessary components.
> * Lower our build times.
> * Interaction with components remains primarily through the same methods
> you'd normally interact (you can build producers/consumers, or whatever).
> * IMO, easier to explain and have people use. I've had a couple people
> pretty easily pick up on it.
>
> I can't share the code, but essentially it looks like (And I'm sure email
> will butcher this, so I'm sorry in advance):
> @ExtendWith({<ExtensionOne.class, ExtensionTwo.class, ...})
> ...
> @BeforeAll
> public static void beforeAll() {
> // Test specific setup, similar to before. But without the cluster
> creation work.
> }
>
> Everything else gets handled under the covers, with the caveat that what I
> needed to do was less involved than some of our tests, so there may be some
> experimenting.
>
> On Wed, May 1, 2019 at 10:59 AM Justin Leet <justinjl...@gmail.com> wrote:
>
> > Hi all,
> >
> > I wanted to start a discussion on something near and dear to all of our
> > hearts: The role of full-dev in our testing cycle.
> >
> > Right now, we require all PRs to have spun up the full-dev environment
> and
> > make sure that things flow through properly. In some cases, this is a
> > necessity, and in other cases, it's a fairly large burden on current and
> > potential contributors.
> >
> > So what do we need to do to reduce our dependence on full dev and
> increase
> > our confidence in our CI process?
> >
> > My initial thoughts on this encompass a few things
> > * Increasing our ability to easily write automated tests. In particular,
> I
> > think our integration testing is fairly hard and has some structural
> issues
> > (e.g. our tests spin up components per Test class, not for the overall
> test
> > suite being run, which also increases build times, etc.). How do we make
> > this easier and have less boilerplate? I have one potential idea I'll
> > reply to this thread with.
> > * Unit tests in general. I'd argue that any area we thing need full-dev
> > spun up to be confident in needs more testing. Does anyone have any
> > opinions on which areas we have the least confidence in? Which areas of
> > code are people afraid to touch?
> > * Our general procedures. Should we not be requiring full-dev on all PRs,
> > but maybe just asking for a justification why not (e.g. "I don't need
> > full-dev for this, because I added unit tests here and here and
> integration
> > tests for the these components?"). And then a reviewer can request a
> > full-dev spin up if needed? Could we stage this rollout (e.g.
> > metron-platform modules require it, but others don't, etc.?) This'll add
> > more pressure onto the release phase, so maybe that involves fleshing
> out a
> > checklist of critical path items to check.
> > * Do we need to improve our docs? Publish Javadocs? After all, if docs
> are
> > better, hopefully we'd see less issues introduced and be more confident
> in
> > changes coming in.
> > * What environment do we even want testing to occur in?
> > https://github.com/apache/metron/pull/1261 has seen a lot of work, and
> > getting it across the finish line may help make the dev environment less
> > onerous, even if still needed.
> >
>
