+1 to Docker and looking at testcontainers.org On Fri, Mar 8, 2019 at 3:03 PM Otto Fowler <ottobackwa...@gmail.com> wrote:
> https://github.com/apache/metron-bro-plugin-kafka/tree/master/docker > > > On March 8, 2019 at 14:28:20, zeo...@gmail.com (zeo...@gmail.com) wrote: > > So most importantly I want to make sure to give Otto credit for being the > one who cleaned up the rudimentary testing steps we had for testing the > plugin and turned it into the docker end to end. Right now we manually run > the tests, as there were a few follow-ons we needed to work through before > it's ready for Travis. In my opinion, once METRON-2003 (PR 26) gets in > it'll be ready to have Travis. There isn't any current Maven use > > Jon > > On Fri, Mar 8, 2019 at 12:26 PM Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > I believe that the TestContainers allows the ide case > > > > > > On March 8, 2019 at 11:38:24, Michael Miklavcic ( > > michael.miklav...@gmail.com) > > wrote: > > > > I'm -1 on #1 unless there's some desperately compelling reason to go that > > route. It would be a regression in our test coverage, and at that point > > it's really just duplicating our unit tests as opposed to checking our > > integration. > > > > I'm good with 3. Gating factors for a successful implementation would be > > that as a developer I can: > > > > 1. Run it in my IDE without having to do anything extra (the beauty of > > the in-mem component is that @BeforeClass spins it up automatically - we > > should keep doing something along those lines) > > 2. Run it via Maven cli > > 3. Run it in Travis as part of our normal build > > > > It's probably worth looking at Kafka's testing infrastructure straight > from > > the source - https://github.com/apache/kafka/blob/trunk/tests/README.md. > > They leverage Docker containers now for system tests. > > > > Best, > > Mike > > > > > > On Fri, Mar 8, 2019 at 7:47 AM Ryan Merriman <merrim...@gmail.com> > wrote: > > > > > I have been researching the effort involved to upgrade to HDP 3. Along > > the > > > way I've found a couple challenging issues that we will need to solve, > > both > > > involving our integration testing strategy. > > > > > > The first issue is Kafka. We are moving from 0.10.0 to 2.0.0 and there > > > have been significant changes to the API. This creates an issue in the > > > KafkaComponent class, which we use as an in-memory Kafka server in > > > integration tests. Most of the classes that were previously used have > > gone > > > away, and to the best of my knowledge, were not supported as public > APIs. > > > I also don't see any publicly documented APIs to replace them. > > > > > > The second issue is HBase. We are moving from 1.1.2 to 2.0.2 so another > > > significant change. This creates an issue in the MockHTable class > > > becausethe HTableInterface class has changed to Table, essentially > > > requiring that MockHTable be rewritten to conform to the new interface. > > > It's my opinion that this class is complicated and difficult to > maintain > > as > > > it is anyways. > > > > > > These 2 issues have the potential to add a significant amount of work > to > > > upgrading Metron to HDP 3. I want to take a step back and review our > > > options before we move forward. Here are some initial thoughts I had on > > > how to approach this. For HBase: > > > > > > 1. Update MockHTable to work with the new HBase API. We would continue > > > using a mock server approach for HBase. > > > 2. Research replacing MockHTable with an in-memory HBase server. > > > 3. Replace MockHTable with a Docker container running HBase. > > > > > > For Kafka: > > > > > > 1. Replace KafkaComponent with a mock server implementation. > > > 2. Update KafkaComponent to work with the new API. We would probably > > > need to leverage some internal Kafka classes. I do not see a testing > > > API > > > documented publicly. > > > 3. Replace KafkaComponent with a Docker container running Kafka. > > > > > > What other options are there? Whatever we choose I think we should > follow > > > a similar approach for both (mock servers, in memory servers, Docker, > > other > > > options I'm not thinking of). > > > > > > This will not shock anyone but I would be in favor of Docker > containers. > > > They have the advantage of classpath isolation, easy upgrades, and > > accurate > > > integration testing. The downside is we will have to adjusts our tests > > and > > > travis script to incorporate these Docker containers into our build > > > process. We have discussed this at length in the past and it has > > generally > > > stalled for various reasons. Maybe if we move a few services at a time > it > > > might be more palatable? As for the other 2 approaches, I think if > either > > > worked well we wouldn't be having this discussion. Mock servers are > hard > > > to maintain and I don't see in memory testing classes documented in > > > javadocs for either service. > > > > > > Thoughts? > > > > > > -- > > Jon Zeolla > -- -Kyle
