"I don't think acceptance tests should loosely associate with real uses, but they should be free to delve into weird non-happy-pathways."
Not following - are you saying they should *tightly* associate with real uses and additonally include non-happy-path? On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella <ceste...@gmail.com> wrote: > It is absolutely not a naive question, Matt. We don't have a lot (or any) > docs about our integration tests; it's more of a "follow the lead" type of > thing at the moment, but that should be rectified. > > The integration tests spin up and down infrastructure in-process, some of > which are real and some of which are mock versions of the services. These > are good for catching some types of bugs, but often things sneak through, > like: > > - Hbase and storm can't exist in the same JVM, so HBase is mocked in > those cases. > - The FileSystem that we get for Hadoop is the LocalRawFileSystem, not > truly HDFS. There are differences and we've run into them..hilariously > at > times. ;) > - Things done statically in a bolt are shared across all bolts because > they all are threads in the same process > > It's good, it catches bugs, it lets us debug things easily, it runs with > every single build automatically via travis. > It's bad because it's awkward to get the dependencies isolated sufficiently > for all of these components to get them to play nice in the same JVM. > > Acceptance tests would be run against a real cluster, so they would: > > - run against real components, not testing or mock components > - run against multiple nodes > > I can imagine a world where we can unify the two to a certain degree in > many cases if we could spin up a docker version of Metron to run as part of > the build, but I think in the meantime, we should focus on providing both. > > I suspect the reference application is possibly inspiring my suggestions > here, but I think the main difference here is that the reference > application is intended to be informational from a end-user perspective: > it's detailing a use-case that users will understand. I don't think > acceptance tests should loosely associate with real uses, but they should > be free to delve into weird non-happy-pathways. > > On Fri, Mar 3, 2017 at 2:16 PM, Matt Foley <ma...@apache.org> wrote: > > > Automating stuff that now has to be done manually gets a big +1. > > > > But, Casey, could you please clarify the relationship between what you > > plan to do and the current “integration test” framework? Will this be in > > the form of additional integration tests? Or a different test framework? > > Can it be done in the integration test framework, rather than creating > new > > mechanism? > > > > BTW, if that’s a naïve question, forgive me, but I could find zero > > documentation for the existing integration test capability, neither wiki > > pages nor READMEs nor Jiras. If there are any docs, please point me at > > them. Or even archived email threads. > > > > There is also something called the “Reference Application” > > https://cwiki.apache.org/confluence/display/METRON/ > > Metron+Reference+Application which sounds remarkably like what you > > propose to automate. Is there / can there / should there be a > relationship? > > > > Thanks, > > --Matt > > > > On 3/3/17, 7:40 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote: > > > > +1 > > > > I agree with Justin’s points. > > > > > > On March 3, 2017 at 08:41:37, Justin Leet (justinjl...@gmail.com) > > wrote: > > > > +1 to both. Having this would especially ease a lot of testing that > > hits > > multiple areas (which there is a fair amount of, given that we're > > building > > pretty quickly). > > > > I do want to point out that adding this type of thing makes the speed > > of > > our builds and tests more important, because they already take up a > > good > > amount of time. There are obviously tickets to optimize these things, > > but > > I would like to make sure we don't pile too much on to every testing > > cycle > > before a PR. Having said that, I think the testing proposed is > > absolutely > > valuable enough to go forward with. > > > > Justin > > > > On Fri, Mar 3, 2017 at 8:33 AM, Casey Stella <ceste...@gmail.com> > > wrote: > > > > > I also propose, once this is done, that we modify the developer > > bylaws > > and > > > the github PR script to ensure that PR authors: > > > > > > - Update the acceptance tests where appropriate > > > - Run the tests as a smoketest > > > > > > > > > > > > On Fri, Mar 3, 2017 at 8:21 AM, Casey Stella <ceste...@gmail.com> > > wrote: > > > > > > > Hi All, > > > > > > > > After doing METRON-744, where I had to walk through a manual test > > of > > > every > > > > place that Stellar touched, it occurred to me that we should > script > > this. > > > > It also occurred to me that some scripts that are run by the PR > > author > > to > > > > ensure no regressions and, eventually maybe, even run on an INFRA > > > instance > > > > of Jenkins would give all of us some peace of mind. > > > > > > > > I am certain that this, along with a couple other manual tests > from > > other > > > > PRs, could form the basis of a really great regression > > acceptance-test > > > > suite and I'd like to propose that we do that, as a community. > > > > > > > > What I'd like to see from such a suite has the following > > characteristics: > > > > > > > > - Can be run on any Metron cluster, including but not limited to > > > > - Vagrant > > > > - AWS > > > > - An existing deployment > > > > - Can be *deployed* from ansible, but must be able to be deployed > > > > manually > > > > - With instructions in the readme > > > > - Tests should be idempotent and independent > > > > - Tear down what you set up > > > > > > > > I think between the Stellar REPL and the fundamental > scriptability > > of > > the > > > > Hadoop services, we can accomplish these tests with a combination > > of > > > shell > > > > scripts and python. > > > > > > > > I propose we break this into the following parts: > > > > > > > > - Acceptance Testing Framework with a small smoketest > > > > - Baseline Metron Test > > > > - Send squid data through the squid topology > > > > - Add an threat triage alert > > > > - Ensure it gets through to the other side with alerts preserved > > > > - + Enrichment > > > > - Add an enrichment in the enrichment pipeline to the above > > > > - + Profiler > > > > - Add a profile with a tick of 1 minute to count per destination > > > > address > > > > - Base PCap test > > > > - Something like the manual test for METRON-743 ( > > > > https://github.com/apache/incubator-metron/pull/467# > > > issue-210285324 > > > > <https://github.com/apache/incubator-metron/pull/467# > > > issue-210285324> > > > > ) > > > > > > > > Thoughts? > > > > > > > > > > > > Best, > > > > > > > > Casey > > > > > > > > > > > > > > > >
