I think this may have come up in another PR already (have to look for it). But maybe we could maintain our flexibility in quick-dev by installing the sensors and not starting them until we need them. I think it's useful to have a quick "genuine" e2e testing environment that doesn't require running through a full install. I'm also not opposed to extracting the integration test functionality into general purpose data generators.
On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen <n...@nickallen.org> wrote: > To Jon's point, I think it would be useful to have a Demo box that uses > generators to produce 3 or 4 types of telemetry that shows up in the Metron > Dashboard. This box would be different from Quick-Dev in that everything > starts automatically, so that a user just has to launch it and the should > start seeing data in the Metron Dashboard right away. In fact, we could > even pre-load the Elasticsearch indices so that the user has more of a > history to mine when using the Demo box. > > On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com <zeo...@gmail.com> > wrote: > > > +1 Ryan and Otto's comments. > > > > I also strongly think we need to make a demo environment easier, but that > > should be different than quick-dev. > > > > Jon > > > > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler <ottobackwa...@gmail.com> > > wrote: > > > > > - create scripts/utilities to easily run a topology locally in an IDE > > > instead of in the VM > > > > > > > > > ^^^^^^^^ THIS. > > > > > > > > > On October 13, 2016 at 12:36:45, Ryan Merriman (merrim...@gmail.com) > > > wrote: > > > > > > Working with the quick-dev vagrant VM recently left a lot to be > desired. > > > All forthcoming comments are made under the assumption that this VM is > > > intended for development purposes. If that is not true, I think we > should > > > consider adding a VM for this purpose (or Docker containers?). Here are > > > the issues I ran into that I think can be improved: > > > > > > - had to upgrade VirtualBox from 5.0.16 to 5.0.20 > > > - had to update to the latest metron/hdp-base Vagrant box > > > - takes forever to spin up > > > - VM is constrained for resources making it unstable > > > - spent a large amount of time troubleshooting sensors (no raw messages > > > in Kafka) > > > - no easy way to debug topologies > > > > > > Fortunately I think we can make this a much better experience without a > > > major effort. Here are my ideas to do this: > > > > > > - update the prereqs for VirtualBox > > > - add a check for the appropriate base box version (Jira has already > > > been created https://issues.apache.org/jira/browse/METRON-497) > > > - don't install any sensors and replace them with a data generator that > > > just loops through sample data and emits to Kafka (could also be used > to > > > replay and troubleshoot edge cases) > > > - everything in monit is off by default except for ES or other critical > > > services > > > - create scripts/utilities to easily run a topology locally in an IDE > > > instead of in the VM > > > - improved documentation with examples of how to run and troubleshoot > > > topologies > > > > > > Is this a worthwhile effort? I think this would also give users an > easier > > > path to demonstrate or tour Metron's capabilities. Are there any other > > > improvements people would like to see? Should we wait for Docker? > > > Thoughts? > > > > > > Ryan Merriman > > > > > -- > > > > Jon > > > > > > -- > Nick Allen <n...@nickallen.org> >
