I really like the idea to replace real sensors in Quick-Dev with data generators; aka spouts that spit-out canned data. The pcap replay mechanism is fairly resource intensive, not to mention all of the sensors like Bro, Snort, etc. Removing these should give us significantly more head room.
At the same time, it is going to be useful in some cases to have a single node that can run end-to-end with sensors. For example, I still want to have a box that I can drop in different pcap files and capture telemetry for testing. I think we should continue to maintain Full-Dev with the current sensor suite. In addition, having these data generators can help with performance testing, if we add the right toggles and switches. Like the ability to control the throughput, etc. On Thu, Oct 13, 2016 at 12:36 PM, Ryan Merriman <[email protected]> wrote: > Working with the quick-dev vagrant VM recently left a lot to be desired. > All forthcoming comments are made under the assumption that this VM is > intended for development purposes. If that is not true, I think we should > consider adding a VM for this purpose (or Docker containers?). Here are > the issues I ran into that I think can be improved: > > - had to upgrade VirtualBox from 5.0.16 to 5.0.20 > - had to update to the latest metron/hdp-base Vagrant box > - takes forever to spin up > - VM is constrained for resources making it unstable > - spent a large amount of time troubleshooting sensors (no raw messages > in Kafka) > - no easy way to debug topologies > > Fortunately I think we can make this a much better experience without a > major effort. Here are my ideas to do this: > > - update the prereqs for VirtualBox > - add a check for the appropriate base box version (Jira has already > been created https://issues.apache.org/jira/browse/METRON-497) > - don't install any sensors and replace them with a data generator that > just loops through sample data and emits to Kafka (could also be used to > replay and troubleshoot edge cases) > - everything in monit is off by default except for ES or other critical > services > - create scripts/utilities to easily run a topology locally in an IDE > instead of in the VM > - improved documentation with examples of how to run and troubleshoot > topologies > > Is this a worthwhile effort? I think this would also give users an easier > path to demonstrate or tour Metron's capabilities. Are there any other > improvements people would like to see? Should we wait for Docker? > Thoughts? > > Ryan Merriman > -- Nick Allen <[email protected]>
