Going forward, the Demo env and data would have implications for testing as well ( gold data sets ) etc.
On October 14, 2016 at 09:52:07, Nick Allen (n...@nickallen.org) wrote: I think based on everyone's input so far, we're describing 4 different builds/images/tools that would each be intended to run on a standard Mac/Linux/Windows laptop. Full Dev - A development environment that performs a full end-to-end deployment of Metron. This is intended for developers working with sensors, deployments, or validating how all Metron components interact with one another. - Starts from base Linux image - Installs Hadoop-y components - Installs Metron - Installs Sensors - Nothing started by default Quick Dev - An environment intended for the developer focusing on the streaming components of Metron; parsing, enrichment, and indexing. - Starts from base image of Linux + Hadoop-y components - Installs Metron - Installs "data generator" spouts - Does not install sensors - Nothing started by default Demo - An environment intended to introduce new users to Metron. The environment should go from nothing to plenty of data in the Metron dashboard in as little "boot" time as possible. - Starts from a base image including Linux + Hadoop-y + Metron + Data Generator Spouts pre-installed - Pre-load Elasticsearch indices so the user has plenty of data to investigate in the dashboard - Does not install sensors - Everything started by default Storm Local Cluster - Otto suggested some scripts/tooling to make it easy to launch the core topologies on a local Storm cluster running on the host OS. I'd be interested to hear if this works for everyone and how this might play into the Ambari mpack + RPM based deployment scheme. On Fri, Oct 14, 2016 at 1:45 AM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I think this may have come up in another PR already (have to look for it). > But maybe we could maintain our flexibility in quick-dev by installing the > sensors and not starting them until we need them. I think it's useful to > have a quick "genuine" e2e testing environment that doesn't require running > through a full install. I'm also not opposed to extracting the integration > test functionality into general purpose data generators. > > On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen <n...@nickallen.org> wrote: > > > To Jon's point, I think it would be useful to have a Demo box that uses > > generators to produce 3 or 4 types of telemetry that shows up in the > Metron > > Dashboard. This box would be different from Quick-Dev in that everything > > starts automatically, so that a user just has to launch it and the should > > start seeing data in the Metron Dashboard right away. In fact, we could > > even pre-load the Elasticsearch indices so that the user has more of a > > history to mine when using the Demo box. > > > > On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com <zeo...@gmail.com> > > wrote: > > > > > +1 Ryan and Otto's comments. > > > > > > I also strongly think we need to make a demo environment easier, but > that > > > should be different than quick-dev. > > > > > > Jon > > > > > > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler <ottobackwa...@gmail.com> > > > wrote: > > > > > > > - create scripts/utilities to easily run a topology locally in an IDE > > > > instead of in the VM > > > > > > > > > > > > ^^^^^^^^ THIS. > > > > > > > > > > > > On October 13, 2016 at 12:36:45, Ryan Merriman (merrim...@gmail.com) > > > > wrote: > > > > > > > > Working with the quick-dev vagrant VM recently left a lot to be > > desired. > > > > All forthcoming comments are made under the assumption that this VM > is > > > > intended for development purposes. If that is not true, I think we > > should > > > > consider adding a VM for this purpose (or Docker containers?). Here > are > > > > the issues I ran into that I think can be improved: > > > > > > > > - had to upgrade VirtualBox from 5.0.16 to 5.0.20 > > > > - had to update to the latest metron/hdp-base Vagrant box > > > > - takes forever to spin up > > > > - VM is constrained for resources making it unstable > > > > - spent a large amount of time troubleshooting sensors (no raw > messages > > > > in Kafka) > > > > - no easy way to debug topologies > > > > > > > > Fortunately I think we can make this a much better experience > without a > > > > major effort. Here are my ideas to do this: > > > > > > > > - update the prereqs for VirtualBox > > > > - add a check for the appropriate base box version (Jira has already > > > > been created https://issues.apache.org/jira/browse/METRON-497) > > > > - don't install any sensors and replace them with a data generator > that > > > > just loops through sample data and emits to Kafka (could also be used > > to > > > > replay and troubleshoot edge cases) > > > > - everything in monit is off by default except for ES or other > critical > > > > services > > > > - create scripts/utilities to easily run a topology locally in an IDE > > > > instead of in the VM > > > > - improved documentation with examples of how to run and troubleshoot > > > > topologies > > > > > > > > Is this a worthwhile effort? I think this would also give users an > > easier > > > > path to demonstrate or tour Metron's capabilities. Are there any > other > > > > improvements people would like to see? Should we wait for Docker? > > > > Thoughts? > > > > > > > > Ryan Merriman > > > > > > > -- > > > > > > Jon > > > > > > > > > > > -- > > Nick Allen <n...@nickallen.org> > > > -- Nick Allen <n...@nickallen.org>
