Really looking forward to dockerized Metron. Thanks ahead. On Wednesday, 19 October 2016, Kyle Richardson <kylerichards...@gmail.com> wrote:
> Sorry, I'm a bit late to the party on this one :). > > +1 on the four builds Nick described. Each would be useful and purpose > built. > > I like the idea of using Docker, especially for local development and quick > testing. Has anyone explored this? I'm envisioning very specific containers > so you could spin up only the components you're actively working on. > Certainly something I would be willing to put some cycles into if there is > interest. > > -Kyle > > On Fri, Oct 14, 2016 at 2:15 PM, Ryan Merriman <merrim...@gmail.com > <javascript:;>> wrote: > > > +1 I like it. Just to clarify, the scripts to run Storm topologies > locally > > in an IDE should be available independent of the environment running. No > > need for a separate build/image. > > > > On Fri, Oct 14, 2016 at 9:12 AM, Otto Fowler <ottobackwa...@gmail.com > <javascript:;>> > > wrote: > > > > > Going forward, the Demo env and data would have implications for > testing > > as > > > well ( gold data sets ) etc. > > > > > > On October 14, 2016 at 09:52:07, Nick Allen (n...@nickallen.org > <javascript:;>) wrote: > > > > > > I think based on everyone's input so far, we're describing 4 different > > > builds/images/tools that would each be intended to run on a standard > > > Mac/Linux/Windows laptop. > > > > > > Full Dev - A development environment that performs a full end-to-end > > > deployment of Metron. This is intended for developers working with > > > sensors, deployments, or validating how all Metron components interact > > with > > > one another. > > > > > > > > > - Starts from base Linux image > > > - Installs Hadoop-y components > > > - Installs Metron > > > - Installs Sensors > > > - Nothing started by default > > > > > > Quick Dev - An environment intended for the developer focusing on the > > > streaming components of Metron; parsing, enrichment, and indexing. > > > > > > > > > - Starts from base image of Linux + Hadoop-y components > > > - Installs Metron > > > - Installs "data generator" spouts > > > - Does not install sensors > > > - Nothing started by default > > > > > > Demo - An environment intended to introduce new users to Metron. The > > > environment should go from nothing to plenty of data in the Metron > > > dashboard in as little "boot" time as possible. > > > > > > > > > - Starts from a base image including Linux + Hadoop-y + Metron + Data > > > Generator Spouts pre-installed > > > - Pre-load Elasticsearch indices so the user has plenty of data to > > > investigate in the dashboard > > > - Does not install sensors > > > - Everything started by default > > > > > > Storm Local Cluster - Otto suggested some scripts/tooling to make it > easy > > > to launch the core topologies on a local Storm cluster running on the > > host > > > OS. > > > > > > > > > I'd be interested to hear if this works for everyone and how this might > > > play into the Ambari mpack + RPM based deployment scheme. > > > > > > > > > On Fri, Oct 14, 2016 at 1:45 AM, Michael Miklavcic < > > > michael.miklav...@gmail.com <javascript:;>> wrote: > > > > > > > I think this may have come up in another PR already (have to look for > > > it). > > > > But maybe we could maintain our flexibility in quick-dev by > installing > > > the > > > > sensors and not starting them until we need them. I think it's useful > > to > > > > have a quick "genuine" e2e testing environment that doesn't require > > > running > > > > through a full install. I'm also not opposed to extracting the > > > integration > > > > test functionality into general purpose data generators. > > > > > > > > On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen <n...@nickallen.org > <javascript:;>> > > wrote: > > > > > > > > > To Jon's point, I think it would be useful to have a Demo box that > > uses > > > > > generators to produce 3 or 4 types of telemetry that shows up in > the > > > > Metron > > > > > Dashboard. This box would be different from Quick-Dev in that > > > everything > > > > > starts automatically, so that a user just has to launch it and the > > > should > > > > > start seeing data in the Metron Dashboard right away. In fact, we > > could > > > > > even pre-load the Elasticsearch indices so that the user has more > of > > a > > > > > history to mine when using the Demo box. > > > > > > > > > > On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com < > zeo...@gmail.com <javascript:;>> > > > > > wrote: > > > > > > > > > > > +1 Ryan and Otto's comments. > > > > > > > > > > > > I also strongly think we need to make a demo environment easier, > > but > > > > that > > > > > > should be different than quick-dev. > > > > > > > > > > > > Jon > > > > > > > > > > > > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler < > > ottobackwa...@gmail.com <javascript:;> > > > > > > > > > > wrote: > > > > > > > > > > > > > - create scripts/utilities to easily run a topology locally in > an > > > IDE > > > > > > > instead of in the VM > > > > > > > > > > > > > > > > > > > > > ^^^^^^^^ THIS. > > > > > > > > > > > > > > > > > > > > > On October 13, 2016 at 12:36:45, Ryan Merriman ( > > > merrim...@gmail.com <javascript:;>) > > > > > > > > > > wrote: > > > > > > > > > > > > > > Working with the quick-dev vagrant VM recently left a lot to be > > > > > desired. > > > > > > > All forthcoming comments are made under the assumption that > this > > VM > > > > is > > > > > > > intended for development purposes. If that is not true, I think > > we > > > > > should > > > > > > > consider adding a VM for this purpose (or Docker containers?). > > Here > > > > are > > > > > > > the issues I ran into that I think can be improved: > > > > > > > > > > > > > > - had to upgrade VirtualBox from 5.0.16 to 5.0.20 > > > > > > > - had to update to the latest metron/hdp-base Vagrant box > > > > > > > - takes forever to spin up > > > > > > > - VM is constrained for resources making it unstable > > > > > > > - spent a large amount of time troubleshooting sensors (no raw > > > > messages > > > > > > > in Kafka) > > > > > > > - no easy way to debug topologies > > > > > > > > > > > > > > Fortunately I think we can make this a much better experience > > > > without a > > > > > > > major effort. Here are my ideas to do this: > > > > > > > > > > > > > > - update the prereqs for VirtualBox > > > > > > > - add a check for the appropriate base box version (Jira has > > > already > > > > > > > been created https://issues.apache.org/jira/browse/METRON-497) > > > > > > > - don't install any sensors and replace them with a data > > generator > > > > that > > > > > > > just loops through sample data and emits to Kafka (could also > be > > > used > > > > > to > > > > > > > replay and troubleshoot edge cases) > > > > > > > - everything in monit is off by default except for ES or other > > > > critical > > > > > > > services > > > > > > > - create scripts/utilities to easily run a topology locally in > an > > > IDE > > > > > > > instead of in the VM > > > > > > > - improved documentation with examples of how to run and > > > troubleshoot > > > > > > > topologies > > > > > > > > > > > > > > Is this a worthwhile effort? I think this would also give users > > an > > > > > easier > > > > > > > path to demonstrate or tour Metron's capabilities. Are there > any > > > > other > > > > > > > improvements people would like to see? Should we wait for > Docker? > > > > > > > Thoughts? > > > > > > > > > > > > > > Ryan Merriman > > > > > > > > > > > > > -- > > > > > > > > > > > > Jon > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Nick Allen <n...@nickallen.org <javascript:;>> > > > > > > > > > > > > > > > > > > > > > -- > > > Nick Allen <n...@nickallen.org <javascript:;>> > > > > > >
