> Does the Ambari MPack provide the right level of ease of use for that?
In your example, Ambari is more like Homebrew and the MPack is more like a
way to extend Ambari (er. Homebrew).

So, yes, but we'd have to take on the effort to make it easy for the user.

-D...


On Fri, Feb 24, 2017 at 9:30 AM, Nick Allen <[email protected]> wrote:

> >
> >
> > we now have a robust
> > ​ ​
> > and flexible means to import enrichment sources and transform their
> > contents as they are inserted into HBase. One of the main motivators for
> > this new functionality was to add the ability to load top domain rankings
> > from sources such as Alexa. The proposal is to make this type of
> enrichment
> > a top-level feature in Metron by introducing it to the Ambari management
> UI
>
>
> (1) In thinking through how the UI should work here, we should consider
> data sources beyond just those that would be loaded in HBase.  I would
> think the UI should be a single view of all data sources, no matter whether
> they load into HBase or not.
>
> It would also be good to think through how the solution might handle
> updating other types of data source, like the geo data, for instance. The
> geo data is something that needs to be updated on a regular basis.  Could
> this solution also manage that?
>
> I know Maxmind has a bit of code to manage updating their data, but I am
> not familiar with what it does or how it works.  Researching that might
> help inform this conversation.
>
>
> > How do folks feel about adding a set of dropdown options in the Ambari UI
> > for loading, updating, and deleting the top domains enrichment?
>
>
> (2) I think if this functionality is truly useful, there is likely going to
> be lots of different data sources that would be made available.  Many of
> which will NOT be applicable or desirable in every environment.
>
> This would be akin to packages or RPMs that are available to install on
> CentOS.  There are many to choose from, but in my specific environment
> there are many that I do not care about.
>
> Is an Ambari drop down scalable considering this usage pattern?
>
> Do we want Ambari to handle only the
> > initial install/load and have end users be responsible on an ongoing
> basis
> > for updates (users would be responsible for copying or distributing the
> > extractor_config.json for instance), or do we want to enable Ambari to
> > manage the configuration ongoing and enable functionality for reloading,
> > updating, and rollback?
>
>
> (3) Whatever solution we land on, it should handle refreshing/reloading the
> data on a regular basis.  This is something that has to be done for almost
> every useful data source and so should be baked into the solution. I don't
> think the functionality is that useful otherwise.
>
> (4) Another thing to consider is extensibility and ease of use.  If we can
> make it really easy to provide a means for loading a data source into
> Metron, then it is more likely that we will have community members willing
> to do that work.
>
> For example, think about the Homebrew project.  They make it stupid simple
> to add a new installable package.  You don't have to know how Homebrew
> works to contribute a package.  The result is they have tons of packages
> available.
>
> Does the Ambari MPack provide the right level of ease of use for that?
>
>
>
>
>
> On Tue, Feb 21, 2017 at 6:31 PM, Michael Miklavcic <
> [email protected]> wrote:
>
> > With the work committed in
> > https://github.com/apache/incubator-metron/pull/445 and
> > https://github.com/apache/incubator-metron/pull/432, we now have a
> robust
> > and flexible means to import enrichment sources and transform their
> > contents as they are inserted into HBase. One of the main motivators for
> > this new functionality was to add the ability to load top domain rankings
> > from sources such as Alexa. The proposal is to make this type of
> enrichment
> > a top-level feature in Metron by introducing it to the Ambari management
> UI
> > as a configurable set of properties in the MPack install. This comes with
> > some options and challenges in how we want to manage the configurations,
> > which I will outline below.
> >
> > *Use cases:*
> >
> >    - Single load of top domains file
> >    - Re-loading top domains file - need to be able to cleanup properly
> >    - Cleaning up/deleting old enrichment data (this is a general feature
> >    that we currently lack - I think it is worth a separate Jira/PR for
> >    creating a MapReduce job that enables cleanup to occur).
> >    - Modifying default top domains file source - there are other options
> >    besides Alexa. And users may want to load a file from local URI since
> > many
> >    data centers do not have direct access to the internet.
> >    - Ability to modify the default extractor config JSON and tune the
> >    Stellar transformations for both the value and indicator transforms.
> > Allows
> >    more flexible handling of data based on other sources.
> >    - Loading multiple top domains source enrichments. (Maybe a separate
> PR
> >    for this if we even think it would be useful)
> >    - Updating the top domain enrichment - This needs to be an atomic
> >    operation in order to prevent incorrect data.
> >    - Rolling back to an older version of the top domains enrichment. Also
> >    needs to be atomic.
> >    - Ability to schedule an enrichment load on schedule - we would like
> to
> >    defer this to an external scheduling mechanism, e.g. cron or Control
> M.
> > The
> >    enrichment loading system should have the necessary features to enable
> > this
> >    type of automation without data integrity issues.
> >
> > *Considerations:*
> >
> >    - As mentioned above, we want to add this feature to the Ambari MPack.
> >    This requires at least 2 parameters to work. We need the ability to
> > specify
> >    a URI as well as an extractor config.
> >    - How do we want to manage the extractor config? The most obvious
> >    solution is to provide a text field in Ambari with a default JSON
> > config.
> >    When a load is initiated, Ambari would place a fresh copy of the
> > extractor
> >    config in the /tmp/ directory. This is an ephemeral file that isn't
> > needed
> >    other than during a load.
> >    - It seems easy enough to have the load occur during the initial
> >    install, however subsequent loads would require a different workflow.
> > How
> >    do folks feel about adding a set of dropdown options in the Ambari UI
> > for
> >    loading, updating, and deleting the top domains enrichment? I believe
> we
> >    are doing something similar for the ElasticSearch templates currently.
> >    - In the case of atomic operations for updates and rollbacks, I
> propose
> >    we add a property to Zookeeper that is reference-able in the
> enrichment
> >    itself. The idea would be to create a "top-domains" property in ZK
> that
> >    points to an enrichment key with a load timestamp associated with it,
> > e.g.
> >    top-domains_20170221042000. This would also allow a mapreduce job to
> be
> >    written that cleans up old enrichments. Another option is to create a
> > new
> >    table in HBase if/when you update the enrichment and change the
> > enrichment
> >    config manually. Deleting an old enrichment would simply be a matter
> of
> >    dropping the table in HBase. A relevant discussion of the tradeoffs of
> >    having many small tables versus 1 large table can be found here -
> >    http://grokbase.com/t/hbase/user/11bjbdw94q/multiple-
> > tables-vs-big-fat-table
> >    - In order to update or rollback an enrichment as mentioned above, we
> >    would also ideally provide a mechanism for changing the rowkey pointed
> > to
> >    by the enrichment.
> >
> > In summary of the use cases and considerations above, this boils down to
> > how we'd like to leverage Ambari here. Do we want Ambari to handle only
> the
> > initial install/load and have end users be responsible on an ongoing
> basis
> > for updates (users would be responsible for copying or distributing the
> > extractor_config.json for instance), or do we want to enable Ambari to
> > manage the configuration ongoing and enable functionality for reloading,
> > updating, and rollback?
> >
> > Best,
> > Mike
> >
>

Reply via email to