I don't really have a firm opinion either way except for the normal
default- if Ambari is going to own a part of it, it needs to maintain the
full (load/update/upgrade/remove) lifecycle.

So, it sounds like you've already decided you want Ambari to do the initial
install and load. The consequence of that is that Ambari should expose the
rest.

-D...


On Tue, Feb 21, 2017 at 6:31 PM, Michael Miklavcic <
[email protected]> wrote:

> With the work committed in
> https://github.com/apache/incubator-metron/pull/445 and
> https://github.com/apache/incubator-metron/pull/432, we now have a robust
> and flexible means to import enrichment sources and transform their
> contents as they are inserted into HBase. One of the main motivators for
> this new functionality was to add the ability to load top domain rankings
> from sources such as Alexa. The proposal is to make this type of enrichment
> a top-level feature in Metron by introducing it to the Ambari management UI
> as a configurable set of properties in the MPack install. This comes with
> some options and challenges in how we want to manage the configurations,
> which I will outline below.
>
> *Use cases:*
>
>    - Single load of top domains file
>    - Re-loading top domains file - need to be able to cleanup properly
>    - Cleaning up/deleting old enrichment data (this is a general feature
>    that we currently lack - I think it is worth a separate Jira/PR for
>    creating a MapReduce job that enables cleanup to occur).
>    - Modifying default top domains file source - there are other options
>    besides Alexa. And users may want to load a file from local URI since
> many
>    data centers do not have direct access to the internet.
>    - Ability to modify the default extractor config JSON and tune the
>    Stellar transformations for both the value and indicator transforms.
> Allows
>    more flexible handling of data based on other sources.
>    - Loading multiple top domains source enrichments. (Maybe a separate PR
>    for this if we even think it would be useful)
>    - Updating the top domain enrichment - This needs to be an atomic
>    operation in order to prevent incorrect data.
>    - Rolling back to an older version of the top domains enrichment. Also
>    needs to be atomic.
>    - Ability to schedule an enrichment load on schedule - we would like to
>    defer this to an external scheduling mechanism, e.g. cron or Control M.
> The
>    enrichment loading system should have the necessary features to enable
> this
>    type of automation without data integrity issues.
>
> *Considerations:*
>
>    - As mentioned above, we want to add this feature to the Ambari MPack.
>    This requires at least 2 parameters to work. We need the ability to
> specify
>    a URI as well as an extractor config.
>    - How do we want to manage the extractor config? The most obvious
>    solution is to provide a text field in Ambari with a default JSON
> config.
>    When a load is initiated, Ambari would place a fresh copy of the
> extractor
>    config in the /tmp/ directory. This is an ephemeral file that isn't
> needed
>    other than during a load.
>    - It seems easy enough to have the load occur during the initial
>    install, however subsequent loads would require a different workflow.
> How
>    do folks feel about adding a set of dropdown options in the Ambari UI
> for
>    loading, updating, and deleting the top domains enrichment? I believe we
>    are doing something similar for the ElasticSearch templates currently.
>    - In the case of atomic operations for updates and rollbacks, I propose
>    we add a property to Zookeeper that is reference-able in the enrichment
>    itself. The idea would be to create a "top-domains" property in ZK that
>    points to an enrichment key with a load timestamp associated with it,
> e.g.
>    top-domains_20170221042000. This would also allow a mapreduce job to be
>    written that cleans up old enrichments. Another option is to create a
> new
>    table in HBase if/when you update the enrichment and change the
> enrichment
>    config manually. Deleting an old enrichment would simply be a matter of
>    dropping the table in HBase. A relevant discussion of the tradeoffs of
>    having many small tables versus 1 large table can be found here -
>    http://grokbase.com/t/hbase/user/11bjbdw94q/multiple-
> tables-vs-big-fat-table
>    - In order to update or rollback an enrichment as mentioned above, we
>    would also ideally provide a mechanism for changing the rowkey pointed
> to
>    by the enrichment.
>
> In summary of the use cases and considerations above, this boils down to
> how we'd like to leverage Ambari here. Do we want Ambari to handle only the
> initial install/load and have end users be responsible on an ongoing basis
> for updates (users would be responsible for copying or distributing the
> extractor_config.json for instance), or do we want to enable Ambari to
> manage the configuration ongoing and enable functionality for reloading,
> updating, and rollback?
>
> Best,
> Mike
>

Reply via email to