With the work committed in
https://github.com/apache/incubator-metron/pull/445 and
https://github.com/apache/incubator-metron/pull/432, we now have a robust
and flexible means to import enrichment sources and transform their
contents as they are inserted into HBase. One of the main motivators for
this new functionality was to add the ability to load top domain rankings
from sources such as Alexa. The proposal is to make this type of enrichment
a top-level feature in Metron by introducing it to the Ambari management UI
as a configurable set of properties in the MPack install. This comes with
some options and challenges in how we want to manage the configurations,
which I will outline below.

*Use cases:*

   - Single load of top domains file
   - Re-loading top domains file - need to be able to cleanup properly
   - Cleaning up/deleting old enrichment data (this is a general feature
   that we currently lack - I think it is worth a separate Jira/PR for
   creating a MapReduce job that enables cleanup to occur).
   - Modifying default top domains file source - there are other options
   besides Alexa. And users may want to load a file from local URI since many
   data centers do not have direct access to the internet.
   - Ability to modify the default extractor config JSON and tune the
   Stellar transformations for both the value and indicator transforms. Allows
   more flexible handling of data based on other sources.
   - Loading multiple top domains source enrichments. (Maybe a separate PR
   for this if we even think it would be useful)
   - Updating the top domain enrichment - This needs to be an atomic
   operation in order to prevent incorrect data.
   - Rolling back to an older version of the top domains enrichment. Also
   needs to be atomic.
   - Ability to schedule an enrichment load on schedule - we would like to
   defer this to an external scheduling mechanism, e.g. cron or Control M. The
   enrichment loading system should have the necessary features to enable this
   type of automation without data integrity issues.

*Considerations:*

   - As mentioned above, we want to add this feature to the Ambari MPack.
   This requires at least 2 parameters to work. We need the ability to specify
   a URI as well as an extractor config.
   - How do we want to manage the extractor config? The most obvious
   solution is to provide a text field in Ambari with a default JSON config.
   When a load is initiated, Ambari would place a fresh copy of the extractor
   config in the /tmp/ directory. This is an ephemeral file that isn't needed
   other than during a load.
   - It seems easy enough to have the load occur during the initial
   install, however subsequent loads would require a different workflow. How
   do folks feel about adding a set of dropdown options in the Ambari UI for
   loading, updating, and deleting the top domains enrichment? I believe we
   are doing something similar for the ElasticSearch templates currently.
   - In the case of atomic operations for updates and rollbacks, I propose
   we add a property to Zookeeper that is reference-able in the enrichment
   itself. The idea would be to create a "top-domains" property in ZK that
   points to an enrichment key with a load timestamp associated with it, e.g.
   top-domains_20170221042000. This would also allow a mapreduce job to be
   written that cleans up old enrichments. Another option is to create a new
   table in HBase if/when you update the enrichment and change the enrichment
   config manually. Deleting an old enrichment would simply be a matter of
   dropping the table in HBase. A relevant discussion of the tradeoffs of
   having many small tables versus 1 large table can be found here -
   http://grokbase.com/t/hbase/user/11bjbdw94q/multiple-tables-vs-big-fat-table
   - In order to update or rollback an enrichment as mentioned above, we
   would also ideally provide a mechanism for changing the rowkey pointed to
   by the enrichment.

In summary of the use cases and considerations above, this boils down to
how we'd like to leverage Ambari here. Do we want Ambari to handle only the
initial install/load and have end users be responsible on an ongoing basis
for updates (users would be responsible for copying or distributing the
extractor_config.json for instance), or do we want to enable Ambari to
manage the configuration ongoing and enable functionality for reloading,
updating, and rollback?

Best,
Mike

Reply via email to