I would like to open a discussion to get the Batch Profiler feature branch merged into master as part of METRON-1699 [1] Create Batch Profiler. All of the work that I had in mind for our first draft of the Batch Profiler has been completed. Please take a look through what I have and let me know if there are other features that you think are required *before* we merge.
Previous list discussions on this topic include [2] and [3]. (Q) What can I do with the feature branch? * With the Batch Profiler, you can backfill/seed profiles using archived telemetry. This enables the following types of use cases. 1. As a Security Data Scientist, I want to understand the historical behaviors and trends of a profile that I have created so that I can determine if I have created a feature set that has predictive value for model building. 2. As a Security Data Scientist, I want to understand the historical behaviors and trends of a profile that I have created so that I can determine if I have defined the profile correctly and created a feature set that matches reality. 3. As a Security Platform Engineer, I want to generate a profile using archived telemetry when I deploy a new model to production so that models depending on that profile can function on day 1. * METRON-1699 [1] includes a more detailed description of the feature. (Q) What work was completed? * The Batch Profiler runs on Spark and was implemented in Java to remain consistent with our current Java-heavy code base. * The Batch Profiler is executed from the command-line. It can be launched using a script or by calling `spark-submit`, which may be useful for advanced users. * Input telemetry can be consumed from multiple sources; for example HDFS or the local file system. * Input telemetry can be consumed in multiple formats; for example JSON or ORC. * The 'output' profile measurements are persisted in HBase and is consistent with the Storm Profiler. * It can be run on any underlying engine supported by Spark. I have tested it both in 'local' mode and on a YARN cluster. * It is installed automatically by the Metron MPack. * A README was added that documents usage instructions. * The existing Profiler code was refactored so that as much code as possible is shared between the 3 Profiler ports; Storm, the Stellar REPL, and Spark. For example, the logic which determines the timestamp of a message was refactored so that it could be reused by all ports. * metron-profiler-common: The common Profiler code shared amongst each port. * metron-profiler-storm: Profiler on Storm * metron-profiler-spark: Profiler on Spark * metron-profiler-repl: Profiler on the Stellar REPL * metron-profiler-client: The client code for retrieving profile data; for example PROFILE_GET. * There are 3 separate RPM and DEB packages now created for the Profiler. * metron-profiler-storm-*.rpm * metron-profiler-spark-*.rpm * metron-profiler-repl-*.rpm * The Profiler integration tests were enhanced to leverage the Profiler Client logic to validate the results. * Review METRON-1699 [1] for a complete break-down of the tasks that have been completed on the feature branch. (Q) What limitations exist? * You must manually install Spark to use the Batch Profiler. The Metron MPack does not treat Spark as a Metron dependency and so does not install it automatically. * You do not configure the Batch Profiler in Ambari. It is configured and executed completely from the command-line. * To run the Batch Profiler in 'Full Dev', you have to take the following manual steps. Some of these are arguably limitations with how Ambari installs Spark 2 in the version of HDP that we run. 1. Install Spark 2 using Ambari. 2. Tell Spark how to talk with HBase. SPARK_HOME=/usr/hdp/current/spark2-client cp /usr/hdp/current/hbase-client/conf/hbase-site.xml $SPARK_HOME/conf/ 3. Create the Spark History directory in HDFS. export HADOOP_USER_NAME=hdfs hdfs dfs -mkdir /spark2-history 4. Change the default input path to `hdfs://localhost:8020/...` to match the port defined by HDP, instead of port 9000. [1] https://issues.apache.org/jira/browse/METRON-1699 [2] https://lists.apache.org/thread.html/da81c1227ffda3a47eb2e5bb4d0b162dd6d36006241c4ba4b659587b@%3Cdev.metron.apache.org%3E [3] https://lists.apache.org/thread.html/d28d18cc9358f5d9c276c7c304ff4ee601041fb47bfc97acb6825083@%3Cdev.metron.apache.org%3E