[GitHub] metron pull request #1187: METRON-1717 Relocate Storm Profiler Code [Feature...
Github user nickwallen closed the pull request at: https://github.com/apache/metron/pull/1187 ---
[GitHub] metron pull request #1187: METRON-1717 Relocate Storm Profiler Code [Feature...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/1187#discussion_r216417073 --- Diff: metron-analytics/metron-profiler-common/README.md --- @@ -0,0 +1,386 @@ + +# Metron Profiler + +* [Introduction](#introduction) +* [Getting Started](#getting-started) +* [Profiles](#profiles) +* [Examples](#examples) + +## Introduction + +The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior. + +This is achieved by summarizing the telemetry data consumed by Metron over tumbling windows. A summary statistic is applied to the data received within a given window. Collecting these values across many windows result in a time series that is useful for analysis. + +Any field contained within a message can be used to generate a profile. A profile can even be produced by combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language. + +There are three separate ports of the Profiler that share this common code base. +* The [Storm Profiler](../metron-profiler-storm/README.md) builds low-latency profiles over streaming data sets. +* The [Spark Profiler](../metron-profiler-spark/README.md) backfills profiles using archived telemetry. +* The [REPL Profiler](../metron-profiler-repl/README.md) allows profiles to be tested and debugged within the Stellar REPL. + +## Getting Started + +1. [Create a profile](../metron-profiler-repl/README.md#getting-started) using the Stellar REPL. Validate your profile using mock data, then apply real, live data. + +1. [Backfill your profile](../metron-profiler-spark/README.md#getting-started) using archived telemetry to see how your profile behaves over time. + +1. [Deploy your profile](../metron-profiler-storm/README.md#getting-started) to Storm to maintain a low-latency profile over a streaming data set. + +1. [Retrieve your profile data](../metron-profiler-client/README.md) using the Stellar API so that you can build enrichments, alert on abnormalities + +1. Explore more ways to create [profiles](#more-examples). + +## Profiles + +Let's start with a simple example. The following profile maintains a count of the number of telemetry messages for each IP source address. A counter is initialized to 0, then incremented each time a message is received for a give IP source address. At regular intervals the count is flushed and stored. Over time this results in a time series describing the amount of telemetry received for each IP source address. --- End diff -- give > given ---
[GitHub] metron pull request #1187: METRON-1717 Relocate Storm Profiler Code [Feature...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/1187#discussion_r216416991 --- Diff: metron-analytics/metron-profiler-common/README.md --- @@ -0,0 +1,386 @@ + +# Metron Profiler + +* [Introduction](#introduction) +* [Getting Started](#getting-started) +* [Profiles](#profiles) +* [Examples](#examples) + +## Introduction + +The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior. + +This is achieved by summarizing the telemetry data consumed by Metron over tumbling windows. A summary statistic is applied to the data received within a given window. Collecting these values across many windows result in a time series that is useful for analysis. + +Any field contained within a message can be used to generate a profile. A profile can even be produced by combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language. + +There are three separate ports of the Profiler that share this common code base. +* The [Storm Profiler](../metron-profiler-storm/README.md) builds low-latency profiles over streaming data sets. +* The [Spark Profiler](../metron-profiler-spark/README.md) backfills profiles using archived telemetry. +* The [REPL Profiler](../metron-profiler-repl/README.md) allows profiles to be tested and debugged within the Stellar REPL. + +## Getting Started + +1. [Create a profile](../metron-profiler-repl/README.md#getting-started) using the Stellar REPL. Validate your profile using mock data, then apply real, live data. + +1. [Backfill your profile](../metron-profiler-spark/README.md#getting-started) using archived telemetry to see how your profile behaves over time. + +1. [Deploy your profile](../metron-profiler-storm/README.md#getting-started) to Storm to maintain a low-latency profile over a streaming data set. + +1. [Retrieve your profile data](../metron-profiler-client/README.md) using the Stellar API so that you can build enrichments, alert on abnormalities --- End diff -- period at the end ---
[GitHub] metron pull request #1187: METRON-1717 Relocate Storm Profiler Code
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1187 METRON-1717 Relocate Storm Profiler Code - This change moves the Storm Profiler module to `metron-analytics/metron-profiler-storm`. - The core Storm Profiler package was renamed to `org.apache.metron.profiler.storm`. - All of the Profiler READMEs have been cleaned-up to contain only content relevant to each project. The main README is now in `metron-analytics/metron-profiler-common` which links to the others as needed. ## Testing 1. Stand-up a development environment. 1. Validate the development environment by ensuring alerts are visible within the Alerts UI and that the Metron Service Check in Ambari passes. 1. Launch the REPL and follow the instructions in the Profiler README to [create and execute a profile in the REPL](https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles). 1. Follow the instructions in the README to[ deploy the same profile in Storm](https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler-storm#getting-started). Ensure that you can retrieve values from HBase using `PROFILE_GET`. 1. Generate the site book and review the READMEs that have changed. ``` cd site-book mvn site ``` ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1717 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1187.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1187 commit 290bc793a4cecb1c7c83ef4cfb77f67f5ef7dbbe Author: Nick Allen Date: 2018-09-05T16:12:51Z METRON-1717 Renamed Storm Profiler package commit b88c0e72974480750255d6e64faed24cf876527b Author: Nick Allen Date: 2018-09-05T17:15:46Z Rename package to org.apache.metron.profiler.storm commit 27e69d41c2e8a982dca23dfc6feca737b0e48c12 Author: Nick Allen Date: 2018-09-05T20:36:26Z Updated READMEs ---
