[
https://issues.apache.org/jira/browse/METRON-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609602#comment-16609602
]
ASF GitHub Bot commented on METRON-1717:
----------------------------------------
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1187#discussion_r216417073
--- Diff: metron-analytics/metron-profiler-common/README.md ---
@@ -0,0 +1,386 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Metron Profiler
+
+* [Introduction](#introduction)
+* [Getting Started](#getting-started)
+* [Profiles](#profiles)
+* [Examples](#examples)
+
+## Introduction
+
+The Profiler is a feature extraction mechanism that can generate a profile
describing the behavior of an entity. An entity might be a server, user,
subnet or application. Once a profile has been generated defining what normal
behavior looks-like, models can be built that identify anomalous behavior.
+
+This is achieved by summarizing the telemetry data consumed by Metron over
tumbling windows. A summary statistic is applied to the data received within a
given window. Collecting these values across many windows result in a time
series that is useful for analysis.
+
+Any field contained within a message can be used to generate a profile. A
profile can even be produced by combining fields that originate in different
data sources. A user has considerable power to transform the data used in a
profile by leveraging the Stellar language.
+
+There are three separate ports of the Profiler that share this common code
base.
+* The [Storm Profiler](../metron-profiler-storm/README.md) builds
low-latency profiles over streaming data sets.
+* The [Spark Profiler](../metron-profiler-spark/README.md) backfills
profiles using archived telemetry.
+* The [REPL Profiler](../metron-profiler-repl/README.md) allows profiles
to be tested and debugged within the Stellar REPL.
+
+## Getting Started
+
+1. [Create a profile](../metron-profiler-repl/README.md#getting-started)
using the Stellar REPL. Validate your profile using mock data, then apply real,
live data.
+
+1. [Backfill your
profile](../metron-profiler-spark/README.md#getting-started) using archived
telemetry to see how your profile behaves over time.
+
+1. [Deploy your
profile](../metron-profiler-storm/README.md#getting-started) to Storm to
maintain a low-latency profile over a streaming data set.
+
+1. [Retrieve your profile data](../metron-profiler-client/README.md) using
the Stellar API so that you can build enrichments, alert on abnormalities
+
+1. Explore more ways to create [profiles](#more-examples).
+
+## Profiles
+
+Let's start with a simple example. The following profile maintains a count
of the number of telemetry messages for each IP source address. A counter is
initialized to 0, then incremented each time a message is received for a give
IP source address. At regular intervals the count is flushed and stored. Over
time this results in a time series describing the amount of telemetry received
for each IP source address.
--- End diff --
give > given
> Relocate Storm Profiler Code
> ----------------------------
>
> Key: METRON-1717
> URL: https://issues.apache.org/jira/browse/METRON-1717
> Project: Metron
> Issue Type: Sub-task
> Reporter: Nick Allen
> Assignee: Nick Allen
> Priority: Major
>
> The Storm port of the Profiler currently lives in
> metron-analytics/metron-profiler. This should be moved to
> metron-analytics/metron-profiler-storm. This would mirror the project names
> for the Spark port (metron-profiler-spark) and the REPL port
> (metron-profiler-repl).
> The package name for the Storm port of the Profiler should be changed to
> org.apache.metron.profiler.storm. This would mimic the package name used for
> Spark; org.apache.metron.profiler.spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
