http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-analytics/metron-profiler-client/index.html
----------------------------------------------------------------------
diff --git
a/site/current-book/metron-analytics/metron-profiler-client/index.html
b/site/current-book/metron-analytics/metron-profiler-client/index.html
index 4fc31d6..d05835a 100644
--- a/site/current-book/metron-analytics/metron-profiler-client/index.html
+++ b/site/current-book/metron-analytics/metron-profiler-client/index.html
@@ -1,233 +1,113 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2018-01-03
- | Rendered using Apache Maven Fluido Skin 1.3.0
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from
src/site/markdown/metron-analytics/metron-profiler-client/index.md at 2018-06-07
+ | Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180103" />
+ <meta name="Date-Revision-yyyymmdd" content="20180607" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Profiler Client</title>
- <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css"
/>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
<link rel="stylesheet" href="../../css/site.css" />
<link rel="stylesheet" href="../../css/print.css" media="print" />
-
-
- <script type="text/javascript"
src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
-
-
-
-<script type="text/javascript">$( document ).ready( function() { $(
'.carousel' ).carousel( { interval: 3500 } ) } );</script>
-
- </head>
- <body class="topBarDisabled">
-
-
-
-
- <div class="container-fluid">
- <div id="banner">
- <div class="pull-left">
- <a href="http://metron.apache.org/";
id="bannerLeft">
-
<img src="../../images/metron-logo.png" alt="Apache Metron"
width="148px" height="48px"/>
- </a>
- </div>
- <div class="pull-right"> </div>
+ <script type="text/javascript"
src="../../js/apache-maven-fluido-1.7.min.js"></script>
+<script type="text/javascript">
+ $( document ).ready( function() { $( '.carousel' ).carousel( {
interval: 3500 } ) } );
+ </script>
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href="http://metron.apache.org/";
id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron"
width="148px" height="48px"/></a></div>
+ <div class="pull-right"></div>
<div class="clear"><hr/></div>
</div>
<div id="breadcrumbs">
<ul class="breadcrumb">
-
-
- <li class="">
- <a href="http://www.apache.org"; class="externalLink"
title="Apache">
- Apache</a>
- </li>
- <li class="divider ">/</li>
- <li class="">
- <a href="http://metron.apache.org/"; class="externalLink"
title="Metron">
- Metron</a>
- </li>
- <li class="divider ">/</li>
- <li class="">
- <a href="../../index.html" title="Documentation">
- Documentation</a>
- </li>
- <li class="divider ">/</li>
- <li class="">Metron Profiler Client</li>
-
-
-
- <li id="publishDate" class="pull-right">Last Published:
2018-01-03</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
-
- </ul>
+ <li class=""><a href="http://www.apache.org"; class="externalLink"
title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="http://metron.apache.org/"; class="externalLink"
title="Metron">Metron</a><span class="divider">/</span></li>
+ <li class=""><a href="../../index.html"
title="Documentation">Documentation</a><span class="divider">/</span></li>
+ <li class="active ">Metron Profiler Client</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span>
Last Published: 2018-06-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.5.0</li>
+ </ul>
</div>
-
-
<div class="row-fluid">
- <div id="leftColumn" class="span3">
+ <div id="leftColumn" class="span2">
<div class="well sidebar-nav">
-
-
- <ul class="nav nav-list">
- <li class="nav-header">User Documentation</li>
-
- <li>
-
- <a href="../../index.html" title="Metron">
- <i class="icon-chevron-down"></i>
- Metron</a>
- <ul class="nav nav-list">
-
- <li>
-
- <a href="../../Upgrading.html" title="Upgrading">
- <i class="none"></i>
- Upgrading</a>
- </li>
-
- <li>
-
- <a href="../../metron-analytics/index.html"
title="Analytics">
- <i class="icon-chevron-down"></i>
- Analytics</a>
- <ul class="nav nav-list">
-
- <li>
-
- <a
href="../../metron-analytics/metron-maas-service/index.html"
title="Maas-service">
- <i class="none"></i>
- Maas-service</a>
- </li>
-
- <li>
-
- <a
href="../../metron-analytics/metron-profiler/index.html" title="Profiler">
- <i class="none"></i>
- Profiler</a>
- </li>
-
- <li class="active">
-
- <a href="#"><i class="none"></i>Profiler-client</a>
- </li>
-
- <li>
-
- <a
href="../../metron-analytics/metron-statistics/index.html" title="Statistics">
- <i class="icon-chevron-right"></i>
- Statistics</a>
- </li>
- </ul>
- </li>
-
- <li>
-
- <a
href="../../metron-contrib/metron-docker/index.html" title="Docker">
- <i class="none"></i>
- Docker</a>
- </li>
-
- <li>
-
- <a href="../../metron-deployment/index.html"
title="Deployment">
- <i class="icon-chevron-right"></i>
- Deployment</a>
- </li>
-
- <li>
-
- <a
href="../../metron-interface/metron-alerts/index.html" title="Alerts">
- <i class="none"></i>
- Alerts</a>
- </li>
-
- <li>
-
- <a
href="../../metron-interface/metron-config/index.html" title="Config">
- <i class="none"></i>
- Config</a>
- </li>
-
- <li>
-
- <a
href="../../metron-interface/metron-rest/index.html" title="Rest">
- <i class="none"></i>
- Rest</a>
- </li>
-
- <li>
-
- <a href="../../metron-platform/index.html"
title="Platform">
- <i class="icon-chevron-right"></i>
- Platform</a>
- </li>
-
- <li>
-
- <a href="../../metron-sensors/index.html"
title="Sensors">
- <i class="icon-chevron-right"></i>
- Sensors</a>
- </li>
-
- <li>
-
- <a
href="../../metron-stellar/stellar-3rd-party-example/index.html"
title="Stellar-3rd-party-example">
- <i class="none"></i>
- Stellar-3rd-party-example</a>
- </li>
-
- <li>
-
- <a
href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
- <i class="icon-chevron-right"></i>
- Stellar-common</a>
- </li>
-
- <li>
-
- <a href="../../use-cases/index.html"
title="Use-cases">
- <i class="icon-chevron-right"></i>
- Use-cases</a>
- </li>
- </ul>
- </li>
- </ul>
-
-
-
- <hr class="divider" />
-
- <div id="poweredBy">
- <div class="clear"></div>
- <div class="clear"></div>
- <div class="clear"></div>
- <a href="http://maven.apache.org/"; title="Built
by Maven" class="poweredBy">
- <img class="builtBy" alt="Built by Maven"
src="../../images/logos/maven-feather.png" />
- </a>
- </div>
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+ <li><a href="../../index.html" title="Metron"><span
class="icon-chevron-down"></span>Metron</a>
+ <ul class="nav nav-list">
+ <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span
class="none"></span>CONTRIBUTING</a></li>
+ <li><a href="../../Upgrading.html" title="Upgrading"><span
class="none"></span>Upgrading</a></li>
+ <li><a href="../../metron-analytics/index.html" title="Analytics"><span
class="icon-chevron-down"></span>Analytics</a>
+ <ul class="nav nav-list">
+ <li><a href="../../metron-analytics/metron-maas-service/index.html"
title="Maas-service"><span class="none"></span>Maas-service</a></li>
+ <li><a href="../../metron-analytics/metron-profiler/index.html"
title="Profiler"><span class="none"></span>Profiler</a></li>
+ <li class="active"><a href="#"><span
class="none"></span>Profiler-client</a></li>
+ <li><a href="../../metron-analytics/metron-statistics/index.html"
title="Statistics"><span class="icon-chevron-right"></span>Statistics</a></li>
+ </ul>
+</li>
+ <li><a href="../../metron-contrib/metron-docker/index.html"
title="Docker"><span class="none"></span>Docker</a></li>
+ <li><a href="../../metron-contrib/metron-performance/index.html"
title="Performance"><span class="none"></span>Performance</a></li>
+ <li><a href="../../metron-deployment/index.html" title="Deployment"><span
class="icon-chevron-right"></span>Deployment</a></li>
+ <li><a href="../../metron-interface/metron-alerts/index.html"
title="Alerts"><span class="none"></span>Alerts</a></li>
+ <li><a href="../../metron-interface/metron-config/index.html"
title="Config"><span class="none"></span>Config</a></li>
+ <li><a href="../../metron-interface/metron-rest/index.html"
title="Rest"><span class="none"></span>Rest</a></li>
+ <li><a href="../../metron-platform/index.html" title="Platform"><span
class="icon-chevron-right"></span>Platform</a></li>
+ <li><a href="../../metron-sensors/index.html" title="Sensors"><span
class="icon-chevron-right"></span>Sensors</a></li>
+ <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html"
title="Stellar-3rd-party-example"><span
class="none"></span>Stellar-3rd-party-example</a></li>
+ <li><a href="../../metron-stellar/stellar-common/index.html"
title="Stellar-common"><span
class="icon-chevron-right"></span>Stellar-common</a></li>
+ <li><a href="../../metron-stellar/stellar-zeppelin/index.html"
title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
+ <li><a href="../../use-cases/index.html" title="Use-cases"><span
class="icon-chevron-right"></span>Use-cases</a></li>
+ </ul>
+</li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href="http://maven.apache.org/"; title="Built by Maven"
class="poweredBy"><img class="builtBy" alt="Built by Maven"
src="../../images/logos/maven-feather.png" /></a>
+ </div>
</div>
</div>
-
-
- <div id="bodyColumn" class="span9" >
-
- <h1>Metron Profiler Client</h1>
+ <div id="bodyColumn" class="span10" >
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<h1>Metron Profiler Client</h1>
<p><a name="Metron_Profiler_Client"></a></p>
-<p>This project provides a client API for accessing the profiles generated by
the <a href="../metron-profiler/index.html">Metron Profiler</a>. This includes
both a Java API and Stellar API for accessing the profile data. The primary use
case is to extract profile data for use during model scoring.</p>
+<p>This project provides a client API for accessing the profiles generated by
the <a href="../metron-profiler/index.html">Metron Profiler</a>. This includes
both a Java API and Stellar API for accessing the profile data. The primary
use case is to extract profile data for use during model scoring.</p>
<div class="section">
<h2><a name="Stellar_Client_API"></a>Stellar Client API</h2>
<div class="section">
<h3><a name="PROFILE_GET"></a><tt>PROFILE_GET</tt></h3>
-<p>The <tt>PROFILE_GET</tt> command allows you to select all of the profile
measurements written. This command takes the following arguments:</p>
+<p>The <tt>PROFILE_GET</tt> command allows you to select all of the profile
measurements written. This command takes the following arguments:</p>
-<div class="source">
-<div class="source">
-<pre>REQUIRED:
+<div>
+<div>
+<pre class="source">REQUIRED:
profile - The name of the profile
entity - The name of the entity
periods - The list of profile periods to grab. These are ProfilePeriod
objects.
@@ -238,123 +118,89 @@ OPTIONAL:
config_overrides - Optional - Map (in curly braces) of name:value pairs,
each overriding the global config parameter
of the same name. Default is the empty Map, meaning no overrides.
</pre></div></div>
-<p>There is an older calling format where <tt>groups_list</tt> is specified as
a sequence of group names, “varargs” style, instead of a List
object. This format is still supported for backward compatibility, but it is
deprecated, and it is disallowed if the optional <tt>config_overrides</tt>
argument is used.</p>
+
+<p>There is an older calling format where <tt>groups_list</tt> is specified as
a sequence of group names, “varargs” style, instead of a List
object. This format is still supported for backward compatibility, but it is
deprecated, and it is disallowed if the optional <tt>config_overrides</tt>
argument is used.</p>
<p>The <tt>periods</tt> field is (likely) the output of another Stellar
function which defines the times to include.</p>
<div class="section">
<h4><a name="Groups_list_argument"></a>Groups_list argument</h4>
-<p>The <tt>groups_list</tt> argument in the client must exactly correspond to
the <a href="../metron-profiler/index.html#groupBy"><tt>groupBy</tt></a>
configuration in the profile definition. If <tt>groupBy</tt> was not used in
the profile, <tt>groups_list</tt> must be empty in the client. If
<tt>groupBy</tt> was used in the profile, then the client <tt>groups_list</tt>
is <b>not</b> optional; it must be the same length as the <tt>groupBy</tt>
list, and specify exactly one selected group value for each <tt>groupBy</tt>
criterion, in the same order. For example:</p>
+<p>The <tt>groups_list</tt> argument in the client must exactly correspond to
the <a href="../metron-profiler/index.html#groupBy"><tt>groupBy</tt></a>
configuration in the profile definition. If <tt>groupBy</tt> was not used in
the profile, <tt>groups_list</tt> must be empty in the client. If
<tt>groupBy</tt> was used in the profile, then the client <tt>groups_list</tt>
is <b>not</b> optional; it must be the same length as the <tt>groupBy</tt>
list, and specify exactly one selected group value for each <tt>groupBy</tt>
criterion, in the same order. For example:</p>
-<div class="source">
-<div class="source">
-<pre>If in Profile, the groupBy criteria are: [
“DAY_OF_WEEK()”, “URL_TO_PORT()” ]
+<div>
+<div>
+<pre class="source">If in Profile, the groupBy criteria are: [
“DAY_OF_WEEK()”, “URL_TO_PORT()” ]
Then in PROFILE_GET, an allowed groups value would be: [ “3”,
“8080” ]
which will select only records from Tuesdays with port number 8080.
-</pre></div></div></div>
+</pre></div></div>
+</div>
<div class="section">
<h4><a
name="Configuration_and_the_config_overrides_argument"></a>Configuration and
the config_overrides argument</h4>
-<p>By default, the Profiler creates profiles with a period duration of 15
minutes. This means that data is accumulated, summarized and flushed every 15
minutes. The Client API must also have knowledge of this duration to correctly
retrieve the profile data. If the Client is expecting 15 minute periods, it
will not be able to read data generated by a Profiler that was configured for 1
hour periods, and will return zero results. </p>
-<p>Similarly, all six Client configuration parameters listed in the table
below must match the Profiler configuration parameter settings from the time
the profile was created. The period duration and other configuration parameters
from the Profiler topology are stored in local filesystem at
<tt>$METRON_HOME/config/profiler.properties</tt>. The Stellar Client API can be
configured correspondingly by setting the following properties in
Metron’s global configuration, on local filesystem at
<tt>$METRON_HOME/config/zookeeper/global.json</tt>, then uploaded to Zookeeper
(at <tt>/metron/topology/global</tt>) by using <tt>zk_load_configs.sh</tt>: </p>
+<p>By default, the Profiler creates profiles with a period duration of 15
minutes. This means that data is accumulated, summarized and flushed every 15
minutes. The Client API must also have knowledge of this duration to correctly
retrieve the profile data. If the Client is expecting 15 minute periods, it
will not be able to read data generated by a Profiler that was configured for 1
hour periods, and will return zero results.</p>
+<p>Similarly, all six Client configuration parameters listed in the table
below must match the Profiler configuration parameter settings from the time
the profile was created. The period duration and other configuration parameters
from the Profiler topology are stored in local filesystem at
<tt>$METRON_HOME/config/profiler.properties</tt>. The Stellar Client API can be
configured correspondingly by setting the following properties in
Metron’s global configuration, on local filesystem at
<tt>$METRON_HOME/config/zookeeper/global.json</tt>, then uploaded to Zookeeper
(at <tt>/metron/topology/global</tt>) by using <tt>zk_load_configs.sh</tt>:</p>
-<div class="source">
-<div class="source">
-<pre> $ cd $METRON_HOME
+<div>
+<div>
+<pre class="source"> $ cd $METRON_HOME
$ bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z node1:2181
</pre></div></div>
-<p>Any of these six Client configuration parameters may be overridden at run
time using the <tt>config_overrides</tt> Map argument in PROFILE_GET. The
primary use case is when historical profiles have been created with a different
Profiler configuration than is currently configured, and the analyst needing to
access them does not want to change the global Client configuration so as not
to disrupt the work of other analysts working with current profiles.</p>
+<p>Any of these six Client configuration parameters may be overridden at run
time using the <tt>config_overrides</tt> Map argument in PROFILE_GET. The
primary use case is when historical profiles have been created with a different
Profiler configuration than is currently configured, and the analyst needing to
access them does not want to change the global Client configuration so as not
to disrupt the work of other analysts working with current profiles.</p>
<table border="0" class="table table-striped">
- <thead>
-
+<thead>
+
<tr class="a">
-
-<th>Key </th>
-
-<th>Description </th>
-
-<th>Required </th>
-
-<th>Default </th>
- </tr>
- </thead>
- <tbody>
-
+<th> Key </th>
+<th> Description
</th>
+<th> Required </th>
+<th> Default </th></tr>
+</thead><tbody>
+
<tr class="b">
-
-<td>profiler.client.period.duration </td>
-
-<td>The duration of each profile period. This value should be defined along
with <tt>profiler.client.period.duration.units</tt>. </td>
-
-<td>Optional </td>
-
-<td>15 </td>
- </tr>
-
+<td> profiler.client.period.duration </td>
+<td> The duration of each profile period. This value should be defined along
with <tt>profiler.client.period.duration.units</tt>. </td>
+<td> Optional </td>
+<td> 15 </td></tr>
<tr class="a">
-
-<td>profiler.client.period.duration.units </td>
-
-<td>The units used to specify the profile period duration. This value should
be defined along with <tt>profiler.client.period.duration</tt>. </td>
-
-<td>Optional </td>
-
-<td>MINUTES </td>
- </tr>
-
+<td> profiler.client.period.duration.units </td>
+<td> The units used to specify the profile period duration. This value should
be defined along with <tt>profiler.client.period.duration</tt>. </td>
+<td> Optional </td>
+<td> MINUTES </td></tr>
<tr class="b">
-
-<td>profiler.client.hbase.table </td>
-
-<td>The name of the HBase table used to store profile data. </td>
-
-<td>Optional </td>
-
-<td>profiler </td>
- </tr>
-
+<td> profiler.client.hbase.table </td>
+<td> The name of the HBase table used to store profile data.
</td>
+<td> Optional </td>
+<td> profiler </td></tr>
<tr class="a">
-
-<td>profiler.client.hbase.column.family </td>
-
-<td>The name of the HBase column family used to store profile data. </td>
-
-<td>Optional </td>
-
-<td>P </td>
- </tr>
-
+<td> profiler.client.hbase.column.family </td>
+<td> The name of the HBase column family used to store profile data.
</td>
+<td> Optional </td>
+<td> P </td></tr>
<tr class="b">
-
-<td>profiler.client.salt.divisor </td>
-
-<td>The salt divisor used to store profile data. </td>
-
-<td>Optional </td>
-
-<td>1000 </td>
- </tr>
-
+<td> profiler.client.salt.divisor </td>
+<td> The salt divisor used to store profile data.
</td>
+<td> Optional </td>
+<td> 1000 </td></tr>
<tr class="a">
-
-<td>hbase.provider.impl </td>
-
-<td>The name of the HBaseTableProvider implementation class. </td>
-
-<td>Optional </td>
-
-<td> </td>
- </tr>
- </tbody>
+<td> profiler.default.value </td>
+<td> The default value to be returned if a profile is not written for a given
period for a profile and entity. </td>
+<td> Optional </td>
+<td> null </td></tr>
+<tr class="b">
+<td> hbase.provider.impl </td>
+<td> The name of the HBaseTableProvider implementation class.
</td>
+<td> Optional </td>
+<td> </td></tr>
+</tbody>
</table></div></div>
<div class="section">
<h3><a name="Profile_Selectors"></a>Profile Selectors</h3>
-<p>You will notice that the third argument for <tt>PROFILE_GET</tt> is a list
of <tt>ProfilePeriod</tt> objects. This list is expected to be produced by
another Stellar function. There are a couple options available.</p>
+<p>You will notice that the third argument for <tt>PROFILE_GET</tt> is a list
of <tt>ProfilePeriod</tt> objects. This list is expected to be produced by
another Stellar function. There are a couple options available.</p>
<div class="section">
<h4><a name="PROFILE_FIXED"></a><tt>PROFILE_FIXED</tt></h4>
-<p>The profiler periods associated with a fixed lookback starting from now.
These are ProfilePeriod objects.</p>
+<p>The profiler periods associated with a fixed lookback starting from now.
These are ProfilePeriod objects.</p>
-<div class="source">
-<div class="source">
-<pre>REQUIRED:
+<div>
+<div>
+<pre class="source">REQUIRED:
durationAgo - How long ago should values be retrieved from?
units - The units of 'durationAgo'.
OPTIONAL:
@@ -363,33 +209,28 @@ OPTIONAL:
e.g. To retrieve all the profiles for the last 5 hours.
PROFILE_GET('profile', 'entity', PROFILE_FIXED(5, 'HOURS'))
</pre></div></div>
-<p>Note that the <tt>config_overrides</tt> parameter operates exactly as the
<tt>config_overrides</tt> argument in <tt>PROFILE_GET</tt>. The only available
parameters for override are:</p>
+<p>Note that the <tt>config_overrides</tt> parameter operates exactly as the
<tt>config_overrides</tt> argument in <tt>PROFILE_GET</tt>. The only available
parameters for override are:</p>
<ul>
-
+
<li><tt>profiler.client.period.duration</tt></li>
-
<li><tt>profiler.client.period.duration.units</tt></li>
</ul></div>
<div class="section">
<h4><a name="PROFILE_WINDOW"></a><tt>PROFILE_WINDOW</tt></h4>
<p><tt>PROFILE_WINDOW</tt> is intended to provide a finer-level of control
over selecting windows for profiles:</p>
-
<ul>
-
+
<li>Specify windows relative to the data timestamp (see the optional
<tt>now</tt> parameter below)</li>
-
<li>Specify non-contiguous windows to better handle seasonal data (e.g. the
last hour for every day for the last month)</li>
-
<li>Specify profile output excluding holidays</li>
-
<li>Specify only profile output on a specific day of the week</li>
</ul>
<p>It does this by a domain specific language mimicking natural language that
defines the windows excluded.</p>
-<div class="source">
-<div class="source">
-<pre>REQUIRED:
+<div>
+<div>
+<pre class="source">REQUIRED:
windowSelector - The statement specifying the window to select.
now - Optional - The timestamp to use for now.
OPTIONAL:
@@ -400,273 +241,213 @@ e.g. To retrieve all the measurements written for
'profile' and 'entity' for the
on the same weekday excluding weekends and US holidays across the last 14
days:
PROFILE_GET('profile', 'entity', PROFILE_WINDOW('1 hour window every 24 hours
starting from 14 days ago including the current day of the week excluding
weekends, holidays:us'))
</pre></div></div>
-<p>Note that the <tt>config_overrides</tt> parameter operates exactly as the
<tt>config_overrides</tt> argument in <tt>PROFILE_GET</tt>. The only available
parameters for override are:</p>
+<p>Note that the <tt>config_overrides</tt> parameter operates exactly as the
<tt>config_overrides</tt> argument in <tt>PROFILE_GET</tt>. The only available
parameters for override are:</p>
<ul>
-
+
<li><tt>profiler.client.period.duration</tt></li>
-
<li><tt>profiler.client.period.duration.units</tt></li>
</ul>
<div class="section">
<h5><a name="The_Profile_Selector_Language"></a>The Profile Selector
Language</h5>
<p>The domain specific language can be broken into a series of clauses, some
optional</p>
-
<ul>
-
+
<li><a href="#Temporal_Window_Width"><span style="color:blue">Total Temporal
Duration</span></a> - The total range of time in which windows may be
specified</li>
-
<li><a href="#InclusionExclusion_specifiers"><span style="color:red">Temporal
Window Width</span></a> - How large each temporal window</li>
-
<li><a href="#Skip_distance"><span style="color:green">Skip
distance</span></a> (optional)- How far to skip between when one window starts
and when the next begins</li>
-
<li><a href="#InclusionExclusion_specifiers"><span
style="color:purple">Inclusion/Exclusion specifiers</span></a> (optional) - The
set of specifiers to further filter the window</li>
</ul>
<p>One <i>must</i> specify either a total temporal duration or a temporal
window width. The remaining clauses are optional. During the course of the
following discussion, we will color code the clauses in the examples and link
them to the relevant section for more detail.</p>
<p>From a high level, the language fits the following three forms, which are
composed of the clauses above:</p>
-
<ul>
-
-<li><a href="#Temporal_Window_Width"><span style="color:red">time_interval
WINDOW?</span></a> <a href="#InclusionExclusion_specifiers"><span
style="color:purple">(INCLUDING specifier_list)? (EXCLUDING
specifier_list)?</span></a></li>
-
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">time_interval
WINDOW?</span></a> <a href="#InclusionExclusion_specifiers"><span
style="color:purple">(INCLUDING specifier_list)? (EXCLUDING
specifier_list)?</span></a></li>
<li><a href="#Temporal_Window_Width"><span style="color:red">time_interval
WINDOW?</span></a> <a href="#Skip_distance"><span style="color:green">EVERY
time_interval</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">FROM time_interval (TO time_interval)?</span></a> <a
href="#InclusionExclusion_specifiers"><span style="color:purple">(INCLUDING
specifier_list)? (EXCLUDING specifier_list)?</span></a></li>
-
<li><a href="#Total_Temporal_Duration"><span style="color:blue">FROM
time_interval (TO time_interval)?</span></a></li>
</ul>
<div class="section">
<h6><a name="Total_Temporal_Duration"></a><span style="color:blue">Total
Temporal Duration</span></h6>
-<p>Total temporal duration is specified by a phrase: <tt>FROM time_interval
AGO TO time_interval AGO</tt> This indicates the beginning and ending of a time
interval. This is an inclusive duration.</p>
-
+<p>Total temporal duration is specified by a phrase: <tt>FROM time_interval
AGO TO time_interval AGO</tt> This indicates the beginning and ending of a time
interval. This is an inclusive duration.</p>
<ul>
-
+
<li><tt>FROM</tt> - Can be the words “from” or “starting
from”</li>
-
<li><tt>time_interval</tt> - A time amount followed by a unit (e.g. 1 hour).
Fractional amounts are not supported. The unit may be “minute”,
“day”, “hour” with any pluralization.</li>
-
<li><tt>TO</tt> - Can be the words “until” or
“to”</li>
-
<li><tt>AGO</tt> - Optionally the word “ago”</li>
</ul>
-<p>The <tt>TO time_interval AGO</tt> portion is optional. If unspecified then
it is expected that the time interval ends now.</p>
-<p>Due to the vagaries of the english language, the from and the to portions,
if both specified, are interchangeable with regard to which one specifies the
start and which specifies the end. </p>
+<p>The <tt>TO time_interval AGO</tt> portion is optional. If unspecified then
it is expected that the time interval ends now.</p>
+<p>Due to the vagaries of the english language, the from and the to portions,
if both specified, are interchangeable with regard to which one specifies the
start and which specifies the end.</p>
<p>In other words “<a href="#Total_Temporal_Duration"><span
style="color:blue">starting from 1 hour ago to 30 minutes
ago</span></a>” and “<a href="#Total_Temporal_Duration"><span
style="color:blue">starting from 30 minutes ago to 1 hour
ago</span></a>” specify the same temporal duration.</p>
<p><b>Examples</b></p>
-
<ul>
-
+
<li>A duration starting 1 hour ago and ending now
-
<ul>
-
+
<li><a href="#Total_Temporal_Duration"><span style="color:blue">from 1 hour
ago</span></a></li>
-
<li><a href="#Total_Temporal_Duration"><span style="color:blue">from 1
hour</span></a></li>
-
<li><a href="#Total_Temporal_Duration"><span style="color:blue">starting from
1 hour ago</span></a></li>
-
<li><a href="#Total_Temporal_Duration"><span style="color:blue">starting from
1 hour</span></a></li>
- </ul></li>
-
+</ul>
+</li>
<li>A duration starting 1 hour ago and ending 30 minutes ago:
-
<ul>
-
+
<li><a href="#Total_Temporal_Duration"><span style="color:blue">from 1 hour
ago until 30 minutes ago</span></a></li>
-
<li><a href="#Total_Temporal_Duration"><span style="color:blue">from 30
minutes ago until 1 hour ago</span></a></li>
-
<li><a href="#Total_Temporal_Duration"><span style="color:blue">starting from
1 hour ago to 30 minutes ago</span></a></li>
-
<li><a href="#Total_Temporal_Duration"><span style="color:blue">starting from
1 hour to 30 minutes</span></a></li>
- </ul></li>
+</ul>
+</li>
</ul></div>
<div class="section">
<h6><a name="Temporal_Window_Width"></a><span style="color:red">Temporal
Window Width</span></h6>
-<p>Temporal window width is the specification of a window. A window is may
either repeat within total temporal duration or may fill the total temporal
duration. This is an inclusive window. A window is specified by the phrase:
<tt>time_interval WINDOW</tt></p>
-
+<p>Temporal window width is the specification of a window. A window is may
either repeat within total temporal duration or may fill the total temporal
duration. This is an inclusive window. A window is specified by the phrase:
<tt>time_interval WINDOW</tt></p>
<ul>
-
-<li><tt>time_interval</tt> - A time amount followed by a unit (e.g. 1 hour).
Fractional amounts are not supported. The unit may be “minute”,
“day”, “hour” with any pluralization.</li>
-
+
+<li><tt>time_interval</tt> - A time amount followed by a unit (e.g. 1 hour).
Fractional amounts are not supported. The unit may be “minute”,
“day”, “hour” with any pluralization.</li>
<li><tt>WINDOW</tt> - Optionally the word “window”</li>
</ul>
<p><b>Examples</b></p>
-
<ul>
-
+
<li>A fixed window starting 2 hours ago and going until now
-
<ul>
-
+
<li><a href="#Temporal_Window_Width"><span style="color:red">2
hour</span></a></li>
-
<li><a href="#Temporal_Window_Width"><span style="color:red">2
hours</span></a></li>
-
<li><a href="#Temporal_Window_Width"><span style="color:red">2 hours
window</span></a></li>
- </ul></li>
-
+</ul>
+</li>
<li>A repeating 30 minute window starting 2 hours ago and repeating every hour
until now. This would result in 2 30-minute wide windows: 2 hours ago and 1
hour ago
-
<ul>
-
+
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute
window</span></a> <a href="#Skip_distance"><span style="color:green">every 1
hour</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">starting from 2 hours ago</span></a></li>
-
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute
windows</span></a> <a href="#Skip_distance"><span style="color:green">every 1
hour</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">from 2 hours ago</span></a></li>
- </ul></li>
-
+</ul>
+</li>
<li>A repeating 30 minute window starting 2 hours ago and repeating every hour
until 30 minutes ago. This would result in 2 30-minute wide windows: 2 hours
ago and 1 hour ago
-
<ul>
-
+
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute
window</span></a> <a href="#Skip_distance"><span style="color:green">every 1
hour</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">starting from 2 hours ago until 30 minutes
ago</span></a></li>
-
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes
window</span></a> <a href="#Skip_distance"><span style="color:green">every 1
hour</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">from 2 hours ago to 30 minutes ago</span></a></li>
-
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes
window</span></a> <a href="#Skip_distance"><span style="color:green">for every
1 hour</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">from 30 minutes ago to 2 hours ago</span></a></li>
- </ul></li>
+</ul>
+</li>
</ul></div>
<div class="section">
<h6><a name="Skip_distance"></a><span style="color:green">Skip
distance</span></h6>
-<p>Skip distance is the amount of time between temporal window beginnings that
the next window starts. It is, in effect, the window period. </p>
+<p>Skip distance is the amount of time between temporal window beginnings that
the next window starts. It is, in effect, the window period.</p>
<p>It is specified by the phrase <tt>EVERY time_interval</tt></p>
-
<ul>
-
-<li><tt>time_interval</tt> - A time amount followed by a unit (e.g. 1 hour).
Fractional amounts are not supported. The unit may be “minute”,
“day”, “hour” with any pluralization.</li>
-
+
+<li><tt>time_interval</tt> - A time amount followed by a unit (e.g. 1 hour).
Fractional amounts are not supported. The unit may be “minute”,
“day”, “hour” with any pluralization.</li>
<li><tt>EVERY</tt> - The word/phrase “every” or “for
every”</li>
</ul>
<p><b>Examples</b></p>
-
<ul>
-
+
<li>A repeating 30 minute window starting 2 hours ago and repeating every hour
until now. This would result in 2 30-minute wide windows: 2 hours ago and 1
hour ago
-
<ul>
-
+
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute
window</span></a> <a href="#Skip_distance"><span style="color:green">every 1
hour</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">starting from 2 hours ago </span></a></li>
-
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes
window</span></a> <a href="#Skip_distance"><span style="color:green">every 1
hour</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">from 2 hours ago </span></a></li>
- </ul></li>
-
+</ul>
+</li>
<li>A repeating 30 minute window starting 2 hours ago and repeating every hour
until 30 minutes ago. This would result in 2 30-minute wide windows: 2 hours
ago and 1 hour ago
-
<ul>
-
+
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute
window</span></a> <a href="#Skip_distance"><span style="color:green">every 1
hour</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">starting from 2 hours ago until 30 minutes
ago</span></a></li>
-
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes
window</span></a> <a href="#Skip_distance"><span style="color:green">every 1
hour</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">from 2 hours ago to 30 minutes ago</span></a></li>
-
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes
window</span></a> <a href="#Skip_distance"><span style="color:green">for every
1 hour</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">from 30 minutes ago to 2 hours ago</span></a></li>
- </ul></li>
+</ul>
+</li>
</ul></div>
<div class="section">
-<h6><a name="InclusionExclusion_specifiers"></a><span
style="color:purple">Inclusion/Exclusion specifiers</span></h6>
+<h6><a name="Inclusion.2FExclusion_specifiers"></a><span
style="color:purple">Inclusion/Exclusion specifiers</span></h6>
<p>Inclusion and Exclusion specifiers operate as filters on the set of
windows. They operate on the window beginning timestamp.</p>
-<p>For inclusion specifiers, windows who are passed by <i>any</i> of the set
of inclusion specifiers are included.<br />inclusion specifiers. Similarly,
windows who are passed by <i>any</i> of the set of exclusion specifiers are
excluded. Exclusion specifiers trump inclusion specifiers.</p>
+<p>For inclusion specifiers, windows who are passed by <i>any</i> of the set
of inclusion specifiers are included.<br />
+inclusion specifiers. Similarly, windows who are passed by <i>any</i> of the
set of exclusion specifiers are excluded. Exclusion specifiers trump inclusion
specifiers.</p>
<p>Specifiers follow one of the following formats depending on if it is an
inclusion or exclusion specifier:</p>
-
<ul>
-
+
<li><tt>INCLUSION specifier, specifier, ...</tt>
-
<ul>
-
+
<li><tt>INCLUSION</tt> can be “include”,
“includes” or “including”</li>
- </ul></li>
-
+</ul>
+</li>
<li><tt>EXCLUSION specifier, specifier, ...</tt>
-
<ul>
-
+
<li><tt>EXCLUSION</tt> can be “exclude”,
“excludes” or “excluding”</li>
- </ul></li>
+</ul>
+</li>
</ul>
<p>The specifiers are a set of fixed specifiers available as part of the
language:</p>
-
<ul>
-
+
<li>Fixed day of week-based specifiers - includes or excludes if the window is
on the specified day of the week
-
<ul>
-
+
<li>“monday” or “mondays”</li>
-
<li>“tuesday” or “tuesdays”</li>
-
<li>“wednesday” or “wednesdays”</li>
-
<li>“thursday” or “thursdays”</li>
-
<li>“friday” or “fridays”</li>
-
<li>“saturday” or “saturdays”</li>
-
<li>“sunday” or “sundays”</li>
-
<li>“weekday” or “weekdays”</li>
-
<li>“weekend” or "“weekends”</li>
- </ul></li>
-
+</ul>
+</li>
<li>Relative day of week-based specifiers - includes or excludes based on the
day of week relative to now
-
<ul>
-
+
<li>“current day of the week”</li>
-
<li>“current day of week”</li>
-
<li>“this day of the week”</li>
-
<li>“this day of week”</li>
- </ul></li>
-
+</ul>
+</li>
<li>Specified date - includes or excludes based on the specified date
-
<ul>
-
+
<li>“date” - Takes up to 2 arguments
-
<ul>
-
+
<li>The day in <tt>yyyy/MM/dd</tt> format if no second argument is
provided</li>
-
<li>Optionally the format to specify the first argument in</li>
-
<li>Example: <tt>date:2017/12/25</tt> would include or exclude December 25,
2017</li>
-
<li>Example: <tt>date:20171225:yyyyMMdd</tt> would include or exclude December
25, 2017</li>
- </ul></li>
- </ul></li>
-
+</ul>
+</li>
+</ul>
+</li>
<li>Holidays - includes or excludes based on if the window starts during a
holiday
-
<ul>
-
+
<li>“holiday” or “holidays”
-
<ul>
-
-<li>Arguments form the jollyday hierarchy of holidays. e.g.
“us:nyc” would be holidays for New York City, USA</li>
-
+
+<li>Arguments form the jollyday hierarchy of holidays. e.g.
“us:nyc” would be holidays for New York City, USA</li>
<li>If none is specified, it will choose based on locale.</li>
-
<li>Countries supported are those supported in <a class="externalLink"
href="https://github.com/svendiedrichsen/jollyday/tree/master/src/main/resources/holidays";>jollyday</a></li>
-
<li>Example: <tt>holiday:us:nyc</tt> would be the holidays of New York City,
USA</li>
-
<li>Example: <tt>holiday:hu</tt> would be the holidays of Hungary</li>
- </ul></li>
- </ul></li>
+</ul>
+</li>
+</ul>
+</li>
</ul>
<p><b>WARNING: Daylight Savings Time effects</b></p>
-<p>While Universal Time (UTC) is nice and constant, many servers are set to
local timezones that enable Daylight Savings Time (DST). This means that twice
a year, on DST transition weekends, “Sunday” is either 23 or 25
hours long. However, durations specified as “7 days ago” are
always interpreted as “7*24 hours ago”. This can lead to some
surprising effects when using days of the week as inclusion or exclusion
specifiers.</p>
+<p>While Universal Time (UTC) is nice and constant, many servers are set to
local timezones that enable Daylight Savings Time (DST). This means that twice
a year, on DST transition weekends, “Sunday” is either 23 or 25
hours long. However, durations specified as “7 days ago” are
always interpreted as “7*24 hours ago”. This can lead to some
surprising effects when using days of the week as inclusion or exclusion
specifiers.</p>
<p>For example, the profile window specified by the phrase “30 minute
window every 24 hours from 7 days ago” will always have 7 thirty-minute
intervals, and these will normally occur on 5 weekdays and 2 weekend days.
However, if you invoke this window at 12:15am any day during the week following
the start of DST, you will get these intervals (supposing you start early on a
Wednesday morning):</p>
-<div class="source">
-<div class="source">
-<pre>Tuesday 12:15am-12:45am (yesterday)
+<div>
+<div>
+<pre class="source">Tuesday 12:15am-12:45am (yesterday)
Monday 12:15am-12:45am
Saturday 11:15pm-11:45pm (skipped Sunday!)
Friday 11:15pm-11:45pm
@@ -674,61 +455,56 @@ Thursday 11:15pm-11:45pm
Wednesday 11:15pm-11:45pm
Tuesday 11:15pm-11:45pm
</pre></div></div>
-<p>Sunday got skipped over because it was only 23 hours long; that is, there
were 24 hours between Saturday 11:15pm and Monday 12:15am. So if you specified
“excluding weekends”, you would get 6 days’ intervals
instead of the expected 5. There are multiple variations on this theme.</p>
-<p>Remember that the underlying time is kept in UTC, so the data is always
correct. It is only when attempting to interpret UTC as local time, date, and
day, that these confusions may occur. They may be eliminated by setting your
server timezone to UTC, or otherwise disabling DST.</p>
+
+<p>Sunday got skipped over because it was only 23 hours long; that is, there
were 24 hours between Saturday 11:15pm and Monday 12:15am. So if you specified
“excluding weekends”, you would get 6 days’ intervals
instead of the expected 5. There are multiple variations on this theme.</p>
+<p>Remember that the underlying time is kept in UTC, so the data is always
correct. It is only when attempting to interpret UTC as local time, date, and
day, that these confusions may occur. They may be eliminated by setting your
server timezone to UTC, or otherwise disabling DST.</p>
<p><b>Examples</b></p>
<p>Assume these are executed at noon.</p>
-
<ul>
-
+
<li>A 1 hour window for the past 8 ‘current day of the week’
-
<ul>
-
+
<li><a href="#Temporal_Window_Width"><span style="color:red">1 hour
window</span></a> <a href="#Skip_distance"><span style="color:green">every 24
hours</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">from 56 days ago</span></a> <a
href="#InclusionExclusion_specifiers"><span style="color:purple">including this
day of the week</span></a></li>
- </ul></li>
-
+</ul>
+</li>
<li>A 1 hour window for the past 8 tuesdays
-
<ul>
-
+
<li><a href="#Temporal_Window_Width"><span style="color:red">1 hour
window</span></a> <a href="#Skip_distance"><span style="color:green">every 24
hours</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">from 56 days ago</span></a> <a
href="#InclusionExclusion_specifiers"><span style="color:purple">including
tuesdays</span></a></li>
- </ul></li>
-
+</ul>
+</li>
<li>A 30 minute window every tuesday at noon starting 14 days ago until now
-
<ul>
-
+
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute
window</span></a> <a href="#Skip_distance"><span style="color:green">every 24
hours</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">from 14 days ago</span></a> <a
href="#InclusionExclusion_specifiers"><span style="color:purple">including
tuesdays</span></a></li>
- </ul></li>
-
+</ul>
+</li>
<li>A 30 minute window every day except holidays and weekends at noon starting
14 days ago until now
-
<ul>
-
+
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute
window</span></a> <a href="#Skip_distance"><span style="color:green">every 24
hours</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">from 14 days ago</span></a> <a
href="#InclusionExclusion_specifiers"><span style="color:purple">excluding
holidays:us, weekends</span></a></li>
-
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute
window</span></a> <a href="#Skip_distance"><span style="color:green">every 24
hours</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">from 14 days ago</span></a> <a
href="#InclusionExclusion_specifiers"><span style="color:purple">including
weekdays excluding holidays:us, weekends</span></a></li>
- </ul></li>
-
+</ul>
+</li>
<li>A 30 minute window at noon every day from 7 days ago including saturdays
and excluding weekends. Because exclusions trump inclusions, the following will
never yield any windows
-
<ul>
-
+
<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute
window</span></a> <a href="#Skip_distance"><span style="color:green">every 24
hours</span></a> <a href="#Total_Temporal_Duration"><span
style="color:blue">from 7 days ago</span></a> <a
href="#InclusionExclusion_specifiers"><span style="color:purple">including
saturdays excluding weekends</span></a></li>
- </ul></li>
+</ul>
+</li>
</ul></div></div></div></div>
<div class="section">
<h3><a name="Errors"></a>Errors</h3>
<p>The most common result of incorrect <tt>PROFILE_GET</tt> arguments or
Client configuration parameters is an empty result set, rather than an error.
The Client cannot effectively validate the arguments, because the Profiler
configuration parameters may be changed and the profile itself does not store
them. The person doing the querying must carry forward the knowledge of the
Profiler configuration parameters from the time of profile creation, and use
corresponding <tt>PROFILE_GET</tt> arguments and Client configuration
parameters when querying the data.</p></div>
<div class="section">
<h3><a name="Examples"></a>Examples</h3>
-<p>The following are usage examples that show how the Stellar API can be used
to read profiles generated by the <a
href="../metron-profiler/index.html">Metron Profiler</a>. This API would be
used in conjunction with other Stellar functions like <a
href="../../metron-stellar/stellar-common/index.html#MAAS_MODEL_APPLY"><tt>MAAS_MODEL_APPLY</tt></a>
to perform model scoring on streaming data.</p>
-<p>These examples assume a profile has been defined called
‘snort-alerts’ that tracks the number of Snort alerts associated
with an IP address over time. The profile definition might look similar to the
following.</p>
+<p>The following are usage examples that show how the Stellar API can be used
to read profiles generated by the <a
href="../metron-profiler/index.html">Metron Profiler</a>. This API would be
used in conjunction with other Stellar functions like <a
href="../../metron-stellar/stellar-common/index.html#MAAS_MODEL_APPLY"><tt>MAAS_MODEL_APPLY</tt></a>
to perform model scoring on streaming data.</p>
+<p>These examples assume a profile has been defined called
‘snort-alerts’ that tracks the number of Snort alerts associated
with an IP address over time. The profile definition might look similar to the
following.</p>
-<div class="source">
-<div class="source">
-<pre>{
+<div>
+<div>
+<pre class="source">{
"profiles": [
{
"profile": "snort-alerts",
@@ -740,46 +516,52 @@ Tuesday 11:15pm-11:45pm
]
}
</pre></div></div>
-<p>During model scoring the entity being scored, in this case a particular IP
address, will be known. The following examples shows how this profile data
might be retrieved. Retrieve all values of ‘snort-alerts’ from
‘10.0.0.1’ over the past 4 hours.</p>
-<div class="source">
-<div class="source">
-<pre>PROFILE_GET('snort-alerts', '10.0.0.1', PROFILE_FIXED(4, 'HOURS'))
+<p>During model scoring the entity being scored, in this case a particular IP
address, will be known. The following examples shows how this profile data
might be retrieved. Retrieve all values of ‘snort-alerts’ from
‘10.0.0.1’ over the past 4 hours.</p>
+
+<div>
+<div>
+<pre class="source">PROFILE_GET('snort-alerts', '10.0.0.1', PROFILE_FIXED(4,
'HOURS'))
</pre></div></div>
+
<p>Retrieve all values of ‘snort-alerts’ from
‘10.0.0.1’ over the past 2 days.</p>
-<div class="source">
-<div class="source">
-<pre>PROFILE_GET('snort-alerts', '10.0.0.1', PROFILE_FIXED(2, 'DAYS'))
+<div>
+<div>
+<pre class="source">PROFILE_GET('snort-alerts', '10.0.0.1', PROFILE_FIXED(2,
'DAYS'))
</pre></div></div>
+
<p>If the profile had been defined to group the data by weekday versus
weekend, then the following example would apply:</p>
<p>Retrieve all values of ‘snort-alerts’ from
‘10.0.0.1’ that occurred on ‘weekdays’ over the
past 30 days.</p>
-<div class="source">
-<div class="source">
-<pre>PROFILE_GET('snort-alerts', '10.0.0.1', PROFILE_FIXED(30, 'DAYS'),
['weekdays'] )
+<div>
+<div>
+<pre class="source">PROFILE_GET('snort-alerts', '10.0.0.1', PROFILE_FIXED(30,
'DAYS'), ['weekdays'] )
</pre></div></div>
-<p>The client may need to use a configuration different from the current
Client configuration settings. For example, perhaps you are on a cluster shared
with other analysts, and need to access a profile that was constructed 2 months
ago using different period duration, while they are accessing more recent
profiles constructed with the currently configured period duration. For this
situation, you may use the <tt>config_overrides</tt> argument:</p>
+
+<p>The client may need to use a configuration different from the current
Client configuration settings. For example, perhaps you are on a cluster
shared with other analysts, and need to access a profile that was constructed 2
months ago using different period duration, while they are accessing more
recent profiles constructed with the currently configured period duration. For
this situation, you may use the <tt>config_overrides</tt> argument:</p>
<p>Retrieve all values of ‘snort-alerts’ from
‘10.0.0.1’ over the past 2 days, with no <tt>groupBy</tt>, and
overriding the usual global client configuration parameters for window
duration.</p>
-<div class="source">
-<div class="source">
-<pre>PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(2, 'DAYS',
{'profiler.client.period.duration' : '2',
'profiler.client.period.duration.units' : 'MINUTES'}), [])
+<div>
+<div>
+<pre class="source">PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(2,
'DAYS', {'profiler.client.period.duration' : '2',
'profiler.client.period.duration.units' : 'MINUTES'}), [])
</pre></div></div>
+
<p>Retrieve all values of ‘snort-alerts’ from
‘10.0.0.1’ that occurred on ‘weekdays’ over the
past 30 days, overriding the usual global client configuration parameters for
window duration.</p>
-<div class="source">
-<div class="source">
-<pre>PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(30, 'DAYS',
{'profiler.client.period.duration' : '2',
'profiler.client.period.duration.units' : 'MINUTES'}), ['weekdays'] )
-</pre></div></div></div></div>
+<div>
+<div>
+<pre class="source">PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(30,
'DAYS', {'profiler.client.period.duration' : '2',
'profiler.client.period.duration.units' : 'MINUTES'}), ['weekdays'] )
+</pre></div></div>
+</div></div>
<div class="section">
<h2><a name="Getting_Started"></a>Getting Started</h2>
-<p>These instructions step through the process of using the Stellar Client API
on a live cluster. These instructions assume that the ‘Getting
Started’ instructions included with the <a
href="../metron-profiler/index.html">Metron Profiler</a> have been followed.
This will create a Profile called ‘test’ whose data will be
retrieved with the Stellar Client API.</p>
-<p>To validate that everything is working, login to the server hosting Metron.
We will use the Stellar Shell to replicate the execution environment of Stellar
running in a Storm topology, like Metron’s Parser or Enrichment
topology. Replace ‘node1:2181’ with the URL to a Zookeeper
Broker. </p>
+<p>These instructions step through the process of using the Stellar Client API
on a live cluster. These instructions assume that the ‘Getting
Started’ instructions included with the <a
href="../metron-profiler/index.html">Metron Profiler</a> have been followed.
This will create a Profile called ‘test’ whose data will be
retrieved with the Stellar Client API.</p>
+<p>To validate that everything is working, login to the server hosting Metron.
We will use the Stellar Shell to replicate the execution environment of
Stellar running in a Storm topology, like Metron’s Parser or Enrichment
topology. Replace ‘node1:2181’ with the URL to a Zookeeper
Broker.</p>
-<div class="source">
-<div class="source">
-<pre>[root@node1 0.4.2]# bin/stellar -z node1:2181
+<div>
+<div>
+<pre class="source">[root@node1 0.4.2]# bin/stellar -z node1:2181
Stellar, Go!
Please note that functions are loading lazily in the background and will be
unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300,
es.date.format=yyyy.MM.dd.HH}
@@ -805,20 +587,21 @@ Returns: The selected profile measurements.
[Stellar]>>> PROFILE_GET('test','192.168.138.158', 1, 'HOURS')
[12078.0, 8921.0, 12131.0]
</pre></div></div>
+
<p>The client API call above has retrieved the past hour of the
‘test’ profile for the entity
‘192.168.138.158’.</p></div>
<div class="section">
<h2><a name="Developing_Profiles"></a>Developing Profiles</h2>
-<p>Troubleshooting issues when programming against a live stream of data can
be difficult. The Stellar REPL is a powerful tool to help work out the kinds of
enrichments and transformations that are needed. The Stellar REPL can also be
used to help when developing profiles for the Profiler.</p>
+<p>Troubleshooting issues when programming against a live stream of data can
be difficult. The Stellar REPL is a powerful tool to help work out the kinds
of enrichments and transformations that are needed. The Stellar REPL can also
be used to help when developing profiles for the Profiler.</p>
<p>Follow these steps in the Stellar REPL to see how it can be used to help
create profiles.</p>
-
<ol style="list-style-type: decimal">
-
+
<li>
-<p>Take a first pass at defining your profile. As an example, in the editor
copy/paste the basic “Hello, World” profile below.</p>
-
-<div class="source">
-<div class="source">
-<pre>[Stellar]>>> conf := SHELL_EDIT()
+
+<p>Take a first pass at defining your profile. As an example, in the editor
copy/paste the basic “Hello, World” profile below.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> conf := SHELL_EDIT()
[Stellar]>>> conf
{
"profiles": [
@@ -832,26 +615,29 @@ Returns: The selected profile measurements.
}
]
}
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
+
<p>Initialize the Profiler.</p>
-
-<div class="source">
-<div class="source">
-<pre>[Stellar]>>> profiler := PROFILER_INIT(conf)
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> profiler := PROFILER_INIT(conf)
[Stellar]>>> profiler
Profiler{1 profile(s), 0 messages(s), 0 route(s)}
</pre></div></div>
-<p>The profiler itself will show the number of profiles defined, the number of
messages applied, and the number of routes taken. </p>
-<p>A route is defined when a message is applied to a specific profile. If a
message is applied and not needed by any profile, then there are no routes. If
a message is needed by one profile, then one route has been defined. If a
message is needed by two profiles, then two routes have been defined. </p></li>
-
+
+<p>The profiler itself will show the number of profiles defined, the number of
messages applied, and the number of routes taken.</p>
+<p>A route is defined when a message is applied to a specific profile. If a
message is applied and not needed by any profile, then there are no routes. If
a message is needed by one profile, then one route has been defined. If a
message is needed by two profiles, then two routes have been defined.</p>
+</li>
<li>
-<p>Create a message to simulate the type of telemetry that you expect to be
profiled. As an example, in the editor copy/paste the JSON below.</p>
-
-<div class="source">
-<div class="source">
-<pre>[Stellar]>>> message := SHELL_EDIT()
+
+<p>Create a message to simulate the type of telemetry that you expect to be
profiled. As an example, in the editor copy/paste the JSON below.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> message := SHELL_EDIT()
[Stellar]>>> message
{
"ip_src_addr": "10.0.0.1",
@@ -859,62 +645,62 @@ Profiler{1 profile(s), 0 messages(s), 0 route(s)}
"length": "10",
"bytes_in": "234"
}
-</pre></div></div></li>
-
+</pre></div></div>
+</li>
<li>
-<p>Apply some telemetry messages to your profiles. The following applies the
same message 3 times.</p>
-
-<div class="source">
-<div class="source">
-<pre>[Stellar]>>> PROFILER_APPLY(message, profiler)
+
+<p>Apply some telemetry messages to your profiles. The following applies the
same message 3 times.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> PROFILER_APPLY(message, profiler)
Profiler{1 profile(s), 1 messages(s), 1 route(s)}
</pre></div></div>
-
-<div class="source">
-<div class="source">
-<pre>[Stellar]>>> PROFILER_APPLY(message, profiler)
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> PROFILER_APPLY(message, profiler)
Profiler{1 profile(s), 2 messages(s), 2 route(s)}
</pre></div></div>
-
-<div class="source">
-<div class="source">
-<pre>[Stellar]>>> PROFILER_APPLY(message, profiler)
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> PROFILER_APPLY(message, profiler)
Profiler{1 profile(s), 3 messages(s), 3 route(s)}
</pre></div></div>
-<p>It is also possible to apply multiple messages at once. This is useful when
testing against a larger set of data. To do this, create a string that contains
a JSON array of messages and pass that to the <tt>PROFILER_APPLY</tt>
function.</p></li>
-
+
+<p>It is also possible to apply multiple messages at once. This is useful
when testing against a larger set of data. To do this, create a string that
contains a JSON array of messages and pass that to the <tt>PROFILER_APPLY</tt>
function.</p>
+</li>
<li>
-<p>Flush the Profiler to see what has been calculated. A flush is what occurs
at the end of each 15 minute period in the Profiler. The result is a list of
profile measurements. Each measurement is a map containing detailed information
about the profile data that has been generated.</p>
-
-<div class="source">
-<div class="source">
-<pre>[Stellar]>>> values := PROFILER_FLUSH(profiler)
+
+<p>Flush the Profiler to see what has been calculated. A flush is what occurs
at the end of each 15 minute period in the Profiler. The result is a list of
profile measurements. Each measurement is a map containing detailed
information about the profile data that has been generated.</p>
+
+<div>
+<div>
+<pre class="source">[Stellar]>>> values := PROFILER_FLUSH(profiler)
[Stellar]>>> values
[{period={duration=900000, period=1669628, start=1502665200000,
end=1502666100000},
profile=hello-world, groups=[], value=3, entity=10.0.0.1}]
</pre></div></div>
-<p>This profile simply counts the number of messages by IP source address.
Notice that the value is ‘3’ for the entity
‘10.0.0.1’ as we applied 3 messages with an
‘ip_src_addr’ of ‘10.0.0.1’. There will always be
one measurement for each [profile, entity] pair.</p></li>
-
+
+<p>This profile simply counts the number of messages by IP source address.
Notice that the value is ‘3’ for the entity
‘10.0.0.1’ as we applied 3 messages with an
‘ip_src_addr’ of ‘10.0.0.1’. There will always be
one measurement for each [profile, entity] pair.</p>
+</li>
<li>
-<p>If you are unhappy with the data that has been generated, then
‘wash, rinse and repeat’ this process. Once you are happy with
the profile that was created, follow the <a
href="../metron-profiler/index.html#Getting_Started">Getting Started</a> guide
to use the profile against your live, streaming data in a Metron
cluster.</p></li>
-</ol></div>
- </div>
- </div>
- </div>
+<p>If you are unhappy with the data that has been generated, then
‘wash, rinse and repeat’ this process. Once you are happy with
the profile that was created, follow the <a
href="../metron-profiler/index.html#Getting_Started">Getting Started</a> guide
to use the profile against your live, streaming data in a Metron cluster.</p>
+</li>
+</ol></div>
+ </div>
+ </div>
+ </div>
<hr/>
-
<footer>
- <div class="container-fluid">
- <div class="row span12">Copyright © 2018
- <a href="https://www.apache.org";>The Apache Software
Foundation</a>.
- All Rights Reserved.
-
+ <div class="container-fluid">
+ <div class="row-fluid">
+é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache,
the Apache feather logo,
+ and the Apache Metron project logo are trademarks of The Apache
Software Foundation.
+ </div>
</div>
-
-
-
- </div>
</footer>
</body>
</html>
