http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-platform/metron-parsers/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-parsers/index.html
b/site/current-book/metron-platform/metron-parsers/index.html
index 23955ac..807a24e 100644
--- a/site/current-book/metron-platform/metron-parsers/index.html
+++ b/site/current-book/metron-platform/metron-parsers/index.html
@@ -1,379 +1,211 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2018-01-03
- | Rendered using Apache Maven Fluido Skin 1.3.0
+ | Generated by Apache Maven Doxia Site Renderer 1.8 from
src/site/markdown/metron-platform/metron-parsers/index.md at 2018-06-07
+ | Rendered using Apache Maven Fluido Skin 1.7
-->
<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180103" />
+ <meta name="Date-Revision-yyyymmdd" content="20180607" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Parsers</title>
- <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css"
/>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" />
<link rel="stylesheet" href="../../css/site.css" />
<link rel="stylesheet" href="../../css/print.css" media="print" />
-
-
- <script type="text/javascript"
src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
-
-
-
-<script type="text/javascript">$( document ).ready( function() { $(
'.carousel' ).carousel( { interval: 3500 } ) } );</script>
-
- </head>
- <body class="topBarDisabled">
-
-
-
-
- <div class="container-fluid">
- <div id="banner">
- <div class="pull-left">
- <a href="http://metron.apache.org/";
id="bannerLeft">
-
<img src="../../images/metron-logo.png" alt="Apache Metron"
width="148px" height="48px"/>
- </a>
- </div>
- <div class="pull-right"> </div>
+ <script type="text/javascript"
src="../../js/apache-maven-fluido-1.7.min.js"></script>
+<script type="text/javascript">
+ $( document ).ready( function() { $( '.carousel' ).carousel( {
interval: 3500 } ) } );
+ </script>
+ </head>
+ <body class="topBarDisabled">
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left"><a href="http://metron.apache.org/";
id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron"
width="148px" height="48px"/></a></div>
+ <div class="pull-right"></div>
<div class="clear"><hr/></div>
</div>
<div id="breadcrumbs">
<ul class="breadcrumb">
-
-
- <li class="">
- <a href="http://www.apache.org"; class="externalLink"
title="Apache">
- Apache</a>
- </li>
- <li class="divider ">/</li>
- <li class="">
- <a href="http://metron.apache.org/"; class="externalLink"
title="Metron">
- Metron</a>
- </li>
- <li class="divider ">/</li>
- <li class="">
- <a href="../../index.html" title="Documentation">
- Documentation</a>
- </li>
- <li class="divider ">/</li>
- <li class="">Parsers</li>
-
-
-
- <li id="publishDate" class="pull-right">Last Published:
2018-01-03</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
-
- </ul>
+ <li class=""><a href="http://www.apache.org"; class="externalLink"
title="Apache">Apache</a><span class="divider">/</span></li>
+ <li class=""><a href="http://metron.apache.org/"; class="externalLink"
title="Metron">Metron</a><span class="divider">/</span></li>
+ <li class=""><a href="../../index.html"
title="Documentation">Documentation</a><span class="divider">/</span></li>
+ <li class="active ">Parsers</li>
+ <li id="publishDate" class="pull-right"><span class="divider">|</span>
Last Published: 2018-06-07</li>
+ <li id="projectVersion" class="pull-right">Version: 0.5.0</li>
+ </ul>
</div>
-
-
<div class="row-fluid">
- <div id="leftColumn" class="span3">
+ <div id="leftColumn" class="span2">
<div class="well sidebar-nav">
-
-
- <ul class="nav nav-list">
- <li class="nav-header">User Documentation</li>
-
- <li>
-
- <a href="../../index.html" title="Metron">
- <i class="icon-chevron-down"></i>
- Metron</a>
- <ul class="nav nav-list">
-
- <li>
-
- <a href="../../Upgrading.html" title="Upgrading">
- <i class="none"></i>
- Upgrading</a>
- </li>
-
- <li>
-
- <a href="../../metron-analytics/index.html"
title="Analytics">
- <i class="icon-chevron-right"></i>
- Analytics</a>
- </li>
-
- <li>
-
- <a
href="../../metron-contrib/metron-docker/index.html" title="Docker">
- <i class="none"></i>
- Docker</a>
- </li>
-
- <li>
-
- <a href="../../metron-deployment/index.html"
title="Deployment">
- <i class="icon-chevron-right"></i>
- Deployment</a>
- </li>
-
- <li>
-
- <a
href="../../metron-interface/metron-alerts/index.html" title="Alerts">
- <i class="none"></i>
- Alerts</a>
- </li>
-
- <li>
-
- <a
href="../../metron-interface/metron-config/index.html" title="Config">
- <i class="none"></i>
- Config</a>
- </li>
-
- <li>
-
- <a
href="../../metron-interface/metron-rest/index.html" title="Rest">
- <i class="none"></i>
- Rest</a>
- </li>
-
- <li>
-
- <a href="../../metron-platform/index.html"
title="Platform">
- <i class="icon-chevron-down"></i>
- Platform</a>
- <ul class="nav nav-list">
-
- <li>
-
- <a
href="../../metron-platform/Performance-tuning-guide.html"
title="Performance-tuning-guide">
- <i class="none"></i>
- Performance-tuning-guide</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-api/index.html" title="Api">
- <i class="none"></i>
- Api</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-common/index.html" title="Common">
- <i class="none"></i>
- Common</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-data-management/index.html"
title="Data-management">
- <i class="none"></i>
- Data-management</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-elasticsearch/index.html"
title="Elasticsearch">
- <i class="none"></i>
- Elasticsearch</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
- <i class="none"></i>
- Enrichment</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-indexing/index.html" title="Indexing">
- <i class="none"></i>
- Indexing</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-management/index.html" title="Management">
- <i class="none"></i>
- Management</a>
- </li>
-
- <li class="active">
-
- <a href="#"><i class="icon-chevron-down"></i>Parsers</a>
- <ul class="nav nav-list">
-
- <li>
-
- <a
href="../../metron-platform/metron-parsers/parser-testing.html"
title="parser-testing">
- <i class="none"></i>
- parser-testing</a>
- </li>
- </ul>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-pcap-backend/index.html"
title="Pcap-backend">
- <i class="none"></i>
- Pcap-backend</a>
- </li>
-
- <li>
-
- <a
href="../../metron-platform/metron-writer/index.html" title="Writer">
- <i class="none"></i>
- Writer</a>
- </li>
- </ul>
- </li>
-
- <li>
-
- <a href="../../metron-sensors/index.html"
title="Sensors">
- <i class="icon-chevron-right"></i>
- Sensors</a>
- </li>
-
- <li>
-
- <a
href="../../metron-stellar/stellar-3rd-party-example/index.html"
title="Stellar-3rd-party-example">
- <i class="none"></i>
- Stellar-3rd-party-example</a>
- </li>
-
- <li>
-
- <a
href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
- <i class="icon-chevron-right"></i>
- Stellar-common</a>
- </li>
-
- <li>
-
- <a href="../../use-cases/index.html"
title="Use-cases">
- <i class="icon-chevron-right"></i>
- Use-cases</a>
- </li>
- </ul>
- </li>
- </ul>
-
-
-
- <hr class="divider" />
-
- <div id="poweredBy">
- <div class="clear"></div>
- <div class="clear"></div>
- <div class="clear"></div>
- <a href="http://maven.apache.org/"; title="Built
by Maven" class="poweredBy">
- <img class="builtBy" alt="Built by Maven"
src="../../images/logos/maven-feather.png" />
- </a>
- </div>
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+ <li><a href="../../index.html" title="Metron"><span
class="icon-chevron-down"></span>Metron</a>
+ <ul class="nav nav-list">
+ <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span
class="none"></span>CONTRIBUTING</a></li>
+ <li><a href="../../Upgrading.html" title="Upgrading"><span
class="none"></span>Upgrading</a></li>
+ <li><a href="../../metron-analytics/index.html" title="Analytics"><span
class="icon-chevron-right"></span>Analytics</a></li>
+ <li><a href="../../metron-contrib/metron-docker/index.html"
title="Docker"><span class="none"></span>Docker</a></li>
+ <li><a href="../../metron-contrib/metron-performance/index.html"
title="Performance"><span class="none"></span>Performance</a></li>
+ <li><a href="../../metron-deployment/index.html" title="Deployment"><span
class="icon-chevron-right"></span>Deployment</a></li>
+ <li><a href="../../metron-interface/metron-alerts/index.html"
title="Alerts"><span class="none"></span>Alerts</a></li>
+ <li><a href="../../metron-interface/metron-config/index.html"
title="Config"><span class="none"></span>Config</a></li>
+ <li><a href="../../metron-interface/metron-rest/index.html"
title="Rest"><span class="none"></span>Rest</a></li>
+ <li><a href="../../metron-platform/index.html" title="Platform"><span
class="icon-chevron-down"></span>Platform</a>
+ <ul class="nav nav-list">
+ <li><a href="../../metron-platform/Performance-tuning-guide.html"
title="Performance-tuning-guide"><span
class="none"></span>Performance-tuning-guide</a></li>
+ <li><a href="../../metron-platform/metron-api/index.html"
title="Api"><span class="none"></span>Api</a></li>
+ <li><a href="../../metron-platform/metron-common/index.html"
title="Common"><span class="none"></span>Common</a></li>
+ <li><a href="../../metron-platform/metron-data-management/index.html"
title="Data-management"><span class="none"></span>Data-management</a></li>
+ <li><a href="../../metron-platform/metron-elasticsearch/index.html"
title="Elasticsearch"><span class="none"></span>Elasticsearch</a></li>
+ <li><a href="../../metron-platform/metron-enrichment/index.html"
title="Enrichment"><span class="icon-chevron-right"></span>Enrichment</a></li>
+ <li><a href="../../metron-platform/metron-indexing/index.html"
title="Indexing"><span class="none"></span>Indexing</a></li>
+ <li><a href="../../metron-platform/metron-management/index.html"
title="Management"><span class="none"></span>Management</a></li>
+ <li class="active"><a href="#"><span
class="icon-chevron-down"></span>Parsers</a>
+ <ul class="nav nav-list">
+ <li><a href="../../metron-platform/metron-parsers/3rdPartyParser.html"
title="3rdPartyParser"><span class="none"></span>3rdPartyParser</a></li>
+ <li><a href="../../metron-platform/metron-parsers/parser-testing.html"
title="parser-testing"><span class="none"></span>parser-testing</a></li>
+ </ul>
+</li>
+ <li><a href="../../metron-platform/metron-pcap-backend/index.html"
title="Pcap-backend"><span class="none"></span>Pcap-backend</a></li>
+ <li><a href="../../metron-platform/metron-writer/index.html"
title="Writer"><span class="none"></span>Writer</a></li>
+ </ul>
+</li>
+ <li><a href="../../metron-sensors/index.html" title="Sensors"><span
class="icon-chevron-right"></span>Sensors</a></li>
+ <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html"
title="Stellar-3rd-party-example"><span
class="none"></span>Stellar-3rd-party-example</a></li>
+ <li><a href="../../metron-stellar/stellar-common/index.html"
title="Stellar-common"><span
class="icon-chevron-right"></span>Stellar-common</a></li>
+ <li><a href="../../metron-stellar/stellar-zeppelin/index.html"
title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li>
+ <li><a href="../../use-cases/index.html" title="Use-cases"><span
class="icon-chevron-right"></span>Use-cases</a></li>
+ </ul>
+</li>
+</ul>
+ <hr />
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+<a href="http://maven.apache.org/"; title="Built by Maven"
class="poweredBy"><img class="builtBy" alt="Built by Maven"
src="../../images/logos/maven-feather.png" /></a>
+ </div>
</div>
</div>
-
-
- <div id="bodyColumn" class="span9" >
-
- <h1>Parsers</h1>
+ <div id="bodyColumn" class="span10" >
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<h1>Parsers</h1>
<p><a name="Parsers"></a></p>
<div class="section">
<h2><a name="Introduction"></a>Introduction</h2>
-<p>Parsers are pluggable components which are used to transform raw data
(textual or raw bytes) into JSON messages suitable for downstream enrichment
and indexing. </p>
+<p>Parsers are pluggable components which are used to transform raw data
(textual or raw bytes) into JSON messages suitable for downstream enrichment
and indexing.</p>
<p>There are two general types types of parsers:</p>
-
<ul>
-
-<li>A parser written in Java which conforms to the <tt>MessageParser</tt>
interface. This kind of parser is optimized for speed and performance and is
built for use with higher velocity topologies. These parsers are not easily
modifiable and in order to make changes to them the entire topology need to be
recompiled.</li>
-
-<li>A general purpose parser. This type of parser is primarily designed for
lower-velocity topologies or for quickly standing up a parser for a new
telemetry before a permanent Java parser can be written for it. As of the time
of this writing, we have:
-
+
+<li>A parser written in Java which conforms to the <tt>MessageParser</tt>
interface. This kind of parser is optimized for speed and performance and is
built for use with higher velocity topologies. These parsers are not easily
modifiable and in order to make changes to them the entire topology need to be
recompiled.</li>
+<li>A general purpose parser. This type of parser is primarily designed for
lower-velocity topologies or for quickly standing up a parser for a new
telemetry before a permanent Java parser can be written for it. As of the time
of this writing, we have:
<ul>
-
+
<li>Grok parser: <tt>org.apache.metron.parsers.GrokParser</tt> with possible
<tt>parserConfig</tt> entries of
-
<ul>
-
+
<li><tt>grokPath</tt> : The path in HDFS (or in the Jar) to the grok
statement</li>
-
<li><tt>patternLabel</tt> : The pattern label to use from the grok
statement</li>
-
<li><tt>timestampField</tt> : The field to use for timestamp</li>
-
<li><tt>timeFields</tt> : A list of fields to be treated as time</li>
-
<li><tt>dateFormat</tt> : The date format to use to parse the time fields</li>
-
<li><tt>timezone</tt> : The timezone to use. <tt>UTC</tt> is default.</li>
- </ul></li>
-
+</ul>
+</li>
<li>CSV Parser: <tt>org.apache.metron.parsers.csv.CSVParser</tt> with possible
<tt>parserConfig</tt> entries of
-
<ul>
-
-<li><tt>timestampFormat</tt> : The date format of the timestamp to use. If
unspecified, the parser assumes the timestamp is ms since unix epoch.</li>
-
-<li><tt>columns</tt> : A map of column names you wish to extract from the CSV
to their offsets (e.g. <tt>{ 'name' : 1, 'profession' : 3}</tt> would be a
column map for extracting the 2nd and 4th columns from a CSV)</li>
-
+
+<li><tt>timestampFormat</tt> : The date format of the timestamp to use. If
unspecified, the parser assumes the timestamp is ms since unix epoch.</li>
+<li><tt>columns</tt> : A map of column names you wish to extract from the CSV
to their offsets (e.g. <tt>{ 'name' : 1, 'profession' : 3}</tt> would be a
column map for extracting the 2nd and 4th columns from a CSV)</li>
<li><tt>separator</tt> : The column separator, <tt>,</tt> by default.</li>
- </ul></li>
-
+</ul>
+</li>
<li>JSON Map Parser: <tt>org.apache.metron.parsers.json.JSONMapParser</tt>
with possible <tt>parserConfig</tt> entries of
-
<ul>
-
-<li><tt>mapStrategy</tt> : A strategy to indicate how to handle
multi-dimensional Maps. This is one of
-
+
+<li><tt>mapStrategy</tt> : A strategy to indicate how to handle
multi-dimensional Maps. This is one of
<ul>
-
+
<li><tt>DROP</tt> : Drop fields which contain maps</li>
-
-<li><tt>UNFOLD</tt> : Unfold inner maps. So <tt>{ "foo" : {
"bar" : 1} }</tt> would turn into <tt>{"foo.bar" :
1}</tt></li>
-
+<li><tt>UNFOLD</tt> : Unfold inner maps. So <tt>{ "foo" : {
"bar" : 1} }</tt> would turn into <tt>{"foo.bar" :
1}</tt></li>
<li><tt>ALLOW</tt> : Allow multidimensional maps</li>
-
<li><tt>ERROR</tt> : Throw an error when a multidimensional map is
encountered</li>
- </ul></li>
-
+</ul>
+</li>
+<li><tt>jsonpQuery</tt> : A <a href="#json_path">JSON Path</a> query string.
If present, the result of the JSON Path query should be a list of messages.
This is useful if you have a JSON document which contains a list or array of
messages embedded in it, and you do not have another means of splitting the
message.</li>
<li>A field called <tt>timestamp</tt> is expected to exist and, if it does
not, then current time is inserted.</li>
- </ul></li>
- </ul></li>
+</ul>
+</li>
+</ul>
+</li>
</ul></div>
<div class="section">
+<h2><a name="Parser_Error_Routing"></a>Parser Error Routing</h2>
+<p>Currently, we have a few mechanisms for either deferring processing of
messages or marking messages as invalid.</p>
+<div class="section">
+<h3><a name="Invalidation_Errors"></a>Invalidation Errors</h3>
+<p>There are two reasons a message will be marked as invalid:</p>
+<ul>
+
+<li>Fail <a href="../metron-common/index.html#validation-framework">global
validation</a></li>
+<li>Fail the parser’s validate function (generally that means to not
have a <tt>timestamp</tt> field or a <tt>original_string</tt> field.</li>
+</ul>
+<p>Those messages which are marked as invalid are sent to the error queue with
an indication that they are invalid in the error message.</p></div>
+<div class="section">
+<h3><a name="Parser_Errors"></a>Parser Errors</h3>
+<p>Errors, which are defined as unexpected exceptions happening during the
parse, are sent along to the error queue with a message indicating that there
was an error in parse along with a stacktrace. This is to distinguish from the
invalid messages.</p></div></div>
+<div class="section">
+<h2><a name="Filtered"></a>Filtered</h2>
+<p>One can also filter a message by specifying a <tt>filterClassName</tt> in
the parser config. Filtered messages are just dropped rather than passed
through.</p></div>
+<div class="section">
<h2><a name="Parser_Architecture"></a>Parser Architecture</h2>
<p><img src="../../images/parser_arch.png" alt="Architecture" /></p>
-<p>Data flows through the parser bolt via kafka and into the
<tt>enrichments</tt> topology in kafka. Errors are collected with the context
of the error (e.g. stacktrace) and original message causing the error and sent
to an <tt>error</tt> queue. Invalid messages as determined by global validation
functions are also treated as errors and sent to an <tt>error</tt> queue.
</p></div>
+<p>Data flows through the parser bolt via kafka and into the
<tt>enrichments</tt> topology in kafka. Errors are collected with the context
of the error (e.g. stacktrace) and original message causing the error and sent
to an <tt>error</tt> queue. Invalid messages as determined by global
validation functions are also treated as errors and sent to an <tt>error</tt>
queue.</p></div>
<div class="section">
<h2><a name="Message_Format"></a>Message Format</h2>
-<p>All Metron messages follow a specific format in order to ingest a message.
If a message does not conform to this format it will be dropped and put onto an
error queue for further examination. The message must be of a JSON format and
must have a JSON tag message like so:</p>
+<p>All Metron messages follow a specific format in order to ingest a message.
If a message does not conform to this format it will be dropped and put onto an
error queue for further examination. The message must be of a JSON format and
must have a JSON tag message like so:</p>
-<div class="source">
-<div class="source">
-<pre>{"message" : message content}
+<div>
+<div>
+<pre class="source">{"message" : message content}
</pre></div></div>
-<p>Where appropriate there is also a standardization around the 5-tuple JSON
fields. This is done so the topology correlation engine further down stream can
correlate messages from different topologies by these fields. We are currently
working on expanding the message standardization beyond these fields, but this
feature is not yet availabe. The standard field names are as follows:</p>
+<p>Where appropriate there is also a standardization around the 5-tuple JSON
fields. This is done so the topology correlation engine further down stream
can correlate messages from different topologies by these fields. We are
currently working on expanding the message standardization beyond these fields,
but this feature is not yet availabe. The standard field names are as
follows:</p>
<ul>
-
+
<li>ip_src_addr: layer 3 source IP</li>
-
<li>ip_dst_addr: layer 3 dest IP</li>
-
<li>ip_src_port: layer 4 source port</li>
-
<li>ip_dst_port: layer 4 dest port</li>
-
<li>protocol: layer 4 protocol</li>
-
<li>timestamp (epoch)</li>
-
<li>original_string: A human friendly string representation of the message</li>
</ul>
-<p>The timestamp and original_string fields are madatory. The remaining
standard fields are optional. If any of the optional fields are not applicable
then the field should be left out of the JSON.</p>
+<p>The timestamp and original_string fields are madatory. The remaining
standard fields are optional. If any of the optional fields are not applicable
then the field should be left out of the JSON.</p>
<p>So putting it all together a typical Metron message with all 5-tuple fields
present would look like the following:</p>
-<div class="source">
-<div class="source">
-<pre>{
+<div>
+<div>
+<pre class="source">{
"message":
{"ip_src_addr": xxxx,
"ip_dst_addr": xxxx,
@@ -384,52 +216,43 @@
"additional-field 1": xxx,
}
}
-</pre></div></div></div>
+</pre></div></div>
+</div>
<div class="section">
<h2><a name="Global_Configuration"></a>Global Configuration</h2>
<p>There are a few properties which can be managed in the global configuration
that have pertinence to parsers and parsing in general.</p>
<div class="section">
<h3><a name="parser.error.topic"></a><tt>parser.error.topic</tt></h3>
<p>The topic where messages which were unable to be parsed due to error are
sent. Error messages will be indexed under a sensor type of <tt>error</tt> and
the messages will have the following fields:</p>
-
<ul>
-
+
<li><tt>sensor.type</tt>: <tt>error</tt></li>
-
<li><tt>failed_sensor_type</tt> : The sensor type of the message which
wasn’t able to be parsed</li>
-
<li><tt>error_type</tt> : The error type, in this case <tt>parser</tt>.</li>
-
<li><tt>stack</tt> : The stack trace of the error</li>
-
<li><tt>hostname</tt> : The hostname of the node where the error happened</li>
-
<li><tt>raw_message</tt> : The raw message in string form</li>
-
<li><tt>raw_message_bytes</tt> : The raw message bytes</li>
-
<li><tt>error_hash</tt> : A hash of the error message</li>
</ul></div></div>
<div class="section">
<h2><a name="Parser_Configuration"></a>Parser Configuration</h2>
<p>The configuration for the various parser topologies is defined by JSON
documents stored in zookeeper.</p>
<p>The document is structured in the following way</p>
-
<ul>
-
+
<li><tt>parserClassName</tt> : The fully qualified classname for the parser to
be used.</li>
-
-<li><tt>filterClassName</tt> : The filter to use. This may be a fully
qualified classname of a Class that implements the
<tt>org.apache.metron.parsers.interfaces.MessageFilter<JSONObject></tt>
interface. Message Filters are intended to allow the user to ignore a set of
messages via custom logic. The existing implementations are:
-
+<li><tt>filterClassName</tt> : The filter to use. This may be a fully
qualified classname of a Class that implements the
<tt>org.apache.metron.parsers.interfaces.MessageFilter<JSONObject></tt>
interface. Message Filters are intended to allow the user to ignore a set of
messages via custom logic. The existing implementations are:
<ul>
-
-<li><tt>STELLAR</tt> : Allows you to apply a stellar statement which returns a
boolean, which will pass every message for which the statement returns
<tt>true</tt>. The Stellar statement that is to be applied is specified by the
<tt>filter.query</tt> property in the <tt>parserConfig</tt>. Example Stellar
Filter which includes messages which contain a the <tt>field1</tt> field:</li>
- </ul></li>
+
+<li><tt>STELLAR</tt> : Allows you to apply a stellar statement which returns a
boolean, which will pass every message for which the statement returns
<tt>true</tt>. The Stellar statement that is to be applied is specified by the
<tt>filter.query</tt> property in the <tt>parserConfig</tt>. Example Stellar
Filter which includes messages which contain a the <tt>field1</tt> field:</li>
+</ul>
+</li>
</ul>
-<div class="source">
-<div class="source">
-<pre> {
+<div>
+<div>
+<pre class="source"> {
"filterClassName" : "STELLAR"
,"parserConfig" : {
"filter.query" : "exists(field1)"
@@ -438,127 +261,119 @@
</pre></div></div>
<ul>
-
-<li><tt>sensorTopic</tt> : The kafka topic to send the parsed messages to. If
the topic is prefixed and suffixed by <tt>/</tt> then it is assumed to be a
regex and will match any topic matching the pattern (e.g. <tt>/bro.*/</tt>
would match <tt>bro_cust0</tt>, <tt>bro_cust1</tt> and <tt>bro_cust2</tt>)</li>
-
-<li><tt>readMetadata</tt> : Boolean indicating whether to read metadata or not
(<tt>false</tt> by default). See below for a discussion about metadata.</li>
-
-<li><tt>mergeMetadata</tt> : Boolean indicating whether to merge metadata with
the message or not (<tt>false</tt> by default). See below for a discussion
about metadata.</li>
-
+
+<li><tt>sensorTopic</tt> : The kafka topic to send the parsed messages to. If
the topic is prefixed and suffixed by <tt>/</tt> then it is assumed to be a
regex and will match any topic matching the pattern (e.g. <tt>/bro.*/</tt>
would match <tt>bro_cust0</tt>, <tt>bro_cust1</tt> and <tt>bro_cust2</tt>)</li>
+<li><tt>readMetadata</tt> : Boolean indicating whether to read metadata or not
(<tt>false</tt> by default). See below for a discussion about metadata.</li>
+<li><tt>mergeMetadata</tt> : Boolean indicating whether to merge metadata with
the message or not (<tt>false</tt> by default). See below for a discussion
about metadata.</li>
<li><tt>parserConfig</tt> : A JSON Map representing the parser implementation
specific configuration.</li>
-
<li><tt>fieldTransformations</tt> : An array of complex objects representing
the transformations to be done on the message generated from the parser before
writing out to the kafka topic.</li>
-
-<li><tt>spoutParallelism</tt> : The kafka spout parallelism (default to
<tt>1</tt>). This can be overridden on the command line.</li>
-
+<li><tt>spoutParallelism</tt> : The kafka spout parallelism (default to
<tt>1</tt>). This can be overridden on the command line.</li>
<li><tt>spoutNumTasks</tt> : The number of tasks for the spout (default to
<tt>1</tt>). This can be overridden on the command line.</li>
-
<li><tt>parserParallelism</tt> : The parser bolt parallelism (default to
<tt>1</tt>). This can be overridden on the command line.</li>
-
<li><tt>parserNumTasks</tt> : The number of tasks for the parser bolt (default
to <tt>1</tt>). This can be overridden on the command line.</li>
-
<li><tt>errorWriterParallelism</tt> : The error writer bolt parallelism
(default to <tt>1</tt>). This can be overridden on the command line.</li>
-
<li><tt>errorWriterNumTasks</tt> : The number of tasks for the error writer
bolt (default to <tt>1</tt>). This can be overridden on the command line.</li>
-
<li><tt>numWorkers</tt> : The number of workers to use in the topology
(default is the storm default of <tt>1</tt>).</li>
-
<li><tt>numAckers</tt> : The number of acker executors to use in the topology
(default is the storm default of <tt>1</tt>).</li>
-
<li><tt>spoutConfig</tt> : A map representing a custom spout config (this is a
map). This can be overridden on the command line.</li>
-
-<li><tt>securityProtocol</tt> : The security protocol to use for reading from
kafka (this is a string). This can be overridden on the command line and also
specified in the spout config via the <tt>security.protocol</tt> key. If both
are specified, then they are merged and the CLI will take precedence.</li>
-
-<li><tt>stormConfig</tt> : The storm config to use (this is a map). This can
be overridden on the command line. If both are specified, they are merged with
CLI properties taking precedence.</li>
+<li><tt>securityProtocol</tt> : The security protocol to use for reading from
kafka (this is a string). This can be overridden on the command line and also
specified in the spout config via the <tt>security.protocol</tt> key. If both
are specified, then they are merged and the CLI will take precedence.</li>
+<li><tt>stormConfig</tt> : The storm config to use (this is a map). This can
be overridden on the command line. If both are specified, they are merged with
CLI properties taking precedence.</li>
+<li><tt>cacheConfig</tt> : Cache config for stellar field transformations.
This configures a least frequently used cache. This is a map with the
following keys. If not explicitly configured (the default), then no cache will
be used.
+<ul>
+
+<li><tt>stellar.cache.maxSize</tt> - The maximum number of elements in the
cache. Default is to not use a cache.</li>
+<li><tt>stellar.cache.maxTimeRetain</tt> - The maximum amount of time an
element is kept in the cache (in minutes). Default is to not use a cache.</li>
+</ul>
+<p>Example of a cache config to contain at max <tt>20000</tt> stellar
expressions for at most <tt>20</tt> minutes.:</p>
+</li>
</ul>
-<p>The <tt>fieldTransformations</tt> is a complex object which defines a
transformation which can be done to a message. This transformation can </p>
+<div>
+<div>
+<pre class="source">{
+ "cacheConfig" : {
+ "stellar.cache.maxSize" : 20000,
+ "stellar.cache.maxTimeRetain" : 20
+ }
+}
+</pre></div></div>
+
+<p>The <tt>fieldTransformations</tt> is a complex object which defines a
transformation which can be done to a message. This transformation can</p>
<ul>
-
+
<li>Modify existing fields to a message</li>
-
<li>Add new fields given the values of existing fields of a message</li>
-
<li>Remove existing fields of a message</li>
</ul>
<div class="section">
<h3><a name="Metadata"></a>Metadata</h3>
-<p>Metadata is a useful thing to send to Metron and use during enrichment or
threat intelligence.<br />Consider the following scenarios:</p>
-
+<p>Metadata is a useful thing to send to Metron and use during enrichment or
threat intelligence.<br />
+Consider the following scenarios:</p>
<ul>
-
+
<li>You have multiple telemetry sources of the same type that you want to
-
<ul>
-
+
<li>ensure downstream analysts can differentiate</li>
-
<li>ensure profiles consider independently as they have different seasonality
or some other fundamental characteristic</li>
- </ul></li>
+</ul>
+</li>
</ul>
<p>As such, there are two types of metadata that we seek to support in
Metron:</p>
-
<ul>
-
+
<li>Environmental metadata : Metadata about the system at large
-
<ul>
-
+
<li>Consider the possibility that you have multiple kafka topics being
processed by one parser and you want to tag the messages with the kafka
topic</li>
-
<li>At the moment, only the kafka topic is kept as the field name.</li>
- </ul></li>
-
+</ul>
+</li>
<li>Custom metadata: Custom metadata from an individual telemetry source that
one might want to use within Metron.</li>
</ul>
<p>Metadata is controlled by two fields in the parser:</p>
-
<ul>
-
-<li><tt>readMetadata</tt> : This is a boolean indicating whether metadata will
be read and made available to Field transformations (i.e. Stellar field
transformations). The default is <tt>false</tt>.</li>
-
-<li>
-<p><tt>mergeMetadata</tt> : This is a boolean indicating whether metadata
fields will be merged with the message automatically.<br />That is to say, if
this property is set to <tt>true</tt> then every metadata field will become
part of the messages and, consequently, also available for use in field
transformations.</p>
-<div class="section">
-<h4><a name="Field_Naming"></a>Field Naming</h4></li>
+
+<li><tt>readMetadata</tt> : This is a boolean indicating whether metadata will
be read and made available to Field transformations (i.e. Stellar field
transformations). The default is <tt>false</tt>.</li>
+<li><tt>mergeMetadata</tt> : This is a boolean indicating whether metadata
fields will be merged with the message automatically.<br />
+That is to say, if this property is set to <tt>true</tt> then every metadata
field will become part of the messages and, consequently, also available for
use in field transformations.</li>
</ul>
-<p>In order to avoid collisions from metadata fields, metadata fields will be
prefixed with <tt>metron.metadata.</tt>.<br />So, for instance the kafka topic
would be in the field <tt>metron.metadata.topic</tt>.</p></div>
+<div class="section">
+<h4><a name="Field_Naming"></a>Field Naming</h4>
+<p>In order to avoid collisions from metadata fields, metadata fields will be
prefixed with <tt>metron.metadata.</tt>.<br />
+So, for instance the kafka topic would be in the field
<tt>metron.metadata.topic</tt>.</p></div>
<div class="section">
<h4><a name="Specifying_Custom_Metadata"></a>Specifying Custom Metadata</h4>
-<p>Custom metadata is specified by sending a JSON Map in the key. If no key is
sent, then, obviously, no metadata will be parsed. For instance, sending a
metadata field called <tt>customer_id</tt> could be done by sending</p>
+<p>Custom metadata is specified by sending a JSON Map in the key. If no key
is sent, then, obviously, no metadata will be parsed. For instance, sending a
metadata field called <tt>customer_id</tt> could be done by sending</p>
-<div class="source">
-<div class="source">
-<pre>{
+<div>
+<div>
+<pre class="source">{
"customer_id" : "my_customer_id"
}
</pre></div></div>
-<p>in the kafka key. This would be exposed as the field
<tt>metron.metadata.customer_id</tt> to stellar field transformations as well,
if <tt>mergeMetadata</tt> is <tt>true</tt>, available as a field in its own
right.</p></div></div>
+
+<p>in the kafka key. This would be exposed as the field
<tt>metron.metadata.customer_id</tt> to stellar field transformations as well,
if <tt>mergeMetadata</tt> is <tt>true</tt>, available as a field in its own
right.</p></div></div>
<div class="section">
<h3><a
name="fieldTransformation_configuration"></a><tt>fieldTransformation</tt>
configuration</h3>
<p>The format of a <tt>fieldTransformation</tt> is as follows:</p>
-
<ul>
-
-<li><tt>input</tt> : An array of fields or a single field representing the
input. This is optional; if unspecified, then the whole message is passed as
input.</li>
-
-<li><tt>output</tt> : The outputs to produce from the transformation. If
unspecified, it is assumed to be the same as inputs.</li>
-
-<li><tt>transformation</tt> : The fully qualified classname of the
transformation to be used. This is either a class which implements
<tt>FieldTransformation</tt> or a member of the <tt>FieldTransformations</tt>
enum.</li>
-
+
+<li><tt>input</tt> : An array of fields or a single field representing the
input. This is optional; if unspecified, then the whole message is passed as
input.</li>
+<li><tt>output</tt> : The outputs to produce from the transformation. If
unspecified, it is assumed to be the same as inputs.</li>
+<li><tt>transformation</tt> : The fully qualified classname of the
transformation to be used. This is either a class which implements
<tt>FieldTransformation</tt> or a member of the <tt>FieldTransformations</tt>
enum.</li>
<li><tt>config</tt> : A String to Object map of transformation specific
configuration.</li>
</ul>
<p>The currently implemented fieldTransformations are:</p>
-
<ul>
-
-<li><tt>REMOVE</tt> : This transformation removes the specified input fields.
If you want a conditional removal, you can pass a Metron Query Language
statement to define the conditions under which you want to remove the
fields.</li>
+
+<li><tt>REMOVE</tt> : This transformation removes the specified input fields.
If you want a conditional removal, you can pass a Metron Query Language
statement to define the conditions under which you want to remove the
fields.</li>
</ul>
<p>Consider the following simple configuration which will remove
<tt>field1</tt> unconditionally:</p>
-<div class="source">
-<div class="source">
-<pre>{
+<div>
+<div>
+<pre class="source">{
...
"fieldTransformations" : [
{
@@ -568,11 +383,12 @@
]
}
</pre></div></div>
+
<p>Consider the following simple sensor parser configuration which will remove
<tt>field1</tt> whenever <tt>field2</tt> exists and whose corresponding equal
to ‘foo’:</p>
-<div class="source">
-<div class="source">
-<pre>{
+<div>
+<div>
+<pre class="source">{
...
"fieldTransformations" : [
{
@@ -587,14 +403,34 @@
</pre></div></div>
<ul>
-
+
+<li><tt>SELECT</tt>: This transformation filters the fields in the message to
include only the configured output fields, and drops any not explicitly
included.</li>
+</ul>
+<p>For example:</p>
+
+<div>
+<div>
+<pre class="source">{
+...
+ "fieldTransformations" : [
+ {
+ "output" : ["field1", "field2" ]
+ , "transformation" : "SELECT"
+ }
+ ]
+}
+</pre></div></div>
+
+<p>when applied to a message containing keys field1, field2 and field3, will
only output the first two. It is also worth noting that two standard fields -
timestamp and original_source - will always be passed along whether they are
listed in output or not, since they are considered core required fields.</p>
+<ul>
+
<li><tt>IP_PROTOCOL</tt> : This transformation maps IANA protocol numbers to
consistent string representations.</li>
</ul>
<p>Consider the following sensor parser config to map the <tt>protocol</tt>
field to a textual representation of the protocol:</p>
-<div class="source">
-<div class="source">
-<pre>{
+<div>
+<div>
+<pre class="source">{
...
"fieldTransformations" : [
{
@@ -604,20 +440,45 @@
]
}
</pre></div></div>
-<p>This transformation would transform <tt>{ "protocol" : 6,
"source.type" : "bro", ... }</tt> into <tt>{
"protocol" : "TCP", "source.type" :
"bro", ...}</tt></p>
+<p>This transformation would transform <tt>{ "protocol" : 6,
"source.type" : "bro", ... }</tt> into <tt>{
"protocol" : "TCP", "source.type" :
"bro", ...}</tt></p>
<ul>
-
-<li><tt>STELLAR</tt> : This transformation executes a set of transformations
expressed as <a href="../metron-common/index.html">Stellar Language</a>
statements.</li>
-</ul></div>
+
+<li>
+
+<p><tt>STELLAR</tt> : This transformation executes a set of transformations
expressed as <a href="../metron-common/index.html">Stellar Language</a>
statements.</p>
+</li>
+<li>
+
+<p><tt>RENAME</tt> : This transformation allows users to rename a set of
fields. Specifically, the config is presumed to be the mapping. The keys to
the config are the existing field names and the values for the config map are
the associated new field name.</p>
+</li>
+</ul>
+<p>The following config will rename the fields <tt>old_field</tt> and
<tt>different_old_field</tt> to <tt>new_field</tt> and
<tt>different_new_field</tt> respectively:</p>
+
+<div>
+<div>
+<pre class="source">{
+...
+ "fieldTransformations" : [
+ {
+ "transformation" : "RENAME",
+ , "config" : {
+ "old_field" : "new_field",
+ "different_old_field" : "different_new_field"
+ }
+ }
+ ]
+}
+</pre></div></div>
+</div>
<div class="section">
<h3><a name="Assignment_to_null"></a>Assignment to <tt>null</tt></h3>
-<p>If, in your field transformation, you assign a field to <tt>null</tt>, the
field will be removed. You can use this capability to rename variables.</p>
+<p>If, in your field transformation, you assign a field to <tt>null</tt>, the
field will be removed. You can use this capability to rename variables. It is
preferred, however, that the <tt>RENAME</tt> field transformation is used in
this situation as it is less awkward.</p>
<p>Consider this example:</p>
-<div class="source">
-<div class="source">
-<pre> "fieldTransformations" : [
+<div>
+<div>
+<pre class="source"> "fieldTransformations" : [
{ "transformation" : "STELLAR"
,"output" : [ "new_field", "old_field"]
,"config" : {
@@ -627,15 +488,16 @@
}
]
</pre></div></div>
+
<p>This would set <tt>new_field</tt> to the value of <tt>old_field</tt> and
remove <tt>old_field</tt>.</p></div>
<div class="section">
<h3><a name="Warning:_Transforming_the_same_field_twice"></a>Warning:
Transforming the same field twice</h3>
-<p>Currently, the stellar expressions are expressed in the form of a map where
the keys define the fields and the values define the Stellar expressions. You
order the expression evaluation in the <tt>output</tt> field. A consequence of
this choice to store the assignments as a map is that the same field cannot
appear in the map as a key twice.</p>
+<p>Currently, the stellar expressions are expressed in the form of a map where
the keys define the fields and the values define the Stellar expressions. You
order the expression evaluation in the <tt>output</tt> field. A consequence of
this choice to store the assignments as a map is that the same field cannot
appear in the map as a key twice.</p>
<p>For instance, the following will not function as expected:</p>
-<div class="source">
-<div class="source">
-<pre> "fieldTransformations" : [
+<div>
+<div>
+<pre class="source"> "fieldTransformations" : [
{ "transformation" : "STELLAR"
,"output" : [ "new_field"]
,"config" : {
@@ -645,23 +507,21 @@
}
]
</pre></div></div>
+
<p>In the above example, the last instance of <tt>new_field</tt> will win and
<tt>TO_LOWER(new_field)</tt> will be evaluated while <tt>TO_UPPER(field1)</tt>
will be skipped.</p></div>
<div class="section">
<h3><a name="Example"></a>Example</h3>
<p>Consider the following sensor parser config to add three new fields to a
message:</p>
-
<ul>
-
+
<li><tt>utc_timestamp</tt> : The unix epoch timestamp based on the
<tt>timestamp</tt> field, a <tt>dc</tt> field which is the data center the
message comes from and a <tt>dc2tz</tt> map mapping data centers to
timezones</li>
-
<li><tt>url_host</tt> : The host associated with the url in the <tt>url</tt>
field</li>
-
<li><tt>url_protocol</tt> : The protocol associated with the url in the
<tt>url</tt> field</li>
</ul>
-<div class="source">
-<div class="source">
-<pre>{
+<div>
+<div>
+<pre class="source">{
...
"fieldTransformations" : [
{
@@ -684,14 +544,15 @@ HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )"
}
}
</pre></div></div>
+
<p>Note that the <tt>dc2tz</tt> map is in the parser config, so it is
accessible in the functions.</p></div>
<div class="section">
<h3><a name="An_Example_Configuration_for_a_Sensor"></a>An Example
Configuration for a Sensor</h3>
<p>Consider the following example configuration for the <tt>yaf</tt>
sensor:</p>
-<div class="source">
-<div class="source">
-<pre>{
+<div>
+<div>
+<pre class="source">{
"parserClassName":"org.apache.metron.parsers.GrokParser",
"sensorTopic":"yaf",
"fieldTransformations" : [
@@ -709,42 +570,38 @@ HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )"
"dateFormat":"yyyy-MM-dd HH:mm:ss.S"
}
}
-</pre></div></div></div></div>
+</pre></div></div>
+</div></div>
<div class="section">
<h2><a name="Parser_Adapters"></a>Parser Adapters</h2>
-<p>Parser adapters are loaded dynamically in each Metron topology. They are
defined in the Parser Config (defined above) JSON file in Zookeeper.</p>
+<p>Parser adapters are loaded dynamically in each Metron topology. They are
defined in the Parser Config (defined above) JSON file in Zookeeper.</p>
<div class="section">
<h3><a name="Java_Parser_Adapters"></a>Java Parser Adapters</h3>
-<p>Java parser adapters are indended for higher-velocity topologies and are
not easily changed or extended. As the adoption of Metron continues we plan on
extending our library of Java adapters to process more log formats. As of this
moment the Java adapters included with Metron are:</p>
-
+<p>Java parser adapters are indended for higher-velocity topologies and are
not easily changed or extended. As the adoption of Metron continues we plan on
extending our library of Java adapters to process more log formats. As of this
moment the Java adapters included with Metron are:</p>
<ul>
-
+
<li>org.apache.metron.parsers.ise.BasicIseParser : Parse ISE messages</li>
-
<li>org.apache.metron.parsers.bro.BasicBroParser : Parse Bro messages</li>
-
<li>org.apache.metron.parsers.sourcefire.BasicSourcefireParser : Parse
Sourcefire messages</li>
-
<li>org.apache.metron.parsers.lancope.BasicLancopeParser : Parse Lancope
messages</li>
</ul></div>
<div class="section">
<h3><a name="Grok_Parser_Adapters"></a>Grok Parser Adapters</h3>
-<p>Grok parser adapters are designed primarly for someone who is not a Java
coder for quickly standing up a parser adapter for lower velocity topologies.
Grok relies on Regex for message parsing, which is much slower than
purpose-built Java parsers, but is more extensible. Grok parsers are defined
via a config file and the topplogy does not need to be recombiled in order to
make changes to them. An example of a Grok perser is:</p>
-
+<p>Grok parser adapters are designed primarly for someone who is not a Java
coder for quickly standing up a parser adapter for lower velocity topologies.
Grok relies on Regex for message parsing, which is much slower than
purpose-built Java parsers, but is more extensible. Grok parsers are defined
via a config file and the topplogy does not need to be recombiled in order to
make changes to them. An example of a Grok perser is:</p>
<ul>
-
+
<li>org.apache.metron.parsers.GrokParser</li>
</ul>
<p>For more information on the Grok project please refer to the following
link:</p>
<p><a class="externalLink"
href="https://github.com/thekrakken/java-grok";>https://github.com/thekrakken/java-grok</a></p>
<p><a name="Starting_the_Parser_Topology"></a></p>
<h1>Starting the Parser Topology</h1>
-<p>Starting a particular parser topology on a running Metron deployment is as
easy as running the <tt>start_parser_topology.sh</tt> script located in
<tt>$METRON_HOME/bin</tt>. This utility will allow you to configure and start
the running topology assuming that the sensor specific parser configuration
exists within zookeeper.</p>
+<p>Starting a particular parser topology on a running Metron deployment is as
easy as running the <tt>start_parser_topology.sh</tt> script located in
<tt>$METRON_HOME/bin</tt>. This utility will allow you to configure and start
the running topology assuming that the sensor specific parser configuration
exists within zookeeper.</p>
<p>The usage for <tt>start_parser_topology.sh</tt> is as follows:</p>
-<div class="source">
-<div class="source">
-<pre>usage: start_parser_topology.sh
+<div>
+<div>
+<pre class="source">usage: start_parser_topology.sh
-e,--extra_topology_options <JSON_FILE> Extra options in
the form
of a JSON file with a
map
for content.
@@ -780,136 +637,129 @@ HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )"
-t,--test <TEST> Run in Test Mode
-z,--zk <ZK_QUORUM> Zookeeper Quroum
URL
(zk1:2181,zk2:2181,...
-</pre></div></div></div></div>
+</pre></div></div>
+</div></div>
<div class="section">
<h2><a name="The_--extra_kafka_spout_config_Option"></a>The
<tt>--extra_kafka_spout_config</tt> Option</h2>
-<p>These options are intended to configure the Storm Kafka Spout more
completely. These options can be specified in a JSON file containing a map
associating the kafka spout configuration parameter to a value. The range of
values possible to configure are:</p>
-
+<p>These options are intended to configure the Storm Kafka Spout more
completely. These options can be specified in a JSON file containing a map
associating the kafka spout configuration parameter to a value. The range of
values possible to configure are:</p>
<ul>
-
-<li><tt>spout.pollTimeoutMs</tt> - Specifies the time, in milliseconds, spent
waiting in poll if data is not available. Default is 2s</li>
-
-<li><tt>spout.firstPollOffsetStrategy</tt> - Sets the offset used by the Kafka
spout in the first poll to Kafka broker upon process start. One of
-
+
+<li><tt>spout.pollTimeoutMs</tt> - Specifies the time, in milliseconds, spent
waiting in poll if data is not available. Default is 2s</li>
+<li><tt>spout.firstPollOffsetStrategy</tt> - Sets the offset used by the Kafka
spout in the first poll to Kafka broker upon process start. One of
<ul>
-
+
<li><tt>EARLIEST</tt></li>
-
<li><tt>LATEST</tt></li>
-
<li><tt>UNCOMMITTED_EARLIEST</tt> - Last uncommitted and if offsets
aren’t found, defaults to earliest. NOTE: This is the default.</li>
-
<li><tt>UNCOMMITTED_LATEST</tt> - Last uncommitted and if offsets
aren’t found, defaults to latest.</li>
- </ul></li>
-
+</ul>
+</li>
<li><tt>spout.offsetCommitPeriodMs</tt> - Specifies the period, in
milliseconds, the offset commit task is periodically called. Default is
15s.</li>
-
<li><tt>spout.maxUncommittedOffsets</tt> - Defines the max number of polled
offsets (records) that can be pending commit, before another poll can take
place. Once this limit is reached, no more offsets (records) can be polled
until the next successful commit(s) sets the number of pending offsets bellow
the threshold. The default is 10,000,000.</li>
-
-<li><tt>spout.maxRetries</tt> - Defines the max number of retrials in case of
tuple failure. The default is to retry forever, which means that no new records
are committed until the previous polled records have been acked. This
guarantees at once delivery of all the previously polled records. By specifying
a finite value for maxRetries, the user decides to sacrifice guarantee of
delivery for the previous polled records in favor of processing more
records.</li>
-
+<li><tt>spout.maxRetries</tt> - Defines the max number of retrials in case of
tuple failure. The default is to retry forever, which means that no new records
are committed until the previous polled records have been acked. This
guarantees at once delivery of all the previously polled records. By
specifying a finite value for maxRetries, the user decides to sacrifice
guarantee of delivery for the previous polled records in favor of processing
more records.</li>
<li>Any of the configs in the Consumer API for <a class="externalLink"
href="http://kafka.apache.org/0100/documentation.html#newconsumerconfigs";>Kafka
0.10.x</a></li>
</ul>
<p>For instance, creating a JSON file which will set the offsets to
<tt>UNCOMMITTED_EARLIEST</tt></p>
-<div class="source">
-<div class="source">
-<pre>{
+<div>
+<div>
+<pre class="source">{
"spout.firstPollOffsetStrategy" : "UNCOMMITTED_EARLIEST"
}
</pre></div></div>
+
<p>This would be loaded by passing the file as argument to
<tt>--extra_kafka_spout_config</tt></p></div>
<div class="section">
<h2><a name="The_--extra_topology_options_Option"></a>The
<tt>--extra_topology_options</tt> Option</h2>
-<p>These options are intended to be Storm configuration options and will live
in a JSON file which will be loaded into the Storm config. For instance, if you
wanted to set a storm property on the config called
<tt>topology.ticks.tuple.freq.secs</tt> to 1000 and <tt>storm.local.dir</tt> to
<tt>/opt/my/path</tt> you could create a file called
<tt>custom_config.json</tt> containing </p>
+<p>These options are intended to be Storm configuration options and will live
in a JSON file which will be loaded into the Storm config. For instance, if
you wanted to set a storm property on the config called
<tt>topology.ticks.tuple.freq.secs</tt> to 1000 and <tt>storm.local.dir</tt> to
<tt>/opt/my/path</tt> you could create a file called
<tt>custom_config.json</tt> containing</p>
-<div class="source">
-<div class="source">
-<pre>{
+<div>
+<div>
+<pre class="source">{
"topology.ticks.tuple.freq.secs" : 1000,
"storm.local.dir" : "/opt/my/path"
}
</pre></div></div>
+
<p>and pass <tt>--extra_topology_options custom_config.json</tt> to
<tt>start_parser_topology.sh</tt>.</p>
<p><a name="Notes_on_Performance_Tuning"></a></p>
<h1>Notes on Performance Tuning</h1>
-<p>Default installed Metron is untuned for production deployment. There are a
few knobs to tune to get the most out of your system.</p>
+<p>Default installed Metron is untuned for production deployment. There are a
few knobs to tune to get the most out of your system.</p>
<p><a name="Notes_on_Adding_a_New_Sensor"></a></p>
<h1>Notes on Adding a New Sensor</h1>
<p>In order to allow for meta alerts to be queries alongside regular alerts in
Elasticsearch 2.x, it is necessary to add an additional field to the templates
and mapping for existing sensors.</p>
<p>Please see a description of the steps necessary to make this change in the
metron-elasticsearch <a
href="../../metron-platform/metron-elasticsearch/index.html#Using_Metron_with_Elasticsearch_2.x">Using
Metron with Elasticsearch 2.x</a></p></div>
<div class="section">
<h2><a name="Kafka_Queue"></a>Kafka Queue</h2>
-<p>The kafka queue associated with your parser is a collection point for all
of the data sent to your parser. As such, make sure that the number of
partitions in the kafka topic is sufficient to handle the throughput that you
expect from your parser topology.</p></div>
+<p>The kafka queue associated with your parser is a collection point for all
of the data sent to your parser. As such, make sure that the number of
partitions in the kafka topic is sufficient to handle the throughput that you
expect from your parser topology.</p></div>
<div class="section">
<h2><a name="Parser_Topology"></a>Parser Topology</h2>
-<p>The enrichment topology as started by the
<tt>$METRON_HOME/bin/start_parser_topology.sh</tt> script uses a default of one
executor per bolt. In a real production system, this should be customized by
modifying the arguments sent to this utility.</p>
-
+<p>The enrichment topology as started by the
<tt>$METRON_HOME/bin/start_parser_topology.sh</tt> script uses a default of one
executor per bolt. In a real production system, this should be customized by
modifying the arguments sent to this utility.</p>
<ul>
-
+
<li>Topology Wide
-
<ul>
-
+
<li><tt>--num_workers</tt> : The number of workers for the topology</li>
-
<li><tt>--num_ackers</tt> : The number of ackers for the topology</li>
- </ul></li>
-
+</ul>
+</li>
<li>The Kafka Spout
-
<ul>
-
+
<li><tt>--spout_num_tasks</tt> : The number of tasks for the spout</li>
-
<li><tt>--spout_p</tt> : The parallelism hint for the spout</li>
-
<li>Ensure that the spout has enough parallelism so that it can dedicate a
worker per partition in your kafka topic.</li>
- </ul></li>
-
+</ul>
+</li>
<li>The Parser Bolt
-
<ul>
-
+
<li><tt>--parser_num_tasks</tt> : The number of tasks for the parser bolt</li>
-
<li><tt>--parser_p</tt> : The parallelism hint for the spout</li>
-
<li>This is bolt that gets the most processing, so ensure that it is
configured with sufficient parallelism to match your throughput
expectations.</li>
- </ul></li>
-
+</ul>
+</li>
<li>The Error Message Writer Bolt
-
<ul>
-
+
<li><tt>--error_writer_num_tasks</tt> : The number of tasks for the error
writer bolt</li>
-
<li><tt>--error_writer_p</tt> : The parallelism hint for the error writer
bolt</li>
- </ul></li>
+</ul>
+</li>
</ul>
<p>Finally, if workers and executors are new to you, the following might be of
use to you:</p>
-
<ul>
-
+
<li><a class="externalLink"
href="http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/";>Understanding
the Parallelism of a Storm Topology</a></li>
</ul></div>
- </div>
- </div>
- </div>
+<div class="section">
+<h2><a name="JSON_Path"></a>JSON Path</h2>
+<blockquote>
- <hr/>
+<dl>
+<dt>“JSONPath expressions always refer to a JSON structure in the same
way as XPath expression are used in combination with an XML
document.”</dt>
+<dd>Stefan Goessner</dd>
+</dl>
+</blockquote>
+<ul>
+
+<li><a class="externalLink" href="http://goessner.net/articles/JsonPath/";>JSON
Path concept</a></li>
+<li><a class="externalLink" href="https://github.com/json-path/JsonPath";>Read
about JSON Path library Apache Metron uses</a></li>
+<li><a class="externalLink" href="http://jsonpath.herokuapp.com";>Try JSON Path
expressions online</a></li>
+</ul></div>
+ </div>
+ </div>
+ </div>
+ <hr/>
<footer>
- <div class="container-fluid">
- <div class="row span12">Copyright © 2018
- <a href="https://www.apache.org";>The Apache Software
Foundation</a>.
- All Rights Reserved.
-
+ <div class="container-fluid">
+ <div class="row-fluid">
+é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache,
the Apache feather logo,
+ and the Apache Metron project logo are trademarks of The Apache
Software Foundation.
+ </div>
</div>
-
-
-
- </div>
</footer>
</body>
</html>
