http://git-wip-us.apache.org/repos/asf/metron/blob/f7a94f2e/site/current-book/metron-platform/metron-pcap-backend/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-pcap-backend/index.html b/site/current-book/metron-platform/metron-pcap-backend/index.html index a36dc50..af673d5 100644 --- a/site/current-book/metron-platform/metron-pcap-backend/index.html +++ b/site/current-book/metron-platform/metron-pcap-backend/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-02-23 + | Generated by Apache Maven Doxia at 2017-06-27 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170223" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Metron PCAP Backend</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -30,14 +30,11 @@ <div class="container-fluid"> <div id="banner"> <div class="pull-left"> - <a href="http://metron.incubator.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> </a> </div> - <div class="pull-right"> <a href="http://incubator.apache.org/"; id="bannerRight"> - <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/> - </a> - </div> + <div class="pull-right"> </div> <div class="clear"><hr/></div> </div> @@ -51,8 +48,8 @@ </li> <li class="divider ">/</li> <li class=""> - <a href="http://metron.incubator.apache.org/"; class="externalLink" title="Metron-Incubating"> - Metron-Incubating</a> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> </li> <li class="divider ">/</li> <li class=""> @@ -64,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.3.1</li> + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> </ul> </div> @@ -78,7 +75,7 @@ <ul class="nav nav-list"> <li class="nav-header">User Documentation</li> - + <li> <a href="../../index.html" title="Metron"> @@ -99,7 +96,7 @@ <i class="icon-chevron-right"></i> Analytics</a> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -113,7 +110,21 @@ <i class="none"></i> Docker</a> </li> - + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + <li> <a href="../../metron-platform/index.html" title="Platform"> @@ -127,13 +138,13 @@ <i class="none"></i> Api</a> </li> - + <li> <a href="../../metron-platform/metron-common/index.html" title="Common"> - <i class="none"></i> + <i class="icon-chevron-right"></i> Common</a> - </li> + </li> <li> @@ -174,9 +185,16 @@ <a href="#"><i class="none"></i>Pcap-backend</a> </li> + + <li> + + <a href="../../metron-platform/metron-writer/index.html" title="Writer"> + <i class="none"></i> + Writer</a> + </li> </ul> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> @@ -207,7 +225,31 @@ <h1>Metron PCAP Backend</h1> <p><a name="Metron_PCAP_Backend"></a></p> -<p>The purpose of the Metron PCAP backend is to create a storm topology capable of ingesting rapidly raw packet capture data directly into HDFS from Kafka.</p> +<p>The purpose of the Metron PCAP backend is to create a storm topology capable of rapidly ingesting raw packet capture data directly into HDFS from Kafka.</p> + +<ul> + +<li><a href="#the-sensors-feeding-kafka">Sensors</a></li> + +<li><a href="#the-pcap-topology">PCAP Topology</a></li> + +<li><a href="#the-files-on-hdfs">HDFS Files</a></li> + +<li><a href="#Configuration">Configuration</a></li> + +<li><a href="#Starting_the_Topology">Starting the Topology</a></li> + +<li><a href="#Utilities">Utilities</a> + +<ul> + +<li><a href="#Inspector_Utility">Inspector Utility</a></li> + +<li><a href="#Query_Filter_Utility">Query Filter Utility</a></li> + </ul></li> + +<li><a href="#Performance_Tuning">Performance Tuning</a></li> +</ul> <div class="section"> <h2><a name="The_Sensors_Feeding_Kafka"></a>The Sensors Feeding Kafka</h2> <p>This component must be fed by fast packet capture components upstream via Kafka. The two supported components shipped with Metron are as follows:</p> @@ -257,15 +299,19 @@ <p>These files contain a set of packet data with headers on them in sequence files.</p></div> <div class="section"> <h2><a name="Configuration"></a>Configuration</h2> -<p>The configuration file for the Flux topology is located at <tt>$METRON_HOME/config/etc/env/pcap.properties</tt> and the possible options are as follows:</p> +<p>The configuration file for the Flux topology is located at <tt>$METRON_HOME/config/pcap.properties</tt> and the possible options are as follows:</p> <ul> <li><tt>spout.kafka.topic.pcap</tt> : The kafka topic to listen to</li> +<li><tt>storm.auto.credentials</tt> : The kerberos ticket renewal. If running on a kerberized cluster, this should be <tt>['org.apache.storm.security.auth.kerberos.AutoTGT']</tt></li> + +<li><tt>kafka.security.protocol</tt> : The security protocol to use for kafka. This should be <tt>PLAINTEXT</tt> for a non-kerberized cluster and probably <tt>SASL_PLAINTEXT</tt> for a kerberized cluster.</li> + <li><tt>kafka.zk</tt> : The comma separated zookeeper quorum (i.e. host:2181,host2:2181)</li> -<li><tt>kafka.pcap.start</tt> : One of <tt>START</tt>, <tt>END</tt>, <tt>WHERE_I_LEFT_OFF</tt> representing where to start listening on the queue.</li> +<li><tt>kafka.pcap.start</tt> : One of <tt>EARLIEST</tt>, <tt>LATEST</tt>, <tt>UNCOMMITTED_EARLIEST</tt>, <tt>UNCOMMITTED_LATEST</tt> representing where to start listening on the queue.</li> <li><tt>kafka.pcap.numPackets</tt> : The number of packets to keep in one file.</li> @@ -301,7 +347,7 @@ <li>fixed</li> -<li>query (Metron Stellar)</li> +<li>query (via Stellar)</li> </ul> <p>The tool is executed via </p> @@ -324,6 +370,7 @@ and end_time. Default is to use time in millis since the epoch. -dp,--ip_dst_port <arg> Destination port + -pf,--packet_filter <arg> Packet filter regex -et,--end_time <arg> Packet end time range. Default is current system time. -nr,--num_reducers <arg> The number of reducers to use. Default @@ -354,7 +401,217 @@ -h,--help Display help -q,--query <arg> Query string to use as a filter -st,--start_time <arg> (required) Packet start time range. -</pre></div></div></div></div></div> +</pre></div></div> +<p>The Query filter’s <tt>--query</tt> argument specifies the Stellar expression to execute on each packet. To interact with the packet, a few variables are exposed:</p> + +<ul> + +<li><tt>packet</tt> : The packet data (a <tt>byte[]</tt>)</li> + +<li><tt>ip_src_addr</tt> : The source address for the packet (a <tt>String</tt>)</li> + +<li><tt>ip_src_port</tt> : The source port for the packet (an <tt>Integer</tt>)</li> + +<li><tt>ip_dst_addr</tt> : The destination address for the packet (a <tt>String</tt>)</li> + +<li><tt>ip_dst_port</tt> : The destination port for the packet (an <tt>Integer</tt>)</li> +</ul></div> +<div class="section"> +<h4><a name="Binary_Regex"></a>Binary Regex</h4> +<p>Filtering can be done both by the packet header as well as via a binary regular expression which can be run on the packet payload itself. This filter can be specified via:</p> + +<ul> + +<li>The <tt>-pf</tt> or <tt>--packet_filter</tt> options for the fixed query filter</li> + +<li>The <tt>BYTEARRAY_MATCHER(pattern, data)</tt> Stellar function. The first argument is the regex pattern and the second argument is the data. The packet data will be exposed via the<tt>packet</tt> variable in Stellar.</li> +</ul> +<p>The format of this regular expression is described <a class="externalLink" href="https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md";>here</a>.</p></div></div></div> +<div class="section"> +<h2><a name="Performance_Tuning"></a>Performance Tuning</h2> +<p>The PCAP topology is extremely lightweight and functions as a Spout-only topology. In order to tune the topology, users currently must specify a combination of properties in pcap.properties as well as configuration in the pcap remote.yaml flux file itself. Tuning the number of partitions in your Kafka topic will have a dramatic impact on performance as well. We ran data into Kafka at 1.1 Gbps and our tests resulted in configuring 128 partitions for our kakfa topic along with the following settings in pcap.properties and remote.yaml (unrelated properties for performance have been removed):</p> +<div class="section"> +<h3><a name="pcap.properties_file"></a>pcap.properties file</h3> + +<div class="source"> +<div class="source"> +<pre>spout.kafka.topic.pcap=pcap +storm.topology.workers=16 +kafka.spout.parallelism=128 +kafka.pcap.numPackets=1000000000 +kafka.pcap.maxTimeMS=0 +hdfs.replication=1 +hdfs.sync.every=10000 +</pre></div></div> +<p>You’ll notice that the number of kakfa partitions equals the spout parallelism, and this is no coincidence. The ordering guarantees for a partition in Kafka enforces that you may have no more consumers than 1 per topic. Any additional parallelism will leave you with dormant threads consuming resources but performing no additional work. For our cluster with 4 Storm Supervisors, we found 16 workers to provide optimal throughput as well. We were largely IO bound rather than CPU bound with the incoming PCAP data.</p></div> +<div class="section"> +<h3><a name="remote.yaml"></a>remote.yaml</h3> +<p>In the flux file, we introduced the following configuration:</p> + +<div class="source"> +<div class="source"> +<pre>name: "pcap" +config: + topology.workers: ${storm.topology.workers} + topology.worker.childopts: ${topology.worker.childopts} + topology.auto-credentials: ${storm.auto.credentials} + topology.ackers.executors: 0 +components: + + # Any kafka props for the producer go here. + - id: "kafkaProps" + className: "java.util.HashMap" + configMethods: + - name: "put" + args: + - "value.deserializer" + - "org.apache.kafka.common.serialization.ByteArrayDeserializer" + - name: "put" + args: + - "key.deserializer" + - "org.apache.kafka.common.serialization.ByteArrayDeserializer" + - name: "put" + args: + - "group.id" + - "pcap" + - name: "put" + args: + - "security.protocol" + - "${kafka.security.protocol}" + - name: "put" + args: + - "poll.timeout.ms" + - 100 + - name: "put" + args: + - "offset.commit.period.ms" + - 30000 + - name: "put" + args: + - "session.timeout.ms" + - 30000 + - name: "put" + args: + - "max.uncommitted.offsets" + - 200000000 + - name: "put" + args: + - "max.poll.interval.ms" + - 10 + - name: "put" + args: + - "max.poll.records" + - 200000 + - name: "put" + args: + - "receive.buffer.bytes" + - 431072 + - name: "put" + args: + - "max.partition.fetch.bytes" + - 8097152 + + - id: "hdfsProps" + className: "java.util.HashMap" + configMethods: + - name: "put" + args: + - "io.file.buffer.size" + - 1000000 + - name: "put" + args: + - "dfs.blocksize" + - 1073741824 + + - id: "kafkaConfig" + className: "org.apache.metron.storm.kafka.flux.SimpleStormKafkaBuilder" + constructorArgs: + - ref: "kafkaProps" + # topic name + - "${spout.kafka.topic.pcap}" + - "${kafka.zk}" + configMethods: + - name: "setFirstPollOffsetStrategy" + args: + # One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST + - ${kafka.pcap.start} + + - id: "writerConfig" + className: "org.apache.metron.spout.pcap.HDFSWriterConfig" + configMethods: + - name: "withOutputPath" + args: + - "${kafka.pcap.out}" + - name: "withNumPackets" + args: + - ${kafka.pcap.numPackets} + - name: "withMaxTimeMS" + args: + - ${kafka.pcap.maxTimeMS} + - name: "withZookeeperQuorum" + args: + - "${kafka.zk}" + - name: "withSyncEvery" + args: + - ${hdfs.sync.every} + - name: "withReplicationFactor" + args: + - ${hdfs.replication} + - name: "withHDFSConfig" + args: + - ref: "hdfsProps" + - name: "withDeserializer" + args: + - "${kafka.pcap.ts_scheme}" + - "${kafka.pcap.ts_granularity}" +spouts: + - id: "kafkaSpout" + className: "org.apache.metron.spout.pcap.KafkaToHDFSSpout" + parallelism: ${kafka.spout.parallelism} + constructorArgs: + - ref: "kafkaConfig" + - ref: "writerConfig" + +</pre></div></div> +<div class="section"> +<h4><a name="Flux_Changes_Introduced"></a>Flux Changes Introduced</h4> +<div class="section"> +<h5><a name="Topology_Configuration"></a>Topology Configuration</h5> +<p>The only change here is <tt>topology.ackers.executors: 0</tt>, which disables Storm tuple acking for maximum throughput.</p></div> +<div class="section"> +<h5><a name="Kafka_configuration"></a>Kafka configuration</h5> + +<div class="source"> +<div class="source"> +<pre>poll.timeout.ms +offset.commit.period.ms +session.timeout.ms +max.uncommitted.offsets +max.poll.interval.ms +max.poll.records +receive.buffer.bytes +max.partition.fetch.bytes +</pre></div></div></div> +<div class="section"> +<h5><a name="Writer_Configuration"></a>Writer Configuration</h5> +<p>This is a combination of settings for the HDFSWriter (see pcap.properties values above) as well as HDFS.</p> +<p><b>HDFS config</b></p> +<p>Component config HashMap with the following properties:</p> + +<div class="source"> +<div class="source"> +<pre>io.file.buffer.size +dfs.blocksize +</pre></div></div> +<p><b>Writer config</b></p> +<p>References the HDFS props component specified above.</p> + +<div class="source"> +<div class="source"> +<pre> - name: "withHDFSConfig" + args: + - ref: "hdfsProps" +</pre></div></div></div></div></div></div> </div> </div> </div> @@ -363,8 +620,9 @@ <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017. - All Rights Reserved. + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. </div>
http://git-wip-us.apache.org/repos/asf/metron/blob/f7a94f2e/site/current-book/metron-platform/metron-writer/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-writer/index.html b/site/current-book/metron-platform/metron-writer/index.html new file mode 100644 index 0000000..ad36a4f --- /dev/null +++ b/site/current-book/metron-platform/metron-writer/index.html @@ -0,0 +1,321 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-06-27 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Writer</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Writer</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-down"></i> + Platform</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../metron-platform/metron-api/index.html" title="Api"> + <i class="none"></i> + Api</a> + </li> + + <li> + + <a href="../../metron-platform/metron-common/index.html" title="Common"> + <i class="icon-chevron-right"></i> + Common</a> + </li> + + <li> + + <a href="../../metron-platform/metron-data-management/index.html" title="Data-management"> + <i class="none"></i> + Data-management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment"> + <i class="none"></i> + Enrichment</a> + </li> + + <li> + + <a href="../../metron-platform/metron-indexing/index.html" title="Indexing"> + <i class="none"></i> + Indexing</a> + </li> + + <li> + + <a href="../../metron-platform/metron-management/index.html" title="Management"> + <i class="none"></i> + Management</a> + </li> + + <li> + + <a href="../../metron-platform/metron-parsers/index.html" title="Parsers"> + <i class="none"></i> + Parsers</a> + </li> + + <li> + + <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend"> + <i class="none"></i> + Pcap-backend</a> + </li> + + <li class="active"> + + <a href="#"><i class="none"></i>Writer</a> + </li> + </ul> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-right"></i> + Sensors</a> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <!-- Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to You under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. --><h1>Writer</h1> +<p><a name="Writer"></a></p> +<div class="section"> +<h2><a name="Introduction"></a>Introduction</h2> +<p>The writer module provides some utilties for writing to outside components from within Storm. This includes managing bulk writing. An implemention is included for writing to HDFS in this module. Other writers can be found in their own modules.</p></div> +<div class="section"> +<h2><a name="HDFS_Writer"></a>HDFS Writer</h2> +<p>The HDFS writer included here expands on what Storm has in several ways. There’s customization in syncing to HDFS, rotation policy, etc. In addition, the writer allows for users to define output paths based on the fields in the provided JSON message. This can be defined using Stellar.</p> +<p>To manage the output path, a base path argument is provided by the Flux file, with the FileNameFormat as follows</p> + +<div class="source"> +<div class="source"> +<pre> - id: "fileNameFormat" + className: "org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat" + configMethods: + - name: "withPrefix" + args: + - "enrichment-" + - name: "withExtension" + args: + - ".json" + - name: "withPath" + args: + - "/apps/metron/" +</pre></div></div> +<p>This means that all output will land in <tt>/apps/metron/</tt>. With no further adjustment, it will be <tt>/apps/metron/<sensor>/</tt>. However, by modifying the sensor’s JSON config, it is possible to provide additional pathing based on the the message itself.</p> +<p>E.g.</p> + +<div class="source"> +<div class="source"> +<pre>{ + "index": "bro", + "batchSize": 5, + "outputPathFunction": "FORMAT('uid-%s', uid)" +} +</pre></div></div> +<p>will land data in <tt>/apps/metron/uid-<uid>/</tt>.</p> +<p>For example, if the data contains uid’s 1, 3, and 5, there will be 3 output folders in HDFS:</p> + +<div class="source"> +<div class="source"> +<pre>/apps/metron/uid-1/ +/apps/metron/uid-3/ +/apps/metron/uid-5/ +</pre></div></div> +<p>The Stellar function must return a String, but is not limited to FORMAT functions. Other functions, such as <tt>TO_LOWER</tt>, <tt>TO_UPPER</tt>, etc. are all available for use. Typically, it’s preferable to do nontrivial transformations as part of enrichment and simply reference the output here.</p> +<p>If no Stellar function is provided, it will default to putting the sensor in a folder, as above.</p> +<p>A caveat is that the writer will only allow a certain number of files to be created at once. HdfsWriter has a function <tt>withMaxOpenFiles</tt> allowing this to be set. The default is 500. This can be set in Flux:</p> + +<div class="source"> +<div class="source"> +<pre> - id: "hdfsWriter" + className: "org.apache.metron.writer.hdfs.HdfsWriter" + configMethods: + - name: "withFileNameFormat" + args: + - ref: "fileNameFormat" + - name: "withRotationPolicy" + args: + - ref: "hdfsRotationPolicy" + - name: "withMaxOpenFiles" + args: 500 +</pre></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html> http://git-wip-us.apache.org/repos/asf/metron/blob/f7a94f2e/site/current-book/metron-sensors/bro-plugin-kafka/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-sensors/bro-plugin-kafka/index.html b/site/current-book/metron-sensors/bro-plugin-kafka/index.html new file mode 100644 index 0000000..6a9654a --- /dev/null +++ b/site/current-book/metron-sensors/bro-plugin-kafka/index.html @@ -0,0 +1,491 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia at 2017-06-27 + | Rendered using Apache Maven Fluido Skin 1.3.0 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20170627" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – Logging Bro Output to Kafka</title> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/site.css" /> + <link rel="stylesheet" href="../../css/print.css" media="print" /> + + + <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> + + + +<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> + + </head> + <body class="topBarDisabled"> + + + + + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"> + <a href="http://metron.apache.org/"; id="bannerLeft"> + <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> + </a> + </div> + <div class="pull-right"> </div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + + + <li class=""> + <a href="http://www.apache.org"; class="externalLink" title="Apache"> + Apache</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> + Metron</a> + </li> + <li class="divider ">/</li> + <li class=""> + <a href="../../index.html" title="Documentation"> + Documentation</a> + </li> + <li class="divider ">/</li> + <li class="">Logging Bro Output to Kafka</li> + + + + <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.0</li> + + </ul> + </div> + + + <div class="row-fluid"> + <div id="leftColumn" class="span3"> + <div class="well sidebar-nav"> + + + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + + <li> + + <a href="../../index.html" title="Metron"> + <i class="icon-chevron-down"></i> + Metron</a> + <ul class="nav nav-list"> + + <li> + + <a href="../../Upgrading.html" title="Upgrading"> + <i class="none"></i> + Upgrading</a> + </li> + + <li> + + <a href="../../metron-analytics/index.html" title="Analytics"> + <i class="icon-chevron-right"></i> + Analytics</a> + </li> + + <li> + + <a href="../../metron-deployment/index.html" title="Deployment"> + <i class="icon-chevron-right"></i> + Deployment</a> + </li> + + <li> + + <a href="../../metron-docker/index.html" title="Docker"> + <i class="none"></i> + Docker</a> + </li> + + <li> + + <a href="../../metron-interface/metron-config/index.html" title="Config"> + <i class="none"></i> + Config</a> + </li> + + <li> + + <a href="../../metron-interface/metron-rest/index.html" title="Rest"> + <i class="none"></i> + Rest</a> + </li> + + <li> + + <a href="../../metron-platform/index.html" title="Platform"> + <i class="icon-chevron-right"></i> + Platform</a> + </li> + + <li> + + <a href="../../metron-sensors/index.html" title="Sensors"> + <i class="icon-chevron-down"></i> + Sensors</a> + <ul class="nav nav-list"> + + <li class="active"> + + <a href="#"><i class="none"></i>Bro-plugin-kafka</a> + </li> + + <li> + + <a href="../../metron-sensors/fastcapa/index.html" title="Fastcapa"> + <i class="none"></i> + Fastcapa</a> + </li> + + <li> + + <a href="../../metron-sensors/pycapa/index.html" title="Pycapa"> + <i class="none"></i> + Pycapa</a> + </li> + </ul> + </li> + </ul> + </li> + </ul> + + + + <hr class="divider" /> + + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> + <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> + </a> + </div> + </div> + </div> + + + <div id="bodyColumn" class="span9" > + + <h1>Logging Bro Output to Kafka</h1> +<p>A Bro log writer that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, to process the data generated by Bro.</p> +<div class="section"> +<h2><a name="Installation"></a>Installation</h2> + +<ol style="list-style-type: decimal"> + +<li> +<p>Install <a class="externalLink" href="https://github.com/edenhill/librdkafka";>librdkafka</a>, a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.9.4.</p> +<p>In order to use this plugin within a kerberized Kafka environment, you will also need <tt>libsasl2</tt> installed and will need to pass <tt>--enable-sasl</tt> to the <tt>configure</tt> script.</p> + +<div class="source"> +<div class="source"> +<pre>curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar xvz +cd librdkafka-0.9.4/ +./configure --enable-sasl +make +sudo make install +</pre></div></div></li> + +<li> +<p>Build the plugin using the following commands.</p> + +<div class="source"> +<div class="source"> +<pre>./configure --bro-dist=$BRO_SRC +make +sudo make install +</pre></div></div></li> + +<li> +<p>Run the following command to ensure that the plugin was installed successfully.</p> + +<div class="source"> +<div class="source"> +<pre>$ bro -N Bro::Kafka +Bro::Kafka - Writes logs to Kafka (dynamic, version 0.1) +</pre></div></div></li> +</ol></div> +<div class="section"> +<h2><a name="Activation"></a>Activation</h2> +<p>The following examples highlight different ways that the plugin can be used. Simply add the Bro script language to your <tt>local.bro</tt> file (for example, <tt>/usr/share/bro/site/local.bro</tt>) as shown to demonstrate the example.</p> +<div class="section"> +<h3><a name="Example_1"></a>Example 1</h3> +<p>The goal in this example is to send all HTTP and DNS records to a Kafka topic named <tt>bro</tt>. </p> + +<ul> + +<li>Any configuration value accepted by librdkafka can be added to the <tt>kafka_conf</tt> configuration table.</li> + +<li>By defining <tt>topic_name</tt> all records will be sent to the same Kafka topic.</li> + +<li>Defining <tt>logs_to_send</tt> will ensure that only HTTP and DNS records are sent.</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>@load Bro/Kafka/logs-to-kafka.bro +redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); +redef Kafka::topic_name = "bro"; +redef Kafka::kafka_conf = table( + ["metadata.broker.list"] = "localhost:9092" +); +</pre></div></div></div> +<div class="section"> +<h3><a name="Example_2"></a>Example 2</h3> +<p>It is also possible to send each log stream to a uniquely named topic. The goal in this example is to send all HTTP records to a Kafka topic named <tt>http</tt> and all DNS records to a separate Kafka topic named <tt>dns</tt>.</p> + +<ul> + +<li>The <tt>topic_name</tt> value must be set to an empty string.</li> + +<li>The <tt>$path</tt> value of Bro’s Log Writer mechanism is used to define the topic name.</li> + +<li>Any configuration value accepted by librdkafka can be added to the <tt>$config</tt> configuration table.</li> + +<li>Each log writer accepts a separate configuration table.</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>@load Bro/Kafka/logs-to-kafka.bro +redef Kafka::topic_name = ""; +redef Kafka::tag_json = T; + +event bro_init() +{ + # handles HTTP + local http_filter: Log::Filter = [ + $name = "kafka-http", + $writer = Log::WRITER_KAFKAWRITER, + $config = table( + ["metadata.broker.list"] = "localhost:9092" + ), + $path = "http" + ]; + Log::add_filter(HTTP::LOG, http_filter); + + # handles DNS + local dns_filter: Log::Filter = [ + $name = "kafka-dns", + $writer = Log::WRITER_KAFKAWRITER, + $config = table( + ["metadata.broker.list"] = "localhost:9092" + ), + $path = "dns" + ]; + Log::add_filter(DNS::LOG, dns_filter); +} +</pre></div></div></div> +<div class="section"> +<h3><a name="Example_3"></a>Example 3</h3> +<p>You may want to configure bro to filter log messages with certain characteristics from being sent to your kafka topics. For instance, Metron currently doesn’t support IPv6 source or destination IPs in the default enrichments, so it may be helpful to filter those log messages from being sent to kafka (although there are <a href="#notes">multiple ways</a> to approach this). In this example we will do that that, and are assuming a somewhat standard bro kafka plugin configuration, such that:</p> + +<ul> + +<li>All bro logs are sent to the <tt>bro</tt> topic, by configuring <tt>Kafka::topic_name</tt>.</li> + +<li>Each JSON message is tagged with the appropriate log type (such as <tt>http</tt>, <tt>dns</tt>, or <tt>conn</tt>), by setting <tt>tag_json</tt> to true.</li> + +<li>If the log message contains a 128 byte long source or destination IP address, the log is not sent to kafka.</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>@load Bro/Kafka/logs-to-kafka.bro +redef Kafka::topic_name = "bro"; +redef Kafka::tag_json = T; + +event bro_init() &priority=-5 +{ + # handles HTTP + Log::add_filter(HTTP::LOG, [ + $name = "kafka-http", + $writer = Log::WRITER_KAFKAWRITER, + $pred(rec: HTTP::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); }, + $config = table( + ["metadata.broker.list"] = "localhost:9092" + ) + ]); + + # handles DNS + Log::add_filter(DNS::LOG, [ + $name = "kafka-dns", + $writer = Log::WRITER_KAFKAWRITER, + $pred(rec: DNS::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); }, + $config = table( + ["metadata.broker.list"] = "localhost:9092" + ) + ]); + + # handles Conn + Log::add_filter(Conn::LOG, [ + $name = "kafka-conn", + $writer = Log::WRITER_KAFKAWRITER, + $pred(rec: Conn::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); }, + $config = table( + ["metadata.broker.list"] = "localhost:9092" + ) + ]); +} +</pre></div></div> +<div class="section"> +<h4><a name="Notes"></a>Notes</h4> + +<ul> + +<li><tt>logs_to_send</tt> is mutually exclusive with <tt>$pred</tt>, thus for each log you want to set <tt>$pred</tt> on, you must individually setup a <tt>Log::add_filter</tt> and refrain from including that log in <tt>logs_to_send</tt>.</li> + +<li>You can also filter IPv6 logs from within your Metron cluster <a href="../../metron-platform/metron-common/index.html#IS_IP">using Stellar</a>. In that case, you wouldn’t apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron.</li> + +<li>It is also possible to use the <tt>is_v6_subnet()</tt> bro function in your predicate, as of their <a class="externalLink" href="https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-5";>2.5 release</a>, however the above example should work on <a class="externalLink" href="https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-4";>bro 2.4</a> and newer, which has been the focus of the kafka plugin.</li> +</ul></div></div></div> +<div class="section"> +<h2><a name="Settings"></a>Settings</h2> +<div class="section"> +<h3><a name="kafka_conf"></a><tt>kafka_conf</tt></h3> +<p>The global configuration settings for Kafka. These values are passed through directly to librdkafka. Any valid librdkafka settings can be defined in this table. The full set of valid librdkafka settings are available <a class="externalLink" href="https://github.com/edenhill/librdkafka/blob/v0.9.4/CONFIGURATION.md";>here</a>.</p> + +<div class="source"> +<div class="source"> +<pre>redef Kafka::kafka_conf = table( + ["metadata.broker.list"] = "localhost:9092", + ["client.id"] = "bro" +); +</pre></div></div></div> +<div class="section"> +<h3><a name="topic_name"></a><tt>topic_name</tt></h3> +<p>The name of the topic in Kafka where all Bro logs will be sent to.</p> + +<div class="source"> +<div class="source"> +<pre>redef Kafka::topic_name = "bro"; +</pre></div></div></div> +<div class="section"> +<h3><a name="max_wait_on_shutdown"></a><tt>max_wait_on_shutdown</tt></h3> +<p>The maximum number of milliseconds that the plugin will wait for any backlog of queued messages to be sent to Kafka before forced shutdown.</p> + +<div class="source"> +<div class="source"> +<pre>redef Kafka::max_wait_on_shutdown = 3000; +</pre></div></div></div> +<div class="section"> +<h3><a name="tag_json"></a><tt>tag_json</tt></h3> +<p>If true, a log stream identifier is appended to each JSON-formatted message. For example, a Conn::LOG message will look like <tt>{ 'conn' : { ... }}</tt>.</p> + +<div class="source"> +<div class="source"> +<pre>redef Kafka::tag_json = T; +</pre></div></div></div> +<div class="section"> +<h3><a name="debug"></a><tt>debug</tt></h3> +<p>A comma separated list of debug contexts in librdkafka which you want to enable. The available contexts are:</p> + +<ul> + +<li>generic</li> + +<li>broker</li> + +<li>topic</li> + +<li>metadata</li> + +<li>queue</li> + +<li>msg</li> + +<li>protocol</li> + +<li>cgrp</li> + +<li>security</li> + +<li>fetch</li> + +<li>feature</li> + +<li>all</li> +</ul></div></div> +<div class="section"> +<h2><a name="Kerberos"></a>Kerberos</h2> +<p>This plugin supports producing messages from a kerberized kafka. There are a couple of prerequisites and a couple of settings to set. </p> +<div class="section"> +<h3><a name="SASL"></a>SASL</h3> +<p>If you are using SASL as a security protocol for kafka, then you must have libsasl or libsasl2 installed. You can tell if sasl is enabled by running the following from the directory in which you have build librdkafka:</p> + +<div class="source"> +<div class="source"> +<pre>examples/rdkafka_example -X builtin.features +builtin.features = gzip,snappy,ssl,sasl,regex +</pre></div></div></div> +<div class="section"> +<h3><a name="Producer_Config"></a>Producer Config</h3> +<p>As stated above, you can configure the producer kafka configs in <tt>${BRO_HOME}/share/bro/site/local.bro</tt>. There are a few configs necessary to set, which are described <a class="externalLink" href="https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka";>here</a>. For an environment where the following is true:</p> + +<ul> + +<li>The broker is <tt>node1:6667</tt></li> + +<li>This kafka is using <tt>SASL_PLAINTEXT</tt> as the security protocol</li> + +<li>The keytab used is the <tt>metron</tt> keytab</li> + +<li>The service principal for <tt>metron</tt> is <tt>met...@example.com</tt></li> +</ul> +<p>The kafka topic <tt>bro</tt> has been given permission for the <tt>metron</tt> user to write:</p> + +<div class="source"> +<div class="source"> +<pre># login using the metron user +kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com +${KAFKA_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:metron --topic bro +</pre></div></div> +<p>The following is how the <tt>${BRO_HOME}/share/bro/site/local.bro</tt> looks:</p> + +<div class="source"> +<div class="source"> +<pre>@load Bro/Kafka/logs-to-kafka.bro +redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); +redef Kafka::topic_name = "bro"; +redef Kafka::tag_json = T; +redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "node1:6667" + , ["security.protocol"] = "SASL_PLAINTEXT" + , ["sasl.kerberos.keytab"] = "/etc/security/keytabs/metron.headless.keytab" + , ["sasl.kerberos.principal"] = "met...@example.com" + ); +</pre></div></div></div></div> + </div> + </div> + </div> + + <hr/> + + <footer> + <div class="container-fluid"> + <div class="row span12">Copyright © 2017 + <a href="https://www.apache.org";>The Apache Software Foundation</a>. + All Rights Reserved. + + </div> + + + + </div> + </footer> + </body> +</html>
