http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-sensors/pycapa/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-sensors/pycapa/index.html b/site/current-book/metron-sensors/pycapa/index.html index 8c46464..f81aee9 100644 --- a/site/current-book/metron-sensors/pycapa/index.html +++ b/site/current-book/metron-sensors/pycapa/index.html @@ -1,313 +1,199 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2018-01-03 - | Rendered using Apache Maven Fluido Skin 1.3.0 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-sensors/pycapa/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180103" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Pycapa</title> - <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> <link rel="stylesheet" href="../../css/site.css" /> <link rel="stylesheet" href="../../css/print.css" media="print" /> - - - <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> - - - -<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> - - </head> - <body class="topBarDisabled"> - - - - - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"> - <a href="http://metron.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> - </a> - </div> - <div class="pull-right"> </div> + <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> - - - <li class=""> - <a href="http://www.apache.org"; class="externalLink" title="Apache"> - Apache</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> - Metron</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="../../index.html" title="Documentation"> - Documentation</a> - </li> - <li class="divider ">/</li> - <li class="">Pycapa</li> - - - - <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.2</li> - - </ul> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Pycapa</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> </div> - - <div class="row-fluid"> - <div id="leftColumn" class="span3"> + <div id="leftColumn" class="span2"> <div class="well sidebar-nav"> - - - <ul class="nav nav-list"> - <li class="nav-header">User Documentation</li> - - <li> - - <a href="../../index.html" title="Metron"> - <i class="icon-chevron-down"></i> - Metron</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../Upgrading.html" title="Upgrading"> - <i class="none"></i> - Upgrading</a> - </li> - - <li> - - <a href="../../metron-analytics/index.html" title="Analytics"> - <i class="icon-chevron-right"></i> - Analytics</a> - </li> - - <li> - - <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> - <i class="none"></i> - Docker</a> - </li> - - <li> - - <a href="../../metron-deployment/index.html" title="Deployment"> - <i class="icon-chevron-right"></i> - Deployment</a> - </li> - - <li> - - <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> - <i class="none"></i> - Alerts</a> - </li> - - <li> - - <a href="../../metron-interface/metron-config/index.html" title="Config"> - <i class="none"></i> - Config</a> - </li> - - <li> - - <a href="../../metron-interface/metron-rest/index.html" title="Rest"> - <i class="none"></i> - Rest</a> - </li> - - <li> - - <a href="../../metron-platform/index.html" title="Platform"> - <i class="icon-chevron-right"></i> - Platform</a> - </li> - - <li> - - <a href="../../metron-sensors/index.html" title="Sensors"> - <i class="icon-chevron-down"></i> - Sensors</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../metron-sensors/fastcapa/index.html" title="Fastcapa"> - <i class="none"></i> - Fastcapa</a> - </li> - - <li class="active"> - - <a href="#"><i class="none"></i>Pycapa</a> - </li> - </ul> - </li> - - <li> - - <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> - <i class="none"></i> - Stellar-3rd-party-example</a> - </li> - - <li> - - <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> - <i class="icon-chevron-right"></i> - Stellar-common</a> - </li> - - <li> - - <a href="../../use-cases/index.html" title="Use-cases"> - <i class="icon-chevron-right"></i> - Use-cases</a> - </li> - </ul> - </li> - </ul> - - - - <hr class="divider" /> - - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> - </a> - </div> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> + <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-down"></span>Sensors</a> + <ul class="nav nav-list"> + <li><a href="../../metron-sensors/fastcapa/index.html" title="Fastcapa"><span class="none"></span>Fastcapa</a></li> + <li class="active"><a href="#"><span class="none"></span>Pycapa</a></li> + </ul> +</li> + <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a> + </div> </div> </div> - - - <div id="bodyColumn" class="span9" > - - <div class="section"> -<h2><a name="Pycapa"></a>Pycapa</h2> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<div class="section"> +<h2><a name="Pycapa"></a>Pycapa</h2> <ul> - + <li><a href="#Overview">Overview</a></li> - <li><a href="#Installation">Installation</a></li> - <li><a href="#Usage">Usage</a> - <ul> - + <li><a href="#Parameters">Parameters</a></li> - <li><a href="#Examples">Examples</a></li> - <li><a href="#Kerberos">Kerberos</a></li> - </ul></li> - +</ul> +</li> <li><a href="#FAQs">FAQs</a></li> </ul> <h1>Overview</h1> -<p>Pycapa performs network packet capture, both off-the-wire and from a Kafka topic, which is useful for the testing and development of <a class="externalLink" href="https://github.com/apache/metron";>Apache Metron</a>. It is not intended for production use. The tool will capture packets from a specified interface and push them into a Kafka Topic. The tool can also do the reverse. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat";>libpcap-compliant file</a> or even to feed directly into a tool like Wireshark to monitor ongoing activity.</p> +<p>Pycapa performs network packet capture, both off-the-wire and from a Kafka topic, which is useful for the testing and development of <a class="externalLink" href="https://github.com/apache/metron";>Apache Metron</a>. It is not intended for production use. The tool will capture packets from a specified interface and push them into a Kafka Topic. The tool can also do the reverse. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat";>libpcap-compliant file</a> or even to feed directly into a tool like Wireshark to monitor ongoing activity.</p> <h1>Installation</h1> <p>General notes on the installation of Pycapa.</p> - <ul> - + <li>Python 2.7 is required.</li> - <li>The following package dependencies are required and can be installed automatically with <tt>pip</tt>. - <ul> - + <li><a class="externalLink" href="https://github.com/confluentinc/confluent-kafka-python";>confluent-kafka-python</a></li> - <li><a class="externalLink" href="https://github.com/CoreSecurity/pcapy";>pcapy</a></li> - </ul></li> - +</ul> +</li> <li>These instructions can be used directly on CentOS 7+.</li> - <li>Other Linux distributions that come with Python 2.7 can use these instructions with some minor modifications.</li> - <li>Older distributions, like CentOS 6, that come with Python 2.6 installed, should install Python 2.7 within a virtual environment and then run Pycapa from within the virtual environment.</li> </ul> - <ol style="list-style-type: decimal"> - + <li> -<p>Install system dependencies including the core development tools, Python libraries and header files, and Libpcap libraries and header files. On CentOS 7+, you can install these requirements with the following command.</p> - -<div class="source"> -<div class="source"> -<pre> yum -y install "@Development tools" python-devel libpcap-devel -</pre></div></div></li> - + +<p>Install system dependencies including the core development tools, Python libraries and header files, and Libpcap libraries and header files. On CentOS 7+, you can install these requirements with the following command.</p> + +<div> +<div> +<pre class="source">yum -y install "@Development tools" python-devel libpcap-devel +</pre></div></div> +</li> <li> + <p>Install Librdkafka at your chosen $PREFIX.</p> - -<div class="source"> -<div class="source"> -<pre> export PREFIX=/usr - wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz - cd librdkafka-0.9.4/ - ./configure --prefix=$PREFIX - make - make install -</pre></div></div></li> - + +<div> +<div> +<pre class="source">export PREFIX=/usr +wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz +cd librdkafka-0.9.4/ +./configure --prefix=$PREFIX +make +make install +</pre></div></div> +</li> <li> + <p>Add Librdkafka to the dynamic library load path.</p> - -<div class="source"> -<div class="source"> -<pre>echo "$PREFIX/lib" >> /etc/ld.so.conf.d/pycapa.conf + +<div> +<div> +<pre class="source">echo "$PREFIX/lib" >> /etc/ld.so.conf.d/pycapa.conf ldconfig -v -</pre></div></div></li> - +</pre></div></div> +</li> <li> -<p>Install Pycapa. This assumes that you already have the Metron source code on the host.</p> - -<div class="source"> -<div class="source"> -<pre>cd metron/metron-sensors/pycapa + +<p>Install Pycapa. This assumes that you already have the Metron source code on the host.</p> + +<div> +<div> +<pre class="source">cd metron/metron-sensors/pycapa pip install -r requirements.txt python setup.py install -</pre></div></div></li> +</pre></div></div> +</li> </ol> <h1>Usage</h1> <p>Pycapa has two primary runtime modes.</p> - <ul> - + <li> -<p><b>Producer Mode</b>: Pycapa can capture packets from a network interface and forward those packets to a Kafka topic. Pycapa embeds the raw network packet data in the Kafka message body. The message key contains the timestamp indicating when the packet was captured in microseconds from the epoch, in network byte order.</p></li> - + +<p><b>Producer Mode</b>: Pycapa can capture packets from a network interface and forward those packets to a Kafka topic. Pycapa embeds the raw network packet data in the Kafka message body. The message key contains the timestamp indicating when the packet was captured in microseconds from the epoch, in network byte order.</p> +</li> <li> -<p><b>Consumer Mode</b>: Pycapa can also perform the reverse operation. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat";>libpcap-compliant file</a> or even to feed directly into a tool like Wireshark to monitor activity.</p></li> + +<p><b>Consumer Mode</b>: Pycapa can also perform the reverse operation. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat";>libpcap-compliant file</a> or even to feed directly into a tool like Wireshark to monitor activity.</p> +</li> </ul> <div class="section"> <h3><a name="Parameters"></a>Parameters</h3> -<div class="source"> -<div class="source"> -<pre>$ pycapa --help +<div> +<div> +<pre class="source">$ pycapa --help usage: pycapa [-h] [-p] [-c] [-k KAFKA_BROKERS] [-t KAFKA_TOPIC] [-o {begin,end,stored}] [-i NETWORK_IFACE] [-m MAX_PACKETS] [-pp PRETTY_PRINT] [-ll LOG_LEVEL] [-X KAFKA_CONFIGS] @@ -335,16 +221,17 @@ optional arguments: -s SNAPLEN, --snaplen SNAPLEN capture only the first X bytes of each packet; default=65535 -</pre></div></div></div> +</pre></div></div> +</div> <div class="section"> <h3><a name="Examples"></a>Examples</h3> <div class="section"> <h4><a name="Example_1"></a>Example 1</h4> -<p>Capture 10 packets from the <tt>eth0</tt> network interface and forward those to a Kafka topic called <tt>pcap</tt> running on <tt>localhost:9092</tt>. The process will not terminate until all messages have been delivered to Kafka.</p> +<p>Capture 10 packets from the <tt>eth0</tt> network interface and forward those to a Kafka topic called <tt>pcap</tt> running on <tt>localhost:9092</tt>. The process will not terminate until all messages have been delivered to Kafka.</p> -<div class="source"> -<div class="source"> -<pre>$ pycapa --producer \ +<div> +<div> +<pre class="source">$ pycapa --producer \ --interface eth0 \ --kafka-broker localhost:9092 \ --kafka-topic pcap \ @@ -353,14 +240,15 @@ INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id INFO:root:Starting packet capture INFO:root:Waiting for '6' message(s) to flush INFO:root:'10' packet(s) in, '10' packet(s) out -</pre></div></div></div> +</pre></div></div> +</div> <div class="section"> <h4><a name="Example_2"></a>Example 2</h4> -<p>Capture packets until SIGINT is received (the interrupt signal sent when entering CTRL-C in the console.) In this example, nothing will be reported as packets are captured and delivered to Kafka. Simply wait a few seconds, then type CTRL-C and the number of packets will be reported.</p> +<p>Capture packets until SIGINT is received (the interrupt signal sent when entering CTRL-C in the console.) In this example, nothing will be reported as packets are captured and delivered to Kafka. Simply wait a few seconds, then type CTRL-C and the number of packets will be reported.</p> -<div class="source"> -<div class="source"> -<pre>$ pycapa --producer \ +<div> +<div> +<pre class="source">$ pycapa --producer \ --interface en0 \ --kafka-broker localhost:9092 \ --kafka-topic pcap @@ -370,14 +258,15 @@ INFO:root:Starting packet capture INFO:root:Clean shutdown process started INFO:root:Waiting for '2' message(s) to flush INFO:root:'21' packet(s) in, '21' packet(s) out -</pre></div></div></div> +</pre></div></div> +</div> <div class="section"> <h4><a name="Example_3"></a>Example 3</h4> -<p>While capturing packets, output diagnostic information every 5 packets. Diagnostics will report when packets have been received from the network interface and when they have been successfully delivered to Kafka.</p> +<p>While capturing packets, output diagnostic information every 5 packets. Diagnostics will report when packets have been received from the network interface and when they have been successfully delivered to Kafka.</p> -<div class="source"> -<div class="source"> -<pre>$ pycapa --producer \ +<div> +<div> +<pre class="source">$ pycapa --producer \ --interface eth0 \ --kafka-broker localhost:9092 \ --kafka-topic pcap \ @@ -399,37 +288,39 @@ INFO:root:'21' packet(s) in, '21' packet(s) out INFO:root:Waiting for '1' message(s) to flush INFO:root:'27' packet(s) in, '27' packet(s) out -</pre></div></div></div> +</pre></div></div> +</div> <div class="section"> <h4><a name="Example_4"></a>Example 4</h4> <p>Consume 10 packets and create a libpcap-compliant pcap file.</p> -<div class="source"> -<div class="source"> -<pre> $ pycapa --consumer \ - --kafka-broker localhost:9092 \ - --kafka-topic pcap \ - --max-packets 10 \ - > out.pcap - $ tshark -r out.pcap - 1 0.000000 199.193.204.147 → 192.168.0.3 TLSv1.2 151 Application Data - 2 0.000005 199.193.204.147 → 192.168.0.3 TLSv1.2 1191 Application Data - 3 0.000088 192.168.0.3 → 199.193.204.147 TCP 66 54788 → 443 [ACK] Seq=1 Ack=86 Win=4093 Len=0 TSval=961284465 TSecr=943744612 - 4 0.000089 192.168.0.3 → 199.193.204.147 TCP 66 54788 → 443 [ACK] Seq=1 Ack=1211 Win=4058 Len=0 TSval=961284465 TSecr=943744612 - 5 0.948788 192.168.0.3 → 192.30.253.125 TCP 54 54671 → 443 [ACK] Seq=1 Ack=1 Win=4096 Len=0 - 6 1.005175 192.30.253.125 → 192.168.0.3 TCP 66 [TCP ACKed unseen segment] 443 → 54671 [ACK] Seq=1 Ack=2 Win=31 Len=0 TSval=2658544467 TSecr=961240339 - 7 1.636312 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df - 8 2.253052 192.175.27.112 → 192.168.0.3 TLSv1.2 928 Application Data - 9 2.253140 192.168.0.3 → 192.175.27.112 TCP 66 55078 → 443 [ACK] Seq=1 Ack=863 Win=4069 Len=0 TSval=961286699 TSecr=967172238 - 10 2.494769 192.168.0.3 → 224.0.0.251 MDNS 82 Standard query 0x0000 PTR _googlecast._tcp.local, "QM" question -</pre></div></div></div> +<div> +<div> +<pre class="source"> $ pycapa --consumer \ + --kafka-broker localhost:9092 \ + --kafka-topic pcap \ + --max-packets 10 \ + > out.pcap + $ tshark -r out.pcap + 1 0.000000 199.193.204.147 → 192.168.0.3 TLSv1.2 151 Application Data + 2 0.000005 199.193.204.147 → 192.168.0.3 TLSv1.2 1191 Application Data + 3 0.000088 192.168.0.3 → 199.193.204.147 TCP 66 54788 → 443 [ACK] Seq=1 Ack=86 Win=4093 Len=0 TSval=961284465 TSecr=943744612 + 4 0.000089 192.168.0.3 → 199.193.204.147 TCP 66 54788 → 443 [ACK] Seq=1 Ack=1211 Win=4058 Len=0 TSval=961284465 TSecr=943744612 + 5 0.948788 192.168.0.3 → 192.30.253.125 TCP 54 54671 → 443 [ACK] Seq=1 Ack=1 Win=4096 Len=0 + 6 1.005175 192.30.253.125 → 192.168.0.3 TCP 66 [TCP ACKed unseen segment] 443 → 54671 [ACK] Seq=1 Ack=2 Win=31 Len=0 TSval=2658544467 TSecr=961240339 + 7 1.636312 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df + 8 2.253052 192.175.27.112 → 192.168.0.3 TLSv1.2 928 Application Data + 9 2.253140 192.168.0.3 → 192.175.27.112 TCP 66 55078 → 443 [ACK] Seq=1 Ack=863 Win=4069 Len=0 TSval=961286699 TSecr=967172238 + 10 2.494769 192.168.0.3 → 224.0.0.251 MDNS 82 Standard query 0x0000 PTR _googlecast._tcp.local, "QM" question +</pre></div></div> +</div> <div class="section"> <h4><a name="Example_5"></a>Example 5</h4> <p>Consume 10 packets from the Kafka topic <tt>pcap</tt> running on <tt>localhost:9092</tt>, then pipe those into Wireshark for DPI.</p> -<div class="source"> -<div class="source"> -<pre>$ pycapa --consumer \ +<div> +<div> +<pre class="source">$ pycapa --consumer \ --kafka-broker localhost:9092 \ --kafka-topic pcap \ --max-packets 10 \ @@ -446,59 +337,58 @@ Capturing on 'Standard input' 9 2.083872 192.30.253.125 → 192.168.0.3 TCP 66 443 → 54671 [ACK] Seq=32 Ack=36 Win=31 Len=0 TSval=2658503087 TSecr=961120495 10 3.173189 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df 10 packets captured -</pre></div></div></div></div> +</pre></div></div> +</div></div> <div class="section"> <h3><a name="Kerberos"></a>Kerberos</h3> -<p>The probe can be used in a Kerberized environment. Follow these additional steps to use Pycapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment.</p> - +<p>The probe can be used in a Kerberized environment. Follow these additional steps to use Pycapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment.</p> <ul> - + <li>The Kafka broker is at <tt>kafka1:6667</tt></li> - <li>Zookeeper is at <tt>zookeeper1:2181</tt></li> - <li>The Kafka security protocol is <tt>SASL_PLAINTEXT</tt></li> - <li>The keytab used is located at <tt>/etc/security/keytabs/metron.headless.keytab</tt></li> - <li>The service principal is <tt>met...@example.com</tt></li> </ul> - <ol style="list-style-type: decimal"> - + <li> + <p>Build Librdkafka with SASL support (<tt>--enable-sasl</tt>) and install at your chosen $PREFIX.</p> - -<div class="source"> -<div class="source"> -<pre>wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz + +<div> +<div> +<pre class="source">wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz cd librdkafka-0.9.4/ ./configure --prefix=$PREFIX --enable-sasl make make install -</pre></div></div></li> - +</pre></div></div> +</li> <li> -<p>Validate Librdkafka does indeed support SASL. Run the following command and ensure that <tt>sasl</tt> is returned as a built-in feature.</p> - -<div class="source"> -<div class="source"> -<pre>$ examples/rdkafka_example -X builtin.features + +<p>Validate Librdkafka does indeed support SASL. Run the following command and ensure that <tt>sasl</tt> is returned as a built-in feature.</p> + +<div> +<div> +<pre class="source">$ examples/rdkafka_example -X builtin.features builtin.features = gzip,snappy,ssl,sasl,regex </pre></div></div> -<p>If it is not, ensure that you have <tt>libsasl</tt> or <tt>libsasl2</tt> installed. On CentOS, this can be installed with the following command.</p> - -<div class="source"> -<div class="source"> -<pre>yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi -</pre></div></div></li> - + +<p>If it is not, ensure that you have <tt>libsasl</tt> or <tt>libsasl2</tt> installed. On CentOS, this can be installed with the following command.</p> + +<div> +<div> +<pre class="source">yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi +</pre></div></div> +</li> <li> -<p>Grant access to your Kafka topic. In this example the topic is simply named <tt>pcap</tt>.</p> - -<div class="source"> -<div class="source"> -<pre>${KAFKA_HOME}/bin/kafka-acls.sh \ + +<p>Grant access to your Kafka topic. In this example the topic is simply named <tt>pcap</tt>.</p> + +<div> +<div> +<pre class="source">${KAFKA_HOME}/bin/kafka-acls.sh \ --authorizer kafka.security.auth.SimpleAclAuthorizer \ --authorizer-properties zookeeper.connect=zookeeper1:2181 \ --add \ @@ -510,65 +400,60 @@ ${KAFKA_HOME}/bin/kafka-acls.sh \ --add \ --allow-principal User:metron \ --group pycapa -</pre></div></div></li> - +</pre></div></div> +</li> <li> + <p>Use Pycapa as you normally would, but append the following three additional parameters</p> - <ul> - + <li><tt>security.protocol</tt></li> - <li><tt>sasl.kerberos.keytab</tt></li> - -<li><tt>sasl.kerberos.principal</tt></li> - </ul> - -<div class="source"> -<div class="source"> -<pre> $ pycapa --producer \ - --interface eth0 \ - --kafka-broker kafka1:6667 \ - --kafka-topic pcap --max-packets 10 \ - -X security.protocol=SASL_PLAINTEXT \ - -X sasl.kerberos.keytab=/etc/security/keytabs/metron.headless.keytab \ - -X sasl.kerberos.principal=metron-met...@metronexample.com - INFO:root:Connecting to Kafka; {'sasl.kerberos.principal': 'metron-met...@metronexample.com', 'group.id': 'ORNLVWJZZUAA', 'security.protocol': 'SASL_PLAINTEXT', 'sasl.kerberos.keytab': '/etc/security/keytabs/metron.headless.keytab', 'bootstrap.servers': 'kafka1:6667'} - INFO:root:Starting packet capture - INFO:root:Waiting for '1' message(s) to flush - INFO:root:'10' packet(s) in, '10' packet(s) out -</pre></div></div></li> +<li><tt>sasl.kerberos.principal</tt> + +<div> +<div> +<pre class="source">$ pycapa --producer \ + --interface eth0 \ + --kafka-broker kafka1:6667 \ + --kafka-topic pcap --max-packets 10 \ + -X security.protocol=SASL_PLAINTEXT \ + -X sasl.kerberos.keytab=/etc/security/keytabs/metron.headless .keytab \ + -X sasl.kerberos.principal=metron-met...@metronexample.com +INFO:root:Connecting to Kafka; {'sasl.kerberos.principal': 'metron-met...@metronexample.com', 'group.id': 'ORNLVWJZZUAA', 'security.protocol': 'SASL_PLAINTEXT', 'sasl.kerberos.keytab': '/etc/security/keytabs/metron.headless.keytab', 'bootstrap.servers': 'kafka1:6667'} +INFO:root:Starting packet capture +INFO:root:Waiting for '1' message(s) to flush +INFO:root:'10' packet(s) in, '10' packet(s) out +</pre></div></div> +</li> +</ul> +</li> </ol> <h1>FAQs</h1></div> <div class="section"> -<h3><a name="How_do_I_get_more_logs"></a>How do I get more logs?</h3> +<h3><a name="How_do_I_get_more_logs.3F"></a>How do I get more logs?</h3> <p>Use the following two command-line arguments to get detailed logging.</p> -<div class="source"> -<div class="source"> -<pre>-X debug=all --log-level DEBUG -</pre></div></div></div> +<div> +<div> +<pre class="source">-X debug=all --log-level DEBUG +</pre></div></div> +</div> <div class="section"> -<h3><a name="When_I_run_Pycapa_against_a_Kafka_broker_with_Kerberos_enabled_why_do_I_get_an_error_like_No_such_configuration_property:_sasl.kerberos.principal"></a>When I run Pycapa against a Kafka broker with Kerberos enabled, why do I get an error like “No such configuration property: ‘sasl.kerberos.principal’”?</h3> -<p>This can be a confusing error message because <tt>sasl.kerberos.principal</tt> is indeed a valid property for librdkafka as defined <a class="externalLink" href="https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md";>here</a>. This is most likely because Pycapa is running against a version of Librdkafka without SASL support enabled. This might happen if you have accidentally installed multiple versions of Librdkafka and Pycapa is unexpectedly using the version without SASL support enabled.</p> +<h3><a name="When_I_run_Pycapa_against_a_Kafka_broker_with_Kerberos_enabled.2C_why_do_I_get_an_error_like_.E2.80.9CNo_such_configuration_property:_.E2.80.98sasl.kerberos.principal.E2.80.99.E2.80.9D.3F"></a>When I run Pycapa against a Kafka broker with Kerberos enabled, why do I get an error like “No such configuration property: ‘sasl.kerberos.principal’”?</h3> +<p>This can be a confusing error message because <tt>sasl.kerberos.principal</tt> is indeed a valid property for librdkafka as defined <a class="externalLink" href="https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md";>here</a>. This is most likely because Pycapa is running against a version of Librdkafka without SASL support enabled. This might happen if you have accidentally installed multiple versions of Librdkafka and Pycapa is unexpectedly using the version without SASL support enabled.</p> <p>Bottom Line: Make sure that Pycapa is running against a version of Librdkafka with SASL support enabled.</p></div></div> - </div> - </div> - </div> - + </div> + </div> + </div> <hr/> - <footer> - <div class="container-fluid"> - <div class="row span12">Copyright © 2018 - <a href="https://www.apache.org";>The Apache Software Foundation</a>. - All Rights Reserved. - + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> </div> - - - - </div> </footer> </body> </html>
http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-stellar/stellar-3rd-party-example/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-stellar/stellar-3rd-party-example/index.html b/site/current-book/metron-stellar/stellar-3rd-party-example/index.html index d60a128..4e19e16 100644 --- a/site/current-book/metron-stellar/stellar-3rd-party-example/index.html +++ b/site/current-book/metron-stellar/stellar-3rd-party-example/index.html @@ -1,211 +1,108 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2018-01-03 - | Rendered using Apache Maven Fluido Skin 1.3.0 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-stellar/stellar-3rd-party-example/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180103" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Introduction</title> - <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> <link rel="stylesheet" href="../../css/site.css" /> <link rel="stylesheet" href="../../css/print.css" media="print" /> - - - <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> - - - -<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> - - </head> - <body class="topBarDisabled"> - - - - - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"> - <a href="http://metron.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> - </a> - </div> - <div class="pull-right"> </div> + <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> - - - <li class=""> - <a href="http://www.apache.org"; class="externalLink" title="Apache"> - Apache</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> - Metron</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="../../index.html" title="Documentation"> - Documentation</a> - </li> - <li class="divider ">/</li> - <li class="">Introduction</li> - - - - <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.2</li> - - </ul> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Introduction</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> </div> - - <div class="row-fluid"> - <div id="leftColumn" class="span3"> + <div id="leftColumn" class="span2"> <div class="well sidebar-nav"> - - - <ul class="nav nav-list"> - <li class="nav-header">User Documentation</li> - - <li> - - <a href="../../index.html" title="Metron"> - <i class="icon-chevron-down"></i> - Metron</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../Upgrading.html" title="Upgrading"> - <i class="none"></i> - Upgrading</a> - </li> - - <li> - - <a href="../../metron-analytics/index.html" title="Analytics"> - <i class="icon-chevron-right"></i> - Analytics</a> - </li> - - <li> - - <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> - <i class="none"></i> - Docker</a> - </li> - - <li> - - <a href="../../metron-deployment/index.html" title="Deployment"> - <i class="icon-chevron-right"></i> - Deployment</a> - </li> - - <li> - - <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> - <i class="none"></i> - Alerts</a> - </li> - - <li> - - <a href="../../metron-interface/metron-config/index.html" title="Config"> - <i class="none"></i> - Config</a> - </li> - - <li> - - <a href="../../metron-interface/metron-rest/index.html" title="Rest"> - <i class="none"></i> - Rest</a> - </li> - - <li> - - <a href="../../metron-platform/index.html" title="Platform"> - <i class="icon-chevron-right"></i> - Platform</a> - </li> - - <li> - - <a href="../../metron-sensors/index.html" title="Sensors"> - <i class="icon-chevron-right"></i> - Sensors</a> - </li> - - <li class="active"> - - <a href="#"><i class="none"></i>Stellar-3rd-party-example</a> - </li> - - <li> - - <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> - <i class="icon-chevron-right"></i> - Stellar-common</a> - </li> - - <li> - - <a href="../../use-cases/index.html" title="Use-cases"> - <i class="icon-chevron-right"></i> - Use-cases</a> - </li> - </ul> - </li> - </ul> - - - - <hr class="divider" /> - - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> - </a> - </div> - </div> - </div> - - - <div id="bodyColumn" class="span9" > - - <h1>Introduction</h1> -<p><a name="Introduction"></a></p> -<p>This is just a sample 3rd party stellar function (which is used in our unit testing). If Stellar changes its API (e.g. the annotation changes classes), then this will need to be built and the output jar of this (in <tt>target</tt>) will need to be placed in <tt>../stellar-common/src/test/classpath-resources/</tt>.</p> - </div> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> + <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li class="active"><a href="#"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a> </div> </div> + </div> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at - <hr/> +http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>Introduction</h1> +<p><a name="Introduction"></a></p> +<p>This is just a sample 3rd party stellar function (which is used in our unit testing). If Stellar changes its API (e.g. the annotation changes classes), then this will need to be built and the output jar of this (in <tt>target</tt>) will need to be placed in <tt>../stellar-common/src/test/classpath-resources/</tt>.</p> + </div> + </div> + </div> + <hr/> <footer> - <div class="container-fluid"> - <div class="row span12">Copyright © 2018 - <a href="https://www.apache.org";>The Apache Software Foundation</a>. - All Rights Reserved. - + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> </div> - - - - </div> </footer> </body> </html> http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html b/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html index eb9e054..88d7ac6 100644 --- a/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html +++ b/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html @@ -1,216 +1,114 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2018-01-03 - | Rendered using Apache Maven Fluido Skin 1.3.0 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-stellar/stellar-common/3rdPartyStellar.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180103" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Custom Stellar Functions</title> - <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../css/apache-maven-fluido-1.7.min.css" /> <link rel="stylesheet" href="../../css/site.css" /> <link rel="stylesheet" href="../../css/print.css" media="print" /> - - - <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script> - - - -<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> - - </head> - <body class="topBarDisabled"> - - - - - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"> - <a href="http://metron.apache.org/"; id="bannerLeft"> - <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> - </a> - </div> - <div class="pull-right"> </div> + <script type="text/javascript" src="../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> - - - <li class=""> - <a href="http://www.apache.org"; class="externalLink" title="Apache"> - Apache</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> - Metron</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="../../index.html" title="Documentation"> - Documentation</a> - </li> - <li class="divider ">/</li> - <li class="">Custom Stellar Functions</li> - - - - <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.2</li> - - </ul> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Custom Stellar Functions</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> </div> - - <div class="row-fluid"> - <div id="leftColumn" class="span3"> + <div id="leftColumn" class="span2"> <div class="well sidebar-nav"> - - - <ul class="nav nav-list"> - <li class="nav-header">User Documentation</li> - - <li> - - <a href="../../index.html" title="Metron"> - <i class="icon-chevron-down"></i> - Metron</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../Upgrading.html" title="Upgrading"> - <i class="none"></i> - Upgrading</a> - </li> - - <li> - - <a href="../../metron-analytics/index.html" title="Analytics"> - <i class="icon-chevron-right"></i> - Analytics</a> - </li> - - <li> - - <a href="../../metron-contrib/metron-docker/index.html" title="Docker"> - <i class="none"></i> - Docker</a> - </li> - - <li> - - <a href="../../metron-deployment/index.html" title="Deployment"> - <i class="icon-chevron-right"></i> - Deployment</a> - </li> - - <li> - - <a href="../../metron-interface/metron-alerts/index.html" title="Alerts"> - <i class="none"></i> - Alerts</a> - </li> - - <li> - - <a href="../../metron-interface/metron-config/index.html" title="Config"> - <i class="none"></i> - Config</a> - </li> - - <li> - - <a href="../../metron-interface/metron-rest/index.html" title="Rest"> - <i class="none"></i> - Rest</a> - </li> - - <li> - - <a href="../../metron-platform/index.html" title="Platform"> - <i class="icon-chevron-right"></i> - Platform</a> - </li> - - <li> - - <a href="../../metron-sensors/index.html" title="Sensors"> - <i class="icon-chevron-right"></i> - Sensors</a> - </li> - - <li> - - <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> - <i class="none"></i> - Stellar-3rd-party-example</a> - </li> - - <li> - - <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"> - <i class="icon-chevron-down"></i> - Stellar-common</a> - <ul class="nav nav-list"> - - <li class="active"> - - <a href="#"><i class="none"></i>3rdPartyStellar</a> - </li> - </ul> - </li> - - <li> - - <a href="../../use-cases/index.html" title="Use-cases"> - <i class="icon-chevron-right"></i> - Use-cases</a> - </li> - </ul> - </li> - </ul> - - - - <hr class="divider" /> - - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /> - </a> - </div> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-right"></span>Deployment</a></li> + <li><a href="../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-down"></span>Stellar-common</a> + <ul class="nav nav-list"> + <li class="active"><a href="#"><span class="none"></span>3rdPartyStellar</a></li> + </ul> +</li> + <li><a href="../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" /></a> + </div> </div> </div> - - - <div id="bodyColumn" class="span9" > - - <h1>Custom Stellar Functions</h1> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>Custom Stellar Functions</h1> <p><a name="Custom_Stellar_Functions"></a></p> -<p>Metron is fundamentally a programmable, extensible system and Stellar is the extension language. We have some great Stellar functions available out of the box and we’ll be adding more over time, but they may not quite scratch quite your particular itch. </p> -<p>Of course, we’d love to have your contribution inside of Metron if you think it general purpose enough, but not every function is general-purpose or it may rely on libraries those licenses aren’t acceptable for an Apache project. In that case, then you will be wondering how to add your custom function to a running instance of Metron.</p> +<p>Metron is fundamentally a programmable, extensible system and Stellar is the extension language. We have some great Stellar functions available out of the box and we’ll be adding more over time, but they may not quite scratch quite your particular itch.</p> +<p>Of course, we’d love to have your contribution inside of Metron if you think it general purpose enough, but not every function is general-purpose or it may rely on libraries those licenses aren’t acceptable for an Apache project. In that case, then you will be wondering how to add your custom function to a running instance of Metron.</p> <div class="section"> <h2><a name="Building_Your_Own_Function"></a>Building Your Own Function</h2> -<p>Let’s say that I need a function that returns the current time in milliseconds since the epoch. I notice that there’s nothing like that currently in Metron, so I embark on the adventure of adding it for my cluster.</p> -<p>I will presume that you have an installed Metron into your local maven repo via <tt>mvn install</tt> . In the future, when we publish to a maven repo, you will not need this. I will depend on 0.4.2 for the purpose of this demonstration</p> +<p>Let’s say that I need a function that returns the current time in milliseconds since the epoch. I notice that there’s nothing like that currently in Metron, so I embark on the adventure of adding it for my cluster.</p> +<p>I will presume that you have an installed Metron into your local maven repo via <tt>mvn install</tt> . In the future, when we publish to a maven repo, you will not need this. I will depend on 0.4.2 for the purpose of this demonstration</p> <div class="section"> -<h3><a name="Hack_Hack_Hack"></a>Hack, Hack, Hack</h3> -<p>I like to use Maven, so we’ll use that for this demonstration, but you can use whatever build system that you like. Here’s my favorite way to build a project with groupId <tt>com.mycompany.stellar</tt> and artifactId of <tt>tempus</tt> <tt>mvn archetype:create -DgroupId=com.mycompany.stellar -DartifactId=tempus -DarchetypeArtifactId=maven-archetype-quickstart</tt></p> +<h3><a name="Hack.2C_Hack.2C_Hack"></a>Hack, Hack, Hack</h3> +<p>I like to use Maven, so we’ll use that for this demonstration, but you can use whatever build system that you like. Here’s my favorite way to build a project with groupId <tt>com.mycompany.stellar</tt> and artifactId of <tt>tempus</tt> <tt>mvn archetype:create -DgroupId=com.mycompany.stellar -DartifactId=tempus -DarchetypeArtifactId=maven-archetype-quickstart</tt></p> <p>First, we should depend on <tt>metron-common</tt> and we can do that by adjusting the <tt>pom.xml</tt> just created:</p> -<div class="source"> -<div class="source"> -<pre><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" +<div> +<div> +<pre class="source"><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> @@ -243,11 +141,12 @@ </dependencies> </project> </pre></div></div> + <p>Let’s add our implementation in <tt>src/main/java/com/mycompany/stellar/TimeFunctions.java</tt> with the following content:</p> -<div class="source"> -<div class="source"> -<pre>package com.notmetron.stellar; +<div> +<div> +<pre class="source">package com.notmetron.stellar; import org.apache.metron.stellar.dsl.Context; import org.apache.metron.stellar.dsl.ParseException; @@ -276,6 +175,7 @@ public class TimeFunction { } } </pre></div></div> + <p>Now we can build the project via <tt>mvn package</tt> which will create a <tt>target/tempus-1.0-SNAPSHOT.jar</tt> file.</p></div></div> <div class="section"> <h2><a name="Install_the_Function"></a>Install the Function</h2> @@ -283,26 +183,22 @@ public class TimeFunction { <div class="section"> <h3><a name="Deploy_the_Jar"></a>Deploy the Jar</h3> <p>First you need to place the jar in HDFS, if we have it on an access node, one way to do that is:</p> - <ul> - + <li><tt>hadoop fs -put tempus-1.0-SNAPSHOT.jar /apps/metron/stellar</tt> This presumes that:</li> - <li>you’ve standardized on <tt>/apps/metron/stellar</tt> as the location for custom jars</li> - <li>you are running the command from an access node with the <tt>hadoop</tt> command installed</li> - <li>you are running from a user that has write access to <tt>/apps/metron/stellar</tt></li> </ul></div> <div class="section"> <h3><a name="Set_Global_Config"></a>Set Global Config</h3> <p>You may not need this if your Metron administrator already has this setup.</p> -<p>With that dispensed with, we need to ensure that Metron knows to look at that location. We need to ensure that the <tt>stellar.function.paths</tt> property in the <tt>global.json</tt> is in place that makes Metron aware to look for Stellar functions in <tt>/apps/metron/stellar</tt> on HDFS. </p> +<p>With that dispensed with, we need to ensure that Metron knows to look at that location. We need to ensure that the <tt>stellar.function.paths</tt> property in the <tt>global.json</tt> is in place that makes Metron aware to look for Stellar functions in <tt>/apps/metron/stellar</tt> on HDFS.</p> <p>This property looks like, the following for a vagrant install</p> -<div class="source"> -<div class="source"> -<pre>{ +<div> +<div> +<pre class="source">{ "es.clustername": "metron", "es.ip": "node1", "es.port": "9300", @@ -310,21 +206,23 @@ public class TimeFunction { "stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/.*.jar", } </pre></div></div> + <p>The <tt>stellar.function.paths</tt> property takes a comma separated list of URIs or URIs with regex expressions at the end. Also, note path is prefaced by the HDFS default name, which, if you do not know, can be found by executing, <tt>hdfs getconf -confKey fs.default.name</tt>, such as</p> -<div class="source"> -<div class="source"> -<pre>[root@node1 ~]# hdfs getconf -confKey fs.default.name +<div> +<div> +<pre class="source">[root@node1 ~]# hdfs getconf -confKey fs.default.name hdfs://node1:8020 -</pre></div></div></div> +</pre></div></div> +</div> <div class="section"> <h3><a name="Use_the_Function"></a>Use the Function</h3> <p>Now that we have deployed the function, if we want to use it, any running topologies that use Stellar will need to be restarted.</p> <p>Beyond that, let’s take a look at it in the REPL:</p> -<div class="source"> -<div class="source"> -<pre>Stellar, Go! +<div> +<div> +<pre class="source">Stellar, Go! Please note that functions are loading lazily in the background and will be unavailable until loaded fully. {es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH, stellar.function.paths=hdfs://node1:8020/apps/metron/stellar/.*.jar, profiler.client.period.duration=1, profiler.client.period.duration.units=MINUTES} [Stellar]>>> # Get the help for NOW @@ -339,23 +237,17 @@ Returns: Timestamp 1488400515655 [Stellar]>>> # Looks like I got a timestamp, success! </pre></div></div></div></div> - </div> - </div> - </div> - + </div> + </div> + </div> <hr/> - <footer> - <div class="container-fluid"> - <div class="row span12">Copyright © 2018 - <a href="https://www.apache.org";>The Apache Software Foundation</a>. - All Rights Reserved. - + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> </div> - - - - </div> </footer> </body> </html>
