http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-deployment/packaging/ambari/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/ambari/index.html b/site/current-book/metron-deployment/packaging/ambari/index.html index 8013381..bac5758 100644 --- a/site/current-book/metron-deployment/packaging/ambari/index.html +++ b/site/current-book/metron-deployment/packaging/ambari/index.html @@ -1,505 +1,331 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2018-01-03 - | Rendered using Apache Maven Fluido Skin 1.3.0 + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20180103" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Ambari Management Pack Development</title> - <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" /> + <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.7.min.css" /> <link rel="stylesheet" href="../../../css/site.css" /> <link rel="stylesheet" href="../../../css/print.css" media="print" /> - - - <script type="text/javascript" src="../../../js/apache-maven-fluido-1.3.0.min.js"></script> - - - -<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script> - - </head> - <body class="topBarDisabled"> - - - - - <div class="container-fluid"> - <div id="banner"> - <div class="pull-left"> - <a href="http://metron.apache.org/"; id="bannerLeft"> - <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/> - </a> - </div> - <div class="pull-right"> </div> + <script type="text/javascript" src="../../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> - - - <li class=""> - <a href="http://www.apache.org"; class="externalLink" title="Apache"> - Apache</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="http://metron.apache.org/"; class="externalLink" title="Metron"> - Metron</a> - </li> - <li class="divider ">/</li> - <li class=""> - <a href="../../../index.html" title="Documentation"> - Documentation</a> - </li> - <li class="divider ">/</li> - <li class="">Ambari Management Pack Development</li> - - - - <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.2</li> - - </ul> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active ">Ambari Management Pack Development</li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> </div> - - <div class="row-fluid"> - <div id="leftColumn" class="span3"> + <div id="leftColumn" class="span2"> <div class="well sidebar-nav"> - - - <ul class="nav nav-list"> - <li class="nav-header">User Documentation</li> - - <li> - - <a href="../../../index.html" title="Metron"> - <i class="icon-chevron-down"></i> - Metron</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../../Upgrading.html" title="Upgrading"> - <i class="none"></i> - Upgrading</a> - </li> - - <li> - - <a href="../../../metron-analytics/index.html" title="Analytics"> - <i class="icon-chevron-right"></i> - Analytics</a> - </li> - - <li> - - <a href="../../../metron-contrib/metron-docker/index.html" title="Docker"> - <i class="none"></i> - Docker</a> - </li> - - <li> - - <a href="../../../metron-deployment/index.html" title="Deployment"> - <i class="icon-chevron-down"></i> - Deployment</a> - <ul class="nav nav-list"> - - <li> - - <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"> - <i class="none"></i> - Kerberos-ambari-setup</a> - </li> - - <li> - - <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"> - <i class="none"></i> - Kerberos-manual-setup</a> - </li> - - <li> - - <a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"> - <i class="none"></i> - Amazon-ec2</a> - </li> - - <li> - - <a href="../../../metron-deployment/other-examples/index.html" title="Other-examples"> - <i class="icon-chevron-right"></i> - Other-examples</a> - </li> - - <li class="active"> - - <a href="#"><i class="none"></i>Ambari</a> - </li> - - <li> - - <a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"> - <i class="none"></i> - Ansible-docker</a> - </li> - - <li> - - <a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"> - <i class="none"></i> - Rpm-docker</a> - </li> - - <li> - - <a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"> - <i class="none"></i> - Packer-build</a> - </li> - - <li> - - <a href="../../../metron-deployment/roles/index.html" title="Roles"> - <i class="icon-chevron-right"></i> - Roles</a> - </li> - - <li> - - <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant"> - <i class="icon-chevron-right"></i> - Vagrant</a> - </li> - </ul> - </li> - - <li> - - <a href="../../../metron-interface/metron-alerts/index.html" title="Alerts"> - <i class="none"></i> - Alerts</a> - </li> - - <li> - - <a href="../../../metron-interface/metron-config/index.html" title="Config"> - <i class="none"></i> - Config</a> - </li> - - <li> - - <a href="../../../metron-interface/metron-rest/index.html" title="Rest"> - <i class="none"></i> - Rest</a> - </li> - - <li> - - <a href="../../../metron-platform/index.html" title="Platform"> - <i class="icon-chevron-right"></i> - Platform</a> - </li> - - <li> - - <a href="../../../metron-sensors/index.html" title="Sensors"> - <i class="icon-chevron-right"></i> - Sensors</a> - </li> - - <li> - - <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> - <i class="none"></i> - Stellar-3rd-party-example</a> - </li> - - <li> - - <a href="../../../metron-stellar/stellar-common/index.html" title="Stellar-common"> - <i class="icon-chevron-right"></i> - Stellar-common</a> - </li> - - <li> - - <a href="../../../use-cases/index.html" title="Use-cases"> - <i class="icon-chevron-right"></i> - Use-cases</a> - </li> - </ul> - </li> - </ul> - - - - <hr class="divider" /> - - <div id="poweredBy"> - <div class="clear"></div> - <div class="clear"></div> - <div class="clear"></div> - <a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"> - <img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" /> - </a> - </div> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-down"></span>Deployment</a> + <ul class="nav nav-list"> + <li><a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"><span class="none"></span>Kerberos-ambari-setup</a></li> + <li><a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"><span class="none"></span>Kerberos-manual-setup</a></li> + <li><a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"><span class="none"></span>Amazon-ec2</a></li> + <li><a href="../../../metron-deployment/ansible/index.html" title="Ansible"><span class="icon-chevron-right"></span>Ansible</a></li> + <li><a href="../../../metron-deployment/development/index.html" title="Development"><span class="icon-chevron-right"></span>Development</a></li> + <li><a href="../../../metron-deployment/other-examples/index.html" title="Other-examples"><span class="icon-chevron-right"></span>Other-examples</a></li> + <li class="active"><a href="#"><span class="icon-chevron-down"></span>Ambari</a> + <ul class="nav nav-list"> + <li><a href="../../../metron-deployment/packaging/ambari/elasticsearch-mpack/index.html" title="Elasticsearch-mpack"><span class="none"></span>Elasticsearch-mpack</a></li> + <li><a href="../../../metron-deployment/packaging/ambari/metron-mpack/index.html" title="Mpack"><span class="none"></span>Mpack</a></li> + </ul> +</li> + <li><a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"><span class="none"></span>Ansible-docker</a></li> + <li><a href="../../../metron-deployment/packaging/docker/deb-docker/index.html" title="Deb-docker"><span class="none"></span>Deb-docker</a></li> + <li><a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"><span class="none"></span>Rpm-docker</a></li> + <li><a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"><span class="none"></span>Packer-build</a></li> + </ul> +</li> + <li><a href="../../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" /></a> + </div> </div> </div> - - - <div id="bodyColumn" class="span9" > - - <h1>Ambari Management Pack Development</h1> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<h1>Ambari Management Pack Development</h1> <p><a name="Ambari_Management_Pack_Development"></a></p> <div class="section"> <h2><a name="Table_of_Contents"></a>Table of Contents</h2> - <ul> - + <li><a href="#Overview">Overview</a></li> - <li><a href="#Ambari_Zookeeper_Config_Lifecycle">Ambari Zookeeper Config Lifecycle</a></li> - <li><a href="#Adding_a_new_property">Adding a new property</a></li> - <li><a href="#How_to_identify_errors_in_MPack_changes">How to identify errors in MPack changes</a></li> - <li><a href="#Testing_changes_without_cycling_Vagrant_build">Testing changes without cycling Vagrant build</a></li> - <li><a href="#Configuration_involving_dependency_services">Configuration involving dependency services</a></li> - <li><a href="#Kerberos">Kerberos</a></li> - <li><a href="#Best_practices">Best practices</a></li> +<li><a href="#Upgrading_MPack_Services">Upgrading MPack Services</a></li> </ul></div> <div class="section"> <h2><a name="Overview"></a>Overview</h2> <p>Typically, Ambari Management Pack development will be done in the Vagrant environments. These instructions are specific to Vagrant, but can be adapted for other environemnts (e.g. make sure to be on the correct nodes for server vs agent files)</p> -<p>There is an <tt>mpack.json</tt> file which describes what services the mpack will contains, versions, etc.</p> -<p>Alongside this are two directories, <tt>addon-services</tt> and <tt>common-services</tt>.</p> -<p>The layout of <tt>/common-services/METRON.CURRENT</tt> is</p> +<p>There are two MPacks:</p> +<ul> +<li>Metron - contains artifacts for deploying the Metron service</li> +<li>Elasticsearch - contains artifacts for installing Elasticsearch and Kibana services</li> +</ul> +<p>There is an <tt>mpack.json</tt> file for each which describes what services the mpack will contain, versions, etc.</p> +<p>Alongside this are two directories, <tt>addon-services</tt> and <tt>common-services</tt>. Below the Metron MPack is described, but this also applies similarly to the Elasticsearch MPack.</p> +<p>The layout of <tt>/common-services/METRON/CURRENT</tt> is</p> <ul> - + <li><tt>/configuration</tt> - <ul> - + <li>This contains a set of <tt>*-env.xml</tt> files, relevent to particular components or the service as a whole. These are where properties are defined.</li> - </ul></li> - +</ul> +</li> <li><tt>/package</tt> - <ul> - + <li><tt>/files</tt> - <ul> - + <li>Contains files that get used as provided, in particular Elasticsearch templates.</li> - </ul></li> - +</ul> +</li> <li><tt>/scripts</tt> - <ul> - + <li>A set of Python scripts that interface with Ambari to manage setup and install</li> - <li><tt>/params</tt> - <ul> - + <li>A set of Python scripts for managing parameters from the <tt>*-env.xml</tt> files</li> - </ul></li> - +</ul> +</li> <li><tt>/templates</tt> - <ul> - + <li>A set of Jinja template files which can be populated with properties</li> - </ul></li> - </ul></li> - </ul></li> - +</ul> +</li> +</ul> +</li> +</ul> +</li> <li><tt>/quicklinks</tt> - <ul> - + <li>Contains <tt>quicklinks.json</tt> to define the Ambari quicklinks that should be used in the UI</li> - </ul></li> - +</ul> +</li> <li><tt>themes</tt> - <ul> - + <li>Manages the Ambari UI themes blocks for organizing the configuration by relevant area.</li> - </ul></li> - +</ul> +</li> <li><tt>kerberos.json</tt> - <ul> - + <li>Defines the keytabs and other Kerberos configuration to be used when Kerberizing a cluster</li> - </ul></li> - +</ul> +</li> <li><tt>metainfo.xml</tt> - <ul> - + <li>Defines the METRON service, along with required packages, services, etc.</li> - </ul></li> - +</ul> +</li> <li><tt>service_advisor.py</tt> - <ul> - + <li>Handles component layout and validation, along with handling some configurations for other services or that needs configs from other services.</li> - </ul></li> </ul> -<p>The layout of <tt>/addon-services/METRON.CURRENT</tt> is</p> - +</li> +</ul> +<p>The layout of <tt>/addon-services/METRON/CURRENT</tt> is</p> <ul> - + <li><tt>/repos</tt> - <ul> - + <li>Contains <tt>repoinfo.xml</tt> that defines repositories to install packages from</li> - </ul></li> - +</ul> +</li> <li><tt>metainfo.xml</tt> - <ul> - -<li>Limited info version of <tt>/common-services/METRON.CURRENT/metainfo.xml</tt></li> - </ul></li> - + +<li>Limited info version of <tt>/common-services/METRON/CURRENT/metainfo.xml</tt></li> +</ul> +</li> <li><tt>role_command_order.json</tt> - <ul> - + <li>Defines the order of service startup and other actions relative to each other.</li> - </ul></li> +</ul> +</li> </ul></div> <div class="section"> <h2><a name="Ambari_Zookeeper_Config_Lifecycle"></a>Ambari Zookeeper Config Lifecycle</h2> <p>The source of truth for Metron configuration resides in Zookeeper as a series of JSON documents. There are actually multiple locations that you can populate the Zookeeper configs from:</p> - <ul> - + <li>$METRON_HOME/config/zookeeper</li> - <li>Stellar REPL</li> - <li>Management UI</li> - <li>Ambari</li> </ul> <p>Any change that works with Metron configuration stored in Zookeeper should always pull the latest config from Zookeeper first before making any changes and publishing them back up to avoid potentially overwriting the latest configs with stale data. The lifecycle for configuration management with Ambari is as follows:</p> - <ul> - + <li>First start/install for parsers, enrichment, indexing, or the profiler - <ul> - + <li>Perform a PUSH from local disk to Zookeeper</li> - </ul></li> - +</ul> +</li> <li>All component start/restart operations for parsers, enrichment, indexing, or the profiler - <ul> - + <li>Perform a JSON patch (RFC 6902) on the configs in Zookeeper and push the patched, pretty formatted config up to Zookeeper. Patching is currently applicable to to global.json only.</li> - <li>Pull all configs to the local file system.</li> - </ul></li> +</ul> +</li> </ul> <p>The patching mechanism we introduced to Ambari for managing the global config is by default additive in nature, though it will also overwrite any existing values with the latest values from Ambari. This means that you can add any properties to the global config outside of Ambari you like, provided you are not modifying values explicitly managed by Ambari. This is important because you will not get any errors when modifying the configuration outside of Ambari. Instead, if you modified es.ip and changed global.json outsided Ambari, you would not see this change in Ambari. Meanwhile, the indexing topology would be using the new value stored in Zookeeper. Remember, Zookeeper is the source of truth. If you were to restart a Metron topology component via Ambari, that es.ip property would now be set back to the value stored in Ambari. So in summary, if it’s managed by Ambari, only change that value via Ambari. If it’s not, then feel free to use whatever mechanism you like, just be sure to pull the latest config from Zookeeper before making any changes. We make no effort to handle race conditions. It’s last commit wins. Below are the global config properties managed by Ambari:</p> - <ul> - + <li>es.clustername</li> - <li>es.ip</li> - <li>es.date.format</li> - <li>parser.error.topic</li> - <li>update.hbase.table</li> - <li>update.hbase.cf</li> - <li>profiler.client.period.duration</li> - <li>profiler.client.period.duration.units</li> </ul></div> <div class="section"> <h2><a name="Adding_a_new_property"></a>Adding a new property</h2> - <ol style="list-style-type: decimal"> - -<li> -<p>Add the property to the appropriate <tt>*-env.xml</tt> file found in <tt>METRON.CURRENT/configuration</tt>.</p> - -<div class="source"> -<div class="source"> -<pre><property> + +<li>Add the property to the appropriate <tt>*-env.xml</tt> file found in <tt>METRON.CURRENT/configuration</tt>. + +<div> +<div> +<pre class="source"><property> <name>new_property</name> <description>New Property description</description> <value>Default Value</value> <display-name>New Property Pretty Name</display-name> </property> </pre></div></div> -<p>The appropriate <tt>*-env.xml</tt> file should be selected based on which component depends on the property. This allows Ambari to accurately restart only the affected components when the property is changed. If a property is in <tt>metron-env.xml</tt>, Ambari will prompt you to restart all Metron components.</p></li> - +</li> +</ol> +<p>The appropriate <tt>*-env.xml</tt> file should be selected based on which component depends on the property. This allows Ambari to accurately restart only the affected components when the property is changed. If a property is in <tt>metron-env.xml</tt>, Ambari will prompt you to restart all Metron components.</p> +<ol style="list-style-type: decimal"> + <li> -<p>Add the property to the <tt>metron_theme.json</tt> file found in <tt>METRON.CURRENT/themes</tt> if the property was added to a component-specific <tt>*-env.xml</tt> file (<tt>metron-parsers-env.xml</tt> for example) and not <tt>metron-env.xml</tt>. This is necessary for the property to be displayed in the correct tab of the Metron Configs section in the Ambari UI. Using other properties as a guide, add the property to <tt>/configuration/placement/configs</tt> for proper placement and also to <tt>/configuration/widgets</tt> for a specific widget type.</p></li> - + +<p>Add the property to the <tt>metron_theme.json</tt> file found in <tt>METRON.CURRENT/themes</tt> if the property was added to a component-specific <tt>*-env.xml</tt> file (<tt>metron-parsers-env.xml</tt> for example) and not <tt>metron-env.xml</tt>. This is necessary for the property to be displayed in the correct tab of the Metron Configs section in the Ambari UI. Using other properties as a guide, add the property to <tt>/configuration/placement/configs</tt> for proper placement and also to <tt>/configuration/widgets</tt> for a specific widget type.</p> +</li> <li> + <p>Reference the property in <tt>METRON.CURRENT/package/scriptes/params/params_linux.py</tt>, unless it will be used in Ambari’s status command. It will be stored in a variable. The name doesn’t have to match, but it’s preferred that it does. Make sure to use replace <tt>metron-env</tt> the correct <tt>*-env</tt> file, as noted above.</p> - -<div class="source"> -<div class="source"> -<pre> new_property = config['configurations']['metron-env']['new_property'] + +<div> +<div> +<pre class="source">new_property = config['configurations']['metron-env']['new_property'] </pre></div></div> -<p>If this property will be used in the status command, instead make this change in <tt>METRON.CURRENT/package/scriptes/params/status_params.py</tt>. Afterwards, in <tt>params_linux.py</tt>, reference the new property:</p></li> -</ol> -<div class="source"> -<div class="source"> -<pre> new_property = status_params.new_property +<p>If this property will be used in the status command, instead make this change in <tt>METRON.CURRENT/package/scriptes/params/status_params.py</tt>. Afterwards, in <tt>params_linux.py</tt>, reference the new property:</p> + +<div> +<div> +<pre class="source">new_property = status_params.new_property </pre></div></div> -<p>This behavior is because Ambari doesn’t send all parameters to the status, so it needs to be explicitly provided. Also note that status_params.py parameters are not automatically pulled into the params_linux.py namespace, so we explicitly choose the variables to include. See <a class="externalLink" href="https://docs.python.org/2/howto/doanddont.html#at-module-level";>https://docs.python.org/2/howto/doanddont.html#at-module-level</a> for more info.</p> -<ol style="list-style-type: decimal"> - +<p>This behavior is because Ambari doesn’t send all parameters to the status, so it needs to be explicitly provided. Also note that status_params.py parameters are not automatically pulled into the params_linux.py namespace, so we explicitly choose the variables to include. See <a class="externalLink" href="https://docs.python.org/2/howto/doanddont.html#at-module-level";>https://docs.python.org/2/howto/doanddont.html#at-module-level</a> for more info.</p> +</li> <li> + <p>Ambari master services can then import the params:</p> - -<div class="source"> -<div class="source"> -<pre> from params import params - env.set_params(params) -</pre></div></div></li> - + +<div> +<div> +<pre class="source">from params import params +env.set_params(params) +</pre></div></div> +</li> <li> + <p>The <tt>*_commands.py</tt> files receive the params as an input from the master services. Once this is done, they can be accessed via the variable we set above:</p> - -<div class="source"> -<div class="source"> -<pre> self.__params.new_property -</pre></div></div></li> + +<div> +<div> +<pre class="source">self.__params.new_property +</pre></div></div> +</li> </ol> <div class="section"> <h3><a name="Env_file_property_walkthrough"></a>Env file property walkthrough</h3> @@ -508,108 +334,113 @@ <h4><a name="Defining_the_property"></a>Defining the property</h4> <p>First the property appears in the appropriate <tt>*-env.xml</tt>, in this case <tt>METRON.CURRENT/configuration/metron-env.xml</tt>.</p> -<div class="source"> -<div class="source"> -<pre><property> +<div> +<div> +<pre class="source"><property> <name>metron_apps_hdfs_dir</name> <value>/apps/metron</value> <description>Metron apps HDFS dir</description> <display-name>Metron apps HDFS dir</display-name> </property> </pre></div></div> -<p>This defines several things</p> +<p>This defines several things</p> <ol style="list-style-type: decimal"> - + <li>The name of the property we’ll be referencing it by in our code.</li> - <li>The default value of the property.</li> - <li>The description of the property for the Ambari UI.</li> - <li>The pretty name that will be shown in Ambari for the property.</li> </ol></div> <div class="section"> <h4><a name="Making_the_property_available_to_scripts"></a>Making the property available to scripts</h4> <p>Second, we set up the property to be available to the code. This happens in <tt>METRON.CURRENT/packages/scripts/params/params_linux.py</tt>. Just add the following line:</p> -<div class="source"> -<div class="source"> -<pre>metron_apps_hdfs_dir = config['configurations']['metron-env']['metron_apps_hdfs_dir'] +<div> +<div> +<pre class="source">metron_apps_hdfs_dir = config['configurations']['metron-env']['metron_apps_hdfs_dir'] </pre></div></div> + <p>There is one catch to this. If we wanted this to be available to Ambari’s status command, we’d need to put that line in <tt>METRON.CURRENT/packages/scripts/params/status_params.py</tt> and reference it in <tt>params_linux.py</tt> like so:</p> -<div class="source"> -<div class="source"> -<pre>metron_apps_hdfs_dir = status_params.metron_apps_hdfs_dir +<div> +<div> +<pre class="source">metron_apps_hdfs_dir = status_params.metron_apps_hdfs_dir </pre></div></div> + <p>This behavior is because Ambari doesn’t send all parameters to the status, so it needs to be explicitly provided.</p> <p>In our case, we don’t use this parameter directly (but it could be if we wanted to use it exactly). We actually append to it before use in <tt>params_linux.py</tt>:</p> -<div class="source"> -<div class="source"> -<pre>from resource_management.libraries.functions import format +<div> +<div> +<pre class="source">from resource_management.libraries.functions import format # ... hdfs_grok_patterns_dir = format("{metron_apps_hdfs_dir}/patterns") </pre></div></div> + <p>The <tt>format</tt> function is a special Ambari function that will let you create a string with properties in curly braces replaced by their values. In this case, <tt>hdfs_grok_patterns_dir</tt> will be <tt>/apps/metron/patterns</tt> and will be what we now follow in this example.</p> <p>Note: There are instances where we deviate a bit from this pattern and use format(format(“{{some_prop}}”)). This is not a mistake. Ambari does not natively support nested properties, and this was a workaround that accomplished being able to initialize values from env.xml files and have them also available to users as Ambari properties later. Here’s an example from params_linux.py:</p> -<div class="source"> -<div class="source"> -<pre>metron_apps_indexed_hdfs_dir = format(format(config['configurations']['metron-indexing-env']['metron_apps_indexed_hdfs_dir'])) +<div> +<div> +<pre class="source">metron_apps_indexed_hdfs_dir = format(format(config['configurations']['metron-indexing-env']['metron_apps_indexed_hdfs_dir'])) </pre></div></div> + <p>and the relavant properties in metron-env.xml and metron-indexing-env.xml</p> <p>metron-env.xml</p> -<div class="source"> -<div class="source"> -<pre><property> +<div> +<div> +<pre class="source"><property> <name>metron_apps_hdfs_dir</name> <value>/apps/metron</value> <description>Metron apps HDFS dir</description> <display-name>Metron apps HDFS dir</display-name> </property> </pre></div></div> + <p>metron-indexing-env.xml</p> -<div class="source"> -<div class="source"> -<pre><property> +<div> +<div> +<pre class="source"><property> <name>metron_apps_indexed_hdfs_dir</name> <value>{{metron_apps_hdfs_dir}}/indexing/indexed</value> <description>Indexing bolts will write to this HDFS directory</description> <display-name>Metron apps indexed HDFS dir</display-name> </property> -</pre></div></div></div> +</pre></div></div> +</div> <div class="section"> <h4><a name="Importing_the_parameters_into_scripts"></a>Importing the parameters into scripts</h4> <p><tt>hdfs_grok_patterns_dir</tt> is used in <tt>METRON.CURRENT/package/scripts/parser_commands.py</tt>, but before we can reference it, we’ll need the params available to <tt>parser_commands.py</tt></p> <p>To make them available, we take them in as part of the <tt>__init__</tt></p> -<div class="source"> -<div class="source"> -<pre>def __init__(self, params): +<div> +<div> +<pre class="source">def __init__(self, params): if params is None: raise ValueError("params argument is required for initialization") self.__params = params # Other initialization </pre></div></div> + <p>This init is called from various Ambari service methods in <tt>METRON.CURRENT/package/scripts/parser_master.py</tt>, e.g.:</p> -<div class="source"> -<div class="source"> -<pre>def stop(self, env, upgrade_type=None): +<div> +<div> +<pre class="source">def stop(self, env, upgrade_type=None): from params import params env.set_params(params) commands = ParserCommands(params) commands.stop_parser_topologies() </pre></div></div> + <p>Once the params are available to <tt>parser_commands.py</tt>, <tt>hdfs_grok_patterns_dir</tt> is by referencing <tt>self.__params.hdfs_grok_patterns_dir)</tt> In our case, this will create and populate <tt>/apps/metron/patterns</tt> on HDFS, owned by the metron user with appropriate permissions. It’ll also log out what it’s doing, which is important for being able to debug.</p> -<div class="source"> -<div class="source"> -<pre>def init_parsers(self): +<div> +<div> +<pre class="source">def init_parsers(self): Logger.info( "Copying grok patterns from local directory '{0}' to HDFS '{1}'".format(self.__params.local_grok_patterns_dir, self.__params.hdfs_grok_patterns_dir)) @@ -622,7 +453,8 @@ hdfs_grok_patterns_dir = format("{metron_apps_hdfs_dir}/patterns") source=self.__params.local_grok_patterns_dir) Logger.info("Done initializing parser configuration") -</pre></div></div></div></div> +</pre></div></div> +</div></div> <div class="section"> <h3><a name="Jinja_Templates_and_properties"></a>Jinja Templates and properties</h3> <p>Jinja templates allow for the ability to have most of a file defined, but allow variables to be filled in from the properties defined in our <tt>*-env.xml</tt> files.</p> @@ -636,36 +468,39 @@ hdfs_grok_patterns_dir = format("{metron_apps_hdfs_dir}/patterns") <h4><a name="Defining_the_property"></a>Defining the property</h4> <p>First, we need the property in the configuration file:</p> -<div class="source"> -<div class="source"> -<pre><property> +<div> +<div> +<pre class="source"><property> <name>metron_log_dir</name> <value>/var/log/metron</value> <description>Log directory for metron</description> <display-name>Metron log dir</display-name> </property> -</pre></div></div></div> +</pre></div></div> +</div> <div class="section"> <h4><a name="Making_the_property_available_to_templates"></a>Making the property available to templates</h4> <p>This property isn’t used in Ambari’s status check, so it was directly added to <tt>METRON.CURRENT/package/scripts/params/params_linux.py</tt>. All we do is add the variable, and reference Ambari’s config object appropriately, making sure to reference <tt>metron-env</tt> as the file where the property is located.</p> -<div class="source"> -<div class="source"> -<pre>metron_log_dir = config['configurations']['metron-env']['metron_log_dir'] +<div> +<div> +<pre class="source">metron_log_dir = config['configurations']['metron-env']['metron_log_dir'] </pre></div></div> + <p>The property is referenced in <tt>metron.j2</tt>.</p> -<div class="source"> -<div class="source"> -<pre>METRON_LOG_DIR="{{metron_log_dir}}" -</pre></div></div></div> +<div> +<div> +<pre class="source">METRON_LOG_DIR="{{metron_log_dir}}" +</pre></div></div> +</div> <div class="section"> <h4><a name="Using_the_template_in_scripts"></a>Using the template in scripts</h4> <p>For that property to actually be used, it is referenced in <tt>rest_master.py</tt>:</p> -<div class="source"> -<div class="source"> -<pre>from resource_management.core.resources.system import File +<div> +<div> +<pre class="source">from resource_management.core.resources.system import File from resource_management.core.source import Template def configure(self, env, upgrade_type=None, config_dir=None): @@ -677,21 +512,23 @@ def configure(self, env, upgrade_type=None, config_dir=None): ) # Do stuff </pre></div></div> + <p>This will create a file on the Ambari agent machine’s file system, <tt>/etc/sysconfig/metron</tt>, with the content of <tt>metron.j2</tt>, after replacing <tt>{{metron_log_dir}}</tt> with the value of the property (<tt>/var/log/metron</tt>)</p> -<div class="source"> -<div class="source"> -<pre>... +<div> +<div> +<pre class="source">... METRON_LOG_DIR="/var/log/metron" ... -</pre></div></div></div></div> +</pre></div></div> +</div></div> <div class="section"> <h3><a name="Defining_presentation_in_the_Ambari_UI"></a>Defining presentation in the Ambari UI</h3> <p>Where and how a property is displayed can be controlled in the <tt>METRON.CURRENT/themes/metron_theme.json</tt> file. Consider the <tt>enrichment_workers</tt> property that is defined in a component specific <tt>*-env.xml</tt> file, in this case <tt>METRON.CURRENT/configuration/metron-enrichment-env.xml</tt>. The property appears in <tt>METRON.CURRENT/themes/metron_theme.json</tt> in two different sections:</p> -<div class="source"> -<div class="source"> -<pre>{ +<div> +<div> +<pre class="source">{ "configuration": { "layouts": [...], "placement": { @@ -714,21 +551,20 @@ METRON_LOG_DIR="/var/log/metron" } } </pre></div></div> + <p>The first setting places the property in the “Storm” section of the “Enrichment” tab in Ambari. Sections are defined in <tt>metron_theme.json</tt> under <tt>/configuration/layouts</tt>.</p> <p>The second setting defines a widget type of <tt>text-field</tt>. See the <a class="externalLink" href="https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Configs";>Ambari Wiki</a> for more detail on widget types.</p> <p>If a property is defined in <tt>metron-env.xml</tt>, it is not necessary to add it to the <tt>metron_theme.json</tt> file. By default the property will be located under the “Advanced” tab in the “Advanced metron-env” section.</p></div></div> <div class="section"> <h2><a name="How_to_identify_errors_in_MPack_changes"></a>How to identify errors in MPack changes</h2> <p>Typically, any errors are thrown at one of two times:</p> - <ol style="list-style-type: decimal"> - + <li>Attempting to install Metron as a whole. These are typically service level definition errors, not property errors. Logs will often be found in <tt>/var/log/ambari-server/ambari-server.log</tt>. Often the log will indicate it was unable to load a particular service or subservice with some more detail.</li> - <li>Running the actual functionality. These typically tend to cause errors in the UI at runtime. Logs are usually found in <tt>/var/log/ambari-agent/ambari-agent.log</tt>.</li> </ol> <p>Unfortunately, because errors tend to occur at runtime, it’s often necessary to add things like logging statements, or even just throw errors to print out in the Ambari UI.</p> -<p>The primary solution to these is to look in the logs for exceptions, see what’s going wrong (Property doesn’t exist? Malformed file couldn’t be loaded?), and adjust appropriately.</p></div> +<p>The primary solution to these is to look in the logs for exceptions, see what’s going wrong (Property doesn’t exist? Malformed file couldn’t be loaded?), and adjust appropriately.</p></div> <div class="section"> <h2><a name="Testing_changes_without_cycling_Vagrant_build"></a>Testing changes without cycling Vagrant build</h2> <p>There are techniques we can use to avoid spinning down and spinning back up Vagrant.</p> @@ -738,73 +574,73 @@ METRON_LOG_DIR="/var/log/metron" <p>Ambari stores the Python files from the service in a couple places. We’ll want to update the files, then have Ambari pick up the updated versions and use them as the new basis. A reinstall of Metron is unnecessary for this type of testing.</p> <p>Specifically, the server files live in</p> -<div class="source"> -<div class="source"> -<pre>/var/lib/ambari-server/resources/mpacks/metron-ambari.mpack-0.4.0.0/common-services +<div> +<div> +<pre class="source">/var/lib/ambari-server/resources/mpacks/metron-ambari.mpack-0.4.0.0/common-services +/var/lib/ambari-server/resources/mpacks/elasticsearch-ambari.mpack-0.4.0.0/common-services /var/lib/ambari-agent/cache/common-services </pre></div></div> + <p>e.g. enrichment_commands.py can be found in:</p> -<div class="source"> -<div class="source"> -<pre>/var/lib/ambari-server/resources/mpacks/metron-ambari.mpack-0.4.0.0/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py +<div> +<div> +<pre class="source">/var/lib/ambari-server/resources/mpacks/metron-ambari.mpack-0.4.0.0/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py /var/lib/ambari-agent/cache/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py </pre></div></div> + <p>A <tt>find</tt> command can also be useful in quickly locating the exact location of a file, e.g.</p> -<div class="source"> -<div class="source"> -<pre>[root@node1 ~]# find /var/lib/ -name enrichment_commands.py +<div> +<div> +<pre class="source">[root@node1 ~]# find /var/lib/ -name enrichment_commands.py /var/lib/ambari-server/resources/mpacks/metron-ambari.mpack-0.4.0.0/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py /var/lib/ambari-agent/cache/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py </pre></div></div> -<p>The steps to update, for anything affecting an Ambari agent node, e.g. setup scripts during the service install process, are:</p> +<p>The steps to update, for anything affecting an Ambari agent node, e.g. setup scripts during the service install process, are:</p> <ol style="list-style-type: decimal"> - + <li>Stop Metron through Ambari. If the property in question is used by the topologies, we’ll want them stopped, so they can be restarted with the new property. This can sometimes be skipped if the change doesn’t affect the topologies themselves, but is a case by case choice.</li> - <li>Edit the file(s) with your changes. The ambari-agent file must be edited, but generally better to update both for consistency.</li> - -<li> -<p>Restart the Ambari Agent to get the cache to pick up the modified file</p> - -<div class="source"> -<div class="source"> -<pre> ambari-agent restart -</pre></div></div></li> - +<li>Restart the Ambari Agent to get the cache to pick up the modified file + +<div> +<div> +<pre class="source">ambari-agent restart +</pre></div></div> +</li> <li>Start Metron through Ambari if it was stopped.</li> </ol></div> <div class="section"> <h3><a name="Reinstalling_the_mpack"></a>Reinstalling the mpack</h3> <p>After we’ve modified files in Ambari and the mpack is working, it is a good idea to reinstall it. Fortunately this can be done without rebuilding the Vagrant environment by following these steps:</p> - <ol style="list-style-type: decimal"> - + <li>Stop Metron through Ambari and remove the Metron service</li> - -<li> -<p>Rebuild the mpack on your local machine and deploy it to Vagrant, ensuring that all changes made directly to files in Ambari were also made in your local environment</p> - -<div class="source"> -<div class="source"> -<pre> cd metron-deployment - mvn clean package - scp packaging/ambari/metron-mpack/target/metron_mpack-0.4.0.0.tar.gz root@node1:~ -</pre></div></div></li> - -<li> -<p>Log in to Vagrant, deploy the mpack and restart Ambari</p> - -<div class="source"> -<div class="source"> -<pre> ssh root@node1 - ambari-server install-mpack --mpack=metron_mpack-0.4.0.0.tar.gz --verbose --force - ambari-server restart -</pre></div></div></li> - +<li>Rebuild the mpack on your local machine and deploy it to Vagrant, ensuring that all changes made directly to files in Ambari were also made in your local environment + +<div> +<div> +<pre class="source">cd metron-deployment +mvn clean package +scp packaging/ambari/metron-mpack/target/metron_mpack-0.4.0.0.tar.gz root@node1:~ +</pre></div></div> +</li> +<li>Log in to Vagrant, deploy the mpack and restart Ambari + +<div> +<div> +<pre class="source">ssh root@node1 +ambari-server install-mpack --mpack=metron_mpack-0.4.0.0.tar.gz --verbose --force +ambari-server restart +</pre></div></div> +</li> <li>Install the mpack through Ambari as you normally would</li> +<li> + +<p>The same steps can be followed for Elasticsearch and Kibana by similary deploying the ES MPack located in elasticsearch-mpack/target.</p> +</li> </ol></div></div> <div class="section"> <h2><a name="Configuration_involving_dependency_services"></a>Configuration involving dependency services</h2> @@ -812,88 +648,350 @@ METRON_LOG_DIR="/var/log/metron" <p>The value is defined in a map in <tt>getSTORMSiteDesiredValues(self, is_secured)</tt>. This map is used by <tt>validateSTORMSiteConfigurations</tt> to ensure that the value used by the component is what we need it to be. Generally, only the map should need to be modified.</p> <p>Properties from the other services can also be used and examined. We use this to build the appropriate URL for Storm, because it differs based on Kerberos security. For example, to retrieve from another service, access the configurations array, retrieve the appropriate config file, retrieve the properties, and finally the desired property.</p> -<div class="source"> -<div class="source"> -<pre>stormUIServerPort = services["configurations"]["storm-site"]["properties"]["ui.port"] +<div> +<div> +<pre class="source">stormUIServerPort = services["configurations"]["storm-site"]["properties"]["ui.port"] </pre></div></div> + <p>The security of the cluster can be checked with</p> -<div class="source"> -<div class="source"> -<pre>is_secured = self.isSecurityEnabled(services) +<div> +<div> +<pre class="source">is_secured = self.isSecurityEnabled(services) </pre></div></div> + <p>Note that the configuration expectations will be properly enforced if the cluster is upgraded (e.g. the Storm URL is properly updated) and it does not need to occur manually.</p></div> <div class="section"> <h2><a name="Kerberos"></a>Kerberos</h2> <p>Any scripts interacting with underlying secured tech needs to have <tt>kinit</tt> run as needed. This includes anything interacting with HDFS, HBase, Kafka, Storm, etc. This <tt>kinit</tt> should be run in a conditional statement, to avoid breaking non-Kerberized clusters.</p> <p>Ambari can run <tt>kinit</tt>, as a given user with the appropriate keytab like so:</p> -<div class="source"> -<div class="source"> -<pre>if self.__params.security_enabled: +<div> +<div> +<pre class="source">if self.__params.security_enabled: metron_security.kinit(self.__params.kinit_path_local, self.__params.metron_keytab_path, self.__params.metron_principal_name, execute_user=self.__params.metron_user) </pre></div></div> + <p>The <tt>security_enabled</tt> param is already made available, along with appropriate keytabs for metron, hdfs, kafka, and hbase users.</p></div> <div class="section"> <h2><a name="Best_practices"></a>Best practices</h2> - <ul> - + <li> + <p>Write scripts to be idempotent. The pattern currently used is to write a file out when a task is finished, e.g. setting up ACLs or tables. For example, when indexing is configured, a file is written out and checked based on a property.</p> - -<div class="source"> -<div class="source"> -<pre> def set_configured(self): - File(self.__params.indexing_configured_flag_file, - content="", - owner=self.__params.metron_user, - mode=0755) -</pre></div></div> -<p>This is checked in the indexing master</p></li> -</ul> -<div class="source"> -<div class="source"> -<pre> if not commands.is_configured(): - commands.init_kafka_topics() - commands.init_hdfs_dir() - commands.set_configured() +<div> +<div> +<pre class="source">def set_configured(self): + File(self.__params.indexing_configured_flag_file, + content="", + owner=self.__params.metron_user, + mode=0755) </pre></div></div> +<p>This is checked in the indexing master</p> + +<div> +<div> +<pre class="source">if not commands.is_configured(): + commands.init_kafka_topics() + commands.init_hdfs_dir() + commands.set_configured() +</pre></div></div> +</li> +<li> + +<p>Ensure ACLs are properly managed. This includes Kafka and HBase. Often this involves a config file written out as above because this isn’t idempotent!</p> <ul> - -<li>Ensure ACLs are properly managed. This includes Kafka and HBase. Often this involves a config file written out as above because this isn’t idempotent! - -<ul> - + <li>Make sure to <tt>kinit</tt> as the correct user for setting up ACLs in a secured cluster. This is usually kafka for Kafka and hbase for HBase.</li> - <li>See <tt>set_hbase_acls</tt> in <tt>METRON.CURRENT/package/scripts/enrichment_commands.py</tt> for an HBase example</li> - -<li>See <tt>init_kafka_acls</tt> in <tt>METRON.CURRENT/package/scripts/enrichment_commands.py</tt> and <tt>METRON.CURRENT/package/scripts/metron_service.py</tt> for an Kafka example</li> - </ul></li> +<li>See <tt>init_kafka_acls</tt> in <tt>METRON.CURRENT/package/scripts/enrichment_commands.py</tt> and <tt>METRON.CURRENT/package/scripts/metron_service.py</tt> for an Kafka example</li> +</ul> +</li> </ul></div> - </div> - </div> - </div> +<div class="section"> +<h2><a name="Upgrading_MPack_Services"></a>Upgrading MPack Services</h2> +<p>Apache Metron currently provides one service as part of its Metron MPack</p> +<ul> - <hr/> +<li>Metron</li> +</ul> +<p>Apache Metron currently provides two services as part of its Elasticsearch MPack</p> +<ul> + +<li>Elasticsearch</li> +<li>Kibana</li> +</ul> +<p>There is currently no mechanism provided for multi-version or backwards compatibility. If you upgrade a service, e.g. Elasticsearch 2.x to 5.x, that is the only version that will be supported by Ambari via MPack.</p> +<p>The main steps for upgrading a service are split into add-on and common services for each service within the MPack as follows:</p> +<ul> + +<li>Update the common services +<ul> + +<li>Change the service directory to use the new product version number</li> +<li>Update metainfo.xml</li> +</ul> +</li> +<li>Update the add-on services +<ul> + +<li>Change the service directory to use the new product version number</li> +<li>Update repoinfo.xml</li> +<li>Update metainfo.xml</li> +</ul> +</li> +<li>Update mpack.json</li> +</ul> +<div class="section"> +<h3><a name="Update_Elasticsearch"></a>Update Elasticsearch</h3> +<div class="section"> +<h4><a name="Update_Common_Services"></a>Update Common Services</h4> +<ol style="list-style-type: decimal"> + +<li> + +<p>Change service directory names for Elasticsearch to the new desired version</p> + +<div> +<div> +<pre class="source">metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/${YOUR_VERSION_NUMBER_HERE} +</pre></div></div> + +<p>e.g.</p> + +<div> +<div> +<pre class="source">metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2 +</pre></div></div> +</li> +<li> + +<p>Update metainfo.xml</p> +<p>Change the version number and package name in <tt>metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/${YOUR_VERSION_NUMBER_HERE}/metainfo.xml</tt>, e.g.</p> + +<div> +<div> +<pre class="source"><version>5.6.2</version> +... +<osSpecifics> + <osSpecific> + <osFamily>any</osFamily> + <packages> + <package> + <name>elasticsearch-5.6.2</name> + </package> + </packages> + </osSpecific> +</osSpecifics> +</pre></div></div> +</li> +</ol></div> +<div class="section"> +<h4><a name="Update_Add-on_Services"></a>Update Add-on Services</h4> +<ol style="list-style-type: decimal"> + +<li> + +<p>Change service directory names for Elasticsearch to the new desired version</p> + +<div> +<div> +<pre class="source">metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/ELASTICSEARCH/${YOUR_VERSION_NUMBER_HERE} +</pre></div></div> + +<p>e.g.</p> + +<div> +<div> +<pre class="source">metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/ELASTICSEARCH/5.6.2 +</pre></div></div> +</li> +<li> + +<p>Update repoinfo.xml</p> +<p>See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html";>https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html</a> for the latest info.</p> +<p>Modify the baseurl and repoid in <tt>metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/ELASTICSEARCH/${YOUR_VERSION_NUMBER_HERE}/repos/repoinfo.xml</tt>, e.g.</p> + +<div> +<div> +<pre class="source"><baseurl>https://artifacts.elastic.co/packages/5.x/yum</baseurl> +<repoid>elasticsearch-5.x</repoid> +<reponame>ELASTICSEARCH</reponame> +</pre></div></div> +</li> +<li> + +<p>Update metainfo.xml</p> +<p>Change the version number in <tt>metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/ELASTICSEARCH/${YOUR_VERSION_NUMBER_HERE}/metainfo.xml</tt>. Also make sure to update the “extends” version to point to the updated common-services version, e.g.</p> + +<div> +<div> +<pre class="source"><name>ELASTICSEARCH</name> +<version>5.6.2</version> +<extends>common-services/ELASTICSEARCH/5.6.2</extends> +</pre></div></div> +</li> +</ol></div> +<div class="section"> +<h4><a name="Update_mpack.json"></a>Update mpack.json</h4> +<ol style="list-style-type: decimal"> +<li>Update the corresponding service_version in the service_versions_map, e.g. + +<div> +<div> +<pre class="source">... +"service_versions_map": [ + { + "service_name" : "ELASTICSEARCH", + "service_version" : "5.6.2", + "applicable_stacks" : [ + ... + ] + }, + ... + ] +... +</pre></div></div> +</li> +</ol></div></div> +<div class="section"> +<h3><a name="Kibana"></a>Kibana</h3> +<p><b>Note:</b> Curator is included with the Kibana service</p> +<div class="section"> +<h4><a name="Update_Common_Services"></a>Update Common Services</h4> +<ol style="list-style-type: decimal"> + +<li> + +<p>Change service directory names for Kibana to the new desired version</p> + +<div> +<div> +<pre class="source">metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/${YOUR_VERSION_NUMBER_HERE} +</pre></div></div> + +<p>e.g.</p> + +<div> +<div> +<pre class="source">metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2 +</pre></div></div> +</li> +<li> + +<p>Update metainfo.xml</p> +<p>Change the version number and package name in <tt>metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/${YOUR_VERSION_NUMBER_HERE}/metainfo.xml</tt>, e.g.</p> + +<div> +<div> +<pre class="source"><version>5.6.2</version> +... +<packages> + ... + <package> + <name>kibana-5.6.2</name> + </package> +</packages> +</pre></div></div> +</li> +</ol></div> +<div class="section"> +<h4><a name="Update_Add-on_Services"></a>Update Add-on Services</h4> +<ol style="list-style-type: decimal"> + +<li> + +<p>Change service directory names for Kibana to the new desired version</p> + +<div> +<div> +<pre class="source">metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/KIBANA/${YOUR_VERSION_NUMBER_HERE} +</pre></div></div> + +<p>e.g.</p> + +<div> +<div> +<pre class="source">metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/KIBANA/5.6.2 +</pre></div></div> +</li> +<li> + +<p>Update repoinfo.xml</p> +<p><b>Note:</b> for Curator, there is a different repo for rhel 6 vs rhel 7</p> +<p>See the following links for current repo information for Kibana and Curator.</p> +<ul> + +<li><a class="externalLink" href="https://www.elastic.co/guide/en/kibana/current/rpm.html";>https://www.elastic.co/guide/en/kibana/current/rpm.html</a></li> +<li><a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/client/curator/current/yum-repository.html";>https://www.elastic.co/guide/en/elasticsearch/client/curator/current/yum-repository.html</a></li> +</ul> +<p>Modify the baseurl’s and repoid’s in <tt>metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/KIBANA/${YOUR_VERSION_NUMBER_HERE}/repos/repoinfo.xml</tt>, e.g.</p> + +<div> +<div> +<pre class="source"><baseurl>https://artifacts.elastic.co/packages/5.x/yum</baseurl> +<repoid>kibana-5.x</repoid> +<reponame>KIBANA</reponame> +... +<baseurl>http://packages.elastic.co/curator/5/centos/6</baseurl> +<repoid>ES-Curator-5.x</repoid> +<reponame>CURATOR</reponame> +</pre></div></div> +</li> +<li> + +<p>Update metainfo.xml</p> +<p>Change the version number in <tt>metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/KIBANA/${YOUR_VERSION_NUMBER_HERE}/metainfo.xml</tt>. Also make sure to update the “extends” version to point to the updated common-services version, e.g.</p> + +<div> +<div> +<pre class="source"><name>KIBANA</name> +<version>5.6.2</version> +<extends>common-services/KIBANA/5.6.2</extends> +</pre></div></div> +</li> +</ol></div> +<div class="section"> +<h4><a name="Update_mpack.json"></a>Update mpack.json</h4> +<ol style="list-style-type: decimal"> + +<li>Update the corresponding service_version in the service_versions_map, e.g. + +<div> +<div> +<pre class="source">... +"service_versions_map": [ + { + "service_name" : "KIBANA", + "service_version" : "5.6.2", + "applicable_stacks" : [ + ... + ] + }, + ... + ] +... +</pre></div></div> +</li> +</ol></div></div></div> + </div> + </div> + </div> + <hr/> <footer> - <div class="container-fluid"> - <div class="row span12">Copyright © 2018 - <a href="https://www.apache.org";>The Apache Software Foundation</a>. - All Rights Reserved. - + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> </div> - - - - </div> </footer> </body> </html>
http://git-wip-us.apache.org/repos/asf/metron/blob/ae1d3eb9/site/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html b/site/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html new file mode 100644 index 0000000..dd97717 --- /dev/null +++ b/site/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html @@ -0,0 +1,337 @@ +<!DOCTYPE html> +<!-- + | Generated by Apache Maven Doxia Site Renderer 1.8 from src/site/markdown/metron-deployment/packaging/ambari/metron-mpack/index.md at 2018-06-07 + | Rendered using Apache Maven Fluido Skin 1.7 +--> +<html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> + <head> + <meta charset="UTF-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /> + <meta name="Date-Revision-yyyymmdd" content="20180607" /> + <meta http-equiv="Content-Language" content="en" /> + <title>Metron – </title> + <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.7.min.css" /> + <link rel="stylesheet" href="../../../../css/site.css" /> + <link rel="stylesheet" href="../../../../css/print.css" media="print" /> + <script type="text/javascript" src="../../../../js/apache-maven-fluido-1.7.min.js"></script> +<script type="text/javascript"> + $( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } ); + </script> + </head> + <body class="topBarDisabled"> + <div class="container-fluid"> + <div id="banner"> + <div class="pull-left"><a href="http://metron.apache.org/"; id="bannerLeft"><img src="../../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/></a></div> + <div class="pull-right"></div> + <div class="clear"><hr/></div> + </div> + + <div id="breadcrumbs"> + <ul class="breadcrumb"> + <li class=""><a href="http://www.apache.org"; class="externalLink" title="Apache">Apache</a><span class="divider">/</span></li> + <li class=""><a href="http://metron.apache.org/"; class="externalLink" title="Metron">Metron</a><span class="divider">/</span></li> + <li class=""><a href="../../../../index.html" title="Documentation">Documentation</a><span class="divider">/</span></li> + <li class="active "></li> + <li id="publishDate" class="pull-right"><span class="divider">|</span> Last Published: 2018-06-07</li> + <li id="projectVersion" class="pull-right">Version: 0.5.0</li> + </ul> + </div> + <div class="row-fluid"> + <div id="leftColumn" class="span2"> + <div class="well sidebar-nav"> + <ul class="nav nav-list"> + <li class="nav-header">User Documentation</li> + <li><a href="../../../../index.html" title="Metron"><span class="icon-chevron-down"></span>Metron</a> + <ul class="nav nav-list"> + <li><a href="../../../../CONTRIBUTING.html" title="CONTRIBUTING"><span class="none"></span>CONTRIBUTING</a></li> + <li><a href="../../../../Upgrading.html" title="Upgrading"><span class="none"></span>Upgrading</a></li> + <li><a href="../../../../metron-analytics/index.html" title="Analytics"><span class="icon-chevron-right"></span>Analytics</a></li> + <li><a href="../../../../metron-contrib/metron-docker/index.html" title="Docker"><span class="none"></span>Docker</a></li> + <li><a href="../../../../metron-contrib/metron-performance/index.html" title="Performance"><span class="none"></span>Performance</a></li> + <li><a href="../../../../metron-deployment/index.html" title="Deployment"><span class="icon-chevron-down"></span>Deployment</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup"><span class="none"></span>Kerberos-ambari-setup</a></li> + <li><a href="../../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup"><span class="none"></span>Kerberos-manual-setup</a></li> + <li><a href="../../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2"><span class="none"></span>Amazon-ec2</a></li> + <li><a href="../../../../metron-deployment/ansible/index.html" title="Ansible"><span class="icon-chevron-right"></span>Ansible</a></li> + <li><a href="../../../../metron-deployment/development/index.html" title="Development"><span class="icon-chevron-right"></span>Development</a></li> + <li><a href="../../../../metron-deployment/other-examples/index.html" title="Other-examples"><span class="icon-chevron-right"></span>Other-examples</a></li> + <li><a href="../../../../metron-deployment/packaging/ambari/index.html" title="Ambari"><span class="icon-chevron-down"></span>Ambari</a> + <ul class="nav nav-list"> + <li><a href="../../../../metron-deployment/packaging/ambari/elasticsearch-mpack/index.html" title="Elasticsearch-mpack"><span class="none"></span>Elasticsearch-mpack</a></li> + <li class="active"><a href="#"><span class="none"></span>Mpack</a></li> + </ul> +</li> + <li><a href="../../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker"><span class="none"></span>Ansible-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/deb-docker/index.html" title="Deb-docker"><span class="none"></span>Deb-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker"><span class="none"></span>Rpm-docker</a></li> + <li><a href="../../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build"><span class="none"></span>Packer-build</a></li> + </ul> +</li> + <li><a href="../../../../metron-interface/metron-alerts/index.html" title="Alerts"><span class="none"></span>Alerts</a></li> + <li><a href="../../../../metron-interface/metron-config/index.html" title="Config"><span class="none"></span>Config</a></li> + <li><a href="../../../../metron-interface/metron-rest/index.html" title="Rest"><span class="none"></span>Rest</a></li> + <li><a href="../../../../metron-platform/index.html" title="Platform"><span class="icon-chevron-right"></span>Platform</a></li> + <li><a href="../../../../metron-sensors/index.html" title="Sensors"><span class="icon-chevron-right"></span>Sensors</a></li> + <li><a href="../../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"><span class="none"></span>Stellar-3rd-party-example</a></li> + <li><a href="../../../../metron-stellar/stellar-common/index.html" title="Stellar-common"><span class="icon-chevron-right"></span>Stellar-common</a></li> + <li><a href="../../../../metron-stellar/stellar-zeppelin/index.html" title="Stellar-zeppelin"><span class="none"></span>Stellar-zeppelin</a></li> + <li><a href="../../../../use-cases/index.html" title="Use-cases"><span class="icon-chevron-right"></span>Use-cases</a></li> + </ul> +</li> +</ul> + <hr /> + <div id="poweredBy"> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> + <div class="clear"></div> +<a href="http://maven.apache.org/"; title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="../../../../images/logos/maven-feather.png" /></a> + </div> + </div> + </div> + <div id="bodyColumn" class="span10" > +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +<p>This provides a Management Pack (MPack) extension for <a class="externalLink" href="https://ambari.apache.org/";>Apache Ambari</a> that simplifies the provisioning, management and monitoring of Metron on clusters of any size.</p> +<p>This allows you to easily install Metron using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos.</p> +<div class="section"> +<div class="section"> +<h3><a name="Prerequisites"></a>Prerequisites</h3> +<ul> + +<li> + +<p>Ambari 2.4.2+</p> +</li> +<li> + +<p>Installable Metron packages (either RPMs or DEBs) located in a repository on each host at <tt>/localrepo</tt>.</p> +</li> +<li> + +<p>A <a class="externalLink" href="https://nodejs.org/en/download/package-manager/";>Node.js</a> repository installed on the host running the Management and Alarm UI.</p> +</li> +</ul></div> +<div class="section"> +<h3><a name="Quick_Start"></a>Quick Start</h3> +<ol style="list-style-type: decimal"> + +<li> + +<p>Build the Metron MPack. Execute the following command from the project’s root directory.</p> + +<div> +<div> +<pre class="source">mvn clean package -Pmpack -DskipTests +</pre></div></div> +</li> +<li> + +<p>This results in the Mpack being produced at the following location.</p> + +<div> +<div> +<pre class="source">metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz +</pre></div></div> +</li> +<li> + +<p>Copy the tarball to the host where Ambari Server is installed.</p> +</li> +<li> + +<p>Ensure that Ambari Server is stopped.</p> +</li> +<li> + +<p>Install the MPack.</p> + +<div> +<div> +<pre class="source">ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz --verbose +</pre></div></div> +</li> +<li> + +<p>Install the Metron packages (RPMs or DEBs) in a local repository on each host where a Metron component is installed. By default, the repository is expected to exist at <tt>/localrepo</tt>.</p> +<p>On hosts where only a Metron client is installed, the local repository must exist, but it does not need to contain Metron packages. For example to create an empty repository for an RPM-based system, run the following commands.</p> + +<div> +<div> +<pre class="source">yum install createrepo +mkdir /localrepo +cd /localrepo +createrepo +</pre></div></div> +</li> +<li> + +<p>Metron will now be available as an installable service within Ambari.</p> +</li> +</ol></div> +<div class="section"> +<h3><a name="Installation_Notes"></a>Installation Notes</h3> +<p>The MPack will make all Metron services available in Ambari in the same manner as any other services in a stack. These can be installed using Ambari’s user interface using “Add Services” or during an initial cluster install.</p> +<div class="section"> +<h4><a name="Kerberization"></a>Kerberization</h4> +<p>The MPack allows Metron to be automatically kerberized in two different ways.</p> +<ul> + +<li>Metron can be installed on a non-kerberized cluster and then the entire cluster can be kerberized using Ambari.</li> +<li>Alternatively, Metron can be installed on top of an already kerberized cluster.</li> +</ul> +<p>Using the MPack is preferred, but instructions for manually Kerberizing a cluster with Metron can be found at <a href="../../Kerberos-manual-setup.html">Kerberos-manual-setup.md</a>.</p> +<div class="section"> +<h5><a name="Metron_Client"></a>Metron Client</h5> +<p>A “Metron Client” must be installed on each supervisor node in a kerberized cluster. This client ensures that the Metron keytab and <tt>client_jaas.conf</tt> get distributed to each node in order to allow reading and writing from Kafka.</p> +<ul> + +<li>When Metron is already installed on the cluster, installation of the “Metron Client” should be done before Kerberizing.</li> +<li>When adding Metron to an already Kerberized cluster, ensure that all supervisor nodes receive a Metron client.</li> +</ul></div> +<div class="section"> +<h5><a name="Restarts"></a>Restarts</h5> +<p>Storm (and the Metron topologies) must be restarted after Metron is installed on an already Kerberized cluster. The restart triggers several Storm configurations to get updated and Metron will be unable to write to Kafka without a restart.</p> +<p>Kerberizing a cluster with a pre-existing Metron, automatically restarts all services during Kerberization. No additional manual restart is needed in this case.</p></div></div> +<div class="section"> +<h4><a name="Kibana_Dashboards"></a>Kibana Dashboards</h4> +<p>The dashboards installed by the Kibana custom action are managed by two JSON files:</p> +<ul> + +<li>metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/dashboard/kibana.template</li> +<li>metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/dashboard/dashboard-bulkload.json</li> +</ul> +<p>The first file, <tt>kibana.template</tt>, is an Elasticsearch template that specifies the proper mapping types for the Kibana index. This configuration is necessary due to a bug in the default dynamic mappings provided by Elasticsearch for long types versus integer that are incompatible with Kibana [1]. The second file, <tt>dashboard-bulkload.json</tt>, contains all of the dashboard metadata necessary to create the Metron dashboard. It is an Elasticsearch bulk-insert formatted file [2] that contains a series of documents necessary for setting up the dashboard in Elasticsearch. The main features installed are index patterns, searches, and a variety of visualizations that are used in the Metron dashboard.</p> +<p>Deploying the existing dashboard is easy. Once the MPack is installed, run the Metron service’s action “Load Template” to install dashboards. This will no longer overwrite the .kibana in Elasticsearch. The bulk load is configured to fail inserts for existing documents. If you want to <i>completely</i> reload the dashboard, you would need to delete the .kibana index and reload again from Ambari.</p> +<ol style="list-style-type: decimal"> + +<li><a class="externalLink" href="https://github.com/elastic/kibana/issues/9888#issuecomment-298096954";>https://github.com/elastic/kibana/issues/9888#issuecomment-298096954</a></li> +<li><a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docs-bulk.html";>https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docs-bulk.html</a></li> +</ol> +<div class="section"> +<h5><a name="Modifying_Pre-Installed_Dashboards"></a>Modifying Pre-Installed Dashboards</h5> +<p>You can modify dashboards in Kibana and bring those changes into the core MPack distribution by performing the following steps:</p> +<ol style="list-style-type: decimal"> + +<li>Export the .kibana index from ES</li> +<li>Convert the data into the ES bulk load format</li> +<li>Replace the dashboard-bulkload.json file in the Metron MPack.</li> +</ol> +<p>You can export the .kibana index using a tool like <a class="externalLink" href="https://github.com/taskrabbit/elasticsearch-dump";>https://github.com/taskrabbit/elasticsearch-dump</a>. The important feature is to have one document per line. Here’s an exmaple export using elasticsearch-dump</p> + +<div> +<div> +<pre class="source">elasticdump \ + --input=http://node1:9200/.kibana \ + --output=~/dashboard-data.json \ + --type=data +</pre></div></div> + +<p>Once you’ve exported the data, you can now format it as a bulk load ES file by running the import/export tool located in metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExport.java. This tool can be run from full-dev as follows</p> + +<div> +<div> +<pre class="source">java -cp $METRON_HOME/lib/metron-elasticsearch-0.4.2-uber.jar org.apache.metron.elasticsearch.bulk.ElasticsearchImportExport \ + ~/dashboard-data.json \ + ~/dashboard-bulkload.json +</pre></div></div> + +<p>Locate the “create” command for setting the default index by searching for “5.6.2”. Change “create” to “index” so that it modifies the existing value. It should look similar to line 1 below.</p> + +<div> +<div> +<pre class="source">{ "index" : { "_id": "5.6.2", "_type": "config" } } +{"defaultIndex":"AV-S2e81hKs1cXXnFMqN"} +</pre></div></div> + +<p>Now copy this file to the Kibana MPack, overwriting the existing bulk load file. That should be everything needed to backup the dashboard.</p> +<p><b>Note</b>: the dashboard Python Pickle binary file is deprecated and no longer used for backing up and restoring Kibana dashboards. The tooling is still provided as of this version but is expected to be removed in the future. A section describing the deprecated backup process remains below.</p></div> +<div class="section"> +<h5><a name="Deprecated_Dashboard_Install.2FBackup_Instructions"></a>Deprecated Dashboard Install/Backup Instructions</h5> +<p>The dashboards installed by the Kibana custom action are managed by the dashboard.p file. This file is created by exporting existing dashboards from a running Kibana instance.</p> +<p>To create a new version of the file, make any necessary changes to Kibana (e.g. on full-dev), and export with the appropriate script.</p> +<p><b>Script Options</b></p> + +<div> +<div> +<pre class="source">[elasticsearch_host] ES host +[elasticsearch_port] ES port number +[input_output_filename] Filename used for reading or writing out pickle file +[-s] Flag to indicate that the .kibana index should be saved locally. Not including this flag will overwrite the .kibana + index completely with the contents of 'input_output_filename'. Careful with this. +</pre></div></div> + +<p><b>Saving a Backup</b></p> + +<div> +<div> +<pre class="source">python packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/dashboard/dashboardindex.py \ +$ES_HOST 9200 \ +~/dashboard.p -s +</pre></div></div> + +<p><b>Restoring From a Backup</b></p> + +<div> +<div> +<pre class="source">python packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/dashboard/dashboardindex.py \ +$ES_HOST 9200 \ +~/dashboard.p +</pre></div></div> + +<p><b>Note</b>: This method of writing the Kibana dashboard to Elasticsearch will overwrite the entire .kibana index. Be sure to first backup the index first using either the new JSON method described above, or writing out the dashboard.p pickle file using the old method (passing -s option to dashboardindex.py) described here.</p></div></div> +<div class="section"> +<h4><a name="Zeppelin_Import"></a>Zeppelin Import</h4> +<p>A custom action is available in Ambari to import Zeppelin dashboards. See the <a href="../../../../metron-platform/metron-indexing/index.html">metron-indexing documentation</a> for more information.</p></div> +<div class="section"> +<h4><a name="Offline_Installation"></a>Offline Installation</h4> +<p>Retrieval of the GeoIP database is the only point during installation that reaches out to the internet. For an offline installation, the URL for the GeoIP database can be manually set to a local path on the file system such as <tt>file:///home/root/geoip/GeoLite2-City.mmdb.gz</tt>.</p> +<p>The RPMs DO NOT reach out to the internet (because there is currently no hosting for them). They look on the local filesystem in <tt>/localrepo</tt>.</p></div> +<div class="section"> +<h4><a name="Limitations"></a>Limitations</h4> +<p>There are a few limitations that should be addressed to improve the Metron MPack installation.</p> +<ul> + +<li> + +<p>There is no external hosting for Metron packages (either RPMs or DEBs). These have to be built locally and installed on each host in a repository located at <tt>/localrepo</tt>.</p> +</li> +<li> + +<p>Several configuration parameters used when installing Metron could retrieved from Ambari rather than requiring user input.</p> +</li> +<li> + +<p>The MPack does not support upgrades.</p> +</li> +</ul></div></div></div> + </div> + </div> + </div> + <hr/> + <footer> + <div class="container-fluid"> + <div class="row-fluid"> +é 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, + and the Apache Metron project logo are trademarks of The Apache Software Foundation. + </div> + </div> + </footer> + </body> +</html>
