http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/Kerberos-manual-setup.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/Kerberos-manual-setup.html b/site/current-book/metron-deployment/Kerberos-manual-setup.html index c07156d..1c1ed0a 100644 --- a/site/current-book/metron-deployment/Kerberos-manual-setup.html +++ b/site/current-book/metron-deployment/Kerberos-manual-setup.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Kerberos Setup</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../metron-deployment/index.html" title="Deployment"> @@ -171,7 +171,7 @@ <i class="icon-chevron-right"></i> Roles</a> </li> - + <li> <a href="../metron-deployment/vagrant/index.html" title="Vagrant"> @@ -201,20 +201,27 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../metron-sensors/index.html" title="Sensors"> <i class="icon-chevron-right"></i> Sensors</a> </li> + + <li> + + <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -222,7 +229,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../use-cases/index.html" title="Use-cases"> @@ -252,7 +259,7 @@ <div id="bodyColumn" class="span9" > <h1>Kerberos Setup</h1> -<p>This document provides instructions for kerberizing Metron’s Vagrant-based development environments; “Quick Dev” and “Full Dev”. These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment <a href="../index.html">README.md</a>.</p> +<p>This document provides instructions for kerberizing Metron’s Vagrant-based development environments. These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment <a href="../index.html">README.md</a>.</p> <ul> @@ -282,7 +289,7 @@ <ol style="list-style-type: decimal"> <li> -<p>Deploy a Vagrant development environment; either <a href="vagrant/full-dev-platform/index.html">Full Dev</a> or <a href="vagrant/quick-dev-platform/index.html">Quick Dev</a>.</p></li> +<p>Deploy the <a href="vagrant/full-dev-platform/index.html">development environment.</a>.</p></li> <li> <p>Export the following environment variables. These need to be set for the remainder of the instructions. Replace <tt>node1</tt> with the appropriate hosts, if you are running Metron anywhere other than Vagrant.</p> @@ -297,7 +304,7 @@ export ELASTICSEARCH=node1:9200 export BROKERLIST=node1:6667 export HDP_HOME="/usr/hdp/current" export KAFKA_HOME="${HDP_HOME}/kafka-broker" -export METRON_VERSION="0.4.1" +export METRON_VERSION="0.4.2" export METRON_HOME="/usr/metron/${METRON_VERSION}" </pre></div></div></li> @@ -343,16 +350,18 @@ sudo -u hdfs hdfs dfs -chmod 770 /user/metron </pre></div></div></li> <li> -<p>Define the host, <tt>node1</tt>, as the KDC.</p> +<p>Define the current host as the KDC.</p> <div class="source"> <div class="source"> -<pre>sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf +<pre>KDC=`hostname` +sed -i.orig 's/kerberos.example.com/'"$KDC"'/g' /etc/krb5.conf cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts </pre></div></div></li> <li> -<p>Ensure the KDC can issue renewable tickets. This can be necessary on a real cluster, but should not be on full-dev. In /var/kerberos/krb5kdc/kdc.conf ensure the following is in the realm section</p> +<p>Ensure that the KDC can issue renewable tickets. This may be necessary on a real cluster, but should not be on <a href="vagrant/full-dev-platform/index.html">Full Dev</a>.</p> +<p>Edit <tt>/var/kerberos/krb5kdc/kdc.conf</tt> and ensure the following is added to the <tt>realm</tt> section</p> <div class="source"> <div class="source"> @@ -360,49 +369,86 @@ cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts </pre></div></div></li> <li> -<p>Do not copy/paste this full set of commands as the <tt>kdb5_util</tt> command will not run as expected. Run the commands individually to ensure they all execute. This step takes a moment. It creates the kerberos database.</p> +<p>Create the KDC principal database. You will be prompted for a password. This step takes a moment.</p> <div class="source"> <div class="source"> <pre>kdb5_util create -s -/etc/rc.d/init.d/krb5kdc start -chkconfig krb5kdc on -/etc/rc.d/init.d/kadmin start +</pre></div></div></li> + +<li> +<p>Start the KDC and ensure that it starts on boot.</p> + +<div class="source"> +<div class="source"> +<pre>/etc/rc.d/init.d/krb5kdc start +chkconfig krb5kdc on +</pre></div></div></li> + +<li> +<p>Start the Kerberos Admin service and ensure that it starts on boot. </p> + +<div class="source"> +<div class="source"> +<pre>/etc/rc.d/init.d/kadmin start chkconfig kadmin on </pre></div></div></li> <li> -<p>Setup the <tt>admin</tt> and <tt>metron</tt> principals. You’ll <tt>kinit</tt> as the <tt>metron</tt> principal when running topologies. Make sure to remember the passwords.</p> +<p>Setup the <tt>admin</tt> principal. You will be prompted for a password; do not forget it.</p> <div class="source"> <div class="source"> <pre>kadmin.local -q "addprinc admin/admin" -kadmin.local -q "addprinc metron" +</pre></div></div></li> + +<li> +<p>Setup the <tt>metron</tt> principal. You will <tt>kinit</tt> as the <tt>metron</tt> principal when running topologies. You will be prompted for a password; do not forget it.</p> + +<div class="source"> +<div class="source"> +<pre>kadmin.local -q "addprinc metron" </pre></div></div></li> </ol></div> <div class="section"> <h2><a name="Verify_KDC"></a>Verify KDC</h2> -<p>Ticket renewal is by default disallowed in many linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron’s Storm topologies:</p> +<ol style="list-style-type: decimal"> + +<li> +<p>Ticket renewal is disallowed by default in many Linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron’s Storm topologies:</p> + <div class="source"> <div class="source"> -<pre>Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The TGT found is not renewable -</pre></div></div> -<p>Ensure the Metron keytab is renewable. Look for the ‘R’ flag from the following command</p> - +<pre>Exception in thread "main" java.lang.RuntimeException: +java.lang.RuntimeException: The TGT found is not renewable +</pre></div></div></li> + +<li> +<p>Ensure the Metron keytab is renewable. Look for the ‘R’ flag in the output of the following command.</p> + <div class="source"> <div class="source"> <pre>klist -f </pre></div></div> -<p>If the ‘R’ flags are present, you may skip to next section.</p> -<p>If the ‘R’ flags are absent, you will need to follow the below steps: If the KDC is already setup, then editing max_life and max_renewable_life in <tt>/var/kerberos/krb5kdc/kdc.conf</tt>, and restarting kadmin and krb5kdc services will not change the policies for existing users. </p> -<p>You need to set the renew lifetime for existing users and krbtgt realm. Modify the appropriate principals to allow renewable tickets using the following commands. Adjust the parameters to match your desired KDC parameters:</p> - + +<ul> + +<li>If the ‘R’ flags are present, you may skip to next section.</li> + +<li>If the ‘R’ flags are absent, you will need to follow the below steps:</li> + </ul></li> + +<li> +<p>If the KDC is already setup, then editing <tt>max_life</tt> and <tt>max_renewable_life</tt> in <tt>/var/kerberos/krb5kdc/kdc.conf</tt>, then restarting <tt>kadmin</tt> and <tt>krb5kdc</tt> services will not change the policies for existing users.</p> +<p>You need to set the renew lifetime for existing users and the <tt>krbtgt</tt> realm. Modify the appropriate principals to allow renewable tickets using the following commands. Adjust the parameters to match your desired KDC parameters:</p> + <div class="source"> <div class="source"> <pre>kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable krbtgt/example....@example.com" kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable met...@example.com" -</pre></div></div></div> +</pre></div></div></li> +</ol></div> <div class="section"> <h2><a name="Enable_Kerberos"></a>Enable Kerberos</h2> @@ -846,7 +892,7 @@ done <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/amazon-ec2/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/amazon-ec2/index.html b/site/current-book/metron-deployment/amazon-ec2/index.html index e81f4f5..4e7e771 100644 --- a/site/current-book/metron-deployment/amazon-ec2/index.html +++ b/site/current-book/metron-deployment/amazon-ec2/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Apache Metron on Amazon EC2</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -171,7 +171,7 @@ <i class="icon-chevron-right"></i> Roles</a> </li> - + <li> <a href="../../metron-deployment/vagrant/index.html" title="Vagrant"> @@ -201,20 +201,27 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> <i class="icon-chevron-right"></i> Sensors</a> </li> + + <li> + + <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -222,7 +229,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../../use-cases/index.html" title="Use-cases"> @@ -582,7 +589,7 @@ fatal: [ec2-52-26-113-221.us-west-2.compute.amazonaws.com]: UNREACHABLE! => { <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/index.html b/site/current-book/metron-deployment/index.html index d1071d7..72723a6 100644 --- a/site/current-book/metron-deployment/index.html +++ b/site/current-book/metron-deployment/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Overview</title> <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li class="active"> <a href="#"><i class="icon-chevron-down"></i>Deployment</a> @@ -171,7 +171,7 @@ <i class="icon-chevron-right"></i> Roles</a> </li> - + <li> <a href="../metron-deployment/vagrant/index.html" title="Vagrant"> @@ -201,20 +201,27 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../metron-sensors/index.html" title="Sensors"> <i class="icon-chevron-right"></i> Sensors</a> </li> + + <li> + + <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -222,7 +229,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../use-cases/index.html" title="Use-cases"> @@ -306,7 +313,7 @@ <p>Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2.</p></div> <div class="section"> <h2><a name="Vagrant"></a>Vagrant</h2> -<p>There are current two Vagrant modes, full-dev and quick-dev. Full-dev installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. Quick-dev re-installs the core Metron Services (Parsing, Enrichment, and Indexing)on a pre-built instance. Use quick-dev for testing out changes to core Metron services.</p> +<p>There is a development environment based on Vagrant that is referred to as “Full Dev”. This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure.</p> <div class="section"> <h3><a name="Prerequsities"></a>Prerequsities</h3> @@ -318,10 +325,7 @@ </ul></div> <div class="section"> <h3><a name="Full-Dev"></a>Full-Dev</h3> -<p>Navigate to <tt>metron/metron-deployment/vagrant/full-dev-platform</tt> and run <tt>vagrant up</tt>.</p></div> -<div class="section"> -<h3><a name="Quick-Dev"></a>Quick-Dev</h3> -<p>Navigate to <tt>metron/metron-deployment/vagrant/quick-dev-platform</tt> and run <tt>vagrant up</tt>.</p></div></div> +<p>Navigate to <tt>metron/metron-deployment/vagrant/full-dev-platform</tt> and run <tt>vagrant up</tt>.</p></div></div> <div class="section"> <h2><a name="Ambari_Management_Pack"></a>Ambari Management Pack</h2> <p>An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update.</p> @@ -348,7 +352,7 @@ <li>A cluster managed by Ambari 2.4.2+</li> -<li>Metron RPMs available on the cluster in the /localrepo directory. See <a href="#RPM">RPM</a> for further information.</li> +<li>Metron RPMs available on the cluster in the /localrepo directory. See <a href="#RPMs">RPMs</a> for further information.</li> <li><a class="externalLink" href="https://nodejs.org/en/download/package-manager/";>Node.js</a> repository installed on the Management UI host</li> </ul></div> @@ -467,7 +471,7 @@ mvn clean package -Pbuild-rpms <div class="section"> <h2><a name="Kibana_Dashboards"></a>Kibana Dashboards</h2> <p>The dashboards installed by the Kibana custom action are managed by the dashboard.p file. This file is created by exporting existing dashboards from a running Kibana instance.</p> -<p>To create a new version of the file, make any necessary changes to Kibana (e.g. on quick-dev), and export with the appropriate script.</p> +<p>To create a new version of the file, make any necessary changes to Kibana (e.g. on full-dev), and export with the appropriate script.</p> <div class="source"> <div class="source"> @@ -522,7 +526,7 @@ packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/pa <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/other-examples/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/other-examples/index.html b/site/current-book/metron-deployment/other-examples/index.html index 7dd9888..3cf67e2 100644 --- a/site/current-book/metron-deployment/other-examples/index.html +++ b/site/current-book/metron-deployment/other-examples/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Other Example Deployments</title> <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../../metron-deployment/index.html" title="Deployment"> @@ -180,7 +180,7 @@ <i class="icon-chevron-right"></i> Roles</a> </li> - + <li> <a href="../../metron-deployment/vagrant/index.html" title="Vagrant"> @@ -210,20 +210,27 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../metron-sensors/index.html" title="Sensors"> <i class="icon-chevron-right"></i> Sensors</a> </li> + + <li> + + <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -231,7 +238,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../../use-cases/index.html" title="Use-cases"> @@ -276,7 +283,7 @@ <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html b/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html index 62128e1..eeadfe6 100644 --- a/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html +++ b/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – </title> <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../../../metron-deployment/index.html" title="Deployment"> @@ -180,7 +180,7 @@ <i class="icon-chevron-right"></i> Roles</a> </li> - + <li> <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant"> @@ -210,20 +210,27 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../../metron-sensors/index.html" title="Sensors"> <i class="icon-chevron-right"></i> Sensors</a> </li> + + <li> + + <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -231,7 +238,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../../../use-cases/index.html" title="Use-cases"> @@ -310,14 +317,20 @@ <div class="source"> <div class="source"> <pre># cat /dev/zero | ssh-keygen -q -N "" 2>/dev/null +# cd ~/.ssh +# cat id_rsa.pub >> authorized_keys </pre></div></div> -<p>If you’re not installing on a single node, add this newly generated key to all the slave nodes:</p> + +<ul> + +<li>If you’re not installing on a single node, add this newly generated key to all the slave nodes:</li> +</ul> <div class="source"> <div class="source"> <pre>ssh-copy-id -i ~/.ssh/id_rsa.pub <replace_with_node_ip> </pre></div></div> -<p><i>Side note:</i> You might have to adapt your sshd_config file and add “PermitRootLogin yes” amongst other parameters if you want passwordless root access, but that’s outside the scope of this document.</p> +<p><b>Side note:</b> You might have to adapt your sshd_config file and add “PermitRootLogin yes” amongst other parameters if you want passwordless root access, but that’s outside the scope of this document.</p> <ul> @@ -376,6 +389,24 @@ Afterwards, run: # grub-install /dev/sda </pre></div></div> + +<ul> + +<li>If you do not want to mess with grub/kernel parameters, add the following to /etc/rc.local:</li> +</ul> + +<div class="source"> +<div class="source"> +<pre>vim /etc/rc.local: +# Disable THP at boot time +if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then + echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled +fi + +if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then + echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag +fi +</pre></div></div> <p>After reboot check that changes were applied (make sure that word “never” is selected in square-brackets):</p> <div class="source"> @@ -393,7 +424,7 @@ always madvise [never] <div class="source"> <div class="source"> -<pre># yum install git wget curl rpm tar unzip bzip2 wget createrepo yum-utils ntp python-pip psutils python-psutil ntp libffi-devel gcc openssl-devel -y +<pre># yum install git wget curl rpm tar unzip bzip2 wget createrepo yum-utils ntp python-pip psutils python-psutil ntp libffi-devel gcc openssl-devel npm -y # pip install --upgrade pip # pip install requests urllib # pip install --upgrade setuptools @@ -401,7 +432,7 @@ always madvise [never] <ul> -<li>Install Maven on main node and on Metron node install java 1.8 (if you don’t know which it is, install it everywhere):</li> +<li>Install Maven 3.3.9 on main node and on Metron node install java 1.8 (if you don’t know which it is, install it everywhere):</li> </ul> <div class="source"> @@ -433,7 +464,7 @@ always madvise [never] <ul> -<li>Download and install Maven:</li> +<li>Download and install Maven 3.3.9:</li> </ul> <div class="source"> @@ -504,11 +535,14 @@ OS name: "linux", version: "3.10.0-514.16.1.el7.x86_64", arc <ul> -<li>Remove ipv4 ‘localhost.localdomain’ from /etc/hosts</li> +<li> +<p>Remove ipv4 ‘localhost.localdomain’ from /etc/hosts</p></li> -<li>Remove ipv6 ‘localhost.localdomain’ from /etc/hosts</li> +<li> +<p>Remove ipv6 ‘localhost.localdomain’ from /etc/hosts</p></li> -<li>Add “127.0.0.1 localhost” to /etc/hosts</li> +<li> +<p>Add “127.0.0.1 localhost” to /etc/hosts</p></li> <li> <p>Install the database we will use for Metron REST:</p></li> @@ -608,6 +642,14 @@ Cleaning up... <div class="source"> <pre># git clone https://github.com/apache/metron </pre></div></div> +<p>If you want to make sure you’re on the 0.4.0 release branch, do:</p> + +<div class="source"> +<div class="source"> +<pre># git clone https://github.com/apache/metron +# cd metron +# git checkout Metron_0.4.0 +</pre></div></div> <ul> @@ -634,17 +676,19 @@ Cleaning up... # cp -rp /root/metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/* /localrepo/ # createrepo /localrepo </pre></div></div> -<p>If you’re doing a multi node install, also copy the packages to the other nodes:</p> +<p>If you’re doing a multi node install, also create localrepo on the nodes and copy the packages to the other nodes:</p> <div class="source"> <div class="source"> -<pre># scp /localrepo/* <replace_with_node_ip>:/localrepo/ -# createrepo /localrepo +<pre># ssh root@node2 mkdir /localrepo +# scp /localrepo/*\.rpm root@node2:/localrepo/. +# ssh root@node2 yum install createrepo -y +# ssh root@node2 createrepo /localrepo </pre></div></div> <ul> -<li>Make sure to run <tt>createrepo /localrepo</tt> on every node!</li> +<li>Make sure to do the above on each node.</li> </ul> <p>Fetch & create logrotate script for Hadoop Services:</p> @@ -673,7 +717,7 @@ Cleaning up... <ul> -<li>Enable time sync, disable firewall and SElinux:</li> +<li>Enable time sync, disable firewall and SElinux on every node:</li> </ul> <div class="source"> @@ -688,7 +732,7 @@ Cleaning up... <ul> -<li>Disable firewall:</li> +<li>Disable firewall on every node:</li> </ul> <div class="source"> @@ -700,7 +744,7 @@ Cleaning up... <ul> -<li>Disable IPv6 firewall:</li> +<li>Disable IPv6 firewall on every node:</li> </ul> <div class="source"> @@ -712,7 +756,7 @@ Cleaning up... <ul> -<li>Disable SElinux</li> +<li>Disable SElinux on every node:</li> </ul> <div class="source"> @@ -1115,7 +1159,7 @@ Metron JDBC username: <DB USERNAME> <ul> <li> -<p>Install everything. Metron REST will probably not work as we still need to add a user and the database to MySQL.</p></li> +<p>Install everything. Metron REST will probably not work as we still need to add a user and the database to MySQL. At this point, make sure that all the services are up. You might have to manually start a few.</p></li> <li> <p>Configure a user for Metron REST in MySQL. On the node where you installed the Metron REST UI, do:</p></li> @@ -1216,23 +1260,11 @@ Bye # yum update -y # yum install python27 -y # scl enable python27 bash - # cd /opt/rh/python27/root/usr/bin/ # LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./pip2.7 install --upgrade pip # LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./pip2.7 install requests - -(# /opt/rh/python27/root/usr/bin/virtualenv py27venv -# source py27venv/bin/activate -# pip install --upgrade pip -# pip install ansible==2.0.0.2 -# ansible --version -# deactivate) -</pre></div></div> - -<div class="source"> -<div class="source"> -<pre># yum install @Development python-virtualenv libpcap-devel libselinux-python -y +# yum install @Development python-virtualenv libpcap-devel libselinux-python -y # mkdir /usr/local/pycapa # cd /usr/local/pycapa # virtualenv pycapa-venv @@ -1240,7 +1272,6 @@ Bye # cp -r /root/metron/metron-sensors/pycapa/. /usr/local/pycapa/. # pip install --upgrade pip # /usr/local/pycapa/pycapa-venv/bin/pip install -r requirements.txt -(# pip install -r requirements.txt) # /usr/local/pycapa/pycapa-venv/bin/python setup.py install # ln -s /usr/local/lib/librdkafka.so.1 /opt/rh/python27/root/usr/lib64 @@ -1258,7 +1289,6 @@ Bye # sed -i 's/--kafka {{ kafka_broker_url }}/--kafka-broker <IP:6667>/' /etc/init.d/pycapa # sed -i 's/--topic {{ pycapa_topic }}/--kafka-topic pcap/' /etc/init.d/pycapa # sed -i 's/{{ pycapa_sniff_interface }}/tap0/' /etc/init.d/pycapa -(# sed -i 's/export LD_LIBRARY_PATH=\/opt\/rh\/python27\/root\/usr\/lib64/export LD_LIBRARY_PATH=\/usr\/local\/lib/' /etc/init.d/pycapa) # chmod 755 /etc/init.d/pycapa # yum install @Development libdnet-devel rpm-build libpcap libpcap-devel pcre pcre-devel zlib zlib-devel glib2-devel -y # yum install kafka -y @@ -1267,9 +1297,9 @@ Bye <div class="source"> <div class="source"> -<pre># wget -O /tmp/bro-2.4.1.tar.gz https://www.bro.org/downloads/release/bro-2.4.1.tar.gz -# /bin/gtar --extract -C /tmp -z -f /tmp/bro-2.4.1.tar.gz -# cd /tmp/bro-2.4.1 +<pre># wget -O /tmp/bro-2.4.2.tar.gz https://www.bro.org/downloads/bro-2.4.2.tar.gz +# /bin/gtar --extract -C /tmp -z -f /tmp/bro-2.4.2.tar.gz +# cd /tmp/bro-2.4.2 # ./configure --prefix=/usr/local/bro # make -j4 # make install @@ -1292,14 +1322,14 @@ Bye <div class="source"> <div class="source"> -<pre># cp -r /root/metron/metron-sensors/bro-plugin-kafka /tmp -# cd /tmp/bro-plugin-kafka +<pre># git clone https://github.com/apache/metron-bro-plugin-kafka /tmp/metron-bro-plugin-kafka +# cd /tmp/metron-bro-plugin-kafka # rm -rf build/ -# ./configure --bro-dist=/tmp/bro-2.4.1 --install-root=/usr/local/bro/lib/bro/plugins/ --with-librdkafka=/usr/local +# ./configure --bro-dist=/tmp/bro-2.4.2 --install-root=/usr/local/bro/lib/bro/plugins/ --with-librdkafka=/usr/local # make -j4 # make install </pre></div></div> -<p>Configure bro-kafka plugin:</p> +<p>Configure metron-bro-plugin-kafka:</p> <div class="source"> <div class="source"> @@ -1476,7 +1506,11 @@ EOF <ul> -<li>I had a problem with Zeppelin after rebooting this machine and had to manually create the Zeppelin run directory:</li> +<li> +<p>There’s currently a bug in Metron 0.4.0 where Metron REST doesn’t start when restarting the Metron services. This bug was fixed in METRON-990 (<a class="externalLink" href="https://github.com/apache/metron/pull/613/commits/1a9b19a0101ada58cb671ab224934f304df6fff8";>https://github.com/apache/metron/pull/613/commits/1a9b19a0101ada58cb671ab224934f304df6fff8</a>) but unfortunately, this fix didn’t make the 0.4.0 release. In order to fix this, edit the file “/etc/rc.d/init.d/metron-rest” and on line 148, change “$0 start” to “$0 start $2”.</p></li> + +<li> +<p>I had a problem with Zeppelin after rebooting this machine and had to manually create the Zeppelin run directory:</p></li> </ul> <div class="source"> @@ -1583,12 +1617,13 @@ EOF <ul> -<li>Install Apache NiFi. Download nifi-1.2.0-bin.tar.gz from <a class="externalLink" href="https://nifi.apache.org/download.html";>https://nifi.apache.org/download.html</a></li> +<li>Install Apache NiFi in /root (You can pretty much use any directory you want). Download nifi-1.2.0-bin.tar.gz from <a class="externalLink" href="https://nifi.apache.org/download.html";>https://nifi.apache.org/download.html</a></li> </ul> <div class="source"> <div class="source"> -<pre># wget http://apache.mirror.iweb.ca/nifi/1.2.0/nifi-1.2.0-bin.tar.gz +<pre># cd /root +# wget http://apache.mirror.iweb.ca/nifi/1.2.0/nifi-1.2.0-bin.tar.gz # tar xf nifi-1.2.0-bin.tar.gz </pre></div></div> <p>Before we run NiFi, we need to change the port as the default port collides with the Ambari port. To do this, we need to change the value “nifi.web.http.port=8080” to “nifi.web.http.port=8089” in the file “nifi-1.1.2/conf/nifi.properties”. Install and start NiFi afterwards:</p> @@ -1634,7 +1669,7 @@ EOF <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/packaging/ambari/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/ambari/index.html b/site/current-book/metron-deployment/packaging/ambari/index.html index 25df81e..8013381 100644 --- a/site/current-book/metron-deployment/packaging/ambari/index.html +++ b/site/current-book/metron-deployment/packaging/ambari/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Ambari Management Pack Development</title> <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../../../metron-deployment/index.html" title="Deployment"> @@ -171,7 +171,7 @@ <i class="icon-chevron-right"></i> Roles</a> </li> - + <li> <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant"> @@ -201,20 +201,27 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../../metron-sensors/index.html" title="Sensors"> <i class="icon-chevron-right"></i> Sensors</a> </li> + + <li> + + <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -222,7 +229,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../../../use-cases/index.html" title="Use-cases"> @@ -253,6 +260,29 @@ <h1>Ambari Management Pack Development</h1> <p><a name="Ambari_Management_Pack_Development"></a></p> +<div class="section"> +<h2><a name="Table_of_Contents"></a>Table of Contents</h2> + +<ul> + +<li><a href="#Overview">Overview</a></li> + +<li><a href="#Ambari_Zookeeper_Config_Lifecycle">Ambari Zookeeper Config Lifecycle</a></li> + +<li><a href="#Adding_a_new_property">Adding a new property</a></li> + +<li><a href="#How_to_identify_errors_in_MPack_changes">How to identify errors in MPack changes</a></li> + +<li><a href="#Testing_changes_without_cycling_Vagrant_build">Testing changes without cycling Vagrant build</a></li> + +<li><a href="#Configuration_involving_dependency_services">Configuration involving dependency services</a></li> + +<li><a href="#Kerberos">Kerberos</a></li> + +<li><a href="#Best_practices">Best practices</a></li> +</ul></div> +<div class="section"> +<h2><a name="Overview"></a>Overview</h2> <p>Typically, Ambari Management Pack development will be done in the Vagrant environments. These instructions are specific to Vagrant, but can be adapted for other environemnts (e.g. make sure to be on the correct nodes for server vs agent files)</p> <p>There is an <tt>mpack.json</tt> file which describes what services the mpack will contains, versions, etc.</p> <p>Alongside this are two directories, <tt>addon-services</tt> and <tt>common-services</tt>.</p> @@ -359,7 +389,61 @@ <li>Defines the order of service startup and other actions relative to each other.</li> </ul></li> +</ul></div> +<div class="section"> +<h2><a name="Ambari_Zookeeper_Config_Lifecycle"></a>Ambari Zookeeper Config Lifecycle</h2> +<p>The source of truth for Metron configuration resides in Zookeeper as a series of JSON documents. There are actually multiple locations that you can populate the Zookeeper configs from:</p> + +<ul> + +<li>$METRON_HOME/config/zookeeper</li> + +<li>Stellar REPL</li> + +<li>Management UI</li> + +<li>Ambari</li> </ul> +<p>Any change that works with Metron configuration stored in Zookeeper should always pull the latest config from Zookeeper first before making any changes and publishing them back up to avoid potentially overwriting the latest configs with stale data. The lifecycle for configuration management with Ambari is as follows:</p> + +<ul> + +<li>First start/install for parsers, enrichment, indexing, or the profiler + +<ul> + +<li>Perform a PUSH from local disk to Zookeeper</li> + </ul></li> + +<li>All component start/restart operations for parsers, enrichment, indexing, or the profiler + +<ul> + +<li>Perform a JSON patch (RFC 6902) on the configs in Zookeeper and push the patched, pretty formatted config up to Zookeeper. Patching is currently applicable to to global.json only.</li> + +<li>Pull all configs to the local file system.</li> + </ul></li> +</ul> +<p>The patching mechanism we introduced to Ambari for managing the global config is by default additive in nature, though it will also overwrite any existing values with the latest values from Ambari. This means that you can add any properties to the global config outside of Ambari you like, provided you are not modifying values explicitly managed by Ambari. This is important because you will not get any errors when modifying the configuration outside of Ambari. Instead, if you modified es.ip and changed global.json outsided Ambari, you would not see this change in Ambari. Meanwhile, the indexing topology would be using the new value stored in Zookeeper. Remember, Zookeeper is the source of truth. If you were to restart a Metron topology component via Ambari, that es.ip property would now be set back to the value stored in Ambari. So in summary, if it’s managed by Ambari, only change that value via Ambari. If it’s not, then feel free to use whatever mechanism you like, just be sure to pull the latest config from Zookeeper before making any changes. We make no effort to handle race conditions. It’s last commit wins. Below are the global config properties managed by Ambari:</p> + +<ul> + +<li>es.clustername</li> + +<li>es.ip</li> + +<li>es.date.format</li> + +<li>parser.error.topic</li> + +<li>update.hbase.table</li> + +<li>update.hbase.cf</li> + +<li>profiler.client.period.duration</li> + +<li>profiler.client.period.duration.units</li> +</ul></div> <div class="section"> <h2><a name="Adding_a_new_property"></a>Adding a new property</h2> @@ -396,7 +480,7 @@ <div class="source"> <pre> new_property = status_params.new_property </pre></div></div> -<p>This behavior is because Ambari doesn’t send all parameters to the status, so it needs to be explicitly provided.</p> +<p>This behavior is because Ambari doesn’t send all parameters to the status, so it needs to be explicitly provided. Also note that status_params.py parameters are not automatically pulled into the params_linux.py namespace, so we explicitly choose the variables to include. See <a class="externalLink" href="https://docs.python.org/2/howto/doanddont.html#at-module-level";>https://docs.python.org/2/howto/doanddont.html#at-module-level</a> for more info.</p> <ol style="list-style-type: decimal"> @@ -468,7 +552,36 @@ # ... hdfs_grok_patterns_dir = format("{metron_apps_hdfs_dir}/patterns") </pre></div></div> -<p>The <tt>format</tt> function is a special Ambari function that will let you create a string with properties in curly braces replaced by their values. In this case, <tt>hdfs_grok_patterns_dir</tt> will be <tt>/apps/metron/patterns</tt> and will be what we now follow in this example.</p></div> +<p>The <tt>format</tt> function is a special Ambari function that will let you create a string with properties in curly braces replaced by their values. In this case, <tt>hdfs_grok_patterns_dir</tt> will be <tt>/apps/metron/patterns</tt> and will be what we now follow in this example.</p> +<p>Note: There are instances where we deviate a bit from this pattern and use format(format(“{{some_prop}}”)). This is not a mistake. Ambari does not natively support nested properties, and this was a workaround that accomplished being able to initialize values from env.xml files and have them also available to users as Ambari properties later. Here’s an example from params_linux.py:</p> + +<div class="source"> +<div class="source"> +<pre>metron_apps_indexed_hdfs_dir = format(format(config['configurations']['metron-indexing-env']['metron_apps_indexed_hdfs_dir'])) +</pre></div></div> +<p>and the relavant properties in metron-env.xml and metron-indexing-env.xml</p> +<p>metron-env.xml</p> + +<div class="source"> +<div class="source"> +<pre><property> + <name>metron_apps_hdfs_dir</name> + <value>/apps/metron</value> + <description>Metron apps HDFS dir</description> + <display-name>Metron apps HDFS dir</display-name> +</property> +</pre></div></div> +<p>metron-indexing-env.xml</p> + +<div class="source"> +<div class="source"> +<pre><property> + <name>metron_apps_indexed_hdfs_dir</name> + <value>{{metron_apps_hdfs_dir}}/indexing/indexed</value> + <description>Indexing bolts will write to this HDFS directory</description> + <display-name>Metron apps indexed HDFS dir</display-name> +</property> +</pre></div></div></div> <div class="section"> <h4><a name="Importing_the_parameters_into_scripts"></a>Importing the parameters into scripts</h4> <p><tt>hdfs_grok_patterns_dir</tt> is used in <tt>METRON.CURRENT/package/scripts/parser_commands.py</tt>, but before we can reference it, we’ll need the params available to <tt>parser_commands.py</tt></p> @@ -772,7 +885,7 @@ METRON_LOG_DIR="/var/log/metron" <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html b/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html index f2045ae..83d98f6 100644 --- a/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html +++ b/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Overview</title> <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../../../../metron-deployment/index.html" title="Deployment"> @@ -171,7 +171,7 @@ <i class="icon-chevron-right"></i> Roles</a> </li> - + <li> <a href="../../../../metron-deployment/vagrant/index.html" title="Vagrant"> @@ -201,20 +201,27 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../../../metron-sensors/index.html" title="Sensors"> <i class="icon-chevron-right"></i> Sensors</a> </li> + + <li> + + <a href="../../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -222,7 +229,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../../../../use-cases/index.html" title="Use-cases"> @@ -253,7 +260,7 @@ <h1>Overview</h1> <p><a name="Overview"></a></p> -<p>The Metron ansible-docker container is provided in an effort reduce the installation burden of deploying Metron in a live envirionment. It is provisioned with software required to sucessfully run the deployment scripts.</p> +<p>The Metron ansible-docker container is provided in an effort reduce the installation burden of building Metron. It may also be used to deploy Metron in a live environment. It is provisioned with software required to sucessfully build metron and run the deployment scripts.</p> <div class="section"> <h2><a name="Building_the_Container"></a>Building the Container</h2> @@ -266,18 +273,35 @@ <li>Build the container <tt>docker build -t ansible-docker:2.0.0.2 .</tt></li> </ol></div> <div class="section"> -<h2><a name="Using_the_Container"></a>Using the Container</h2> -<p>Full instructions are found on the wiki at <a class="externalLink" href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361";>https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361</a></p> -<p>tl;dr:</p> +<h2><a name="Using_the_Container_to_build_metron"></a>Using the Container to build metron</h2> +<p>anytime after building the container you can run it with the following command</p> +<p><tt>docker run -it -v \<project-directory\>:/root/metron ansible-docker:2.0.0.2 bash</tt></p> +<p>If you are going to build metron multiple times, you may want to map the /root/.m2 maven repo from outside of the container so that you don’t start with an empty repo every build and have to download the world.</p> +<p><tt>docker run -it -v \<project-directory\>:/root/metron -v \<your .m2 directory\>:/root/.m2 ansible-docker:2.0.0.2 bash</tt></p> +<p>After running the container:</p> <ol style="list-style-type: decimal"> -<li>docker run -it -v <project-directory>:/root/metron ansible-docker:2.0.0.2 bash</li> - <li>cd /root/metron</li> -<li>mvn clean package -DskipTests</li> -</ol></div> +<li>run build commands, for example: + +<ul> + +<li>build metron without tests : <tt>mvn clean package -DskipTests</tt></li> + +<li>build metron and build the rpms as well : <tt>mvn clean install && cd metron-deployment && mvn package -P build-rpms</tt></li> + </ul></li> +</ol> +<p>If you wish to use this build with a vagrant instance, then after building with rpms as above, modify your usual vagrant up command to skip the build role, as so:</p> +<p><tt>vagrant --ansible-skip-tags="build" up</tt></p></div> +<div class="section"> +<h2><a name="Using_the_Container_for_deployment"></a>Using the Container for deployment</h2> + +<blockquote> +<p>Note these instructions are outdated</p> +</blockquote> +<p>Full instructions are found on the wiki at <a class="externalLink" href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361";>https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361</a></p></div> </div> </div> </div> @@ -286,7 +310,7 @@ <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html b/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html index c216970..53b2885 100644 --- a/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html +++ b/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Overview</title> <link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../../../../metron-deployment/index.html" title="Deployment"> @@ -171,7 +171,7 @@ <i class="icon-chevron-right"></i> Roles</a> </li> - + <li> <a href="../../../../metron-deployment/vagrant/index.html" title="Vagrant"> @@ -201,20 +201,27 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../../../metron-sensors/index.html" title="Sensors"> <i class="icon-chevron-right"></i> Sensors</a> </li> + + <li> + + <a href="../../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -222,7 +229,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../../../../use-cases/index.html" title="Use-cases"> @@ -273,7 +280,7 @@ <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved. http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/packaging/packer-build/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/packer-build/index.html b/site/current-book/metron-deployment/packaging/packer-build/index.html index d85fa3e..066073c 100644 --- a/site/current-book/metron-deployment/packaging/packer-build/index.html +++ b/site/current-book/metron-deployment/packaging/packer-build/index.html @@ -1,13 +1,13 @@ <!DOCTYPE html> <!-- - | Generated by Apache Maven Doxia at 2017-09-15 + | Generated by Apache Maven Doxia at 2018-01-03 | Rendered using Apache Maven Fluido Skin 1.3.0 --> <html xmlns="http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> - <meta name="Date-Revision-yyyymmdd" content="20170915" /> + <meta name="Date-Revision-yyyymmdd" content="20180103" /> <meta http-equiv="Content-Language" content="en" /> <title>Metron – Build Metron Images</title> <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" /> @@ -61,8 +61,8 @@ - <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li> - <li id="projectVersion" class="pull-right">Version: 0.4.1</li> + <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li> + <li id="projectVersion" class="pull-right">Version: 0.4.2</li> </ul> </div> @@ -103,7 +103,7 @@ <i class="none"></i> Docker</a> </li> - + <li> <a href="../../../metron-deployment/index.html" title="Deployment"> @@ -171,7 +171,7 @@ <i class="icon-chevron-right"></i> Roles</a> </li> - + <li> <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant"> @@ -201,20 +201,27 @@ <i class="none"></i> Rest</a> </li> - + <li> <a href="../../../metron-platform/index.html" title="Platform"> <i class="icon-chevron-right"></i> Platform</a> </li> - + <li> <a href="../../../metron-sensors/index.html" title="Sensors"> <i class="icon-chevron-right"></i> Sensors</a> </li> + + <li> + + <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example"> + <i class="none"></i> + Stellar-3rd-party-example</a> + </li> <li> @@ -222,7 +229,7 @@ <i class="icon-chevron-right"></i> Stellar-common</a> </li> - + <li> <a href="../../../use-cases/index.html" title="Use-cases"> @@ -259,8 +266,6 @@ <ul> <li>base-centos-6.7: Centos 6.7 + HDP. Used in the full-dev-platform Vagrant image</li> - -<li>quick-dev-centos-6.7: Centos 6.7 + HDP + Metron. Used for the quick-dev-platform Vagrant image.</li> </ul></div> <div class="section"> <h2><a name="Prerequisites"></a>Prerequisites</h2> @@ -270,10 +275,14 @@ <li><a class="externalLink" href="https://www.packer.io/";>Packer</a> 0.12.2</li> <li><a class="externalLink" href="https://www.virtualbox.org/";>Virtualbox</a> 5.0.16+ (Tested with 5.0.20)</li> -</ul> -<p>Build Both Images ———————- Navigate to <your-project-directory>/metron-deployment/packer-build Execute bin/bento build</p> -<p>Packer will build both images and export .box files to the ./builds directory.</p> -<p>Build Single Images ———————- Navigate to <i>your-project-directory</i>/metron-deployment/packer-build</p> +</ul></div> +<div class="section"> +<h2><a name="Build_Both_Images"></a>Build Both Images</h2> +<p>Navigate to <your-project-directory>/metron-deployment/packer-build Execute bin/bento build</p> +<p>Packer will build both images and export .box files to the ./builds directory.</p></div> +<div class="section"> +<h2><a name="Build_Single_Images"></a>Build Single Images</h2> +<p>Navigate to <i>your-project-directory</i>/metron-deployment/packer-build</p> <ul> @@ -283,18 +292,10 @@ <div class="source"> <div class="source"> <pre>bin/bento build base-centos-6.7.json -</pre></div></div> - -<ul> - -<li>Quick Dev</li> -</ul> - -<div class="source"> -<div class="source"> -<pre>bin/bento build quick-dev-centos-6.7.json -</pre></div></div> -<p>Using Your New Box File ———————- Modify the relevant Vagrantfile (full-dev-platform or quick-dev-platform) replacing the lines:</p> +</pre></div></div></div> +<div class="section"> +<h2><a name="Using_Your_New_Box_File"></a>Using Your New Box File</h2> +<p>Modify the relevant Vagrantfile (full-dev-platform) replacing the lines:</p> <div class="source"> <div class="source"> @@ -319,7 +320,7 @@ config.ssh.insert_key = true</code></pre> <footer> <div class="container-fluid"> - <div class="row span12">Copyright © 2017 + <div class="row span12">Copyright © 2018 <a href="https://www.apache.org";>The Apache Software Foundation</a>. All Rights Reserved.
